How to Use Selenium and Cypress for Canvas Automation

The Canvas element in HTML (or ) is majorly used for rendering graphics, animations, and other interactive content on a web page. Though the Canvas element is a part of the DOM, content inside the Canvas is not a part of it! This essentially means that the traditional DOM-based validations do not hold well for the automated handling of Canvas elements.

Over & above, automating Canvas interactions with Cypress is a tad easier than Selenium. This is because Cypress runs within the browser’s context, providing you seamless access to JavaScript APIs like the Canvas API. In this blog, we will be covering using Selenium (Python) and Cypress for Canvas element automation with hands-on examples.

What Is Canvas Element in HTML?

The Canvas () element is used for drawing graphics or other visual content on-the-fly on a web page. Along with animations, the Canvas element can also be used for drawing charts, animations, and other custom visualizations on the page.

The Canvas element was introduced as a part of the HTML5 standard for reducing the dependencies on plugins like Adobe Flash and Microsoft SilverLight, both of which have already reached EOL (End Of Life).

The element, which is essentially a blank container, heavily relies on JavaScript for drawing any graphics and performing interactive actions. As stated in the Canvas API Documentation, the Canvas APIs primarily focus on 2D graphics. WebGL (Web Graphics Library), a JavaScript API for rendering high-performance interactive 2D & 3D graphics, also makes use of the element.

Example 1:

As shown below, graphics in the LambdaTest Bug Smasher Game are housed in the *element. After locating the Canvas element using the ID — *document.getElementById(“unity-canvas”), we get the details (i.e., dimensions/size) of the said element.

The above command, when run on the Browser console, provides the following output:

<canvas id="unity-canvas" width="1920" height="1200" style="width: 960px; height: 600px; cursor: default;"></canvas>

The Canvas element is identified using the ID attribute (i.e., unity-canvas)
The width=”1920″ and *height=”1200″ *attributes define the actual size of the Canvas element used for drawing/rendering
style=”width: 960px; height: 600px;” *defines the display size of the Canvas element on the page. In this case, a scaling factor of 0.5 (*width = 960/1920 and height = 600/1200) is used when locating elements in the Canvas via its coordinates

Example 2:

Another example of the Canvas element is a rendering of a Bar Graph in CanvasJS. After locating the Canvas element using Selenium locators document.getElementById(“themes-chart”), we get the details (i.e., dimensions/size) of the said element.

The above command, when run on the browser console provides the following output:

<div id="themes-chart" style="width: 100%; height: 441px;"><div class="canvasjs-chart-container" style="position: relative; text-align: left; cursor: auto; direction: ltr;"><canvas class="canvasjs-chart-canvas" width="1892" height="882" style="width: 946px; height: 441px; position: absolute; user-select: none;"></canvas>...</div>

The Canvas element is identified using the ID attribute (i.e., themes-cart)
The width=”1892″ and *height=”882″ *attributes define the actual size of the Canvas element used for drawing/rendering
style=”width: 946px; height: 441px;” *defines the display size of the Canvas element on the page. In this case, a scaling factor of 0.5 (*width = 946/1892 and height = 441/882) is used when locating elements in the Canvas via its coordinates

As seen from the above examples, the internal resolution and display size difference are different (i.e., scaling — 0.5) and can affect the quality of the rendering. When automating interactions with Canvas elements, it is important to take the scaling into account to ensure the accuracy of mouse movement actions.

In this blog, I will automate Canvas elements on the local Chrome browser and/or Chrome installed on the LambdaTest cloud grid.

LambdaTest is an AI-based test orchestration and execution platform that allows automated tests to be run at scale in more than 3,000 environments. It also supports automating Canvas elements on multiple browsers using popular frameworks like Selenium and Cypress.

The getBoundingClientRect() Method

Though the element is a part of the DOM, the elements rendered inside it are not represented in the DOM. This makes automating Canvas challenging, as the elements within the DOM have to be located using the co-ordinates.

This is where the getBoundingClientRect() method of the Element class can be leveraged, as it provides the size of the element and its properties relative to the viewport. The *getBoundingClientRect() *method returns a *DOMRect *object that contains the following properties (in pixels):

Shown below is the pictorial view of the BoundingRect() method that deep dives in the approach used for calculating the properties:

When automating interactions with the Canvas element, set a consistent viewport size (e.g., 1024 * 768) and maintain a constant zoom-level (100% preferred) for reliable automation results.

Let’s fetch the DOMRect properties, by executing the *document. getElementById(“unity-canvas”).getBoundingClientRect *command on the browser console. Shown below is the command in action for the LambdaTest BugSmasher game:

Now, let’s resize the browser window and check the properties contained in the DOMRect object:

As seen from the above screenshots, certain DOMRect properties, namely X, Y, top, right, bottom, and left, change with a resized browser window. Like other Selenium best practices, it is recommended to instantiate the respective browser in maximized mode so that the DOMRect *properties (i.e., *top, right, bottom, and left) are consistent across subsequent runs.

To summarize, the getBoundingClientRect() method plays a pivotal role in handling Canvas elements by providing accurate size and coordinates relative to the viewport.

Tracking Mouse Movements in Canvas Elements

As seen so far, interaction with Canvas content is based on precise coordinates rather than element-based tracking (which is normally used with DOM-based interactions). This is precisely why tracking mouse positions becomes essential to ensure that automation mimics real user behavior.

During the process of writing this blog, I tried automating different Canvas element scenarios with Selenium as well as Cypress. Since Cypress tests run directly in the browser, I found it relatively easy to use Cypress for Canvas automation.

As far as automation with Selenium is concerned, this insightful StackoverFlow thread helped a lot in calculating Canvas element coordinates more robustly. As stated in the W3C specification, the offset of action commands is calculated from the center of a WebElement.

Regardless of the automation framework used, such as Selenium or Cypress for Canvas automation, it is imperative to track mouse movements so that coordinates can be adjusted as needed.

I used the prompt “*Show a dot wherever the mouse pointer is.” *This code will be used for Selenium and Cypress for Canvas automation and fed into the AI tool that assisted me with code development.

Here is the AI-generated code that will show a Red (#FF0000) dot with every mouse movement.

/* Customized command to show the click on the element */
Cypress.Commands.add('showClickCoordinates', (element, x, y) => {
 cy.wrap(element).then(($el) => {
   // Reference - https://developer.mozilla.org/en-US/docs/Web/API/Element/getBoundingClientRect
   const rect = $el[0].getBoundingClientRect();


   cy.window().then((win) => {
     // Create a new dot element
     const dot = win.document.createElement('div');
     dot.style.position = 'absolute';
     dot.style.width = '10px';
     dot.style.height = '10px';
     dot.style.backgroundColor = 'red';
     dot.style.borderRadius = '50%';
     dot.style.zIndex = '9999';


     // Adjust the top and left coordinates to be relative to the element
     dot.style.top = `${rect.top + y}px`;
     dot.style.left = `${rect.left + x}px`;


     // Append the dot to the body
     win.document.body.appendChild(dot);


     // Remove the dot after 3 seconds
     setTimeout(() => {
       dot.remove();
     }, 3000);
   });
 });
});

Here is a short gist of the above AI-generated snippet, the primary purpose of which is to position a Red (#FF0000) colored dot relative to an element in the Canvas. Let’s dive deep into the essential aspects of the snippet:

Step 1. Cypress.Commands.add adds a Cypress custom command (named showClickCoordinates) that can be reused throughout the tests. It takes three input parameters:

element: DOM element where the click needs to be highlighted
x: X-coordinate calculated relative to the element’s bounding box
y: Y-coordinate calculated relative to the element’s bounding box

Step 2: getBoundingClientRect() provides the element’s size and position relative to the viewport. For the current use case, we extract the top, left, width, and height properties of the said element.

Step 3: cy.window() helps provide access to the browser’s window object (i.e., win)

Step 4: This is the crux of the code where win.document.createElement helps in creating a new div element in the DOM (to act as a dot). Here is the styling of the dot:

position: absolute: Places the dot at an exact mouse position
width *and *height: Defines the size of the dot
backgroundColor: ‘red’: Makes the dot visible as red (#FF0000)
borderRadius: ‘50%’: Makes the dot a circle
zIndex: ‘9999’: Ensures the dot appears above other elements

Step 5: The top (rect.top) and left (rect.left) styles are calculated to position the dot relative to the element in the viewport. Here, x and y are the coordinates offset inside the element.

The dot is finally appended to the body of the document, making it visible on the page.

Step 6: dot.remove() removes the displayed dot to clean up the DOM after a set timeout of 3 seconds.

Shown below is a sample use of the Mouse Movement tracking code when used for automating Canvas elements in the LambdaTest Bug Smasher game:

We will use the above AI-generated code with Selenium and Cypress for Canvas automation examples, which will be demonstrated in later sections of the blog. AI can help speed up code generation and streamline workflows, allowing QA’s to leverage its benefits by generating tests using AI. Since Canvas interactions are heavily based on coordinates, you could also use Chrome/Brave extensions like Page Ruler to measure distances (in pixels) on a webpage.

Now, let’s get our hands dirty with some hands-on experiments with Selenium and Cypress or Canvas automation!

Using Cypress for Canvas Automation

For this demonstration, we will automate the play of LambdaTest Bug Smasher. If you are new to Cypress on the LambdaTest cloud, you can check out the blog on how to write your first Cypress test, which dives into the nuances of Cypress execution on LambdaTest.

Canvas automation demonstration should work seamlessly on Cypress LambdaTest cloud as well as on Cypress installed on your local machine. In both cases, Chrome is chosen as the browser for the test. The configuration for executing the Canvas tests on LambdaTes

Author Of article : himanshuseth004 Read full article

How to Use Selenium and Cypress for Canvas Automation

What Is Canvas Element in HTML?

The getBoundingClientRect() Method

Tracking Mouse Movements in Canvas Elements

Using Cypress for Canvas Automation

Read Next

Integrations: From Simple Data Transfer To Modern Composable Architectures

Injecting Environment Variables in a Deployed React App Without Rebuilding.