Hello everyone! I’m Noam, an SDK developer on the Applitools JS-SDKs team. While my day-to-day focus is on core engineering, I work closely with our field teams and occasionally join technical deep-dive sessions with customers.

In these conversations, we frequently encounter questions about performance and the engineering philosophy behind our integration. Specifically, there is often curiosity about how to make visual testing feel more “Playwright-native” and natural to developers.

In this post, I’ll share the design logic behind these architectural choices so you can apply these patterns in your own CI pipelines in a way that fits your organization’s needs.

Adding unresolved to Playwright

Integrating visual regression testing into Playwright requires combining two different status models: Playwright’s binary Pass/Fail and the visual testing concept of unresolved.

In visual testing, instead of having two (passed and failed) states, there’s an additional third state: unresolved. This state indicates a difference was detected, but a human decision is required to determine if it is a bug or a valid change that should be approved as a new baseline.

​Playwright doesn’t support this third state out of the box. Visual test maintenance using Playwright’s native toHaveScreenshot API forces the developer into a cumbersome cycle requiring three separate test executions:

1. First, the developer needs to run to see the failure.
2. Then, they need to run with the --update-snapshots flag to create new baseline images.
3. Then, most developers would run again to validate that everything works with the updated baseline as expected—which isn’t always the case, because the Playwright native comparison method (pixelmatch) tends to be very flaky, unlike Visual AI.

​After this local cycle, the developer must commit the new baseline images to the repository—bloating the git history—and wait for a new CI execution to provide final feedback. For dev-centered organizations that focus on feedback loop velocity, this workflow is… suboptimal. Personally, I believe that’s one of the reasons visual testing isn’t as popular as it should be among Playwright users.

​When we engineered the Applitools fixture, one of our goals was to support this Unresolved state natively, without disrupting Playwright’s core lifecycle—specifically its Worker Processes and Retry mechanisms.

The solution rests on two key engineering decisions: moving rendering to the background (async architecture) and giving developers control over the exit signal and performance tradeoffs (failTestsOnDiff).

We don’t block test execution when Applitools is rendering

The core value of visual testing lies in AI-based comparison to eliminate false positives and multi-platform rendering.

Architecturally, these processes are cloud-native services.

• AI-as-a-Service: Just like massive LLMs or other generative models, the Visual AI engine runs on specialized cloud infrastructure optimized for heavy inference. It cannot simply be “installed” on a lightweight CI agent.
• Platform Constraints: Authentic cross-platform rendering (e.g., iOS Safari on a Linux CI agent) is physically impossible on a single local machine.

Since these operations inherently occur remotely, performing them synchronously would force the local test runner to idle while waiting for network round-trips and cloud processing.

To solve this, we designed the fixture around an asynchronous architecture:

• ​Instant Capture: When eyes.check() is called, we synchronously capture the DOM and CSS resources (instead of a rasterized image). This operation is extremely fast.
• ​Immediate Release: We purposefully use soft assertions by design. We release the Playwright test thread immediately so the functional logic can proceed to the next step or test case without blocking.
• ​Background Heavy Lifting: The heavy work—uploading assets, rendering across different browsers and operating systems, and performing the AI comparison in the Applitools cloud—starts immediately in the background, managed by the Worker process.

The “Draining Queue” Effect

​This architecture explains why the Playwright Worker sometimes remains active after the final test completes.

​The background tasks are limited only by your account’s concurrency settings, and the screenshot size. For example, when rendering a 10,000 px page on a small mobile device, the rendering infrastructure might need time for scrolling and stitching. If your functional tests execute faster than the background workers can process the queue (rendering & comparing), the Worker process stays alive at the end solely to “drain the queue” and ensure data integrity.

While it does ensure your test logic runs at maximum speed, offloading the processing cost to the background, this experience might cause friction and frustration as the developers see that workers are “hanging” after tests are completed. When facing such issues, our support team is here to advise and assist with various solutions—we can investigate execution logs and if needed even make custom suggestions to tailor Eyes-Playwright to your needs.

Solving the Matrix Problem

​Standard Playwright documentation recommends defining multiple projects in playwright.config.ts to cover different browsers (Chromium, Firefox, WebKit) and various viewport sizes.

​While this ensures coverage, it introduces a linear performance penalty (O(N)). To test three browsers across two viewports, your CI must execute the functional logic (clicks, waits, navigation) six times. It’s 6x more load on the CI machine and the testing environment.

​We recommend shifting this workload to the Ultrafast Grid (UFG).

​In this mode, you execute the Playwright test once, typically on Chromium. We upload the DOM state, and our cloud infrastructure renders that state across all configured browsers and viewports in parallel.

This transforms an O(N) execution problem into an O(1) execution problem, significantly shortening the feedback loop.

The Strategy: failTestsOnDiff

​Since the actual comparison happens asynchronously and potentially completes after the test logic finishes, we need a mechanism to map the visual result back to the Playwright status.

​This is controlled by the failTestsOnDiff flag. It’s not just a boolean; it’s a strategic choice for your CI pipeline.

​Strategy A—Recommended for CI: failTestsOnDiff: false

• ​The Logic: This is the configuration our own Front-End team uses. We believe that Visual Change ≠ Test Failure.
• ​Behavior: The Playwright test passes (Green). The unresolved status is reported externally via our SCM integration (GitHub/GitLab).
• ​Why: Retrying a visual test is computationally wasteful—the pixels won’t change on the second run. By keeping the test “Green,” we avoid triggering Playwright’s retry mechanism. The decision is moved to the Pull Request, where it belongs.

Read more about SCM integration or hop directly to our GitHub, Bitbucket, Gitlab or Azure Devops articles.

Strategy B—Recommended for Strict Pipelines: failTestsOnDiff: 'afterAll'

• ​The Logic: You need a “Red” pipeline to block deployment, but you want to avoid the noise of retries and gain a significant performance improvement.
• ​Behavior: Individual tests pass, but the Worker Process exits with a failure code if any diffs were found in the suite.
• ​Why: This provides a hard gatekeeper for the build status. It allows the Eyes rendering farms to continue processing visual test results in the background without blocking the execution thread, allowing the worker to move on to handle more tests efficiently.

​Strategy C—Recommended for Local Debugging: failTestsOnDiff: 'afterEach'

• ​The Logic: Immediate feedback loop.
• ​Behavior: Fails the test immediately in the afterEach hook.
• ​Why: Best for local development where you want to see the failure immediately in the console. It is also useful if you use the trace: retainOnFailure setting in Playwright, as it ensures traces are preserved for unresolved visual assertions. Not recommended for CI due to the retry loops described above.

TL;DR – When to use each setting

Closing the Visibility Gap: The Custom Reporter

​If you adopt Strategy A (false) or Strategy B (afterAll), you introduce a secondary challenge: Visibility.

Since Playwright technically marks these tests as Passed to avoid retries, the standard Playwright HTML Report will show them as “Green,” potentially masking unresolved visual differences that require attention.

​To bridge this gap without forcing developers to switch context, we developed a Custom Applitools Reporter.

​This reporter extends the standard Playwright HTML report. It injects the actual visual status (Passed, Failed, or unresolved) directly into the test results view.

• ​True Status: You see which tests have visual diffs, even if the Playwright exit code was successful.
• ​Direct Links: It provides a direct link from the test report to the specific batch results in the Applitools Dashboard.
• ​Context: It enriches the report with UFG render status and batch information.

​This ensures you get the best of both worlds: The optimization of a “Green” CI run (no retries), with the transparency of a report that highlights exactly where manual review is needed.

Summary

​The Applitools Playwright fixture is designed to be non-blocking and scalable. By leveraging asynchronous architecture and Applitools UltraFast Grid, we offload the heavy lifting from your CI. By correctly configuring failTestsOnDiff, you ensure that your pipeline reflects your team’s engineering culture—whether that’s strict gating or modern, PR-based visual review.

The post Engineering a Playwright-Native Developer Experience: One Flag, Three Strategies appeared first on AI-Powered End-to-End Testing | Applitools.

Have you ever encountered a situation where you try to take a screenshot, but instead of the beautifully well-crafted UI, all you’ve got is an image of a spinner/skeleton/loading screen? Handling animations and loading artifacts in visual testing can be daunting.

Don’t worry – it can happen to anyone, and we’re not here to judge you

One of the SDK engineers here at Applitools, Noam, breaks this down into a hands-on tutorial, hoping it will help you get a better understanding of the industry’s best practices around visual testing of rich and dynamic UI experiences.

Let’s dive right in!

Framework Native Solutions

Most frameworks already have different mechanisms to handle animations and loading artifacts. Keeping things simple is often the best way to achieve code stability and maintainability. Using your framework’s built-in tools would most often be the best approach.

For example:

Playwright JS

// Playwright: wait for spinner to be removed
await page.waitForSelector('.spinner', { state: 'detached' });
await eyes.check()

Cypress

// Cypress: wait for spinner to not exist
cy.get('.spinner').should('not.exist'); 
cy.eyesCheck()

Selenium JS

// Selenium: wait for spinner to be invisible
const spinnerElements = await driver.findElements(By.css('.spinner'));
if (spinnerElements.length) {
  await driver.wait(until.elementIsNotVisible(spinnerElements[0]), 5000);
}
await eyes.check()

A Common Pitfall

Even if the UI appears visually unchanged, frontend frameworks like React, Vue, and Angular may re-render elements under the hood. This can lead to stale element references, especially when capturing regions right after a DOM change.

Consider the following example:

cy.get('.main').then($el => {
  cy.get('.spinner').should('not.exist'); // spinner disappears after main was located

  cy.eyesCheckWindow({
    tag: 'main',
    target: 'region',
    element: $el, // stale reference if .main was replaced
  });
});

1. First, Cypress locates .main
2. Then, Cypress waits for the spinner to disappear
3. This example would fail (even if the new element has the exact same properties) if the main element is replaced by another element

How to avoid that?

When possible (e.g., Playwright), it’s preferred that you use locators instead of selectors. If you can’t, it’s better if you use selectors instead of DOM references (element: ‘.main’).

Videos, CSS Animations, GIFs

There are many techniques to eliminate other types of dynamic behaviour in web pages. Playwright, for example, provides a Clock API that allows pausing JavaScript time-related events (including JS-driven animations). It’s also possible to install custom CSS snippets to pause and reset CSS-related animations. Other JS specialized crafted snippets would be required for resetting GIFs, videos, and so on – you get the idea.

This never-ending cat-and-mouse game can be prevented by using Applitools Ultrafast Grid (UFG). Instead of rendering web pages on locally executed browsers, the UFG team maintains specialized logics and fine-tuned commands that ensure a stable and consistent rendering experience. While UFG offers more than just rendering stability, it’s worth noting that classic screenshots can still achieve stable results. UFG just makes it easier!

Algo-Based Solutions

If you intentionally want to capture dynamic content (e.g., animations, changes), a smarter strategy is to embrace that variability and use smart matching algorithms to compare just what you need, like those found in Applitools Eyes.

Any match level can be used for the entire screenshot or specific regions of the screen. Read more in the Match Level Best Practices tutorial. For example, algorithms like the Layout match level can drastically improve your experience with localization testing.

The waitBeforeCapture Setting

Performing wait operations can become more complicated when:

1. Testing with no-code visual testing SDKs (e.g eyes-storybook)
2. Testing with advanced Eyes features like lazyLoading and layoutBreakpoints

The waitBeforeCapture setting was invented for these types of use cases (and a few others).

This setting can receive three types of arguments:

1. Milliseconds – the simplest approach. While it’s not always the most innovative or sophisticated pattern, in many cases, it “does the trick.” In general, waiting for explicit timeouts during tests is not recommended. However, when compared to clock manipulations and code injections, sometimes the simplicity and stability is worth the longer run-time.
2. Selector – when we’re waiting for something to appear, most SDKs support passing a selector, and Applitools Eyes will automatically wait for an element that matches this selector to appear in the web page.
3. Custom function – see code example

// eyes-storybook
  waitBeforeCapture: async () => {
    while (document.querySelector('.spinner')) {
      await new Promise((resolve) => setTimeout(resolve, 100))
    }
    return true;
  }

// eyes-playwright
await eyes.check({
  name: 'my-step',
  async waitBeforeCapture() {
    await page.locator('.spinner').waitFor({state: 'hidden'})
  },
})

The waitBeforeCapture setting can be defined in your applitools.config file, in your eyes.check settings, using the Target settings builder, and in other similar places. Please refer to the documentation of the specific SDK you’re using for concrete examples.

Storybook Play Functions

A nice eyes-storybook-specific trick to achieve a desired rendering state would be a Storybook Play Function.

Applitools Eyes will run your play functions and wait for them to finish before capturing anything on the screen. Use Play Functions to navigate the story to an interesting state and wait for the story to be stable inside the play function to help Eyes understand what the best time is to capture the screenshot.

Applitools is Here to Help

We hope you’ve found this article interesting, and maybe it solved some of the most common visual testing issues you may have encountered. Go ahead and try these examples out for free with Applitools Eyes.

However, if something isn’t clear or if you’d like advice regarding the best way to incorporate visual testing into your organization, please don’t hesitate to reach out to our experts! Testing is our passion, and we’re here to help.

Quick Answers

The post Handling Animations and Loading Artifacts in Visual Testing appeared first on AI-Powered End-to-End Testing | Applitools.

Understanding the problem

In recent months, we improved Applitools Eyes’ ability to perform visual testing on images which are semi-transparent. In the past, Eyes would test an entire screen or defined region, but now using the Storybook SDK, users can automatically test each component separately, without needing to define a test for each component.

For example, when testing a gallery component, Eyes can identify visual bugs and regressions over all screen elements, including the appearance of buttons, controls, fonts, shadows, images, as well as backgrounds that include a transparency gradient.

Figure 1: Transparent Background

After implementing the transparency feature, a visual bug was reported.  In a screen capture of a semi-transparent screen region, unexpected grid lines appeared on top of the tested image.

The root cause of these lines wasn’t clear, so as a first step, we developed a test plan to reproduce the issue. I created a semi-transparent image, all gray (rgb = 127,127,127), with a constant alpha (transparency) channel (alpha=0.5). Fortunately, the bug was easily reproduced and I managed to create easily identifiable grid lines:

As I experimented with different transparency settings, it was clear that the color of the grid lines was the same as the color of the image, and it became stronger as the image transparency was lower.

After further investigation, I discovered that the image viewer component uses tiles to represent large images, and the tiles had one pixel overlap. In the past, when all images were RBG images with no transparency, the overlapping pixel was not visible to the human eye.  When I added semi-transparency, as the adjacent tiles were stacked on top of each other, sampling the outcoming color inside the grid line produced a 192 grayscale value, which is the exact outcome of stacking two half transparent gray layers over a white background:

(white ⋅ 0.5 + gray ⋅ 0.5) ⋅ 0.5 + gray ⋅ 0.5
given (white:=255, gray:=128)

Solving the preliminary bug

To resolve this bug, I recalculated the position and scale of each region so that there would not be an overlap and the line between regions would not appear.
For example, if the first tile had width: 480px and left: 0, the next adjacent tile should be positioned using left: 480, so that there is a zero pixel overlap between tiles.

I tested the results on my local (Linux) machine and assumed that the issue was resolved.

I didn’t realize that when I fixed this bug, I had also created a new issue which would have been almost impossible to anticipate.

How UFG identified the bug I created before deployment

At Applitools, we understand the importance of quality visual testing across browsers, so before deployment, every code change that impacts the Eyes interface must be tested by Applitools Eyes using UFG.

We are proud to “eat our own dogfood.” We rely on our visual testing tools to make sure that our products are visually perfect before release.

Our integration pipeline is configured to use UFG to test the UI change on multiple devices and screen settings so that we can confirm that the interface is consistent on every browser, operating system, and screen size.

We discovered that fixing the bug of a one pixel overlap created a new bug on certain systems where there was a gap visible between tiles.  Frustratingly, this bug was not reproducible in any of the devices used in development, and could not have been discovered with conventional manual visual testing.

The bug was only visible on screens with Retina Display which uses HiDPI.

What was interesting about this bug, is that it highlighted an inconsistency in the way that the same browser (Chrome) displays the same UI on different screen types.

What happened?

The post How We Identified and Resolved a Bug Before Release Using Applitools Ultrafast Grid appeared first on AI-Powered End-to-End Testing | Applitools.