Hello everyone! I’m Noam, an SDK developer on the Applitools JS-SDKs team. While my day-to-day focus is on core engineering, I work closely with our field teams and occasionally join technical deep-dive sessions with customers.

In these conversations, we frequently encounter questions about performance and the engineering philosophy behind our integration. Specifically, there is often curiosity about how to make visual testing feel more “Playwright-native” and natural to developers.

In this post, I’ll share the design logic behind these architectural choices so you can apply these patterns in your own CI pipelines in a way that fits your organization’s needs.

Adding unresolved to Playwright

Integrating visual regression testing into Playwright requires combining two different status models: Playwright’s binary Pass/Fail and the visual testing concept of unresolved.

In visual testing, instead of having two (passed and failed) states, there’s an additional third state: unresolved. This state indicates a difference was detected, but a human decision is required to determine if it is a bug or a valid change that should be approved as a new baseline.

​Playwright doesn’t support this third state out of the box. Visual test maintenance using Playwright’s native toHaveScreenshot API forces the developer into a cumbersome cycle requiring three separate test executions:

1. First, the developer needs to run to see the failure.
2. Then, they need to run with the --update-snapshots flag to create new baseline images.
3. Then, most developers would run again to validate that everything works with the updated baseline as expected—which isn’t always the case, because the Playwright native comparison method (pixelmatch) tends to be very flaky, unlike Visual AI.

​After this local cycle, the developer must commit the new baseline images to the repository—bloating the git history—and wait for a new CI execution to provide final feedback. For dev-centered organizations that focus on feedback loop velocity, this workflow is… suboptimal. Personally, I believe that’s one of the reasons visual testing isn’t as popular as it should be among Playwright users.

​When we engineered the Applitools fixture, one of our goals was to support this Unresolved state natively, without disrupting Playwright’s core lifecycle—specifically its Worker Processes and Retry mechanisms.

The solution rests on two key engineering decisions: moving rendering to the background (async architecture) and giving developers control over the exit signal and performance tradeoffs (failTestsOnDiff).

We don’t block test execution when Applitools is rendering

The core value of visual testing lies in AI-based comparison to eliminate false positives and multi-platform rendering.

Architecturally, these processes are cloud-native services.

• AI-as-a-Service: Just like massive LLMs or other generative models, the Visual AI engine runs on specialized cloud infrastructure optimized for heavy inference. It cannot simply be “installed” on a lightweight CI agent.
• Platform Constraints: Authentic cross-platform rendering (e.g., iOS Safari on a Linux CI agent) is physically impossible on a single local machine.

Since these operations inherently occur remotely, performing them synchronously would force the local test runner to idle while waiting for network round-trips and cloud processing.

To solve this, we designed the fixture around an asynchronous architecture:

• ​Instant Capture: When eyes.check() is called, we synchronously capture the DOM and CSS resources (instead of a rasterized image). This operation is extremely fast.
• ​Immediate Release: We purposefully use soft assertions by design. We release the Playwright test thread immediately so the functional logic can proceed to the next step or test case without blocking.
• ​Background Heavy Lifting: The heavy work—uploading assets, rendering across different browsers and operating systems, and performing the AI comparison in the Applitools cloud—starts immediately in the background, managed by the Worker process.

The “Draining Queue” Effect

​This architecture explains why the Playwright Worker sometimes remains active after the final test completes.

​The background tasks are limited only by your account’s concurrency settings, and the screenshot size. For example, when rendering a 10,000 px page on a small mobile device, the rendering infrastructure might need time for scrolling and stitching. If your functional tests execute faster than the background workers can process the queue (rendering & comparing), the Worker process stays alive at the end solely to “drain the queue” and ensure data integrity.

While it does ensure your test logic runs at maximum speed, offloading the processing cost to the background, this experience might cause friction and frustration as the developers see that workers are “hanging” after tests are completed. When facing such issues, our support team is here to advise and assist with various solutions—we can investigate execution logs and if needed even make custom suggestions to tailor Eyes-Playwright to your needs.

Solving the Matrix Problem

​Standard Playwright documentation recommends defining multiple projects in playwright.config.ts to cover different browsers (Chromium, Firefox, WebKit) and various viewport sizes.

​While this ensures coverage, it introduces a linear performance penalty (O(N)). To test three browsers across two viewports, your CI must execute the functional logic (clicks, waits, navigation) six times. It’s 6x more load on the CI machine and the testing environment.

​We recommend shifting this workload to the Ultrafast Grid (UFG).

​In this mode, you execute the Playwright test once, typically on Chromium. We upload the DOM state, and our cloud infrastructure renders that state across all configured browsers and viewports in parallel.

This transforms an O(N) execution problem into an O(1) execution problem, significantly shortening the feedback loop.

The Strategy: failTestsOnDiff

​Since the actual comparison happens asynchronously and potentially completes after the test logic finishes, we need a mechanism to map the visual result back to the Playwright status.

​This is controlled by the failTestsOnDiff flag. It’s not just a boolean; it’s a strategic choice for your CI pipeline.

​Strategy A—Recommended for CI: failTestsOnDiff: false

• ​The Logic: This is the configuration our own Front-End team uses. We believe that Visual Change ≠ Test Failure.
• ​Behavior: The Playwright test passes (Green). The unresolved status is reported externally via our SCM integration (GitHub/GitLab).
• ​Why: Retrying a visual test is computationally wasteful—the pixels won’t change on the second run. By keeping the test “Green,” we avoid triggering Playwright’s retry mechanism. The decision is moved to the Pull Request, where it belongs.

Read more about SCM integration or hop directly to our GitHub, Bitbucket, Gitlab or Azure Devops articles.

Strategy B—Recommended for Strict Pipelines: failTestsOnDiff: 'afterAll'

• ​The Logic: You need a “Red” pipeline to block deployment, but you want to avoid the noise of retries and gain a significant performance improvement.
• ​Behavior: Individual tests pass, but the Worker Process exits with a failure code if any diffs were found in the suite.
• ​Why: This provides a hard gatekeeper for the build status. It allows the Eyes rendering farms to continue processing visual test results in the background without blocking the execution thread, allowing the worker to move on to handle more tests efficiently.

​Strategy C—Recommended for Local Debugging: failTestsOnDiff: 'afterEach'

• ​The Logic: Immediate feedback loop.
• ​Behavior: Fails the test immediately in the afterEach hook.
• ​Why: Best for local development where you want to see the failure immediately in the console. It is also useful if you use the trace: retainOnFailure setting in Playwright, as it ensures traces are preserved for unresolved visual assertions. Not recommended for CI due to the retry loops described above.

TL;DR – When to use each setting

Closing the Visibility Gap: The Custom Reporter

​If you adopt Strategy A (false) or Strategy B (afterAll), you introduce a secondary challenge: Visibility.

Since Playwright technically marks these tests as Passed to avoid retries, the standard Playwright HTML Report will show them as “Green,” potentially masking unresolved visual differences that require attention.

​To bridge this gap without forcing developers to switch context, we developed a Custom Applitools Reporter.

​This reporter extends the standard Playwright HTML report. It injects the actual visual status (Passed, Failed, or unresolved) directly into the test results view.

• ​True Status: You see which tests have visual diffs, even if the Playwright exit code was successful.
• ​Direct Links: It provides a direct link from the test report to the specific batch results in the Applitools Dashboard.
• ​Context: It enriches the report with UFG render status and batch information.

​This ensures you get the best of both worlds: The optimization of a “Green” CI run (no retries), with the transparency of a report that highlights exactly where manual review is needed.

Summary

​The Applitools Playwright fixture is designed to be non-blocking and scalable. By leveraging asynchronous architecture and Applitools UltraFast Grid, we offload the heavy lifting from your CI. By correctly configuring failTestsOnDiff, you ensure that your pipeline reflects your team’s engineering culture—whether that’s strict gating or modern, PR-based visual review.

The post Engineering a Playwright-Native Developer Experience: One Flag, Three Strategies appeared first on AI-Powered End-to-End Testing | Applitools.

Why Test Maintenance Breaks at Scale

Test maintenance at scale slows releases. Teams that rely on coded assertions spend more time updating tests than improving coverage. Brittle locators, environment drift, and false positives all add up—turning automation into a maintenance cycle.

Neglecting maintenance is like skipping car care: small issues snowball into costly downtime. A smarter approach replaces manual review and locator-based scripts with automated, visual validation that adapts as your UI evolves.

How Visual AI Delivers Test Maintenance at Scale

Visual AI replaces dozens of coded assertions with a single checkpoint that mimics how humans see. It validates full UI states, detecting layout shifts, missing elements, and text overlaps automatically.

By consolidating validations into one Visual AI check, teams cut review time, reduce false positives, and gain faster feedback cycles.

Scale Reviews with Ultrafast Grid and Grouping

Running tests one browser at a time no longer scales. The Applitools Ultrafast Grid executes a single test once, then validates results across every browser and device combination in parallel.

Batching and grouping features make reviews equally efficient—approve or reject similar changes across entire runs in just a few clicks.

How it works

• Replace assertions with one visual checkpoint
• Run once across all browsers and devices
• Batch results for unified review
• Approve or reject in bulk
• Tune match levels for dynamic content

Together, these capabilities eliminate redundant effort and make large-scale testing faster to maintain.

Customer Results: 78% Less Maintenance

Teams that adopt this approach see measurable ROI. At Peloton, replacing a legacy visual testing tool with Applitools Visual AI produced a 78% reduction in maintenance time and saved about 130 hours per month.

With dynamic leaderboards, live data, and responsive layouts across web and native mobile, Peloton maintains quality at scale without expanding test overhead.

Three Features That Change Maintenance

“Ultrafast Grid, Visual AI match levels, and bulk grouping—those three change the game.”

Mike Millgate, Smarter Test Maintenance at Scale

These three deliver flexible validation, fast execution, and effortless maintenance. Each removes manual steps and accelerates the feedback loop that keeps releases reliable.

Smarter Maintenance for Modern Teams

Smarter test maintenance isn’t about writing more code—it’s about automating intelligently. Visual AI reduces flakiness, speeds reviews, and scales across devices and environments.

To see what’s next, explore Applitools Eyes 10.22, featuring faster review cycles, new Storybook and Figma integrations, and even shorter feedback loops for test maintenance at scale.

The post Test Maintenance at Scale: How Visual AI Cuts Review Time and Flakiness appeared first on AI-Powered End-to-End Testing | Applitools.

Software leaders often face a false choice: should testing be code-driven or no-code? The truth is, the strongest strategies use code and no-code test automation together. By letting each approach play to its strengths, teams cut bottlenecks, empower more contributors, and deliver quality software faster.

The Pitfalls of Choosing One Approach

When organizations lean too heavily on one side—whether code or no-code—the same challenges show up again and again:

• Skill gaps: Engineers and testers bring different levels of coding expertise, which creates dependencies and slows progress.
• Silos: Developers, QA, and manual testers often work separately, with little shared visibility.
• Maintenance overhead: Purely coded frameworks can be fragile and time-consuming to update, while a no-code-only strategy can limit flexibility for advanced scenarios.

Instead of streamlining releases, testing becomes another obstacle—especially when teams frame it as code versus no-code instead of embracing code and no-code test automation as a unified strategy.

The Strengths of Code-Based Automation

Code-based frameworks like Selenium, Cypress, and Playwright remain essential for complex cases. They provide:

• Flexibility and customization to test virtually any scenario.
• Fine-grained control over selectors, browser behavior, and environments.
• Precision that’s critical when working with complex workflows.

For engineering teams, code is still the best tool for edge cases and advanced automation.

The Strengths of No-Code Automation

No-code testing platforms such as Applitools Autonomous thrive on speed and accessibility. With plain-language test authoring and visual interfaces, they allow non-technical testers to contribute directly. This makes them ideal for:

• Regression and smoke tests that repeat across releases.
• Routine workflows that don’t require custom code.
• Broad participation across QA and business testers.

The benefit: engineers aren’t pulled into repetitive work, freeing them to focus on higher-value challenges.

Code + No-Code in Action

The difference becomes clear when comparing the two side by side. In one demo, a Selenium test for a simple e-commerce checkout flow took nearly an hour to script. Using Autonomous, the same flow—with assertions—was built in just two minutes.

The takeaway isn’t that one should replace the other. No-code handles what’s fast and repeatable; code handles the complex and custom. Together, they balance speed and depth.

Watch Code & No-Code Journeys: The Collaboration Campground now on-demand.

Real-World Proof: EVERSANA

EVERSANA INTOUCH, a global life sciences agency, illustrates what this balance looks like in practice. Faced with strict compliance requirements and fragmented workflows, they needed to unify testing across teams worldwide.

• First step: Adopted Applitools Eyes (code-based visual testing).
• Next step: Expanded to Autonomous, allowing global manual testers to build end-to-end tests in the browser.

Result: A 65%+ reduction in regression testing time, faster validation across browsers and environments, and a new “Autonomous-first” policy before assigning engineering resources.

The biggest change wasn’t only speed—it was collaboration. Developers, testers, and compliance began working from shared results, cutting duplicate effort and improving trust across the organization.

Read more about how EVERSANA INTOUCH cut regression testing time by 65% in the customer case study.

Takeaway for QA and Engineering Leaders

The question isn’t “code or no-code.” It’s how best to integrate both. For many teams, this means adopting code and no-code test automation to scale testing with confidence. By using no-code for regression and repeatable flows, and code for complex scenarios, teams reduce bottlenecks, shorten feedback cycles, and scale their testing with confidence.

For mid-size to enterprise teams, this balanced approach delivers:

• Faster test creation and execution.
• Greater collaboration across roles and skill levels.
• A testing strategy that keeps pace with modern release cycles.

Next Steps

Identify where no-code can relieve your engineers, and where code provides the precision you need. The future of testing isn’t about choosing sides—it’s about working smarter with both. Start your own code and no-code journey with Applitools Autonomous.

The post Why the Future of Test Automation is Code AND No-Code appeared first on AI-Powered End-to-End Testing | Applitools.

Applitools was named CIO Review’s 2025 AI-Powered Test Automation Platform of the Year—not because we chased buzzwords, but because the platform is fundamentally AI-native, built for engineering scale, and designed to help developers test smarter without slowing down.

For developers, testers, and QA engineers, the award reflects what actually matters:

• Reducing test flakiness
• Automating visual and functional checks in parallel
• Scaling test execution across browsers and devices
• Plugging into CI/CD pipelines without disrupting existing workflows

Let’s break down what makes this platform stand out from a developer’s perspective.

AI-Native Testing, Not Bolt-On AI

Applitools isn’t a traditional test framework with AI sprinkled on top. It’s purpose-built to use Visual AI plus code-aware intelligence for smarter test coverage. That means:

• You can catch regressions that DOM diffs would miss
• You write fewer assertions, yet spot more visual and layout issues
• You reduce false positives and test flakiness—without relying on brittle selectors

It’s AI-native automation that understands what the user sees, not just what the code renders.

Built for Real Engineering Workflows

Applitools supports every major language and framework, including:

• Languages: JavaScript, TypeScript, Java, Python, C#, Ruby
• Frameworks: Cypress, Playwright, Selenium, WebdriverIO, and more
• Mobile: Appium and native frameworks

You don’t need to rip and replace. Applitools plugs directly into your current test suite with minimal setup and no test rewrites required.

Ultrafast Grid = Multi-Platform Testing Without the Bottlenecks

You run your tests once. Applitools executes them across dozens of browser, OS, and device combinations in parallel—via the Ultrafast Grid, not your CI or local machine.

That means:

• Fast, scalable cross-browser coverage
• Smart DOM diffing combined with Visual AI
• Consistent UX testing across breakpoints and devices

No emulators. No stitched screenshots. Just reliable results, fast.

Seamless CI/CD Integration

Applitools fits natively into DevOps workflows with:

• GitHub Actions, GitLab, Jenkins, CircleCI, Azure Pipelines, Bitbucket, TeamCity
• Rich CLI tooling for custom pipelines
• Git-based test baselines and approval workflows
• Smart diffing and auto-approvals to keep noisy builds out of your way

For more, explore our Integrations Hub.

This is test automation that moves with your code, not one that slows it down.

Dev Teams Are Reporting…

Here’s what teams have seen after adopting Applitools:

• Up to 80% reduction in test maintenance overhead
• 10x faster execution across browsers and devices
• 70% fewer visual bugs escaping into production
• Faster code reviews with fewer test-related delays

Whether you’re validating a single feature branch or running thousands of tests in parallel, Applitools is built to support real scale—without compromising on accuracy.

Why This Award Actually Matters

The CIO Review award isn’t about hype. It’s a reflection of what forward-looking engineering teams need from test automation in 2025—More confidence. Less friction. AI that works.

If you’re building modern apps, you deserve modern testing. Applitools gives you a platform that evolves with your code, scales with your team, and delivers confidence without the test fatigue.

Applitools Resources for Developers

• Docs & SDKs
• Start a Free Trial
• Cory House’s Creating Automated Tests with AI webinar
• Adam Carmi’s Building No-Code Autonomous End-to-End Tests webinar

The post AI Test Automation Platform for Developers: Why Applitools Won in 2025 appeared first on AI-Powered End-to-End Testing | Applitools.

Visual testing is crucial for ensuring seamless user experiences by identifying regressions early and maintaining the visual integrity of applications. In our recent webinar, Playwright Visual Testing Best Practices, Cory House—a software developer, architect, and Playwright ambassador—shared essential insights and practical strategies to enhance your testing process. 

Why Visual Testing Matters

Visual testing focuses on what users see when interacting with an application. Unlike functional tests, which validate “if it works,” visual testing ensures “if it looks right.” From detecting subtle CSS issues to validating themes and layouts across devices, visual testing complements unit, integration, and end-to-end testing strategies. Cory outlined why visual testing is essential:

• Catch visual regressions early—Identify unintended changes in your application’s design.
• Validate consistency—Ensure design components like themes, layouts, and responsiveness work across browsers and devices.
• Boost confidence in refactoring—With robust visual testing, developers gain the confidence to improve features without fear of introducing visual errors.

Two Approaches to Visual Testing

During the webinar, two approaches to visual testing were discussed:

1. Playwright’s Built-in Snapshot Testing 

  Playwright offers a developer-friendly built-in snapshot testing tool. It’s great for smaller projects or teams just getting started with visual testing. However, challenges like layout shifts and dynamic content require additional custom configurations.

  Key Advantage: Minimal setup for developers and local execution for quick validation. 

  Key Limitation: Susceptible to test flakiness, false positives, and limited handling of dynamic content. 

2. Applitools Eyes’ AI-powered Visual Testing 

  For advanced needs, Applitools Eyes provides enterprise-grade features like AI algorithms to handle dynamic content, floating elements, and layout changes. Its cloud-based platform ensures consistent testing across environments, reducing false positives caused by machine or browser differences.

  Key Advantages: 

• Handles dynamic content effortlessly. 
• Detects meaningful changes instead of minor pixel differences. 
• Enables accessibility and A/B testing support. 
• Simplifies debugging with an intuitive dashboard.

  Key Limitation: Requires integration efforts, but delivers unmatched scalability and precision.

“Playwright snapshots are great for getting started with visual testing, but if you’re dealing with complex dynamic content or working in large teams, Applitools’ advanced capabilities are a game-changer.” – Cory House

Overcoming Common Visual Testing Challenges

Layout Shifts

Challenge: Element displacement due to new features or changes can lead to failed tests. 

Solution: 

• With Playwright: Snapshot individual components instead of entire pages. 
• With Applitools: Use “ignore” region feature to avoid unnecessary failures from harmless shifts.

Dynamic Content

Challenge: Changing data, like user balances in a dashboard, can cause test flakiness. 

Solution: 

• Playwright users can mask dynamic areas or use mock data. 
• Applitools simplifies this with AI that can ignore insignificant dynamic changes or confirm text data matches preconfigured or custom patterns.

Test Flakiness

Challenge: Slight differences in rendering across machines and environments lead to false failures. 

Solution: 

• Run tests in consistent CI environments to avoid discrepancies between local setups. 
• Applitools’ cloud-based testing offers consistent rendering across environments for each run and automatically accounts for differences between browsers, devices, and screen sizes. 

Merge Conflicts

Challenge: Snapshot diffs can lead to conflicts during team collaboration. 

Solution: 

• Use Applitools’ diffing tool for a visual representation of changes, allowing easy resolution.

Debugging

Challenge: Debugging failed tests on full-page snapshots can be overwhelming. 

Solution: 

• Focus on smaller, modular components with Playwright. 
• Leverage Applitools’ annotations and layout-focused match levels for better clarity.

Want more detail on any of these challenges?

Watch the full webinar recording.

Strategic Takeaway for Teams

Visual testing is not a silver bullet but a powerful addition to your testing toolkit. Cory highlighted the need for a balanced testing strategy, integrating visual testing with unit, integration, and end-to-end tests. Choosing the right tools and configuration can accelerate development cycles while maintaining quality.

Next Steps

Want to learn more about implementing visual testing in your workflow? Book a personalized demo of Applitools today and explore how AI-powered testing can transform your testing strategy.

The post Recap: Playwright Visual Testing Best Practices appeared first on AI-Powered End-to-End Testing | Applitools.

Learn what visual testing is, why visual testing is important, how to catch visual bugs, the differences between visual and functional testing, and how you can get started with automated visual testing today.

What is Meant By Visual Testing?

Visual testing evaluates the visible output of an application and compares that output against the results expected by design. In other words, it helps catch “visual bugs” in the appearance of a page or screen, which are distinct from strictly functional bugs. Automated visual testing tools, like Applitools, can help speed this visual testing up and reduce errors that occur with manual verification.

You can run visual tests at any time on any application with a visual user interface. Most developers run visual tests on individual components during development, and on a functioning application during end-to-end tests.

In today’s world, in the world of HTML, web developers create pages that appear on a mix of browsers and operating systems. Because HTML and CSS are standards, front-end developers want to feel comfortable with a ‘write once, run anywhere’ approach to their software. Which also translates to “Let QA sort out the implementation issues.” QA is still stuck checking each possible output combination for visual bugs.

This explains why, when I worked in product management, QA engineers would ask me all the time, “Which platforms are most important to test against?” If you’re like most QA team members, your test matrix has probably exploded: multiple browsers, multiple operating systems, multiple screen sizes, multiple fonts—and dynamic responsive content that renders differently on each combination.

If you are with me so far, you’re starting to answer the question: why do visual testing?

Why is Visual Testing Important?

We do visual testing because visual errors happen—more frequently than you might realize. Take a look at this visual bug on Instagram’s app:

The text and ad are crammed together. If this was your ad, do you think there would be a revenue impact? Absolutely.

Visual bugs happen at other companies too: Amazon. Google. Slack. Robin Hood. Poshmark. Airbnb. Yelp. Target. Southwest. United. Virgin Atlantic. OpenTable. These aren’t cosmetic issues. In each case, visual bugs are blocking revenue.

If you need to justify spending money on visual testing, share these examples with your boss.

All these companies are able to hire some of the smartest engineers in the world. If it happens to Google, Instagram, or Amazon, it probably can happen to you, too.

Why do these visual bugs occur? Don’t they do functional testing? They do — but it’s not enough.

Visual bugs are rendering issues. Rendering validation is not what functional testing tools are designed to catch. Functional testing measures functional behavior.

Why can’t functional tests cover visual issues?

Sure, functional test scripts can validate the size, position, and color scheme of visual elements. But if you do this, your test scripts will soon balloon in size due to checkpoint bloat.

To see what I mean, let’s look at an Instagram ad screen that’s properly rendered. There are 21 visual elements by my count—various icons, and text. (This ignores iOS elements at the top like WiFi signal and time, since those aren’t controlled by the Instagram app.)

If you used traditional checkpoints in a functional testing tool like Selenium Webdriver, Cypress, WebdriverIO, or Appium, you’d have to check the following for each of those 21 visual elements:

1. Visible (true/false)
2. Upper-left x,y coordinates
3. Height
4. Width
5. Background color

That means you’d need the following number of assertions:

21 visual elements x 5 assertions per element = 105 lines of assertion code

Even with all this assertion code, you wouldn’t be able to detect all visual bugs. Such as whether a visual element can’t be accessed because it’s being covered up, which blocked revenue in the above examples from Yelp, Southwest, United, and Virgin Atlantic. And, you’d miss subtleties like the brand logo, or the red dot under the heart.

But it gets worse: if OS, browser, screen orientation, screen size, or font size changes, your app’s appearance will change as a result. That means you have to write another 105 lines of functional test assertions. For EACH combination of OS/browser/font size/screen size/screen orientation/font size.

You could end up with thousands of lines of assertion code — any of which might need to change with a new release. Trying to maintain that would be sheer madness. No one has time for that.

You need visual testing because visual errors occur. And you need visual testing because you cannot rely on functional tests to catch visual errors.

What is Manual Visual Testing?

Because automated functional testing tools are poorly suited for finding visual bugs, companies find visual glitches using manual testers. Lots of them (more on that in a bit).

For these manual testers, visual testing behaves a lot like this spot-the-difference game:

To understand how time-consuming visual testing can be, get out your phone and time how long it takes for you to find all six visual differences. I took a minute to realize that the writing in the panels doesn’t count. It took me about 3 minutes to spot all six. Or, you can cheat and look at the answers.

Why does it take so long? Some differences are difficult to spot. In other cases, our eyes trick us into finding differences that don’t exist.

Manual visual testing means comparing two screenshots, one from your known good baseline image, and another from the latest version of your app. For each pair of images, you have to invest time to ensure you’ve caught all issues. Especially if the page is long, or has a lot of visual elements. Think “Where’s Waldo”…

Challenges of manual testing

If you’re a manual tester or someone who manages them, you probably know how hard it is to visually test.

If you are a test engineer reading this paragraph, you already know this: web page testing only starts with checking the visual elements and their function on a single operating system, browser, browser orientation, and browser dimension combination. Then continue on to other combinations. And, that’s where a huge amount of test effort lies – not in the functional testing, but in the inspection of visual elements across the combination of an operating system, browser, screen orientation, and browser dimensions.

To put it in perspective, imagine you need to test your app on:

• 5 operating systems: Windows, MacOS, Android, iOS, and Chrome.
• 5 popular browsers: Chrome, Firefox, Internet Explorer (Windows only) Microsoft Edge (Windows Only), and Safari (Mac only).
• 2 screen orientations for mobile devices: portrait and landscape.
• 10 standard mobile device display resolutions and 18 standard desktop/laptop display resolutions from XGA to 4G.

If you’re doing the math, you think it’s the browsers running on each platform (a total of 21 combinations) multiplied by the two orientations of the ten mobiles (2×10)=20 added to the 18 desktop display resolutions.

21 x (20+18) = 21 x 38 = 798 Unique Screen Configurations to test

That’s a lot of testing—for just one web page or screen in your mobile app.

Except that it’s worse. Let’s say your app has 100 pages or screens to test.

798 Screen Configurations x 100 Screens in-app = 79,800 Screen Configurations to test

Meanwhile, companies are releasing new app versions into production as frequently as once a week, or even once a day.

How many manual testers would you need to test 79,800 screen configurations in a week? Or a day? Could you even hire that many people?

Wouldn’t it be great if there was a way to automate this crazy-tedious process?

Well, yes there is…

What is Automated Visual Testing?

Automated visual testing uses software to automate the process of comparing visual elements across various screen combinations to uncover visual defects.

Automated visual testing piggybacks on your existing functional test scripts running in a tool like Selenium Webdriver, Cypress, WebdriverIO, or Appium. As your script drives your app, your app creates web pages with static visual elements. Functional testing changes visual elements, so each step of a functional test creates a new UI state you can visually test.

Automated visual testing evolved from functional testing. Rather than descending into the madness of writing assertions to check the properties of each visual element, automated visual testing tools visually check the visual appearance of an entire screen with just one assertion statement. This leads to test scripts that are MUCH simpler and easier to maintain.

But, if you’re not careful, you can go down an unproductive rat hole. I’m talking about Snapshot Testing.

What is Snapshot Testing?

First-generation automated visual testing uses a technology called snapshot testing. With snapshot testing, a bitmap of a screen is captured at various points of a test run and its pixels are compared to a baseline bitmap.

Snapshot testing algorithms are very simplistic: iterate through each pixel pair, then check if the color hex code is the same. If the color codes are different, raise a visual bug.

Because they can be built relatively easily, there are a number of open-source and commercial snapshot testing tools. Unlike human testers, snapshot testing tools can spot pixel differences quickly and consistently. And that’s a step forward. A computer can highlight the visual differences in the Hocus Focus cartoon easily. A number of these tools market themselves as enabling “pixel-perfect testing”.

Sounds like a good idea, right?

What are the Problems With Snapshot Testing?

Alas, pixels aren’t visual elements. Font smoothing algorithms, image resizing, graphics cards, and even rendering algorithms generate pixel differences. And that’s just static content. The actual content can vary between any two interfaces. As a result, a tool that expects exact pixel matches between two images can be filled with pixel differences.

If you want to see some examples of bitmap differences affecting snapshot testing, take a look at the blog post we wrote on this topic last year.

Unfortunately, while you might think snapshot testing makes intuitive sense, practitioners like you are finding that the conditions for running successful bitmap comparisons require a stationary target, while your company continues to develop dynamic websites across a range of browsers and operating systems. You can try to force your app to behave a certain way – but you may not always succeed.

Can you share some details of Snapshot Testing Problems?

For example, when testing on a single browser and operating system:

• Identify and isolate (mute) fields that change over time, such as radio signal strength, battery state, and blinking cursors.
• Ignore user data that might otherwise change over time, such as visitor count.
• Determine how to support testing content on your site that must change frequently – especially if you are a media company or have an active blog.
• Consider how different hardware or software affects antialiasing.

When doing cross-browser testing, you must also consider:

• Text wrapping, because you cannot guarantee the locations of text wrapping between two browsers using the same specifications. The text can break differently between two browsers, even with identical screen sizes.
• Image rendering software, which can affect the pixels of font antialiasing as well as images and can vary from browser to browser (and even on a single browser among versions).
• Image rendering hardware, which may render bitmaps differently.
• Variations in browser font size and other elements that affect the text.

If you choose to pursue snapshot testing in spite of these issues, don’t be surprised if you end up joining the group of experienced testers who have tried, and then ultimately abandoned, snapshot testing tools.

Can I See Some Snapshot Testing Problems In Real Life?

Here are some quick examples of these real-life bitmap issues.

If you use pixel testing for mobile apps, you’ll need to deal with the very dynamic data at the top of nearly every screen: network strength, time, battery level, and more:

When you have dynamic content that shifts over time — news, ads, user-submitted content — where you want to check to ensure that everything is laid out with proper alignment and no overlaps. Pixel comparison tools can’t test for these cases. Twitter’s user-generated content is even more dynamic, with new tweets, likes, retweets, and comment counts changing by the second.

Your app doesn’t even need to change to confuse pixel tools. If your baselines and test screenshots were captured on different machines with different display settings for anti-aliasing, that can turn pretty much the entire page into a false positive, like this:

Source: storybook.js.org

If you’re using pixel tools and you still have to track down false positives and expose false negatives, what does that say about your testing efficiency?

For these reasons, many companies throw out their pixel tools and go back to manual visual testing, with all of its issues.

There’s a better alternative: using AI—specifically computer vision—for visual testing.

How Do I Use AI for Automated Visual Testing?

The current generation of automated visual testing uses a class of artificial intelligence algorithms called computer vision as a core engine for visual comparison. Typically these algorithms are used to identify objects with images, such as with facial recognition. We call them visual AI testing tools.

AI-powered automated visual testing combines a learning algorithm to interpret the relationship between a rendered page and the intended display of visual elements with actual visual elements and locations. Like pixel tools, AI-powered automated visual testing takes page snapshots as your functional tests run. Unlike pixel-based comparators, AI-powered automated visual test tools use algorithms instead of pixels to determine when errors have occurred.

Unlike snapshot testers, AI-powered automated visual testing tools do not need special environments that remain static to ensure accuracy. Testing and real-world customer data show that AI testing tools have a high degree of accuracy even with dynamic content because the comparisons are based on relationships and not simply pixels.

Here’s a comparison of the kinds of issues that AI-powered visual testing tools can handle compared to snapshot testing tools:

Some AI-powered test tools have been tested at a false positive rate of 0.001% (or 1 in every 100,000 errors).

AI-Powered Test Tools In Action

An AI-powered automated visual testing tool can test a wide range of visual elements across a range of OS/browser/orientation/resolution combinations. Just running the first baseline of rendering and functional test on a single combination is sufficient to guide an AI-powered tool to test results across the range of potential platforms

Here are some examples of how AI-powered automated visual testing improves visual test results by awareness of content.

This is a comparison of two different USA Today homepage images. When an AI-powered tool looks at the layout comparison, the layout framework matters, not the content. Layout comparison ignores content differences; instead, layout comparison validates the existence of the content and relative placement. Compare that with a bitmap comparison of the same two pages (also called “exact comparison:):

Literally, every non-whites pace (and even some of the white space) is called out.

Which do you think would be more useful in your validation of your own content?

When Should I Use Visual Testing?

You can do automated visual testing with each check-in of front-end code, after unit testing and API testing, and before functional testing — ideally as part of your CI/CD pipeline running in Jenkins, Travis, or another continuous integration tool.

How often? On days ending with “y”.

Because of the accuracy of AI-powered automated visual testing tools, they can be deployed in more than just functional and visual testing pre-production. AI-powered automated visual testing can help developers understand how visual element components will render across various systems. In addition to running in development, test engineers can also validate new code against existing platforms and new platforms against running code.

AI-powered tools like Applitools allow different levels of smart comparison.

AI-powered visual testing tools are a key validation tool for any app or web presence that requires regular changes in content and format. For example, media companies change their content as frequently as twice per hour and use AI-powered automated testing to isolate real errors that affect paying customers without impacting them. AI-powered visual test tools are key tools in the test arsenal for any app or web presence going through brand revision or merger, as the low error rate and high accuracy let companies identify and fix problems associated with major DOM, CSS, and Javascript changes that are core to those updates.

Talk to Applitools

Applitools is the pioneer and leading vendor in AI-powered automated visual testing. Applitools has a range of options to help you become incredibly productive in application testing. We can help you test components in development. We can help you find the root cause of the visual errors you have encountered. And, we can run your tests on an Ultrafast Grid that allows you to recreate your visual test in one environment across a number of others on various browser and OS configurations. Our goal is to help you realize the vision we share with our customers – you need to create functional tests for only one environment and let Applitools run the validation across all your customer environments after your first test has passed. We’d love to talk testing with you – feel free to reach out to contact us anytime.

More To Read About Visual Testing

If you liked reading this, here are some more Applitools posts and webinars for you.

1. Visual Testing for Mobile Apps by Angie Jones
2. Visual Assertions – Hype or Reality? – by Anand Bagmar
3. The Many Uses of Visual Testing by Angie Jones
4. Visual UI Testing as an Aid to Functional Testing by Gil Tayar
5. Visual Testing: A Guide for Front End Developers by Gil Tayar

Find out more about Applitools. Set up a live demo with us, or if you’re the do-it-yourself type, sign up for a free Applitools account and follow one of our tutorials.

Quick Answers

The post What is Visual Testing? appeared first on AI-Powered End-to-End Testing | Applitools.

In our recent webinar, Building the Ideal CI/CD Pipeline: Achieving Visual Perfection, we explored the transformative power of Visual AI testing for CI/CD pipelines. Aimed at software engineering managers and team leads, the session provided a deep dive into the limitations of traditional functional testing and how Visual AI testing can surpass these to achieve visual perfection in software development.

Technical Customer Success Manager Brandon Murray shared expert strategies and highlighted the benefits of integrating Visual AI testing, offering guidance on constructing the optimal CI/CD pipeline. He explored the intricacies of Visual AI testing, illuminating its critical role in enhancing software quality and performance.

Challenges in Traditional Functional Testing

Murray began by identifying the bottlenecks commonly encountered in traditional functional testing. These include:

• High maintenance efforts
• Slow feedback cycles
• Limited UI coverage
• Tedious manual testing

The Power of Visual AI Testing

Visual AI testing offers a revolutionary approach to overcome these challenges. By capturing screenshots and using AI to compare these snapshots to a baseline ‘golden image’, Visual AI testing ensures:

• Reduced Test Development and Maintenance Time: Automating UI comparisons dramatically decreases the time spent on writing and maintaining tests.
• Complete UI Coverage: Screenshots ensure that every aspect of the UI is tested, eliminating blind spots.
• Enhanced Operational Efficiency: Faster feedback loops lead to quicker identification and resolution of issues, facilitating faster product releases.

Other Strategies to Supplement Visual AI Testing:

• Self-Healing: Automatically corrects flaky tests by adjusting for locator changes, vastly improving test stability
• Lazy Loading: Helps to ensure the entire page content is loaded
• Parallel Test Execution: Enables the execution of multiple tests simultaneously, significantly speeding up the testing process

Integration into the Development Workflow

Integrating Visual AI testing into existing development workflows, particularly with pull request checks, is pivotal for agile environments. The webinar emphasized the importance of instant feedback for swift issue resolution, leading to accelerated development cycles.

Tools and Technologies Highlighted:

• Cypress: Innovative testing framework for both developers and QA engineers
• GitHub Actions: Continuous integration and continuous delivery (CI/CD) platform enabling automation directly in GitHub repositories
• Figma Designs: Useful for collaborative design reviews and direct comparison against implementations

The session underscored the cost-effectiveness of using browsers on cloud infrastructure containers, especially when dealing with cross-browser coverage. Notably, the Filter Fast Grid was mentioned as an effective solution for this purpose.

Comparing Visual AI Testing to Traditional Methods

Attendees were eager to learn how Visual AI testing compares to snapshot tests and other traditional methods. The webinar demonstrated how Visual AI testing offers:

• Greater Accuracy: By leveraging AI for pixel-perfect comparisons
• Higher Efficiency: Through automated and parallel testing routes

In particular, using commodity CI solutions like GitHub Actions or CircleCI was recommended for their affordability and versatility.

Building the Ideal CI/CD Pipeline: Achieving Visual Perfection highlighted the transformative potential of Visual AI testing in optimizing CI/CD pipelines. Software engineering managers and team leads are strongly encouraged to evaluate how AI-powered tools like Applitools can elevate their testing processes, enhance product quality, and expedite delivery timelines. For those interested, a free trial of Applitools is available to experience the benefits firsthand.

The post Recap: Building the Ideal CI/CD Pipeline appeared first on AI-Powered End-to-End Testing | Applitools.

Creating a flawless UI/UX experience across a myriad of devices and platforms is absolutely critical to the success of a business in today’s digitally dominated world. This task has become increasingly complicated with the swift advancement of mobile technologies and the ever-evolving expectations of users. 

Our recent webinar with Martin Kowalewski, Global Sales Engineering Lead at Applitools, and Frank Moyer, CTO of Kobiton, delved into these complexities, offering attendees a comprehensive look at the latest trends in mobile development and testing, with a special focus on the revolutionary impact of AI. 

The session introduced participants to the partnership between Kobiton and Applitools, showcasing a robust solution designed to meet the expansive testing needs of customers. This powerful integration leverages AI to provide a full spectrum solution for continuous testing across all platforms. We’ve highlighted a few of the key takeaways from the webinar below.

Revolutionizing Mobile App Testing with Visual AI Technology

The webinar kicked off by diving into the world of Visual AI technology, with a strong emphasis on its pivotal role in boosting the accuracy of UI and UX testing. Participants were provided with in-depth insights into how Visual AI is revolutionizing the field of software testing, offering examples of its application and the significant improvements it brings to the quality engineering process. Attendees left with a clear understanding of the transformative impact Visual AI has on enhancing user experience and interface design.

The conversation then shifted towards a comprehensive examination of Applitools’ Visual AI platform, showcasing its revolutionary approach to automating visual testing. This platform stands out by providing advanced monitoring and management tools specifically designed for the visual aspects of applications. By doing so, it sets the benchmark in the industry, offering an unparalleled level of precision and efficiency in detecting and addressing visual discrepancies, thereby significantly enhancing the quality assurance process for software developers.

The Power of Integrating Kobiton and Applitools for Enhanced Testing

The presentation emphasized the advantages of utilizing Kobiton for real device testing, particularly how Kobiton’s comprehensive mobile device cloud and on-premises device laboratories facilitate testing conditions that closely mimic real-world usage. By offering access to an extensive selection of mobile devices, Kobiton significantly improves the testing process. This access allows developers and testers to ensure their applications perform well across a diverse spectrum of devices, thereby enhancing user experience and satisfaction. Kobiton’s platform not only streamlines the testing cycle but also aids in detecting and resolving potential issues before release, making it an invaluable tool in the development process.

Streamlining Development: Practical Tips for Implementing Continuous Visual Testing

Finally, the session provided detailed practical advice on effectively integrating Applitools and Kobiton. This integration facilitates seamless, continuous visual testing, which is crucial for ensuring that applications render as intended on a variety of devices. By addressing this integration, developers can tackle one of the most significant challenges in mobile app development head-on, enhancing user experience by guaranteeing that apps look and function correctly across different devices and use scenarios. This approach not only improves the quality of mobile apps but also streamlines the development and testing process, making it more efficient.

The event highlighted the remarkable teamwork between Kobiton and Applitools, providing developers and testers with the necessary tools to excel in this highly competitive technology world. This collaboration has paved the way for enhanced testing capabilities, ensuring that applications are not only functional but also visually appealing, setting a new standard for success. Watch the full webinar here.

The post Recap: Leveraging AI for Mobile App Testing on Real Devices appeared first on AI-Powered End-to-End Testing | Applitools.

As a test automation engineer, finding the right tools and frameworks is crucial to building a successful test automation strategy. Playwright is an end-to-end testing framework that provides a robust set of features to create fast, reliable, and maintainable tests.

In a recent webinar, Playwright Ambassador and TAU instructor Renata Andrade shared several use cases and best practices for using the framework. Here are some of the most valuable takeaways for test automation engineers:

Use Playwright’s built-in locators for resilient tests.
Playwright recommends using attributes like “text”, “aria-label”, “alt”, and “placeholder” to find elements. These locators are less prone to breakage, leading to more robust tests.

Speed up test creation with the code generator.
The Playwright code generator can automatically generate test code for you. This is useful when you’re first creating tests to quickly get started. You can then tweak and build on the generated code.

Debug tests and view runs with UI mode and the trace viewer.
Playwright’s UI mode and VS Code extension provide visibility into your test runs. You can step through tests, pick locators, view failures, and optimize your tests. The trace viewer gives you a detailed trace of all steps in a test run, which is invaluable for troubleshooting.

Add visual testing with Applitools Eyes.
For complete validation, combine Playwright with Applitools for visual and UI testing. Applitools Eyes catches unintended changes in UI that can be missed by traditional test automation.

Handle dynamic elements with the right locators.
Use a combination of attributes like “text”, “aria-label”, “alt”, “placeholder”, CSS, and XPath to locate dynamic elements that frequently change. This enables you to test dynamic web pages.

Set cookies to test personalization.
You can set cookies in Playwright to handle scenarios like A/B testing where the web page or flow differs based on cookies. This is important for testing personalization on websites.

Playwright provides a robust set of features to build, run, debug, and maintain end-to-end web tests. By leveraging the use cases and best practices shared in the webinar, you can power up your test automation and build a successful testing strategy using Playwright. Watch the full recording and see the session materials.

The post Power Up Your Test Automation with Playwright appeared first on AI-Powered End-to-End Testing | Applitools.

Learn why cross-browser testing is so important and an approach you can take to make cross-browser testing with Selenium much faster.

What is Cross Browser Testing?

Cross-browser testing is a form of functional testing in which an application is tested on multiple browsers (Chrome, Firefox, Edge, Safari, IE, etc.) to validate that functionality performs as expected.

In other words, it is designed to answer the question: Does your app work the way it’s supposed to on every browser your customers use?

Why is Cross Browser Testing Important?

While modern browsers generally conform to key web standards today, important problems remain. Differences in interpretations of web standards, varying support for new CSS or other design features, and rendering discrepancies between the different browsers can all yield a user experience that is different from one browser to the next.

A modern application needs to perform as expected across all major browsers. Not only is this a baseline user expectation these days, but it is critical to delivering a positive user experience and a successful app.

At the same time, the number of screen combinations (between screen sizes, devices and versions) is rising quickly. In recent years the number of screens required to test has exploded, rising to an industry average of 81,480 screens and reaching 681,296 for the top 30% of companies.

Ensuring complete coverage of each screen on every browser is a common challenge. Effective and fast cross-browser testing can help alleviate the bottleneck from all these screens that require testing.

How to Perform Modern Cross Browser Testing in Selenium with Visual Testing

Traditional approaches to cross-browser testing in Selenium have existed for a while, and while they still work, they have not scaled well to handle the challenge of complex modern applications. They can be time-consuming to build, slow to execute and challenging to maintain in the face of apps that change frequently.

Applitools Developer Advocate and Test Automation University Director Andrew Knight (AKA Pandy Knight) recently conducted a hands-on workshop where he explored the history of cross-browser testing, its evolution over time and the pros and cons of different approaches.

Andrew then explores a modern cross-browser testing solution with Selenium and Applitools. He walks you through a live demo (which you can replicate yourself by following his shared Github repo) and explains the benefits and how to get started. He also covers how you can accelerate test automation with integration into CI/CD to achieve Continuous Testing.

Check out the workshop below, and follow along with the Github repo here.

More on Cross Browser Testing in Cypress, Playwright or Storybook

At Applitools we are dedicated to making software testing faster and easier so that testers can be more effective and apps can be visually perfect. That’s why we use our industry-leading Visual AI and built the Applitools Ultrafast Grid, a key component of the Applitools Test Cloud that enables ultrafast cross-browser testing. If you’re looking to do cross-browser testing better but don’t use Selenium, be sure to check out these links too for more info on how we can help:

• Testing Storybook Components in Any Browser – Without Writing Any New Tests!
• Running Lightning-Fast Cross-Browser Playwright Tests Against any Browser
• How to Run Cross Browser Tests with Cypress on All Browsers (and follow-up Q&A)
• Learn More about How to Use Applitools for Cross-Browser Testing

The post Ultrafast Cross Browser Testing with Selenium Java appeared first on AI-Powered End-to-End Testing | Applitools.

Learn about common localization bugs, the traditional challenges involved in finding them, and solutions that can make localization testing far easier.

What is Localization?

Localization is the process of customizing a software application that was originally designed for a domestic market so that it can be released in a specific foreign market.

How to Get Started with Localization

Localization testing usually involves substantial changes of the application’s UI, including the translation of all texts to the target language, replacement of icons and images, and many other culture, language, and country-specific adjustments, that affect the presentation of data (e.g., date and time formats, alphabetical sorting order, etc.). Due to the lack of in-house language expertise, localization usually involves in-house personnel as well as outside contractors, and localization service providers.

Before a software application is localized for the first time, it must undergo a process of Internationalization.

What is Internationalization?

Internationalization often involves an extensive development and re-engineering effort which goal is to allow the application to operate in localized environments and to correctly process and display localized data. In addition, locale-specific resources such as texts, images and documentation files, are isolated from the application code and placed in external resource files, so they can be easily replaced without requiring further development efforts.

Once an application is internationalized, the engineering effort required to localize it to a new language or culture is drastically reduced. However, the same is not true for UI localization testing.

The Challenge of UI Localization Testing

Every time an application is localized to a new language, the application changes, or the resources of a supported localization change, the localized UI must be thoroughly tested for localization and internationalization (LI) bugs.

Common Localization and Internationalization Bugs Most Testers can Catch

LI bugs which can be detected by testers that are not language experts include:

• Broken functionality – the execution environment, data or translated resources of a new locale, may uncover internationalization bugs can prevent the application from running or break some of its functionality.
• Untranslated text – text appearing in text fields or images of the localized UI is left untranslated. This indicates that certain resources were not translated or that the original text is hard-coded in the UI and not exported to the resource files.
• Text overlap / overflow – the translated text may require more space than available in its containing control, resulting with the text overflowing the bounds of the control and possibly overlapping or hiding other UI elements.
• Layout corruption – UI controls dynamically adjust their size and position to the expanded or contracted size of the localized text, icons or images, resulting with misaligned, overlapping, missing or redundant UI artifacts.
• Oversized windows and dialogs – multiple expanded texts and images can result with oversized tooltips, dialogs and windows. In extreme situations, expanded dialogs and windows may only be partially visible in low screen resolutions.
• Inadequate fonts – a control’s font cannot properly display some characters of the target language. This usually results with question marks or glyphs being displayed instead of the expected text.

Localization and Internationalization Bugs Requiring Language Expertise

Other common LI bugs which can only be detected with the help of a language expert include:

• Mistranslation – translated text that appears once in the resource files, may appear multiple times in different parts of the application. The context in which the text appears can vary its meaning and require a different translation.
• Wrong images and icons – images and icons were replaced with wrong or inappropriate graphics.
• Text truncation – the translated text may require more space than available in its containing control, resulting with a truncated string.
• Locale violations – wrong date, time, number and currency formats, punctuation, alphabetical sort order, etc.

Localization and Internationalization Bugs are Hard to Find

An unfortunate characteristic of LI bugs, is that they require a lot of effort to find. To uncover such bugs, a tester (assisted by a language expert) must carefully inspect each and every window, dialog, tooltip, menu item, and any other UI state of the application. Since most of these bugs are sensitive to the size and layout of the application, tests must be repeated on a variety of execution environments (e.g., different operating systems, web browsers, devices, etc.) and screen resolutions. Furthermore, if the application window is resizable, tests should also be repeated for various window sizes.

Why is UI Localization Testing Hard?

There are several other factors that contribute to the complexity of UI Localization testing:

• Lack of automation – most of the common LI bugs listed above are visual and cannot be effectively detected by traditional functional test automation tools. Manual inspection of the localized UI is slower than with a non-localized UI because it is  unreadable to the tester.
• Lack of in-house language expertise – since many of the common LI bugs can only be detected with the help of external language experts, which are usually not testers and are not familiar with the application under test, LI testing often requires an in-house tester to perform tests together with a language expert. In many cases, these experts work on multiple projects for multiple customers in parallel, and their occasional lack of availability can substantially delay test cycles and product releases. Similarly, delays can occur while waiting for the translation of changed resources, or while waiting for translation bugs to be fixed.
• Time constraints – localization projects usually begin at late stages of the development lifecycle, after the application UI has stabilized. In many cases, testers are left with little time to properly perform localization tests, and are under constant pressure to avoid delaying the product release.
• Bug severity – UI localization bugs such as missing or garbled text are often considered critical, and therefore must be fixed and verified before the product is released.

Due to these factors, maintaining multiple localized application versions and adding new ones, incurs a huge overhead on quality assurance teams.

Fortunately, there is a modern solution that can make localization testing significantly easier – Automated Visual Testing.

How to Automate Localization Testing with Visual Testing

Visual test automation tools can be applied to UI localization testing to eliminate unnecessary manual involvement of testers and language experts, and drastically shorten test cycles.

To understand this, let’s first understand what visual testing is, and then how to apply visual testing to localization testing.

What is Visual Testing?

Visual testing is the process of validating the visual aspects of an application’s User Interface (UI).

In addition to validating that the UI displays the correct content or data, visual testing focuses on validating the layout and appearance of each visual element of the UI and of the UI as a whole. Layout correctness means that each visual element of the UI is properly positioned on the screen, is of the right shape and size, and doesn’t overlap or hide other visual elements. Appearance correctness means that the visual elements are of the correct font, color, or image.

Visual Test Automation tools can automate most of the activities involved in visual testing. They can easily detect many common UI localization bugs such as text overlap or overflow, layout corruptions, oversized windows and dialogs, etc. All a tester needs to do is to drive the Application Under Test (AUT) through its various UI states and submit UI screenshots to the tool for visual validation.

For simple websites, this can be as easy as directing a web browser to a set of URLs. For more complex applications, some buttons or links should be clicked, or some forms should be filled in order to reach certain screens. Driving the AUT through its different UI states can be easily automated using a variety of open-source and commercial tools (e.g., Selenium, Cypress, etc.). If the tool is properly configured to rely on internal UI object identifiers, the same automation script/program can be used to drive the AUT in all of its localized versions.

So, how can we use this to simplify UI localization testing?

How Automated Visual Testing Simplifies UI Localization Testing

• Preparation – in order to provide translators with the context required to properly localize the application, screenshots of the application’s UI are often delivered along with the resource files to be localized. The process of manually collecting these screenshots is laborious, time consuming, and error prone. When a visual test automation tool is in place, updated screenshots of all UI states are always available and can be shared with translators with a click of a button. When an application changes, the tool can highlight only those screens (in the source language) that differ from the previous version so that only those screens are provided to translators. Some visual test automation tools also provide animated “playbacks” of tests showing the different screens, and the human activities leading from one screen to the next (e.g., clicks, mouse movements, keyboard strokes, etc.).  Such animated playbacks provide much more context than standalone screenshots and are more easily understood by translators, which are usually not familiar with the application being localized. Employing a visual test automation tool can substantially shorten the localization project’s preparation phase and assist in producing higher quality preliminary translations, which in turn can lead to fewer and shorter test cycles.

• Testing localization changes – visual test automation tools work by comparing screenshots of an application against a set of previously approved “expected” screenshots called the baseline. After receiving the translated resources and integrating them with the application, a visual test of the updated localized application can be automatically executed using the previous localized version as a baseline. The tool will then report all screens that contain visual changes and will also highlight the exact changes in each of the changed screens. This report can then be inspected by testers and external language experts without having to manually interact with the localized application. By only focusing on the screens that changed, a huge amount of time and effort can be saved. As we showed above, most UI localization bugs are visual by nature and are therefore sensitive to the execution environment (browser, operating system, device, screen resolution, etc.). Since visual test automation tools automatically execute tests in all required execution environments, testing cycles can be drastically shortened.

• Testing new localizations – when localizing an application for a new language, no localized baseline is available to compare with. However, visual test automation tools can be configured to perform comparisons at the layout level, meaning that only layout inconsistencies (e.g., missing or overflowing text, UI elements appearing out of place, broken paragraphs or columns, etc.) are flagged as differences. By using layout comparison, a newly localized application can be automatically compared with its domestic version, to obtain a report indicating all layout inconsistencies, in all execution environments and screen resolutions.

• Incremental validation – when localization defects are addressed by translators and developers, the updated application must be tested again to make sure that all reported defects were fixed and that no new defects were introduced. By using the latest localized version as the baseline with which to compare the newly updated application, testers can easily identify the actual changes between the two versions, and quickly verify their validity, instead of manually testing the entire application.

• Regression testing – whenever changes are introduced to a localized application, regression testing must be performed to make sure that no localization bugs were introduced, even if no direct changes were made to the application’s localizable resources. For example, a UI control can be modified or replaced, the contents of a window may be repositioned, or some internal logic that affects the application’s output may change. It is practically impossible to manually perform these tests, especially with today’s Agile and continuous delivery practices, which dictate extremely short release cycles. Visual test automation tools can continuously verify that no unexpected UI changes occur in any of the localized versions of the application, after each and every change to the application.

• Collateral material – in additional to localizing the application itself, localized versions of its user manual, documentation and other marketing and sales collateral must be created. For this purpose, updated screenshots of the application must be obtained. As described above, a visual test automation tool can provide up-to-date screenshots of any part of the application in any execution environment. The immediate availability of these screenshots significantly reduces the chance of including out-of-date application images in collaterals and eliminates the manual effort involved in obtaining them after each application change.

Application localization is notoriously difficult and complex. Manually testing for UI localization bugs, during and between localization projects, is extremely time consuming, error-prone, and requires the involvement of external language experts.

Visual test automation tools are a modern breed of test automation tools that can effectively eliminate unnecessary manual involvement, drastically shorten the duration of localization projects, and increase the quality of localized applications.

Applitools Automated Visual Testing and Localization Testing

Applitools has pioneered the use of Visual AI to deliver the best visual testing in the industry. You can learn more about how Applitools can help you with localization testing, or to get started with Applitools today, request a demo or sign up for a free Applitools account.

Editor’s Note: Parts of this post were originally published in two parts in 2017/2018, and have since been updated for accuracy and completeness.

The post Getting Started with Localization Testing appeared first on AI-Powered End-to-End Testing | Applitools.

In this guide, learn everything you need to know about cross-browser testing, including examples, a comparison of different implementation options and how you can get started with cross-browser testing today.

What is Cross Browser Testing?

Cross Browser Testing is a testing method for validating that the application under test works as expected on different browsers, at varying viewport sizes, and devices. It can be done manually or as part of a test automation strategy. The tooling required for this activity can be built in-house or provided by external vendors.

Why is Cross Browser Testing Important?

When I began in QA I didn’t understand why cross-browser testing was important. But it quickly became clear to me that applications frequently render differently at different viewport sizes and with different browser types. This can be a complex issue to test effectively, as the number of combinations required to achieve full coverage can become very large.

A Cross Browser Testing Example

Here’s an example of what you might look for when performing cross-browser testing. Let’s say we’re working on an insurance application. I, as a user, should be able to view my insurance policy details on the website, using any browser on my laptop or desktop. 

This should be possible while ensuring:

• The features remain the same
• The look and feel, UI or cosmetic effects are the same
• Security standards are maintained

How to Implement Cross Browser Testing 

There are various aspects to consider while implementing your cross-browser testing strategy.

Understand the scope == Data!

“Different devices and browsers: chrome, safari, firefox, edge”

Thankfully IE is not in the list anymore (for most)!

You should first figure out the important combinations of devices and browsers and viewport sizes your userbase is accessing your application from. 

PS: Each team member should have access to the analytics data of the product to understand patterns of usage of the product. This data, which includes OS, browser details (type, version, viewport sizes) are essential to plan and test proactively, instead of later reacting to situations (= defects).

This will tell you the different browser types, browser versions, devices, viewport sizes you need to consider in your testing and test automation strategy.

Cross Browser Testing Techniques

There are various ways you can perform cross-browser testing. Let’s understand them.

Local Setup -> On a Single (Dev / QA Machine)

We usually have multiple browsers on our laptop / desktops. While there are other ways to get started, it is probably simplest to start implementing your cross browser tests here. You also need a local setup to enable debugging and maintaining / updating the tests. 

If mobile-web is part of the strategy, then you also need to have the relevant setup available on local machines to enable that.

Setting up the Infrastructure

While this may seem the easiest, it can get out of control very quickly. 

Examples:

• You may not be able to install all supported browsers on your computer (ex: Safari is not supported on Windows OS). 
• Browser vendors keep releasing new versions very frequently. You need to keep your browser drivers in sync with this.
• Maintaining / using older versions of the browsers may not be very straightforward.
• If you need to run tests on mobile devices, you may not have access to all the variety of devices. So setting up local emulators may be a way to proceed.

The choices can actually vary based on the requirements of the project and on a case by case basis.

As alternatives, we have the liberty to choose and create either an in-house testing solution, or go for a platform / license / third party tool to support our device farm needs.

In-House Setup of Central Infrastructure

You can set up a central infrastructure of browsers and emulators or real devices in your organization that can be leveraged by the teams. You will also need some software to manage the usage and allocation of these browsers and devices. 

This infrastructure can potentially be used in the following ways:

• Triggered from local machine
  Tests can be triggered from any dev / QA machine to run on the central infrastructure.

• For CI execution
  Tests triggered via Continuous Integration (CI), like Jenkins, CircleCI, Azure DevOps, TeamCity, etc. can be run against browsers / emulators setup on the central infrastructure. 

Cloud Solution    

You can also opt to run the tests against browsers / devices in a cloud-based solution. You can select different device / browser options offered by various providers in the market that give you the wide coverage as per your requirements, without having to build / maintain / manage the same. This can also be used to run tests triggered from local machines, or from CI.

Modern, AI-Based Cross Browser Testing Solution: Applitools Ultrafast Test Cloud 

It is important to understand the evolution of browsers in recent years. 

• They have started conforming to the W3C standard. 
• They seem to have started adopting Continuous Delivery – well, at least releasing new versions at a very fast pace, sometimes multiple versions a week.
• In a major development a lot of major browsers are adopting and building on the Chromium codebase. This makes these browsers very similar, except the rendering part – which is still pretty browser specific.

We need to factor this change in our cross browser testing strategy. 

In addition, AI-based cross-browser testing solutions are becoming quite popular, which use machine learning to help scale your automation execution and get deep insights into the results – from a functional, performance and user-experience perspective.

To get hands-on experience in this, I signed-up for a free Applitools account, which uses a powerful Visual AI, and implemented a few tests using this tutorial as a reference.

How Does Applitools Visual AI Work as a Solution for Cross Browser Testing

Integration with Applitools

Integrating Applitools with your functional automation is extremely easy. Simply select the relevant Applitools SDK based on your functional automation tech stack from here, and follow the detailed tutorial to get started.

Now, at any place in your test execution where you need functional or visual validation, add methods like eyes.checkWindow(), and you are set to run your test against any browser or device of your choice.

Reference: https://app14743.cloudwayssites.com/tutorials/overview/how-it-works.html

AI-Based Cross Browser Testing

Now you have your tests ready and running against a specific browser or device, scaling for cross-browser testing is the next step.

What if I told you with just the addition of the different device combinations, you can leverage the same single script to give you the functional and visual test results on the variety of combinations specified, covering the cross browser testing aspect as well.

Seems too far-fetched?

It isn’t. That is exactly what Applitools Ultrafast Test Cloud does!

The addition of lines of code below will do the magic. You can also go about changing the configurations, as per your requirements. 

(Below example is from the Selenium-Java SDK. Similar configuration can be supplied for the other SDKs.)

// Add browsers with different viewports
config.addBrowser(800, 600, BrowserType.Chrome);
config.addBrowser(700, 500, BrowserType.FIREFOX);
config.addBrowser(1600, 1200, BrowserType.IE_11);
config.addBrowser(1024, 768, BrowserType.EDGE_CHROMIUM);
config.addBrowser(800, 600, BrowserType.SAFARI);

// Add mobile emulation devices in Portrait mode
config.addDeviceEmulation(DeviceName.iPhone_X, ScreenOrientation.PORTRAIT;
config.addDeviceEmulation(DeviceName.Pixel_2, ScreenOrientation.PORTRAIT;

// Set the configuration object to eyes
eyes.setConfiguration(config);

Now when you run the test again, say against Chrome browser on your laptop, in the Applitools dashboard, you will see results for all the browser and device combinations provided above.

You may be wondering, the test ran just once on the Chrome browser. How did the results from all other browsers and devices come up? And so fast?

This is what Applitools Ultrafast Grid (a part of the Ultrafast Test Cloud) does under the hood:

• When the test starts, the browser configuration is passed from the test execution to the Ultrafast Grid.
• For every eyes.checkWindow call, the information captured (DOM, CSS, etc.) is sent to the Ultrafast Grid.
• The Ultrafast Grid will render the same page / screen on each browser / device provided by the test – (think of this as playing a downloaded video in airplane mode).
• Once rendered in each browser / device, a visual comparison is done and the results are sent to the Applitools dashboard.

What I like about this AI-based solution, is that:

• I create my automation scripts for different purposes – functional, visual, cross browser testing, in one go
• There is no need of maintaining devices 
• There is no need to create different set-ups for different types of testing
• The AI algorithms start providing results from the first run – “no training required”
• I can leverage the solution on any kind of setup 
  • i.e. running the scripts through my IDE, terminal, or CI/CD 
• I can leverage the solution for web, mobile web, and native apps
• I can integrate Visual Testing results in as part of my CI execution
• Rich information available in the dashboard including ease of updating the baselines, doing Root Cause Analysis, reporting defects in Jira or Rally, etc.
• I can ensure there are no Contrast issues (part of Accessibility testing) in my execution at scale

Here is the screenshot of the Applitools dashboard after I ran my sample tests:

Cross Browser Testing Tools and Applitools Visual AI

The Ultrafast Test Grid and Applitools Visual AI can be integrated into many popular and free and open source test automation frameworks to easily supercharge their effectiveness as cross-browser testing tools.

Cross Browser Testing in Selenium

As you saw above in my code sample, Ultrafast Grid is compatible with Selenium. Selenium is the most popular open source test automation framework. It is possible to perform cross browser testing with Selenium out of the box, but Ultrafast Grid offers some significant advantages. Check out this article for a full comparison of using an in-house Selenium Grid vs using Applitools.

Cross Browser Testing in Cypress

Cypress is another very popular open source test automation framework. However, it can only natively run tests against a few browsers at the moment – Chrome, Edge and Firefox. The Applitools Ultrafast Grid allows you to expand this list to include all browsers. See this post on how to perform cross-browser tests with Cypress on all browsers.

Cross Browser Testing in Playwright

Playwright is an open source test automation framework that is newer than both Cypress and Selenium, but it is growing quickly in popularity. Playwright has some limitations on doing cross-browser testing natively, because it tests “browser projects” and not full browsers. The Ultrafast Grid overcomes this limitation. You can read more about how to run cross-browser Playwright tests against any browser.

Pro and Cons of Each Technique (Table of Comparison)

My Learning from this Experience

• A good cross browser testing strategy allows you to reduce the risk of functionality and visual experience not working as expected on the browsers and devices used by your users. A good strategy will also optimize the testing efforts required to do this. To allow this, you need data to provide the insights from your users.
• Having a holistic view of how your team will be leveraging cross browser testing (ex: manual testing, automation, local executions, CI-based execution, etc.) is important to know before you start off with your implementation.
• Sometimes the easiest way may not be the best – ex: Using the browsers on your computer to automate against that will not scale. At the same time, using technology like Applitools Ultrafast Test Cloud is very easy – you end up writing less code and get increased functional and visual coverage at scale. 
• You need to think about the ROI of your approach and if it achieves the objectives of the need for cross browser testing. ROI calculation should include:
  • Effort to implement, maintain, execute and scale the tests
  • Effort to set up, and maintain the infrastructure (hardware and software components)
  • Ability to get deterministic & reliable feedback from from test execution

Summary

Depending on your project strategy, scope, manual or automation requirements and of course, the hardware or infrastructure combinations, you should make a choice that not only suits the requirements but gives you the best returns and results. 

Based on my past experiences, I am very excited about the Applitools Ultrafast Test Cloud – a unique way to scale test automation seamlessly. In the process, I ended up writing less code, and got amazingly high test coverage, with very high accuracy. I recommend everyone to try this and experience it themselves!

Get Started Today

Want to get started with Applitools today? Sign up for a free account and check out our docs to get up and running today, or schedule a demo and we’ll be happy to answer any questions you may have.

Editor’s Note: This post was originally published in January 2022, and has been updated for accuracy and completeness.

The post What is Cross Browser Testing? Examples & Best Practices appeared first on AI-Powered End-to-End Testing | Applitools.