Hello everyone! I’m Noam, an SDK developer on the Applitools JS-SDKs team. While my day-to-day focus is on core engineering, I work closely with our field teams and occasionally join technical deep-dive sessions with customers.

In these conversations, we frequently encounter questions about performance and the engineering philosophy behind our integration. Specifically, there is often curiosity about how to make visual testing feel more “Playwright-native” and natural to developers.

In this post, I’ll share the design logic behind these architectural choices so you can apply these patterns in your own CI pipelines in a way that fits your organization’s needs.

Adding unresolved to Playwright

Integrating visual regression testing into Playwright requires combining two different status models: Playwright’s binary Pass/Fail and the visual testing concept of unresolved.

In visual testing, instead of having two (passed and failed) states, there’s an additional third state: unresolved. This state indicates a difference was detected, but a human decision is required to determine if it is a bug or a valid change that should be approved as a new baseline.

​Playwright doesn’t support this third state out of the box. Visual test maintenance using Playwright’s native toHaveScreenshot API forces the developer into a cumbersome cycle requiring three separate test executions:

1. First, the developer needs to run to see the failure.
2. Then, they need to run with the --update-snapshots flag to create new baseline images.
3. Then, most developers would run again to validate that everything works with the updated baseline as expected—which isn’t always the case, because the Playwright native comparison method (pixelmatch) tends to be very flaky, unlike Visual AI.

​After this local cycle, the developer must commit the new baseline images to the repository—bloating the git history—and wait for a new CI execution to provide final feedback. For dev-centered organizations that focus on feedback loop velocity, this workflow is… suboptimal. Personally, I believe that’s one of the reasons visual testing isn’t as popular as it should be among Playwright users.

​When we engineered the Applitools fixture, one of our goals was to support this Unresolved state natively, without disrupting Playwright’s core lifecycle—specifically its Worker Processes and Retry mechanisms.

The solution rests on two key engineering decisions: moving rendering to the background (async architecture) and giving developers control over the exit signal and performance tradeoffs (failTestsOnDiff).

We don’t block test execution when Applitools is rendering

The core value of visual testing lies in AI-based comparison to eliminate false positives and multi-platform rendering.

Architecturally, these processes are cloud-native services.

• AI-as-a-Service: Just like massive LLMs or other generative models, the Visual AI engine runs on specialized cloud infrastructure optimized for heavy inference. It cannot simply be “installed” on a lightweight CI agent.
• Platform Constraints: Authentic cross-platform rendering (e.g., iOS Safari on a Linux CI agent) is physically impossible on a single local machine.

Since these operations inherently occur remotely, performing them synchronously would force the local test runner to idle while waiting for network round-trips and cloud processing.

To solve this, we designed the fixture around an asynchronous architecture:

• ​Instant Capture: When eyes.check() is called, we synchronously capture the DOM and CSS resources (instead of a rasterized image). This operation is extremely fast.
• ​Immediate Release: We purposefully use soft assertions by design. We release the Playwright test thread immediately so the functional logic can proceed to the next step or test case without blocking.
• ​Background Heavy Lifting: The heavy work—uploading assets, rendering across different browsers and operating systems, and performing the AI comparison in the Applitools cloud—starts immediately in the background, managed by the Worker process.

The “Draining Queue” Effect

​This architecture explains why the Playwright Worker sometimes remains active after the final test completes.

​The background tasks are limited only by your account’s concurrency settings, and the screenshot size. For example, when rendering a 10,000 px page on a small mobile device, the rendering infrastructure might need time for scrolling and stitching. If your functional tests execute faster than the background workers can process the queue (rendering & comparing), the Worker process stays alive at the end solely to “drain the queue” and ensure data integrity.

While it does ensure your test logic runs at maximum speed, offloading the processing cost to the background, this experience might cause friction and frustration as the developers see that workers are “hanging” after tests are completed. When facing such issues, our support team is here to advise and assist with various solutions—we can investigate execution logs and if needed even make custom suggestions to tailor Eyes-Playwright to your needs.

Solving the Matrix Problem

​Standard Playwright documentation recommends defining multiple projects in playwright.config.ts to cover different browsers (Chromium, Firefox, WebKit) and various viewport sizes.

​While this ensures coverage, it introduces a linear performance penalty (O(N)). To test three browsers across two viewports, your CI must execute the functional logic (clicks, waits, navigation) six times. It’s 6x more load on the CI machine and the testing environment.

​We recommend shifting this workload to the Ultrafast Grid (UFG).

​In this mode, you execute the Playwright test once, typically on Chromium. We upload the DOM state, and our cloud infrastructure renders that state across all configured browsers and viewports in parallel.

This transforms an O(N) execution problem into an O(1) execution problem, significantly shortening the feedback loop.

The Strategy: failTestsOnDiff

​Since the actual comparison happens asynchronously and potentially completes after the test logic finishes, we need a mechanism to map the visual result back to the Playwright status.

​This is controlled by the failTestsOnDiff flag. It’s not just a boolean; it’s a strategic choice for your CI pipeline.

​Strategy A—Recommended for CI: failTestsOnDiff: false

• ​The Logic: This is the configuration our own Front-End team uses. We believe that Visual Change ≠ Test Failure.
• ​Behavior: The Playwright test passes (Green). The unresolved status is reported externally via our SCM integration (GitHub/GitLab).
• ​Why: Retrying a visual test is computationally wasteful—the pixels won’t change on the second run. By keeping the test “Green,” we avoid triggering Playwright’s retry mechanism. The decision is moved to the Pull Request, where it belongs.

Read more about SCM integration or hop directly to our GitHub, Bitbucket, Gitlab or Azure Devops articles.

Strategy B—Recommended for Strict Pipelines: failTestsOnDiff: 'afterAll'

• ​The Logic: You need a “Red” pipeline to block deployment, but you want to avoid the noise of retries and gain a significant performance improvement.
• ​Behavior: Individual tests pass, but the Worker Process exits with a failure code if any diffs were found in the suite.
• ​Why: This provides a hard gatekeeper for the build status. It allows the Eyes rendering farms to continue processing visual test results in the background without blocking the execution thread, allowing the worker to move on to handle more tests efficiently.

​Strategy C—Recommended for Local Debugging: failTestsOnDiff: 'afterEach'

• ​The Logic: Immediate feedback loop.
• ​Behavior: Fails the test immediately in the afterEach hook.
• ​Why: Best for local development where you want to see the failure immediately in the console. It is also useful if you use the trace: retainOnFailure setting in Playwright, as it ensures traces are preserved for unresolved visual assertions. Not recommended for CI due to the retry loops described above.

TL;DR – When to use each setting

Closing the Visibility Gap: The Custom Reporter

​If you adopt Strategy A (false) or Strategy B (afterAll), you introduce a secondary challenge: Visibility.

Since Playwright technically marks these tests as Passed to avoid retries, the standard Playwright HTML Report will show them as “Green,” potentially masking unresolved visual differences that require attention.

​To bridge this gap without forcing developers to switch context, we developed a Custom Applitools Reporter.

​This reporter extends the standard Playwright HTML report. It injects the actual visual status (Passed, Failed, or unresolved) directly into the test results view.

• ​True Status: You see which tests have visual diffs, even if the Playwright exit code was successful.
• ​Direct Links: It provides a direct link from the test report to the specific batch results in the Applitools Dashboard.
• ​Context: It enriches the report with UFG render status and batch information.

​This ensures you get the best of both worlds: The optimization of a “Green” CI run (no retries), with the transparency of a report that highlights exactly where manual review is needed.

Summary

​The Applitools Playwright fixture is designed to be non-blocking and scalable. By leveraging asynchronous architecture and Applitools UltraFast Grid, we offload the heavy lifting from your CI. By correctly configuring failTestsOnDiff, you ensure that your pipeline reflects your team’s engineering culture—whether that’s strict gating or modern, PR-based visual review.

The post Engineering a Playwright-Native Developer Experience: One Flag, Three Strategies appeared first on AI-Powered End-to-End Testing | Applitools.

TL;DR

• Reducing test maintenance and improving result accuracy are becoming core evaluation criteria for autonomous testing platforms
• Visual validation is increasingly used to ensure UI accuracy across web, mobile, and native applications
• These capabilities help teams maintain release confidence and reduce risk in complex and dynamic, user-facing experiences at scale

Modern software teams ship faster than ever, and testing teams need tooling that keeps up. In Q4 2025, Forrester published The Forrester Wave: Autonomous Testing Platforms, Q4 2025, evaluating autonomous testing platform providers.

Applitools is named a Strong Performer in this evaluation.

The momentum behind autonomous testing

Teams now build and ship across more devices, frameworks, and release cadences. That reality pushes quality practices toward higher automation, better maintenance efficiency, and faster feedback loops.

Forrester frames this market shift directly:

“This is why we changed this Forrester Wave category from ‘continuous automation testing platforms’ to ‘autonomous testing platforms.’”

The Forrester Wave: Autonomous Testing Platforms, Q4 2025, Forrester Research, Inc., Q4 2025.

What buyers should look for in autonomous testing platforms

When you evaluate autonomous testing platforms in 2025, three practical questions usually help teams make sense of the space:

• Platform fit: Can the platform support your mix of apps and test types, plus your workflows across engineering and QA?
• AI-infused automation: Does the platform reduce authoring and maintenance effort in a way you can trust and govern?
• Testing AI-enabled experiences: As more teams ship AI-enabled features, can your testing approach keep pace with new failure modes and higher variability?

These questions help teams connect product capabilities to real delivery constraints: speed, coverage, confidence, and operating cost.

How the report characterizes Applitools

This report describes Applitools’ approach through Visual AI and ML-resilience oriented toward UI accuracy and maintenance reduction:

“(Applitools) It features Visual AI to validate UI accuracy across web, mobile, and native apps and support modern digital experiences at scale.”

The Forrester Wave: Autonomous Testing Platforms, Q4 2025, Forrester Research, Inc., Q4 2025.

It also cites a strategy emphasis on reducing maintenance and improving accuracy:

“Applitools stands out for innovation, gaining an above-par score due to its Visual AI and ML-driven resilience that reduce test maintenance and improve accuracy.”

The Forrester Wave: Autonomous Testing Platforms, Q4 2025, Forrester Research, Inc., Q4 2025.

What this can mean for engineering, QA, and design teams in 2025

Engineering teams can treat autonomous testing as a way to protect delivery speed. When teams reduce flaky failures and avoid constant test repairs, they shorten the path from code change to deployable signal.

QA teams can prioritize scalability and governance. As test suites grow, teams need tools and workflows that improve coverage without creating unsustainable maintenance load.

Design teams can connect UI intent to release confidence. When teams validate UI accuracy consistently across browsers, devices, and releases, they reduce risk in UX-heavy, customer-facing journeys.

Across all three groups, teams can get more value when they align on what “quality” means for the product and then choose automation approaches that enforce that definition consistently.

Read the report

While you’re evaluating autonomous testing priorities for 2025, read the full report to understand the evaluation criteria, methodology, and vendor profiles in context.

Forrester does not endorse any company, product, brand, or service included in its research publications and does not advise any person to select the products or services of any company or brand based on the ratings included in such publications. Information is based on the best available resources. Opinions reflect judgment at the time and are subject to change. For more information, read about Forrester’s objectivity here.

The post Applitools Named a Strong Performer in The Forrester Wave™: Autonomous Testing Platforms Report, Q4 2025 appeared first on AI-Powered End-to-End Testing | Applitools.

TL;DR

• Code-centric automation continues to slow teams down as UI changes multiply, making stability and evidence hard to maintain.
• AI code generators don’t solve the problem because they still produce brittle test code that requires constant oversight.
• Live LLM-driven execution introduces unpredictability. Regulated teams need deterministic runs, not improvisation
• A clearer path is intent-driven authoring paired with deterministic engines and Visual AI that detects visual drift and preserves audit-ready evidence.

Request our Governance Readiness Checklist

Teams in regulated environments face a familiar strain. Applications grow in complexity, expectations for fast releases keep rising, and every update requires clarity about what changed and whether required elements still appear as intended. Traditional automation wasn’t built for that pace or level of oversight, and the recent wave of AI coding tools hasn’t solved the core challenges.

A better model is emerging—one that uses AI to reduce the workload of authoring and maintaining tests while keeping execution deterministic, reviewable, and aligned with how people evaluate digital experiences.

This post breaks down why the legacy testing model is hitting its limits and how AI can support a more stable, more trustworthy approach.

Why traditional automation keeps slowing teams down

As digital experiences expand across pages, portals, member journeys, and product flows, test code becomes difficult to scale. Even minor UI changes break locators and assertions, creating unpredictable test runs, delayed reviews, and long maintenance cycles.

Developers are often asked to take on more of the testing responsibility. While this can improve feedback loops, it does not reduce the burden of maintaining code that reacts poorly to UI changes. And when teams already lack time, context switching between product development and test diagnostics becomes expensive.

The result is a predictable bottleneck: too many tests tied directly to implementation details and not enough stability across releases.

Why AI-generated test code hasn’t fixed the problem

The last few years have produced a surge of tools that promise to generate automation code automatically. But teams report the same issues repeating in a new form. LLMs can produce code quickly, yet the resulting output still inherits all the maintenance challenges of coded automation.

AI code generators also excel more at producing new code than updating existing flows. They struggle with assertions, hallucinate element behavior, and require human supervision to validate every step. For regulated teams that must show repeatability and generate evidence for every release, inconsistency becomes a risk rather than a convenience.

If the goal is to escape brittle code, producing more of it is not the answer.

Why live LLM-driven execution creates instability

Another idea gaining attention is allowing an LLM to operate the UI directly during test execution. In theory, this removes the need to write code. In practice, teams quickly run into new risks: undefined steps, inconsistent interactions, slow decision-making, and no reliable way to debug.

Execution in regulated environments must be predictable. It must be reviewable. And it must produce evidence that can be traced, explained, and defended. Live improvisation during a test run undermines each of these requirements.

Determinism matters more than novelty. A testing approach must produce the same result today, tomorrow, and during an audit review.

A clearer path forward: intent-driven authoring with deterministic execution

A more reliable model is emerging that uses AI to simplify authoring without relying on AI to make real-time decisions during execution.

Teams describe test intent in natural language. An AI system translates that intent into structured steps during authoring, where humans can review and adjust. Execution is then handled by deterministic engines and Visual AI that observe the rendered UI and detect visual changes, required-element presence, placement consistency, and contrast.

This separation delivers two advantages:

• People write and maintain far fewer lines of test code
• Test runs become stable, repeatable, and easier to verify

Visual AI provides a complete view of the screen state and compares each run against an approved baseline. When something changes, the system surfaces the difference, captures evidence, and supports reviewer approvals. When the change is expected, one acceptance updates the baseline and applies it across browsers and devices.

The outcome is a testing layer that is easier to maintain and easier to trust.

What this looks like in practice

Teams adopting this approach typically see changes across several parts of their workflow:

• Tests are written in plain language, without selectors or framework setup
• Visual AI validates full screens for layout, presence, placement, and readability
• Changes are highlighted automatically to reduce manual inspection
• Evidence is captured through screenshots, diffs, timestamps, and logs
• Debugging takes place in an environment where runs behave the same every time
• Reusable flows and data-driven steps integrate into the same natural-language format

Instead of managing a growing volume of fragile code, teams maintain intent-level descriptions supported by deterministic execution.

What this means for oversight and compliance

For teams in financial services, healthcare, insurance, or life sciences, the benefits go beyond efficiency.

A visually grounded testing model helps confirm that required notices, disclosures, language-access elements, and other regulated UI content remain present and placed as expected. It documents what changed and preserves evidence for review. It supports consistent experiences across browsers, devices, and PDFs without checking whether values, data, or regulatory text are correct.

Most importantly, it delivers predictable results.

Regulated environments depend on clarity and traceability. When every test run yields reviewable outputs, and every change is captured with context, teams can maintain confidence and release with speed.

If you’re assessing how well your testing workflow supports stability and audit readiness, request our Governance Readiness Checklist. We’ll share the version designed for your stage—whether you’re evaluating Applitools or optimizing an existing deployment.

The post AI Testing in Regulated Environments: Smarter Testing Starts With Stability, Not More Code appeared first on AI-Powered End-to-End Testing | Applitools.

Why Test Maintenance Breaks at Scale

Test maintenance at scale slows releases. Teams that rely on coded assertions spend more time updating tests than improving coverage. Brittle locators, environment drift, and false positives all add up—turning automation into a maintenance cycle.

Neglecting maintenance is like skipping car care: small issues snowball into costly downtime. A smarter approach replaces manual review and locator-based scripts with automated, visual validation that adapts as your UI evolves.

How Visual AI Delivers Test Maintenance at Scale

Visual AI replaces dozens of coded assertions with a single checkpoint that mimics how humans see. It validates full UI states, detecting layout shifts, missing elements, and text overlaps automatically.

By consolidating validations into one Visual AI check, teams cut review time, reduce false positives, and gain faster feedback cycles.

Scale Reviews with Ultrafast Grid and Grouping

Running tests one browser at a time no longer scales. The Applitools Ultrafast Grid executes a single test once, then validates results across every browser and device combination in parallel.

Batching and grouping features make reviews equally efficient—approve or reject similar changes across entire runs in just a few clicks.

How it works

• Replace assertions with one visual checkpoint
• Run once across all browsers and devices
• Batch results for unified review
• Approve or reject in bulk
• Tune match levels for dynamic content

Together, these capabilities eliminate redundant effort and make large-scale testing faster to maintain.

Customer Results: 78% Less Maintenance

Teams that adopt this approach see measurable ROI. At Peloton, replacing a legacy visual testing tool with Applitools Visual AI produced a 78% reduction in maintenance time and saved about 130 hours per month.

With dynamic leaderboards, live data, and responsive layouts across web and native mobile, Peloton maintains quality at scale without expanding test overhead.

Three Features That Change Maintenance

“Ultrafast Grid, Visual AI match levels, and bulk grouping—those three change the game.”

Mike Millgate, Smarter Test Maintenance at Scale

These three deliver flexible validation, fast execution, and effortless maintenance. Each removes manual steps and accelerates the feedback loop that keeps releases reliable.

Smarter Maintenance for Modern Teams

Smarter test maintenance isn’t about writing more code—it’s about automating intelligently. Visual AI reduces flakiness, speeds reviews, and scales across devices and environments.

To see what’s next, explore Applitools Eyes 10.22, featuring faster review cycles, new Storybook and Figma integrations, and even shorter feedback loops for test maintenance at scale.

The post Test Maintenance at Scale: How Visual AI Cuts Review Time and Flakiness appeared first on AI-Powered End-to-End Testing | Applitools.

The excuse “we don’t have time to write tests” doesn’t hold up anymore. AI has reshaped the way teams approach software testing, making it faster, smarter, and more accessible than ever. Tools like GitHub Copilot, ChatGPT, and Applitools Autonomous can generate reliable automated tests without slowing down your development flow.

If you’ve ever struggled with limited testing resources or hesitated to adopt AI-enhanced workflows, now is the perfect time to embrace AI-powered testing.

How GitHub Copilot Helps Accelerate Unit Test Creation

GitHub Copilot can dramatically speed up unit test creation. It can generate unit tests directly in your editor with a single prompt. For example, typing “create unit tests for Hello.tsx” in VS Code can instantly produce functional test cases using React Testing Library.

While Copilot’s first drafts were impressive—correctly using accessible locators and matching key UI elements—it’s important to note that AI-generated tests often require slight refinements.

Expecting a one-shot from AI is probably unrealistic—but in my experience, it gets you pretty darn close.

Copilot typically picks up on your dependencies, infers structure, and outputs readable, executable tests. If the results aren’t perfect, for instance, using fragile selectors or inconsistent naming, you can quickly iterate. Adjusting your prompt often resolves these issues. In many cases, reprompting is faster than manual edits.

Accessible locators and consistent naming can be enforced through clearer prompting or by storing preferences in a centralized configuration file

The key? Good prompts make a big difference. Prompting Copilot to use best practices, like favoring accessible selectors, resulted in much cleaner and more reliable output.

Taking Testing Further with Playwright and Copilot

Beyond unit tests, AI can support end-to-end testing for full user flows. Using Copilot with a framework like Playwright, you can prompt test generation by simply referencing a live URL and desired interactions.

For example, pointing Copilot to a public demo app like TodoMVC and requesting end-to-end tests will often result in tests for adding, completing, deleting, and filtering tasks—all without writing code manually.

To further improve coverage, ChatGPT can help by generating a requirements document for the app. This doc acts as a guide to ensure tests align with expected behaviors.

The better the input we provide the LLM, the better output we’re likely to get. A requirements doc is a really important piece of input.

Once the requirements are defined, you can direct the AI to use them when generating tests, producing more complete and targeted coverage. Just remember to include your preferences for things like locator strategy and naming conventions in your prompt or project config.

The message is clear: Combining ChatGPT and Copilot creates a powerful AI-assisted workflow for test generation. This approach cuts down on manual scripting while improving test depth.

Boosting End-to-End Testing with Applitools Autonomous

Applitools Autonomous handles creating automated tests with AI differently. Instead of writing code or interacting with the DOM, you provide a URL, and the system automatically scans the app. It generates visual and functional tests and organizes results into a centralized dashboard.

Highlights of what Autonomous can do include:

• Crawl an entire application from just a URL and automatically generate visual and functional tests
  
• Use plain English commands to create, edit, and validate tests (no coding needed)
  
• Validate UI, behavior, and API responses in one workflow
  
• Capture dynamic data like confirmation IDs, verify API responses, and support parameterization without code

Unlike traditional recording tools, Autonomous intelligently builds stable, scalable tests while seamlessly validating across browsers. It even flags hidden 404 errors—showcasing the tool’s ability to catch issues early.

Another key point is that anyone, regardless of technical background, can create sophisticated tests using natural language. At the same time, it maintains the depth and flexibility senior developers demand.

Key Takeaways for Modern Testing Workflows

Today’s AI software testing tools are designed for real-world developer needs:

• Copilot accelerates unit and E2E test creation with natural language prompts.
• ChatGPT fills documentation gaps by drafting requirements for better test coverage.
• Applitools Autonomous redefines E2E testing, combining visual validation and functional flows—from UI to visual to API—and plain-English test authoring. It integrates these into a single, no-install SaaS platform.

AI doesn’t replace the tester’s critical thinking — it augments your workflow, helping you focus on improving test quality, not just checking boxes.

In Summary…

The landscape of automated testing is still evolving. With tools like Copilot, ChatGPT, and Applitools Autonomous, building and maintaining high-quality automated tests no longer has to be a slow, painful process. Whether you’re a front-end engineer, QA lead, or tech manager, adopting AI-powered workflows will free up your team’s time. It will increase your confidence in releases and bring better quality to every sprint.

Want to learn more about how to create automated tests with AI? Watch the full session on demand to see in-depth demos.

The post Creating Automated Tests with AI: How to Use Copilot, Playwright, and Applitools Autonomous appeared first on AI-Powered End-to-End Testing | Applitools.

We’re proud to share that Applitools has been named AI-Powered Test Automation Platform of the Year 2025 by CIO Review.

Selected by a panel of C-level executives, industry thought leaders, and the editorial team at CIO Review, this recognition highlights the meaningful progress we’re making toward truly intelligent, AI-driven testing.

“We see this as validation of our vision—to move testing beyond automation and toward intelligent systems that know what to test, when, and why.” – Alex Berry, Applitools CEO

At Applitools, our mission is to help teams ship high-quality software with greater speed and confidence. From Visual AI to Applitools Autonomous, our Intelligent Testing Platform is designed to reduce test maintenance, streamline workflows, and help teams scale testing without scaling complexity.

Read the full feature article.

As we continue evolving what’s possible in software testing, we’re honored to be recognized by industry leaders who are shaping the future of technology.

The post Applitools Named AI-Powered Test Automation Platform of the Year by CIO Review appeared first on AI-Powered End-to-End Testing | Applitools.

Efficient software testing plays a crucial role in the digital financial industry, where customer trust relies on reliable systems and smooth user experiences. AI-powered tools are revolutionizing software testing for online banking. Learn how Applitools’ Autonomous Testing Platform, an AI-driven testing solution, can boost efficiency and ensure seamless functionality for digital banking platforms.

AI-Powered Testing for Visual Verification

Modern interfaces are becoming more dynamic, making software testing more challenging than ever. Applitools uses AI-powered end-to-end testing to deliver accurate, human-like visual validation. With this advanced approach, teams can easily detect visual bugs, handle dynamic content, and solve multi-device rendering issues. Compared to traditional testing tools, Applitools’ AI testing solution offers a faster, more reliable way to ensure seamless user experiences across all platforms.

Visual AI “sees” applications as a human would, identifying not just design inconsistencies, but functional issues within complex environments. This human-eye accuracy is especially critical for financial applications, where misdisplayed data or minor UI errors can compromise user trust.

Visual Verification with AI gives QA teams:

• Precise identification of visual bugs, such as misaligned content or missing elements.
• Compatibility across a range of devices, screen sizes, and browsers.
• Reduced false positives, ensuring only actionable defects are flagged.

Challenges in Automated Testing for Financial Apps

Some of the key challenges QA teams face in the financial services industry include testing data-heavy dashboards, personalized user experiences, and meeting strict compliance requirements. QA teams also navigate the complexity of testing constantly changing data, such as account balances and transactions, while ensuring seamless functionality across multiple devices and screen sizes. These unique testing challenges highlight the importance of effective QA processes in delivering reliable, user-friendly financial services.

Adding to these complexities are regulatory requirements, like accessibility compliance mandated by guidelines such as the European Accessibility Act. Traditional tools often fall short in dynamically adapting to such changes, leading to increased bottlenecks.

Common Obstacles Faced by QA Teams:

• Exponential growth in test scenarios due to dynamic UI states.
• Maintaining coverage amid rapid deployment cycles.
• Lacking tools to validate compliance and localization needs.

The Role of Visual AI and Autonomous in Testing

Applitools’ Visual AI is transforming the way teams approach UI testing. Developed with 11 years of research and development, this powerful technology compares both visual elements and DOM structure to deliver comprehensive UI validation. Key features like automated baselining and self-healing tests make maintaining tests easier, especially during frequent code updates or UI changes.

The Autonomous platform takes Visual AI testing to the next level by integrating it with end-to-end test workflows. The platform simplifies visual, functional, API, and accessibility testing. With Autonomous, teams can automate testing, schedule tests, validate results, and efficiently manage issues—all from a user-friendly interface.

Features of Visual AI & Testing with Autonomous:

• Self-healing capabilities to adapt tests to UI updates without manual adjustments.
• Natural language authoring that allows easy collaboration between technical and non-technical team members.
• Supports functional flows like login processes while ensuring personalized data fields meet validation patterns.

Efficiency and Coverage Improvements

A top US bank improved testing efficacy using Applitools Autonomous. By adopting Visual AI and scaling to test its entire ecosystem, the bank achieved:

• 5x Test Coverage Expansion across all devices and browsers.
• 35% More Defects Caught weekly within earlier production stages.
• Up to 999 Hours Saved Per Release by reducing manual test generation and maintenance efforts.

This expansive coverage allowed the team to uncover bugs that would otherwise have been missed, delivering higher-quality software quicker.

Visit the event archive to see more of the case study and watch the full session on-demand.

Security and Integration Features

Data security and adaptiveness in distinct environments are primary concerns in financial testing. The webinar addressed how Applitools ensures client protection with private cloud implementations and a secure architecture for test data management. The platform seamlessly integrates with firewalls and various development ecosystems, making it an ideal choice regardless of organizational infrastructure.

Ensuring Data Privacy and Flexibility:

• Test data secured using encryption and private cloud options.
• Compatible with diverse environments, both behind firewalls and in SaaS architectures.
• Allows scalability without compromising compliance adherence.

Strengthen Financial Application Testing

AI-assisted testing is helping overcome the hurdles posed by traditional testing. By integrating Visual AI with a broad end-to-end testing workflow, financial institutions can drastically enhance their QA practices. More importantly, these improvements do not require trade-offs between coverage, speed, and accuracy.

If you’re ready to elevate your testing game, particularly in high-demand sectors like online banking, platforms like Autonomous present an opportunity you can’t afford to miss.

Explore the full webinar on-demand or reach out to learn how Applitools Autonomous can streamline your QA efforts and ensure the flawless delivery of digital experiences.

The post End-to-End Testing Solutions for Online Banking Software Applications appeared first on AI-Powered End-to-End Testing | Applitools.

Modern ecommerce applications are more than digital storefronts. They’re immersive, complex experiences designed to captivate and engage customers—customers that spent well over $1 trillion in 2024 (over $220 billion during seasonal shopping surges).

Ecommerce applications are now multifaceted ecosystems that require high reliability, fast updates, and seamless performance across devices. With features like multi-step checkout, dynamic promotions, and personalized recommendations, these applications present a unique challenge for testing teams: ensuring consistent quality in an environment of constant change.

From adaptive content testing to streamlined cross-platform validations, AI-powered testing helps software test and development teams deliver personalized, high-quality experiences to their shoppers.

Step 1: Addressing the Core Challenges in Ecommerce Application Testing

To understand how AI-powered testing can enhance quality, we first need to think about the primary complexities in ecommerce applications:

• Dynamic Content
  Promotions, recommendations, and A/B testing create a dynamic landscape where content changes rapidly, which can lead to issues in test reliability.
• Multifaceted User Journeys
  A typical user flow can include browsing, filtering products, adding items to a cart, and various checkout steps, each with variations like promotional codes and payment methods.
• Cross-Platform Consistency
  Ensuring a consistent experience across mobile, desktop, and tablet adds another layer of complexity, requiring cross-device and cross-browser validations to prevent regressions. 

These challenges make e-commerce application testing labor-intensive, especially for teams relying on traditional methods–manual or automated.

Step 2: Leveraging AI to Manage Complexity in Ecommerce Testing

Let’s explore how AI-powered testing can help address the common pain points of ecommerce application testing:

Autonomous Testing for Comprehensive Flow Coverage

Autonomous testing provides broad page coverage by automatically creating visual tests for each page from a single URL. For more complex, critical user journeys, custom flow tests blend human oversight with AI assistance to ensure thorough validation.

AI tools, like Applitools Autonomous, can learn the structure of complex user journeys—like multi-step checkouts or specific promo code applications—and adapt to new scenarios with minimal manual intervention. For developers, this means reduced time writing scripts and a faster feedback loop, as the AI detects critical paths and validates user interactions autonomously.

Test Tip: Use autonomous testing to cover frequently changing paths like checkout, where many variations (promo codes, payment methods) need to be validated with each update.

Visual AI for Dynamic Content Testing

AI-powered visual testing focuses on meaningful changes, helping to identify only the differences that impact user experience. Traditional test automation often flags minor layout shifts that don’t affect usability, leading to unnecessary noise. On the other hand, Visual AI detects layout issues that matter, minimizing false positives and simplifying testing for high-variance content.

Use Cases: Visual AI is particularly effective in handling promotional banners, personalized ads, and product recommendations. It ensures that these elements render correctly without producing redundant alerts.

Cross-Browser and Cross-Device Testing

Ensuring consistency across devices and browsers is crucial in ecommerce. Testing with a tool, teams can validate both functionality and appearance across multiple environments in parallel, enabling broader test coverage with significantly reduced runtime.

Tool Tip: For each release, use the Ultrafast Grid to run tests across all targeted browsers and devices simultaneously. By testing in parallel, you can catch visual and functional discrepancies early, ensuring consistent experiences across your audience’s preferred devices.

AI for Personalized and Edge Case Testing

AI-powered testing can adapt to personalized and localized content variations. This allows you to verify account-based and user-specific scenarios, like tailored recommendations, language translation, and geolocation-based offers. The AI dynamically generates test scenarios for real-world edge cases, validating content presentation, accuracy, and functionality across users.

A/B Testing and Experimentation

AI testing tools can run tests alongside A/B experiments to quickly validate different scenarios and identify potential issues without extensive setup or downtime. Integrated into CI/CD pipelines, AI can seamlessly support experimentation, allowing you to test new features or changes with agility.

Test Tip: Use AI for A/B testing by validating that both variations display correctly and function as expected. This can streamline experiment verification and minimize risk.

Accessibility Compliance with AI

Accessibility is key to ensuring inclusivity for all of your shoppers when you consider that 4.4% of the world’s population is colorblind. Even more shoppers have contrast sensitivity issues (we don’t know about you but some of us are just getting old). AI-driven testing identifies accessibility issues early, such as missing alt text, improper color contrast, and navigation difficulties. With accessibility testing integrated into automated pipelines, developers can confidently meet compliance requirements while optimizing the experience for users with disabilities.

Step 3: Applying AI Testing in Real-World Ecommerce Scenarios

Ensuring that every scenario receives accurate validation across all paths and user types can be hard but your team can quickly improve test coverage by integrating AI into the below scenarios:

• Multi-Path Checkout Testing
  AI testing tools validate complex checkout flows, including various payment methods, promo codes, and cart adjustments. This reduces the risk of abandoned carts due to broken checkout paths and ensures seamless user experiences.
• Product Filtering and Sorting
  Filtering and sorting are essential for user navigation. AI testing validates the functionality and accuracy of filters by testing different combinations. This ensures the results match user criteria and display correctly.
• Personalized Content Validation
  Personalized recommendations, banners, and product listings are critical for user engagement. AI-powered testing verifies that these elements display correctly based on user data, ensuring consistency without disrupting the layout.

Step 4: Getting Started with AI-Powered Testing

To maximize the benefits of AI in ecommerce application testing, you can follow these best practices:

• Integrate AI into your CI/CD Pipeline
  Embedding AI testing into the CI/CD process ensures that every change undergoes rigorous testing before reaching production. This allows for fast iteration and high-quality releases.
• Prioritize Dynamic and High-Risk Areas
  Focus AI testing on paths that involve dynamic content, personalized features, and checkout flows. These areas tend to have the highest variance and the most critical impact on user experience.
• Leverage Parallel Testing
  Run cross-platform tests in parallel to streamline testing cycles and get immediate insights into how changes impact various devices and browsers.
• Use AI for Regression and Smoke Tests
  Routine regression and smoke testing can become resource-intensive. By automating these with AI, you free up resources for more complex testing, ensuring each release is stable.

To Wrap Up…

As AI becomes an integral part of the testing process you can shift your focus toward building new features and enhancing shopper experiences by maintaining a high standard of quality:

• Reduced Manual Work and Faster Releases
  By automating repetitive tasks and handling edge cases, AI-powered testing allows QA teams to focus on high-impact scenarios, reducing time to market.
• Increased Test Coverage and Accuracy
  AI’s ability to adapt to dynamic content and complex scenarios broadens test coverage and increases the accuracy of results, ultimately reducing production issues.
• Adaptability to Ongoing Changes
  AI-powered testing lets ecommerce sites handle constant updates and seasonal changes without extensive reconfiguration so that shoppers have a seamless experience.

AI-powered testing is now an essential strategy for ecommerce test and dev teams to meet user expectations and stay competitive. You can try out these strategic steps with a free trial of Applitools Autonomous.

The post A Guide to AI-Powered Ecommerce Application Testing appeared first on AI-Powered End-to-End Testing | Applitools.

Quick Answers

The post Top 10 Visual Testing Tools appeared first on AI-Powered End-to-End Testing | Applitools.

In our recent webinar, Building the Ideal CI/CD Pipeline: Achieving Visual Perfection, we explored the transformative power of Visual AI testing for CI/CD pipelines. Aimed at software engineering managers and team leads, the session provided a deep dive into the limitations of traditional functional testing and how Visual AI testing can surpass these to achieve visual perfection in software development.

Technical Customer Success Manager Brandon Murray shared expert strategies and highlighted the benefits of integrating Visual AI testing, offering guidance on constructing the optimal CI/CD pipeline. He explored the intricacies of Visual AI testing, illuminating its critical role in enhancing software quality and performance.

Challenges in Traditional Functional Testing

Murray began by identifying the bottlenecks commonly encountered in traditional functional testing. These include:

• High maintenance efforts
• Slow feedback cycles
• Limited UI coverage
• Tedious manual testing

The Power of Visual AI Testing

Visual AI testing offers a revolutionary approach to overcome these challenges. By capturing screenshots and using AI to compare these snapshots to a baseline ‘golden image’, Visual AI testing ensures:

• Reduced Test Development and Maintenance Time: Automating UI comparisons dramatically decreases the time spent on writing and maintaining tests.
• Complete UI Coverage: Screenshots ensure that every aspect of the UI is tested, eliminating blind spots.
• Enhanced Operational Efficiency: Faster feedback loops lead to quicker identification and resolution of issues, facilitating faster product releases.

Other Strategies to Supplement Visual AI Testing:

• Self-Healing: Automatically corrects flaky tests by adjusting for locator changes, vastly improving test stability
• Lazy Loading: Helps to ensure the entire page content is loaded
• Parallel Test Execution: Enables the execution of multiple tests simultaneously, significantly speeding up the testing process

Integration into the Development Workflow

Integrating Visual AI testing into existing development workflows, particularly with pull request checks, is pivotal for agile environments. The webinar emphasized the importance of instant feedback for swift issue resolution, leading to accelerated development cycles.

Tools and Technologies Highlighted:

• Cypress: Innovative testing framework for both developers and QA engineers
• GitHub Actions: Continuous integration and continuous delivery (CI/CD) platform enabling automation directly in GitHub repositories
• Figma Designs: Useful for collaborative design reviews and direct comparison against implementations

The session underscored the cost-effectiveness of using browsers on cloud infrastructure containers, especially when dealing with cross-browser coverage. Notably, the Filter Fast Grid was mentioned as an effective solution for this purpose.

Comparing Visual AI Testing to Traditional Methods

Attendees were eager to learn how Visual AI testing compares to snapshot tests and other traditional methods. The webinar demonstrated how Visual AI testing offers:

• Greater Accuracy: By leveraging AI for pixel-perfect comparisons
• Higher Efficiency: Through automated and parallel testing routes

In particular, using commodity CI solutions like GitHub Actions or CircleCI was recommended for their affordability and versatility.

Building the Ideal CI/CD Pipeline: Achieving Visual Perfection highlighted the transformative potential of Visual AI testing in optimizing CI/CD pipelines. Software engineering managers and team leads are strongly encouraged to evaluate how AI-powered tools like Applitools can elevate their testing processes, enhance product quality, and expedite delivery timelines. For those interested, a free trial of Applitools is available to experience the benefits firsthand.

The post Recap: Building the Ideal CI/CD Pipeline appeared first on AI-Powered End-to-End Testing | Applitools.

Context Switching

As a result of various obstacles, team members must switch between issues in context. When the team switches context frequently, they are not working efficiently. To maintain focus, the appropriate adjustments should be made in this case. A huge reason for context switching is the process around fixing bugs in the development process. Sometimes the development process makes development teams use tools that make it hard to remain in the context of the development process. Most of the times, it is the testing cycle that leads to context switching due to the lack of adequate integrations across the development lifecycle. 

As you may have observed, there is one common thing that we found becomes an obstacle in achieving better engineering productivity metrics. The desire to drop cycle time and deploy frequently is usually done at a compromise of testing coverage. Eventually, it leads to more rework ratio and more context switching for the teams. Most teams that scaled their engineering process start by paying acute attention to the testing process. The idea is to automate what can be automated with tools that can allow developers to move faster.

Testing fast and at scale is the key to engineering efficiency. Spotify coined the term for this called “Quality at Speed”. To maintain Quality at Speed, a Quality Engineering counterforce is required. At Applitools, our customers have helped in achieving quality at ultrafast speed. Visual AI from Applitools provides you with the ability to extend human eyes on the testing process without having to increase the QA/Dev ratio in the team.  Some quotes from engineering managers that have used Applitools for building products include:

“Any engineering team can reduce the manual testing resource and time by at least 70%. It also avoids overloading of SDETs.”

Engineering Manager at Dunzo (India)

“At Pushpay, our success stems from a technology-forward culture which drives our behavior, how we solve problems, and what tools we use to solve them. Since partnering with Applitools over 5 years ago, we have been able to improve quality, gain productivity and thus save time and money. We could not be more pleased with the efficiency boost our team has experienced since adopting Applitools and more recently, the Ultrafast Grid.”

Engineering Manager at Pushpay (New Zealand)

Using visual AI to address these factors

If the above is something you wish to improve within your team, then you will be surprised that it takes just a few days to get to this degree of speed at quality. 

The picture below shows how Applitools integrates with your application.

I will not get into how to install Applitools, as that is fairly well described in the tutorials. This also includes how you can integrate Applitools within your CI/CD pipeline. In the remainder of this article, I will like to tell you about some great examples of improving these metrics using Applitools. 

Most efficient teams would start a day with the below dashboard. This gives you a comprehensive view of all your tests that have been executed across your entire coverage list. Having high coverage is made easy by using an ultrafast grid that reduces your rework ratio later for devices or browsers that may not have been included before. At last a number of re-work happens due to poor coverage of testing in the first phase of development. Of course there is an element of scope creep that leads to the re-work as well which can be easily avoided by involving cross-functional teams in the development process. Applitools provide a visual abstraction of your application that can be accessed by everyone on a GUI. This drastically brings down the areas of scope creep unless the requirements have totally changed by business.

When you are doing testing for high coverage it becomes important that you are not getting slowed down by the process of reviewing the bugs. A big reason why deployment frequency gets reduced is because of the time it takes to review and fix the issues. This is exactly where Visual AI plays a big role.

Visual AI also lets you troubleshoot the defects really quickly.

Finally, developers and testers can use the same platform which integrates seamlessly with Jira or their preferred communication channel for faster feedback. Email and Slack notifications help the team get the feedback fast without any context switching.

To conclude, the engineering manager needs to explore more deeply how engineering processes are structured as the engineering team grows. Businesses are demanding faster releases of standard quality products, and Visual AI is an effective method of improving both efficiency and coverage of testing.

Learn about more about visual AI with Applitools Eyes.

The post Improving Engineering Productivity with Visual AI appeared first on AI-Powered End-to-End Testing | Applitools.

The current state of testing

Software testing is often seen in businesses as a necessary evil at the end of the software development lifecycle to find issues before they reach production. Chris and I strongly agree that software quality is crucial to modern businesses to help to achieve modern businesses’ goals and needs to be thought of throughout the software development lifecycle.

Digital transformation and continuous delivery

The largest and most relevant companies today have embraced digital transformation and technology to run their businesses and meet their customers’ needs. To keep up with digital transformation, you need modern software development practices like DevOps and continuous delivery. DevOps requires continuous testing to be successful, but we’re seeing that reliance on manual testing is the biggest challenge that organizations face when adopting DevOps.

Software testing and quality assurance

The software world is changing, so we need to change how we deliver technology. That requires modern software development approaches, which requires modern software testing and software quality approaches. Thankfully, testing and quality are far more top of mind now than in previous years.

From the audience poll results from our webinar, we see that continuous delivery is here to stay. When asked how often do you deploy changes to production, over 40% of people stated that they deploy either daily or multiple times per day. This wasn’t the case 10 to 20 years ago for most organizations unless technology was the product. Now, daily deployments are pretty common for most organizations.

Low rates of test automation

However, we’re seeing that getting high test automation coverage is still a huge challenge. In the survey, 55% of the respondents automate less than half of their testing.

The numbers may have a bit of sample bias because Applitools users actually automate on average 1.7 times higher than other respondents. The responses align with anecdotal experience that a lot of organizations are still in lower test automation coverage around 20% to 50%.

Testing complexity and the amount of testing needed are going up increasingly, and this shows in our survey as well. More than 50% of the respondents test two languages or more, three browsers or more, three devices or more, and two applications or more.

With two thirds of the respondents saying that one of the hardest parts of testing UIs is that they are constantly changing, traditional automation tools can’t handle testing at that speed scale and complexity.

Using AI in testing

We know that there’s a lot of excitement around AI tools, and the survey shows that.

When asked What parts of the testing process are you supporting or planning on supporting with AI, test offering, test prioritization, test execution, test management, and visual regression we all mentioned. The top two answers among respondents were test execution and visual regression.

It’s important to remember that continuous testing is about more than just test automation.

While test automation is key to the process, you still need to incorporate other software testing and software quality practices like load testing, security, user experience, and accessibility.

The future of testing

What we’re trying to achieve in the future of testing is to support modern software development practices. The best way we’re seeing to do this is to have software testing and software quality more tightly integrated into the process. Let’s talk about what a modern software approach looks like.

Testing sooner and more frequently

To get tighter integration of quality into the process, testing can’t just be an activity at the end of the development lifecycle. Testing has to happen continuously for teams to be able to provide fast feedback across disciplines and ensure a quality product. When this is done, we see increased speed not just of testing, but of overall software development and deployment. We also see reduced costs and increased quality.  More defects are found before production, and we see quicker responses when finding defects in production.

Increasing how much testing is automated

Traditional testing approaches tend to be done mostly manually. Increasing test coverage doesn’t mean manual testing goes away.

Automating your test cases frees up time to do more exploratory testing. Exploratory testing should be assisted by different tools, so AI has a good role to play here. Tools like ChatGPT are useful to brainstorm things like what to test next. Obviously we want to increase test automation coverage at all levels, including unit, API, and UI. Intelligent automated UI tests provide us more information than functional tests alone.

Knowing when to use AI in testing

What does the future of testing with AI look like? It’s a combination of people, processes, and technology. Software testers need to be thinking about what skills we need to have to support these new ways of testing and delivering quality software.

We need to uncover if a use case is better served by AI and machine learning than an algorithmic solution. To do this, we need to ask the following questions:

• What is the pain point that we are trying to solve?
• Are there good algorithmic or heuristic solutions available to address the pain point?
• Are there enterprise ready AI and machine learning solutions that help?
  • If there are solutions, what happens if the AI doesn’t work?
  • If there aren’t solutions yet, how do we position ourselves to be ready for them?

“It’s quite trendy to talk about artificial intelligence, but the reason why we’re partnered with Applitools is that they apply real machine learning and artificial intelligence to a problem that is not well solved by other types of solutions on the market.”

Chris Rolls, CEO, Americas, TTC

Integrating visual AI into test automation

Let’s talk about how we can integrate AI into our testing to get some of those advantages of increased speed and coverage discussed earlier.

What is visual AI?

I like to explain visual AI visually. Do you remember those spot-the-difference pictures we had in our activity books from when we were kids?

As humans, we could sit around and play with this to find the differences manually. But what we want is to be able to find these differences immediately. And that is what visual AI has the power to do.

Even when the images are a little bit skewed or off by a couple pixels here or there, visual AI can pinpoint any differences between one view and another.

Now you might be thinking, Andy, that’s cute, but how’s this technology gonna help me in the real world? Is it just gonna solve little activity book problems? Well, think about all the apps that you would develop – whether web, mobile, desktop, whatever you have – and all the possible ways that you could have visual bugs in your apps.

Here we’ve got three different views from mobile apps. One for Chipotle, one for a bank, and another one for a healthcare provider.

Visual AI test automation

You can see that visual bugs are pervasive and they come in all different shapes and sizes. Sometimes the formatting is off, sometimes a particular word or phrase or title is just nulled out. What’s really pesky is that sometimes you might have overlapping text.

Traditional automation struggles to find these issues because traditional automation usually hinges on text content purely or on particular attributes of elements on a page. So as long as something appears and it’s enactable, most traditional scripts will pass. Even though we as humans visually can inspect and see when something is completely broken and unuseable.

This is where visual AI can help us, because what we can do is we can take snapshots of our app over time and use visual AI to detect when we have visual regressions. Because if, let’s say, one day the title went from being your bank’s name to null, it’ll pick it up right away in your continuous testing.

Visual AI in action

In the webinar, I gave a live demo of automated visual testing using Applitools Eyes. In case you missed the demo, you can check it out here:

The future will be autonomous

So all this really cool stuff is powered by visual AI, a real world application of AI looking at images and being able to find things in them like they were humanized. Now you may think this is really cool, but what’s even cooler is this is just the beginning of what we can do with the power of AI and machine learning in the testing and automation space.

What we’re going to be seeing in the next couple years is a new thing called autonomous testing where not only are we automating our tests, but we’re automating the process of developing and maintaining our tests. The tests are kind of almost writing themselves in a sense. And visual AI is going to be a key part of that, because if testing is interaction and verification, what we want to make autonomous is both interaction and verification. And visual AI has already made verification autonomous. We’re halfway there, folks.

Be sure to check out our upcoming events page for new webinars coming soon!Learn how Applitools Eyes uses AI to catch visual differences between releases while reducing false positives. Happy testing!

Read more

Let the Engineers Speak! Part 5: Audience Q&A

Articles — 01.11.2023

Let the Engineers Speak! Part 4: Changing Frameworks

Articles — 01.05.2023

Let the Engineers Speak! Part 3: Favorite Test Integrations

Articles — 12.28.2022

Let the Engineers Speak! Part 2: The Biggest Challenges

Articles — 12.22.2022

The post Enhancing UI/UX Testing with AI appeared first on AI-Powered End-to-End Testing | Applitools.