TL;DR

• AI is now foundational to software testing, but more AI often creates more noise.
• AI-assisted development increases code volume and pressure on QA teams.
• The biggest bottleneck in testing today is signal-to-noise, not execution speed.
• Successful testing strategies in 2026 prioritize trust, explainability, and reliable results.

AI has quietly moved from the edges of software testing into the center of it. For most teams, it’s no longer a question of whether AI plays a role in testing, but how deeply—and how intentionally.

Quality and Engineering leaders are feeling this shift firsthand. AI-assisted development is increasing the volume and pace of code changes. Release cycles are accelerating. At the same time, testing teams are being asked to scale confidence without scaling headcount.

In this environment, speed alone is not the differentiator. Trust is. 

In AI-driven testing, speed without trust slows teams down.

AI is no longer optional in testing

Across the software delivery lifecycle, AI is already embedded in day-to-day workflows. Teams are using it to generate test cases from requirements, assist with automation, create test data, and analyze results. In many organizations, this adoption didn’t start with QA—it started with developers.

What’s changed is that AI is no longer experimental or isolated. It’s shaping how testing actually happens.

This matters because AI-assisted coding changes the scale of the testing problem. More code is being produced, faster than before, and not all of it is high quality. That shift pushes pressure downstream, straight onto QA and QE teams.

More AI hasn’t reduced pressure on QA—it’s increased it

For many Engineering Managers, AI has delivered productivity gains on the development side while increasing complexity on the testing side. Test suites grow larger. Pipelines generate more results. Failures are harder to interpret.

As Applitools CEO Anand Sundaram recently described, the imbalance is real:

“You have more code to be tested, sometimes not the best code, more coverage required, and fewer people.”

– Shaping Your 2026 Testing Strategy

This combination exposes a deeper issue. As tooling improves, teams don’t just get more data, they get more noise. And noise is expensive.

The real bottleneck is signal-to-noise

Most mature teams are no longer blocked by how fast they can run tests. They’re blocked by how confidently they can interpret the results. 

As AI accelerates development, signal quality matters more than test volume.

False positives, flaky tests, and inconsistent outcomes force teams into defensive behaviors: re-running pipelines, manually validating changes, and delaying releases “just to be safe.” Over time, automation stops accelerating delivery and starts slowing it down.

This is where many AI-driven testing initiatives struggle. AI can generate more tests and more output, but without reliable signals, that output doesn’t lead to better decisions.

Not all AI is suitable for testing decisions

One clear theme for 2026 is that AI is not a single, interchangeable capability. Different phases of the testing lifecycle have very different requirements.

Large language models excel at tasks that tolerate variation: generating test ideas, creating data, summarizing results, and assisting with analysis. But test execution and release decisions demand consistency, repeatability, and explainability.

This distinction becomes especially clear when you look at test execution. Unlike test generation or analysis, execution depends on consistent behavior and repeatable outcomes.

When test outcomes change run to run, teams lose trust. When failures can’t be reproduced, debugging slows down. And when decisions can’t be explained clearly, confidence erodes—both within engineering and with leadership.

Trust, explainability, and repeatability matter more than novelty

As AI adoption grows, testing teams are being forced to answer harder questions. Can we trust these results? Can we explain them? Can we confidently make release decisions based on them?

These questions matter in regulated and high-risk environments, but they’re just as relevant for any team shipping customer-facing software at speed. Reliability is not a constraint on velocity—it’s what makes velocity sustainable.

Teams operating under stricter compliance requirements have already learned that explainability and repeatability are non-negotiable for AI-driven testing decisions. (Read more—AI Testing in Regulated Environments: Smarter Testing Starts With Stability, Not More Code.)

This is why many teams are rethinking how they apply AI to testing. Deterministic approaches—systems that behave consistently and predictably—make it easier to reduce noise, identify real failures, and move faster with confidence.

What this means for testing strategy in 2026

The takeaway for Quality and Engineering leaders isn’t to slow down AI adoption. It’s to be more intentional about it.

Successful testing strategies in 2026 will share a few characteristics:

• AI is treated as foundational, not experimental
• Different phases of testing use different kinds of AI
• Reliability and explainability are prioritized where decisions are made
• Signal quality and maintenance reduction are explicit goals

Not all AI belongs everywhere. Choosing where reliability matters most is becoming a core leadership responsibility for engineering and quality teams. The biggest risk in AI-driven testing isn’t lack of automation—it’s lack of trust.

Choosing progress over noise

AI is reshaping software testing whether teams are ready or not. The challenge now is judgment. Knowing where AI accelerates quality—and where it quietly undermines it—is what separates teams that scale confidently from those that drown in noise.

The fastest teams aren’t the ones chasing the newest tools. They’re the ones that trust what their tests are telling them.

Watch Shaping Your 2026 Testing Strategy now.

The post AI Testing in 2026: Why Signal, Trust, and Intentional Choices Matter More Than Ever appeared first on AI-Powered End-to-End Testing | Applitools.

TL;DR

• Code-centric automation continues to slow teams down as UI changes multiply, making stability and evidence hard to maintain.
• AI code generators don’t solve the problem because they still produce brittle test code that requires constant oversight.
• Live LLM-driven execution introduces unpredictability. Regulated teams need deterministic runs, not improvisation
• A clearer path is intent-driven authoring paired with deterministic engines and Visual AI that detects visual drift and preserves audit-ready evidence.

Request our Governance Readiness Checklist

Teams in regulated environments face a familiar strain. Applications grow in complexity, expectations for fast releases keep rising, and every update requires clarity about what changed and whether required elements still appear as intended. Traditional automation wasn’t built for that pace or level of oversight, and the recent wave of AI coding tools hasn’t solved the core challenges.

A better model is emerging—one that uses AI to reduce the workload of authoring and maintaining tests while keeping execution deterministic, reviewable, and aligned with how people evaluate digital experiences.

This post breaks down why the legacy testing model is hitting its limits and how AI can support a more stable, more trustworthy approach.

Why traditional automation keeps slowing teams down

As digital experiences expand across pages, portals, member journeys, and product flows, test code becomes difficult to scale. Even minor UI changes break locators and assertions, creating unpredictable test runs, delayed reviews, and long maintenance cycles.

Developers are often asked to take on more of the testing responsibility. While this can improve feedback loops, it does not reduce the burden of maintaining code that reacts poorly to UI changes. And when teams already lack time, context switching between product development and test diagnostics becomes expensive.

The result is a predictable bottleneck: too many tests tied directly to implementation details and not enough stability across releases.

Why AI-generated test code hasn’t fixed the problem

The last few years have produced a surge of tools that promise to generate automation code automatically. But teams report the same issues repeating in a new form. LLMs can produce code quickly, yet the resulting output still inherits all the maintenance challenges of coded automation.

AI code generators also excel more at producing new code than updating existing flows. They struggle with assertions, hallucinate element behavior, and require human supervision to validate every step. For regulated teams that must show repeatability and generate evidence for every release, inconsistency becomes a risk rather than a convenience.

If the goal is to escape brittle code, producing more of it is not the answer.

Why live LLM-driven execution creates instability

Another idea gaining attention is allowing an LLM to operate the UI directly during test execution. In theory, this removes the need to write code. In practice, teams quickly run into new risks: undefined steps, inconsistent interactions, slow decision-making, and no reliable way to debug.

Execution in regulated environments must be predictable. It must be reviewable. And it must produce evidence that can be traced, explained, and defended. Live improvisation during a test run undermines each of these requirements.

Determinism matters more than novelty. A testing approach must produce the same result today, tomorrow, and during an audit review.

A clearer path forward: intent-driven authoring with deterministic execution

A more reliable model is emerging that uses AI to simplify authoring without relying on AI to make real-time decisions during execution.

Teams describe test intent in natural language. An AI system translates that intent into structured steps during authoring, where humans can review and adjust. Execution is then handled by deterministic engines and Visual AI that observe the rendered UI and detect visual changes, required-element presence, placement consistency, and contrast.

This separation delivers two advantages:

• People write and maintain far fewer lines of test code
• Test runs become stable, repeatable, and easier to verify

Visual AI provides a complete view of the screen state and compares each run against an approved baseline. When something changes, the system surfaces the difference, captures evidence, and supports reviewer approvals. When the change is expected, one acceptance updates the baseline and applies it across browsers and devices.

The outcome is a testing layer that is easier to maintain and easier to trust.

What this looks like in practice

Teams adopting this approach typically see changes across several parts of their workflow:

• Tests are written in plain language, without selectors or framework setup
• Visual AI validates full screens for layout, presence, placement, and readability
• Changes are highlighted automatically to reduce manual inspection
• Evidence is captured through screenshots, diffs, timestamps, and logs
• Debugging takes place in an environment where runs behave the same every time
• Reusable flows and data-driven steps integrate into the same natural-language format

Instead of managing a growing volume of fragile code, teams maintain intent-level descriptions supported by deterministic execution.

What this means for oversight and compliance

For teams in financial services, healthcare, insurance, or life sciences, the benefits go beyond efficiency.

A visually grounded testing model helps confirm that required notices, disclosures, language-access elements, and other regulated UI content remain present and placed as expected. It documents what changed and preserves evidence for review. It supports consistent experiences across browsers, devices, and PDFs without checking whether values, data, or regulatory text are correct.

Most importantly, it delivers predictable results.

Regulated environments depend on clarity and traceability. When every test run yields reviewable outputs, and every change is captured with context, teams can maintain confidence and release with speed.

If you’re assessing how well your testing workflow supports stability and audit readiness, request our Governance Readiness Checklist. We’ll share the version designed for your stage—whether you’re evaluating Applitools or optimizing an existing deployment.

The post AI Testing in Regulated Environments: Smarter Testing Starts With Stability, Not More Code appeared first on AI-Powered End-to-End Testing | Applitools.

Update & TL;DR

This post was written while Forrester’s research on agentic and autonomous testing was still emerging. Since publication, Applitools has been included in The Forrester Wave: Autonomous Testing Platforms, Q4 2025. The perspective outlined below reflects how this shift has since been validated and formalized by independent industry analysts.

• Agentic automation shifts testing from brittle, script-driven execution to intelligent systems that adapt based on change, risk, and context.
• AI augments human intent rather than replacing QA teams, enabling people to focus on quality strategy, governance, and risk decisions.
• This model is increasingly shaping how autonomous testing platforms are evaluated in the market.

Forrester, a leading global research and advisory firm, identified a major turning point in software testing in its Autonomous Testing Platforms Landscape, Q3 2025. The research describes a shift from traditional scripted automation to AI-augmented systems that can learn, adapt, and act under human guidance. This shift signals the rise of agentic automation: intelligent systems that create, run, and optimize tests within defined boundaries.

As delivery cycles compress and complexity grows, quality and engineering leaders are redefining what effective testing means in practice. Agentic automation bridges human intent with machine-driven precision—transforming testing from a reactive maintenance task into a proactive engine for reliability, speed, and continuous improvement.

From Automation to Intelligence

Traditional automation accelerated execution but left teams managing brittle scripts and endless maintenance. AI-augmented testing changes that dynamic. These systems:

• Learn continuously from results and application change.
• Adapt test scope and prioritization based on business risk.
• Optimize coverage while maintaining human oversight.

The result is testing that behaves less like a checklist and more like a self-improving quality partner, one that scales reliability across every release.

The Three Business Values Driving This Shift

Forrester highlights three outcomes motivating investment in more intelligent testing systems:

1. Accelerate Time to Value – AI-driven generation and self-healing shorten feedback loops and reduce maintenance.
2. Reduce Strategic Risk – Risk-based orchestration and built-in governance connect quality metrics directly to business priorities.
3. Democratize Testing – Low-code authoring and natural-language interaction let non-developers participate in quality, closing skill gaps.

Agentic automation brings these together: human-directed intent, machine-driven efficiency, and transparent oversight.

How AI-Augmented Systems Complement Human Expertise

AI in testing works best as augmentation, not replacement. By handling repetitive execution and maintenance, intelligent systems free QA professionals to focus on:

• Defining risk and coverage strategy.
• Establishing governance frameworks that maintain trust.
• Collaborating earlier with product and development teams.

Agentic automation shifts QA leadership from running tests to steering quality outcomes.

The Role of Visual and Experience Validation

Intelligent automation depends on reliable validation signals. Traditional assertions can’t always capture what matters to real users: layout, accessibility, and experience consistency. 

Visual and experience validation fill that gap, giving AI-augmented systems context they can trust. When machines validate what users actually experience, teams gain both speed and confidence—without rigid pixel-level comparison.

Building Toward AI-Augmented Readiness

Forrester describes this as a maturing market: organizations are blending traditional automation with AI capabilities to move toward greater autonomy over time. QA leaders can start by:

1. Stabilizing automation foundations and addressing flakiness.
2. Adopting AI-assisted detection of UI and data changes.
3. Integrating experience-level validation for richer feedback.
4. Connecting quality analytics to business metrics for continuous improvement.

Each step builds the trust and data maturity required for agentic automation to succeed under human orchestration. As adoption increases, these maturity steps align with how leaders in the market are being evaluated on autonomous capabilities.

What QA Leaders Can Do Next

Forward-looking teams are already experimenting with:

• Adaptive execution that prioritizes tests dynamically.
• Governance dashboards linking coverage, risk, and compliance.
• Visual AI that helps systems understand real user impact.

The goal isn’t full autonomy—it’s AI-augmented confidence: testing that’s faster, smarter, and more inclusive across roles. Read the full report now.

The post Agentic Automation: Preparing QA Leaders for the Next Leap in Testing appeared first on AI-Powered End-to-End Testing | Applitools.

TL;DR — Applitools brings visual, functional, and autonomous testing together in a single AI-powered platform. Backed by 11+ years of refinement and a dataset of 4 billion real-world images, our Visual AI delivers unmatched accuracy and reliability for enterprise-grade software testing.

Testing today isn’t just about coverage—it’s about confidence, speed, and scaling quality across teams. Whether you’re a developer chasing faster feedback, a QA lead reducing maintenance overhead, or a product owner focused on release velocity, Applitools helps modern teams deliver software that looks right, works right, and evolves with ease.

Here’s how Visual, Functional, and Autonomous Testing all come together in one powerful platform.

Trusted Visual AI with Proven Accuracy

Applitools sets the standard in Visual Testing. Our Visual AI engine delivers 99.9999% accuracy, eliminating false positives and catching bugs others miss.

• 5.8x more efficient than pixel-based tools
• Detect both functional and visual bugs in a single test
• Works with all major frameworks: Selenium, Cypress, Playwright, and more

We didn’t just add AI—we’ve spent 11+ years perfecting it.

A Complete Platform for End-to-End Testing

Applitools goes far beyond screenshots. Our Intelligent Testing Platform includes Autonomous Test Creation, Visual Validation, Cross-Browser + Device Testing, and Accessibility Testing—all in one cloud-based solution.

• Run tests across browsers, devices, and screen sizes in parallel
• Built-in accessibility and compliance testing
• Fully scalable with enterprise-grade performance

Less Test Maintenance with Self-Healing, Smart Grouping & Predictive Analytics

Spend less time fixing broken tests and more time delivering value. Applitools minimizes test upkeep so your team can focus on building.

• Self-healing locators reduce test flakiness and manual updates
• Root cause analysis, smart grouping, and predictive analytics streamline debugging
• Tests automatically adapt as your UI evolves—no babysitting required

Collaborative Testing: How Developers, PMs, Designers & Marketers All Work Smarter with Applitools

Testing shouldn’t be a bottleneck—or limited to just QA. Applitools empowers developers, designers, product managers, and even marketers to collaborate with ease.

• Intuitive UI for reviewing results and managing baselines
• Seamless sharing of results and issue tracking
• Codeless and code-based authoring, no deep technical expertise needed

More than a Decade of AI Leadership

AI isn’t new to us—it’s the foundation of our platform. Unlike newer tools making AI promises, we’ve been building, training, and refining Visual AI to solve real testing challenges at scale for more than a decade.

• Mature, battle-tested AI that improves with scale
• Trusted by engineering teams at Microsoft, Tesla, Salesforce, eBay, and 400+ more
• Over 1 billion visual tests run with our platform

Seamless Integrations & Dev Experience

Great testing fits into your workflow—not the other way around. Our AI-powered test automation works with your tools, languages, and CI/CD pipelines to scale quality without slowing you down. Applitools integrates with:

• Every major framework: Selenium, Cypress, Playwright, Puppeteer, WebdriverIO
• CI/CD tools: GitHub Actions, Jenkins, GitLab, Azure DevOps
• SDKs for Java, JavaScript, Python, C#, and more

Whether you’re in code or no-code workflows, we plug into your stack and scale with you.

24/7 Support That Doesn’t Disappear

Whether you’re mid-sprint or troubleshooting a release, help is always within reach. Get expert guidance anytime—no hoops, no waiting.

• Around-the-clock global technical support
• Extensive documentation, how-tos, and real-time guidance
• Active community forum and dedicated Customer Success Managers (not just for enterprise)

Compare that to competitors with limited support, slow response times, or no dedicated resources unless you’re a top-tier customer.

Smart Investment, Real Value

Our pricing is flexible, predictable, and scales with your needs. You’ll see ROI fast:

• Save hours of test maintenance per sprint
• Eliminate manual bug hunts and false positives
• Deliver faster releases without compromising quality

Explore our current pricing structure, or speak with a testing specialist to build a package that’s right for your team.

“We reduced our testing time from days to hours. Applitools changed how we think about QA.”
— QA Lead, Global Retail Brand

Visual, Functional, and Autonomous Testing—The Applitools Advantage

We combine Visual AI, Autonomous Testing, and a developer-friendly platform into one powerful, scalable solution. With Applitools, your team gets:

• Smarter test creation
• Less maintenance
• Better collaboration
• Faster releases
• And trusted results every time

See What’s New with Applitools Autonomous and What’s Coming with Applitools Eyes

Ready to Test Smarter?

In a crowded automation landscape, it’s not enough to have “AI-powered” features. You need real results. With over a billion visual tests run and trusted by leading enterprises across industries, Applitools isn’t experimenting with AI—it’s already delivering.

Whether you’re starting fresh or looking to scale smarter, Applitools gives your team the tools to automate with confidence and speed.

Ready to see it in action? Start your free trial, book a personalized demo, or explore the platform today.

Applitools helps you test like it’s 2025. Join the world’s top teams already doing it.

The post Visual, Functional, and Autonomous Testing—All in One appeared first on AI-Powered End-to-End Testing | Applitools.

Manual testing remains an integral part of software development but the increasing complexity of applications demands faster and more efficient testing methodologies. This is where Artificial Intelligence (AI) comes in, offering innovative ways to enhance manual testing efforts.

AI is not here to replace manual testers; instead, it acts as a force multiplier, augmenting their capabilities, reducing repetitive work, and improving accuracy. Something that has been proven multiple times is that AI cannot test or tell the look and feel of an application as well as a human.

In this blog, we will explore how AI can augment manual testing, making the process smarter, faster, and more effective.

The Role of Manual Testing

Manual testing involves human testers executing test cases without automation tools. It is essential for:

• Usability testing – Ensuring a seamless user experience.
• Exploratory testing – Identifying edge cases and unpredictable scenarios.
• Ad-hoc testing – Finding defects that automated scripts may miss.
• Accessibility testing – Evaluating how applications accommodate diverse user needs.

While manual testing is indispensable, it also comes with some challenges like the tests being time-consuming and repetitive testing that can take a lot of effort. It is also error-prone and can also miss some defects having a defect leakage in extreme scenarios.

In addition, with all the new and evolving technologies out there manual testing is not scalable. Therefore, AI helps address these challenges by complementing human testers, allowing them to focus on more strategic tasks.

How AI Augments Manual Testing

Test Case Generation and Optimization
Creating test cases manually can be labor-intensive and inefficient. AI-driven tools can:

• Historical defect data analysis to suggest optimal test scenarios.
• Dynamic generation of test cases from application changes.
• Optimizing test coverage by identifying redundant test cases.

Intelligent Bug Detection
AI can improve defect identification by:

• Analyzing log, UI, and user behavior to detect anomalies.
• Detecting potential failure points before they occur.
• Auto-classifying bugs to prioritize critical defects.

Automated Test Execution Suggestions
AI can assist manual testers by:

• Recommending test cases based on failure probabilities.
• Identifying high-risk regions that must be tested more.
• Proposing exploratory test paths based on real user activity.

Self-Healing Test Scripts
One of the biggest pain points in automation is script maintenance. AI-powered automation tools can:

• Automatically modify test scripts when the UI or functionality is changed.
• Reduce false positives via tuning to minor changes.
• Support script learning from previous runs.

Enhanced Exploratory Testing
AI does not replace a tester but rather amplifies them. Exploratory testing still relies on a tester’s experience and intuition while AI enhances this by:

• Providing test suggestions and hints based on application behavior.
• Building real-world usage scenarios for greater testing coverage.
• Identification of probable weak areas from historical trends.

Smarter Test Data Management
AI can streamline test data creation by:

• Synthesizing test data from application requirements.
• Identification of missing test data scenarios for better coverage.
• Masking sensitive data for security and regulatory purposes.

Visual and UI Testing
Ensuring a consistent user experience across multiple devices is challenging. AI-based visual testing tools can:

• Identifies UI anomalies and layout shifts on different screen sizes.
• Identifies color contrast issues for accessibility compliance.
• Baseline screenshot comparison with new builds to highlight differences.

Predictive Analysis for Risk-Based Testing
AI can help teams focus on high-risk areas by:

• Analyzing past test run data to predict probable failure points.
• Recommending test priorities based on defect trends.
• Removing redundant tests with optimal risk coverage.

This allows testers to focus their efforts on the most impactful tests, improving efficiency.

Chatbots for Test Execution and Assistance
AI-driven chatbots can:

• Provide instant visibility into test results and defect patterns.
• Execute test cases on-demand via natural interfaces.
• Assist the author in building and optimizing test scripts.

The Future of AI-Augmented Testing, The Perfect Combination

AI is transforming the way testing is conducted, but human testers remain indispensable. It would be a great challenge for a tester to now start adapting to the new trends, just like in the past we have had many opinions about automation until we actually saw how it helped our testing. 

The future lies in:

• Human-AI Collaboration – AI handles repetitive tasks, while testers focus on critical thinking and user experience.
• More Adaptive AI Models – AI will continue to learn from test results and user behavior, improving over time.
• AI-Driven Test Orchestration – Seamless integration of AI into DevOps for continuous testing and delivery.

Artificial Intelligence (AI) is transforming software testing but it remains a hot debate among testers. While AI enhances manual testing by automating repetitive tasks, improving accuracy, and speeding up defect detection some professionals still hesitate to embrace it.

However, instead of fearing AI testers should embrace it as a powerful ally. AI eliminates tedious tasks, improves efficiency, and allows testers to focus on critical thinking and creative problem-solving.

In Summary

AI is not replacing manual testers—it is empowering them. By automating repetitive tasks, optimizing test execution, enhancing defect detection, and improving exploratory testing, AI allows testers to focus on what truly matters: ensuring a seamless user experience.

As AI continues to evolve, testers who embrace AI-driven tools will be better equipped to deliver high-quality software faster and more efficiently. The key is to strike the right balance between human expertise and AI-powered augmentation, ensuring that software testing remains intelligent, adaptive, and effective.

Are you ready to embrace AI in your testing workflows?

The post How AI Can Augment Manual Testing appeared first on AI-Powered End-to-End Testing | Applitools.

As applications become more complex and UI consistency becomes critical, ensuring that the user interface appears as expected across multiple environments is key. Applitools Eyes, when integrated with the Playwright SDK, provides a powerful, efficient, and streamlined approach to visual testing in your Playwright tests. The combination of Playwright, a popular end-to-end testing framework, and Applitools Eyes, a visual AI-powered testing tool, makes visual validation easier, faster, and more scalable.

Applitools Eyes is a visual AI-powered testing tool that helps address these UI consistency challenges. It uses advanced AI-driven image comparison to detect visual differences. Unlike traditional visual testing tools that perform pixel-by-pixel image comparisons, Applitools Eyes uses AI algorithms to determine if the visual differences are actual bugs.

Applitools Playwright SDK introduces several improvements designed to streamline the process of setting up and running visual tests, making the experience more efficient and user-friendly.

In this blog, we will discuss when it is appropriate to use visual testing with Playwright and which cases will be suitable for using Applitools.

Overview of Visual Testing

Visual testing is a technique performed to ensure the visual appearance of a given website or application matches the provided design and layout . This includes the comparison of the actual UI against a standard image or a reference UI image. This type of testing primarily focuses on detecting any visual anomalies, which include alignment issues , font problems, color discrepancies , images, or structural shifts that can disrupt the user experience.

How Visual Testing Differs from Functional Testing

Visual testing and functional testing are both crucial if the goal is to deliver a high-quality application; visual testing checks the visual appeal of the application, while functional testing verifies how the application functions.

Let’s see some major differences between visual and functional testing.

Before delving into how we can perform visual testing using Playwright and Applitools, let’s review some of the new improvements that Applitools has introduced .

What’s New In Applitools Playwright SDK?

The Applitools Playwright SDK is a library that integrates Applitools’ visual testing capabilities with the Playwright automation framework.

Let’s explore the improvements done by Applitools to streamline the process of setting up and running visual tests, making the experience more efficient and user-friendly. 

Here’s a detailed breakdown of what’s new in the updated SDK:

Test Fixtures:

• Test fixtures Centralize and reuse setup code across multiple tests, making the setup for visual tests more consistent and reducing redundant configuration.
• This feature is particularly valuable when running tests on multiple pages or scenarios, as it simplifies the initialization of Applitools Eyes in each test.

CLI Onboarding:

• The Command-Line Interface (CLI) simplifies the initial setup process by automating configuration steps, helping users quickly integrate Applitools Eyes with Playwright. 
• This is especially helpful for new users, reducing the complexity of the setup and enabling faster onboarding into visual test creation.

Config Object Setup:

• The configuration object setup automates the insertion of Applitools Eyes settings into the Playwright configuration file (e.g., playwright.config.js or playwright.config.ts). 
• This feature eliminates manual setup, reducing the risk of errors, ensuring accurate configuration from the outset, and saving developers valuable time.

Custom HTML Reporter:

• The custom HTML reporter enhances Playwright’s default test report by integrating visual test results from Applitools Eyes. 
• This allows developers to view a direct comparison between the baseline image and the current test screenshot, helping identify visual regressions more effectively. 
• By adding visual results to the standard Playwright report, it simplifies the review process, offering a comprehensive overview of the test outcomes.

Before we deep dive into how Applitools helps in visual testing, let’s first see how we can perform visual testing using Playwright.

Visual Testing Using Playwright

Playwright comes equipped with tools to make visual testing effortless, allowing developers to take and compare screenshots of web pages or specific web page elements. 

Here’s how visual testing works in Playwright:

1. First Run: Playwright captures and saves a screenshot of the page or specific UI elements. This is known as the base image.
2. Subsequent Runs: In the next run, Playwright takes a new screenshot and compares it against the base image.
   • If there are no differences, the test passes.
   • If differences exist, the test fails, flagging the areas where changes occurred.

Prerequisites

Ensure the following tools are installed:

Node.js: Download and install from Node.js.

Visual Studio Code (Optional): Recommended IDE for coding.

Install Playwright:

npm i @playwright/test

Write the Simple Visual Regression Test Using Playwright
Create a new JavaScript file in your test folder, e.g., demo.spec.js.

Use the page.screenshot() method to capture the screenshot and the expect.toMatchSnapshot() assertion from the @playwright/test module to compare images.

const { test, expect } = require('@playwright/test'); 
test('Visual Regression Test Example', async ({ page }) => { 
   // Navigate to the website 
   await page.goto('https://playwright.dev'); 
   // Capture a screenshot of the page 
   const screenshot = await page.screenshot(); 
   expect(screenshot).toMatchSnapshot(); 
}); 

When we execute the above code for the first time, the test case fails with the below error, and Playwright captures and saves a screenshot of the page or specific UI elements. This is known as the base image.

When we execute the same test case again, it will pass.

Visual Regression Test with maxDiffPixels option 

There might be the case when there are some minor differences in the images and the test case starts failing; to handle this situation we have an option, the ‘maxDiffPixels option,’ that we can pass. It allows you to specify the maximum number of pixels that can differ between two images for the comparison to still be considered successful.

const { test, expect } = require('@playwright/test');
test('Visual Regression Test Example', async ({ page }) => {
  // Navigate to the website
  await page.goto('https://playwright.dev');
  // Capture a screenshot of the page
  const screenshot = await page.screenshot();
  expect(screenshot).toMatchSnapshot({ maxDiffPixels: 100 });
}); 

const { test, expect } = require('@playwright/test');
test('Visual Regression Test Example', async ({ page }) => {
  // Navigate to the website
  await page.goto('https://playwright.dev');
  // Capture a screenshot of the page
  const screenshot = await page.screenshot();
  expect(screenshot).toMatchSnapshot({ threshold:0.5} );
}); 

Visual Regression Test with Threshold option

Passing ‘threshold’ as an option is another option to avoid failing the test case. This is particularly useful for handling slight differences that occur due to rendering variations.

const { test, expect } = require('@playwright/test');
test('Visual Regression Test Example', async ({ page }) => {
  // Navigate to the website
  await page.goto('https://playwright.dev');
  // Capture a screenshot of the page
  const screenshot = await page.screenshot();
  expect(screenshot).toMatchSnapshot({ threshold:0.5} );
}); 

So far we have seen how we can automate the visual UI using Playwright. In the next section you will see how we can use Applitools Eyes and Playwright SDK together to automate visual testing.

Visual Testing using Applitools Eyes and Playwright SDK

Visual testing is a crucial aspect of modern software quality assurance. It ensures that a web application’s UI renders correctly across different devices, screen sizes, and browsers. Tools like Applitools Eyes and frameworks like Playwright SDK simplify this process by providing robust solutions for automated visual regression testing.

Below are the steps to set up Applitools Eyes and Playwright SDK.

Install Playwright

npm init playwright@latest

Install Applitools Eyes SDK 

npm install -D @applitools/eyes-playwright

Add the below line in .spec.js file 

To configure your project for Applitools Eyes, we have to add the below line.

import { test } from '@applitools/eyes-playwright/fixture';

In the next section you will see examples to explain how we can use SDK with Playwright tests. 

Before moving in the detail of various types of matchLeavel, let see one basic example to understand how we do the visual testing in Applitool,  and the method used in Applitools, to capture and validate a checkpoint in your application’s UI.

Below is code that integrates Applitools Eyes with Playwright to perform a visual test.

import { test } from '@applitools/eyes-playwright/fixture';
test('My first visual test Using Applitools with matchLevel: Dynamic', async ({ page, eyes }) => {
await page.goto('https://app14743.cloudwayssites.com/helloworld/');
  // Visual check
  await eyes.check('Landing Page', {
    fully: true,
    matchLevel: 'Dynamic',
  });
});

In the above code we mainly use two things: one is the method eyes. check() and two options: {fully: true}, and {matchLevel: ‘Dynamic’}

eyes.check(): Performs a visual snapshot of the page or a specific element and compares it with the baseline image stored in Applitools.

Options:

• fully: true:
  • Captures the entire page, including scrollable sections.
  • Ensures that all content on the page is included in the visual test.
• matchLevel: ‘Dynamic’:
  • A matching algorithm that focuses on the content while ignoring dynamic changes, such as text or layout differences.
  • Useful for pages with frequently changing content, like user-generated text or dynamic data.

Below are the options that we can pass in our test case. These options provide flexibility and precision, enabling robust visual testing tailored to different scenarios and challenges.

This table summarizes each configuration, making it easier to understand their purposes.

Examples with different matchLevel

MatchLevel determines how the captured visual output is compared with the baseline image.Let’s see the examples with different options.

1. Visual Testing with matchLevel: Strict

This ensures pixel-perfect comparison between the baseline and current screenshot, highlighting any visual difference, no matter how small.

test('My first visual test Using Applitools with matchLevel: Strict', async ({ page, eyes }) => {
  await page.goto('https://app14743.cloudwayssites.com/helloworld/');
  await eyes.check('Landing Page', {
    fully: true,
    matchLevel: 'Strict',
  });
});

• Purpose: Validates the entire page with Strict match level.
• Strict Match Level: Identifies even small differences, such as minor pixel or layout changes, ensuring high precision in visual validation.

2. Visual Testing with matchLevel: Dynamic

Focuses on content consistency by ignoring minor layout differences, such as text movement, while ensuring key visual elements remain unchanged.

In the below example you can see even if the text color is changed after clicking on link ‘?diff2’ test case still works perfectly fine because we have set matchLevel: Dynamic. This ensures the visual test focuses on the content and structure of the text rather than superficial changes like color.

test('My first visual test Using Applitools with matchLevel: Dynamic', async ({ page, eyes }) => {
 await page.goto('https://app14743.cloudwayssites.com/helloworld/');
 await page.getByRole('link', { name: '?diff2' }).click();
 await eyes.check('Homepage', {
   fully: true,
   matchLevel: 'Dynamic',
 });
});

• Purpose: Validates the entire page with Dynamic match level.
• Dynamic Match Level: Ignores minor content/layout differences (e.g., text changes, animations) and focuses on high-level structural comparisons.
• fully: true: Captures a screenshot of the full page, not just the visible viewport.

3. Visual Testing for a Specific Region/Component

Limits visual comparisons to a particular area or component of the application.

test('My first visual test Using Applitools with particular region/Component', async ({ page, eyes }) => {
  await page.goto('https://app14743.cloudwayssites.com/helloworld/');
  const component = page.locator('.fancy.title.primary');
  await eyes.check('My Component', {
    region: component,
  });
});

• Purpose: Tests only a specific component or region on the page.
• region: Focuses the validation on the area defined by the component locator (.fancy.title.primary), instead of the entire page.

4. Visual Testing with Ignored Regions

Excludes specific areas from comparison, preventing false positives caused by expected dynamic changes in those regions.

test('Visual test Using Applitools with ignoring the region', async ({ page, eyes }) => {
  await page.goto('https://app14743.cloudwayssites.com/helloworld/');
  const dynamicContent = page.locator('.fancy.title.primary');
  await eyes.check('Homepage', {
    fully: true,
    matchLevel: 'Strict',
    ignoreRegions: [dynamicContent],
  });
});

• Purpose: Tests the page while ignoring specific dynamic regions.
• ignoreRegions: Excludes certain areas from the visual validation (e.g., areas with dynamic content like dates, ads, or animations).
• fully: true: Captures the entire page for validation.

About dynamic match level

Dynamic match level is a feature in Applitools Eyes that verifies text by matching it against predefined or custom patterns, ensuring the content adheres to a specific format. It focuses on format validation rather than content changes, making it ideal for dynamic text like dates, emails, or numbers.

Type of dynamic match level

Dynamic match levels allow for flexible verification of text by matching it against predefined or custom patterns. The default types include:

• TextField: Text inside input fields.
• Number: Numeric values like ZIP codes or phone numbers.
• Date: Validates text as a date in proper format.
• Link: Hyperlinks or URLs.
• Email: Checks for a valid email address format.
• Currency: Recognizes monetary values.

These types ensure accurate validation by focusing on the format rather than the content changes, such as dynamic updates to dates or numbers.

Custom dynamic pattern

To create a custom dynamic pattern we have to follow below steps in Applitools dashboard 

Here are the steps to create a custom dynamic pattern:

1. Open the Settings Window: In the Page Navigator, select Apps & Tests.

In the list of applications on the left, hover over an application and click > Settings.

2. Add a Custom Type: Click Add custom type.

3. Define the Custom Pattern:

Enter a name and a Regex pattern.

4. Save the Custom Pattern:

In the below screenshot you can see we have set a custom pattern for Zip Code. Once the custom pattern is created, its name can be used in the code to reference the pattern.

Click Add will save the entered pattern. 

5. Apply the Settings:

Once all required patterns have been selected, click Apply settings.

Applitools Dashboard: Execute visual testing using the Applitools Playwright SDK

Applitools’ Dashboard is a powerful interface designed to manage and analyze test results efficiently, especially for visual testing and monitoring user interfaces.

Below are the steps that we normally have in the Applitools Dashboard when we execute any visual test case.

Precondition : Run the below command to export the key 

export APPLITOOLS_API_KEY='your_api_key_here'

Once the key is exported, the next step is to execute the test cases using the below command.

npx playwright test --ui tests/visual.spec.js

First Run: Playwright captures and saves a screenshot of the page or specific UI elements. This is known as the base image.

Subsequent Runs: In the next run, Playwright takes a new screenshot and compares it against the base image.

• If there are no differences, the test passes.
• If differences exist, the test fails, flagging the areas where changes occurred.

In the below screenshot in the Applitools Dashboard, you can see all the above four test cases are executed successfully.

Applitools Playwright SDK with Example in Detail

Let’s take an example of a particular component in the page. In the below screenshot, the component focuses on a specific component or region of the page. You can see only the ’Hello World’ part is tested. In case there is any change in this component, the test case will fail.

In the below example we have updated the component with the text ‘Happy World’ 

test('Visual test Using Applitools with Updating Component UI', async ({ page, eyes }) => {
await page.goto('https://app14743.cloudwayssites.com/helloworld/');
await page.getByRole('link', { name: '?diff2' }).click();
await eyes.check('Homepage', {
  fully: true,
  matchLevel: 'Strict',
 });
});

When we execute the above code, it will not execute successfully because the UI of the component is changed after clicking on the link ‘diff2.’

To fix the above problem, we have marked it resolved from the Applitools Dashboard. Once we accept the change, it becomes the base image.

Now when we execute the test case again, it gets executed successfully with the updated UI.

When Playwright’s Visual Testing Features Are Useful 

Playwright’s built-in visual testing features are well-suited for scenarios where:

Simple Visual Comparisons:

• Comparing entire pages or specific UI components with baseline images for detecting pixel-level differences.
• Example: Ensuring the homepage layout hasn’t shifted after a CSS update.

Static UIs:

• Testing applications with minimal dynamic or animated content, as these are less prone to false positives caused by animations or transient elements.

Fast Feedback:

• When quick pixel-by-pixel comparisons in CI pipelines are sufficient without advanced AI-powered analysis.
• Example: Smoke testing in a fast-moving development cycle.

Budget-Friendly:

• Playwright’s native screenshot and comparison capabilities don’t require additional tools or subscriptions.

When Playwright’s Visual Testing May Fall Short

Dynamic Content:

• UIs with dynamic or frequently changing elements (e.g., time, randomized content, or animations) can cause false positives due to exact pixel mismatches.
• Example: A dashboard with live-updating graphs.

Complex Visual Changes:

• Playwright’s threshold-based comparison may miss subtle or semantic changes that don’t exceed the defined pixel difference threshold.
• Example: Slightly altered typography or colour changes.

Ignored Regions:

• While Playwright allows element-specific testing, ignoring specific dynamic regions within a larger page requires custom logic.
• Example: Excluding advertisements or live widgets from visual comparison.

How Applitools Handles Situations Better

Applitools offers superior visual testing capabilities compared to Playwright, especially when handling dynamic content, ensuring pixel-perfect validation, focusing on specific components, and reducing false positives. 

Dynamic Content Handling:

• Match Levels: Applitools’ AI-powered matchLevel options (e.g., Strict, Layout, Content) allow intelligent comparison by focusing on structure and ignoring irrelevant pixel differences.
• Example: Testing a real-time dashboard with dynamic data but a consistent layout.

Ignored Regions:

• Applitools allows you to define ignored regions to exclude specific parts of a page from comparison (e.g., ads or timestamps).
• Example: Excluding a “current time” widget from visual testing.

Advanced Visual Validation:

• Features like dynamic regions and AI-based semantic analysis help detect visual regressions that go beyond pixel-by-pixel differences.
• Example: Identifying a button misalignment that might not be detected by Playwright’s threshold settings.

Visual Testing Across Components:

• Allows you to target specific regions or components for testing without manually cropping or capturing screenshots.
• Example: Testing the styling of a navigation bar or modal dialog.

Conclusion

Playwright’s built-in visual testing features are suitable for simple cases where you need pixel precision in image comparisons of static UIs. However, it may struggle with complicated designs, whether they involve minor changes within the site or more intricate design requirements .

Applitools Eyes, which employs artificial intelligence, is more suitable for overcoming these difficulties. It performs extraordinarily in cases with dynamic content; it easily identifies subtle visual regressions and is equipped with extra functionalities such as ignored areas and cross-browser testing. For teams with significant testing and enhancement requirements, where the built-in visual testing capabilities of Playwright may fall short for complex and large-scale application testing, Applitools serves as an invaluable complement. It ensures and elevates the visual quality of dynamic applications, making it an ideal choice for robust testing and quality assurance.

About the Author

Kailash Pathak (Applitools Ambassador | Cypress Ambassador)

Senior QA Lead Manager with over 15 years of experience in QA engineering and automation. Kailash holds certifications including PMI-ACP®, ITIL®, PRINCE2 Practitioner®, ISTQB, and AWS (CFL).

As an active speaker and workshop conductor, Kailash shares his expertise through blogs on platforms like Medium, Dzone, Talent500, The Test Tribe, and his personal site https://qaautomationlabs.com/

The post Leveraging Applitools for Seamless Visual Testing in Playwright appeared first on AI-Powered End-to-End Testing | Applitools.

Wondering how to choose between Playwright vs Selenium for your test automation? Read on to see a comparison between the two popular test automation tools.

When it comes to web test automation, Selenium has been the dominant industry tool for several years. However, there are many other automated testing tools on the market. Playwright is a newer tool that has been gaining popularity. How do their features compare, and which one should you choose?

What is Selenium?

Selenium is a long-running open source tool for browser automation. It was originally conceived in 2004 by Jason Huggins, and has been actively developed ever since. Selenium is a widely-used tool with a huge community of users, and the Selenium WebDriver interface even became an official W3C Recommendation in 2018.

The framework is capable of automating and controlling web browsers and interacting with UI elements, and it’s the most popular framework in the industry today. There are several tools in the Selenium suite, including:

• Selenium WebDriver: WebDriver provides a flexible collection of open source APIs that can be used to easily test web applications
• Selenium IDE: This record-and-playback tool enables rapid test development for both engineers and non-technical users
• Selenium Grid: The Grid lets you distribute and run tests in parallel on multiple machines

The impact of Selenium goes even beyond the core framework, as a number of other popular tools, such as Appium and WebDriverIO, have been built directly on top of Selenium’s API.

Selenium is under active development and recently unveiled a major version update to Selenium 4. It supports just about all major browsers and popular programming languages. Thanks to a wide footprint of use and extensive community support, the Selenium open source project continues to be a formidable presence in the browser automation space.

What is Playwright?

Playwright is a relatively new open source tool for browser automation, with its first version released by Microsoft in 2020. It was built by the team behind Puppeteer, which is a headless testing framework for Chrome/Chromium. Playwright goes beyond Puppeteer and provides support for multiple browsers, among other changes.

Playwright is designed for end-to-end automated testing of web apps. It’s cross-platform, cross-browser and cross-language, and includes helpful features like auto-waiting. It is specifically engineered for the modern web and generally runs very quickly, even for complex testing projects.

While far newer than Selenium, Playwright is picking up steam quickly and has a growing following. Due in part to its young age, it supports fewer browsers/languages than Selenium, but by the same token it also includes newer features and capabilities that are more aligned with the modern web. It is actively developed by Microsoft.

Selenium vs Playwright

Selenium and Playwright are both capable web automation tools, and each has its own strengths and weaknesses. Depending on your needs, either one could serve you best. Do you need a wider array of browser/language support? How much does a long track record of support and active development matter to you? Is test execution speed paramount? 

Each tool is open source, cross-language, and developer friendly. Both support CI/CD (via Jenkins, Azure Pipelines, etc.), and advanced features like screenshot testing and automated visual testing. However, there are some key architectural and historical differences between the two that explain some of their biggest differences.

Selenium Architecture and History

• Architecture: Selenium uses the WebDriver API to interact between web browsers and browser drivers. It operates by translating test cases into JSON and sending them to the browsers, which then execute the commands and send an HTTP response back.
• History: Selenium has been in continuous operation and development for 18+ years. As a longstanding open source project, it offers broad support for browsers/languages, a wide range of community resources and an ecosystem of support.

Playwright Architecture and History

• Architecture: Playwright uses a WebSocket connection rather than the WebDriver API and HTTP. This stays open for the duration of the test, so everything is sent on one connection. This is one reason why Playwright’s execution speeds tend to be faster.
• History: Playwright is fairly new to the automation scene. It is faster than Selenium and has capabilities that Selenium lacks, but it does not yet have as broad a range of support for browsers/languages or community support. It is open source and backed by Microsoft.

Comparing Playwright vs Selenium Features

It’s important to consider your own needs and pain points when choosing your next test automation framework. The table below will help you compare Playwright vs Selenium.

Should You Use Selenium or Playwright for Test Automation?

Is Selenium better than Playwright? Or is Playwright better than Selenium? Selenium and Playwright both have a number of things going for them – there’s no easy answer here. When choosing between Selenium vs Playwright, it’s important to understand your own requirements and research your options before deciding on a winner.

Selenium vs Playwright: Let the Code Speak

A helpful way to go beyond lists of features and try to get a feel for the practical advantages of each tool is to go straight to the code and compare real-world examples side by side. At Applitools, our goal is to make test automation easier for you – so that’s what we did! 

In the video below, you can see a head-to-head comparison of Playwright vs Selenium. Angie Jones and Andrew Knight take you through ten rounds of a straight-to-the-code battle, with the live audience deciding the winning framework for each round. Check it out for a unique look at the differences between Playwright and Selenium.

If you like these code battles and want more, we’ve also pitted Playwright vs Cypress and Selenium vs Cypress – check out all our versus battles here.

In fact, our original Playwright vs Cypress battle (recap here) was so popular that we’ve even scheduled our first rematch. Who will win this time? Register for the Playwright vs Cypress Rematch now to join in and vote for the winner yourself!

Learn More about Playwright vs Selenium

Want to learn more about Playwright or Selenium? Keep reading below to dig deeper into the two tools.

• Your Favorite Questions about Playwright, Answered [blog]
• Playwright: A New Test Automation Framework for the Modern Web [webinar w/team at Microsoft]
• Running Lightning-Fast Cross-Browser Playwright Tests Against any Browser [blog]
• Modern Cross Browser Testing with Selenium in Java [webinar]
• Playwright with JavaScript [free course]
• Selenium WebDriver with Java [free course]
• Playing with Playwright [blog]
• Playing with Playwright – Java API and Playwright vs Selenium [blog]

The post Playwright vs Selenium: What are the Main Differences and Which is Better? appeared first on AI-Powered End-to-End Testing | Applitools.

What is Functional Testing?

In functional testing, testers evaluate an application’s basic functionalities against a predetermined set of specifications. Using Black Box Testing techniques, functional tests measure whether a given input returns the desired output, regardless of any other details. Results are binary: tests pass or fail.

Why is Functional Testing Important?

Functional testing is important because, without it, you may not accurately understand whether your application functions as intended. An application may pass non-functional tests and otherwise perform well. Still, if the application doesn’t deliver the key expected outputs to the end-user, it cannot be considered working.

What is the Difference between Functional and Non-Functional Testing?

Functional tests check if the application meets specified functional requirements, while non-functional tests assess aspects like performance, security, scalability, and overall quality. To put it another way, functional testing is concerned with whether key functions are operating, and non-functional tests are more concerned with how the operations take place.

Examples of Functional Testing Types

There are many types of functional tests that you may want to complete as you test your application. 

A few of the most common include:

Unit Testing

Unit testing breaks down the desired outcome into individual units, allowing you to test if a small number of inputs (sometimes just one) produce the desired output. These tests tend to be small and quick to write and execute. Each one is designed to cover a single section of code (a function, method, object, etc.) and verify that code’s functionality.

Smoke Testing

Testers perform smoke testing to verify that the most critical parts of the application work as intended. It’s a first pass through the testing process and isn’t meant to be exhaustive. Smoke tests ensure that the application is operational on a basic level. If it’s not, there’s no need to progress to more detailed testing, and the application can go right back to the development team for review.

Sanity Testing

Sanity testing acts as a cousin to smoke testing, verifying basic functionality to potentially bypass detailed testing on broken software. Unlike smoke tests, sanity tests occur later in the process to confirm whether a new code change achieves its intended effect. This ‘sanity check’ ensures the new code roughly performs as expected. 

Integration Testing

Integration testing determines whether combinations of individual software modules function properly together. Individual modules may already have passed independent tests, but when they are dependent on other modules to operate successfully, this kind of testing is necessary to ensure that all parts work together as expected.

Regression Testing

Regression testing makes sure that the addition of new code does not break existing functionalities. In other words, did your new code cause the quality of your application to “regress” or go backward? Regression tests focus on recent changes and ensure that the whole application remains stable and functions as expected.

User Acceptance Testing (UAT)/Beta Testing

Usability testing involves exposing your application to a limited group of real users in a production environment. Teams use feedback from live users—who have no prior experience with the application and may uncover critical bugs unknown to internal teams—to make further changes before a full launch.

UI/UX Testing 

UI/UX testing evaluates the application’s graphical user interface. The performance of UI components such as menus, buttons, text fields, and more are verified to ensure that the user experience is ideal for the application’s users. UI/UX testing is also known as visual testing and can be manual or automated.

Other classifications of functional testing include black box testing, white box testing, component testing, API testing, system testing, and production testing.

How to Perform Functional Testing

The essence of a functional test involves three steps:

• Determine the desired test input values
• Execute the tests
• Evaluate the resulting test output values

Essentially, when you execute a task with input (e.g., enter an email address into a text field and click submit), does your application generate the expected output (e.g., the user is subscribed and a thank-you page is displayed)?

We can understand this further with a quick example.

Functional Testing Example

Let’s begin with a straightforward application: a calculator. 

To create a set of functional tests, you would need to:

• Evaluate all the possible inputs – such as numbers and mathematical symbols – and design assertions to test their functionality
• Execute the tests (either automated or manually)
• Ensure that the application generates the desired outputs—for example, each mathematical function works as intended, the final result appears correctly in all cases, and the formula history displays accurately.

For more on how to create a functional test, you can see a full guide on how to write an automated functional test for this example.

Functional Testing Techniques 

There are many functional testing techniques you might use to design a test suite for this:

• Boundary value tests evaluate what happens if inputs are received outside of specified limits – such as a user entering a number that was too large (if there is a specified limit) or attempting to enter non-numeric input
• Decision-based tests verify the results after a user decides to take an action, such as clearing the history
• User-based tests evaluate how components work together within an application – if the calculator’s history was stored in the cloud, this kind of test would verify that it did so successfully
• Ad-hoc tests can be done at the end to try and discover bugs other methods did not uncover by seeking to break the application and check its response

Other common functional testing techniques include equivalence testing, alternate flow testing, positive testing and negative testing.

Automated Functional Testing vs Manual Functional Testing

Manual functional testing requires a developer or test engineer to design, create, and execute every test by hand. It is flexible and can be powerful with the right team. However, as software grows in complexity and release windows get shorter, a purely manual testing strategy will face challenges in keeping up a large degree of test coverage.

Automated functional testing automates many parts of the testing process, allowing tests to run continuously without human interaction – and with less chance for human error. Recent improvements in AI mean that an increasing share of the design and analysis load can be handled autonomously with the right tool.

How to Use Automated Visual Testing for Functional Tests

One way to automate your functional tests is by using automated visual testing. Automated visual testing uses Visual AI to view software in the same way a human would, and can automatically highlight any unexpected differences with a high degree of accuracy.

Visual testing allows you to test for visual bugs, which are otherwise extremely challenging to uncover with traditional functional testing tools. For example, if an unrelated change shifts a ‘submit’ button to the far right of the page, making it unclickable for the user yet still technically present with the correct identifier, it would pass a traditional functional test. Visual testing, however, would catch this bug and ensure that functionality remains unaffected by visual regressions.

How to Choose an Automated Testing Tool?

Here are a few key considerations to keep in mind when choosing an automated testing tool:

• Ease of Use: Is it something easy for your existing QA team to use, or easy to hire for? Does it require an extensive learning curve or can it be picked up quickly?
• Flexibility: Can it be used across different platforms? Can it easily integrate with your current testing environment, and does it allow you the freedom to change your environment in the future?
• Reusability/AI Assistance: How easy is it to reuse tests, particularly if the UI changes? Is there meaningful AI that can help you test more efficiently, particularly at the scale you need?
• Support: What level of customer support do you require, and how easily can you receive it from the provider of your tool?

Automated testing tools can be paid or open source. Some popular open source tools include Selenium for web testing and Appium for mobile testing.

Why Choose Automated Visual Testing with Applitools

Applitools has pioneered the best Visual AI in the industry, and it’s able to automatically detect visual and functional bugs just as a human would. Our Visual AI has been trained on billions of images with 99.9999% accuracy. Applitools includes advanced features to reduce test flakiness and save time, even across the most complicated test suites.

You can find out more about the power of Visual AI through our free report on the Impact of Visual AI on Test Automation. Check out the entire Applitools platform and sign up for your own free account today.

Happy Testing!

Keep Learning

Looking to learn more about Functional Testing? Check out the resources below to find out more.

• Applitools Eyes: Modern Functional Testing [Video, 1:40]

• Modern Functional Test Automation Through Visual AI [Free Course]
• Why Should Software Testers Understand Unit Testing? [Blog]
• The End of Smoke, Sanity and Regression [Blog]
• The Impact of Visual AI On Test Automation [webinar recap]
• Cutting-edge Functional UI Testing Techniques [webinar recap]

The post What is Functional Testing? Types and Example (Full Guide) appeared first on AI-Powered End-to-End Testing | Applitools.

This is the second post in our Hello World introduction series to Appium, and we’ll discuss how to create your first Appium test for Android. You can read the first post where we discussed what Appium is, including its core concepts and how to set up the Appium server. You can also read the next post on setting up your first Appium iOS test.

Key Takeaways

In this post, we’ll build on top of earlier basics and focus on the following areas:

• Setup Android SDK
• Setup Android emulator/real devices
• Setup Appium Inspector
• Setup our first Android project with framework components

We have lots to cover but don’t worry, by the end of this post, you will have run your first Appium-based Android test. Excited? ? Let’s go.

Setup Android SDK

To run Android tests, we need to set up an Android SDK, ADB (Android debug bridge), and some other utilities. 

• Android SDK will set up Android on your machine and provide you with all the tools required to do the development or in this case automation.
• Android Debug Bridge (ADB) is a CMD line tool that lets you communicate with the Android device (either emulator or physical device).

The easiest way to set these up is to go to the Android site and download Android Studio (An IDE to develop Android apps), which will install all the desired libraries and also give us everything we need to run our first Android test.

Setup SDK Platforms and Tools

Once downloaded and installed, open Android Studio, click on Configure, and then SDK Manager. Using this you can download any Android API version from SDK Platforms.

Also, you can install any desired SDK Tools from here. 

We’ll go with the defaults for now. This tool can also install any required updates for these tools which is quite a convenient way of upgrading.

Add Android Home to Environment Variables

The Appium server needs to know where the Android SDK and other tools like Emulator, Platform Tools are present to help us run the tests. 

We can do so by adding the variables below in the system environment.

On Mac/Linux:

• Add the below environment variables in the shell of your choice (.bash_profile for bash or .zshrc for zsh). These are usually the paths where Android Studio installs these.
• Run source <shell_profile_file_name> for e.g. source.zshrc

export ANDROID_HOME=$HOME/Library/Android/sdk
export PATH=$ANDROID_HOME/emulator:$PATH
export PATH=$ANDROID_HOME/tools:$PATH
export PATH=$ANDROID_HOME/tools/bin:$PATH
export PATH=$ANDROID_HOME/platform-tools:$PATH

If you are on Windows, you’ll need to add the path to Android SDK in the ANDROID_HOME variable under System environment variables.

Once done, run the adb command on the terminal to verify ADB is set up:

➜  appium-fast-boilerplate git:(main) adb
Android Debug Bridge version 1.0.41
Version 30.0.5-6877874
Installed as /Users/gauravsingh/Library/Android/sdk/platform-tools/adb

These are a lot of tedious steps and in case you want to set these up quickly, you can execute this excellent script written by Anand Bagmar.

Set up an Android Emulator or Real Device

Our Android tests will run either on an emulator or a real Android device plugged in. Let’s see how to create an Android emulator image.

• Open Android Studio
• Click Configure
• Click on AVD Manager

You’ll be greeted with a blank screen with no virtual devices listed. Tap on Create a virtual device to launch the Virtual device Configuration flow:

Next, select an Android device like TV, Phone, Tablet, etc., and the desired size and resolution combination. 

It’s usually a good idea to set up an emulator with Play Store services available (see the Play Store icon under the column) as certain apps might need the latest play services to be installed.

We’ll go with Pixel 3a with Play Store available.

Next, we’ll need to select which Android version this emulator should have. You can choose any of the desired versions and download the image. We’ll choose Android Q (10.0 version).

You need to give this image a name. We’ll need to use this later in Appium capabilities so you can give any meaningful name or go with the default. We’ll name it Automation.

Nice, ?We have created our emulator. You can fire it up by tapping the Play icon under the Actions section.

You should see an emulator boot up on your device similar to a real phone. 

Setting up a Real Device

While the emulator is an in-memory image of the Android OS that you can quickly spin up and destroy, it does take physical resources like Memory, RAM, and CPU. It’s always a good idea to verify your app on a real device.

We’ll see how to set up a real device so that we can run automation on it.

You need to connect your device to your machine via USB. Once done:

• Go to About Phone.
• Tap on the Android build number multiple times until developer mode is enabled, you’ll see a Toast message show up as you get closer to enabling this mode.
• Go to Developer Options and Enable USB Debugging.

And thats all you need to potentially run our automation on a connected real device.

Setting up Appium Inspector

Appium comes with a nifty inspector desktop app that can inspect your Application under test and help you identify element locators (i.e. ways to identify elements on the screen) and even play around with the app. 

It can easily run on any running Appium server and is really a cool utility to help you identify element locators and develop Appium scripts.

Download and Install Appium Inspector

You can download it by just going to the Appium Github repo, and searching for appium-inspector.

Go to release and find the latest .dmg (on Mac) or .exe (on Windows) to download and install.

On Mac, you may get a warning stating: “Appium Inspector” can’t be opened because Apple cannot check it for malicious software. 

To mitigate this, just go to System preferences > Security & Privacy > General and say Open Anyway for Appium Inspector. For more details see Issue 1217.

Setting up Desired Capabilities

Once you launch Appium inspector, you’ll be greeted with the below home screen. If you see there is a JSON representation sector under Desired Capabilities section.

What are the Desired Capabilities?

Think of them as properties that you want your driver’s session to have. For example, you may want to reset your app after your test or launch the app in a particular orientation. These are all achievable by specifying a Key-Value pair in a JSON that we provide to the Appium server session. 

Please see Appium docs for more ideas on these capabilities.

Below are some sample capabilities we can give for Android:

{
    "platformName": "android",
    "automationName": "uiautomator2",
    "platformVersion": "10",
    "deviceName": "Automation",
    "app": "/<absolute_path_to_app>/ApiDemos-debug.apk",
    "appPackage": "io.appium.android.apis",
    "appActivity": "io.appium.android.apis.ApiDemos"
}

For this post, we’ll use the sample apps provided by Appium. You can see them here, once you’ve downloaded them. Keep track of its absolute path and update it in-app key in the JSON.

We’ll add this JSON under JSON representation and then tap on the Save button.

It would be a good idea to save this config for the future. You can tap on ‘Save as’ and give it a meaningful name.

Starting the Inspector Session

To start the inspection session, You need to have an Appium server run locally, run it via typing appium on the command line:

➜  appium-fast-boilerplate git:(main) appium
[Appium] Welcome to Appium v2.0.0-beta.25
[Appium] Attempting to load driver uiautomator2...
[Appium] Appium REST http interface listener started on 0.0.0.0:4723
[Appium] Available drivers:
[Appium]   - uiautomator2@2.0.1 (automationName 'UiAutomator2')
[Appium] No plugins have been installed. Use the "appium plugin" command to install the one(s) you want to use.

Let’s make sure our emulator is also up and running. We can start the emulator via AVD Manager in Android studio, or in case you are more command-line savvy, I have written an earlier post on how to do this via CMDline as well.

Once done, tap on the Start Session button, this should launch the API Demos app and show the inspector home screen.

Using this you can tap on any element in the app and see its Element hierarchy and elements properties. This is very useful to author Appium scripts and I’ll encourage you to explore each section of this tool and get familiar since you’ll be using this a lot.

Writing Our First Android Test Using Appium

Phew, that seemed like a lot of setup steps but don’t worry, you only have to do this once, and now we can get down to the real business of writing our first Automated test on Android.

You can download the project using Github from appium-fast-boilerplate.

We’ll also understand what the fundamental concepts of writing page objects and tests are based on these. Let’s take a look at the high-level architecture.

What Would This Test Do?

Before automating any test, we need to be clear on what is the purpose of that test. I’ve found the Arrange Act Assert pattern quite useful to reason about it. Read this post by Andrew Knight in case you are interested to know more about it.

Our test would perform the below:

• Opens API Demos app
• Taps around and navigates to Log Text box and taps on Add once
• Verify the log text is displayed

Our Test Class

Let’s start by seeing our test.

import constants.TestGroups;
import org.testng.Assert;
import org.testng.annotations.Test;
import pages.apidemos.home.APIDemosHomePage;

public class AndroidTest extends BaseTest {

    @Test(groups = {TestGroups.ANDROID})
    public void testLogText() {
        String logText = new APIDemosHomePage(this.driver)
                .openText()
                .tapOnLogTextBox()
                .tapOnAddButton()
                .getLogText();

        Assert.assertEquals(logText, "This is a test");
    }
}

There are a few things to notice above:

public class AndroidTest extends BaseTest

Our class extends a BaseTest, this is useful since we can perform common setup and tear-down functions, including setting up driver sessions and closing them once our script is done. 

This ensures that the tests are as simple as possible and does not overload the reader with any more details than they need to see.

String logText = new APIDemosHomePage(this.driver)
                .openText()
                .tapOnLogTextBox()
                .tapOnAddButton()
                .getLogText();

We see our tests read like plain English with a series of actions following each other. This is called a Fluent pattern and we’ll see how this is set up in just a moment.

Base Test and Driver Setup

Let’s see our BaseTest class:

import constants.Target;
import core.driver.DriverManager;
import core.utils.PropertiesReader;
import exceptions.PlatformNotSupportException;
import io.appium.java_client.AppiumDriver;
import org.testng.ITestContext;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;

import java.io.IOException;

public class BaseTest {
    protected AppiumDriver driver;
    protected PropertiesReader reader = new PropertiesReader();

    @BeforeMethod(alwaysRun = true)
    public void setup(ITestContext context) {
        context.setAttribute("target", reader.getTarget());

        try {
            Target target = (Target) context.getAttribute("target");
            this.driver = new DriverManager().getInstance(target);
        } catch (IOException | PlatformNotSupportException e) {
            e.printStackTrace();
        }
    }

    @AfterMethod(alwaysRun = true)
    public void teardown() {
        driver.quit();
    }
}

Let’s unpack this class.

protected AppiumDriver driver;

We set our driver instance as protected so that all test classes will have access to it.

protected PropertiesReader reader = new PropertiesReader();

We create an instance of PropertiesReader class to read relevant properties. This is useful since we want to be able to switch our driver instances based on different test environments and conditions. If curious, please see its implementation here.

Target target = (Target) context.getAttribute("target");
this.driver = new DriverManager().getInstance(target);

We get the relevant Target and then using that gets an instance of AppiumDriver from a class called DriverManager.

Driver Manager to Setup Appium Driver

We’ll use this reusable class to:

• Read capabilities JSON file based on platform (Android/iOS)
• Setup a local driver instance with these capabilities
• This class could independently evolve to setup desired driver instances, either local, the house or remote cloud lab

package core.driver;

import constants.Target;
import exceptions.PlatformNotSupportException;
import io.appium.java_client.AppiumDriver;
import org.openqa.selenium.remote.DesiredCapabilities;

import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.HashMap;

import static core.utils.CapabilitiesHelper.readAndMakeCapabilities;

public class DriverManager {
    private static AppiumDriver driver;
    // For Appium < 2.0, append /wd/hub to the APPIUM_SERVER_URL
    String APPIUM_SERVER_URL = "http://127.0.0.1:4723";

    public AppiumDriver getInstance(Target target) throws IOException, PlatformNotSupportException {
        System.out.println("Getting instance of: " + target.name());
        switch (target) {
            case ANDROID:
                return getAndroidDriver();
            case IOS:
                return getIOSDriver();
            default:
                throw new PlatformNotSupportException("Please provide supported target");
        }
    }

    private AppiumDriver getAndroidDriver() throws IOException {
        HashMap map = readAndMakeCapabilities("android-caps.json");
        return getDriver(map);
    }

    private AppiumDriver getIOSDriver() throws IOException {
        HashMap map = readAndMakeCapabilities("ios-caps.json");
        return getDriver(map);
    }

    private AppiumDriver getDriver(HashMap map) {
        DesiredCapabilities desiredCapabilities = new DesiredCapabilities(map);

        try {
            driver = new AppiumDriver(
                    new URL(APPIUM_SERVER_URL), desiredCapabilities);
        } catch (MalformedURLException e) {
            e.printStackTrace();
        }

        return driver;
    }
}

You can observe:

• getInstance method takes a target and based on that tries to get either an Android or an IOS Driver. In the future, if we want to run our tests against a cloud provider like Headspin, SauceLabs, BrowserStack, Applitools, then this class could handle creating the relevant session.
• Both getAndroidDriver and getIOSDriver read a JSON file with similar capabilities as we saw in the Inspector section and then convert it into a Java Hashmap which could be passed into getDriver method that returns an Appium Instance. With this, we could later pass say an environmental context and based on that choose a different capabilities file.

A Simple Page Object with the Fluent Pattern

Let’s take a look at the example of a page object that enables a Fluent pattern.

package pages.apidemos.home;

import core.page.BasePage;
import io.appium.java_client.AppiumDriver;
import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import pages.apidemos.logtextbox.LogTextBoxPage;

public class APIDemosHomePage extends BasePage {
    private final By textButton = By.xpath("//android.widget.TextView[@content-desc=\"Text\"]");
    private final By logTextBoxButton = By.xpath("//android.widget.TextView[@content-desc=\"LogTextBox\"]");

    public APIDemosHomePage(AppiumDriver driver) {
        super(driver);
    }

    public APIDemosHomePage openText() {
        WebElement text = getElement(textButton);
        click(text);

        return this;
    }

    public LogTextBoxPage tapOnLogTextBox() {
        WebElement logTextBoxButtonElement = getElement(logTextBoxButton);
        waitForElementToBeVisible(logTextBoxButtonElement);

        click(logTextBoxButtonElement);

        return new LogTextBoxPage(driver);
    }
}

Notice the following:

Above is an example page object class:

• With 2 XPath locators defined with By clause
• We init the driver in BasePage class
• The test actions return either this (i.e. current page) or next page object to enable a Fluent pattern. Following this, we can create a sort of action graph where any action that a test takes connects to the next set of actions this page can take and makes writing test scripts a breeze.

Base Page Class

package core.page;

import io.appium.java_client.AppiumDriver;
import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

import java.util.List;

public class BasePage {
    protected AppiumDriver driver;

    public BasePage(AppiumDriver driver) {
        this.driver = driver;
    }

    public void click(WebElement elem) {
        elem.click();
    }

    public WebElement getElement(By by) {
        return driver.findElement(by);
    }

    public List<WebElement> getElements(By by) {
        return driver.findElements(by);
    }

    public String getText(WebElement elem) {
        return elem.getText();
    }

    public void waitForElementToBeVisible(WebElement elem) {
        WebDriverWait wait = new WebDriverWait(driver, 10);
        wait.until(ExpectedConditions.visibilityOf(elem));
    }

    public void waitForElementToBePresent(By by) {
        WebDriverWait wait = new WebDriverWait(driver, 10);
        wait.until(ExpectedConditions.presenceOfElementLocated(by));
    }

    public void type(WebElement elem, String text) {
        elem.sendKeys(text);
    }
}

Every page object inherits from a BasePage that wraps Appium methods.

• This provides us with an abstraction and allows us to create our own project-specific reusable actions and methods. This is a very good pattern to follow for a few reasons
  • Say we want to provide some custom functionality like adding a logger line in our own logging infrastructure whenever an Appium action is performed, we can wrap those methods and perform this.
  • Also if in the future Appium breaks an API, this change is not cascaded to all our page objects, but rather only to this class where we can handle it in an appropriate manner to provide backward compatibility.
• A word of caution : Do not dump every method in this class, try to compose only relevant actions.

Congratulations, you’ve written your first Appium Android test. You can run this either via the IDE or via a Gradle command

./gradlew clean build runTests -Dtag="ANDROID" -Ddevice="ANDROID"

Conclusion

You can see the entire project on the Github appium-fast-boilerplate, clone it, and play around with it. Hopefully, this post helps you a little bit in starting on Android automation using Appium. 

In the next post, we’ll dive into the world of iOS Automation with Appium and write our first Hello World test.

You could also check out https://automationhacks.io for other posts that I’ve written about software engineering and testing. 

As always, do share this with your friends or colleagues and if you have thoughts or feedback, I’d be more than happy to chat over at Twitter or elsewhere. Until next time. Happy testing and coding.

The post Writing Your First Appium Test For Android Mobile Devices appeared first on AI-Powered End-to-End Testing | Applitools.

Learn what visual testing is, why visual testing is important, how to catch visual bugs, the differences between visual and functional testing, and how you can get started with automated visual testing today.

What is Meant By Visual Testing?

Visual testing evaluates the visible output of an application and compares that output against the results expected by design. In other words, it helps catch “visual bugs” in the appearance of a page or screen, which are distinct from strictly functional bugs. Automated visual testing tools, like Applitools, can help speed this visual testing up and reduce errors that occur with manual verification.

You can run visual tests at any time on any application with a visual user interface. Most developers run visual tests on individual components during development, and on a functioning application during end-to-end tests.

In today’s world, in the world of HTML, web developers create pages that appear on a mix of browsers and operating systems. Because HTML and CSS are standards, front-end developers want to feel comfortable with a ‘write once, run anywhere’ approach to their software. Which also translates to “Let QA sort out the implementation issues.” QA is still stuck checking each possible output combination for visual bugs.

This explains why, when I worked in product management, QA engineers would ask me all the time, “Which platforms are most important to test against?” If you’re like most QA team members, your test matrix has probably exploded: multiple browsers, multiple operating systems, multiple screen sizes, multiple fonts—and dynamic responsive content that renders differently on each combination.

If you are with me so far, you’re starting to answer the question: why do visual testing?

Why is Visual Testing Important?

We do visual testing because visual errors happen—more frequently than you might realize. Take a look at this visual bug on Instagram’s app:

The text and ad are crammed together. If this was your ad, do you think there would be a revenue impact? Absolutely.

Visual bugs happen at other companies too: Amazon. Google. Slack. Robin Hood. Poshmark. Airbnb. Yelp. Target. Southwest. United. Virgin Atlantic. OpenTable. These aren’t cosmetic issues. In each case, visual bugs are blocking revenue.

If you need to justify spending money on visual testing, share these examples with your boss.

All these companies are able to hire some of the smartest engineers in the world. If it happens to Google, Instagram, or Amazon, it probably can happen to you, too.

Why do these visual bugs occur? Don’t they do functional testing? They do — but it’s not enough.

Visual bugs are rendering issues. Rendering validation is not what functional testing tools are designed to catch. Functional testing measures functional behavior.

Why can’t functional tests cover visual issues?

Sure, functional test scripts can validate the size, position, and color scheme of visual elements. But if you do this, your test scripts will soon balloon in size due to checkpoint bloat.

To see what I mean, let’s look at an Instagram ad screen that’s properly rendered. There are 21 visual elements by my count—various icons, and text. (This ignores iOS elements at the top like WiFi signal and time, since those aren’t controlled by the Instagram app.)

If you used traditional checkpoints in a functional testing tool like Selenium Webdriver, Cypress, WebdriverIO, or Appium, you’d have to check the following for each of those 21 visual elements:

1. Visible (true/false)
2. Upper-left x,y coordinates
3. Height
4. Width
5. Background color

That means you’d need the following number of assertions:

21 visual elements x 5 assertions per element = 105 lines of assertion code

Even with all this assertion code, you wouldn’t be able to detect all visual bugs. Such as whether a visual element can’t be accessed because it’s being covered up, which blocked revenue in the above examples from Yelp, Southwest, United, and Virgin Atlantic. And, you’d miss subtleties like the brand logo, or the red dot under the heart.

But it gets worse: if OS, browser, screen orientation, screen size, or font size changes, your app’s appearance will change as a result. That means you have to write another 105 lines of functional test assertions. For EACH combination of OS/browser/font size/screen size/screen orientation/font size.

You could end up with thousands of lines of assertion code — any of which might need to change with a new release. Trying to maintain that would be sheer madness. No one has time for that.

You need visual testing because visual errors occur. And you need visual testing because you cannot rely on functional tests to catch visual errors.

What is Manual Visual Testing?

Because automated functional testing tools are poorly suited for finding visual bugs, companies find visual glitches using manual testers. Lots of them (more on that in a bit).

For these manual testers, visual testing behaves a lot like this spot-the-difference game:

To understand how time-consuming visual testing can be, get out your phone and time how long it takes for you to find all six visual differences. I took a minute to realize that the writing in the panels doesn’t count. It took me about 3 minutes to spot all six. Or, you can cheat and look at the answers.

Why does it take so long? Some differences are difficult to spot. In other cases, our eyes trick us into finding differences that don’t exist.

Manual visual testing means comparing two screenshots, one from your known good baseline image, and another from the latest version of your app. For each pair of images, you have to invest time to ensure you’ve caught all issues. Especially if the page is long, or has a lot of visual elements. Think “Where’s Waldo”…

Challenges of manual testing

If you’re a manual tester or someone who manages them, you probably know how hard it is to visually test.

If you are a test engineer reading this paragraph, you already know this: web page testing only starts with checking the visual elements and their function on a single operating system, browser, browser orientation, and browser dimension combination. Then continue on to other combinations. And, that’s where a huge amount of test effort lies – not in the functional testing, but in the inspection of visual elements across the combination of an operating system, browser, screen orientation, and browser dimensions.

To put it in perspective, imagine you need to test your app on:

• 5 operating systems: Windows, MacOS, Android, iOS, and Chrome.
• 5 popular browsers: Chrome, Firefox, Internet Explorer (Windows only) Microsoft Edge (Windows Only), and Safari (Mac only).
• 2 screen orientations for mobile devices: portrait and landscape.
• 10 standard mobile device display resolutions and 18 standard desktop/laptop display resolutions from XGA to 4G.

If you’re doing the math, you think it’s the browsers running on each platform (a total of 21 combinations) multiplied by the two orientations of the ten mobiles (2×10)=20 added to the 18 desktop display resolutions.

21 x (20+18) = 21 x 38 = 798 Unique Screen Configurations to test

That’s a lot of testing—for just one web page or screen in your mobile app.

Except that it’s worse. Let’s say your app has 100 pages or screens to test.

798 Screen Configurations x 100 Screens in-app = 79,800 Screen Configurations to test

Meanwhile, companies are releasing new app versions into production as frequently as once a week, or even once a day.

How many manual testers would you need to test 79,800 screen configurations in a week? Or a day? Could you even hire that many people?

Wouldn’t it be great if there was a way to automate this crazy-tedious process?

Well, yes there is…

What is Automated Visual Testing?

Automated visual testing uses software to automate the process of comparing visual elements across various screen combinations to uncover visual defects.

Automated visual testing piggybacks on your existing functional test scripts running in a tool like Selenium Webdriver, Cypress, WebdriverIO, or Appium. As your script drives your app, your app creates web pages with static visual elements. Functional testing changes visual elements, so each step of a functional test creates a new UI state you can visually test.

Automated visual testing evolved from functional testing. Rather than descending into the madness of writing assertions to check the properties of each visual element, automated visual testing tools visually check the visual appearance of an entire screen with just one assertion statement. This leads to test scripts that are MUCH simpler and easier to maintain.

But, if you’re not careful, you can go down an unproductive rat hole. I’m talking about Snapshot Testing.

What is Snapshot Testing?

First-generation automated visual testing uses a technology called snapshot testing. With snapshot testing, a bitmap of a screen is captured at various points of a test run and its pixels are compared to a baseline bitmap.

Snapshot testing algorithms are very simplistic: iterate through each pixel pair, then check if the color hex code is the same. If the color codes are different, raise a visual bug.

Because they can be built relatively easily, there are a number of open-source and commercial snapshot testing tools. Unlike human testers, snapshot testing tools can spot pixel differences quickly and consistently. And that’s a step forward. A computer can highlight the visual differences in the Hocus Focus cartoon easily. A number of these tools market themselves as enabling “pixel-perfect testing”.

Sounds like a good idea, right?

What are the Problems With Snapshot Testing?

Alas, pixels aren’t visual elements. Font smoothing algorithms, image resizing, graphics cards, and even rendering algorithms generate pixel differences. And that’s just static content. The actual content can vary between any two interfaces. As a result, a tool that expects exact pixel matches between two images can be filled with pixel differences.

If you want to see some examples of bitmap differences affecting snapshot testing, take a look at the blog post we wrote on this topic last year.

Unfortunately, while you might think snapshot testing makes intuitive sense, practitioners like you are finding that the conditions for running successful bitmap comparisons require a stationary target, while your company continues to develop dynamic websites across a range of browsers and operating systems. You can try to force your app to behave a certain way – but you may not always succeed.

Can you share some details of Snapshot Testing Problems?

For example, when testing on a single browser and operating system:

• Identify and isolate (mute) fields that change over time, such as radio signal strength, battery state, and blinking cursors.
• Ignore user data that might otherwise change over time, such as visitor count.
• Determine how to support testing content on your site that must change frequently – especially if you are a media company or have an active blog.
• Consider how different hardware or software affects antialiasing.

When doing cross-browser testing, you must also consider:

• Text wrapping, because you cannot guarantee the locations of text wrapping between two browsers using the same specifications. The text can break differently between two browsers, even with identical screen sizes.
• Image rendering software, which can affect the pixels of font antialiasing as well as images and can vary from browser to browser (and even on a single browser among versions).
• Image rendering hardware, which may render bitmaps differently.
• Variations in browser font size and other elements that affect the text.

If you choose to pursue snapshot testing in spite of these issues, don’t be surprised if you end up joining the group of experienced testers who have tried, and then ultimately abandoned, snapshot testing tools.

Can I See Some Snapshot Testing Problems In Real Life?

Here are some quick examples of these real-life bitmap issues.

If you use pixel testing for mobile apps, you’ll need to deal with the very dynamic data at the top of nearly every screen: network strength, time, battery level, and more:

When you have dynamic content that shifts over time — news, ads, user-submitted content — where you want to check to ensure that everything is laid out with proper alignment and no overlaps. Pixel comparison tools can’t test for these cases. Twitter’s user-generated content is even more dynamic, with new tweets, likes, retweets, and comment counts changing by the second.

Your app doesn’t even need to change to confuse pixel tools. If your baselines and test screenshots were captured on different machines with different display settings for anti-aliasing, that can turn pretty much the entire page into a false positive, like this:

Source: storybook.js.org

If you’re using pixel tools and you still have to track down false positives and expose false negatives, what does that say about your testing efficiency?

For these reasons, many companies throw out their pixel tools and go back to manual visual testing, with all of its issues.

There’s a better alternative: using AI—specifically computer vision—for visual testing.

How Do I Use AI for Automated Visual Testing?

The current generation of automated visual testing uses a class of artificial intelligence algorithms called computer vision as a core engine for visual comparison. Typically these algorithms are used to identify objects with images, such as with facial recognition. We call them visual AI testing tools.

AI-powered automated visual testing combines a learning algorithm to interpret the relationship between a rendered page and the intended display of visual elements with actual visual elements and locations. Like pixel tools, AI-powered automated visual testing takes page snapshots as your functional tests run. Unlike pixel-based comparators, AI-powered automated visual test tools use algorithms instead of pixels to determine when errors have occurred.

Unlike snapshot testers, AI-powered automated visual testing tools do not need special environments that remain static to ensure accuracy. Testing and real-world customer data show that AI testing tools have a high degree of accuracy even with dynamic content because the comparisons are based on relationships and not simply pixels.

Here’s a comparison of the kinds of issues that AI-powered visual testing tools can handle compared to snapshot testing tools:

Some AI-powered test tools have been tested at a false positive rate of 0.001% (or 1 in every 100,000 errors).

AI-Powered Test Tools In Action

An AI-powered automated visual testing tool can test a wide range of visual elements across a range of OS/browser/orientation/resolution combinations. Just running the first baseline of rendering and functional test on a single combination is sufficient to guide an AI-powered tool to test results across the range of potential platforms

Here are some examples of how AI-powered automated visual testing improves visual test results by awareness of content.

This is a comparison of two different USA Today homepage images. When an AI-powered tool looks at the layout comparison, the layout framework matters, not the content. Layout comparison ignores content differences; instead, layout comparison validates the existence of the content and relative placement. Compare that with a bitmap comparison of the same two pages (also called “exact comparison:):

Literally, every non-whites pace (and even some of the white space) is called out.

Which do you think would be more useful in your validation of your own content?

When Should I Use Visual Testing?

You can do automated visual testing with each check-in of front-end code, after unit testing and API testing, and before functional testing — ideally as part of your CI/CD pipeline running in Jenkins, Travis, or another continuous integration tool.

How often? On days ending with “y”.

Because of the accuracy of AI-powered automated visual testing tools, they can be deployed in more than just functional and visual testing pre-production. AI-powered automated visual testing can help developers understand how visual element components will render across various systems. In addition to running in development, test engineers can also validate new code against existing platforms and new platforms against running code.

AI-powered tools like Applitools allow different levels of smart comparison.

AI-powered visual testing tools are a key validation tool for any app or web presence that requires regular changes in content and format. For example, media companies change their content as frequently as twice per hour and use AI-powered automated testing to isolate real errors that affect paying customers without impacting them. AI-powered visual test tools are key tools in the test arsenal for any app or web presence going through brand revision or merger, as the low error rate and high accuracy let companies identify and fix problems associated with major DOM, CSS, and Javascript changes that are core to those updates.

Talk to Applitools

Applitools is the pioneer and leading vendor in AI-powered automated visual testing. Applitools has a range of options to help you become incredibly productive in application testing. We can help you test components in development. We can help you find the root cause of the visual errors you have encountered. And, we can run your tests on an Ultrafast Grid that allows you to recreate your visual test in one environment across a number of others on various browser and OS configurations. Our goal is to help you realize the vision we share with our customers – you need to create functional tests for only one environment and let Applitools run the validation across all your customer environments after your first test has passed. We’d love to talk testing with you – feel free to reach out to contact us anytime.

More To Read About Visual Testing

If you liked reading this, here are some more Applitools posts and webinars for you.

1. Visual Testing for Mobile Apps by Angie Jones
2. Visual Assertions – Hype or Reality? – by Anand Bagmar
3. The Many Uses of Visual Testing by Angie Jones
4. Visual UI Testing as an Aid to Functional Testing by Gil Tayar
5. Visual Testing: A Guide for Front End Developers by Gil Tayar

Find out more about Applitools. Set up a live demo with us, or if you’re the do-it-yourself type, sign up for a free Applitools account and follow one of our tutorials.

Quick Answers

The post What is Visual Testing? appeared first on AI-Powered End-to-End Testing | Applitools.

The post Resources to Help You Stay on Top of Testing Trends and Techniques appeared first on AI-Powered End-to-End Testing | Applitools.

  cy.get('h2')
  cy.get('.board') // class
  cy.get('#board-1') // id
  cy.get('[data-cy=board-item]') // attribute

  cy.contains('My Shopping') // text
  cy.contains('My') // find the first one
  cy.contains('.board', 'My') // specify element to find

// Filter an element by index
cy.xpath('(//div(contains(@class, "board") ]) [1]')

// Select an element containing a specific child element
cy.xpath('//div [contains(@class, "list")] [.//div[contains(@class, "card")]]')

// Select an element by text
cy.xpath('//*[text()[contains(., "My Boards")]]')

// Select an element after a specific element
cy.xpath('//div[contains(@class, "card")][preceding::div[contains(., "milk")]]')

//Filter an element by index
cy.get('.board').eq(0)

// Select an element containing a specific child element
cy.get(".card").parents('.list')

// Select an element by text
cy.contains('My Boards')

// Select an element after a specific element
cy.contains('.card', 'milk').next('.card')

it.only('Find an element on page', () => {

  cy.visit('/board/1')

  // find last card
  cy.get('[data-cy=card]')
    .last()

  // find parent element
  cy.contains('[data-cy=card]', 'Soap')
    .parents('[data-cy=list]')

  // find next element
  cy.contains('[data-cy=card]', 'Milk')
    .next()

  // find next element
  cy.contains('[data-cy=card]', 'Bread')
    .prev()

});

it('Dealing with flaky situations', () => {

  cardsLoadRandomly(10000)

  cy.visit('/board/1')

  cy.get('[data-cy=card]')
    .last()
    .should('contain.text', 'Soap')

});

it('closes side panel that is in shadow DOM', () => {

  cy.visit('https://lit.dev/playground/#sample=docs%2Fwhat-is-lit&view-mode=preview')

  cy.get('litdev-drawer', { includeShadowDom: true })
    .find('#openCloseButton', { includeShadowDom: true })
    .click()

});

it('dealing with iframes', () => {

  cy.visit('https://kitchen.applitools.com/ingredients/iframe')

  cy.iframe('#the-kitchen-table')
    .find('section')
    .should('contain.text', 'The Kitchen')

});

it('check home screen', () => {

  cy.eyesOpen({
    appName: 'Trello',
  })

  cy.visit('/')

  cy.get('[data-cy=board-item]')
    .should('be.visible')

  cy.eyesCheckWindow({
    ignore: {
      selector: 'hide-in-applitools'
    }
  })

  cy.eyesClose()

});

Learn more

So that covers our expert’s takeaways on locating web elements using Cypress. If you want to watch the demos, you can access the on-demand webinar recording.

If you want to learn more about any of these tools, frameworks like Cypress, WebDriverIO, or specifically web element locator strategies, be sure to check out Test Automation University. All the courses and content are free.
Be sure to register for the upcoming Let the Engineers Speak webinar series installment on Test Maintainability coming in May. Engineers Maaret Pyhäjärvi from Selenium and Ed Manlove from Robot Framework will be discussing test maintenance and test maintainability with a live Q&A.

Let the Engineers Speak! Part 5: Audience Q&A

Articles — 01.11.2023

Let the Engineers Speak! Part 4: Changing Frameworks

Articles — 01.05.2023

Let the Engineers Speak! Part 3: Favorite Test Integrations

Articles — 12.28.2022

The post Let the Engineers Speak: Selectors in Cypress appeared first on AI-Powered End-to-End Testing | Applitools.