Core concepts

Understanding Eyes fundamentals helps you design stable, maintainable visual tests for component libraries.

Applitools Visual AI and advanced match levels

Traditional visual testing tools often rely on pixel-to-pixel comparisons, which can result in numerous false positives due to minor rendering differences. These can be caused by factors such as anti-aliasing, font smoothing, or subtle layout shifts, making tests brittle and time-consuming to maintain. Such methods struggle to scale and require constant adjustment of diff thresholds, which are difficult to configure and maintain reliably.

Advanced image comparison algorithms

Applitools Visual AI overcomes these challenges by utilizing advanced image comparison algorithms that have been developed and refined over more than a decade. These algorithms simulate human vision, allowing Applitools Eyes to detect only meaningful visual differences that could impact the user experience. The key match levels provided by Applitools are:

• Strict (Default): Detects any meaningful differences that are visible to the human eye. Ideal for static pages where any change is significant.
• Layout: Ignores content changes but detects shifts in layout or structure. Useful for pages with dynamic content or responsive designs, and for localization testing where content varies but layout should remain consistent.
• Ignore Colors: Ignores color differences but detects changes in content and layout. Helpful when color variations are expected or irrelevant, such as in applications with theming support.
• Dynamic (New!): Designed to handle dynamic content gracefully by automatically detecting and suppressing diffs due to dynamic data such as emails, dates, credit card numbers, etc.

These match levels can be applied to entire pages or mixed and matched on different regions of the page, giving you granular control over your visual comparisons.

Superiority over pixel-to-pixel matching

By leveraging Visual AI and these advanced match levels, Applitools Eyes addresses the limitations of traditional pixel-to-pixel comparison:

• Reduced false positives: By ignoring insignificant differences that are invisible to the human eye, Visual AI minimizes unnecessary test failures caused by minor rendering variations.
• Enhanced test stability: Tests are less brittle and more resilient to changes in rendering environments, making them stable across different browsers, devices, and operating systems.
• Improved scalability: There's no need to configure diff thresholds or constantly adjust your tests to account for environmental differences, allowing your test suite to scale efficiently.
• Efficient maintenance: Visual AI reduces the maintenance burden by eliminating flakiness, and when intentional changes occur, you can approve them once in the Applitools Dashboard, propagating the acceptance across all relevant tests and checkpoints.
• Contextual understanding: Applitools Eyes understands the content and structure of your application's UI, focusing on perceptible changes that a human user would notice, rather than pixel-level differences.

By adopting Applitools Visual AI, you can trust that your visual tests are highlighting real issues that could affect your users, allowing you to focus on delivering a high-quality user experience without being bogged down by false positives or maintenance overhead.

Baseline images and management

A core concept in Applitools Eyes is the use of baselines, which represent the expected visual appearance of your application at specific points in time.

Automatic baseline management

When you run tests with Applitools Eyes, the SDK automatically detects the baseline signature based on factors such as the application name, test name, browser, operating system, viewport size, and branch. This ensures that the correct baseline is used for each specific test scenario.

When the only change between runs is the host application or OS version (for example, Chrome 130.x → 131.x or iOS 14.x → 15.x) and there isn’t an exact baseline for the new version yet, Eyes can fall back to the closest available version within the same browser/OS family so your tests keep comparing meaningfully during rollouts.

• Version-fallback scope: same browser family (Chrome→Chrome, Safari →Safari), same OS/device class, same viewport. It does not cross different browsers (Chrome↔Firefox) or platforms (iOS↔Android).
• First run on a new version: once you review/approve, Eyes saves a baseline for that specific version so subsequent runs compare against it directly.

Alignment with checkpoint images
Eyes detects new or missing checkpoints and associates them with the appropriate baselines, highlighting any discrepancies for review.

Baseline creation and updating

• First run (baseline creation):
  • When you run a test for the first time, Applitools Eyes captures baseline images for each visual checkpoint.
  • These baselines are stored securely in the Applitools cloud, eliminating the need to manage image files locally or within your code repository.
• Subsequent runs (visual comparison):
  • In subsequent test runs, Applitools Eyes captures new screenshots and compares them to the existing baselines.
  • If differences are detected, they are highlighted in the test results for review.
• Updating baselines:
  • If changes are intentional (e.g., a UI redesign), you can accept the new images in the Applitools Dashboard.
  • Accepted images become the new baselines for future comparisons.
  • Automated test maintenance allows you to accept changes once and propagate them across all relevant tests, devices, and browsers, saving time and effort.

By automatically managing baselines, Applitools Eyes simplifies the maintenance of your visual tests, ensuring accurate comparisons without the overhead of manual baseline management.

Match levels for tailored comparisons

Match levels determine how strictly Applitools Eyes compares new screenshots to the baselines, allowing you to balance between catching important differences and ignoring insignificant changes.

Available match levels

As previously mentioned, Applitools offers several match levels:

• Strict (Default): Detects any meaningful differences that are visible to the human eye. Ideal for static pages where any change is significant.
• Layout: Ignores content changes but detects shifts in layout or structure. Useful for testing highly dynamic pages and localization, where content varies but layout should remain consistent.
• Ignore Colors: Ignores color differences but detects changes in content and layout. Helpful when color variations are expected or irrelevant, such as in applications with theming support.
• Dynamic (New!): Designed to handle dynamic content gracefully by automatically detecting and suppressing diffs due to dynamic data such as emails, dates, credit card numbers, etc.

These match levels can be applied to entire pages or mixed and matched on different regions of the page, giving you granular control over your visual comparisons.

Choosing the appropriate match level

Selecting the right match level depends on the nature of your application and the specific page or component you're testing.

• Dynamic content: Use Layout or Content match levels to avoid false positives due to changing text or images.
• Brand consistency: Use Strict match level to ensure that styling and branding elements remain consistent.
• Theming support: Use Ignore Colors if your application supports different themes or color schemes.

Setting match levels

You can set the match level per story or per component.

storyConfiguration: [
  {
    stories: ({ storyTitle, id, name, kind, parameters }) =>
      storyTitle.includes('image'),
    matchLevel: 'layout',
    target: 'region',
    selector: { selector: '.some-selector' },
    browser: [{ width: 1024, height: 600 }],
  },
  {
    stories: ({ storyTitle, id, name, kind, parameters }) =>
      id === 'button--with-text',
    matchLevel: 'strict',
    browser: [{ name: 'firefox', width: 1024, height: 600 }],
  },
];

By choosing the appropriate match level, you can effectively manage dynamic content and focus on the changes that matter most.

Branching and merging

Applitools supports branching and merging to align with your development workflow, facilitating collaboration and maintaining consistency across your application.

Pull request workflow and baseline management

When working with feature branches and pull requests, Applitools Eyes facilitates an effective workflow:

• Branch-specific baselines: Each branch can maintain its own set of baselines, allowing you to develop and test features in isolation without impacting the main baseline.
• Baseline continuity: When you create a new branch, Applitools automatically creates a copy of the baseline from the parent branch, or the default default branch. This ensures consistency and reduces the need to re-establish baselines.
• Automatic merging: Upon merging a feature branch into the main branch, Applitools can automatically merge the baselines, integrating any accepted changes from the feature branch into the main baseline.
• Conflict resolution: If there are conflicting changes in the baselines (e.g., the same element was modified differently in both branches), Applitools provides tools to resolve these conflicts, allowing you to choose which changes to keep.

Handling a pull request with visual differences

1. Develop on a feature branch:
   • Create a new branch in your version control system.
   • Applitools Eyes automatically uses this branch name to manage baselines.
   • Heads-up (SCM integration required): Automatic branch/PR detection works when you connect your SCM to Eyes. Set up the integration for your provider (e.g., GitHub App, GitLab, Azure DevOps, Bitbucket). Otherwise, set branchName in your config.
2. Run your visual tests:
   • Any visual changes are compared against the baseline for the branch.
   • Review and accept intentional changes in the Applitools Dashboard.
3. Create a pull request:
   • Applitools can integrate with your CI/CD pipeline to run tests on pull requests.
   • Visual test results can be linked in the pull request for reviewers to assess.
4. Review and resolve conflicts:
   • If there are conflicts between the feature branch and the main branch baselines, use the Applitools Dashboard to resolve them.
   • Decide whether to accept changes from the feature branch, main branch, or a combination.
5. Merge the pull request:
   • Baselines are merged according to your resolutions.
   • The main branch baseline is updated to include the accepted changes.

By supporting branching and merging workflows, Applitools Eyes ensures that your visual tests align with your development processes, facilitating collaboration and maintaining consistency across your application.