Local dev is where most UI changes happen (and where regressions sneak in). States drift, styles diverge, and tiny tweaks pile up until something breaks in CI. The Applitools Storybook Addon brings AI-powered visual testing straight into Storybook so you can catch issues as you code, approve the good changes quickly, and keep your CI/CD pipelines green.

AI-Powered Visual Testing Inside Storybook

Open your Storybook and run visual tests from an Applitools Eyes tab – no context switching. Results are grouped by component to mirror your Storybook structure, and a reporter widget highlights what needs attention first so you can review diffs in minutes, not hours. Learn more on our Storybook Component Testing with Applitools page.

• Catch bugs where you build. Validate component states during local development and avoid surprises later.
• Review faster with Visual AI. See only meaningful, human-perceptible UI changes without pixel-to-pixel noise. Tune sensitivity with AI match levels when you need to.
• Scale coverage painlessly. Run once; render everywhere with Ultrafast Grid across browsers, devices, and viewports in parallel.

How to Use the Applitools Eyes Storybook Addon

Getting started takes just a couple of minutes.

1. Install the SDK & Addon
   Add Applitools Eyes to your project and enable the Storybook addon (React, Vue, Angular supported). See the installation instructions in the Eyes Storybook Addon docs.
2. Run Applitools Visual Tests in Storybook
   Open Storybook, switch to the Applitools Eyes tab, and trigger tests for a single story or an entire component. Results stream back in real time with automatic grouping by component.
3. Review & Maintain
   Use Visual AI diffs, side-by-side views, and auto-maintenance to approve or reject changes in bulk. Prioritized sorting surfaces what needs attention first.
4. Scale Across Browsers/Devices
   Turn on Ultrafast Grid to parallelize renders across Chrome, Firefox, Safari, Edge, and mobile sizes – without extra local setup.

Applitools Storybook Addon Use Case Playbook

Below are the three most common ways teams use the Eyes Storybook Addon each – with a quick, practical flow pulled right from the product.

Use Case: Guard Your Design System

As you refactor tokens or update themes, run visual tests on every component state. Spot unintended changes across the library instantly.

How to do it in Storybook

1. Start Storybook and open your design‑system component in the Applitools Eyes tab.
2. Click Run from the tab (or use Run in the left sidebar test module). The addon tests the stories and streams results inline for every browser/device in your applitools.config.js.
3. In the sidebar, filter by Unresolved to zero in on changes across the library (Green = passed, Orange = unresolved, Red = failed).
4. Open a story’s result and use Side‑by‑Side or the Slider to spot subtle spacing/typography diffs.
5. Approve legit theme updates with Thumbs Up (or use ⋯ → Review actions to approve the whole story/batch). Reject regressions with Thumbs Down and fix.

Pro tip: Use the tab ⋯ → Configuration to confirm you’re validating the right browser matrix and server URL. See more options in the docs.

Use Case: Fix Fast During Local Dev

Working on a feature branch? Validate your component in Storybook before you commit.

How to do it in Storybook

1. Open your feature’s stories, then hit Run in the Applitools tab for the component you’re touching.
2. Watch statuses update inline; click the status buttons to filter to Unresolved so you only look at what changed.
3. Click into any row to open compare tools: Diff Image, Actual Image, Expected Image, Side‑by‑Side, or Slider.
4. If the change is intended, Thumbs Up to approve; otherwise Thumbs Down to flag and keep iterating.
5. When you’re happy locally, push your branch. You can scale the same setup in CI using your existing Storybook build/preview URL.

Heads‑up: To view baselines or approve/reject, sign in to your Applitools account in the same browser that’s running Storybook (you’ll be prompted if not).

Use Case: Ship Multi‑Browser Confidence

One click, many targets. Validate layout and responsive behavior across browsers and viewports – early.

How to do it in Storybook

1. In ⋯ → Configuration, verify your browsers/devices list (Chrome, Firefox, Safari, Edge; add viewports you care about).
2. Hit Run for representative stories (states, theming, interactive). Results come back grouped by each browser/device so differences are obvious.
3. Filter the sidebar by Unresolved and scan. Use Side‑by‑Side or Slider to compare layout at different sizes.
4. Approve good changes in bulk (⋯ → Review actions) to keep maintenance low.
5. For broader coverage, run the same setup in CI and expand the matrix.

Why Visual AI > Pixel Diffs for Storybook

Pixel-to-pixel tools are fragile with dynamic content and minor rendering differences. Applitools Visual AI mimics human vision to highlight only meaningful UI changes (structure, layout, content) while ignoring the noise. You can still dial sensitivity up or down with match levels whenever needed. Less flake, more signal.

Try AI-Powered Visual Testing in Storybook Today

Run your first component tests in minutes, review diffs right in Storybook, and expand coverage with Ultrafast Grid – without slowing delivery.

• Explore the Applitools Storybook Component Testing solution
• Read the Applitools Storybook Addon docs
• Compare Applitools Eyes against Chromatic

The post Test Your Components Where You Build with the Applitools Storybook Addon appeared first on AI-Powered End-to-End Testing | Applitools.

The new Applitools Eyes 10.22 release brings Visual AI testing directly to the tools teams already use to design and develop digital experiences. This update strengthens how teams build, validate, and release with three major enhancements:

• Scale quality without slowing down delivery using the new Storybook Addon.
• Align development and design with the new Figma Plugin, bridging intent and implementation.
• Improve traceability and accountability with smarter Dashboard Optimizations for QA and leadership.

Together, these updates make visual testing faster, sharper, and more connected across the product pipeline, helping teams fully embrace shift-left testing and collaborate around a shared baseline for visual quality.

Scale Quality Without Slowing Down Delivery: Storybook Addon

Test Where You Build with the Storybook Addon

The new Storybook Addon for Applitools Eyes brings storybook visual testing directly into the development workflow. Developers can now validate UI components where they build them—no CI/CD wait times, no switching between tools.

Core Capabilities

• Run and review visual tests directly inside your component workspace.
• Approve or reject diffs inline and validate changes locally before merging.
• Leverage higher concurrency for faster feedback.
• Catch visual issues early and prevent CI failures that slow releases.

Use Case Example:
A frontend developer refactoring a shared button component runs Eyes tests directly in Storybook to verify that the update didn’t alter its appearance across the design system. By catching issues immediately, they avoid regressions on the main branch and prevent unnecessary rework later in the release cycle.

The Applitools Eyes Storybook Addon allows teams to scale quality by expanding coverage while maintaining release velocity. By embedding visual validation where developers work, organizations can expand test coverage and reduce release timelines without compromising accuracy.

See the newest features in action during the Platform Pulse webinar, coming October 22.

Align Development and Design: Figma Plugin

Bridge Design & Code with the Figma Plugin

The new Applitools Figma Plugin connects design intent with implementation through seamless Figma design testing and design-to-code comparison. Designers and developers can now validate visual consistency earlier—and with far less manual effort.

What You Can Do

• Export Figma frames directly into Eyes for automated design-to-code validation.
• Compare design-to-design versions to manage iteration cycles.
• Maintain shared baselines so designers and developers work from the same source of truth.
• Designers can validate implementations without writing code, while developers see exactly what “done” looks like.

Use Case Example:
A designer exports a component from Figma and compares it with the implemented version in Eyes to confirm that spacing, colors, and typography match the original design. Any differences appear immediately, allowing the designer and developer to resolve them early instead of during QA or after release.

This plugin bridges design and development, reducing handoff friction and review cycles. Teams gain a single source of truth for UI quality and deliver products that match design intent from concept to code.

Improve Traceability and Accountability: Dashboard Optimizations

Context-Rich Results with Dashboard Optimizations

Eyes 10.22 introduces dashboard enhancements that bring clarity and accountability to every test result. These updates help QA engineers, SDETs, and technical leaders focus on what matters most.

Key Highlights

• Commit SHA and branch info appear directly in batch details for instant traceability.
• “Required Attention First” sorting automatically surfaces unresolved and failed tests.
• Auto-grouping mirrors your Storybook structure, making results easier to navigate.
• Streamlined triage reduces noise from passed tests and accelerates review.

Use Case Example:
A QA engineer reviewing test results spots a visual regression and uses the Eyes dashboard to trace it back to the specific commit that introduced the change. With commit and branch details in context, the engineer can alert the right developer and close the feedback loop quickly.

These context-rich results strengthen accountability and compliance across teams. Organizations gain the visibility they need for auditability—particularly valuable for industries with governance or regulatory requirements.

Explore all the updates and improvements in the Eyes 10.22 Release Notes.

Broader Strategic Impact

Eyes 10.22 delivers improvements that go beyond new capabilities. Together, these updates create a foundation for stronger collaboration and faster releases across the organization:

• Align Development & Design Around a Single Source of Truth – With the Storybook addon and Figma plugin, teams collaborate around shared visual baselines.
• Scale Quality Without Slowing Delivery – Native testing inside development tools ensures visual coverage grows with velocity.
• Improve Traceability & Accountability – Git-linked context and prioritized review queues make it easier to understand, resolve, and communicate changes.
• Drive Shift-Left Adoption Across the Organization – With validation embedded where work happens, teams can catch issues earlier and release with confidence.

Quality That Scales: Visual AI for Every Team

With Applitools Eyes 10.22, teams can test where they build—bringing Visual AI testing directly into their design and development workflows. By embedding validation inside Storybook, Figma, and the Eyes dashboard, organizations can scale software quality without adding maintenance or slowing delivery. Developers, designers, and QA now share one intelligent workflow powered by Visual AI—delivering expanded test coverage, faster feedback, and less maintenance across every stage of the testing lifecycle. Learn more about Applitools Eyes.

The post Test Where You Build with Eyes 10.22: Visual AI for Storybook & Figma appeared first on AI-Powered End-to-End Testing | Applitools.

Have you ever encountered a situation where you try to take a screenshot, but instead of the beautifully well-crafted UI, all you’ve got is an image of a spinner/skeleton/loading screen? Handling animations and loading artifacts in visual testing can be daunting.

Don’t worry – it can happen to anyone, and we’re not here to judge you

One of the SDK engineers here at Applitools, Noam, breaks this down into a hands-on tutorial, hoping it will help you get a better understanding of the industry’s best practices around visual testing of rich and dynamic UI experiences.

Let’s dive right in!

Framework Native Solutions

Most frameworks already have different mechanisms to handle animations and loading artifacts. Keeping things simple is often the best way to achieve code stability and maintainability. Using your framework’s built-in tools would most often be the best approach.

For example:

Playwright JS

// Playwright: wait for spinner to be removed
await page.waitForSelector('.spinner', { state: 'detached' });
await eyes.check()

Cypress

// Cypress: wait for spinner to not exist
cy.get('.spinner').should('not.exist'); 
cy.eyesCheck()

Selenium JS

// Selenium: wait for spinner to be invisible
const spinnerElements = await driver.findElements(By.css('.spinner'));
if (spinnerElements.length) {
  await driver.wait(until.elementIsNotVisible(spinnerElements[0]), 5000);
}
await eyes.check()

A Common Pitfall

Even if the UI appears visually unchanged, frontend frameworks like React, Vue, and Angular may re-render elements under the hood. This can lead to stale element references, especially when capturing regions right after a DOM change.

Consider the following example:

cy.get('.main').then($el => {
  cy.get('.spinner').should('not.exist'); // spinner disappears after main was located

  cy.eyesCheckWindow({
    tag: 'main',
    target: 'region',
    element: $el, // stale reference if .main was replaced
  });
});

1. First, Cypress locates .main
2. Then, Cypress waits for the spinner to disappear
3. This example would fail (even if the new element has the exact same properties) if the main element is replaced by another element

How to avoid that?

When possible (e.g., Playwright), it’s preferred that you use locators instead of selectors. If you can’t, it’s better if you use selectors instead of DOM references (element: ‘.main’).

Videos, CSS Animations, GIFs

There are many techniques to eliminate other types of dynamic behaviour in web pages. Playwright, for example, provides a Clock API that allows pausing JavaScript time-related events (including JS-driven animations). It’s also possible to install custom CSS snippets to pause and reset CSS-related animations. Other JS specialized crafted snippets would be required for resetting GIFs, videos, and so on – you get the idea.

This never-ending cat-and-mouse game can be prevented by using Applitools Ultrafast Grid (UFG). Instead of rendering web pages on locally executed browsers, the UFG team maintains specialized logics and fine-tuned commands that ensure a stable and consistent rendering experience. While UFG offers more than just rendering stability, it’s worth noting that classic screenshots can still achieve stable results. UFG just makes it easier!

Algo-Based Solutions

If you intentionally want to capture dynamic content (e.g., animations, changes), a smarter strategy is to embrace that variability and use smart matching algorithms to compare just what you need, like those found in Applitools Eyes.

Any match level can be used for the entire screenshot or specific regions of the screen. Read more in the Match Level Best Practices tutorial. For example, algorithms like the Layout match level can drastically improve your experience with localization testing.

The waitBeforeCapture Setting

Performing wait operations can become more complicated when:

1. Testing with no-code visual testing SDKs (e.g eyes-storybook)
2. Testing with advanced Eyes features like lazyLoading and layoutBreakpoints

The waitBeforeCapture setting was invented for these types of use cases (and a few others).

This setting can receive three types of arguments:

1. Milliseconds – the simplest approach. While it’s not always the most innovative or sophisticated pattern, in many cases, it “does the trick.” In general, waiting for explicit timeouts during tests is not recommended. However, when compared to clock manipulations and code injections, sometimes the simplicity and stability is worth the longer run-time.
2. Selector – when we’re waiting for something to appear, most SDKs support passing a selector, and Applitools Eyes will automatically wait for an element that matches this selector to appear in the web page.
3. Custom function – see code example

// eyes-storybook
  waitBeforeCapture: async () => {
    while (document.querySelector('.spinner')) {
      await new Promise((resolve) => setTimeout(resolve, 100))
    }
    return true;
  }

// eyes-playwright
await eyes.check({
  name: 'my-step',
  async waitBeforeCapture() {
    await page.locator('.spinner').waitFor({state: 'hidden'})
  },
})

The waitBeforeCapture setting can be defined in your applitools.config file, in your eyes.check settings, using the Target settings builder, and in other similar places. Please refer to the documentation of the specific SDK you’re using for concrete examples.

Storybook Play Functions

A nice eyes-storybook-specific trick to achieve a desired rendering state would be a Storybook Play Function.

Applitools Eyes will run your play functions and wait for them to finish before capturing anything on the screen. Use Play Functions to navigate the story to an interesting state and wait for the story to be stable inside the play function to help Eyes understand what the best time is to capture the screenshot.

Applitools is Here to Help

We hope you’ve found this article interesting, and maybe it solved some of the most common visual testing issues you may have encountered. Go ahead and try these examples out for free with Applitools Eyes.

However, if something isn’t clear or if you’d like advice regarding the best way to incorporate visual testing into your organization, please don’t hesitate to reach out to our experts! Testing is our passion, and we’re here to help.

Quick Answers

The post Handling Animations and Loading Artifacts in Visual Testing appeared first on AI-Powered End-to-End Testing | Applitools.

Curious about the latest updates in Storybook.js, including the upcoming Storybook 7? In this post, which will be continuously updated, we sum up the latest Storybook news.

The highly anticipated Storybook 7.0 is currently in alpha, and there is a lot to get excited about. Let’s take a look at everything we know so far.

What’s Coming in Storybook 7

Storybook 7 promises significant changes. In fact, the Storybook team describes it as “a full rework of Storybook’s core with fast build and next-generation interaction testing.” Interaction testing was first included in the most recent Storybook 6.5 release, but we can expect further development there as Storybook 7 develops.

A Storybook 7 Design Preview

Storybook’s developers have just revealed a “sneak peek” at the design and layout in Storybook 7 [update 8/18: these changes are now available in the latest alpha]. Here’s a few of the changes that were highlighted:

• 3.5% more screen space for the Canvas (where components are developed in isolation).
• A “Reload” tool to reload a selected story (component) without refreshing the whole browser.
• Better access for integrators to the same design patterns used to develop Storybook.
• 196 icons that can be used in all projects, each redrawn from scratch. This is 20 more icons than the previous set.
• Form components like Toggle and Slider conform to the new design language.
• Pre-bundling of Storybook to deliver faster start times and avoid dependency issues (dependencies were also audited to reduce bundle size).

Storybook 7 Release Date

We don’t yet know the exact release date for Storybook 7, but we can guess based on their recent development history.

Storybook Version 6 was originally released in August 2020. Since then, there have been 5 major updates, most recently ending in version 6.5 which was released in May 2022. On the journey to the Storybook 6 release, the development team hit the following milestones:

• 47 alphas between January 2020 and April 2020
• 47 betas between April 2020 and July 2020
• 30 release candidates between July 2020 and August 2020

Storybook 7 began its first alpha in June 2022, and as of this writing is on alpha-26. If it follows the same trajectory is Storybook 6, we can estimate that it will enter beta in September, RC in December and official release in January 2023. Of course, time will tell, and we’ll update this post when any concrete information becomes available.

How Visual Testing Helps Component Testing

Component testing is form of software testing that focuses on software components in isolation. Component testing takes each rendered state (or Storybook story) and tests it.

Visual testing of components allows teams to find bugs earlier – and without writing any additional test code. It works across a variety of browsers and viewports at speeds almost as fast as unit testing.

You can learn more about how you can save time by using Applitools and our AI-powered visual testing with Storybook here:

• Visual Testing with Storybook
• Testing Storybook Components in Any Browser – Without Writing Any New Tests!
• How to Add Applitools Eyes to Storybook

Stay Tuned

We’ll be sure to keep this page updated with the latest on what’s new in Storybook 7, so check back often. And of course, we’re working hard to ensure our own Applitools SDKs for Storybook React, Storybook Angular and Storybook Vue are always compatible with the latest Storybook features.

Last Updated: August 26th, 2022

The post What’s New in Storybook 7? appeared first on AI-Powered End-to-End Testing | Applitools.

Summer is a time for new things and a time for play. We’re excited to announce that the Applitools Storybook SDK now supports Play Functions, giving modern frontend teams even more power when it comes to testing their component systems before production. Play Functions enable rich functionality in your Storybook designs, enabling you to interact with components and test scenarios that otherwise required user intervention. This means interacting and testing interactions such as form fills or date pickers in your component system is now possible! This capability was made available in Storybook version 6.4 and now is available in Applitools Storybook SDK version 3.28.

Applitools Storybook SDK can now consume these interactions through Play Functions and apply Visual AI to help your team spot any visual regression or defect in a component. For stories that have the play function, Applitools will automatically take a screenshot after the play function is finished.

To learn more about this specific feature, you can read our Storybook readme on NPM or the official Storybook Play Article.

Learn how to automatically do ultrafast cross-browser testing for Storybook components without needing to write any new test automation code in Testing Storybook Components in Any Browser by Andrew Knight.

Happy testing!

The post Storybook Play Functions Now Supported in Applitools appeared first on AI-Powered End-to-End Testing | Applitools.

We’re approaching the end of May, which means we’re just a handful of weeks the midpoint of 2022 already. If you’re like me, you’re wondering where the year has gone. Maybe it has to do with life in the northeastern US where I live, where we’ve really just had our first week of warm weather. Didn’t winter just end?

As always, the year is flying by, and it can be hard to keep up with all the great videos or events you might have wanted to watch or attend. To help you out, we’ve rounded up some of our most popular test automation videos of 2022 so far. These are all top-notch workshops or webinars with test automation experts sharing their knowledge and their stories – you’ll definitely want to check them out.

Cross Browser Test Automation

Cross-browser testing is a well-known challenge to test automation practitioners. Luckily, Andy Knight, AKA the Automation Panda, is here to walk you through a modern approach to getting it done. Whether you use Cypress, Playwright, or are testing Storybook components, we have something for you.

Modern Cross Browser Testing with Cypress

For more, see this blog post: How to Run Cross Browser Tests with Cypress on All Browsers (plus bonus post specifically covering the live Q&A from this workshop).

Modern Cross Browser Testing in JavaScript Using Playwright

For more, see this blog post: Running Lightning-Fast Cross-Browser Playwright Tests Against any Browser.

Modern Cross Browser Testing for Storybook Components

For more, see this blog post: Testing Storybook Components in Any Browser – Without Writing Any New Tests!

Test Automation with GitHub or Chrome DevTools

GitHub and Chrome DevTools are both incredibly popular with the developer and testing communities – odds are if you’re reading this you use one or both on a regular basis. We recently spoke with developer advocates Rizel Scarlett of GitHub and Jecelyn Yeen of Google as they explained how you can leverage these extremely popular tools to become a better tester and improve your own testing experience. Click through for more info about each video and get watching.

Make Testing Easy with GitHub

For more, see this blog post: Using GitHub Copilot to Automate Tests.

Automating Tests with Chrome DevTools Recorder

For more, see this blog post: Creating Your First Test With Google Chrome DevTools Recorder.

Test Automation Stories from Our Customers

When it comes to implementing and innovating around test automation, you’re never alone, even though it doesn’t always feel that way. Countless others are struggling with the same challenges that you are and coming up with solutions. Sometimes all it takes is hearing how someone else solved a similar problem to spark an idea or gain a better understanding of how to solve your own.

Accelerating Visual Testing

Nina Westenbrink, Software Engineer at a leading European telecom, talks about how the visual time to test the company’s design system was decreased and simplified, offering helpful tips and tricks along the way. Nina also speaks about her career as a woman in testing and how to empower women and overcome biases in software engineering.

Continuously Testing UX for Enterprise Platforms

Govind Ramachandran, Head of Testing and Quality Assurance for Asia Technology Services at Manulife Asia, discusses challenges around UI/UX testing for enterprise-wide digital programs. Check out his blueprint for continuous testing of the customer experience using Figma and Applitools.

This is just a taste of our favorite videos that we’ve shared with the community from 2022. What were yours? You can check out our full video library here, and let us know your own favorites @Applitools.

The post Our Best Test Automation Videos of 2022 (So Far) appeared first on AI-Powered End-to-End Testing | Applitools.

Learn how to automatically do ultrafast cross-browser testing for Storybook components without needing to write any new test automation code.

Let’s face it: modern web apps are complex. If a team wants to provide a seamless user experience on a deadline, they need to squeeze the most out of the development resources they have. Component libraries help tremendously. Developers can build individual components for small things like buttons and big things like headers to be used anywhere in the frontend with a consistent look and feel.

Storybook is one of the most popular tools for building web components. It works with all the popular frameworks, like React, Angular, and Vue. With Storybook, you can view tweaks to components as you develop their “stories.” It’s awesome! However, manually inspecting components only works small-scale when you, as the developer, are actively working on any given component. How can a team test their Storybook components at scale? And how does that fit into a broader web app testing strategy?

What if I told you that you could automatically do cross-browser testing for Storybook components without needing to define any new tests or write any new automation code? And what if I told you that it could fit seamlessly into your existing development workflow? You can do this with the power of Applitools and your favorite CI tool! Let’s see how.

Adding Visual Component Testing to Your Strategy

Historically, web app testing strategies divide functional testing into three main levels:

1. Unit testing
2. Integration testing for APIs
3. End-to-end testing for UIs and APIs

These three levels make up the classic Testing Pyramid. Each level of testing mitigates a unique type of risk. Unit tests pinpoint problems in code, integration tests catch problems where entities meet, and end-to-end tests exercise behaviors like a user.

The rise of frontend component libraries raises an interesting question: Where do components belong among these levels? Components are essentially units of the UI. In that sense, they should be tested individually as “UI units” to catch problems before they become widespread across multiple app views. One buggy component could unexpectedly break several pages. However, to test them properly, they should be rendered in a browser as if they were “live.” They might even call APIs indirectly. Thus, arguably, component testing should be sandwiched between traditional integration and end-to-end testing.

Wait, another level of testing? Nobody has time for that! It’s hard enough to test adequate coverage at the three other levels, let alone automate those tests. Believe me, I understand the frustration. Unfortunately, component libraries bring new risks that ought to be mitigated.

Thankfully, Applitools provides a way to visually test all the components in a Storybook library with the Applitools Eyes SDK for Storybook. All you need to do is install the @applitools/eyes-storybook package into your web app project, configure a few settings, and run a short command to launch the tests. Applitools Eyes will turn each story for each component into a visual test case. On the first run, it will capture a visual snapshot for each story as a “baseline” image. Then, subsequent runs will capture “checkpoint” snapshots and use Visual AI to detect any changes. You don’t need to write any new test code – tests become a side effect of creating new components and stories!

In this sense, visual component testing with Applitools is like autonomous testing. Test generation and execution is completely automated, and humans review the results. Since testing can be done autonomously, component testing is easy to add to an existing testing strategy. It mitigates lots of risk for low effort. Since it covers components very well, it can also reduce the number of tests at other layers. Remember, the goal of a testing strategy is not to cover all the things but rather to use available resources to mitigate as much risk as possible. Covering a whole component library with an autonomous test run frees up folks to focus on other areas.

Adding Applitools Eyes to Your Web App

Let’s walk through how to set up visual component tests for a Storybook library. You can follow the steps below to add visual component tests to any web app that has a Storybook library. Give it a try on one of your own apps, or use my example React app that I’ll use as an example below. You’ll also need Node.js installed as a prerequisite.

To get started, you’ll need an Applitools account to run visual tests. If you don’t already have an Applitools account, you can register for free using your email or GitHub account. That will let you run visual tests with basic features.

Once you get your account, store your API key as an environment variable. On macOS or Linux, use this command:

export APPLITOOLS_API_KEY=<your-api-key>

On Windows:

set APPLITOOLS_API_KEY=<your-api-key>

Next, you need to add the eyes-storybook package to your project. To install this package into a new project, run:

npm install --save-dev @applitools/eyes-storybook

Finally, you’ll need to add a little configuration for the visual tests. Add a file named applitools.config.js to the project’s root directory, and add the following contents:

module.exports = {
    concurrency: 1,
    batchName: "Visually Testing Storybook Components"
}

The concurrency setting defines how many visual snapshot comparisons the Applitools Ultrafast Test Cloud will perform in parallel. (With a free account, you are limited to 1.) The batchName setting defines a name for the batch of tests that will appear in the Applitools dashboard. You can learn about these settings and more under Advanced Configuration in the docs.

That’s it! Now, we’re ready to run some tests. Launch them with this command:

npx eyes-storybook

Note: If your components use static assets like image files, then you will need to append the -s option with the path to the directory for static files. In my example React app, this would be -s public.

The command line will print progress as it tests each story. Once testing is done, you can see all the results in the Applitools dashboard:

Run the tests a second time for checkpoint comparisons:

If you change any of your components, then tests should identify the changes and report them as “Unresolved.” You can then visually compare differences side-by-side in the Applitools dashboard. Applitools Eyes will highlight the differences for you. Below is the result when I changed a button’s color in my React app:

You can give the changes a thumbs-up if they are “right” or a thumbs-down if they are due to a regression. Applitools makes it easy to pinpoint changes. It also provides auto-maintenance features to minimize the number of times you need to accept or reject changes.

Adding Cross-Browser Tests for All Components

When Applitools performs visual testing, it captures snapshots from tests running on your local machine, but it does everything else in the Ultrafast Test Cloud. It rerenders those snapshots – which contain everything on the page – against different browser configurations and uses Visual AI to detect any changes relative to baselines.

If no browsers are specified for Storybook components, Applitools will run visual component tests against Google Chrome running on Linux. However, you can explicitly tell Applitools to run your tests against any browser or mobile device.

You might not think you need to do cross-browser testing for components at first. They’re just small “UI units,” right? Well, however big or small, different browsers render components differently. For example, a button may have rectangular edges instead of round ones. Bigger components are more susceptible to cross-browser inconsistencies. Think about a navbar with responsive rendering based on viewport size. Cross-browser testing is just as applicable for components as it is for full pages.

Configuring cross-browser testing for Storybook components is easy. All you need to do is add a list of browser configs to your applitools.config.js file like this:

module.exports = {
  concurrency: 1,
  batchName: "Visually Testing Storybook Components",
  browser: [
    // Desktop
    {width: 800, height: 600, name: 'chrome'},
    {width: 700, height: 500, name: 'firefox'},
    {width: 1600, height: 1200, name: 'ie11'},
    {width: 1024, height: 768, name: 'edgechromium'},
    {width: 800, height: 600, name: 'safari'},
    // Mobile
    {deviceName: 'iPhone X', screenOrientation: 'portrait'},
    {deviceName: 'Pixel 2', screenOrientation: 'portrait'},
    {deviceName: 'Galaxy S5', screenOrientation: 'portrait'},
    {deviceName: 'Nexus 10', screenOrientation: 'portrait'},
    {deviceName: 'iPad Pro', screenOrientation: 'landscape'},
  ]
}

This declaration includes ten unique browser configurations: five desktop browsers with different viewport sizes, and five mobile devices with both portrait and landscape orientations. Every story will run against every specified browser. If you run the test suite again, there will be ten times as many results!

As shown above, my batch included 90 unique test instances. Even though that’s a high number of tests, Applitools Ultrafast Test Cloud ran them in only 32 seconds! That really is ultrafast for UI tests.

Running Visual Component Tests Autonomously

Applitools Eyes makes it easy to run visual component tests, but to become truly autonomous, these tests should be triggered automatically as part of regular development workflows. Any time someone makes a change to these components, tests should run, and the team should receive feedback.

We can configure Continuous Integration (CI) tools like Jenkins, CircleCI, and others for this purpose. Personally, I like to use GitHub Actions because they work right within your GitHub repository. Here’s a GitHub Action I created to run visual component tests against my example app every time a change is pushed or a pull request is opened for the main branch:

name: Run Visual Component Tests

on:
  push:
  pull_request:
    branches:
      - main

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v2
      
      - name: Set up Node.js
        uses: actions/setup-node@v2

      - name: Install dependencies
        run: npm install

      - name: Run visual component tests
        run: npx eyes-storybook -s public
        env:
          APPLITOOLS_API_KEY: ${{ secrets.APPLITOOLS_API_KEY }}

The only extra configuration needed was to add my Applitools API key as a repository secret. 

Maximizing Your Testing Value

Components are just one layer of complex modern web apps. A robust testing strategy should include adequate testing at all levels. Thankfully, visual testing with Applitools can take care of the component layer with minimal effort. Unit tests can cover how the code works, such as a component’s play method. Integration tests can cover API requests, and end-to-end tests can cover user-centric behaviors. Tests at all these levels together provide great protection for your app. Don’t neglect any one of them!

The post Testing Storybook Components in Any Browser – Without Writing Any New Tests! appeared first on AI-Powered End-to-End Testing | Applitools.

Component libraries and design systems are important development and design tools that allow teams to focus on building consistent and high quality experiences. Storybook is a tool that helps us with those experiences, but as your library and system grows, it becomes more difficult to maintain that level of high quality.

With the Applitools Eyes Storybook SDK, we’re able to provide visual testing coverage for all stories in your Storybook. The cool thing is this works for whatever UI framework you choose! Whether you’re using Storybook with React, Vue, Angular, Svelte, or even React Native, Applitools is able to support any framework Storybook supports.

Let’s dig in and find out what exactly Storybook is, where it helps, and how Applitools can easily provide coverage for any number of stories that a Storybook library supports.

What is Storybook?

Storybook is a JavaScript-based tool that helps teams build component libraries, design systems, and beyond.

It does this by providing a library-like interface, that shows different components and pages in isolation allowing developers, designers, testers, and users of the library to consume and work in each “story” in a focused environment.

Each Storybook story is composed of a component or set of components that represent one piece of what someone would interact with or visualize in an application.

Testing and validating UI at the component level

While most stories don’t show an entire page built with these components, each of them are fit together in various ways to build the actual interfaces that are used by visitors of that app.

This gives the development team or users of the library an easy and flexible way to make sure each component works as expected from the start, where they can then have confidence to use it however they’d like in the application.

But the tricky part is making sure we’re providing proper coverage of each of the components in a way that will help us build that confidence along with finding a scalable solution as the library grows.

Solutions exist to provide code-based testing, where we might validate the string-based output of a component or test the components virtually mounted, but they’re not actually capturing what the users of the components are actually seeing.

Visual testing helps us solve that. We can capture what the user is seeing right in the browser, and use that to make sure our components and stories are working exactly like they should, giving us that needed confidence.

AI-Powered Visual Testing and Storybook

With the Applitools Eyes Storybook SDK, we’re able to easily provide coverage for any number of stories that our Storybook library supports.

After installing @applitools/eyes-storybook, all you need to do is simply run the following command:

npx eyes-storybook

Once run, Applitools will find all stories and start to capture each story much like other SDKs available for Eyes.

Because Storybook is ultimately web-based, where it renders any UI framework it supports right in a browser, the Eyes SDK is able to capture a DOM snapshot of that rendered output.

That means, the Eyes Storybook SDK can support any framework that Storybook can support!

The SDK uses Puppeteer, similar to the Eyes Puppeteer SDK, where each story is loaded in a new Puppeteer tab.

It then collects a DOM snapshot which is used to render each page cross-browser in the Applitools cloud, where you’ll then get the same AI-powered Visual Testing coverage that you would with any other Applitools integration.

This helps so that we can visually identify the issues with our application, based on what people are actually seeing. Saving us time when trying to solve where the issue occurred or if it’s a false positive in the first place.

Getting started with Applitools Eyes and Storybook

If you want to get started with integrating Applitools Eyes with your Storybook library, we have a variety of resources including a walkthrough for adding Applitools Eyes to Storybook along with dedicated tutorials for React, React using CSF, Vue, and Angular.

You can also learn more about the importance of design systems with my free webinar Bringing Quality Design Systems to Life with Storybook & Applitools.

The post How to Maintain High Quality Design Systems with Storybook by Leveraging Visual AI appeared first on AI-Powered End-to-End Testing | Applitools.

Creating component and design systems are an important way to make sure you’re delivering consistent experiences to the people visiting your website or application. Storybook makes this easy with its JavaScript API, but how can we make sure we’re tracking the quality of those stories with automated testing?

Just looking for an example? Check out the available Applitools tutorials for React, Vue, and Angular on GitHub.com.

What is Storybook?

Storybook is a JavaScript-powered web tool that gives teams the ability to build component and design systems for their websites and applications.

By adding Storybook to your project, you can develop individual components of your UI in a focused, self-documenting environment, that helps everyone from design to development to product release carefully craft an experience.

It supports a variety of the development community’s most popular frameworks including React, Vue, and Angular. It also has a few documented ways to add testing to those components, but that typically involves meticulously writing tests that could be fragile and not necessarily representing what people are using in their applications.

That’s where Applitools and automated visual testing come in!

How does Storybook work with Applitools?

Pretty much out of the box, Applitools will give you full automated visual testing coverage of all of the components of your Storybook library.

By installing the Applitools Eyes Storybook SDK and running a simple command, Applitools will automatically pull in all of those stories, take a snapshot, and check for regressions in the Applitools cloud.

What are we going to build?

We’re going to start off with an existing React project that already has Storybook installed.

For more information about installing Storybook, visit the Storybook docs at: https://storybook.js.org/docs/react/get-started/install

In this walkthrough, we’ll be using a Storybook React application, but both Storybook and Applitools support more frameworks like Vue and Angular!

But with our project ready to go, we’ll first install the Applitools Eyes Storybook SDK using npm.

Once installed, we’ll show an example of how you can use Applitools Eyes to add Visual Testing coverage to your Storybook.

Step 1: Installing the Applitools Eyes Storybook SDK

To get started with the Applitools Eyes Storybook SDK on a project, first install the SDK from npm:

npm install @applitools/eyes-storybook --save-dev

This will pull down the SDK from npm and add it as a package to your project.

Step 2: Running Applitools Eyes for automated Storybook visual testing

Now that our SDK is installed, we actually don’t need to do anything else to get set up, we can dive right into running our tests!

To run our tests, make sure you first head over to the applitools.com dashboard and find your API key, then run the following in your terminal:

APPLITOOLS_API_KEY="[Your Key]" npx eyes-storybook

After that command finishes, you can see that by default, Applitools will go through each story, capture a snapshot, and check for visual differences.

Because this was our first time running our tests, all of our snapshots are new.

If we head over into our Applitools dashboard, we can even see these tests!

We can see that inside Applitools, all of our tests are new and green, which is a good thing. These will serve as the baseline for all of our tests.

Now to see the magic of how Applitools works, we can try breaking one of our components.

Inside Header.js,on line 30, let’s change our user condition to:

{!user ? (

We’re essentially switching around the logic, where if this was happening on production, we would want this bug caught!

Let’s run our tests again:

APPLITOOLS_API_KEY="[Your Key]" npx eyes-storybook

We can see this time, we have differences in our tests.

If we go into the Applitools dashboard, not only can we see that we now have Unresolved tests, we can open up one of those tests, and see the difference.

Applitools Eyes was able to see exactly where that issue was and highlight the change, so we can then see that there is a bug in our application!

Inside of the dashboard, we can click the “thumbs down” button, which tells Applitools we reject that change, meaning it failed.

But once we revert our application change and run our tests again, we can see that our tests are now passing again, because they’re working exactly how they should!

What else can you do with Applitools and Storybook?

Ignoring regions

One of the powerful features of Applitools is the ability to ignore specific regions of your visual tests by passing in selectors.

By using the ignoreRegions property inside of your Story’s eyes configuration, you can specify any selector you’d like to ignore in that Story.

Learn more about ignoring regions on npmjs.com

Specifying region types

In addition to simply ignoring regions, you can also set a custom match level for specific regions, such as if you wanted to define a region to include dynamic content, so that Applitools can use AI to understand if the content changed or if the layout changed.

Learn more about configuring regions for content, layout, and more on npmjs.com.

Get Started Today

Want to try out Applitools Eyes with Storybook for yourself? You can get started today at the link below – the account is free forever.

The post How to Add Visual Testingto Storybook appeared first on AI-Powered End-to-End Testing | Applitools.

Recently I’ve put together a bunch of tutorials: an overview of Applitools, how to test Storybook with Angular, how to quickly troubleshoot bugs in React apps, and how to test Vue apps with Cypress.

If you’ve gone through these, you’ll notice that the visual tests complete in seconds — faster than Nic Cage can jack your ride.

How so fast?

The answer is simple: Applitools Ultrafast Grid. Each time you run a visual test in Applitools, Ultrafast Grid renders that page across a broad range of browsers and configurations. In parallel. In seconds.

Applitools Visual AI then grabs a screenshot of each page, compares it to its baseline screenshot, and determines visual differences and the root cause unpinning them. In less than a minute, your cross-browser testing is done and your developers have what they need to fix any visual bugs.

It’s our very own ludicrous mode (a.k.a. how a Tesla can go from zero to 60 mph in a little over two seconds).

But more than a cool technical trick, Ultrafast Grid can help you keep your app development projects on schedule.

In this article, I will demonstrate the Ultrafast Grid system with test runs and a variety of configuration settings, such as multiple browsers, viewports, and devices, to show Applitools’ efficiency in delivering results.

First off, make sure you have the following installed and configured on your machine:

• Angular
• Storybook
• Applitools

Before we delve into writing code, let’s take a behind-the-scenes look at how Applitools Ultrafast Grid works.

Life before Ultrafast Grid

Before the arrival of the Ultrafast Grid, visual tests were run in this specific order:

• A test run checked the pages one by one.
• A screenshot was taken for each page.
• The screenshot was uploaded to the backend server, where it was analyzed (that is, compared to previous baselines).
• Based on the results, the test would either fail or keep running, checking other pages.

So far, so good!

But problems arise when hundreds of screenshots are uploaded to the backend server to be analyzed. Varying parameters ranging from different viewports, different browsers, and different device emulators can also slow things down.

To help solve this, Applitools designed the Ultrafast Grid system to run tests quickly.

Life with Ultrafast Grid

Applitools Ultrafast Grid is split into two components: server-side and client-side. Here’s a diagram of how it works:

The Ultrafast Grid client is a Node JS library, used internally by both Applitools Storybook and Cypress SDKs. Here’s more on the Ultrafast Grid client.

That code snippet instantiates a new Ultrafast Grid client and returns an object. In this case, we are only interested in the openEyes() method returned.

When called, the openEyes() method starts a new test run and sends the Ultrafast Grid backend server details about it– including, but not limited to the application name, batch or test name, the browsers used, and different variations of browser settings to be adapted while running the tests. This method returns a Promise to an object with the following functions: checkWindow() and close().

This method call instructs the Ultrafast Grid backend server to prepare a testing environment for a new test run that requires running the test over two different browsers. The first browser emulates an iPhone X, while the second specifies the desktop viewport to run the test.

To start collecting snapshots, the test issues a call to checkWindow() function. The Ultrafast Grid client handles this call and collects all the needed resources to store the snapshot locally.

At the end of the test, you issue a call to close() method to close and finish the test run.

At this moment, all the DOM Snapshots collected locally during the test run are sent to the Ultrafast Grid backend server. Based on the test run requirements, sent previously with the openEyes() method, the Ultrafast Grid:

• Launches a number of browsers with different viewports and device emulators.
• Runs all of the DOM snapshots in parallel.
• Collects image screenshots.
• Sends all the screenshots to the Applitools AI server to do all the testing analysis and returns the results.

Finally, it’s worth mentioning that Ultrafast Grid sits behind a family of Applitools SDKs including the following:

• Applitools for Selenium IDE

• Java 3 Selenium SDK
• JS4 Selenium SDK
• C# Selenium SDK
• Python Selenium SDK
• Ruby Selenium SDK
• WebdriverIO SDK
• Cypress
• Storybook

Now that you understand how the Ultrafast Grid system works so efficiently, let’s throw a spanner in the works by running tests with a variety of browser configuration settings. Then, we’ll review the results on the Applitools Test Manager.

Source code

For this article, I’ve chosen to write a few Storybook stories and use Applitools for the custom Angular ContentEditable component. I’ve described and used this in my previous article on Mixing Storybook with Angular with a sprinkle of Applitools.

There have been some changes to the Applitools Eyes SDK for Storybook npm package since my previous article was published. I’d recommend removing the old package and installing the new one by following these steps:

Step 1: Remove the old @applitools/eyes.storybook NPM package by issuing the following command:

npm remove @applitools/eyes.storybook

copy code to clipboard

Step 2: Install the new @applitools/eyes-storybook NPM package by issuing the following command:

npm i @applitools/eyes-storybook

copy code to clipboard

That’s it!

Applitools Ultrafast Grid Demo

Before writing our tests, let’s run the application and make sure it’s working fine.

Step 1: Clone the application repo.

git clone https://github.com/thisdot/storybook-angular-applitools.git

copy code to clipboard

Step 2: Install all app dependencies by issuing the following command:

Go to the folder /storybook-angular-applitools then run this command:

npm install

copy code to clipboard

Step 3: Run the app by issuing the following command:

ng serve

copy code to clipboard

Now that we are sure the application is up and running switch back to the Terminal and hit Ctrl+c twice. This terminates the running application.

Step 4: Run the Storybook stories by issuing the following command:

npm run storybook

copy code to clipboard

Step 5: Run the Applitools Eyes SDK for Storybook by issuing the following command:

npx eyes-storybook

copy code to clipboard

Make sure you get an Applitool API Key and store it on your machine.

To set the APPLITOOLS_API_KEY environment variable locally on your machine you can use the export command on Mac OS or set command on Windows as follows.

For a complete tutorial on how to install and run Applitools Storybook SDK for Angular, check out this article: Storybook Angular Tutorial.

Back to the command above, this command simply opens the Storybook stories, runs them one by one, and then sends all the DOM snapshots to Applitools Ultrafast Grid, which then renders them.

If you expand the first test run, you get the following results:

Now that the app, Storybook stories, and Applitools are running locally, let’s start playing with our visual tests.

Visual UI tests with custom viewports

Keeping in mind that Applitools with Storybook doesn’t need any kind of configuration, the important test configurations are auto-inferred by Applitools. However, if you want to play around with the test configuration, you have two options to specify:

• Environment variables
• The applitools.config.js file

For this demonstration, I will make use of the applitools.config.js file. For a full list of test configuration, check the Advanced Configuration section on the @applitools/eyes-storybook repo on Github.

When you’re using Applitools with Storybook, there’s no need to add any specific Applitools commands or set test configuration to enable visual testing.

Applitools Eyes SDK runs the Storybook stories, takes a snapshot of each, and sends the DOM snapshots to the Ultrafast Grid backend server. It saves the test run with the default test configuration.

Step 1: Add a new JavaScript file to the root of your application. Give it a name of applitools.config.js.

Step 2: Add the appName and batchName configuration settings as follows:

Step 3: Run the test case on two different browsers with two different viewports by adding the following:

Consequently, Ultrafast Grid launches a set of instances for each of the browsers and runs the tests simultaneously. The first time these tests are run, we create baseline images to compare against during subsequent test runs.

Step 4: Save and run the tests again by issuing the following command:

npx eyes-storybook

copy code to clipboard

Step 5: Check the test run in Applitools Test Manager.

I’ve switched the Test Manager view to the Batch Summary View.

Notice how Applitools Ultrafast Grid ran your tests twice, as per the test configuration file above. The test configuration above specified two different browsers with two different viewports. This means that the Ultrafast Grid runs every test twice– once per browser configuration set.

Running this test yields an execution time of just four seconds! To be clear, this is what we spent rendering baseline images, once the DOM snapshots were uploaded. Even so, that’s really fast.

It’s the duty of the Ultrafast Grid to analyze the test configuration file attached to the test and spawn, in parallel, a number of browser instances to handle the test run efficiently.

Visual UI tests with device emulation

Let’s run another test. This time, instead of specifying the browser viewports, we will specify a device emulation for the browser. Device emulation uses a Chrome browser, with the viewport dimensions, and user agent string, specific to that device.

Step 1: Replace the content of the applitools.config.js file with the following configuration:

In this test configuration, we are specifying to the Ultrafast Grid to run this test over a Chrome browser with a device emulation of an iPhone X using a landscape screen orientation.

Ultrafast Grid supports only the Chrome browser for running tests over a device emulation, so you can skip the name of the browser. I’m explicitly specifying the browser name to make it very clear that this is a browser emulation and not a real device.

For a complete list of device emulators supported by the Chrome browser, take a look at this page: Chromium Device Emulators.

Step 2: Save and run the tests by issuing the following command:

npx eyes-storybook

copy code to clipboard

Step 3: Check the test run in Applitools Test Manager.

I’ve switched the Test Manager view to the Batch Summary View.

As per the new test configuration file, the Ultrafast Grid runs your tests on a Chrome browser with a device emulation for iPhone X. Hence, the browser name shown is Safari 11.0 and landscape with a viewport of 812×375.

Notice that running this test yields an execution time of just two seconds! Again, this is once the DOM snapshot has been uploaded. But still…

Finally, in Applitools, baseline snapshots are stored as per test configuration settings. Different test configuration settings generate different baseline snapshots. You can’t compare apples to oranges, right?

That’s why, if you click on the snapshot above (after switching back to Batch steps view) and show both the baseline and the current test run, you won’t get any baseline snapshot–because it’s the first time we’re running the test with these configuration settings.

100 Visual UI tests

Let’s go crazy and run 100 Visual/UI tests with a combination of test configurations.

Step 1: Replace the content of the applitools.config.js file with the following configuration:

In this test configuration, we are running 100 tests with 50 unique view ports on two major browsers: Firefox and Google Chrome.

Step 2: Save and run the tests by issuing the following command:

npx eyes-storybook

copy code to clipboard

During this time, the Eyes Storybook SDK will read all Storybook stories in the application, run them one by one using the Storybook engine, generate a DOM snapshot for each test, and finally upload the results to the Eyes server. At this moment, the Eyes server will run each and every DOM snapshot, render an image snapshot, and store it as a baseline snapshot.

The Eyes SDK/Server took only 21 seconds to complete all the above tasks, create baselines, and report the results on Applitools Test Manager Dashboard, for the entire batch of 100 test runs for a single Storybook story.

Step 3: Check the test run in Applitools Test Manager.

I’ve switched the Test Manager view to the Batch Summary View.

Let’s see the power of the Eyes server when running a regression testing cycle.

Step 4: Let’s introduce a visual change on the Storybook story and run the Eyes SDK once again.

Replace the content of the /src/stories/index.stories.js file with the following:

view rawregression-testing-cycle hosted with by GitHub

Now the ContentEditableComponent should render with a Blue background.

Step 5: Run the tests again by issuing the following command:

npx eyes-storybook

copy code to clipboard

Once again, the Eyes SDK/Server took only 21 seconds to upload the DOM snapshot, run the entire batch of visual UI tests, and display the results on the Applitools Test Manager Dashboard.

Notice the error message generated saying that “A total of 100 differences were found”.

Because the change we introduced affected the entire batch of test runs, therefore, all the current tests failed in comparison with the baseline snapshots, due to the visual change introduced earlier.

Step 6: Check the test run in Applitools Test Manager.

The Test Manager highlights the differences between the baseline snapshot and the current test run and displays a mismatch icon near the snapshots.

At this moment, you can either accept the new changes and make them the new baseline or reject them and keep the old baseline.

Once again, it takes the power of the Ultrafast Grid to analyze the test configuration file and set up the testing environment as needed.

See for yourself

Here’s a step-by-step tutorial video of how I did all the steps above. You can see how it took just 20 seconds for the Applitools Storybook SDK to upload all the DOM snapshots, then use Applitools Visual AI to look for visual regressions:

Conclusion

By now, you have some insight into how Applitools Ultrafast Grid functions behind the scenes when running your visual tests.

Want to get run your own tests ludicrously fast? Applitools Ultrafast grid is GA for Storybook, Cypress, and Selenium IDE. If you use Selenium WebDriver with Java, JavaScript, C#, Python, or Ruby, request to join our early access program.

Happy Testing!

Photo by Sander Jeurissen on Unsplash

The post How I ran 100 UI tests in just 20 seconds [Revisited] appeared first on AI-Powered End-to-End Testing | Applitools.

How do you test a design system? You got here because you either have a design system or know you need one. But, the key is knowing how to test its behavior.

Marie Drake, Principal Test Automation Engineer at News UK, presented her webinar, “Roadmap To Testing A Design System”, where she discussed this topic in some detail.

Marie is many things. In addition to her work at News UK, she is a Cypress Ambassador and organizer of the Cypress UK community group. If you want to know more about using Cypress, she’s a great speaker. In addition, she blogs about testing and tech at her own blog, mariedrake.com.

This post summarizes her webinar and highlights some of the key points.

Who Is News UK?

News UK is the UK subsidiary of News Corp, the large global publishing and media company. Marie’s team supports the sites that develop and deliver online versions of The Sun, The Times, and The Sunday Times. They also run the Wireless media site. Marie supports the various development teams that deliver news and information that change regularly.

Why A Design System At News UK?

Think of a design system as building blocks. A design system provides a repository for design components used to construct your web application. Or, more precisely, applications. By using a design system, you can eliminate redundant work across different parts of your web application.

Marie gave the example of “share bars” at News UK. Share bars let you share content to social networks like Twitter, Facebook, Instagram, and WhatsApp. You likely have seen share bars on blogs or media pages. Inside News UK, design groups had coded their own share bars. They found19 different share bars in use across different parts of the News UK business.

The implication of lots of redundant code written by different people involves the cost of maintenance. Sure, having 19 different teams write 19 different parts of the app sounds like great division of labor. But, when you get 19 different sound bars – how do you maintain them? How do you choose one for your next part of your web business? What happens when you decide to resize the share bars across your site?

Fast Coding Does Not Equal Agility

Marie showed an even more problematic example of a web business, Fabulous, that wanted to change their brand color from #E665BF to #EA6596. When engineers looked at the potentially impacted code and the areas requiring post-change validation, they estimated the change would take six months. Half a year for a color change?

The coding effort at Fabulous involved two code bases. First, they had the website that needed to be updated. Second, they had the tests that needed to be updated to match the new site. A large part of the test change – even with no functional or other visual change – just required code inspection to ensure that the desired branding color change had been applied as expected across the entire site.

Marie’s seven-plus years in software quality led her to understand that raw coding speed rarely correlated with agility. Here, “agility” means something different from “agile.” In software, agility comes from the ability to make quick changes and have confidence on both their impact on intended behavior and avoiding unexpected changes. While many software developers can write code quickly, few write thoughtfully in ways that make code maintainable – especially across the entire site.

Benefits of the News UK Design System

In describing the design system deployed by News UK, Marie quickly pointed out its benefits.

• Cost efficient. Once you set up a design system, you have standardized building blocks for building your site. If you can use the design system to customize, your teams can consume the building blocks instead of rewriting from scratch. And, you reduce software maintenance costs.
• Reusable. A good design system allows you to re-use code.
• Speed To Market. As mentioned in the section on agility, the design system reduces the amount of code you need to write from scratch. It also reduces the amount of code you will manually change as you make updates.
• Scalable. A good design system lets multiple users access the system – making the developers much more efficient.
• Standard Way Of Working. With a design system, you standardize the process of writing code. You can help new people get up to speed on existing code and simplify the code maintenance process.
• Consistency. In the end, you can look to the design system to ensure consistent behavior (visual, functional) from your applications.

Marie showed a loop of the design system at News UK. The components get developed and maintained in Storybook. Developers can grab elements and add them into applications being built. The playground feature in Storybook makes it easy for developers and designers to play with Storybook components to mock up the functioning web application before it gets built.

As Marie pointed out, consistency in the components simplifies both development and testing.

Testing A Design System – Requirements

If a design system should make code easier to create and maintain, how do you test a design system?

Marie started outlining the testing requirements developed by News UK.

• Test different components easily. Expect the system to mature and develop over time. Some components will be entirely visual, and some may include audio. Make sure all this works.
• Test cross-browser. News UK needed this capability as they knew their content got consumed on mobile devices and a range of platforms.
• Visual Tests – Write visual tests with less maintenance in mind to reduce impact on testing workflow and speed the process of testing small changes that touch lots of components.
• Deliver a high-performing build pipeline – build plus test concludes within 10 minutes
• Integrate design review earlier in the process to improve collaboration, find misunderstandings and differences between design and development early in the process.
• Test for accessibility on both the component and site level for all users.
• Catch functional issues early.
• Have all tests written before deploying a feature. There are 2 full-time QA engineers on the Product Platforms team, so they need to share QA responsibility with developers.

Testing A Design System – Strategy

From here, Marie outlined the strategy to run tests of the design system.

• First, unit testing. Developers must write unit tests for each component and component modification.
• Second, snapshot testing. Capture snapshots and validate the status of component groups.
• Third, component testing. All components need to be validated for functionality, visual behavior, and accessibility.
• Fourth, front-end testing.  Make sure the site behaves correctly with updated components. Validate for functionality, visual behavior, and accessibility.
• Fifth – cross-browser tests. Ensure there are no unexpected differences on customer platforms.

Testing A Design System – Challenges

Marie described some of the challenges with different test approaches.

Purely functional tests can include lots of code. Marie’s pseudocode shows this problem. The more comprehensive your functional tests, the more code that exists in those tests. Assertion code – the code used to inspect the DOM for visual elements – becomes a burden for your team going forward.

Visual testing serves a strategic function, except that most visual testing approaches suffer from inaccuracy. Marie showed an example of a “spot-the-differences” game, which highlighted the challenges of a manual visual test. Then, she showed pixel differences, which she found become problematic on cross-browser tests. From a user’s perspective, the pages looked fine. The pixel differences highlighted differences that, after time-consuming inspection, her team judged as inconsequential pixel variations.

Another visual testing inaccuracy Marie described involved visual testing of dynamic data. On news sites, content changes frequently as news stories get updated. When the data changes, does the visual test fail?

Marie and her team had chosen to use available open-source tools for visual testing. Marie showed some of the visual errors that got through her testing system. These had passed functional tests but weren’t caught visually.  

So, Marie and her team discovered that their existing tests let visual bugs through. They knew they needed to solve their visual testing problem.

Choosing New Tools

Marie’s team looked at three potential solutions to their visual testing problem: Screener, Applitools and Happo.  After putting all three through their paces, the team settled on Applitools for accuracy. Being way more accurate helped Marie write up the use case for News UK to purchase a commercial tool instead of adopting an open-source solution.

The team also looked at UI testing tools. They looked at Puppeteer, Selenium, and Cypress for driving web application behavior. As a team, they chose Cypress. They could have used any of these tools with Applitools. Marie’s team chose Cypress because its developer-friendly user experience made it easy for developers to write and maintain tests.  

The final test suite included:

• Jest for managing test flow
• Cypress for running tests
• Applitools for visual tests
• Google Lighthouse and Cypress AXE for accessibility testing
• WAVE for testing Doc accessibility
• Safari Voiceover for sightless test validation

Using Applitools

Next, Marie shared the approach her team used for deploying Applitools.

Prior to using any part of Applitools, the team needed to deploy an API key. This key, found on the Applitools console, permits users the access the Applitools API. Once read into the test environment, the key grants the tests access to the Applitools service.

The team needed to add the Eyes code to Storybook for component tests and to Cypress for the site-level tests.

Component Tests

Next, Marie demonstrated the code for validating the Storybook components. The tests involved cycling through Storybook and having each component captured by Applitools. Individual component tests either matches in Applitools, or showed differences. The test team would walk through the inspected differences to either approve the changes and update the baseline image with the new capture, or rejected the change and send the component back to the developers.

Cypress Tests

Similar to the component tests, the Cypress tests integrated Applitools into captures of the built site using the new components. Again, Applitols compared each capture against the existing baseline to find differences.

For Marie’s team, one great part about using Applitools involved the built-in cross-browser testing using the Applitools Ultrafast Grid. Simply by specifying a file of target browsers and viewport sizes, Applitools could automatically capture images for the targets separately and compare each against its baseline.

Auto Maintenance

Marie talked about one of the great features in Applitools – Auto Maintenance. When Applitools discovers a visual difference, it looks for similar differences on other pages captured during a test run. When an Applitools user finds a visual difference and approves it, Auto Maintenance lists the other captures for which the identical difference exists. The Applitools user can then batch-approve the identical changes found elsewhere. A single user, in a single step, can approve site-wide menu, logo, color, and other common changes all at once.

Handling Dynamic Changes

Another benefit of Applitools involves pages with dynamic data. In addition to the example of news items updating regularly, Marie showed an example of the new Internet radio service offered by News UK. The player page can sometimes show different progress in a progress bar during different captures, depending on data being read when taking a screen capture.

Applitools has a layout mode that ensures that all the items exist in a layout, including the relative location of the items, but layout mode ignores content changes within the layout.

Accessibility Tests

Next, Marie talked about accessibility tests.

Marie demonstrated component accessibility testing with Cypress AXE. She showed that, once integrated with Cypress, AXE can cycle through components. Unfortunately, AXE and other automated tests uncover only about 20% of accessibility tests.

Lighthouse and other tests get run manually to validate the rest of visual accessiblity.

She also showed the Safari screen reader accessibility testing.

Workflow Integration

Marie then described workflow, and how the workflow integration mattered to the Product Platforms team.

She made the team’s first point – quality is everyone’s responsibility. For the product platforms team, the two quality engineers serve as leads on approaches and best practices. Developers must deliver quality in the design system.

To accentuate this point, she explained that the team had developed pull request guidelines. Check-ins and pull requests required documentation and testing checklist of unit, component, and page-level tests. Everyone agreed to this level of work for a pull request.

Next, Marie showed the workflow for a pull request. Each pull request at the component level required a visual validation of the component before merging. She explained how Applitools could maintain separate baselines for each branch and manage checkpoints independently. Then, she showed the full develop workflow build pipeline.

Finally, she showed how Github integration linked visual testing fit into the entire Circle CI build. She also showed how the buld process linked to Slack, so that the team could be notified if the build or testing encountered problems. The build, including all the tests, needed to complete within 10 minutes.

Overall Feedback

Marie provided her team’s general feedback about using Applitools. They concluded that they required Applitools for visual validation of the component-level and site-level tests. Developers appreciated how easily they could use Applitools with Cypress, and how they could run 60 component tests in under 5 minutes across a range of browsers. The design team also uses Applitools to validate the design, and they found the learning curve was fast for figuring out the visual elements.

As users, they did have feedback for improvement to share with the Applitools product team. One of the most interesting came from the design team, who wondered if they could use UI design tools (Sketch, Figma, Abstract, etc.) to seed the initial baseline for an application.

Beyond Applitools, the accessibility testing has helped ensure that News UK can deliver visual and audio accessibility for their users.

Conclusion

Marie Drake made a strong case for using a design system whenever there are multiple design and development groups working independently on a common web application. The design system eliminates redundancy and helps speed the rate of change whenever groups want to roll out application enhancements.

She also made a strong case for building testing into every phase of the design system, from component-level unit, functional, visual, and accessibility tests all the way to page-level tests of the same. For testing speed, testing accuracy, ease of test maintenance, and cross-browser tests, Marie made a strong case for using Applitools as the visual test solution for the News UK design system.

For More Information

• How Do You Simplify End-To-End Test Maintenance?
• Measure Your Test Automation Maturity
• Adding Log Analysis to QA Will Improve Your Processes
• How Do You Catch More Bugs In Your End-To-End Tests?
• How Can 2 Hours Now Save You 1,000 Hours In The Next Year?
• Request an Applitools Demo
• Sign up for a free Applitools account.

The post How Do You Test A Design System? appeared first on AI-Powered End-to-End Testing | Applitools.

Cypress, among other things, validates the structure of your DOM. It verifies that a button is visible or that it has the correct text. But what about the look and styling of our app? How can we test that our application looks good visually? We can use Cypress to verify that it has the correct CSS properties but then our code would look very long and complex. It will be a guarantee that engineers will avoid maintaining this test ?

Visual Testing

This is why we need visual testing as part of our test strategy. With visual testing, we are validating what the user sees on different browsers/viewports/mobile devices. However, it’s very time consuming when you do it manually.

Imagine if someone told you that you have to test the below image manually.

There are 30 differences. You can probably find them after quite some time. But then this provides a really slow feedback loop. The obvious solution would be of course to automate this!

Now, automated visual testing is not new. There are so many tools out there which can help you with visual testing. These tools use pixel-by-pixel comparison to compare the two images. The idea is you have a baseline image which is your source of truth. Every time there is a code change and our automated tests are run, a test image will be taken. The visual testing tool will then compare this test image with the baseline image and then check the differences. At the end, it will report to us whether our test passed or failed.

Problem with pixel-by-pixel visual testing

The problem with pixel-by-pixel visual testing though is it’s very sensitive even with small changes. Even if you have a 1px difference, your test will fail even though on the human eye, the two images look exactly similar.

You also get the issue that if you try to run these tests on your build pipelines, you might see a lot of pixel differences especially if the base image is generated locally such as the image above. Looking at this test image, if you ignore the mismatch image in the middle and compare the two images on left and the right, some of the changes that were reported look ok.  But because these images were taken on different machine setups , the tool has reported a lot of pixel differences. You can use Docker to solve this and generate the base image using the same configuration as the test image, but from personal experience, I still get flakiness with using pixel-by-pixel comparison tools.

Also, what if you have dynamic data? In the test image above, we have a change of data here but the overall layout is similar. You can probably set the mismatch threshold to be slightly higher so your tests will fail only if they reach the threshold that you defined. The problem with this though is that you might miss actual visual bugs.

Cross Browser Visual Validation

Most of the existing open source tools for visual testing only runs on 1 or 2 browsers.  For example, one of the tools that we were using before, called BackstopJS,  which is a popular visual testing framework, only runs visual tests on Chrome headlessly.  AyeSpy, which is a tool that was actually created by one of the teams here at News UK, is another visual testing tool, which hooks into your Selenium Grid to run your visual tests on different browsers. But still, it’s a bit limited. If you are using the Selenium Docker images, they only have images for Chrome and Firefox. What if you want to run your visual tests on Safari or Internet Explorer?  You can definitely verify this yourself, but again as mentioned, it’s time consuming.

How can we solve these different visual testing issues?

Applitools

This is where Applitools comes in. Unlike existing visual tools, Applitools uses AI comparison algorithms so images are compared just like a human would. It was founded in 2013 and pretty much integrates with almost all testing frameworks out there. You name it – Selenium, Cypress, Appium, WebdriverIO, Espresso, XCUITest, even Storybook! With Applitools, you can validate visual differences on a wide range of browsers and viewports. By using different comparison algorithms (exact, strict, layout and content), you have different options to compare images and can cater different scenarios such as dynamic data or animations.

Rather than taking a screenshot, Applitools will extract a snapshot of your DOM. This is one of the reasons why visual tests are fast to run in Applitools. Once the DOM snapshots have been uploaded to the Applitools Cloud, the Applitools Ultrafast Grid, which offers users a way to render screens in multiple browsers and viewports simultaneously to generate the screenshots and do the AI powered image comparison.

Getting Started

To get you started, you need to install the following package to your project. This is specific to Cypress so you would need to install the correct package depending on your test framework of choice.

npm i --D @applitools/eyes-cypress

Once this has been installed, you need to configure the plugin and the easiest way to do this is to run the following command on your terminal

npx eyes-setup

This will automatically add the necessary imports needed to get Applitools working in your Cypress project.

Writing our first test with just Cypress

Let’s start doing some coding and add some validations on a sample react image app that I created a while back. It is a simple image gallery which uses unsplash API for the backend. An example github repo which contains the different code examples can be found here.

Our Cypress test can look like the following code snippet. Keep in mind, this only asserts the application to some extent. I can add more code to verify that it has the correct CSS attributes but I don’t want to make the code too lengthy.

Writing our first test with Cypress and Applitools

Now, let’s look at how we can write the test using Cypress and Applitools.

Applitools provides the following commands to Cypress as a minimum setup. `cy.eyesOpen` initiates the Cypress eyes SDK and we pass some arguments such as our app name, batch name and the browser size (defaults to Chrome). The command `cy.eyesCheckWindow` will take a DOM snapshot so every call to this command means a DOM snapshot will be generated. You can call this every time you do an action such as visiting your page under test or clicking buttons, and dropdown menus. Finally, once you are finished, you just call `cy.eyesClose`. To know more about the Cypress eyes SDK, please visit their documentation here to find more information.

In order to run this in Applitools, you need to export an API key which is detailed on this article. Once you have the API key, on your terminal you need to run:

export APPLITOOLS_API_KEY=<your_key>

npx cypress open

Once the test is finished, if you go to the Applitools dashboard, you should see your test being run. The first time you run it, there will be no baseline image then when you reran the test and everything looks good, you should see the following.

Handling Dynamic Data

Since we are using the unsplash API, we don’t have control as to what data gets returned. When we refresh the app, we might get different results. As an example, the request that I am making to unsplash will get the popular photos on a given day. If I reran my test again tomorrow, then the images will be different like the one shown below.

The good thing is we can apply a layout region so the actual data will be ignored or we can also set the match level as Layout in our test which we can preview on the dashboard. If the layout of our image gallery has changed, Applitools will report it as an issue.

Making code changes to our application

Now, let’s create some changes in our application (code references found here) and introduce the following:

• Update the background colour of the search bar (bug)
• Add a new footer component (new feature)
• Remove camera icon on the subtitle (bug)

If we run the test where we only use Cypress, how many of these changes do you think your test will catch? Will it catch that there is a new footer component? How about updating the background colour of the search bar? How about the missing header icon? Probably not because we didn’t write any assertions for it. Now, let’s rerun the test written in Cypress and Applitools.

Looking at the image above, it caught all the changes and we didn’t had to update our test since all the maintenance is done on the Applitools side. Any issues can be raised directly on the dashboard and you can also configure it to integrate to your JIRA projects.

Cross Browser Visual Validation

To run the same test on different browsers, you just need to specify the browser options on your Applitools configuration. I’ve refactored the code a bit and created a file called `applitools.config.js` and moved some of the setup we added initially in our `cy.eyesOpen` in this class.

Just simply reran your test and check the results in the Applitools dashboard.

Final Thoughts

This is just an introductory post on how you can use Applitools so if you want to know more about its other features, check out the following resources:

• Applitools Master Class Webinar
• Applying Match Levels
• Applying Match Regions
• Test Automation University – different courses on using Applitools with other frameworks

While open source pixel by pixel comparison tools can help you get started with visual testing, using Applitools can modernize the way you do testing. As always, do a thorough analysis of a tool to see if it will meet your testing needs ?

The post Modern Cross Browser Testing with Cypress and Applitools appeared first on AI-Powered End-to-End Testing | Applitools.