Hello everyone! I’m Noam, an SDK developer on the Applitools JS-SDKs team. While my day-to-day focus is on core engineering, I work closely with our field teams and occasionally join technical deep-dive sessions with customers.

In these conversations, we frequently encounter questions about performance and the engineering philosophy behind our integration. Specifically, there is often curiosity about how to make visual testing feel more “Playwright-native” and natural to developers.

In this post, I’ll share the design logic behind these architectural choices so you can apply these patterns in your own CI pipelines in a way that fits your organization’s needs.

Adding unresolved to Playwright

Integrating visual regression testing into Playwright requires combining two different status models: Playwright’s binary Pass/Fail and the visual testing concept of unresolved.

In visual testing, instead of having two (passed and failed) states, there’s an additional third state: unresolved. This state indicates a difference was detected, but a human decision is required to determine if it is a bug or a valid change that should be approved as a new baseline.

​Playwright doesn’t support this third state out of the box. Visual test maintenance using Playwright’s native toHaveScreenshot API forces the developer into a cumbersome cycle requiring three separate test executions:

1. First, the developer needs to run to see the failure.
2. Then, they need to run with the --update-snapshots flag to create new baseline images.
3. Then, most developers would run again to validate that everything works with the updated baseline as expected—which isn’t always the case, because the Playwright native comparison method (pixelmatch) tends to be very flaky, unlike Visual AI.

​After this local cycle, the developer must commit the new baseline images to the repository—bloating the git history—and wait for a new CI execution to provide final feedback. For dev-centered organizations that focus on feedback loop velocity, this workflow is… suboptimal. Personally, I believe that’s one of the reasons visual testing isn’t as popular as it should be among Playwright users.

​When we engineered the Applitools fixture, one of our goals was to support this Unresolved state natively, without disrupting Playwright’s core lifecycle—specifically its Worker Processes and Retry mechanisms.

The solution rests on two key engineering decisions: moving rendering to the background (async architecture) and giving developers control over the exit signal and performance tradeoffs (failTestsOnDiff).

We don’t block test execution when Applitools is rendering

The core value of visual testing lies in AI-based comparison to eliminate false positives and multi-platform rendering.

Architecturally, these processes are cloud-native services.

• AI-as-a-Service: Just like massive LLMs or other generative models, the Visual AI engine runs on specialized cloud infrastructure optimized for heavy inference. It cannot simply be “installed” on a lightweight CI agent.
• Platform Constraints: Authentic cross-platform rendering (e.g., iOS Safari on a Linux CI agent) is physically impossible on a single local machine.

Since these operations inherently occur remotely, performing them synchronously would force the local test runner to idle while waiting for network round-trips and cloud processing.

To solve this, we designed the fixture around an asynchronous architecture:

• ​Instant Capture: When eyes.check() is called, we synchronously capture the DOM and CSS resources (instead of a rasterized image). This operation is extremely fast.
• ​Immediate Release: We purposefully use soft assertions by design. We release the Playwright test thread immediately so the functional logic can proceed to the next step or test case without blocking.
• ​Background Heavy Lifting: The heavy work—uploading assets, rendering across different browsers and operating systems, and performing the AI comparison in the Applitools cloud—starts immediately in the background, managed by the Worker process.

The “Draining Queue” Effect

​This architecture explains why the Playwright Worker sometimes remains active after the final test completes.

​The background tasks are limited only by your account’s concurrency settings, and the screenshot size. For example, when rendering a 10,000 px page on a small mobile device, the rendering infrastructure might need time for scrolling and stitching. If your functional tests execute faster than the background workers can process the queue (rendering & comparing), the Worker process stays alive at the end solely to “drain the queue” and ensure data integrity.

While it does ensure your test logic runs at maximum speed, offloading the processing cost to the background, this experience might cause friction and frustration as the developers see that workers are “hanging” after tests are completed. When facing such issues, our support team is here to advise and assist with various solutions—we can investigate execution logs and if needed even make custom suggestions to tailor Eyes-Playwright to your needs.

Solving the Matrix Problem

​Standard Playwright documentation recommends defining multiple projects in playwright.config.ts to cover different browsers (Chromium, Firefox, WebKit) and various viewport sizes.

​While this ensures coverage, it introduces a linear performance penalty (O(N)). To test three browsers across two viewports, your CI must execute the functional logic (clicks, waits, navigation) six times. It’s 6x more load on the CI machine and the testing environment.

​We recommend shifting this workload to the Ultrafast Grid (UFG).

​In this mode, you execute the Playwright test once, typically on Chromium. We upload the DOM state, and our cloud infrastructure renders that state across all configured browsers and viewports in parallel.

This transforms an O(N) execution problem into an O(1) execution problem, significantly shortening the feedback loop.

The Strategy: failTestsOnDiff

​Since the actual comparison happens asynchronously and potentially completes after the test logic finishes, we need a mechanism to map the visual result back to the Playwright status.

​This is controlled by the failTestsOnDiff flag. It’s not just a boolean; it’s a strategic choice for your CI pipeline.

​Strategy A—Recommended for CI: failTestsOnDiff: false

• ​The Logic: This is the configuration our own Front-End team uses. We believe that Visual Change ≠ Test Failure.
• ​Behavior: The Playwright test passes (Green). The unresolved status is reported externally via our SCM integration (GitHub/GitLab).
• ​Why: Retrying a visual test is computationally wasteful—the pixels won’t change on the second run. By keeping the test “Green,” we avoid triggering Playwright’s retry mechanism. The decision is moved to the Pull Request, where it belongs.

Read more about SCM integration or hop directly to our GitHub, Bitbucket, Gitlab or Azure Devops articles.

Strategy B—Recommended for Strict Pipelines: failTestsOnDiff: 'afterAll'

• ​The Logic: You need a “Red” pipeline to block deployment, but you want to avoid the noise of retries and gain a significant performance improvement.
• ​Behavior: Individual tests pass, but the Worker Process exits with a failure code if any diffs were found in the suite.
• ​Why: This provides a hard gatekeeper for the build status. It allows the Eyes rendering farms to continue processing visual test results in the background without blocking the execution thread, allowing the worker to move on to handle more tests efficiently.

​Strategy C—Recommended for Local Debugging: failTestsOnDiff: 'afterEach'

• ​The Logic: Immediate feedback loop.
• ​Behavior: Fails the test immediately in the afterEach hook.
• ​Why: Best for local development where you want to see the failure immediately in the console. It is also useful if you use the trace: retainOnFailure setting in Playwright, as it ensures traces are preserved for unresolved visual assertions. Not recommended for CI due to the retry loops described above.

TL;DR – When to use each setting

Closing the Visibility Gap: The Custom Reporter

​If you adopt Strategy A (false) or Strategy B (afterAll), you introduce a secondary challenge: Visibility.

Since Playwright technically marks these tests as Passed to avoid retries, the standard Playwright HTML Report will show them as “Green,” potentially masking unresolved visual differences that require attention.

​To bridge this gap without forcing developers to switch context, we developed a Custom Applitools Reporter.

​This reporter extends the standard Playwright HTML report. It injects the actual visual status (Passed, Failed, or unresolved) directly into the test results view.

• ​True Status: You see which tests have visual diffs, even if the Playwright exit code was successful.
• ​Direct Links: It provides a direct link from the test report to the specific batch results in the Applitools Dashboard.
• ​Context: It enriches the report with UFG render status and batch information.

​This ensures you get the best of both worlds: The optimization of a “Green” CI run (no retries), with the transparency of a report that highlights exactly where manual review is needed.

Summary

​The Applitools Playwright fixture is designed to be non-blocking and scalable. By leveraging asynchronous architecture and Applitools UltraFast Grid, we offload the heavy lifting from your CI. By correctly configuring failTestsOnDiff, you ensure that your pipeline reflects your team’s engineering culture—whether that’s strict gating or modern, PR-based visual review.

The post Engineering a Playwright-Native Developer Experience: One Flag, Three Strategies appeared first on AI-Powered End-to-End Testing | Applitools.

Local dev is where most UI changes happen (and where regressions sneak in). States drift, styles diverge, and tiny tweaks pile up until something breaks in CI. The Applitools Storybook Addon brings AI-powered visual testing straight into Storybook so you can catch issues as you code, approve the good changes quickly, and keep your CI/CD pipelines green.

AI-Powered Visual Testing Inside Storybook

Open your Storybook and run visual tests from an Applitools Eyes tab – no context switching. Results are grouped by component to mirror your Storybook structure, and a reporter widget highlights what needs attention first so you can review diffs in minutes, not hours. Learn more on our Storybook Component Testing with Applitools page.

• Catch bugs where you build. Validate component states during local development and avoid surprises later.
• Review faster with Visual AI. See only meaningful, human-perceptible UI changes without pixel-to-pixel noise. Tune sensitivity with AI match levels when you need to.
• Scale coverage painlessly. Run once; render everywhere with Ultrafast Grid across browsers, devices, and viewports in parallel.

How to Use the Applitools Eyes Storybook Addon

Getting started takes just a couple of minutes.

1. Install the SDK & Addon
   Add Applitools Eyes to your project and enable the Storybook addon (React, Vue, Angular supported). See the installation instructions in the Eyes Storybook Addon docs.
2. Run Applitools Visual Tests in Storybook
   Open Storybook, switch to the Applitools Eyes tab, and trigger tests for a single story or an entire component. Results stream back in real time with automatic grouping by component.
3. Review & Maintain
   Use Visual AI diffs, side-by-side views, and auto-maintenance to approve or reject changes in bulk. Prioritized sorting surfaces what needs attention first.
4. Scale Across Browsers/Devices
   Turn on Ultrafast Grid to parallelize renders across Chrome, Firefox, Safari, Edge, and mobile sizes – without extra local setup.

Applitools Storybook Addon Use Case Playbook

Below are the three most common ways teams use the Eyes Storybook Addon each – with a quick, practical flow pulled right from the product.

Use Case: Guard Your Design System

As you refactor tokens or update themes, run visual tests on every component state. Spot unintended changes across the library instantly.

How to do it in Storybook

1. Start Storybook and open your design‑system component in the Applitools Eyes tab.
2. Click Run from the tab (or use Run in the left sidebar test module). The addon tests the stories and streams results inline for every browser/device in your applitools.config.js.
3. In the sidebar, filter by Unresolved to zero in on changes across the library (Green = passed, Orange = unresolved, Red = failed).
4. Open a story’s result and use Side‑by‑Side or the Slider to spot subtle spacing/typography diffs.
5. Approve legit theme updates with Thumbs Up (or use ⋯ → Review actions to approve the whole story/batch). Reject regressions with Thumbs Down and fix.

Pro tip: Use the tab ⋯ → Configuration to confirm you’re validating the right browser matrix and server URL. See more options in the docs.

Use Case: Fix Fast During Local Dev

Working on a feature branch? Validate your component in Storybook before you commit.

How to do it in Storybook

1. Open your feature’s stories, then hit Run in the Applitools tab for the component you’re touching.
2. Watch statuses update inline; click the status buttons to filter to Unresolved so you only look at what changed.
3. Click into any row to open compare tools: Diff Image, Actual Image, Expected Image, Side‑by‑Side, or Slider.
4. If the change is intended, Thumbs Up to approve; otherwise Thumbs Down to flag and keep iterating.
5. When you’re happy locally, push your branch. You can scale the same setup in CI using your existing Storybook build/preview URL.

Heads‑up: To view baselines or approve/reject, sign in to your Applitools account in the same browser that’s running Storybook (you’ll be prompted if not).

Use Case: Ship Multi‑Browser Confidence

One click, many targets. Validate layout and responsive behavior across browsers and viewports – early.

How to do it in Storybook

1. In ⋯ → Configuration, verify your browsers/devices list (Chrome, Firefox, Safari, Edge; add viewports you care about).
2. Hit Run for representative stories (states, theming, interactive). Results come back grouped by each browser/device so differences are obvious.
3. Filter the sidebar by Unresolved and scan. Use Side‑by‑Side or Slider to compare layout at different sizes.
4. Approve good changes in bulk (⋯ → Review actions) to keep maintenance low.
5. For broader coverage, run the same setup in CI and expand the matrix.

Why Visual AI > Pixel Diffs for Storybook

Pixel-to-pixel tools are fragile with dynamic content and minor rendering differences. Applitools Visual AI mimics human vision to highlight only meaningful UI changes (structure, layout, content) while ignoring the noise. You can still dial sensitivity up or down with match levels whenever needed. Less flake, more signal.

Try AI-Powered Visual Testing in Storybook Today

Run your first component tests in minutes, review diffs right in Storybook, and expand coverage with Ultrafast Grid – without slowing delivery.

• Explore the Applitools Storybook Component Testing solution
• Read the Applitools Storybook Addon docs
• Compare Applitools Eyes against Chromatic

The post Test Your Components Where You Build with the Applitools Storybook Addon appeared first on AI-Powered End-to-End Testing | Applitools.

Even the best design systems can fall short when a layout moves from Figma to code. Fonts shift, buttons resize, and colors look a little off. These small issues result in visual drift and long review cycles between design, development, and QA.

Figma design testing with Applitools Eyes closes that gap. Export Figma frames directly to Eyes to compare what you designed with what you built using the same visual testing tools your QA teams already trust.

Design-to-Code Testing in One Place

The plugin lets you send Figma frames, including individual components, pages, or entire prototypes, straight into Applitools Eyes. Each exported frame becomes a visual baseline, the same kind used in automated tests.

Developers can run their regular visual tests against these baselines to confirm that what they’ve built matches the approved design. Meanwhile, Designers can export each new version of a design to see what changed between iterations. Everyone reviews results in the same Eyes dashboard, where visual differences appear side by side.

This shared view reduces guesswork and keeps teams aligned around what “correct” actually looks like.

How to Use the Applitools Eyes Figma Plugin

Getting started takes just a couple of minutes.

1. Install the Plugin

Open the plugin from the Figma Store, or open the Figma desktop app and select Plugins → Manage Plugins → Search “Applitools Eyes” → Install.

2. Connect Your Applitools Account

Launch the plugin and enter your Applitools API key and server URL (default: https://eyes.applitools.com). These settings are saved for future use.

3. Select Figma Frames to Export

You can export a single frame, multiple frames, or a full design. The plugin automatically names them based on your Figma file, or you can customize names with dynamic parameters like {figma_filename}, {figma_page}, or {figma_frame}.

4. Adjust Settings

Optional configurations include:

• Match level: strict, dynamic, layout, ignore colors, exact, or none
• Contrast level: accessibility comparison thresholds
• Auto-accept baselines: mark first exports as approved
• And more…

5. Export and Review

Lastly, click Export to Eyes to send your selections to Applitools. Frames appear in the Eyes dashboard under the “Figma” environment. Designers and Devs can view differences directly and decide whether to accept or reject them.

Three Use Cases for QA Teams

1. Design-to-Implementation Validation

Once designs are uploaded, developers can link automated tests to the same baseline using the “baseline environment name” provided by the plugin. When they run their tests, Eyes compares the live UI against the design reference.

Result: Teams catch spacing, text, or layout differences before they reach production.

2. Design-to-Design Version Comparison

Designers often revisit earlier layouts or explore small variations. Exporting both versions to Eyes highlights the exact visual differences, making it easy to review and choose the preferred version.

Result: Faster review cycles and fewer overlooked design changes.

3. Shared Visual Baselines for Collaboration

Designers, developers, and QA teams can all access the same Eyes dashboard. Instead of passing screenshots or notes, they can comment on the same visual checkpoints.

Result: Clearer handoffs and fewer miscommunications between design and engineering.

Why Visual Testing from Design to Code Matters

Designs are often reviewed visually, while code is tested functionally. However, the Figma plugin connects these two disciplines by giving both teams a consistent, visual source of truth.

For designers, it’s a way to confirm that their layouts are faithfully implemented without manually comparing screenshots. The plugin provides a reference that removes ambiguity about spacing, colors, or typography for developers. For QA teams, it introduces an additional layer of confidence that each release matches approved specifications.

This integration fits naturally into existing workflows: designs are exported once, developers test as usual, and visual checks happen automatically. What was once a manual review step becomes part of the team’s regular quality process.

Try Design-to-Code Testing for Yourself

The Applitools Eyes Figma Plugin brings visual testing into the design process, helping teams maintain consistency from mockup to release. It’s a straightforward way for design and development to share one accurate reference for how an interface should look in order to reduce manual review and give everyone confidence that what’s shipped matches what was designed.

Install the Applitools Eyes Figma Plugin and start validating your designs before code ships.

The post Validate Your Figma Designs Before Code Ships with the Applitools Eyes Plugin appeared first on AI-Powered End-to-End Testing | Applitools.

Applitools was named CIO Review’s 2025 AI-Powered Test Automation Platform of the Year—not because we chased buzzwords, but because the platform is fundamentally AI-native, built for engineering scale, and designed to help developers test smarter without slowing down.

For developers, testers, and QA engineers, the award reflects what actually matters:

• Reducing test flakiness
• Automating visual and functional checks in parallel
• Scaling test execution across browsers and devices
• Plugging into CI/CD pipelines without disrupting existing workflows

Let’s break down what makes this platform stand out from a developer’s perspective.

AI-Native Testing, Not Bolt-On AI

Applitools isn’t a traditional test framework with AI sprinkled on top. It’s purpose-built to use Visual AI plus code-aware intelligence for smarter test coverage. That means:

• You can catch regressions that DOM diffs would miss
• You write fewer assertions, yet spot more visual and layout issues
• You reduce false positives and test flakiness—without relying on brittle selectors

It’s AI-native automation that understands what the user sees, not just what the code renders.

Built for Real Engineering Workflows

Applitools supports every major language and framework, including:

• Languages: JavaScript, TypeScript, Java, Python, C#, Ruby
• Frameworks: Cypress, Playwright, Selenium, WebdriverIO, and more
• Mobile: Appium and native frameworks

You don’t need to rip and replace. Applitools plugs directly into your current test suite with minimal setup and no test rewrites required.

Ultrafast Grid = Multi-Platform Testing Without the Bottlenecks

You run your tests once. Applitools executes them across dozens of browser, OS, and device combinations in parallel—via the Ultrafast Grid, not your CI or local machine.

That means:

• Fast, scalable cross-browser coverage
• Smart DOM diffing combined with Visual AI
• Consistent UX testing across breakpoints and devices

No emulators. No stitched screenshots. Just reliable results, fast.

Seamless CI/CD Integration

Applitools fits natively into DevOps workflows with:

• GitHub Actions, GitLab, Jenkins, CircleCI, Azure Pipelines, Bitbucket, TeamCity
• Rich CLI tooling for custom pipelines
• Git-based test baselines and approval workflows
• Smart diffing and auto-approvals to keep noisy builds out of your way

For more, explore our Integrations Hub.

This is test automation that moves with your code, not one that slows it down.

Dev Teams Are Reporting…

Here’s what teams have seen after adopting Applitools:

• Up to 80% reduction in test maintenance overhead
• 10x faster execution across browsers and devices
• 70% fewer visual bugs escaping into production
• Faster code reviews with fewer test-related delays

Whether you’re validating a single feature branch or running thousands of tests in parallel, Applitools is built to support real scale—without compromising on accuracy.

Why This Award Actually Matters

The CIO Review award isn’t about hype. It’s a reflection of what forward-looking engineering teams need from test automation in 2025—More confidence. Less friction. AI that works.

If you’re building modern apps, you deserve modern testing. Applitools gives you a platform that evolves with your code, scales with your team, and delivers confidence without the test fatigue.

Applitools Resources for Developers

• Docs & SDKs
• Start a Free Trial
• Cory House’s Creating Automated Tests with AI webinar
• Adam Carmi’s Building No-Code Autonomous End-to-End Tests webinar

The post AI Test Automation Platform for Developers: Why Applitools Won in 2025 appeared first on AI-Powered End-to-End Testing | Applitools.

TL;DR — Applitools brings visual, functional, and autonomous testing together in a single AI-powered platform. Backed by 11+ years of refinement and a dataset of 4 billion real-world images, our Visual AI delivers unmatched accuracy and reliability for enterprise-grade software testing.

Testing today isn’t just about coverage—it’s about confidence, speed, and scaling quality across teams. Whether you’re a developer chasing faster feedback, a QA lead reducing maintenance overhead, or a product owner focused on release velocity, Applitools helps modern teams deliver software that looks right, works right, and evolves with ease.

Here’s how Visual, Functional, and Autonomous Testing all come together in one powerful platform.

Trusted Visual AI with Proven Accuracy

Applitools sets the standard in Visual Testing. Our Visual AI engine delivers 99.9999% accuracy, eliminating false positives and catching bugs others miss.

• 5.8x more efficient than pixel-based tools
• Detect both functional and visual bugs in a single test
• Works with all major frameworks: Selenium, Cypress, Playwright, and more

We didn’t just add AI—we’ve spent 11+ years perfecting it.

A Complete Platform for End-to-End Testing

Applitools goes far beyond screenshots. Our Intelligent Testing Platform includes Autonomous Test Creation, Visual Validation, Cross-Browser + Device Testing, and Accessibility Testing—all in one cloud-based solution.

• Run tests across browsers, devices, and screen sizes in parallel
• Built-in accessibility and compliance testing
• Fully scalable with enterprise-grade performance

Less Test Maintenance with Self-Healing, Smart Grouping & Predictive Analytics

Spend less time fixing broken tests and more time delivering value. Applitools minimizes test upkeep so your team can focus on building.

• Self-healing locators reduce test flakiness and manual updates
• Root cause analysis, smart grouping, and predictive analytics streamline debugging
• Tests automatically adapt as your UI evolves—no babysitting required

Collaborative Testing: How Developers, PMs, Designers & Marketers All Work Smarter with Applitools

Testing shouldn’t be a bottleneck—or limited to just QA. Applitools empowers developers, designers, product managers, and even marketers to collaborate with ease.

• Intuitive UI for reviewing results and managing baselines
• Seamless sharing of results and issue tracking
• Codeless and code-based authoring, no deep technical expertise needed

More than a Decade of AI Leadership

AI isn’t new to us—it’s the foundation of our platform. Unlike newer tools making AI promises, we’ve been building, training, and refining Visual AI to solve real testing challenges at scale for more than a decade.

• Mature, battle-tested AI that improves with scale
• Trusted by engineering teams at Microsoft, Tesla, Salesforce, eBay, and 400+ more
• Over 1 billion visual tests run with our platform

Seamless Integrations & Dev Experience

Great testing fits into your workflow—not the other way around. Our AI-powered test automation works with your tools, languages, and CI/CD pipelines to scale quality without slowing you down. Applitools integrates with:

• Every major framework: Selenium, Cypress, Playwright, Puppeteer, WebdriverIO
• CI/CD tools: GitHub Actions, Jenkins, GitLab, Azure DevOps
• SDKs for Java, JavaScript, Python, C#, and more

Whether you’re in code or no-code workflows, we plug into your stack and scale with you.

24/7 Support That Doesn’t Disappear

Whether you’re mid-sprint or troubleshooting a release, help is always within reach. Get expert guidance anytime—no hoops, no waiting.

• Around-the-clock global technical support
• Extensive documentation, how-tos, and real-time guidance
• Active community forum and dedicated Customer Success Managers (not just for enterprise)

Compare that to competitors with limited support, slow response times, or no dedicated resources unless you’re a top-tier customer.

Smart Investment, Real Value

Our pricing is flexible, predictable, and scales with your needs. You’ll see ROI fast:

• Save hours of test maintenance per sprint
• Eliminate manual bug hunts and false positives
• Deliver faster releases without compromising quality

Explore our current pricing structure, or speak with a testing specialist to build a package that’s right for your team.

“We reduced our testing time from days to hours. Applitools changed how we think about QA.”
— QA Lead, Global Retail Brand

Visual, Functional, and Autonomous Testing—The Applitools Advantage

We combine Visual AI, Autonomous Testing, and a developer-friendly platform into one powerful, scalable solution. With Applitools, your team gets:

• Smarter test creation
• Less maintenance
• Better collaboration
• Faster releases
• And trusted results every time

See What’s New with Applitools Autonomous and What’s Coming with Applitools Eyes

Ready to Test Smarter?

In a crowded automation landscape, it’s not enough to have “AI-powered” features. You need real results. With over a billion visual tests run and trusted by leading enterprises across industries, Applitools isn’t experimenting with AI—it’s already delivering.

Whether you’re starting fresh or looking to scale smarter, Applitools gives your team the tools to automate with confidence and speed.

Ready to see it in action? Start your free trial, book a personalized demo, or explore the platform today.

Applitools helps you test like it’s 2025. Join the world’s top teams already doing it.

The post Visual, Functional, and Autonomous Testing—All in One appeared first on AI-Powered End-to-End Testing | Applitools.

The challenges of building UI

Modern UIs today are built by a diverse set of teams that work together at different parts of the process. The pace at which these design, development, QA, operations, marketing, and product teams ship their work is continuing to accelerate – creating new challenges around communication, collaboration, and validation across the workflow.

Getting from design mock-ups in Figma to live UI is a process that includes a lot of feedback and testing. It starts with the designer who passes to the product manager for approval before the developer can start building. Feedback in the development process requires rework to make those updates before it can get approval from the product manager. This is all before the testing team has even started their review.

You can see the game of telephone that’s played through different stakeholders into production, and we get something that’s slightly different at multiple levels. This makes measuring what actually happened and what actually needs to change incredibly hard, making it a huge burden on teams to ship clean UIs at a fast pace. Some of our main challenges here are:

• Lack of communication between the growing group of stakeholders
• Breadth of technology during implementation causing inconsistencies
• No continued source of truth across tooling as the app UI evolves

The post Announcing Applitools Centra: UI Validation From Design to Implementation appeared first on AI-Powered End-to-End Testing | Applitools.

Learn about common localization bugs, the traditional challenges involved in finding them, and solutions that can make localization testing far easier.

What is Localization?

Localization is the process of customizing a software application that was originally designed for a domestic market so that it can be released in a specific foreign market.

How to Get Started with Localization

Localization testing usually involves substantial changes of the application’s UI, including the translation of all texts to the target language, replacement of icons and images, and many other culture, language, and country-specific adjustments, that affect the presentation of data (e.g., date and time formats, alphabetical sorting order, etc.). Due to the lack of in-house language expertise, localization usually involves in-house personnel as well as outside contractors, and localization service providers.

Before a software application is localized for the first time, it must undergo a process of Internationalization.

What is Internationalization?

Internationalization often involves an extensive development and re-engineering effort which goal is to allow the application to operate in localized environments and to correctly process and display localized data. In addition, locale-specific resources such as texts, images and documentation files, are isolated from the application code and placed in external resource files, so they can be easily replaced without requiring further development efforts.

Once an application is internationalized, the engineering effort required to localize it to a new language or culture is drastically reduced. However, the same is not true for UI localization testing.

The Challenge of UI Localization Testing

Every time an application is localized to a new language, the application changes, or the resources of a supported localization change, the localized UI must be thoroughly tested for localization and internationalization (LI) bugs.

Common Localization and Internationalization Bugs Most Testers can Catch

LI bugs which can be detected by testers that are not language experts include:

• Broken functionality – the execution environment, data or translated resources of a new locale, may uncover internationalization bugs can prevent the application from running or break some of its functionality.
• Untranslated text – text appearing in text fields or images of the localized UI is left untranslated. This indicates that certain resources were not translated or that the original text is hard-coded in the UI and not exported to the resource files.
• Text overlap / overflow – the translated text may require more space than available in its containing control, resulting with the text overflowing the bounds of the control and possibly overlapping or hiding other UI elements.
• Layout corruption – UI controls dynamically adjust their size and position to the expanded or contracted size of the localized text, icons or images, resulting with misaligned, overlapping, missing or redundant UI artifacts.
• Oversized windows and dialogs – multiple expanded texts and images can result with oversized tooltips, dialogs and windows. In extreme situations, expanded dialogs and windows may only be partially visible in low screen resolutions.
• Inadequate fonts – a control’s font cannot properly display some characters of the target language. This usually results with question marks or glyphs being displayed instead of the expected text.

Localization and Internationalization Bugs Requiring Language Expertise

Other common LI bugs which can only be detected with the help of a language expert include:

• Mistranslation – translated text that appears once in the resource files, may appear multiple times in different parts of the application. The context in which the text appears can vary its meaning and require a different translation.
• Wrong images and icons – images and icons were replaced with wrong or inappropriate graphics.
• Text truncation – the translated text may require more space than available in its containing control, resulting with a truncated string.
• Locale violations – wrong date, time, number and currency formats, punctuation, alphabetical sort order, etc.

Localization and Internationalization Bugs are Hard to Find

An unfortunate characteristic of LI bugs, is that they require a lot of effort to find. To uncover such bugs, a tester (assisted by a language expert) must carefully inspect each and every window, dialog, tooltip, menu item, and any other UI state of the application. Since most of these bugs are sensitive to the size and layout of the application, tests must be repeated on a variety of execution environments (e.g., different operating systems, web browsers, devices, etc.) and screen resolutions. Furthermore, if the application window is resizable, tests should also be repeated for various window sizes.

Why is UI Localization Testing Hard?

There are several other factors that contribute to the complexity of UI Localization testing:

• Lack of automation – most of the common LI bugs listed above are visual and cannot be effectively detected by traditional functional test automation tools. Manual inspection of the localized UI is slower than with a non-localized UI because it is  unreadable to the tester.
• Lack of in-house language expertise – since many of the common LI bugs can only be detected with the help of external language experts, which are usually not testers and are not familiar with the application under test, LI testing often requires an in-house tester to perform tests together with a language expert. In many cases, these experts work on multiple projects for multiple customers in parallel, and their occasional lack of availability can substantially delay test cycles and product releases. Similarly, delays can occur while waiting for the translation of changed resources, or while waiting for translation bugs to be fixed.
• Time constraints – localization projects usually begin at late stages of the development lifecycle, after the application UI has stabilized. In many cases, testers are left with little time to properly perform localization tests, and are under constant pressure to avoid delaying the product release.
• Bug severity – UI localization bugs such as missing or garbled text are often considered critical, and therefore must be fixed and verified before the product is released.

Due to these factors, maintaining multiple localized application versions and adding new ones, incurs a huge overhead on quality assurance teams.

Fortunately, there is a modern solution that can make localization testing significantly easier – Automated Visual Testing.

How to Automate Localization Testing with Visual Testing

Visual test automation tools can be applied to UI localization testing to eliminate unnecessary manual involvement of testers and language experts, and drastically shorten test cycles.

To understand this, let’s first understand what visual testing is, and then how to apply visual testing to localization testing.

What is Visual Testing?

Visual testing is the process of validating the visual aspects of an application’s User Interface (UI).

In addition to validating that the UI displays the correct content or data, visual testing focuses on validating the layout and appearance of each visual element of the UI and of the UI as a whole. Layout correctness means that each visual element of the UI is properly positioned on the screen, is of the right shape and size, and doesn’t overlap or hide other visual elements. Appearance correctness means that the visual elements are of the correct font, color, or image.

Visual Test Automation tools can automate most of the activities involved in visual testing. They can easily detect many common UI localization bugs such as text overlap or overflow, layout corruptions, oversized windows and dialogs, etc. All a tester needs to do is to drive the Application Under Test (AUT) through its various UI states and submit UI screenshots to the tool for visual validation.

For simple websites, this can be as easy as directing a web browser to a set of URLs. For more complex applications, some buttons or links should be clicked, or some forms should be filled in order to reach certain screens. Driving the AUT through its different UI states can be easily automated using a variety of open-source and commercial tools (e.g., Selenium, Cypress, etc.). If the tool is properly configured to rely on internal UI object identifiers, the same automation script/program can be used to drive the AUT in all of its localized versions.

So, how can we use this to simplify UI localization testing?

How Automated Visual Testing Simplifies UI Localization Testing

• Preparation – in order to provide translators with the context required to properly localize the application, screenshots of the application’s UI are often delivered along with the resource files to be localized. The process of manually collecting these screenshots is laborious, time consuming, and error prone. When a visual test automation tool is in place, updated screenshots of all UI states are always available and can be shared with translators with a click of a button. When an application changes, the tool can highlight only those screens (in the source language) that differ from the previous version so that only those screens are provided to translators. Some visual test automation tools also provide animated “playbacks” of tests showing the different screens, and the human activities leading from one screen to the next (e.g., clicks, mouse movements, keyboard strokes, etc.).  Such animated playbacks provide much more context than standalone screenshots and are more easily understood by translators, which are usually not familiar with the application being localized. Employing a visual test automation tool can substantially shorten the localization project’s preparation phase and assist in producing higher quality preliminary translations, which in turn can lead to fewer and shorter test cycles.

• Testing localization changes – visual test automation tools work by comparing screenshots of an application against a set of previously approved “expected” screenshots called the baseline. After receiving the translated resources and integrating them with the application, a visual test of the updated localized application can be automatically executed using the previous localized version as a baseline. The tool will then report all screens that contain visual changes and will also highlight the exact changes in each of the changed screens. This report can then be inspected by testers and external language experts without having to manually interact with the localized application. By only focusing on the screens that changed, a huge amount of time and effort can be saved. As we showed above, most UI localization bugs are visual by nature and are therefore sensitive to the execution environment (browser, operating system, device, screen resolution, etc.). Since visual test automation tools automatically execute tests in all required execution environments, testing cycles can be drastically shortened.

• Testing new localizations – when localizing an application for a new language, no localized baseline is available to compare with. However, visual test automation tools can be configured to perform comparisons at the layout level, meaning that only layout inconsistencies (e.g., missing or overflowing text, UI elements appearing out of place, broken paragraphs or columns, etc.) are flagged as differences. By using layout comparison, a newly localized application can be automatically compared with its domestic version, to obtain a report indicating all layout inconsistencies, in all execution environments and screen resolutions.

• Incremental validation – when localization defects are addressed by translators and developers, the updated application must be tested again to make sure that all reported defects were fixed and that no new defects were introduced. By using the latest localized version as the baseline with which to compare the newly updated application, testers can easily identify the actual changes between the two versions, and quickly verify their validity, instead of manually testing the entire application.

• Regression testing – whenever changes are introduced to a localized application, regression testing must be performed to make sure that no localization bugs were introduced, even if no direct changes were made to the application’s localizable resources. For example, a UI control can be modified or replaced, the contents of a window may be repositioned, or some internal logic that affects the application’s output may change. It is practically impossible to manually perform these tests, especially with today’s Agile and continuous delivery practices, which dictate extremely short release cycles. Visual test automation tools can continuously verify that no unexpected UI changes occur in any of the localized versions of the application, after each and every change to the application.

• Collateral material – in additional to localizing the application itself, localized versions of its user manual, documentation and other marketing and sales collateral must be created. For this purpose, updated screenshots of the application must be obtained. As described above, a visual test automation tool can provide up-to-date screenshots of any part of the application in any execution environment. The immediate availability of these screenshots significantly reduces the chance of including out-of-date application images in collaterals and eliminates the manual effort involved in obtaining them after each application change.

Application localization is notoriously difficult and complex. Manually testing for UI localization bugs, during and between localization projects, is extremely time consuming, error-prone, and requires the involvement of external language experts.

Visual test automation tools are a modern breed of test automation tools that can effectively eliminate unnecessary manual involvement, drastically shorten the duration of localization projects, and increase the quality of localized applications.

Applitools Automated Visual Testing and Localization Testing

Applitools has pioneered the use of Visual AI to deliver the best visual testing in the industry. You can learn more about how Applitools can help you with localization testing, or to get started with Applitools today, request a demo or sign up for a free Applitools account.

Editor’s Note: Parts of this post were originally published in two parts in 2017/2018, and have since been updated for accuracy and completeness.

The post Getting Started with Localization Testing appeared first on AI-Powered End-to-End Testing | Applitools.

Learn how you can improve and simplify your UI testing using micro frontends and new Pactflow bi-directional API contract testing.

End-to-End Testing within Microservices

When you are writing end-to-end tests with Cypress, you want to make sure your tests are not flaky, run quickly and are independent of any dependencies. What if you could add contract tests to stabilise, speed up and isolate your UI tests? There’s a new kid on the block – now this is possible with Pactflow’s new feature, bi-directional contracts. UI tests often offer the confidence of the application working end-to-end, which is why utilising contract tests can eliminate some of the challenges mentioned above. To attempt to simplify the explanation of this testing approach, I’m using a recipe web application to describe the interactions between the consumer (web app) and provider (api service). If you want to learn more about API Contract Testing checkout the Pactflow docs.

Microservices was a term first coined in 2011, and microservices have since become a popular way to build web services. With the adoption of microservices, testing techniques have had to adapt as well. Integration tests become really important when testing microservices, ensuring that any changes don’t impact the consuming services or applications.

Micro Frontends started being recognised around 2016. Often when building microservices you need a micro frontend to make the application truly independent. In this setup the integration between the web app and the API service is much easier to test in isolation. The benefits of an architecture that uses micro frontends and microservices together mean you can release changes quickly and with confidence. Add in contract testing to the mix, and you can apply the independent approach to end-to-end testing as well.

Traditionally the running of end-to-end tests looks a little something similar to this:

How to Simplify Your UI Tests with Contract Testing

Using this traditional approach, integration points are covered from the end-to-end tests which can take quite a while to run, are difficult to maintain and are often costly to run within the continuous integration pipeline. Contract Testing does not replace the need for integration tests but minimises the amount of tests needed at that level. 

The introduction of bi-directional contract tests now means you can generate contracts from the UI component tests or end-to-end tests. A great opportunity to utilise the existing tests you already have, this will then provide the confidence that the application works end-to-end without running a large suite of end-to-end tests. The contracts could also be used as stubs within your Cypress tests once they’ve been generated.

In my podcast, I spoke to a developer advocate from Pactflow who told me that they realized there was a barrier to getting started with contract testing. Which was that engineers already had tools which were defining contract interactions such as in mocks or pre-defined openAPI specifications. The duplication of adding pact code to generate these contracts seemed like a lot of work when the contracts had already been defined. Often development teams realise the potential of introducing contracts between services but don’t quite know how to get started or what the true benefits are.

What Benefits Do API Contract Tests Bring to Your UI Tests?

• End-to-end tests can run in isolation, while retaining the confidence of fully integrated tests
• Service providers will verify any API changes before deploying making dependent applications more stable
• How the consumer app interacts with the API service is visualised and better understood as a result
• Versioning and tagging contracts allows you to deploy safely between environments

In a world of micro frontends and microservices, it’s important to isolate services while ensuring quality is not impacted. By adding contract tests to your UI testing suite, not only do you gain the benefits listed you also gain time and costs. Running tests in isolation means your tests are faster to run, with a shorter feedback loop and no need to rely on a dedicated integration environment, reducing environment costs.

The Benefits of Bi-Directional Contract Testing

When building the example recipe app, two teams were involved in defining the API schema. An API contract was documented on the teams’ wiki, which presents the ingredients for a specific cake recipe. Both teams go away and build their parts of the application in line with the API documentation. 

The frontend team uses mocks to test and build the recipe Micro Frontend¹. They want to deploy their Micro Frontend to an environment to see whether they can successfully integrate with the ingredients API service². Also during the development process they realized they needed another field within the ingredients service³, so they communicated with the API team and the developer on the team made the change in the code which generates a new swagger openAPI document⁴ (however they didn’t update the documentation). 

From this scenario there are a couple of things to draw attention to (see numbers 1-4 above):

1. Mocks are often used to test integrations which can be utilised within bi-directional contract testing as test scenarios
2. With contract testing you don’t need a dedicated environment in order to test the interactions between web app and API service
3. Specifications defined before development often change during implementation which can be documented and continuously updated within a centralised contract store such as Pactflow
4. Generated openAPI specifications generated by code can be uploaded to the pact broker as well which can be compared directly with the frontend mocks

As mentioned earlier, the introduction of bi-directional contract testing allows you to generate contracts from your existing tests. Pactflow now provides adaptors which you can use to generate contracts from your mocks for example using Cypress:

describe('Great British Bake Off', () => {
    before(() => {
        cy.setupPact('bake-off-ui', 'ingredients-api')
        cy.intercept(`http://localhost:5000/ingredients/chocolate`,
        {
          statusCode: 200,
          body: ["sugar"],
          headers: { 'access-control-allow-origin': '*' }
        }).as('ingredients')
    })

    it('Cake ingredients', () => {
        cy.visit('/ingredients/chocolate')
        cy.get('button').click()
        cy.usePactWait('ingredients').its('response.statusCode').should('eq', 200)
        cy.contains('li', 'sugar').should('be.visible')
    })
})

Once you have generated a contract from your end-to-end tests, the interactions with the service are now passed to the API provider via the contract store hosted in Pactflow. Sharing the contracts means that the interactions of how the web app behaves after implementation aligns with the API service or if any changes occur post initial development. Think of it like sharing test scenarios with the backend engineers which they will replay on the service they have built. The contract document looks similar to this:

{
    "consumer": {
        "name": "bake-off-ui"
    },
    "provider": {
        "name": "ingredients-api"
    },
    "interactions": [
        {
            "description": "Cake ingredients",
            "request": {
                "method": "GET",
                "path": "/ingredients/chocolate",
                "headers": {
                    "accept": "application/json"
                },
            },
            "response": {
                "status": 200,
                "body": [
                    "sugar"
                ]
            }
        }
    ]
}

Once the openAPI specification has been uploaded by the API service and the contracts have been uploaded by the web application to Pactflow, there is just one more step remaining to call can-i-deploy, which will compare both sides and check that everything is as expected. Voila, the process is complete! You can now safely run tests which are verified by the API service provider and reflect the actual behaviour of the web application.

Changing the Mindset of API Test Responsibility

I know it’s a lot to take in and can be a bit confusing to get your head around this testing approach, especially when you are used to the traditional way of testing integrations with a dedicated test environment or by calling the endpoints directly from within your tests. I encourage you to read more about contract testing on my blog, and to listen to my podcast where we talk about how to get started with contract testing.

When you are building software, quality is everyone’s responsibility and everyone is working towards the same goal. When you look at it like that then interactions between integrations are the responsibility of everyone. I have often been involved in conversations where the development team building the API service have said it’s not their responsibility what happens outside of their code and vice versa. The introduction of contracts to your UI tests allows you to break down this perception and start having conversations with the API development team to speak the same language. 

For me, the biggest benefit that comes from implementing contract tests is the conversations that come out of it. Having these conversations about API design early, with clear examples, makes developing microservices and micro frontends much easier.

The post How to Simplify UI Tests with Bi-Directional Contract Testing appeared first on AI-Powered End-to-End Testing | Applitools.

The visual aspect of a website or an app is the first thing that end users will encounter when using the application. For businesses to deliver the best possible user experience, having appealing and responsive websites is an absolutely necessity.

More than ever, customers expect apps and sites to be intuitive, fast, and visually flawless. The number of screens across applications, websites, and devices is growing faster, with the cost of testing rising high. Managing visual quality effectively is now becoming a MUST.

Visual testing is the automated process of comparing the visible output of an app or website against an expected baseline image.

In its most basic form, visual testing, sometimes referred to as Visual UI testing, Visual diff testing or Snapshot testing, compares differences in a website page or device screen by looking at pixel variations. In other words, testing a web or native mobile application by looking at the fully rendered pages and screens as they appear before customers.

The Different Approaches Of Visual Testing

While visual testing has been a popular solution for validating UIs, there have been many flaws in the traditional methods of getting it done. In the past, there have been two traditional methods of visual testing: DOM Diffs and Pixel Diffs. These methods have led to an enormous amount of false positives and lack of confidence from the teams that have adopted them.

Applitools Eyes, the only visual testing solution to use Visual AI, solves for all the shortcomings of visual testing – vastly improves test creation, execution, and maintenance.

The Pixel-Matching Approach

This refers to Pixel-by-pixel comparisons, in which the testing framework will flag literally any difference it sees between two images, regardless of whether the difference is visible to the human eye, or not.

While such comparisons provide an entry-level into visual testing, it tends to be flaky and can lead to a lot of false positives, which is time-consuming.

When working with the web, you must take into consideration that things tend to render slightly different between page loads and browser updates. If the browser renders the page off by 1 pixel due to a rendering change, your text cursor is showing, or an image renders differently, your release may be blocked due to these false positives.

Here are some examples of what this approach cannot handle:

Pixel-based comparisons exhibit the following deficiencies:

• They will be considered successful ONLY if the compared image/checkpoint and the baseline image are identical, which means that every single pixel of every single component has been placed in the exact same way. 
• These types of comparisons are very sensitive, so if anything changes (the font, colors, component size) or the page is rendered differently, you will get a false positive.
• As mentioned above, these comparisons cannot handle dynamic content, shifting elements or different screen sizes, so it’s not a good approach for modern responsive websites.

Take for instance these two examples:

1. When a “-” sign used in a line of text is changed to a “+” sign, many browsers will add literally single digit pixels of padding around the line based on formatting rules. This small change will throw off your entire baseline and render the entire page a massive bug. 
2. When the version of your favorite browser updates, oftentimes the engine it uses to transform colors can improve and throw off small changes that are not even visible to the human eye into the pixels of your UI. This means that colors that have made no perceptible changes will fail visual tests.

The DOM-Based Approach

In this approach, the tool captures the DOM of the page and compares it with the DOM captured of a previous version of the page.

Comparing DOM snapshots does not mean the output in the browser is visually identical. Your browser renders the page from the HTML, CSS and JavaScript, which comprises the DOM. Identical DOM structures can have different visual outputs and different DOM outputs can render identically.

Some differences that a DOM diff misses:

•  IFrame changes but the filename stays the same
•  Broken embedded content
•  Cross-browser issues
•  Dynamic content behavior (DOM is static)

DOM comparators exhibit three clear deficiencies:

1. Code can change and yet render identically, and the DOM comparator flags a false positive.
2. Code can be identical and yet render differently, and the DOM comparator ignores the difference, leading to a false negative.
3. The impact of responsive pages on the DOM. If the viewport changes or the app is loaded on a different device, components size and location may change, this will flag another set of false positives.

In short, DOM diffing ensures that the page structure remains the same from page to page. DOM comparisons on their own are insufficient for ensuring visual integrity.

A combination of Pixel and DOM diffs can mitigate some of these limitations (e.g. identify DOM differences that render identically) but are still suspect to many false-positive results.

The Visual AI Approach

Modern approaches have incorporated artificial intelligence, known as Visual AI, to view as a human eye would and avoid false positives.

Visual AI is a form of computer vision invented by Applitools in 2013 to help quality engineers test and monitor today’s modern apps at the speed of CI/CD. It is a combination of hundreds of AI and ML algorithms that help identify when things go wrong in your UI that actually matter. Visual AI inspects every page, screen, viewport, and browser combination for both web and native mobile apps and reports back any regression it sees. Visual AI looks at applications the same way the human eye and brain do, but without tiring or making mistakes.  It helps teams greatly reduce false positives that arise from small, inconceivable differences in regressions, which has been the biggest challenge for teams adopting visual testing

Visual AI overcomes the problems of pixel and DOM for visual validations, and has 99.9999% accuracy to be used in production functional testing. Visual AI captures the screen image, breaks it into visual elements using AI, compares the visual elements with an older screen image broken into visual elements (using AI), and identifies visible differences.

Each given page renders as a visual image composed of visual elements. Visual AI treats elements as they appear:

• Text, not a collection of pixels
• Geometric elements (rectangles, circles), not a collection of pixels
• Pictures as images, not a collection of pixels

Check Entire Page With One Test

QA Engineers can’t reasonably test the hundreds of UI elements on every page of a given app, they are usually forced to test a subset of these elements, leading to a lot of production bugs due to lack of coverage.

With Visual AI, you take a screenshot and validate the entire page. This limits the tester’s reliance on DOM locators, labels, and messages. Additionally, you can test all elements rather than having to pick and choose. 

Fine Tune the Sensitivity Of Tests

Visual AI identifies the layout at multiple levels – using thousands of data points between location and spacing. Within the layout, Visual AI identifies elements algorithmically. For any checkpoint image compared against a baseline, Visual AI identifies all the layout structures and all the visual elements and can test at different levels. Visual AI can swap between validating the snapshot from exact preciseness to focusing differences in the layout, as well as differences within the content contained within the layout.

Easily Handle Dynamic Content

Visual AI can intelligently test interfaces that have dynamic content like ads, news feeds, and more with the fidelity of the human eye. No more false positives due to a banner that constantly rotates or the newest sale pop-up your team is running.

Quickly Compare Across Browsers & Devices

Visual AI also understands the context of the browser and viewport for your UI so that it can accurately test across them at scale. Visual testing tools using traditional methods will get tripped up by the small, inconsistencies in browsers and your UIs elements. Visual AI understands them and can validate across hundreds of different browser combinations in minutes.

Automate Maintenance At Scale

One of the unique and cool features of Applitools is the power of the automated maintenance capabilities that prevent the need to approve or reject the same change across different screens/devices. This significantly reduces the overhead involved with managing baselines from different browsers and device configurations.  

When it comes to reviewing your test results, this is a major step towards saving team’s and testers time, as it will help to apply the same change on a large number of tests and will identify this same change for future tests as well. Reducing the amount of time required to accomplish these tasks translates to reducing the cost of the project.

Use Cases of Visual AI

Testing eCommerce Sites

ECommerce websites and applications are some of the best candidates for visual testing, as buyers are incredibly sensitive to poor UI/UX. But previously, eCommerce sites had too many moving parts to be practically tested by visual testing tools that use DOM Diffs or Pixel Diffs. Items that are constantly changing and going in and out of stock, sales that happening all the time, and the growth of personalization in digital commerce has made it impossible to validate with AI. Too many things get flagged on each change!

Using Visual AI, tests can omit entire sections of the UI from tripping up tests, validate only layouts, or dynamically assert changing data. 

Testing Dashboards 

Dashboards can be incredibly difficult to test via traditional methods due to the large amount of customized data that can change in real-time.

Visual AI can help not only visually test around these dynamic regions of heavy data, but it can actually replace many of the repeated and customized assertions used on dashboards with a single line of code. 

Let’s take the example of a simple bank dashboard below.

It has hundreds of different assertions, like the Name, Total Balance, Recent Transactions, Amount Due, and more. With visual AI, you can assign profiles to full-page screenshots meaning that the entire UI of “Jack Gomez’s” bank dashboard can be tested via a single assertion. 

Testing Components Across Browsers

Design Systems are a common way to have design and development collaborate on building frontends in a fast, consistent manner. Design Systems output components, which are reusable pieces of UI, like a date-picker or a form entry, that can be mixed and matched together to build application screens and interfaces.

Visual AI can test these components across hundreds of different browsers and mobile devices in just seconds, making sure that they are visibly correct on any size screen. 

Testing PDF Documents 

PDFs are still a staple of many business and legal transactions between businesses of all sizes. Many PDFs get generated automatically and need to be manually tested for accuracy and correctness. Visual AI can scan through hundreds of pages of PDFs in just seconds making sure that they are pixel-perfect. 

Conclusion

DOM-based tools don’t make visual evaluations. DOM-based tools identify DOM differences. These differences may or may not have visual implications. DOM-based tools result in false positives – differences that don’t matter but require human judgment to render a decision that the difference is unimportant. They also result in false negatives, which means they will pass something that is visually different.

Pixel-based tools don’t make evaluations, either. Pixel based tools highlight pixel differences. They are liable to report false positives due to pixel differences on a page. In some cases, all the pixels shift due to an enlarged element at the beginning – pixel technology cannot distinguish the elements as elements, this means pixel technology cannot see the forest from the trees.

Automated Visual Testing powered by Visual AI, can successfully work with the challenges of Digital Transformation and CI-CD by driving higher testing coverage while at the same time helping teams increase their release velocity and improve visual quality.

Be mindful when selecting the right tool for your team and/or project, and always take into consideration:

• Organizational maturity and opportunities for test tool support 
• Appropriate objectives for test tool support 
• Analyze tool information against objectives and project constraints 
• Estimate the cost-benefit ratio based on a solid business case 
• Identify compatibility of the tool with the current system under test components

The post The Benefits of Visual AI over Pixel-Matching & DOM-Based Visual Testing Solutions appeared first on AI-Powered End-to-End Testing | Applitools.

What are the skills you need to begin your career as a software tester? Learn how to get started in software testing and what’s really important.

What is a Software Tester?

A software tester is someone who analyzes software to uncover bugs and defects, or unexpected functionality, in the software’s output.

They help ensure that the software’s functionality and performance are operating as they should. They use various testing techniques and test automation tools to achieve this.

Why Should You Become a Software Tester?

Getting your first role as a software tester can be extremely difficult. Unfortunately for people new to the field, a lot of employers only want people with experience.

But in fact, a lot of employers actually want someone with the right skills so that the new person can (eventually) add value to the team. 

By learning the new skills you need to become a software tester, not only do you make sure you have something to offer your employer, but you also get an idea of what it will be like to work as a software tester.

After all, why would you need to learn these skills if you weren’t actually going to have to apply these skills in the workplace.

But which skills do you need to learn?

What are the Skills Required to Get Started as a Software Tester?

Here are some key skills to start off with when you want to get your first role as a software tester. 

We need to break this down into two categories: see diagram below.

It’s a question of which skills are employers looking for and which skills will actually be useful once you start working as a software tester.

It may be surprising that there can be a difference between the what employers look for and what will be most useful for you. The difference can arise because the person writing the job ad might not always know what skills a software tester needs. 

What Employers Often Look for When Hiring Testers

1. Experience with specific tools

You may have seen job ads looking for experience with specific tools such as API testing tools and test case management tools.

2. Experience in writing test cases

For many people in the software testing industry, writing and executing test cases is the only way you can “properly” test software.

Most Useful Skills for a Software Tester

1. Can Give Effective Actionable Feedback
2. People Skills
3. Able to assess risk
4. Not afraid to ask questions

What Employers Often Look for that is also Useful as a Software Tester

1. Test Automation Skills
2. How To Work in an Agile environment
3. Experience in Writing Bug Reports

How Can You Learn the Skills that are Most Useful to You as a Software Tester?

In this article, I will focus on learning the skills that you will find most useful as a software tester.

How to Start Learning Test Automation Skills

It can be overwhelming to try and figure out where to start. Analysis paralysis can cause you to keep on doing research into what you should learn, instead of actually spending time upskilling.

If you are currently on a project, I suggest you start by learning a programming language that is used on your project and if there is a test automation framework already in place, then that test automation framework.

If you are not currently on a project, I suggest you choose one of the following:

1. Selenium Webdriver in Java
2. Cypress in Javascript
3. API Testing in Python 
4. Playwright in Javascript

Aside from the fact there are some great courses on Test Automation University on these frameworks, these are very popular test automation frameworks with extensive documentation. Therefore, when you get stuck, you have plenty of resources online to enable you to get past any obstacles you may face.

More importantly, it’s important to know which tests you should automate.

Angie Jones has done an excellent talk on the topic.

Long story short, it depends.

According to her talk, there are some key factors you should consider including:

1. What is the risk?
   probability (how often would customers come across this?) vs impact (if broken, how would this affect the customers?)
2. Value
   distinctness (does this test give us new information?) vs induction to action (how quickly would this failure be fixed?)
3. Cost-Efficiency
   quickness (how quickly can we write this test?) vs ease (how easy would it be to write a test for this?)
4. History
   similar to weak areas (have there been a lot of failures in similar areas?) vs frequency of breaks (how often would this test have failed if it was already in place?)

How to Work in an Agile Environment

Realistically, this is pretty hard to define since there are many different flavors of Agile and judging by a job advertisement, it’ll be hard to tell exactly how any particular company interprets how one should work in an Agile environment.

Often, when people refer to “Agile” they are referring to a Scrum team. But be wary that people may claim to work in an Agile environment just because they have daily standups.

It helps to ask questions, so you can better understand their expectations here.   

Getting Experience in Writing Bug Reports

A large part of being a software tester is writing clear, compelling, reproducible bug reports. 

I’ve written a blog post on how to write a bug report. 

A few keys things to highlight when it comes to writing bug reports:

A bug report is a form of written communication – to write a good bug report you need to have good writing skills.

If you want to improve your bug reports you should look into:

• Your use of words. 
• Formatting, so that things are more clear. For example: I like to use bold formatting for subheadings in my bug descriptions. Bullet points can also be useful.  
• Being clear so that it’s easy for the reader to understand what you are trying to say. Don’t expect the reader of your bug to always read it thoroughly before deciding what to do with it (whether that is assign it, start fixing it, reject it), to be safe assume your reader will scan it.

If you want to find a place to practice writing bug reports – you can sign up to Crowdsourced Testing sites like uTest.

If you are already on a project, ask for feedback on your bug reports – see what your team thinks could be done better. 

While it can be scary to ask for feedback, as you don’t know what you’ll find out (about yourself and others’ perception of you), it helps to know how your work is being received by your team. 

Giving Effective Actionable Feedback as a Software Tester

While it’s important to ask for feedback so you can improve, it’s also very important that you can give effective feedback. 

Many people associate Toastmasters with giving speeches. However, improving your public speaking skills isn’t the only way you can benefit from going to Toastmasters. Toastmasters is a great way to learn how to give effective, actionable feedback. At Toastmasters, you give people feedback on their speeches but then you, as an evaluator, also get feedback on how you delivered your feedback. 

You’ll get feedback on many aspects of your feedback including:

• The structure
• Your tone
• The clarity

As an evaluator, your goal isn’t just to help the speaker improve, but also to help lift them up (you don’t want to drag someone down with your feedback). If we were to tie this back to the workplace, that is often the desired result of feedback as well.

Developing Your People Skills

It’ll be easier to do your job as a software tester if you have strong people skills.

While this isn’t an exhaustive list of things that will help you with your people skills, here are a few things I have found useful. 

• Listen to what people are saying and then acknowledge that you had heard them. 
• Be curious – get to know people. 
• Ask those closest to you how they perceive you. This will give you an idea of how you come across to others. Make sure to ask people who you know will be open and honest with you. 

Understanding how to Assess Risk in Testing

If you were to take a risk-based approach to testing, it means you would prioritize testing areas and scenarios that are highly probable and/or have a high impact.

To do this, you need to know where the risks lie.

This often comes with experience.

Some questions to ask yourself when considering risk include:

• Which use cases are the most common?
• Is there any payment involved in any use cases?
• Which use cases would customers expect to work? (Where would they be not-so-forgiving if it did not work?)
• Which areas have had problems in the past?

Don’t be Afraid to Ask Questions

You’ll learn a lot by asking questions. You’ll also be surprised by how many people can shy away from asking questions out of fear of looking stupid.

Karen N. Johnson has done an excellent talk on The Art of Asking Questions.

Here are a few things to keep in mind when asking questions:

• Timing matters. When and where you ask a question can impact what you get for a response. Keep this in mind before asking a question.
• Be careful of relying on only one source for information. You may find that you asked the wrong person.

In Summary

There are a lot of ways in which you can upskill to become an effective software tester. While there are a few skills that are difficult to gain without experience as a tester, there are still plenty you can start learning as you look for your first role (or even after).

The post The Top Skills You Need To Become A Software Tester In 2022 appeared first on AI-Powered End-to-End Testing | Applitools.

In this guide, we’ll explore Visual Artificial Intelligence (AI) and what it means. Read on to learn what Visual AI is, how it’s being applied today, and why it’s critical across a range of industries – and in particular for software development and testing.

From the moment we open our eyes, humans are highly visual creatures. The visual data we process today increasingly comes in digital form. Whether via a desktop, a laptop, or a smartphone, most people and businesses rely on having an incredible amount of computing power available to them and the ability to display any of millions of applications that are easy to use.

The modern digital world we live in, with so much visual data to process, would not be possible without Artificial Intelligence to help us. Visual AI is the ability for computer vision to see images in the same way a human would. As digital media becomes more and more visual, the power of AI to help us understand and process images at a massive scale has become increasingly critical.

What is AI? Background on Artificial Intelligence and Machine Learning

Artificial Intelligence refers to a computer or machine that can understand its environment and make choices to maximize its chance of achieving a goal. As a concept, AI has been with us for a long time, with our modern understanding informed by stories such as Mary Shelley’s Frankenstein and the science fiction writers of the early 20^th century. Many of the modern mathematical underpinnings of AI were advanced by English mathematician Alan Turing over 70 years ago.

Since Turing’s day, our understanding of AI has improved. However, even more crucially, the computational power available to the world has skyrocketed. AI is able to easily handle tasks today that were once only theoretical, including natural language processing (NLP), optical character recognition (OCR), and computer vision.

What is Visual Artificial Intelligence (Visual AI)?

Visual AI is the application of Artificial Intelligence to what humans see, meaning that it enables a computer to understand what is visible and make choices based on this visual understanding.

In other words, Visual AI lets computers see the world just as a human does, and make decisions and recommendations accordingly. It essentially gives software a pair of eyes and the ability to perceive the world with them.

As an example, seeing “just as a human does” means going beyond simply comparing the digital pixels in two images. This “pixel comparison” kind of analysis frequently uncovers slight “differences” that are in fact invisible – and often of no interest – to a genuine human observer. Visual AI is smart enough to understand how and when what it perceives is relevant for humans, and to make decisions accordingly.

How is Visual AI Used Today?

Visual AI is already in widespread use today, and has the potential to dramatically impact a number of markets and industries. If you’ve ever logged into your phone with Apple’s Face ID, let Google Photos automatically label your pictures, or bought a candy bar at a cashierless store like Amazon Go, you’ve engaged with Visual AI. 

Technologies like self-driving cars, medical image analysis, advanced image editing capabilities (from Photoshop tools to TikTok filters) and visual testing of software to prevent bugs are all enabled by advances in Visual AI.

How Does Visual AI Help?

One of the most powerful use cases for AI today is to complete tasks that would be repetitive or mundane for humans to do. Humans are prone to miss small details when working on repetitive tasks, whereas AI can repeatedly spot even minute changes or issues without loss of accuracy. Any issues found can then either be handled by the AI, or flagged and sent to a human for evaluation if necessary. This has the dual benefit of improving the efficiency of simple tasks and freeing up humans for more complex or creative goals.

Visual AI, then, can help humans with visual inspection of images. While there are many potential applications of Visual AI, the ability to automatically spot changes or issues without human intervention is significant. 

Cameras at Amazon Go can watch a vegetable shelf and understand both the type and the quantity of items taken by a customer. When monitoring a production line for defects, Visual AI can not only spot potential defects but understand whether they are dangerous or trivial. Similarly, Visual AI can observe the user interface of software applications to not only notice when changes are made in a frequently updated application, but also to understand when they will negatively impact the customer experience.

How Does Visual AI Help in Software Development and Testing Today?

Traditional testing methods for software testing often require a lot of manual testing. Even at organizations with sophisticated automated testing practices, validating the complete digital experience – requiring functional testing, visual testing and cross browser testing – has long been difficult to achieve with automation. 

Without an effective way to validate the whole page, Automation Engineers are stuck writing cumbersome locators and complicated assertions for every element under test. Even after that’s done, Quality Engineers and other software testers must spend a lot of time squinting at their screens, trying to ensure that no bugs were introduced in the latest release. This has to be done for every platform, every browser, and sometimes every single device their customers use. 

At the same time, software development is growing more complex. Applications have more pages to evaluate and increasingly faster – even continuous – releases that need testing. This can result in tens or even hundreds of thousands of potential screens to test (see below). Traditional testing, which scales linearly with the resources allocated to it, simply cannot scale to meet this demand. Organizations relying on traditional methods are forced to either slow down releases or reduce their test coverage.

At Applitools, we believe AI can transform the way software is developed and tested today. That’s why we invented Visual AI for software testing. We’ve trained our AI on over a billion images and use numerous machine learning and AI algorithms to deliver 99.9999% accuracy. Using our Visual AI, you can achieve automated testing that scales with you, no matter how many pages or browsers you need to test. 

That means Automation Engineers can quickly take snapshots that Visual AI can analyze rather than writing endless assertions. It means manual testers will only need to evaluate the issues Visual AI presents to them rather than hunt down every edge and corner case. Most importantly, it means organizations can release better quality software far faster than they could without it.

How Visual AI Enables Cross Browser/Cross Device Testing

Additionally, due to the high level of accuracy, and efficient validation of the entire screen, Visual AI opens the door to simplifying and accelerating the challenges of cross browser and cross device testing. Leveraging an approach for ‘rendering’ rather than ‘executing’ across all the device/browser combinations, teams can get test results 18.2x faster using the Applitools Ultrafast Test Cloud than traditional execution grids or device farms.

How Will Visual AI Advance in the Future?

As computing power increases and algorithms are refined, the impact of Artificial Intelligence, and Visual AI in particular, will only continue to grow.

In the world of software testing, we’re excited to use Visual AI to move past simply improving automated testing – we are paving the way towards autonomous testing. For this vision (no pun intended), we have been repeatedly recognized as an industry leader by the industry and our customers.

Keep Reading: More about Visual AI and Visual Testing

What is Visual Testing (blog)

The Path to Autonomous Testing (video)

What is Applitools Visual AI (learn)

Why Visual AI Beats Pixel and DOM Diffs for Web App Testing (article)

How AI Can Help Address Modern Software Testing (blog)

The Impact of Visual AI on Test Automation (report)

How Visual AI Accelerates Release Velocity (blog)

Modern Functional Test Automation Through Visual AI (free course)

Computer Vision defined (Wikipedia)

The post What is Visual AI? appeared first on AI-Powered End-to-End Testing | Applitools.

Learn why unit testing isn’t only for developers, the importance of unit testing to software testers and quality engineers, and how you can improve your skills by building better unit tests.

The responsibility for product quality frequently falls on software testers. Yet, software testers are often divorced or even excluded from conversations around the cheapest and easiest way to inject quality into the product and the entire software development lifecycle, right from the beginning: unit testing. In this article, we’ll explore why it’s important for software testers to be able to speak clearly about unit tests and how this can help deliver better quality.

Why Unit Tests Are Important

Unit tests form the solid base of the testing pyramid. They are the cheapest kinds of tests to run, and can be run frequently throughout the deployment pipeline. Unit tests allow us to find errors the soonest, and to fix them before they bubble up in other, more expensive kinds of testing like functional or UI tests, which take much longer to complete and run than unit tests.

Unit Testing Frameworks

Most developers know how to write unit tests in the language in which they develop, and most languages have several libraries to choose from, depending on the type and complexity of testing. For example, Python has pytest, pyunit, unittest(inspired by Java’s JUnit), Nose2, and hypothesis (for property tests, a non-example based type of unit test). These are just some of the choices available, and every language has a number of possible unit testing frameworks to choose from.

You don’t need to know everything about a unit testing library, or even how to write unit tests, to get value from understanding the basics of the unit testing framework. A lot of value can be gained from knowing what framework is being used, and what kinds of assertions can be made within the framework. Also, does the framework support table tests or property-style tests? Understanding what is supported can help you better understand what aspects of your test design might be best handled in the unit-testing phase. 

Unit Testing Is the Developers Job

Yes, developers typically write unit tests. However, they are largely responsible for writing these tests to ensure that the code works – most developer tests are likely to cover happy-path and obvious negative cases. They may not think to write tests for edge or corner cases, as they are working to meet deadlines for code delivery. This is where software testers with unit test knowledge can help to make the unit tests more robust, and perhaps decrease testing that might otherwise be done at integration or functional levels.

The first step, if you are unfamiliar with the code, is to request a walkthrough of the unit tests. Understanding what developers have done and what they are testing will help you to make recommendations about what other tests might be included. Remember, adding tests here is the cheapest and fastest place to do it, especially if there are tests you want run quickly on every code change that a developer makes. 

If you are familiar with the codebase and version control systems, then you can also look for the unit tests in the code. These are often stored in a test directory, and typically named so it is easy to identify what is being tested. Quality teams can be coached to review unit tests, and compare those with their test plans. Once coached, teams can make recommendations to developers to improve unit tests and make test suites more robust. Some team members may even expand their skills by adding tests and making pull requests/merge requests for unit tests. There are many ways to participate in making unit tests more effective, involving writing no code or writing a lot of code; it’s up to you to decide what most benefits you and your team. 

But What if There Are No Unit Tests?

If you are responsible for software quality and you discover that your team or company is not doing unit testing, this can be both painful, but also a great opportunity for growth. The first step in having the conversations around developing unit tests can revolve around the efficiency, efficacy, and speed of unit tests. The next step is building awareness and fluency about quality and testing as a part of development, which is a difficult task to tackle alone, and it may not work without buy-in from key people. However, if you can get understanding and buy-in on the importance of building testing and testability into the product starting with unit-tests as the foundation, further discussions about code quality can be opened up.

Better Quality is the Goal

At the end of every day, every member of the team should be responsible for quality. However, that responsibility rests with different people in different organizations, and often with the person who has the word “quality” in their title is the person who is ultimately held responsible in the end. If you are responsible for quality, understanding the basics of how unit tests work in your code base will help you to have better discussions with developers about how to improve software quality in the fastest, cheapest way possible – directly from the code.

The post Why Should Software Testers Understand Unit Testing? appeared first on AI-Powered End-to-End Testing | Applitools.