{"id":12878,"date":"2018-07-23T16:21:34","date_gmt":"2018-07-23T16:21:34","guid":{"rendered":"https:\/\/app14743.cloudwayssites.com\/blog\/?p=2878"},"modified":"2023-01-26T12:16:20","modified_gmt":"2023-01-26T20:16:20","slug":"why-screenshot-image-comparison-tools-fail","status":"publish","type":"post","link":"https:\/\/app14743.cloudwayssites.com\/blog\/why-screenshot-image-comparison-tools-fail\/","title":{"rendered":"Why Screenshot Image Comparison Tools Fail With Dynamic Visual Content"},"content":{"rendered":"

The QA team is the last frontier before your customers get their hands on your latest product version. Assuming your testing team has already completed smoke testing, integration tests, functional tests, end-to-end tests, acceptance tests and performance tests (not necessarily in that order) there’s still one piece of the puzzle: visual UI testing of your app front end.<\/span><\/p>\n

The way many companies visually test their applications using image comparison tools. Some of these are free and some are paid. But in either case, these tools are not always reliable and often deliver false-positives (false alarms) that only frustrate both frontend developers and testers, and ultimately making you ship software late.<\/span><\/p>\n

How Does Image Comparison Work?<\/strong><\/h2>\n

<\/p>\n

Image comparison is simply pixel-by-pixel comparison, nothing more. Each and every pixel color in the base image is compared to the equivalent pixel in the checkpoint image. If all pixel colors match both images are identical. The chance of this technique working flawlessly is very slim. For this reason, image comparison tools give the user parameters to adjust such as pixel\/color tolerance, which is the number of pixels that are allowed to differ between the two images. <\/span><\/p>\n

But that\u2019s not enough.<\/span><\/p>\n

There are many ways in which images might seem to be identical but the pixels don\u2019t match. Why does image comparison fail? There are many reasons:<\/span>
<\/span><\/p>\n

Mouse Hovers<\/strong><\/h2>\n

Even if the mouse points is not grabbed in a screenshot it may actually be hovering over an element that can be affected by it. Because changing the appearance of an element is trivial to do in CSS<\/a>, JavaScript<\/a>, JQuery<\/a>, React<\/a>, Angular<\/a>, Vue<\/a>, and other front-end developer technologies, this pattern is quite common in web apps. <\/span><\/p>\n

Here\u2019s an example from a website you might be familiar with. In the top screenshot, the mouse isn\u2019t hovering over the Google Search<\/strong> button; in the bottom screenshot, it is. See the difference?<\/span><\/p>\n

\"Why<\/p>\n

\"Why<\/p>\n

Input Caret<\/strong><\/h2>\n

If you have a text element that is in focus it may have a blinking caret in it. Suppose when you did your baseline test, you captured a screenshot of the cursor. Then when you do your test, your screenshot captures the page at the instant when the cursor is invisible. A simplistic pixel comparison will fail, even though the UI doesn\u2019t have a bug.<\/span><\/p>\n

Here\u2019s an example of this. In the top screenshot, the cursor is visible; in the bottom, it isn\u2019t.<\/span><\/p>\n

\"Why<\/p>\n

\"Why<\/p>\n

Font Anti-Aliasing<\/strong><\/h2>\n

Each operating system renders fonts differently to make the font smoother and easier to read. It\u2019s not just a Mac-versus-Windows thing; even an operating system upgrade can change font smoothing<\/a>. <\/span>Additionally, your testers might change a font smoothing setting without realizing how it will impact visual testing.<\/span><\/p>\n

The point is, any of these can make simplistic pixel comparisons fail, even when there isn\u2019t a visual UI bug.<\/span><\/p>\n

For this reason, it\u2019s crucial for testers to understand how to control font smoothing on their test platforms. On macOS, the font smoothing setting is in <\/span>System Preferences > General: <\/b><\/p>\n

\"Font<\/p>\n

In Windows, ClearType<\/a> is the technology used for font smoothing and can be turned on and off in the settings<\/a>. The exact method varies by Windows version.<\/p>\n

\"ClearType<\/p>\n

If pixel-based bitmap comparison is part of your test automation, you\u2019ll want to ensure a consistent font smoothing setting programmatically. Here\u2019s how to do that on macOS<\/a>, and how to do it in Windows<\/a> (if these techniques seem complicated — that\u2019s one of many reasons customers use Applitools. More on that below). <\/span><\/p>\n

User Interface Themes<\/strong><\/h2>\n

If your testing team switches the user interface visual theme on a test computer, that will change image screenshots and cause simple pixel comparison tools to fail. These changes are very easy to make on both macOS<\/a> and Windows<\/a>, meaning <\/span>it\u2019s all too easy for testers to introduce changes that will visual UI tests.<\/span><\/p>\n

Different Monitors<\/strong><\/h2>\n

Higher resolution screens like the Apple Retina Display<\/a> <\/span>have more pixel density, so much so that pixels are not visible to the human eye. A screenshot taken on retina screen and a monitor with less pixel density will fail a simple pixel comparison, even if, to the human eye, there\u2019s no apparent difference.<\/span><\/p>\n

Here\u2019s an example. The image on the top is a non-retina display on an iPhone 3GS. The image on the bottom is from an iPhone 4\u2019s retina display.<\/span><\/p>\n

\"200px
200px Non-Retina Display<\/em><\/figcaption><\/figure>\n

 <\/p>\n

\"200px
200px Retina Display<\/em><\/figcaption><\/figure>\n

Neither one of these has a visual bug, but it\u2019s obvious that they wouldn\u2019t pass a simple pixel comparison, even if the battery levels were the same.<\/span><\/p>\n

Graphics Cards<\/strong><\/h2>\n

When different computers have different graphics cards, they can produce different screenshots. Or fuzzier screenshots<\/a>. And even if you don\u2019t change your graphics card, an older card that is about to fail can introduce screen glitches<\/a> that lead to<\/span> different screenshots over time.<\/span><\/p>\n

Visually Testing Dynamic Content<\/strong><\/h2>\n

In the real world your apps are not static and content changes constantly, using image comparison will not cut it to validate all pages in your app since you can’t automatically create a baseline for your tests. It takes more than pixel-by-pixel comparison to achieve that and manual testing is not a scalable solution either.<\/span><\/p>\n

A good example of this is Gannett<\/a>, the company behind USA Today and dozens of other news publications<\/a>. They <\/span>can\u2019t use pixel comparison, because their screenshots constantly change as breaking news stories are added to their websites. For USA Today and other new sites, there is no constant but change itself.<\/span><\/p>\n

AI To The Rescue<\/strong><\/h2>\n

By emulating the human eye and brain, Applitools AI powered Visual UI Testing technology only reports differences that are perceptible to users and reliably ignore invisible rendering, size and position differences. The algorithms can instantly validate entire application pages, detect layout issues, and process the most complex and dynamic pages.<\/span><\/p>\n

When writing an automated visual UI test, sometimes we will want to change the comparison method between our test and its baseline image<\/a>, <\/span>especially when dealing with applications that contain dynamic content. <\/span><\/p>\n

Applitools Eyes can test the UI in four different comparison levels<\/a>: <\/span><\/p>\n

\"Applitools<\/p>\n

Here is how those comparison levels vary:<\/p>\n

    \n
  1. Exact<\/strong>: pixel-to-pixel comparison. If you really want to do exact pixel comparisons, you can still use Applitools.<\/li>\n
  2. Strict<\/strong>: compares everything including content (text), fonts, layout, colors and position of each of the elements. Strict knows to ignore rendering changes that are not visible to people. Strict is the recommended match level when running regression tests on the same browser\/OS.<\/li>\n
  3. Content<\/strong>: works in a similar way to Strict except for the fact that it ignores colors.<\/li>\n
  4. Layout<\/strong>: compares the layouts (i.e. structure) of the baseline and actual images. It validates the alignment and relative position of all elements on the page, such as buttons, menus, text areas, paragraphs, images, and columns. It ignores the content, color and other style changes between the pages. Gannett uses layout mode extensively<\/a> to test the layout of USA Today and other news sites.<\/li>\n<\/ol>\n

    Layout: Changing the Game<\/strong><\/h2>\n

    Applitools\u2019 layout mode algorithm reverse-engineers the structure of the compared images by applying complex image processing algorithms resulting with the logical page structure of each image that includes all the components of the page. Then, it proceeds to map each component with its counterpart in the other image which allows detection of missing or new elements as well as misplaced ones. All such elements are considered different and are highlighted as such in the comparison result.<\/span><\/p>\n

    Layout mode will have greater benefits than other match levels in certain scenarios, which most notably include: dynamic content, language localization, responsive design, and cross-environment testing (including operating systems, browsers, devices, and form factors) since it is a more flexible approach of visual testing will have greater benefits.<\/span><\/p>\n

    But don’t take my word for it. In the following example (video below) I run a search in Google Images for two different keywords, grab screenshots of both results using Applitools Eyes, and show how Exact match level (which is doing pixel-by-pixel comparison) fails and Layout match level pass. Applitools Eyes is able to reverse engineer the image layout and the relation between all the UI elements on the page while ignoring the actual content of those images and text.<\/span><\/p>\n