Almost everyone in software testing knows about Test Automation University (TAU). Powered by Applitools, TAU is one of the best resources for building up your testing and automation skills. It offers dozens of courses from the world’s leading instructors all available for free. It’s awesome!

Last month, I wrote an article listing the 10 most popular TAU courses ranked by completions over the past year. However, there are several other excellent courses in our catalog that also deserve attention. In this article, I want to highlight a small selection of those courses that I think are fantastic. These courses are listed in no particular order, and of course, all our other courses are excellent as well. Let’s dive in!

#1: Playwright with JavaScript

First on this list is Playwright with JavaScript by Ixchel Meza. Playwright is a hot, new web testing framework from Microsoft. It’s a modern framework with automatic waiting, built-in tracing, and cross-browser support. Plus, it’s fast – really, really fast for UI testing! It’s quickly gaining industry traction. In this course, Ixchel teaches you how to get a Playwright project up and running in JavaScript. The distinction of saying “in JavaScript” is important because Playwright also supports Java, C#, and Python.

#2: Intro to Testing Machine Learning Models

Next up is one of my personal favorites: Intro to Testing Machine Learning Models by Carlos Kidman. Machine learning (ML), artificial intelligence (AI), “algorithms” – whatever you call it – is indispensable to software products today. Just like every business plan in 1999 needed a website, every new app these days seems to have an ML component. But how do we test ML models when they seem so opaque and, at times, questionable? In this course, Carlos breaks down where testers fit in this brave new world of ML and all the techniques for properly testing models.

#3: UI Automation with WebdriverIO v7

UI Automation with WebdriverIO v7 by Julia Pottinger is actually a special course: it’s the first course that we “refreshed” for TAU. Julia’s original course covered a now-older version of WebdriverIO, so she developed a new edition to keep up with the project. WebdriverIO is a Node.js-based browser automation tool that can use either the WebDriver Protocol or the Chrome DevTools protocol. Alongside Selenium WebDriver, Cypress, and Playwright, it’s one of the most popular web testing tools available! Julia covers all the ins and outs of WebDriverIO v7 in this course.

#4: The Basics of Visual Testing

Visual testing is indispensable for testing any app with a user interface (UI) these days – which is basically every app. In The Basics of Visual Testing, Matt Jasaitis teaches how to incorporate visual testing techniques as part of a well-balanced test automation strategy. You’ll build a Selenium Java project together with Applitools Eyes chapter by chapter, learning the best practices for visual testing along the way. Visual testing will help you catch all sorts of bugs like overlapping text, missing elements, and skewed layouts that traditional automation struggles to find.

#5: Introduction to Observability for Test Automation

“Observability” means the ability to understand what is happening in a software system based on telemetry being reported in real time. Just like any other software system, test automation and build pipelines have telemetry worth analyzing for improvements. In her course Introduction to Observability for Test Automation, Abby Bangser teaches how to expose metrics, logs, traces, and events of an automated test suite written in Java running through GitHub Actions. She provides excellent guidance on what is worth capturing and how to capture it.

#6: Introduction to Blockchain Testing

A blockchain is a distributed ledger that links records or “blocks” together in sequence using cryptographic signatures. One of the most popular blockchain applications is cryptocurrencies like Bitcoin and Ethereum. Blockchains also serve as a cornerstone for web3 and web5. Although blockchain technologies are controversial due to environmental concerns from high energy consumption and due to perceived scams around cryptocurrencies and non-fungible tokens (NFTs), they are nevertheless ingrained in our tech landscape. In Introduction to Blockchain Testing, Rafaela Azevedo shows you how to get your hands dirty with blockchains by setting up an Ethereum blockchain on your local machine, writing your first contract with Solidity, and finally testing your contract.

#7: Testing from the Inside: Unit Testing Edition

The final course to round off this list is Testing from the Inside: Unit Testing Edition by Tariq King. Unit testing is such a vital part of software development. It’s one of the first lines of defense in catching bugs in code. In this course, Tariq takes a unique approach to unit testing: thinking “inside the box” of the software product under test to explore the actual code and its data. Tariq covers foundational principles as well as practical techniques like parameterizing, mocking, and spying.

What other courses are available?

The post 7 Awesome Test Automation University Courses Not To Miss appeared first on AI-Powered End-to-End Testing | Applitools.

Test Automation University (also called “TAU”) is one of the best online platforms for learning testing and automation skills. TAU offers dozens of courses from the world’s leading instructors, and everything is available for free. The platform is proudly powered by Applitools. As of November 2022, nearly 140,000 students have signed up! TAU has become an invaluable part of the testing community at large. Personally, I know many software teams who use TAU courses as part of their internal onboarding and mentorship programs.

So, which TAU courses are currently the most popular? In this list, we’ll count down the top 10 most popular courses, ranked by the total number of course completions over the past year. Let’s go!

#10: Selenium WebDriver with Java

Starting off the list at #10 is Selenium WebDriver with Java by none other than Angie Jones. Even with the rise of alternatives like Cypress and Playwright, Selenium WebDriver continues to be one of the most popular tools for browser automation, and Java continues to be one of its most popular programming languages. Selenium WebDriver with Java could almost be considered the “default” choice for Web UI test automation.

In this course, Angie digs deep into the WebDriver API, teaching everything from the basics to advanced techniques. It’s a great course for building a firm foundation in automation with Selenium WebDriver.

#9: Python Programming

#9 on our list is one of our programming courses: Python Programming by Jess Ingrassellino. Python is hot right now. On whatever ranking, index, or article you find these days for the “most popular programming languages,” Python is right at the top of the list – often vying for the top spot with JavaScript. Python is also quite a popular language for test automation, with excellent frameworks like pytest, libraries like requests, and bindings for browser automation tools like Selenium WebDriver and Playwright.

In this course, Dr. Jess teaches programming in Python. This isn’t a test automation course – it’s a coding course that anyone could take. She covers both structured programming and object-oriented principles from the ground up. After two hours, you’ll be ready to start coding your own projects!

#8: API Test Automation with Postman

The #8 spot belongs to API Test Automation with Postman by Beth Marshall. In recent years, Postman has become the go-to tool for building and testing APIs. You could almost think of it as an IDE for APIs. Many test teams use Postman to automate their API test suites.

Beth walks through everything you need to know about automating API tests with Postman in this course. She covers basic features, mocks, monitors, workspaces, and more. Definitely take this course if you want to take your API testing skills to the next level!

#7: Introduction to Cypress

Lucky #7 is Introduction to Cypress by Gil Tayar. Cypress is one of the most popular web testing frameworks these days, even rivaling Selenium WebDriver. With its concise syntax, rich debugging features, and JavaScript-native approach, it’s become the darling end-to-end test framework for frontend developers.

It’s no surprise that Gil’s Cypress course would be in the top ten. In this course, Gil teaches how to set up and run tests in Cypress from scratch. He covers both the Cypress app and the CLI, and he even covers how to do visual testing with Cypress.

#6: Exploring Service APIs through Test Automation

The sixth most popular TAU course is Exploring Service APIs through Test Automation by Amber Race. API testing is just as important as UI testing, and this course is a great way to start learning what it’s all about. In fact, this is a great course to take before API Test Automation with Postman.

This course was actually the second course we launched on TAU. It’s almost as old as TAU itself! In it, Amber shows how to explore APIs first and then test them using the POISED strategy.

#5: IntelliJ for Test Automation Engineers

Coming in at #5 is IntelliJ for Test Automation Engineers by Corina Pip. Java is one of the most popular languages for test automation, and IntelliJ is arguably the best and most popular Java IDE on the market today. Whether you build frontend apps, backend services, or test automation, you need proper development tools to get the job done.

Corina is a Java pro. In this course, she teaches how to maximize the value you get out of IntelliJ – and specifically for test automation. She walks through all those complicated menus and options you may have ignored otherwise to help you become a highly efficient engineer.

#4: Java Programming

Our list is winding down! At #4, we have Java Programming by Angie Jones. For the third time, a Java-based course appears on this list. That’s no surprise, as we’ve said before that Java remains a dominant programming language for test automation.

Like the Python Programming course at spot #9, Angie’s course is a programming course: it teaches the fundamentals of the Java language. Angie covers everything from “Hello World” to exceptions, polymorphism, and the Collections Framework. Clocking in at just under six hours, this is also one of the most comprehensive courses in the TAU catalog. Angie is also an official Java Champion, so you know this course is top-notch.

#3: Introduction to JavaScript

It’s time for the top three! The bronze medal goes to Introduction to JavaScript by Mark Thompson. JavaScript is the language of the Web, so it should be no surprise that it is also a top language for test automation. Popular test frameworks like Cypress, Playwright, and Jest all use JavaScript.

This is the third programming course TAU offers, and also the top one in this ranking! In this course, Mark provides a very accessible onramp to start programming in JavaScript. He covers the rock-solid basics: variables, conditionals, loops, functions, and classes. These concepts apply to all other programming languages, too, so it’s a great course for anyone who is new to coding.

#2: Web Element Locator Strategies

I’m partial to the course in second place – Web Element Locator Strategies by me, Andrew Knight! This was the first course I developed for TAU, long before I ever joined Applitools.

In whatever test framework or language you use for UI-based test automation, you need to use locators to find elements on the page. Locators can use IDs, CSS selectors, or XPaths to uniquely identify elements. This course teaches all the tips and tricks to write locators for any page, including the tricky stuff!

#1: Setting a Foundation for Successful Test Automation

It should come as no surprise that the #1 course on TAU in terms of course completions is Setting a Foundation for Successful Test Automation by Angie Jones. This course was the very first course published to TAU, and it is the first course in almost all the Learning Paths.

Before starting any test automation project, you must set clear goals with a robust strategy that meets your business objectives. Testing strategies must be comprehensive – they include culture, tooling, scaling, and longevity. While test tools and frameworks will come and go, common-sense planning will always be needed. Angie’s course is a timeless classic for teams striving for success with test automation.

What can we learn from these trends?

What other courses are popular?

The post The Top 10 Test Automation University Courses appeared first on AI-Powered End-to-End Testing | Applitools.

What does it take to make a difference in the lives of 75,000 people?

Applitools has reached 75,000 students enrolled in Test Automation University, a global online platform led by Angie Jones that provides free courses on things test automation. Today, more engineers understand how to create, manage, and maintain automated tests.

What Engineers Have Learned on TAU

Engineers have learned how to automate UI, mobile, and API tests. They have learned to write tests in specific languages, including Java, JavaScript, Python, Ruby, and C#. They have applied tests through a range of frameworks including Selenium, Cypress, WebdriverIO, TestCafe, Appium, and Jest.

75,000 engineers would exceed the size of some 19,000 cities and towns in the United States. They work at large, established companies and growing startups. They work on every continent with the possible exception of Antarctica.

What makes Test Automation University possible? Contributors, who create all the coursework.

Thank You, Instructors

As of this writing, Test Automation University consists of 54 courses taught by 39 different instructors. Each instructor has contributed knowledge and expertise. You can find the list of authors on the Test Automation University home page.

Here are the instructors of the most recently added courses to TAU.

Author	Course	Details	Chapters
Corina Pip	JUnit 5	Learn to execute and verify your automated tests with JUnit 5	17
Matt Chiang	WinAppDriver	Learn how to automate Windows desktop testing with WinAppDriver	10
Marie Drake	Test Automation for Accessibility	Learn the fundamentals of automated accessibility testing	8
Lewis Prescott	API Testing In JavaScript	Learn how to mock and test APIs in JavaScript	5
Andrew Knight	Introduction to pytest	Learn how to automate tests using pytest	10
Moataz Nabil	E2E Web Testing with TestCafe	Learn how to automate end-to-end testing with TestCafe	15
Aparna Gopalakrishnan	Continuous Integration with Jenkins	Learn how to use Jenkins for Continuous Integration	5
Moataz Nabil	Android Test Automation with Espresso	Learn how to automate Android tests with Espresso	11
Mark Thompson	Introduction to JavaScript	Learn how to program in JavaScript	6
Dmitri Harding	Introduction to NightwatchJS	Learn to automate web UI tests with NightwatchJS	8
Rafaela Azevedo	Contract Tests with Pact	Learn how to implement contract tests using Pact	8
Simon Berner	Source Control for Test Automation with Git	Learn the basics of source control using Git	8
Paul Merrill	Robot Framework	Learn to use Robot Framework for robotic process automation (RPA)	7
Brendan Connolly	Introduction to Nunit	Learn to execute and verify your auotmated tests with nUnit	8
Gaurav Singh	Automated Visual Testing with Python	Learn how to automate visual testing in Python with Applitools	11

Thank You, Students

As engineers and thinkers, the students continue to expand their knowledge through TAU coursework.

Each course contains quizzes of several questions per chapter. Each student who completes a course gets credit for questions answered correctly. Students who have completed the most courses and answered the most questions successfully make up the TAU 100.

Some of the students who lead on the TAU 100 include:

Student		Credits	Rank
	Osanda Nimalarathna Founder @MaxSoft Ambalangoda Sri Lanka	44,300	Griffin
	Patrick Döring Sr. QA Engineer @Pro7 Munich Germany	44,300	Griffin
	Darshit Shah Sr. QA Engineer @N/A Ahmedabad India	40,250	Griffin
	Adha Hrustic QA Engineer @Klika Bosnia and Herzegovina	39,575	Griffin
	Ho Sang Principal Technical Test Engineer @N/A Kuala Lumpur Malaysia	38,325	Griffin
	Gopi Srinivasan Senior SDET Lead @Trimble Inc Chennai India	38,075	Griffin
	Ivo Dimitrov Sr. QA Engineer @IPD Sofia Bulgaria	37,875	Griffin
	Malith Karunaratne Technical Specialist – QE @Pearson Lanka Sri Lanka	36,400	Griffin
	Stéphane Colson Freelancer @Testing IT Lyon France	35,325	Griffin
	Tania Pilichou Sr. QA Engineer @Workable Athens Greece	35,025	Griffin

Join the 75K!

Get inspired by the engineers around the world who are learning new test automation skills through Test Automation University.

Through the courses on TAU, you’ll not only learn how to automate tests, but more importantly, you’ll learn to eliminate redundant tests, add automation into your continuous integration processes, and make your testing an integral part of your build and delivery processes.

Learn a new language. Pick up a new testing framework. Know how to automate tests for each part of your development process – from unit and API tests through user interface, on-device, and end-to-end tests.

No matter what you learn, you will become more valuable to your team and company with your skills on how to improve quality through automation.

The post Test Automation University is now 75,000 students strong appeared first on AI-Powered End-to-End Testing | Applitools.

I remember the feeling when I submitted my entry for Holiday Shopping Hackathon. Sure, there is always a bit of relief once you wrap something up. But mostly I was just proud that I managed to handle each task from Hackathon instructions. 

I wasn’t eyeing any of the prices nor expected to ever hear back from judges. I simply saw Applitools Holiday Shopping Hackathon as a learning opportunity and went for it. This sense of pride was coming from having my learning mission accomplished. 

But Hackathon ended up being much more for me, besides getting fine JavaScript and Cypress practice and getting introduced to this amazing visual testing tool, now I also have lifetime bragging rights and a bit of self esteem boost to keep up with my new tech goals.

Why did I need new tech goals in the first place?

I have been in managerial and leadership roles in IT industry for over 12 years. Even though I hold Master’s degree in Computer Science and my first role after graduation was as Software Engineer, 12 years is a lot of time to not be actively developing software.   

All this time I have been making constant efforts to build and enhance an array of soft skills, to accumulate industry specific knowledge (for at least 5 industries) and to be able to actively participate in tech discussions. It turned out that this was not enough, at least not for the tech part, as I started getting feedback that I am “behind with the tech side”. 

One thing was clear, I needed to craft a plan which will turn the things around.

The things I tried

I know by now that the best way to learn something is to start practicing it actively and to combine theory with practice. My work is not leaving me with much room for something like getting hands on experience with new cool frameworks. So all the learning and practicing had to happen in the evenings and over the weekend.

I subscribed to several podcasts and blogs and I handpicked some development courses which seemed related to technologies currently used with my team. I was investing a lot of time and was absolutely sure that there is no significant improvement. Courses I choose were either focused on very basic examples or were too demanding in terms of mandatory coursework. Even if I managed to stretch my time and cover self-assignments, whatever I learned there would fade away shortly because I was not actively using it. 

Then came Hackathon

The hackathon just sounded like a good idea. The instructions were very specific; it was very clear what was expected from participants. Timeframe for submission was very generous – since learning about Hackathon, I had several weeks to complete my submission, so I didn’t need to pause on rest of my life and get behind with the sleep (something I have been associating with Hackathons until now).

For Cypress part I relied on Introduction to Cypress course from Test Automation University. Mr. Gil Tayar did a great job!

Visual testing in Applitools

I admit that I ignored the manual and relied on exploring Applitools myself. Overall, I find the app to be intuitive and easy to use. All information about test runs is very well structured and easy to navigate through. 

Multi-browser testing worked like a charm. It took me no time to set this up, and speed of multi-browser testing was more than I hoped for.

For one of Hackathon tasks, I figured out how bugs work. That was straight forward. Potential issues were very obviously highlighted. They scream action.

Another task was related to root causes. I didn’t figure this one in first attempt, but I have obviously excelled on that second try.

Visual testing before I knew Applitools

I recall scenarios where QA team on my projects was using Selenium to automate tests. Idea was to automate UI tests as well. 

There were too many visual issues which tests were not detecting. Even the issues important for the end user were being undetected by these tests. QA Engineers were explaining the causes for this, coming up with workarounds how to increase the test coverage with limited time investments. This didn’t sit that well with the client.

Conclusion

What can I say, this experience has turned me into advocate for Applitools. I see a lot of potential of this kind of testing. I recognize the benefit for the current project my team is working on. And looking back I see there were many cases over the years where it would have helped QA Engineers I have worked with. It shortens the time to set up UI tests and it probably shortens running time. Plus, it provides better coverage. 

Also, I find Test Automation University to be one of the best things that happened in testing community lately. Thank you for doing this, Applitools!

As for my personal development, Hackathon was a great boost for me. It helped me carry on with my learning trajectory. And I expect more Hackathons in my future.

Lead image by Antonis Kousoulas from Pixabay

The post More than a Hackathon to me – Ivana Dilparic appeared first on AI-Powered End-to-End Testing | Applitools.

If you started using Cypress in recent years, chances are that you heard about “app actions” being preferred over using page objects. In fact, if you google “Cypress page objects” the first article you’ll see is Gleb Bahmutov’s blog titled:

Stop using Page Objects and Start using App Actions

I see this statement confusing some people, making them think that using Page Object Model (POM) is some sort of anti-pattern in Cypress. The truth is, that using POM does no harm in Cypress and can be really helpful. But I like that Gleb has made a strong argument for other options. Because using them unleashes some Cypress superpowers. In this post, I’d like to explore and describe both of these options.

Page objects

Gil Tayar has made an excellent demonstration of how to use page objects in Cypress. If you are new to this concept or haven’t yet used page objects in Cypress, I suggest you watch that video. In fact, go watch the full course on Test Automation University. It’s free! To sum things up for this article, let me just give you a very quick TL;DR version of how to create page objects.

For demonstration, you can take a look at my Trello clone app. Make sure you check out the page-objects-app-actions branch.

Let’s say we want to work with the login/signup part of our app. Normally this is the part where many of our test efforts start. To create a page object, create a separate .js or .ts file where all your functions for a particular component will be stored.

I will create a login.ts file that will look like this:

I am creating a Login class, which contains two functions. One will open my login window, and the other clicks a link to the signup page. So basically, I’m grouping sequences of Cypress commands into these functions. These help me divide my test code into smaller chunks that I can later reuse.

Paraphrasing Gleb from Cypress again –

It’s all JavaScript, you can do whatever you want.

And he’s right. I can add actions, assertions, or pass arguments into these functions. Let’s see what these two functions do in our application (I slowed the video down a little so it’s clear to see):

In my test, I want to open the login modal window, and then click on the „Sign up here“ text. This is what our test code looks like, using functions from our page object:

Notice how our functions are chained together. This is because in each of our page object functions I’m using 

return this;

This way, each of our functions returns to the original context and will be chained off our Login class.

Now, whenever I want to create a flow that includes clicking on a “Log in” button, I can use a function from my page object. Since I have this action abstracted in page object, in theory, I can now write tests for login, signup, reset password, logout, and many other user stories.

Also, whenever something in our login flow would change, I can just edit my page object and that change translates everywhere. This helps avoid situations when I need to rewrite multiple tests because of a single change in the tested app.

Page objects are also a great way to get your app into the desired state that you want to test functionally or with visual tests.

These are some main reasons why testers abstract their UI actions into page objects. So why wouldn’t you use them in Cypress?

Why don’t just stick with page objects?

The main reason for choosing a different approach is to take advantage of how Cypress is built. Cypress runs inside the browser, which is the main difference compared to Selenium-based test automation. That means that Cypress can actually get access to a lot of what’s happening inside of our application.

For example, we can call the exact same function that gets called when we click on the “log in” button. In other words, we can open our login modal window without actually doing the click. That way, we can skip interacting with UI through our page objects, and just start our app in any state we want. This enables us to avoid doing UI actions that are not actually a part of our test.

In our app, let’s say we want to do a visual test for our signup view. In order to create such a test, we need to:

1. Click “log in” button
2. Click “sign up here” link
3. Do our visual test

Notice, that steps 1 and 2 are not actually part of what we want to achieve. We are here just for the state that we want to test visually. Furthermore, I imagine this is an example of a very simple flow. There may be some hard-to-reach places in your app that require you to do multiple steps. So how about we just:

1. Set our app to the desired state
2. Do our visual test?

This is what app actions actually do. Instead of clicking, typing, and interacting with our app, we will set it up the way we want.

Before we write our first app action, let’s examine how our app actually works.

Looking into the app

Our app is built with Vue.js. If you are not familiar with it, I suggest you check it out. It provides a great developer experience. For some reason, Vue was the easiest to learn for me as a tester out of the most popular frameworks. But if you are working with Angular or React, principles are pretty much the same.

Vue.js comes with some amazing developer tools. These enable you to change the state of your app right from devtools. Look at how I’m able to toggle the visibility of our login element with these tools:

You can see that I have an attribute called showLoginModule that can be set to true or false. With Cypress, we are going to change the state of our app in a very similar way.

To do that, we need to first expose our app to our window context. This may sound like a challenge, but in our Vue app is just done by adding the last line in this piece of code:

Whenever we now open our dev tools and type window.app into our console, we will see our attributes, app functions, and much more information about our app. In fact, if we decide to type 

window.app.showLoginModule = true 

into our console, our login modal window will open. Go ahead and try that!

Creating an app action

Now, we can do the exact same thing using Cypress, like this:

And just like that, our login modal is open! We have made our first app action. In other words, we have set our app to the desired state. Let’s go one step further and get to our signup page. Now, instead of simply setting our attribute to the desired state, we will call a function that changes the state for us.

To save us some time, I have created a helper custom command in our app, that will select one of our components:

• root
• Navbar
• Login
• board-collection
• board

These are the same components you can see in Vue.js dev tools. You can use the command simply by calling 

cy.component('Login')

This command will return our component with all its data and functions:

Notice that this component contains a logSignSwitch() function. If you look into the code, you’ll see that this is a function responsible for switching between “login” and “signup” view in our modal window.

We will now call this function from within our test using the following code:

This will now open our app in desired state, which is our signup screen.

Why bother though?

This is a legitimate question. Seems like the setup is more complicated than setting up page objects. However, one does not rule the other. You can combine both approaches and use the one that is more appropriate for your test. It is however good to design your tests in a way that will shorten them by starting it at a desired point.

App actions can also be quite fast. During my comparison of these two tests, I got to finish my tests in 0.34 seconds for page objects and 0.18 seconds for app actions. Although both are pretty fast, things can really make a difference when running thousands of tests.

App actions also skip the part where e.g. your inputs need to become interactive, our your buttons need to become clickable. There are no incomplete input fields or misclicks because we are skipping this kind of interaction altogether.

Of course, we cannot write an end-to-end test without interacting with our application directly. For these parts, abstracting common actions make absolute sense. But for the parts of our app that are already tested, we can save some time and skip a few steps.

I find app actions especially useful for setting up your app for a visual test. You may save yourself a few headaches by skipping UI interaction and jumping straight into your desired state.

If you liked this blog, be sure to check out my page at filiphric.com, I write a Cypress tip every week and do all kinds of fun stuff ?

The post Page objects vs. App actions in Cypress appeared first on AI-Powered End-to-End Testing | Applitools.

As we complete 2020, we want to share our take on the past year. We had a number of achievements in 2020. And, we celebrated a number of milestones.

Any year in review article must include the effects of the pandemic, along with the threats on social justice. We also want to give thanks to our customers for their support.

Achievements: Product Releases in 2020

Ultrafast Grid

Among our achievements in 2020, Applitools launched the production version of Ultrafast Grid and the Ultrafast Test Cloud Platform. With Ultrafast Grid, you can validate your UI across multiple desktop client operating systems, browsers, and viewport sizes using only a single test run. We take care of the validation and image management, and you don’t need to set up and manage that infrastructure.

Ultrafast Grid works so quickly because we assume your application uses a common server response for all your clients. You only need to capture one server response. Ultrafast Grid captures the DOM state each snapshot and compares that snapshot in parallel across every client/operating system/viewport combination you wish to test. A single test run means less server time. Parallel validation means less test time. Ultrafast Grid simultaneously increases your test coverage while reducing both your test run time and infrastructure requirements.

A/B Testing

We introduced a new feature to support A/B testing. As more of our customers use A/B testing to conduct live experiments on customer conversion and retention, Applitools now supports the deployment and visual validation of parallel application versions.

GitHub, Microsoft, and Slack Integrations

Applitools now integrates with Slack, adding to our range of application and collaboration integrations. Applitools can now send alerts to your engineering team members, including highlights of changes and the test runs on which they occurred.

As a company, we also announced integrations with GitHub Actions and Microsoft Visual Studio App Center.  The integrations allow developers to seamlessly add Visual AI-powered testing to every build and pull request (PR), resulting in greater UI version control and improved developer workflows. As we have seen, this integration into the software build workflow provides visual testing at code check-in time. Instead of waiting for end-to-end tests to expose rendering problems and conflicts, developers can use Applitools to validate prior to code merge.

Auto Maintenance and Smart Assist

We also introduced major improvements with Auto Maintenance and Smart Assist. With Smart Assist, we help you deploy your tests to address unique visual test challenges, such as dynamic data and graphical tests. With Auto Maintenance, you can validate an intended visual change in one page of your application and then approve that change on every other page where that change occurs. If you update your logo or your color scheme, you can validate identical changes across your entire application in one click. Smart Assist and Auto Maintenance reduce the time and effort you need to maintain your visual tests – saving hours of effort in your development and release process.

Achievements: Milestones in 2020

Applitools also achieved a number of major milestones in 2020.

1,000,000,000 Page Images Collected

We recorded one billion page images collected across our customer base. Many of our customers now include Applitools validation as part of every CICD check-in and build. You will find out more in our customer insights discussion, below. We celebrated that achievement earlier in 2020.

Test Automation University

We launched Test Automation University (TAU) as a way to help expand test knowledge among practitioners. Among our achievements in 2020, TAU now has over 50 courses to teach test techniques and programming languages. You can take any of these courses free of charge. Whether you are an experienced test programmer or just getting started, you will find a range of courses to match your interests and abilities. We introduced 19 new courses in 2020. We also saw significant numbers of new students using Test Automation University. In early 2020, we announced that we had 35,000 students taking courses. Later in the year we celebrated reaching the 50,000 user milestone. Look forward to another announcement in early 2021.

Hackathons

In 2019, Applitools launched our Visual AI Rockstar Hackathon. Hackathon participants ran a series of test cases comparing legacy locator-based functional testing with Applitools visual validation. In 2020, we shared the results of that Hackathon.  Engineers wrote tests faster, wrote test code that ran more quickly, and wrote tests that required less maintenance over time. We were able to show your achievements in 2020.

Also in 2020, we hosted a cross browser test hackathon.  Participant results demonstrated that Ultrafast Grid sets up more easily than a traditional farm of multiple browsers. The real value of Ultrafast Grid, though, comes with test maintenance as applications update over time. In November, we hosted a hackathon based on a retail shopping application. We look forward to sharing the insights from that hackathon in early 2021.

Future of Testing

Lastly in 2020 Applitools launched the Future of Testing Conference. Applitools gathered engineering luminaries across a range of industries and companies – from brand names like Microsoft and Sony to tech leaders like GoodRX and Everfi. Their stories show how companies continue to deliver quality products more quickly by using the right approaches and the right tools. Applitools has planned more Future of Testing Conferences for 2021.

Achievement: Customer Growth In 2020

Another of our achievements in 2020 involved customers. We want to thank our customers for their commitment to using Applitools in 2020. Not only did we pass the 1,000,000,000 page capture mark, but we also learned about the many exciting ways our customers are using Applitools.

During the COVID-19 coronavirus pandemic, our customers have appreciated how we have worked to ensure that they continued to get full use and value from Applitools. Though our support team worked largely from home during the year, we used tools to ensure that our customers got the support they needed to succeed with Applitools.

We continued to see our existing customers use more and more page checks over time. A number of companies run Applitools to validate code check-in on daily, and even hourly, code builds. Our customers are also using Applitools to validate component libraries they are building and modifying in React, Angular, and Vue.

We also saw a large number of companies experimenting with and adopting Cypress for development validation. Some companies used Cypress in development to complement an existing Selenium test infrastructure. Others were starting their Cypress validation in new areas or on new products.

Our World in Review – 2020

While many issues affected the world in 2020, two dominated the Applitools world.

The first issue, the COVID-19 pandemic, required our team to work from home for much of the year. Dan Levy offered his suggestions on how to work from home more efficiently.  As we continued to work remotely, we saw how the pandemic affected our team and the world around us. At this point, all of us know people who have been infected. Some in our circles have been hospitalized. And, some have died.

As a company, we are fortunate that Applitools has provided its employees with the ability to work from home. As a company, we want to thank the first responders and health care workers who cannot shelter in safety. We thank them for risking their lives to make all of us safe.

And, we also share condolences with those of you who have lost family, friends, and other loved ones in 2020.

The second issue, social justice, has continued to capture the spirit of our company and its employees. For 8 minutes 46 seconds, the world saw one human’s casual indifference while kneeling on another human’s neck. While not the only incident of 2020, the video of George Floyd’s struggle to live affected all of us. How can there be justice if our civil guardians cannot treat all of us equally? If we want a just world, we need to support those who advocate for social justice.

Applitools and its employees support creating a more just world for all. We continue to encourage our employees to support social justice movements for all. They can support Black Lives Matter, or any other organization actively combatting racism and injustice.

We know there are some who sow division for their own gain. As a company, we think we are stronger together.

Next Up – Customer Insights in 2020

In our next blog post, learn more about Applitools customers. We will share some details we learned about Applitools driving our customers’ productivity. We will be sharing more details in 2021 with a series of customer success stories. Before we release those, read our next blog post on customer insights. Learn how your peers and colleagues benefit from using our highly-accurate Visual AI infrastructure.

The post Leading With Visual AI – Applitools Achievements In 2020 appeared first on AI-Powered End-to-End Testing | Applitools.

I often get questions from those starting new test automation projects querying which programming language they should choose. I never have a cut and dry answer to this because it depends on a few factors such as which language the product features are being developed in, who will be writing the tests and which language are they most comfortable in, and how much support and tooling is available for a given language.

In this post, I’ll share which programming languages are most used for test automation as it gives some insight into industry usage. However, do not take this to mean “best programming language”, as the best language is the one that is best for your context.

The Data

You may be wondering where the data is from. Good question! More than half of the top companies in software, financial services, and healthcare verticals use Applitools for their web and mobile test automation needs. From the millions of tests that run in our cloud every week, I’ve analyzed which languages the tests were written in and aggregated the results at the team level (not the test level).

The Results

1. Java
   Java remains the most popular programming language for test automation. Java held its lead with 43% of our users opting to write their tests in this language. In last year’s review, Java was used by 44% of our customers, so a slight decline but nevertheless this language managed to keep the crown in 2020.
2. JavaScript
   Coming in as the 2nd most popular programming language for test automation is JavaScript with 35% of our users writing their tests in this language. This is a huge increase from last year where only 15% of our users tested in JS! According to StackOverflow, JavaScript is the most popular technology used by professional developers, so I expect to see an increased usage of JS for testing in the years to come.
3. C#
   With Java and JavaScript accounting for 78% of usage combined, there’s not much market share left for the other languages. So, we see quite the jump with the 3rd place language, C#, being used by 8.8% of our users. This is rather interesting because last year’s results showed 13% of our customers using C#, which means almost a third of these users have likely opted for a different language this year.
4. Python
   Right behind C# is Python, with 8% of our customers using Python as their language of choice for test automation. This is exactly the same percentage of usage we saw last year. What’s most surprising about this stat is that Python is gaining popularity year after year with professional developers and has become the fastest-growing major programming language – even edging out Java for the first time this year! Perhaps we’ll eventually see this trend in software testing as well.
5. Ruby
   Only 4.2% of our customers use Ruby for test automation. This is a stunning 40% decrease from Ruby test automation usage last year. StackOverflow shows Ruby’s popularity down to 8.9% with professional developers and it appears Ruby is even less popular in the testing space.

Remember…

While the data here doesn’t necessarily indicate which is the best programming language for test automation, it does highlight which ones are most used for testing amongst the hundreds of companies and open source projects surveyed.

The post 2020’s Most Popular Programming Languages for UI Test Automation appeared first on AI-Powered End-to-End Testing | Applitools.

Cypress is a top choice for many developers that create modern web applications written in popular JavaScript frameworks. It is a tool that greatly fits into developer workflow, but has found love among test automation engineers and QAs too.

Cypress benefits greatly from being part of the JavaScript and Node.js ecosystem. There are tons of plugins which you can just install via npm and use them within your tests. They expand functionality, add new commands, add missing features and so much more.

Installing a Cypress plugin

Most Cypress plugins can be installed in 3 steps:

1. open terminal and type `npm install <name of your plugin>`
2. in your cypress project, go to support/index.js file and import/require your plugin
3. go to plugins/index.js file and include your plugin inside module.exports function

Some of the plugins only require you to do step #2 or step #3. The installation is often very well documented by the authors, so even if you are new to Cypress, you’ll have good guidance.

There are tons of options, and it’s easy to get lost In the vast universe of Cypress plugins. In this post I’d like to share my pick for the very best.

Add drag and drop functionality

There is no .drag()  function in Cypress, but the Cypress Drag Drop plugin will add it for you. These come in super handy if you have a list that you can reorder by drag and dropping. You can simply select an element and then drag it onto another like this:

Upload files

Similarly to the previous, the Cypress File Upload plugin adds missing functionality. With cypress-file-upload, you can take any file input element and upload a file within your tests. What’s really cool about this one, is that it plays well with Cypress fixtures. Just add the name of your file that is located in the fixtures folder as an argument and voila!

Testing emails with Mailosaur

Have you ever tested “forgot password” functionality? These flows can be a real challenge since you need to access an email inbox. With Mailosaur service, you have an inbox created for you. You can access it by using this awesome Cypress Mailosaur plugin. It will allow you to search or delete your emails and so much more. When searching for a message, this plugin will return your email parsed as a JSON with data such as links, recipients or email subject.

Library of awesome commands

Cypress is all JavaScript. You can expand it any way you like, create your own logic and do whatever JavaScript enables you to do. With custom commands, you can create your own set of commands that fit in your project. Or you can choose to import an existing command set like the one offered by the Testing Library plugin.

Testing Library is an awesome project that contains utilities for better testing, suitable for many different frameworks, and it has its Cypress module too. You’ll be able to select elements based on their placeholder, label, role attribute and even more.

Docs in your terminal

The documentation of Cypress project is a masterpiece. It is basically a tutorial on how to use Cypress that will get you from A to Z. I open Cypress docs many times during the day.

With TypeScript and intellisense I can access the documentation right as I type. With the Cy Search plugin, I’m getting something little extra. I am able to search Cypress docs from the command line. I just type the keyword and pick the article I want to open. Fun fact: I helped improve this plugin ?

Choose a spec to run headlessly

While we are still in the terminal, I’ve got another plugin tip for you. If you want to run a single test in your Cypress project in headless mode, you can just type `cypress run –spec <path to your spec>` but that gets tedious when you have multiple projects or you want to run multiple files at once. This can be easily solved by the Cyrun plugin, which enables you to pick a single spec or run a whole folder in Cypress headless mode.

Skip a test

There are some tests that you may want to run only on localhost. Or only in a certain browser. Or only in headless mode.

Rather than adding conditional statements that may pollute your test code, you can use the Cypress Skip Test plugin. It allows you to run a test or set of tests based on various conditions. You can choose to skip a test or decide to only run it if a condition is met. Out of the box you can decide based on stuff like operating system, browser or url, but you can basically choose any condition you like.

Bonus plugin: Come to the dark side

Big fan of dark mode? There’s one for Cypress too, Cypress Dark! It even has a spooky Halloween theme that laughs at you when your test fails. How rude! ?

There are of course a ton of great plugins and I suggest you explore them. You‘ll definitely find the one that helps you write your tests faster or makes the process more comfortable. If you liked this article, come visit my blog where I share Cypress tips on a weekly basis.

Cover photo by James Orr on Unsplash

The post 8 Cypress Plugins You Should Know appeared first on AI-Powered End-to-End Testing | Applitools.

Coding recipes using the newest automation testing tool

There’s a new kid on the block in the world of automation testing tools….Playwright!

Created by Microsoft, Playwright is an open source browser automation framework that allows JavaScript engineers to test their web applications on Chromium, Webkit, and Firefox browsers.

While Playwright is still very new, there’s quite a buzz around the tool. So, I decided to try it out for myself.

The Playwright documentation contains a ton of information about how to get started as well as details on their APIs. So, I won’t rehash all of that here. Instead, I will automate real scenarios from the Automation Bookstore application, which will demonstrate examples of using the Playwright APIs together.

Launching an Application

First things first…launching the browser and navigating to the application’s url.

The first step to do this is to declare which browser engines you’d like to run against. Again, Playwright allows you to run your tests against Chromium, Firefox, and Webkit.

I’ll only run against Chromium for now.

Sidenote: By default, Playwright runs in headless mode. If you’d like to actually see the browser as it’s executing the test, set headless to false in the launch call:

Next, I declare variables for the browser and the page objects.

An important thing to note about Playwright, is that all of their APIs are asynchronous, so async/await is needed to call them. So, I utilize this pattern to launch the browser and initial the page object.

Now that I have the page object, I can go to my application using page.goto()

Adding an Assertion Library

A key thing to note about Playwright, is like many other automated testing tools, it is designed to automate browser interaction, but you must use an assertion tool for your verifications. I’ll use Mocha for this example.

I’ll also add a suite (using ‘describe’) and move the Playwright calls inside of there into before/after functions.

Accessing Elements

When trying a new tool, I like to start with a very simple example. Basically, the “Hello World” of web test automation is verifying the title on the page. So, let’s start there and make the first test!

Playwright offers many ways to access elements including the typical ones of CSS and Xpath selectors.

When inspecting the DOM of this application, the element that displays the title has an id of ‘page-title’, and the text I want to verify is the inner text of this element.

So, I’ll use page.innerText and pass in the CSS selector for this id.

And voila, just like that, I have my very first test!

To run a Playwright test, type npm test from your terminal.

Entering Text

Now that I’m up and running, I’d like to do something a little more interesting, such as search for a book.

To do so, I need to enter text into the Filter Books field (which has an id of ‘searchBar’).

To enter text, Playwright provides the fill() API, which accepts a selector as well as the text to enter.

Waiting for Elements

Playwright does a great job of waiting for the elements to be ready before attempting to interact with them, so I don’t have to worry about that. However, sometimes explicitly waiting is still needed.

 For example, when I enter text into the field, Playwright will wait for the field since that’s the element we’re interacting with, but the verification needs to occur on the search results.

There is a slight delay between the time that the text is entered and when the results are shown, so my script must account for that.

Fortunately, in the latest release of Playwright (1.4), the waitForSelector API was introduced. I can use this to wait until hidden books become present in the DOM.

Getting List of Elements

Now that I have the search results, I want to verify that there is only one book returned. To get a list of the elements, I use the $$ API on the page object to query for all elements matching a given selector.

Now, that I have this list of elements, I can verify that it only contains one book.

Accessing Children Elements

In addition to verifying the quantity, I also need to verify that the title is as expected. Since I have a list of all of the visible books, I don’t know exactly what the ID will be for the book at runtime, so I’d prefer to just get the child element from the visible book that holds the title. This is an h2 element, as seen on line 4.

To access descendant elements in Playwright, you can use Clauses. Clauses are selectors that are separated by >>, where each clause is a selector that is relative to the one before it.

So, in my case, where I’d like to get h2 that is a child of this particular li, I can do so with ‘li:not(.ui-screen-hidden) >> h2’

And now the second test is done! On line 5 is where the parent selector is defined, and line 8 is where I expand to use a clause to target a child node.

This particular scenario used a list that only contains one element, so I added one more scenario that demonstrates how to work with a list of multiple elements.

Visual Testing

That last scenario got a bit long, and honestly, I’m only verifying a fraction of what I should be verifying. For more thorough tests that not only verify the count and title, but also verify everything else on the page in addition to making sure it’s displayed correctly, I’ll use visual testing.

Note that Playwright offers image and video capture, but it’s more meant as a visual logging mechanism, and can’t be used in the assertions themselves. So, I’m going to use a proper visual testing assertion library, Applitools.

After installing Applitools, I specify the Applitools classes required on line 3.

I declare and initialize the ‘eyes’ object (on line 3), which is the API that does the visual testing.

Then redo the last test using visual testing.

Cross Platform Tests

While Playwright provides support for Chromium, Webkit, and Firefox, there is no built-in support for IE.

When integrating Playwright with Applitools’ Ultrafast Grid, I can now get further coverage for my tests!

As opposed to using the Applitools Classic Runner as I have above, I’m going to modify it to use the Visual Grid Runner. On line 5, notice I use the number 10. This indicates the number of tests I’d like to run in parallel…making my run even faster!

Now for the good part! I can specify all of the browsers, devices, and viewports I’d like to run against – giving me the ultimate cross-platform coverage for my tests. Not only verifying them functionally, but also visually as well.

No changes are needed to the tests themselves, they will automatically run across all of the specified configurations.

Play with Playwright Yourself

Reading blog posts helped me understand what Playwright is but when I played around with it to explore realistic scenarios is when I truly got a feel for the tool! I recommend you do the same.

You can use the code examples from this blog post as well as Playwright’s documentation to get started. Good luck!

The post Playing with Playwright appeared first on AI-Powered End-to-End Testing | Applitools.

A data-centric view into the most popular tools for automated testing

Asking “what’s the best test framework?” is a sure way to get into a heated debate. Everyone has their favorite automation tool and wants to let the world know that it’s the best!

Instead of giving you yet another opinion on the matter, I am going to provide the statistics, because after all, numbers don’t lie.

What our data is saying

Many companies are using Applitools for the functional and visual testing of their web and mobile applications. With millions of tests running in our cloud every week, we’ve analyzed how widely used the most-talked about automation testing tools really are.

Most popular web test automation framework

Selenium WebDriver

There’s been a lot of discussion about the future of Selenium WebDriver when there are so many new kids on the block in terms of framework choices. Is Selenium WebDriver losing momentum?

According to our data, only slightly. However, make no mistake, Selenium Webdriver is by far, still the most widely used web automation tool with 75% of our users executing their tests with this framework.

Besides being the tried and true web automation framework for over a decade, Selenium WebDriver also supports all major browsers (Chrome, IE, Edge, Firefox, and Safari) and boasts support for Java, Python, C#, Ruby, JavaScript, and Kotlin whereas most other web automation tools only provide support for a limited number of browsers and a single programming language.

Given this, it’s expected that Selenium WebDriver will generally have more users, as it has a much larger target audience.

So to be fair, let’s look at the JavaScript usage, since most of the newer web automation frameworks are for that demographic.

JavaScript Automation Testing

If we only look at the JavaScript world, we see a much different story.

Cypress

Leading the pack is relative newcomer, Cypress, with just shy of half (47%) of our JavaScript users choosing this tool for their web testing. This is quite impressive, considering up until a couple of months ago (February 2020), Cypress’ browser support was limited to Chrome only.

WebdriverIO

WebdriverIO, which is an independent JavaScript implementation of the WebDriver API, comes in second place with 25% of our JS users choosing this framework for their test automation needs.

WebDriverJS

Selenium WebDriver’s official JavaScript implementation, WebdriverJS, is close behind with 23% of JavaScript tests being executed with this tool.

TestCafe

After the Big Three of JavaScript web automation tools, comes newcomer, TestCafe, with 5%.

Honorable Mention

Playwright

The youngest web automation framework in the race is Playwright by Microsoft. Playwright has only been available for a few months (May 2020) but we’re beginning to see usage. This is definitely one to keep your eye on.

Puppeteer

While Puppeteer is not on our list, we do realize that teams use this framework for their automation needs. However, from our observation, many are utilizing this for automating browsers (i.e. generating PDFs, crawling sites, etc) as opposed to test automation.

So, what’s the most popular test automation tool for web testing?

1. Selenium WebDriver

   Although technically a web automation framework (not a testing framework), Selenium WebDriver still reigns supreme. With support for a plurality of browsers and languages, WebDriver is by far the most popular choice overall and has seen a 9% increase this year.

2. Cypress

   Cypress rules the JavaScript web test automation world, and comes in second in overall web automation tools. Yet, it is a far way off from attracting the wide audience that Selenium WebDriver does as a whole. However, its growth is quite impressive, with an 41% increase so far this year.

3. WebDriverIO
   WebDriverIO is the third most popular web automation tool amongst our users, and saw an increase of 10% this year.

To hear more about the state of web automation tools, their similarities and differences, join the Future of Testing 2020 virtual event on September 17, where a panel of experts from Selenium, Cypress, Playwright, and Applitools will discuss the future of test frameworks.

For More Information

Applitools works with all popular test automation frameworks. We allow you to add accurate visual validation to your existing automated tests with ease. Check out some of these Applitools tutorials.

• Selenium Java Tutorial
• Selenium JavaScript Tutorial
• Selenium Python Tutorial
• Cypress Tutorial
• WebdriverIO 5 Tutorial
• Testcafé Tutorial
• Sign up for a free Applitools account
• Request an Applitools demo

The post 2020 Most Popular Front End Automation Testing Tools appeared first on AI-Powered End-to-End Testing | Applitools.

I took Nyran Moodie’s course on Test Automation University: Automated Visual Testing with WebdriverIO. If you want the explicit link to the course, here it is: https://testautomationu.applitools.com/automated-visual-testing-javascript-webdriverio/index.html

Course Introduction

If you use WebdriverIO regularly, and you are unfamiliar with the basics of using Applitools for automated visual testing, you will appreciate this course. Nyran focuses you on how to add visual test automation with Applitools to your WebdriverIO tests.

Nyran expects you to know WebdriverIO. If you want to learn WebdriverIO, check out Julia Pottinger’s excellent course – UI Automation with WebdriverIO on Test Automation University.

Nyran uses JavaScript for his examples and Visual Studio Code as his IDE. In case you didn’t know, you can use Applitools with a range of test languages. No matter what your test language of choice or IDE, I think you will find Nyran’s code and coding approach fairly intuitive.

Course Structure

Nyran breaks the course into eight main chapters. He gives this description in the course overview:

• 1 – I am going to start by giving you an introduction to visual testing.
• 2 – We are going to look at how we can get our environment set up to start using Applitools eyes.
• 3 – We are going to create and run our first visual test using Applitools
• 4 – I want to introduce you to the different match levels that Applitools has and the concept of viewport sizes
• 5 – I will be talking about checkpoints.
• 6 – I will be looking at how we can organize our visual tests using batches
• 7 – We will look at how we can analyze our test results using the Test Manager.
• 8 – We will be looking at the integration and collaborations that Applitools provides.

Each of these chapters provides a methodical approach to getting going with Applitools. I’ll give a quick overview of each.

Chapter 1 – Why Visual Testing

If you read this review or take the course, you know why visual testing. Your UI and end-to-end tests result in rendered output. You can write all the functional tests that grab locators, enter data, effect action, and cause the appropriate output locators to have the appropriate values. But, until you actually look at the result, you cannot tell if the input and output conform to design and usability expectations.

Nyran did not explain the most frequently-experienced reason for visual testing – unintended consequences of code changes over time. Our experience shows us that most expected application changes get tested, but unintended changes cause problems.

Chapter 2 – Getting Started With Applitools

Nyran does a nice job explaining how to get started. You need an Applitools API key, which you can get from the Applitools console. Nyran explains why you set up a local environment variable for your API key (so you do not need to include your API key in your test code directly). He also points to the github repo he uses for all the examples in the course.

Chapter 3 – Create And Run A Visual Test with Applitools

Chapter 3 involves the first coding examples for setting up Applitools. With a simple:

npm install  @applitools/eyes.webdriverio

You get the the node instructions for installing the Applitools Eyes to your WebdriverIO setup. After this you can install the Applitools Eyes service to your tests. He shows code examples of what test code looks like when calling Applitools:

Once he walks you through a test page example and fills in the tests, he gets code that looks like a full test.

Finally, he shows you the Applitools UI and how it highlights differences found during a test. To do this, he shows a test site with the ability to show different content, and he shows how Applitools highlights the differences.

Nyran makes it clear that Applitools can find and highlight all the visual changes on a comparison page (the checkpoint) versus the original capture (the baseline). And, he explains that Applitools lets you accept the checkpoint as the new baseline, or reject it.

Chapter 4 – Viewports and Match Levels

Nyran breaks chapter 4 into two sections. In Section 4.1, Nyran goes through the idea of viewport sizes. If you build a responsive app, you want to run your app for 4K down to mobile device sizes. How do you validate the different device size views? Applitools makes it easy to add a JavaScript file that specifies all the viewport sizes you want to validate. Applitools runs all the captures for you.

Next, Nyran writes about match levels. Applitools default comparison level, called “strict”, compares visually noticeable changes between a checkpoint and baseline. Strict uses Applitools Visual AI to break items on a page into elements that it then compares.

However, sometimes strict is too strict. In “content” match level, Applitools checks text and structures but ignores color variations. This match level helps when you apply a global change to color and want to ensure that – color outstanding – no other changes have taken place. And, if they have, you want those changes highlighted quickly.

In “layout” match level, Nyran shows, Applitools lets you validate pages that have dynamic content sharing a common structure. For example, you might have a retail shoppping page that shows user-specific and hot items updating from test run to test run. Or, you have a news site that updates top stories regularly. Layout match level pays attention to the layout structure (relationship of sections and text sizes) without comparing the specific contents of any set of elements within that structure.

Chapter 5 – Checkpoints

Nyran spends this chapter reviewing the ways Applitools lets  you make visual captures. First, ou can capture the visible page – the current viisble screen. Alternatively, you can capture the full page. Applitools runs through the app horizontally and vertically and stitches the images into a single screen baseline and checkpoint.

Next, you can capture individual web elements. Finally, you can capture frames on a page. Your flexibility in setting checkpoints gives you plenty of power to control the details of your inspection at different times in your testing life cycle.

Chapter 6 – Batches

Batches provide a way of organizing common tests inside the Applitools Test Manager. Nyran explains how to code tests into batches. He also shows how batched tests get grouped inside the Test Manager. When you use batches, your test results become easier to interpret.

Nyran implies two things about batches. First, grouping tests into batches make your testing much easier to understand. Second, If you want the benefit of batches, you need to code those batch definitions yourself.

Chapter 7 – Using the Test Manager

The Test Manager is the Applitools service user interface. Your user login provides you an API key, and tests run under your API key collect in your view in Test Manager.

Nyran shows you the basics, as well as some cool tricks in the Test Manager. He shows you how to compare differences between the checkpoint and the baseline. You can approve expected changes and update the baseline to the checkpoint. Or, you can reject changes and have them routed back to development as bugs.

Next, Nyran shows you how tests get grouped and how to use statistics. He also shows you how to override the existing match level on all or part of a test. Finally, he shows you Automated Test Maintenance.

Automated Test Maintenance gives you huge powers of scale for validating changes to your application. When you find a change in one checkpoint and approve it as a valid change, Applitools finds all other similar changes and gives you the power to approve those identical changes at the same time. For example – you change your menu bar at the top of your app, and the change affects 145 pages. Following your validation of a change on one page, Applitools asks you if you want to approve the other 144 pages with the identical change. That’s powerful.

Chapter 8 – Integrations

In the last chapter, Nyran shows how Applitools integrates with other tools in development – especially your CICD workflows.

Nyran shows the range of Applitools test framework and language SDKs that work on web and mobile test frameworks. Applitools lets you capture screenshots to use Visual AI if you use an unsupported framework. Applitools even lets you compare PDFs to ensure that your document generators behave correctly even as you update your application.

Next, you see how to link Applitools with Jira. You can link issues found in Applitools with incidents in Jira. Also, you can link GitHub and Applitools to tie your image comparisons to the source code GitHub pull requests. Finally, you see the standard approach to having Applitools link with your favorite CICD workflow manager.

Conclusion

Nyran wrote a nice course on how to use Applitools for automated visual testing. He makes clear that he used WebdriverIO and JavaScript because he knew these well. However, he knows the range of choices available to you.

Nyran really doesn’t cover:

• Setting up tests in WebdriverIO,
• Approaches for running and managing tests and test results
• Managing WebdriverIO in your CICD workflow

You can find other courses to address these issues.I enjoyed taking Nyran’s course. Having taken other JavaScript testing courses, I think Nyran provides good examples of Applitools. He wants you to know how to get the most out of Applitools when you use WebdriveriO.

As always, I include my certificate of completion from Test Automation University:

For More Information

• Visual Testing – FAQ
• Visual Testing: The Missing Piece of the Puzzle – with Gil Tayar
• 20 Highly Qualified Test Automation Superstars
• How Do You Simplify End-To-End Test Maintenance?
• Five Data-Driven Reasons To Add Visual AI To Your End-To-End Tests
• Sign up for a free Applitools account
• Request an Applitools demo

The post Review – Automated Visual Testing With WebdriverIO appeared first on AI-Powered End-to-End Testing | Applitools.

How do you test a design system? You got here because you either have a design system or know you need one. But, the key is knowing how to test its behavior.

Marie Drake, Principal Test Automation Engineer at News UK, presented her webinar, “Roadmap To Testing A Design System”, where she discussed this topic in some detail.

Marie is many things. In addition to her work at News UK, she is a Cypress Ambassador and organizer of the Cypress UK community group. If you want to know more about using Cypress, she’s a great speaker. In addition, she blogs about testing and tech at her own blog, mariedrake.com.

This post summarizes her webinar and highlights some of the key points.

Who Is News UK?

News UK is the UK subsidiary of News Corp, the large global publishing and media company. Marie’s team supports the sites that develop and deliver online versions of The Sun, The Times, and The Sunday Times. They also run the Wireless media site. Marie supports the various development teams that deliver news and information that change regularly.

Why A Design System At News UK?

Think of a design system as building blocks. A design system provides a repository for design components used to construct your web application. Or, more precisely, applications. By using a design system, you can eliminate redundant work across different parts of your web application.

Marie gave the example of “share bars” at News UK. Share bars let you share content to social networks like Twitter, Facebook, Instagram, and WhatsApp. You likely have seen share bars on blogs or media pages. Inside News UK, design groups had coded their own share bars. They found19 different share bars in use across different parts of the News UK business.

The implication of lots of redundant code written by different people involves the cost of maintenance. Sure, having 19 different teams write 19 different parts of the app sounds like great division of labor. But, when you get 19 different sound bars – how do you maintain them? How do you choose one for your next part of your web business? What happens when you decide to resize the share bars across your site?

Fast Coding Does Not Equal Agility

Marie showed an even more problematic example of a web business, Fabulous, that wanted to change their brand color from #E665BF to #EA6596. When engineers looked at the potentially impacted code and the areas requiring post-change validation, they estimated the change would take six months. Half a year for a color change?

The coding effort at Fabulous involved two code bases. First, they had the website that needed to be updated. Second, they had the tests that needed to be updated to match the new site. A large part of the test change – even with no functional or other visual change – just required code inspection to ensure that the desired branding color change had been applied as expected across the entire site.

Marie’s seven-plus years in software quality led her to understand that raw coding speed rarely correlated with agility. Here, “agility” means something different from “agile.” In software, agility comes from the ability to make quick changes and have confidence on both their impact on intended behavior and avoiding unexpected changes. While many software developers can write code quickly, few write thoughtfully in ways that make code maintainable – especially across the entire site.

Benefits of the News UK Design System

In describing the design system deployed by News UK, Marie quickly pointed out its benefits.

• Cost efficient. Once you set up a design system, you have standardized building blocks for building your site. If you can use the design system to customize, your teams can consume the building blocks instead of rewriting from scratch. And, you reduce software maintenance costs.
• Reusable. A good design system allows you to re-use code.
• Speed To Market. As mentioned in the section on agility, the design system reduces the amount of code you need to write from scratch. It also reduces the amount of code you will manually change as you make updates.
• Scalable. A good design system lets multiple users access the system – making the developers much more efficient.
• Standard Way Of Working. With a design system, you standardize the process of writing code. You can help new people get up to speed on existing code and simplify the code maintenance process.
• Consistency. In the end, you can look to the design system to ensure consistent behavior (visual, functional) from your applications.

Marie showed a loop of the design system at News UK. The components get developed and maintained in Storybook. Developers can grab elements and add them into applications being built. The playground feature in Storybook makes it easy for developers and designers to play with Storybook components to mock up the functioning web application before it gets built.

As Marie pointed out, consistency in the components simplifies both development and testing.

Testing A Design System – Requirements

If a design system should make code easier to create and maintain, how do you test a design system?

Marie started outlining the testing requirements developed by News UK.

• Test different components easily. Expect the system to mature and develop over time. Some components will be entirely visual, and some may include audio. Make sure all this works.
• Test cross-browser. News UK needed this capability as they knew their content got consumed on mobile devices and a range of platforms.
• Visual Tests – Write visual tests with less maintenance in mind to reduce impact on testing workflow and speed the process of testing small changes that touch lots of components.
• Deliver a high-performing build pipeline – build plus test concludes within 10 minutes
• Integrate design review earlier in the process to improve collaboration, find misunderstandings and differences between design and development early in the process.
• Test for accessibility on both the component and site level for all users.
• Catch functional issues early.
• Have all tests written before deploying a feature. There are 2 full-time QA engineers on the Product Platforms team, so they need to share QA responsibility with developers.

Testing A Design System – Strategy

From here, Marie outlined the strategy to run tests of the design system.

• First, unit testing. Developers must write unit tests for each component and component modification.
• Second, snapshot testing. Capture snapshots and validate the status of component groups.
• Third, component testing. All components need to be validated for functionality, visual behavior, and accessibility.
• Fourth, front-end testing.  Make sure the site behaves correctly with updated components. Validate for functionality, visual behavior, and accessibility.
• Fifth – cross-browser tests. Ensure there are no unexpected differences on customer platforms.

Testing A Design System – Challenges

Marie described some of the challenges with different test approaches.

Purely functional tests can include lots of code. Marie’s pseudocode shows this problem. The more comprehensive your functional tests, the more code that exists in those tests. Assertion code – the code used to inspect the DOM for visual elements – becomes a burden for your team going forward.

Visual testing serves a strategic function, except that most visual testing approaches suffer from inaccuracy. Marie showed an example of a “spot-the-differences” game, which highlighted the challenges of a manual visual test. Then, she showed pixel differences, which she found become problematic on cross-browser tests. From a user’s perspective, the pages looked fine. The pixel differences highlighted differences that, after time-consuming inspection, her team judged as inconsequential pixel variations.

Another visual testing inaccuracy Marie described involved visual testing of dynamic data. On news sites, content changes frequently as news stories get updated. When the data changes, does the visual test fail?

Marie and her team had chosen to use available open-source tools for visual testing. Marie showed some of the visual errors that got through her testing system. These had passed functional tests but weren’t caught visually.  

So, Marie and her team discovered that their existing tests let visual bugs through. They knew they needed to solve their visual testing problem.

Choosing New Tools

Marie’s team looked at three potential solutions to their visual testing problem: Screener, Applitools and Happo.  After putting all three through their paces, the team settled on Applitools for accuracy. Being way more accurate helped Marie write up the use case for News UK to purchase a commercial tool instead of adopting an open-source solution.

The team also looked at UI testing tools. They looked at Puppeteer, Selenium, and Cypress for driving web application behavior. As a team, they chose Cypress. They could have used any of these tools with Applitools. Marie’s team chose Cypress because its developer-friendly user experience made it easy for developers to write and maintain tests.  

The final test suite included:

• Jest for managing test flow
• Cypress for running tests
• Applitools for visual tests
• Google Lighthouse and Cypress AXE for accessibility testing
• WAVE for testing Doc accessibility
• Safari Voiceover for sightless test validation

Using Applitools

Next, Marie shared the approach her team used for deploying Applitools.

Prior to using any part of Applitools, the team needed to deploy an API key. This key, found on the Applitools console, permits users the access the Applitools API. Once read into the test environment, the key grants the tests access to the Applitools service.

The team needed to add the Eyes code to Storybook for component tests and to Cypress for the site-level tests.

Component Tests

Next, Marie demonstrated the code for validating the Storybook components. The tests involved cycling through Storybook and having each component captured by Applitools. Individual component tests either matches in Applitools, or showed differences. The test team would walk through the inspected differences to either approve the changes and update the baseline image with the new capture, or rejected the change and send the component back to the developers.

Cypress Tests

Similar to the component tests, the Cypress tests integrated Applitools into captures of the built site using the new components. Again, Applitols compared each capture against the existing baseline to find differences.

For Marie’s team, one great part about using Applitools involved the built-in cross-browser testing using the Applitools Ultrafast Grid. Simply by specifying a file of target browsers and viewport sizes, Applitools could automatically capture images for the targets separately and compare each against its baseline.

Auto Maintenance

Marie talked about one of the great features in Applitools – Auto Maintenance. When Applitools discovers a visual difference, it looks for similar differences on other pages captured during a test run. When an Applitools user finds a visual difference and approves it, Auto Maintenance lists the other captures for which the identical difference exists. The Applitools user can then batch-approve the identical changes found elsewhere. A single user, in a single step, can approve site-wide menu, logo, color, and other common changes all at once.

Handling Dynamic Changes

Another benefit of Applitools involves pages with dynamic data. In addition to the example of news items updating regularly, Marie showed an example of the new Internet radio service offered by News UK. The player page can sometimes show different progress in a progress bar during different captures, depending on data being read when taking a screen capture.

Applitools has a layout mode that ensures that all the items exist in a layout, including the relative location of the items, but layout mode ignores content changes within the layout.

Accessibility Tests

Next, Marie talked about accessibility tests.

Marie demonstrated component accessibility testing with Cypress AXE. She showed that, once integrated with Cypress, AXE can cycle through components. Unfortunately, AXE and other automated tests uncover only about 20% of accessibility tests.

Lighthouse and other tests get run manually to validate the rest of visual accessiblity.

She also showed the Safari screen reader accessibility testing.

Workflow Integration

Marie then described workflow, and how the workflow integration mattered to the Product Platforms team.

She made the team’s first point – quality is everyone’s responsibility. For the product platforms team, the two quality engineers serve as leads on approaches and best practices. Developers must deliver quality in the design system.

To accentuate this point, she explained that the team had developed pull request guidelines. Check-ins and pull requests required documentation and testing checklist of unit, component, and page-level tests. Everyone agreed to this level of work for a pull request.

Next, Marie showed the workflow for a pull request. Each pull request at the component level required a visual validation of the component before merging. She explained how Applitools could maintain separate baselines for each branch and manage checkpoints independently. Then, she showed the full develop workflow build pipeline.

Finally, she showed how Github integration linked visual testing fit into the entire Circle CI build. She also showed how the buld process linked to Slack, so that the team could be notified if the build or testing encountered problems. The build, including all the tests, needed to complete within 10 minutes.

Overall Feedback

Marie provided her team’s general feedback about using Applitools. They concluded that they required Applitools for visual validation of the component-level and site-level tests. Developers appreciated how easily they could use Applitools with Cypress, and how they could run 60 component tests in under 5 minutes across a range of browsers. The design team also uses Applitools to validate the design, and they found the learning curve was fast for figuring out the visual elements.

As users, they did have feedback for improvement to share with the Applitools product team. One of the most interesting came from the design team, who wondered if they could use UI design tools (Sketch, Figma, Abstract, etc.) to seed the initial baseline for an application.

Beyond Applitools, the accessibility testing has helped ensure that News UK can deliver visual and audio accessibility for their users.

Conclusion

Marie Drake made a strong case for using a design system whenever there are multiple design and development groups working independently on a common web application. The design system eliminates redundancy and helps speed the rate of change whenever groups want to roll out application enhancements.

She also made a strong case for building testing into every phase of the design system, from component-level unit, functional, visual, and accessibility tests all the way to page-level tests of the same. For testing speed, testing accuracy, ease of test maintenance, and cross-browser tests, Marie made a strong case for using Applitools as the visual test solution for the News UK design system.

For More Information

• How Do You Simplify End-To-End Test Maintenance?
• Measure Your Test Automation Maturity
• Adding Log Analysis to QA Will Improve Your Processes
• How Do You Catch More Bugs In Your End-To-End Tests?
• How Can 2 Hours Now Save You 1,000 Hours In The Next Year?
• Request an Applitools Demo
• Sign up for a free Applitools account.

The post How Do You Test A Design System? appeared first on AI-Powered End-to-End Testing | Applitools.