Selenium Archives - AI-Powered End-to-End Testing | Applitools

Let the Engineers Speak! Part 4: Changing Frameworks

Let the Engineers Speak! Part 3: Favorite Test Integrations

Let the Engineers Speak! Part 2: The Biggest Challenges

The post Let the Engineers Speak! Part 4: Changing Frameworks appeared first on AI-Powered End-to-End Testing | Applitools.

Let the Engineers Speak! Part 3: Favorite Test Integrations

Andrew Knight — Wed, 28 Dec 2022 16:06:28 +0000

In part 3 of our Cypress, Playwright, Selenium, or WebdriverIO? Let the Engineers Speak recap series, we will recap our panelists’ favorite integrations in their test automation projects, as well as share our audience’s favorite integrations. Be sure to read our previous post, where our panelists discussed their biggest challenges in test automation.

The experts

I’m Andrew Knight – the Automation Panda – and I moderated this panel. Here were the panelists and the frameworks they represented:

Gleb Bahmutov (Cypress) – Senior Director of Engineering at Mercari US
Carter Capocaccia (Cypress) – Senior Engineering Manager – Quality Automation at Hilton
Tally Barak (Playwright) – Software Architect at YOOBIC
Steve Hernandez (Selenium) – Software Engineer in Test at Q2
Jose Morales (WebdriverIO) – Automation Architect at Domino’s

The discussion

Andy Knight (moderator):

Alrighty. So I mean, this, our conversation is now kind of going in so many good directions. This is awesome. We keep talking about integrations and how our frameworks and our tools can connect with other things. So I guess my next question I would like to ask as a big popcorn question would be: what are some of your favorite integrations with your test projects? I’m talking about things like CI hooks, special reports that y’all have, maybe things for visual testing, accessibility testing.

And not only is this a question for the panel speakers, but also for the audience. We have another Slido word cloud, so head over to the Slido tab. We’d love to hear what your favorite integrations are for your test projects.

But I’d like to now direct this question back to our panel and ask them to pick one. Tell me why you love it so much. Tell me how y’all use it and what value y’all get out of it.

Carter Capocaccia (Cypress):

Okay, I’ll hop in here. I think for me with Cypress, it’s just under the hood, it’s just JavaScript, right? So most anywhere is able to hook into that immediately. So when we talk about integrations, let’s just use GitHub for example here. If you wanted a GitHub Actions pipeline, you can just go ahead and start doing that immediately. And Cypress out of the box will provide you a Docker image to go ahead and use right inside of your GitHub Actions pipeline.

So for me, I guess I’m not going to focus on any one specific tool for integration here. I think the way that Cypress kind of makes itself agnostic to all external tools is what makes it really, really nice. You know, it doesn’t say, hey, you’ve gotta use us with this particular, like an Azure pipeline or any kind of other DevOps tool. It says, hey, under the hood, all we are is a JavaScript framework. If you want to execute our JavaScript inside of some kind of Docker image, create your own Docker image. Gleb knows all about this. He publishes a new Docker image like every day it seems.

And we have, you know, the ability to take all that, take the results, and the base of what comes out of Cypress is just JSON. We’ll shoot that over to an open-source dashboard. So there’s Report Portal out there if you wanted to take those results, and then put them into a portal. I see on here [the poll results] Allure reports. Well, okay, so you take it, you just put it into any kind of integration there.
So I think for me personally – I actually wrote a whole blog post about this where I was able to take Vercel, GitHub Actions, and Cypress and set up a CI/CD pipeline completely for free with automated deployments, preview environments, automated testing, the whole nine yards. And so it made it to where for my personal little website, all I’ve got to do is submit a pull request. It kicks off all my tests. As soon as it’s done, it merges it, and then it’s immediately deployed with Vercel. So that was a really cool thing. So if I have to name a tool, I’m gonna say Vercel and GitHub Actions.

Gleb Bahmutov (Cypress):

Can I second that? GitHub Actions are amazing. Honestly, I went through the whole, you know, self-running Jenkins to TeamCity, Buildkite, Circle CI. GitHub Actions are the best. It’s amazing how much you can do with just JavaScript on UCI. And I was the one writing all integrations for the Cypress team at the time.

I want to bring one more integration. I’ve written it. It’s [an] integration between Cypress and LaunchDarkly feature flags environment, because during the test, sometimes you want to test an experiment behind a feature flag. And for us now, it’s an open-source plugin. It’s like this, here’s my user, here are the experiment values I want to enable. Let’s see this functionality, and it’s working.

Tally Barak (Playwright):

[My] answer is GitHub Actions as well. I don’t have that much experience derived with different tools. I used to work with Travis and a bit with CircleCI. But GitHub Actions is just answering everything we need – and Playwright obviously – because it just requires a node for the JavaScript version. You just install it with Docker, and you are good to go and do whatever you want with the outputs and so on. So yeah, GitHub actions and I’m actually interested in the other one that you mentioned about the feature flag. That’s worth looking into.

And of course, Cucumber. I also have a repository that is a starter report for working with Cucumber and Playwright. I’ll post it on the channel if anyone is interested to try it out. If you want to stick with the Gherkin and the BDD-style test. Otherwise, go with the Playwright test runner.

Andy Knight (moderator):

Tally, I want to plus one of both things you said there. First of all, everyone knows that I love BDD, I love Cucumber, I love Gherkin, so yes. But also, I want to go back to what everyone is harping about GitHub Actions. Y’all in the audience, if you haven’t tried GitHub Actions or used to yet, definitely check it out. It is very straightforward, low-to-no cost CI baked into your repo. I mean, like anytime I’ve done open-source projects or example projects, just putting a GitHub Actions to automatically run your tests from a trigger, oh, it’s beautiful. It can be a little intimidating at first because you’ve got that YAML structure you’ve got to learn. But I have found it very easy to pick up. I feel like personally – and speakers back me up on this – I never sat down and read a manual on how to write a GitHub Action. I just kind of looked at GitHub Actions and copied them and tweaked them, and it worked.

Tally Barak:

And then you went to the marketplace and looked for some.

Andy Knight (moderator):

Yes, yes, yes. You know, and so you can find things that are out there already. And I’ve done it not just with Cypress and Playwright, but literally like every single test framework out there that I’ve touched. You know, if it’s like a Selenium thing, if it’s a Playwright thing, a Cypress thing. Any kind of test unit integration, end-to-end, you name it. Like, it’s awesome. But anyway, I just wanted a plus one to both of those and now I’ll yield to the rest of the speakers’ favored integrations and why.

Steve Hernandez (Selenium):

Can I give a plug for LivingDoc? We use a lot of the SpecFlow stuff in our stack, as you saw, like the runner. There’s a great SpecFlow+ Runner report that gets generated and can show how each of the threads are performing in your test suite so you can look at performance. But one of the things just really simply is the Azure Log Analytics ingestion API. So right in your test pipeline, you can just throw up a PowerShell script or some console application and send that data that you would have in your CI tool over into something like log analytics. I suppose there’s an equivalent for Amazon. But then you can correlate, you know, your telemetry from your application along with test data and start to see where performance issues are introduced.

But I guess coming back to my plug for LivingDoc is, I feel in a lot of cases our automated tests are actually the only documentation for some parts of the application as far as how they’re supposed to work because you’ve captured the requirements in a test. So they’re kind of like living documentation and LivingDoc literally is that. And it’s something that you can share with the business and the developers to see what you have for coverage and it’s human readable. I think that’s really important.

Carter Capocaccia (Cypress):

Well, now with GPT3, everything is human readable.

Andy Knight (moderator):

So I wanted to ask about visual testing and specifically with Applitools. I saw and heard that some of y’all had used visual testing with Applitools in your projects. In line with talking about integrations that you love, how do y’all use Applitools if you use it, and what value do you find you get out of it? Go ahead.

Carter Capocaccia (Cypress):

Yes, I want to bring up kind of why Applitools is a different breed of product when it comes to visual testing, though. So you really have two flavors of visual testing. One is a bitmap comparison where you take a screenshot of an image and it takes it and basically breaks it down into computer readable code and it compares a known standard against the new version. If there’s a certain percentage with diversions between the old version and the new version, it flags it.You can kind of already tell, well, if all I know about is just basically one to zeros and comparing the results here, it’s not really all that intelligent. Because I could have a piece of the page, let’s say it’s a headline. The headline updates every five minutes. Well, I don’t have time to just continually update this baseline screenshot.

Well, what a tool like Applitools does is it actually takes your screenshots and puts them through computer vision. And with computer vision you can now provide a lot more instruction to what do you want to compare? What’s important to you? Do you want to ignore certain parts of the page? When I compare X to Y, do I really consider, let’s say, a carousel where the images are being delivered by a CMS and they’re being changed on a daily basis? Do I really care that yesterday it was a picture of let’s say somebody’s front porch and today it’s a picture of a tree? So there’s a lot more intelligence that’s built into a tool like Applitools than there is with standard bitmap comparison. I think it takes the right kind of customer for Applitools. And when they do have that right kind of customer, it’s a pretty huge game changer, considering that it allows you to make a lot more informed decisions about the health of the visual appearance of your application versus, like I said, a bitmap comparison.

Basically, if you want to compare the two, one is analog, the other is digital. The analog is the map comparison. The digital way is Applitools. If we’re being honest with ourselves, the Applitools way is probably the best and right way to do this. I just know that there are a lot of teams that are making smaller scale applications that can get away with assuming higher risk in using a tool like a bitmap comparison tool.

Jose Morales (WebdriverIO):

I would like to add that we have been using Applitools for six months or so, and it was a great addition to our framework. It was reducing a lot of code because we were coding a lot of assertions. So for example, if you go to the Domino’s website, you have the store locator, and then the store locator basically based on your zip code automatically populates the city region and the store closest to you. There is a lot of functionality there. And I remember that we were validating every single text fill, every single button, every single drop and menu for the content. In English and in Spanish and the different languages. So for us it was very helpful to use Applitools because with WebdriverIO, we can simulate the location from the customer and then move the store locator for that particular zip code and tell Applitools, hey, take a screenshot for this one.

And in one single shot, we are validating all the colors, buttons, text fields, content, and we were very careful to see what is worth it for this scenario, right? So in all the framework, what we are doing is okay, let’s test this particular functionality for this scenario. Let’s use WebdriverIO to move the customer until this particular checkpoint and then take a screenshot for that particular state in our application. And in one single shot, we can validate this in all the devices. In Chrome, Firefox, Edge. Pixel, iPhones. So with that particular action, and because we are reducing a lot of assertions, we are writing less code, it’s easy to maintain, it’s easy to extend, and our test cases are more accurate and smaller. And I think for us Applitools was a great addition and I fell in love with Applitools and currently we’re trying to incorporate the mobile validations, meaning iOS and Android as well.

Did anyone ever change their framework?

We heard popular tools mentioned like GitHub Actions and Cucumber, as well as some lesser known tools. A few of our panelists are also using Applitools Eyes in their projects to provide faster test coverage of their frontend. In the next article, we’ll cover the answers these experts gave when we posed a sticky question: “Did you ever consider changing your test framework?”

Let the Engineers Speak! Part 5: Audience Q&A

Let the Engineers Speak! Part 4: Changing Frameworks

Let the Engineers Speak! Part 3: Favorite Test Integrations

Let the Engineers Speak! Part 2: The Biggest Challenges

The post Let the Engineers Speak! Part 3: Favorite Test Integrations appeared first on AI-Powered End-to-End Testing | Applitools.

Let the Engineers Speak! Part 2: The Biggest Challenges

Andrew Knight — Thu, 22 Dec 2022 14:20:00 +0000

In part 2 of our Cypress, Playwright, Selenium, or WebdriverIO? Let the Engineers Speak recap series, we will recap what our panelists shared as the biggest challenges they currently face in testing and automation. We’ll also cover some of the biggest challenges our webinar audience members have faced. Be sure to read our previous post, where we introduced the experts and their test automation projects.

The experts

I’m Andrew Knight – the Automation Panda – and I moderated this panel. Here were the panelists and the frameworks they represented:

Gleb Bahmutov (Cypress) – Senior Director of Engineering at Mercari US
Carter Capocaccia (Cypress) – Senior Engineering Manager – Quality Automation at Hilton
Tally Barak (Playwright) – Software Architect at YOOBIC
Steve Hernandez (Selenium) – Software Engineer in Test at Q2
Jose Morales (WebdriverIO) – Automation Architect at Domino’s

The discussion

Carter Capocaccia (Cypress):

So I think one of the biggest challenges that I personally faced was that a lot of the times – I think initially when you’re starting out with automation – the tech problems that you face are where all the focus gets put on. And people lose sight that there’s an entire component to process around this. Meaning that, well, once you’ve written, you know, your hundred tests, you’ve got your architecture stable, you’ve figured out how to seed your database and correct your APIs into all the stuff that you need to do to make your tests run. Well, what do you do with those results? How does it actually impact your app? And I think a lot of people lose sight of that.

Of course, with the advent of CI/CD pipelines, it becomes a little bit easier because now you can basically prevent merges or prevent releases based on the results of these tests. But I think that gets forgotten until we’re way down the line. So, you know, you’ve written all your tests and you come back to your dev teams and you say, “Hey, I’m gonna start failing your pipelines now, and you’re not gonna be able to merge, you’re not gonna be able to to deploy.” And then you immediately have resentment against this process, right? So I think maybe the marketing of why you’re doing what you’re doing, how it’s gonna impact the process, and what is the positive result that comes out of this is equally as important as tackling those technical hurdles.

Tally Barak (Playwright):

Yeah I think I will go somehow along those lines. Besides some technical difficulties that I already shared about the shadow DOM and trivial actions like dragging, again, some of the things that we encountered. The biggest challenge was creating a culture of testing in the organization, making sure that people take care of the test. Because of some flakiness, we didn’t want to prevent merges. I mean, [for] a long time, we built more and more into that, but at the beginning it was like, “People, this is your helper. This is something to help you write better code.” And it didn’t come very easily, I would say.

Poll results from over 60 webinar participants depicting their biggest challenges while testing

Gleb Bahmutov (Cypress):

Can I comment on this slide [of the] poll? I see selectors as one of the challenges and it’s [a] pretty big one, apparently, for a lot of people. I just want to say what we’ve done at Mercari US. So, sometimes [when] you’re writing an end-to-end test, you come to a part of a page and there [are] some dynamic classes, no good selector specific for testing. So what we do in our test [is] we select however we can – by text, by position, you name it. But we do add a to-do comment. So in our test repo, we have lots of to-do [comments] like “needs better test selector”.

And we have [an] automated script that actually counts it and puts it in a rhythm. So we know how many to-dos for selectors we have total. And one of our objectives as a team is to drive a number down. So we [are] adding selectors and it’s kind of a good small task to do, but having good selectors is a key. So don’t just kind of assume someone else will add it for you. Have it as one of your automated priorities.

Andy Knight (moderator):

That’s really cool to actually have it kind of scan through and kind of pinpoint. So it’s almost like you’re self-linting. Wow, that’s awesome. If I may ask another question off of this topic of the challenges specifically around selectors. What other challenges have y’all faced with selectors or innovative solutions to try to get on top of them? Like, does anybody use smart selectors or visual selectors or anything at all?

Carter Capocaccia (Cypress):

Can I share my really staunch opinion on this? I personally don’t allow merge requests that have XPath selectors in them. I disallow it. It is a forbidden practice at our organization. We actually start with – we have a selector hierarchy that we try and follow. And I believe it’s Testing Library. And know it’s a really generic name for everybody in this room, but genuinely, it’s called Testing Library. And they have a selector pattern basically as a hierarchical representation based. I think it says, you know, accessibility selectors, and then at the very bottom, it even goes down to classes and then IDs. But it starts with accessibility selectors, and then content and then, you know, some other pieces that you should go through. And nowhere in that list is XPath.

I know that a lot of folks feel like XPaths are necessary, but I think that you’ve gotta find a way around them. I’m sorry if that hurts you there, Pandy. But yeah, in my opinion, I think XPaths are sometimes used as a crutch. And in today’s world where you have components that are conditionally rendered or potentially even, like for situations that I’m working in, it depends on what part of the country you’re in or what part of the world you’re in. Your page may render it completely differently than another. So therefore I cannot rely on a DOM structure to not be shifting at a moment’s notice, or there may be a CMS tied into this so they can add a component to the page. It’s gonna completely change the DOM structure. So we have to do better than that and, you know, tests more and more nowadays are being kept right alongside the application code.

And I think that, speaking back to what I was saying about kind of changing the process around what you do. Well, if I’m a test engineer, if I really can’t get to a DOM element any other way, why can’t I go into that component and add a data test ID attribute? I should have that ability as a test engineer, just like a UI dev should have the ability to come into my test and modify that. So that’s I think a way you can kind of shift culture when you even think about something as simple as DOM selectors. But yeah, the long-winded answer to say, I personally think that if you want to have maintainable selectors, avoid XPaths.

Jose Morales (WebdriverIO):

I would like to share my opinion in two ways. One is the challenges that we have in Domino’s, and another one is the selectors.

The challenge is the maintenance of the test cases. And basically what we are following is a good process. And with a good process, I mean we have a team that is reviewing the executions and we do analysis about the results. We figured out if we need to fix the test cases, we need to change the test cases because the feature is now different – or what is the root cause of the problem, right? We are following good practices in terms of development, like code reviews, pull requests. We’re doing peer programming. If some automation engineer is facing some kind of challenging problem/blocker, we have working sessions they requested for those working sessions. And we work together, right, as a team to figure out what’s the problem, what [are] the possible solutions that we can do. And it’s very active.

And commenting about the selectors, we encourage to use the ID. And in Domino’s we have a custom ID is the UUID that all the web developers add that particular identifier for us to use. So there is a close communication between the developers and the automation engineers to work together as a team, how we can help to [better test] our product. That’s another thing that we normally do. And respect as much as possible the page-object design pattern. That is what we are doing in Domino’s.

Tally Barak (Playwright):

The slide that was showing actually three items that were shining there, which is flakiness, maintainability, and selectors. And I think they’re all related to some extent. Because if your selectors are too accurate, like XPath with all the DOM structure, then you have a maintainability problem and you might have flakiness. And I have to say, if you are talking about the tools, this is probably the one thing that Playwright has solved in just a way that makes writing tests really easy. I was hearing the other people here and they went like, “Oh, we already forgot about this problem after working with two and a half years.” We used to work with XPaths because that was the only thing that could pierce the shadow DOM.

Now we are working with plain selectors. We’re keeping them to the minimum and mostly the user-facing selectors. So things like text usually. Maybe something like a header component and then a text. And we will know how to address the very exact selector. And that reduces the maintenance because this is the last part that is likely to change. And because of the way Playwright is working with the auto waiting, it’ll just wait, we don’t need to put some time out. It’ll know to wait until a certain time out for the component to show up.

You can write very small and very concise selectors. And the one other thing is that they have the strict mode, which means that if it finds two elements that correspond to the same selector, it’ll break the test. And one of the problems that we’ve seen previously is that it would go with the first element. So it might click a different button or something else, and the test will continue and it’ll only fade like three steps or five steps ahead. And this is really hard to understand what is the problem. Playwright here will just tell you. Of course, you can change if you want it or not. It’s an option. But if you do that, you get a lot less flakiness in your test because the first time it’ll encounter the element, it’ll tell you, okay, “I have two here. You’re probably on the wrong path. Go ahead and fix your test.” So all these three things, the maintenance, the flakiness, the selectors is one of the, I think, the greatest advantages of Playwright in these terms.

Andy Knight (moderator):

Awesome. So I know we’ve spent a lot of time talking about selectors and DOM and modeling the page. What design patterns do y’all use for or within your test projects? I know, I think it was Jose mentioned like they use page objects and they stick to it very rigidly – or maybe rigidly is not the best word, but they stick with page objects. Are y’all using page objects or are y’all using Screenplay? I know Cypress has those action or command methods, I forget. What are y’all using to help make your interactions using those selectors better?

Steve Hernandez (Selenium):

Screenplay.

Gleb Bahmutov (Cypress):

We don’t try to make the test super dry, right? Eventually, if we see a duplication of code, we will write a custom command. For example, we have custom commands for selecting by data test id. So it’s really easy by data test ID or ARIA label, we use custom commands. We almost never write page objects. We prefer little reusable functions, right? We need to create a user. Here’s a function, creates a user for you.

There is one advantage, though, because I like writing Cypress plugins, we can abstract something that is really specific and just move it to [a] third party public open source plugin, but we’ll reuse in our project. So we’ve been doing that a lot. I just looked, we have more than 20 Cypress plugins with our test report, but 17 out of them is what I’ve written. So, we take that abstraction and we make it and just have it by itself so it becomes more powerful and reusable. But small functions and custom commands for us.

Tally Barak (Playwright):

We are working with Cucumber. So this is basically our structure. Each step has the automation behind the scene, and then obviously you can reuse because the step is reusable in any of your tests. And we do try to build them sometimes a bit more, you know, a sophisticated kind of code, so you can reuse them in different places.

Jose Morales (WebdriverIO):

[About] using other design patterns – another thing that really helped us. And I saw a very interesting question about how do you integrate a backend with a JavaScript frontend? And that’s a very interesting question because for us, we try to consume APIs as well. So for example, in Domino’s, we have an application that is an interface for configuration for our stores. So every manager in our stores can configure certain parameters for the store, right? Like the store phone number. So they can go to a specific website and then change the store’s number, for example. So if you think about the testing perspective, you can go to the UI and change a particular store number and then hit an endpoint to refresh the backend for that particular store configuration.

And then you go to another backend to hit the store profile and then get the store profile with a request and read the JSON back, right? So you don’t need to interact with the UI all the time, but you can also integrate with the backend as much as possible. So that’s a very clean strategy, because it helps you to test faster and test better and reduce the flakiness.

And another thing that I recommend is to use the plugins, right? So explore the plugins that you can get in the industry. For example, in the backend and if you are interested in measuring the performance, you can use Lighthouse with WebdriverIO. So you can – with the standards in the industry – measure what is the performance for your web application, right? So we don’t need to reinvent the wheel. And that’s my recommendation.

Andy Knight (moderator):

Awesome. Awesome. So I know, Steve, right off the bat, you had mentioned the Screenplay pattern. I wanted to come back to that. What have you found has been good about using Screenplay over something like raw calls or page objects or something else?

Steve Hernandez (Selenium):

Personally, I haven’t used a lot of raw calls. I’ve had the luxury of using Boa Constrictor. It kind of wraps the Selenium stuff in a warm hug, and it handles a lot of the timing issues that you might run into. Boa Constrictor has a lot of built-in – I hear you guys talk about plug-ins – we have a lot of built-in interactions, they’re called. So they can be questions that you’ll ask of an API endpoint. They can be questions that you’ll ask of the page in the form of, “I’m gonna scrape this column here that I use over and over to compare to something in the future”.

So we’ll have our Gherkin steps and essentially backing each of those step definitions are these different reusable components or questions or interactions. That works very well with the level of parallel parallelism we’re doing. You know, if you’re firing off 50 threads at a time or 80 or more, we need stuff to be fast, not step on each other’s toes. So we write the interactions in a way where they’re very atomic and they’re not going to affect something else that’s happening on another thread. As far as page objects, I haven’t used those a whole lot.

What about integrations?

We’ve heard some common themes from both our panelists and our webinar viewers around their test automation challenges – selectors, maintenance, and flakiness. Gaining a culture of testing that embraces and adheres to the process is another challenge our engineers have faced in their projects. Our panelists each touched a little bit on how they approach these challenges in this recap. In the next article, we’ll cover the panelists’ favorite integrations for their test automation projects.

Let the Engineers Speak! Part 5: Audience Q&A

Let the Engineers Speak! Part 4: Changing Frameworks

Let the Engineers Speak! Part 3: Favorite Test Integrations

Let the Engineers Speak! Part 2: The Biggest Challenges

The post Let the Engineers Speak! Part 2: The Biggest Challenges appeared first on AI-Powered End-to-End Testing | Applitools.

Let the Engineers Speak! Part 1: Real-World Projects

Andrew Knight — Mon, 19 Dec 2022 16:27:39 +0000

Over the past two years, Applitools has hosted a series of code battles called Let the Code Speak. We pitted the most popular web testing frameworks against each other through rounds of code challenges, and we invited the audience to vote for which did them best. Now, to cap off the series, we hosted a panel of test automation experts to share their real-world experiences working with these web test frameworks. We called it Cypress, Playwright, Selenium, or WebdriverIO? Let the Engineers Speak!

Here were our panelists and the frameworks they represented:

Gleb Bahmutov (Cypress) – Senior Director of Engineering at Mercari US
Carter Capocaccia (Cypress) – Senior Engineering Manager – Quality Automation at Hilton
Tally Barak (Playwright) – Software Architect at YOOBIC
Steve Hernandez (Selenium) – Software Engineer in Test at Q2
Jose Morales (WebdriverIO) – Automation Architect at Domino’s

I’m Andrew Knight – the Automation Panda – and I moderated this panel.

In this blog series, we will recap what these experts had to say about the biggest challenges they face, their favorite integrations, and if they have ever considered changing their test framework. In this article, we’ll kick off the series by introducing each speaker with an overview of their projects in their own words.

Gleb’s Cypress Project at Mercari US

Gleb:

I’m in charge of the web automation group; we test our web application. Mercari US is an online marketplace where you can list an item, buy items, ship [items], and so on. I joined about a year and a half [ago], and my goal was to make it work so it doesn’t break.

There were previous attempts to automate web application testing that were not very successful, and they hired me specifically because I brought Cypress experience. Before that, I worked with Cypress for years, so I’m kind of partial. So you can kind of guess which framework I chose to implement end-to-end tests.

We currently run about 700 end-to-end tests every day multiple times per feature with each pull request. On average, our flake – you know, how many tests kind of pass or fail per test run – is probably half a percent, maybe below that. So we actually invested a lot of time in making sure it’s all green and working correctly. So that’s my story.

Carter’s Cypress Project at Hilton

Carter:

Nice to meet you all today and thank you for the opportunity to speak with everybody. So I need to lead off with a little bit of a disclaimer here. Hilton does not necessarily endorse or use any of the products you may speak of today, and all the opinions that I’m going to talk about are my own.

So Hilton is a large corporation. We operate 7,000 properties in 122 countries. So when we talk about some of the concerns that I face as a quality automation engineer, we’re talking about accessibility, localization, analytics, [and] a variety of application architectures. We might have some that are server-side rendered. We might have some that are CMS-driven pages and even multiple types of CMS. We also have, you know, things that are like that filter out bot traffic. We may use mock data, live data. We have availability engines, pricing engines. So things get pretty complex and therefore our strategy has to follow this complexity in a way that makes things simple for us to interpret results and act upon them quickly.

To give you guys an idea of the scale here, we typically are much like Gleb in this regard, where every single one of our changes runs all of our tests. So we are constantly executing tests all of the time. I think this year – I was just running these reports the day to figure out for, you know, interview review stuff – I think I pulled up that we had so far in this year run about 1.2 million tests on our applications. That’s quite a large quantity. So, you know, if anything, that speaks to our architecture – [it] has some stability built into it – the fact that we were even able to execute that quantity of tests.

But yeah, I’m a Cypress user. I’ve been a Cypress user now for about three years. Much like Gleb, I got faced with the challenge of, “Well, this only runs in Chrome. It’s a young company, young team.” And yeah, I think, you know, we stuck with it, and obviously today Cypress has grown with us and we’re really, really happy to still be using the tool. We were a Webdriver shop before, so I’m really interested to see what Jose is doing. And there probably is still some Webdriver code hanging around here at Hilton. But yeah, I’m really excited to be here today, so thank you for the opportunity.

Tally’s Playwright Project at YOOBIC

Tally:

I work at YOOBIC. YOOBIC is a digital platform for frontline employees – that’s a way to say people like sellers in shops. And they use our application in their daily job – for the tasks they need to do, for their training, for their communication needs, and so on.

For this application to be useful, we have a very rich UI that we are using. And for that, we have decided to use web components. We’re using extensive JS and building web components. This is important because when we tried to automate the testing, this was one of the biggest barriers that we had to cross.

We used to work with WebdriverIO, which was good to some extent. But the thing that really was problematic for us was the shadow DOM support – and shadow root. So I got into the habit for once every fortnight or monthly to go into the web and just Google “end-to-end testing shadow DOM support” and voila! In one of these Googlings, it actually worked out and I stumbled upon Playwright – and it said, “Yes, we have Shadow DOM support.” So I went like, “Okay, I have to try that.” And I just looked at the documentation. It was like shadow DOM was no biggie. It was just like built/baked in naturally. I started testing it, and it worked like magic.

That was almost two and a half years ago. Playwright had a lot less than what it has today. It was still on version 1.2 or something. At the time, it was only the browser automation side; it didn’t have the test runner. And our tests were already written in Cucumber, and Cucumber is a BDD-style kind of testing. So this is the way we actually describe our functional test. So user selects mission, and then user starts, and so on – this is the test. We already had that from WebdriverIO, and basically we just changed the automation underneath to work with Playwright instead of Selenium WebdriverIO.

And this is still how we work. We are using Cucumber for running those tests. We use Playwright as the browser automation. We have our tested app and we are testing against server and the database. We have about 200 scenarios. We run them on every PR. It takes about 20 or 25 minutes with sharding, but this is because they are really long tests.

It also solved a lot of other problems that we had previously, like the fact that you can open multiple browsers. Previously what we were doing, if you wanted to test multiple users, we would log in, log out, and log in with the other user. And here, all of a sudden, we could just spawn multiple windows and do that.

And one [additional] thing that I want to mention besides the functional test, this end-to-end test, we also run component tests. This is different; we are using the Playwright Test Runner, in fact. This is because we work with Angular, and Playwright component testing is not supported with Angular yet. We are using Storybook, and we test with Playwright on top of Storybook. So yeah, that’s my story.

Steve’s Selenium Project at Q2

Steve:

I actually started as a full stack developer at Precision Lender. We’re part of Q2, which is a big FinTech company. If you had priced any skyscrapers or went to get a loan for a skyscraper, there’s a very good chance that our software was used recently – I’m sure you’re all doing that.

Working on the front end, it’s a JavaScript-based application using Knockout but increasingly Vue. I worked over in that for about two and a half years at the company, and I saw the cool stuff that Andy and our other Andy were doing. And I was excited about it. I wanted to move over [and] see what it was all about. And I believe strongly in the power of test to not only improve the quality of the application, but also just the software development lifecycle and the whole experience for everyone and the developers.

Q2 test automation architecture diagram from the Let the Engineers Speak webinar

So what you guys have in front of you is a typical CI pipeline that we use. We use Git and we use TeamCity. And on a typical day, we run about 20,000 tests. We have about 6,000 unique tests and upon one commit to our main developed branch, we kick off about 1,500 tests and it takes about 20 to 25 minutes, depending on errors, test failures. On that check-in to Team City, we use a stack with SpecFlow and SpecFlow+ Runner kicks off our tests.

As I said before, we’re using Boa Constrictor underneath with the Screenplay Pattern. It acts as our actor, which is controlling the Selenium Webdriver. And then the Selenium Webdriver farms out the test work in parallel to the Selenium Grid and our little nodes; it’s an AKS cluster run through all of our tests and spit out a whole bunch of great documentation. One of my favorites being the SpecFlow+ Runner report and other good logging things like Azure Log Analytics.

Jose’s WebdriverIO Project at Domino’s

Jose:

Thank you from all [of us at] Domino’s. Well, Domino’s is a big company, right? We have 4 billion dollars in sales every single year. And – only in the Super Bowl – we sell 2 million pizzas just in the United States. So we have a big product and our operations are very critical. So I decided to use WebdriverIO for our product. And this is the architecture that we created in Domino’s.

Domino’s test automation architecture diagram from the Let the Engineers Speak webinar

In Domino’s, we have different environments. We have QA, we have pre-production, and we have production. And for selecting different kinds of environments, we have different kinds of configuration. And in that configuration, Jenkins reads the configuration for that particular environment, we get the code from a stash, and then we are using virtual machines – it could be inside of the Domino’s network or it could be in Source Labs. Then we have a kind of report. And one thing that I really love about WebdriverIO is the fact that it’s open source, and it has huge libraries.

So in Domino’s, we’re using WebdriverIO in three projects – Domino’s website, in the Domino’s re-architecture, and in the Domino’s call center application. [In fact,] we are using WebdriverIO with two different kinds of reports, and we’re using [it] with two different kinds of BDD frameworks. One is, of course, Cucumber; that is the most used in the industry. And another one is with Mocha. And in both, we have very successful results. We have close to 200 test scenarios right now. And we are executing every single week. And we are generating reports for those specific scenarios. We have a 90% success pass rate in all scenarios.

And we’re using Applitools as an integration for discovering issues – visual issues and visual AI validations. And basically in all those scenarios, we’re validating the results in all those browsers. We’re normally using Chrome, Safari, Firefox, Edge, and of course, we’re using mobile as well, in the flavors of Android and iOS.

And the cool thing is that it really depends about what is the usage for customers, right? Most of the customers that we have are using Chrome, and we are seeing what is our market, what our customers are using, and we are validating our products in those particular customers.

So I am really happy with WebdriverIO, because I was able to use it in other problems that we were facing. For example, in Malaysia we have some stores as well, and in the previous year we had a problem with performance in Malaysia. So I was able to integrate a plugin – Lighthouse from Google – in order to measure the performance for that particular market in Malaysia using WebdriverIO. So far for us [at] Domino’s, WebdriverIO is doing great. And I am really happy with the solution and the results and the framework that we selected. And that’s everything from my end. Thank you so much.

What are their biggest challenges?

So we’ve gotten an overview of the projects these test experts are using and a little bit of their histories with their test framework of choice. This will help set a foundation to get to know our engineers and understand where they’re coming from as we continue our Cypress, Playwright, Selenium, or WebdriverIO? Let The Engineers Speak series. In the next article, we’ll cover the biggest challenges these experts face in their current test automation projects.

Let the Engineers Speak! Part 5: Audience Q&A

Let the Engineers Speak! Part 4: Changing Frameworks

Let the Engineers Speak! Part 3: Favorite Test Integrations

Let the Engineers Speak! Part 2: The Biggest Challenges