AI & ML in the QA Industry: Friday 11/20/20 10am, PST

On November 20, 2020, Women in Big Data kicked off the “Better Data for Better App” series with Jonathan Lipps, on “AI and ML in the QA Industry”.

Jonathan is the Director of Learning at HeadSpin and focuses his time in three key areas:

1. He is the architect and maintainer of Appium, a tool that facilitates the automation of applications. It gives developers & QA engineers the ability to write automated steps for any kind of application, which is a key component of the process of automated testing of any application.
2. He is also in charge of HeadSpin University where the mission is to provide great content for the QA and related Industries.
3. Jonathan also spends a lot of time writing about Appium and mobile test automation at Appium Pro.

The talk covered a topic that marries two things – QA & AI and the promise AI holds for various industries and some of the applications of ML in the QA industry. Jonathan spoke in depth about what some of the products that have been marketed to the QA Industry as AI have been and evaluating it from a lens of a QA expert as to what is myth vs. reality.

QA Industry

The Software development cycle consists of the following stages:

Ideate Feature ⇔ Build Feature ⇔ Test App ⇔ Release App ⇔ Evaluate user response

This is an iterative process where you are improving a product and coming up with new things based on how things went with your last iteration. QA Industry is about Quality assurance and Quality control, figuring out what works and what doesn’t. In software development, the testing stage of the cycle is owned by the QA team.

Types of QA

• Manual QA: People using the app the way a user would, checking manually for bugs. Walking through a variety of predetermined user flows manually to make sure everything works the way it should.
• Automated QA: Over time there was pressure to increase the speed and accuracy which led Manual QA to evolve into automation. This involves checking the quality application by writing code that exercises various levels or aspects of the app (Including UI). These would exactly mimic how a user would use the app.

When people are talking about applying AI/ML to the QA Industry, they are usually talking about processes and ML methods that assist with automated QA or maybe replace automated QA. Manual QA is becoming obsolete due to the need of speed and efficiency in the software development cycle.

The AI question: Sometimes we can go a little bit overboard in thinking how AI is applied to actual problems we want to solve. Is AI == BS?. Sometimes what people promote as AI is really Software development under the hood; it’s not anything different from what we’ve had for years in coming up with algorithmic solutions to problems: hand-coded solutions to problems that use regular old software development practices.

Categories of AI solutions in QA

• AI in marketing only: Intelligently designed software that doesn’t use machine learning models. Example: scrape production user activity log to generate test cases. Capture multiple selectors for elements to increase test robustness.
• AI/ML in supporting roles: ML models are used to support features, not as a replacement for test authoring. Example: Image recognition models to detect visual differences. Video quality models give feedback on user-perceived quality.
• AI/ML as the primary driver of automation: Tests are written and bugs found by autonomous bots acting on pre- or post-trained ML models. Example: you hand off the app to the AI with no additional metadata and it sends you back bug reports.

Do you need “AI” in your testing? Why?

Jonathan believes one way to answer this question is to evaluate technologies based on their actual ROI, not how well they claim the hype of the zeitgeist. Also an important question to ask when doing your diligence on a product that claims to be AI is: what corpus was used to train the ML Model? Jonathan believes that most ROI for the products will be from AI/ML in supporting roles for a while.

I would highly encourage you to watch Jonathan’s talk for a deeper dive on the topic. A PDF version is available here.

Also, if you’d like to learn more about the QA and test automation industry, write automated tests yourself or make a career move, Appium Pro and HeadSpin University are great resources.

Thank you, Jonathan! Grateful for your time.