Chetan Shridhar

Regression Web Testing

0→1Regression testing for configurations

Engineers test configuration changes on staging before they reach production.

ACT landing page: a search bar, news, a configuration check and Jira tickets, with recent test suites marked running, passed and failed in a panel on the right

At a glance

  • ProductACT, an internal web app for regression testing configuration changes
  • UsersSolutions Engineers, expert users who test changes to customers’ configurations
  • RoleUX Designer
  • TeamSolutions Architect (subject-matter expert), Engineering Manager
  • ResearchQuestionnaire and feature-ranking survey with 37 prospective users
  • Visual designA new theme for a standalone app, specced for engineers in InVision

The problem

An untested configuration change could slow a customer’s website, or take it down.

Customers have thousands of configuration files between them. Before a change reaches the production network, Solutions Engineers test it on staging. The tool they had made that hard.

  • Test casesFinding and managing the test cases for a configuration was very difficult.
  • VariablesSingle and grouped variables were hard to manage and use correctly.
  • VersionsTest suites could not be versioned.
  • ReportsWeak test reports led to customer incidents.

Engineers knew the old tool well. The new one had to do much more without a steep learning curve, and bring their test cases across without loss.

My role

I designed the entire app, understood the problem and use cases from the Solutions Architect, reviewed proposals from a closed user group, and checked for feasibility at every point.

Research

We started from the product owners’ spec, then asked the people who would use the tool.

Brainstorming sessions with product owners and closed user groups set the goals and pain points. A questionnaire asked prospective users what was wrong with the existing tool, and how they expected its information to be organised.

The list of wanted features was long. So 37 prospective users, nominated by the product owners and the user groups, ranked each one on a scale from must-have to not necessary.

Whiteboard sketch of four views in a row: landing page, search result, test suite page with a run button, and test result with pass and fail marks. Below, notes on searching by config name, account and contract, and on what the test suite needs
An early session with stakeholders: the four key views, and what each had to hold.

Structure first

Before the screens, I mapped how the app’s information fits together.

Closed user groups and subject-matter experts checked that the structure matched how they think about the work. The database team checked it could be built from the data sources they had.

App structure

Accounts, contracts and configurations follow the structure engineers already knew from other systems in the company, such as Salesforce. Every configuration has its own test suite. A global template set holds test cases for each product feature, so ACT can suggest test cases for a configuration.

The app’s structure On the customer side, an account holds contracts, and a contract holds configurations. A configuration is an XML file with staging and production versions, and has one test suite. Under it are hostnames and CP codes read from the file, test cases and test results. The global template set holds products, each product has features, each feature has a test case, and each test case has test conditions. ACT reads the product a configuration uses and suggests its test cases. Customer side Global template set Account Contract Configuration XML, in staging and productionversions; one test suite each Hostnames read from the file CP codes read from the file Test cases Test results Product each with preset features Feature each with its own test case Test case Test conditions ACT reads the product a configuration uses, and suggests its test cases
Blue comes from the customer’s contracts, green from the template set; teal test cases connect the two. Swipe to see all of it.

Basic workflow

Find a configuration, manage its test cases, run a test every time the configuration changes, then read the results.

The basic workflow From the landing page, an engineer searches for a configuration. Search results match it by name, account or contract. In the test suite the engineer writes and manages test cases, then runs the test. The test result shows a pass or fail for each test case and condition. From the result, the engineer edits the test suite and runs it again. Landing Search results Test suite Test result Search for aconfiguration Match by name,account or contract Write and managetest cases Pass or fail for eachcase and condition Run Edit the suite and run again
The loop engineers repeat for every change to a configuration. Swipe to see all of it.

Key views

The workflow set four key views. Each was wireframed and reviewed with stakeholders before the visual design.

Landing page wireframe: the ACT logo above a search bar, and a list of recent tests on the right, each with staging and production versions, product, account and a pass or fail mark
Landing. A search bar, recently run test suites with their status, and favourites. The landing page wireframe. The finished screen opens this page.
Search results for an account with six contracts, each contract a card listing four of its twelve configuration files with a view all button
Search results. Configurations matched by name, account or contract and grouped by contract, with filters, drill-downs and breadcrumbs. Results for an account, grouped by its contracts. Customer names are blurred.
Test suite workspace: test cases listed on the left, the selected test case’s hostname, URL, request headers and lettered test conditions in the middle, joined by a logical operation, and host name variables on the right
Test suite. The workspace where engineers spend most of their time, writing test cases and their conditions. Reviews with stakeholders ranked the information into primary, secondary and tertiary levels, and the layout follows that order. A test case with three conditions, joined by a logical operation. Variables sit on the right.
Test result view: counts of running, passed, warning and failed test cases, each test condition marked pass or fail, and the DNS and HTTP request and response logs on the right
Test result. The same structure as the test suite, with logs and a pass or fail for each condition, so engineers can trace a failure to its cause. A failed test case: which condition failed, and the logs beside it.

Many more workflows went into the app than this page shows, such as test cases that apply to every test suite.

Grouped variables

Test cases in a suite reuse values, so engineers can save them as variables.

Some values belong together, such as hostnames and their paths. A grouped variable holds them as a table: a key, then any number of values. A test case that uses a grouped variable runs once for each value in it.

This was one of the complex workflows, and we brainstormed together to arrive at it.

Adding many keys and values to a two-dimensional structure had to stay easy. The wireframes walk through adding hosts, adding a path key, setting a value as a variable, and deleting a key.

Six wireframes of an Add Variable Group dialog: a host key with one value, then four host values, a path key added beside them, a value being set as a variable, the path key being deleted, and the host column on its own
Wireframes of the grouped variable interaction.

The full story

There is more to it than one page. I’m glad to walk you through the whole process on a call.