Regression Testing Web App is a web-based tool designed to enable quick and easy regression testing of metadata changes. Solutions Engineers use it to regression test a metadata change on the Staging Network before it's pushed to the Production Network, to ensure that your change did not inadvertently break anything else in the metadata file. This is an internal app for superusers, which hosts a lot of complex Use Cases compared to the customer-facing app.
The Problem
Untested configurations can impact a customer's website performance or sometimes have the potential to completely pull the customer's website down. Challenge was to build a sophisticated app where engineers could perform complex network related tests with minimal cognitive load. Supporting tools to troubleshoot, build test cases to be easily accessible and contextually made available.
Engineers were already used to an existing app and we had to learn a great deal from their existing mental models and usage patterns, and craft a solution with more capabilities without leading to a steep learning curve for engineers.
Also, migrating test cases & other data from the older apps to new should be seamless
Problems with the existing Application
- Finding and managing test cases per configuration is extremely difficult.
- Managing variables (single + grouped) & using them appropriately
Versioning the test suites was not possible. - Customer incidents happening due to poor Test Report features.
Discovery
This is the primary phase of this project where time was spent on understanding existing approaches, usage patterns, and users' mental models. We started with a functional spec document provided by the Product Owners which mentioned what the goal of this app is, Use Cases, and required features.
Various brainstorming sessions were conducted with Closed User Groups and product owners to understand goals and pain points.
2
Questionnaire
Questionnaires were circulated amongst closed user groups and prospective users to understand:
a. Problems in existing app
b. Information Hierarchy
1
Feature Ranking Survey
We had a huge list of features to be developed for the new app, so we did a feature ranking survey amongst users to find the most-in-demand features. Participants were asked to choose from from a scale of 'must-have' to 'not- necessary'
37
Participants for survey
37 prospective users were nominated for these surveys by the Product Owners and members of Closed User Groups. This was a decent chunk of users to understand the pulse of requirements.
n
Brainstorming Sessions
There were several brainsotrming sessions conducted to understand and groom the requirements.
Fundamental Features Required
The following features are the key-functional features that had multiple sub-features within each which is not covered in detail in this case study. For eg., Global Test Case Management requires a workflow that enables super-users to create Test Cases that can be applied to all Test Suites (all configurations).
Multi-dimensional search feature
We have numerous customers with multiple configuration files (ranging from thousands across all customers). A User should be able to search the right configuration by various dimensions like Customer Name, Account ID, Contract ID, hostnames, and a few other parameters.
Multi-dimensional search feature
Once a test is run, all Test Cases and its Pass/Fail statuses along with appropriate technical messaging for engineers to further drill down the cause for failure.
Powerful Testing Workspace (Test Suite)
For every configuration, Engineers will require a space to create and manage various test cases and their conditions.
Test Result page
There are various test cases that could be applied to various configurations that contain variables that can be tuned to a per configuration level. A space to manage this.
Information & Interaction Design
A very important phase of Design is where we structure the task-oriented hierarchy, proximity, and grouping of elements and the navigation scheme through methods like card sorting, information architecture diagrams, key views of an app that also communicates the information placement w.r.t its hierarchy and more.
Sometimes we rely on principles of Entity-Relationship Diagrams & Data Modelling. Usually, some UX teams ignore this aspect as it is assumed to be a technical task done by a Database expert. However, as a practice, we involve SM E's, closed User groups and the DB experts during this phase of the project. Here we look forward to validation of these models from two major vantage points:
Dependency
How Users perceive the hierarchy of elements about a certain process and its alignment at the organization level.
Structure/Hierarchy
DB experts, depending on the kind of Data Sources available to them, validate to tell if the proposed models are workable.
Information Architecture Diagrams
There were 2 main aspects of Information Design in this project, that is:
App's Fundamental Structure
This structure defines what the app encompasses at a high level. It acts as the baseplate to identify and build the key views upon. Since this is an internal app being built for Engineers, it is important that the structure is coherent with other established systems (like SalesForce, etc.,) in our org.
Test Suite Structure
Test Suite is the heart of this application where all the action happens, it is critical to building a widely accepted, understandable information structure so that it makes sense to core technical users.
Key Views’ Wireframes
Once the information architecture is finalised and signed-off, wireframes of key views of the app is prepared to replicate Information hirearchy & flow in the layout. The wireframes are reviewed with stakeholders to ensure correct Information hierarchy and structure.
I. Landing Page
3 main elements were a part of the landing page:
- Search Bar
- Recently Visited/Run Test Suites
- Favorite Test Suites
II. Search Result Page
As there is a need to search for configurations or Test Suites by various attributes, we added filters, drill-downs, and breadcrumbs at the Search Result level to aid easy search.
III. Test Suite Page
Heart of the entire app where Users spend most time by creating Test Cases and Test Conditions. We have a "workspace" kind of view where all the information required to add Test Cases are available handy. The layout was reviewed with the stakeholders to identify Primary,
Secondary &Tertiary levels of information required, respectively the information was logically placed on the layout.
IV. Test Result Page
Result-page contained a lot of information for the Users to investigate Test Case failures. Efforts were made to keep the Test Case structure/navigation similar to Test Suite page, but with Log information, and Test Condition Pass/Fail results.
Workflow Design
Planning workflows and interactions is another critical phase of Product Design where we map how a User accomplishes a certain task, and at what step of a workflow what actions are available to the User and to which User Types. There were many workflow diagrams prepared. The first step is to identify and list the tasks a User would have to accomplish this app, and its corresponding workflow is charted. Only in cases where the interaction is complex will we prepare a prototype to explain the workflow to stakeholders.
Basic Workflow
At a very high level, the basic workflow for this app is that a User finds a configuration, creates & manages test cases, and runs a test every time there is a new change in the configuration file.
Grouped Variable Creation
As there will be multiple Test Cases within a Test Suite, there is a need for re-use of various elements which gave rise to the need for Users to be able to create Variables. Adding to this feature, there was also a need for Grouped Variables as there were certain values to be added as a group and looped between multiple Test Cases. For Eg."Hostnames" and their 'respective paths".
If any value in the Test Case uses a "Grouped Variable" name, that particular Test Case runs multiple times depending on the contents of the number of variables in the grouped variable list.
Grouped Variable is a 2-dimensional structure and we had to come up with a solution so that Users could add multiple [key:valuel:value2:::value'n'] pairs with easy and intuitive interaction.
Visual Design
It was not required for us to be compliant with an existing Visual Design system as this was a standalone app, hence we came up with a different design theme for this app from scratch. Also, as this is a bespoke design theme, a Design Guide was not built and the design specs were communicated to the Front End Engineers through Invision specs.
Mock-ups
This document is not complete but large, and a lot to consume. Thanks for your time and patience to go through my work. Please reach out to me to discuss further.
Thanks for your time!
Let's Talk
+971 509076140 (UAE) OR +91 9379416140 (INDIA - please message on Whatsapp)
chetanshridhar@gmail.com