How to always win by conducting a Proof of Concept?
What is the purpose of conducting a Proof of Concept? What should I consider before conducting it? What conclusions should I expect to obtain?
For those of us involved in Appian projects, we see how the dedication of testers needs to be adjusted in specific aspects: spikes in deliveries, gradual increase for regression testing, etc…
And when we have to start a project, in honesty, quality is usually not of the highest priority, neither is manual testing nor automated testing (check our articles about this on this blog), but…
But we know it's going to be something fundamental and will require a specific resource load that we need to measure, and we know what happens if we don't get the measuring right!
If you have a lot of experience in Appian projects, you can accurately estimate how big your QA team has to be. Generally, if the development team is experienced, the requirements are more or less in good shape, and you don't automate, you'll need approximately 0.4 testers per designer.
And if you do automate... Do you have enough experience with the product you're going to use to estimate the size of the QA team? Generally, the answer is no, either because you have experience with the product but not in Appian projects - where the pace of change is higher than in other technologies - or vice versa, you have experience in Appian but not with the product.
This is precisely the objective of Proof of Concept (PoC).
With a PoC, you need to confirm the path you're taking, how agile it makes you, how quickly you progress, how it adapts to your methodology, how the entire team participates, and the return it will give you.
STAGES:
A PoC is almost always a small project, and if you want to get the most out of it, you need to prepare it well.
You'll have to progress through a set of stages that I'll briefly explain.
General preliminary analysis
You need to consider the project's characteristics.
For example, there is a big difference between a new phase of an existing project and starting a project from scratch. There's a big difference between inheriting a project from another provider that hasn't been renewed and a project to change the infrastructure of an application.
You have to assess the quality of the requirements, the volume of changes, the team's experience, the business objectives, calendar and budget constraints... everything!
With this information, you can design the framework in which to execute the PoC and a first set of requirements you'll make to your tool.
Selection of the typical case
If in the previous point we touched on more architectural aspects, here we'll dive into the practical aspects of design: you have to choose the typical case that will represent the highest percentage of use.
It's not effective, when searching for automation tools, to think of especially complex cases if they represent an exceptional situation in the project; there's always someone who suggests that automating is a very difficult functionality.
This, in general, is a waste of time. If something is very difficult to automate, it shouldn't be automated, period.
The goal of automation is to be able to cover a high amount of functionality very quickly and make it easy to maintain and adapt to changes that arise. This is where you'll have a truly significant impact on the return on investment.
Exceptional cases have no impact on the overall outcome.
Preparation and requirements for the PoC
For example, if your process is analyzing operations that need validation:
- Choose a high-impact operation, as they usually initiate these types of flows.
- Measure how quickly a tester without programming knowledge can automate it. A regular screen shouldn't take more than 5 minutes.
- See how easy it will be for a Product Owner to understand the generated test. They should be able to recognize the actions and validations present.
- Ensure that you can create the test without having to wait for the designer to create their content, and that you can adhere to a BDD methodology without issues.
- Validate how it integrates with your user stories. You should have a functional view.
- Test how it generates evidence: word files with screenshots, videos of navigation, integration with test suites like ALM, etc…
- And now, have each team member try to execute the test: designers, analysts, the Product Owner, and testers. Everyone should be able to execute a test.
- Simulate a requirements change
- Count how quickly anyone in the project, starting with the analysts, can find the tests impacted by the change (impact analysis).
- Have anyone on the team measure the degree of impact. You don't want specialists; you want quality to be present throughout the team!
- Measure the time it takes to make the change; of course, it should always be much shorter than on the Appian side, shorter than the impact of creating it from scratch.
- Evaluate whether the change on the Appian side was made on a reusable component and if you had the same level of reusability in the test.
- Now simulate an Appian version change.
- General-purpose testing tools don't understand Appian and don't know what it means for all components to change their internal structure overnight. Every time there's a version change, automated processes may stop working.
- AAT has a configuration parameter where you indicate which version of Appian you're using, supporting from 18.3 up to the newest versions. Version changes are transparent to you.
- Now evaluate the "industrialization".
- Budget delegating the creation of a test to an external professional.
- Measure the learning curve for the different profiles available to you, whether in your company or in the market, with or without programming experience.
- Evaluate what it entails to add new environments, migrate to a local environment without an internet connection, whether you need databases or other infrastructures, etc.
- Consider how it will integrate with a DevOps pipeline, if it can be executed in Kubernetes (or similar) to scale indefinitely.
- Budget the licenses by calculating the involvement of the entire team and without limits on parallel executions.
- Make sure that each team can run their complete tests every night to discuss everything during the daily meetings.
- And finally, prepare your numbers:
- Prepare KPIs to calculate the test creation and update time.
- Create the cost model for training, licenses, and execution.
- Ensure alignment with BDD methodologies and others that may be required.
- Measure the incidents you have before and after automation, and their improvement ratio.
- ...
Recommendations
- Have an expert guide you and help you start on the right foot: whoever automates tests in Appian needs to know the platform and the characteristics of Appian projects.
- The tool needs to recognize Appian platform objects.
- Version changes (at least 4 per year) are critical for your test, and your automation solution needs to guarantee an almost negligible impact. General-purpose tools usually fail in this regard.
Conclusions and Decision Making
As you've seen, a PoC is not trivial.
Once you have it defined and have executed it, you should receive a summary of the PoC from the provider and prepare the conclusions and focus on decision making.
We always prepare a report with the discussed details. Our team knows Appian and understands the projects required of us, which is why we help our clients evaluate their conclusions, emphasizing the risks that general-purpose products completely ignore.
In any case, with our PoC, you'll gain an innovative perspective carried out by a team of professionals dedicated exclusively to Appian projects.
When you need it, with AAT, you'll have a specialized tool ready to help you, allowing you to maintain a dedication to quality proportional to the development team, not the project's history.
And remember,
not everything needs to be automated. There are as many adoption models for a tool as there are projects. What's important is that in each model, the return on investment is substantial.
You should seek maximum flexibility and not be tied to the idea that if you don't automate a certain quantity, it's not worth automating.
