Greetings and welcome to a brand new segment on our blog: Under the Hood featuring industry experts. In this series, we’ll be peeling back the layers and diving deep into the exciting new features that we’re developing here at Foretellix.
To kick things off, we’re thrilled to have Bindesh Patel, Foretellix’s Senior Product Manager for Foretify Manager join us. Bindesh has been instrumental in spearheading the development of the new triage features in Foretify Manager, a powerful new addition to our product line.
Today’s interview takes place in a Q&A format, giving you the chance to gain firsthand insights from Bindesh on the inspiration behind triage, its technical aspects, and how it will benefit customers. So, put on your thinking caps and get ready to explore the exciting world of V&V triage.
Welcome Bindesh.
Thanks for having me. Foretellix offers a powerful platform called Foretify that lets you scale verification and validation (V&V) testing to handle a massive number of test scenarios. Alongside this, we introduced the concept of coverage metrics to objectively measure how complete your V&V process is. This was implemented with Foretify Manager acting as the central hub, reporting all these metrics throughout the V&V process.
Now, tracking completeness is important, but V&V projects also need to address the issues and bugs that get uncovered. Foretify expands the scope and depth of tests, along with the scalability, which naturally exposes more bugs. This is a good thing – we want to find bugs! But to handle this effectively, we need a whole new set of features. That’s why we recently added triage functionality to Foretify Manager. It automates the identification and clustering of all these issues into more manageable groups that are actionable.
“Actionable” means they can be easily assigned to team members for further debugging and analysis, and can also connect to other tools like ticketing systems. The benefit of all this is a boost in engineering productivity. Tasks that were previously done manually outside the workflow, in a fragmented and tedious way, can now be unified under Foretify Manager. Additional features have also been added to automate many of these triage-related tasks, leading to even greater efficiency.
Please explain the triage features of Foretify Manager
When you run scenario-based testing at scale, you typically get slammed with a huge number of errors, especially early on in the process. Foretellix technology lets us scale up the number of tests we run, and that naturally exposes more bugs – which is a good thing, we want to find them!
The challenge is figuring out which errors are the most common, and more importantly, which ones actually affect the system under test (SUT) and which don’t. Maybe they’re caused by the testing environment or other factors. Manually sorting through this at scale is nearly impossible, so it needed automation.
Foretify Manager tackles this by offering guided automation and data views that can be shared among users. This whole package minimizes the need for all those external tools like TDS scripts and spreadsheets.
Why is triage a significant pain point?
Triage in Foretify Manager is basically like giving you a toolbox to sort and group your issues. This is crucial. You can categorize them based on severity, the team or person responsible, priority, or even whether it’s a real bug or not. There are many factors at play, and Foretify Manager lets you tackle them all with this categorization system.
Now, this toolbox is built around the idea of “guided automation” through rules. You basically code these rules, and once they’re set up, they can automatically be applied to future tests and regression runs, saving you a lot of time.
How does the failure triage feature assist engineers in prioritizing and managing the large volume of issues reported from testing at scale?
Typically, users upload a test suite run to Foretify Manager for analysis. They’ve always done this to check completeness (coverage analysis) and analyze issues. The team would first use our features, which I mentioned before, to cluster and categorize issues into unique sets of similar problems that need further investigation.
This clustering is automated using pre-defined rules you create in Foretify Manager. It’s important because it saves time. Plus, the triage process is iterative, meaning the status of these clusters is constantly monitored and refined for quicker resolution – that’s the ultimate goal!
Once you have these automated clusters, they can be assigned to specific owners and even linked to other systems like JIRA, a bug tracking tool. After assignment, owners can easily use Foretify Manager’s existing debug features with just one click. This helps them get right to the root cause of the issue and ultimately fix the system under test (SUT).
Here’s another benefit: If a new regression test appears, say tomorrow or next week, Foretify Manager lets you compare it with previous runs using the same rules, clustering, and views. This helps you ensure your overall V&V process is improving over time.
How does the triage feature work with Foretify Manager?
We’ve mentioned a few features here and there, but let’s dive deeper into some key ones that make this all possible. One of the big things is the concept of attributes. These can be system attributes, like whether a test run passed or failed according to Foretellix. They basically act like key-value pairs.
Now, with triage, we let you create your own custom attributes. These can be set anywhere, from the runs table to the triage views we’ve discussed. But the main way to set them is through rules. You can define these rules either through the user interface or our Python SDK API.
These rules basically tell the system how to set attributes based on other data. The whole point of attributes and rules is to allow you to group similar test runs together.
Once you have these attributes set and the runs are clustered, you can use the customizable triage views inside Foretify Manager. These are basically hierarchical views of the clusters you’ve created based on all those attribute settings. They’re basically clusters that make sense for you – that’s the essence of clustering and categorization. And from there, you can take action on each cluster, as we discussed earlier.
On top of that, you can also set up user-defined aggregations. These let you easily answer complex questions, like “show me the collision rate in cutting scenarios where the speed was above 90 kilometers per hour.” You can easily set up these “aggregates” to display and analyze this kind of data. We also have powerful built-in filtering capabilities, which let you focus only on the relevant information.
All of this boils down to being able to compare test results from different runs, say from different points in time. This allows you to track the trends you’re interested in.
How does the triage feature boost efficiency and scalability in development?
Traditionally, test suite results are uploaded to Foretify Manager for analyzing coverage and key performance indicators (KPIs). Now, with this new built-in triage component, you can seamlessly perform triage tasks for your V&V projects within the same platform. It’s important to remember that without this, most of this work would be done manually and outside the system, maybe using spreadsheets or custom scripts.
Foretify Manager brings everything together. The key benefit? It eliminates data fragmentation, where information gets scattered and lost.
As we’ve discussed, you can set attributes within triage views. These attributes can define things like owners and even link to other tools like JIRA, a popular ticket tracking system. This way, individual owners can focus on the key issues assigned to them. They’ll see the corresponding JIRA ticket right there in Foretify Manager. With a single click, they can open that ticket directly in JIRA.
How do you recommend training engineers to effectively use the triage feature?
To help you get started, we offer a dedicated training module on failure triage. This covers both the methodology itself and how to use our tool effectively. It’s a great resource!
On top of that, our Applications Engineering Team (AET) has a wealth of experience in advanced V&V. They’re also very familiar with this new failure triage feature. They work closely with our customers to help them adopt both the capabilities and the methodologies around it.
Thank you for joining us today, Bindesh, and explaining the new triage features of Foretify Manager. Please visit the Foretify Manager page for more information.
