After a release
Run the suite against production straight after a release. If checkout, search or login broke on the way out, the result names the scenario and the step that failed.
Testing Suite · Early access
Smoke-test your key journeys after every release, in real browsers.
Your site changes every week. Run Testing Suite after a release and it re-runs your customer journeys in the same real browsers, with a result for each one. When checkout breaks on the way out, you see the step that failed, with the session video and logs to fix it. Automatic release gates and Web Vitals budgets are planned.
Testing Suite is in early access. Written for the engineering team? Yes. Forward it.
What it does
Build once. Re-run after every release.
Performance testing tells you whether your site stays fast under load. That's once a release, maybe weekly. The other 95% of the week, you ship code without testing what users actually see. Testing Suite is built for that 95%.
After a deploy, run the scenarios you already built (homepage to checkout, search and add to cart, login and dashboard) as one suite. Same real browser. Same isolation. Each scenario runs once and they all start together, so a suite of your scenarios is smaller and faster than a full performance test. Suites run from the UK or the EU, with your choice of locale, time zone, desktop screen size and pacing.
Scenarios can check what a customer should see along the way, such as the URL, the page title, a line of text, an element and how many there are, an attribute, or the value in a field. When a required step or check fails, the scenario fails and the report names the step. That is the layer unit tests miss. Today a failed run can email your team; with the planned GitHub and GitLab integrations, it will be able to halt the release as well.
And because the scenarios are the same ones you performance test with, there is one place to maintain them. No duplicated test code between QA and performance testing.
What you get back
A verdict that links to the evidence.
Every run you let finish ends in one of three results. Passed means every scenario passed. Failed lists the scenarios that failed and, when a step or check broke, names that step and the error it hit. Errored means a scenario produced no result at all, a problem on our side, and it is reported separately from failures.
Open a failed scenario to watch its session video, step through a timeline that jumps to the failure, and read the console and network logs. The bug that broke the journey is one click away. Then re-run the whole suite, or only the scenarios that failed.
Results can go by email to your teammates, after every run or only when something fails. Each suite tracks its pass rate, streak and run duration, keeps a scenario-by-run grid of its recent runs, and flags scenarios that flip between pass and fail as flaky.
Where it fits
Six ways to use it today.
Each one builds on scenarios you create once in Evaluat, with session video and logs for the scenarios that run.
Pre-release smoke
Build a suite from your staging journeys and run it before you promote a release. Green means every journey still works; red shows you where one stopped.
Before a sale goes live
Schedule a one-off run for the morning a campaign starts, up to four weeks ahead. Your team can get the result by email, or hear only if something fails.
After a platform update
A Magento upgrade, a new extension or a theme change can break checkout without anyone noticing. Re-run the journeys that take money and see which step stopped working.
Regression hunting
When a journey starts failing, the scenario-by-run grid of recent runs shows the first red run. Open it to watch the video and read the logs.
Flaky scenarios
A scenario that both passed and failed in recent runs is flagged as flaky. Re-run only the failed scenarios from the report, and fix the noise instead of ignoring it.
Roadmap
What's planned.
None of this is built yet. We'll update this page as each piece lands.
GitHub Action Planned
Will run your suite against a pull request's preview deployment and post the result as a comment.
GitLab CI Planned
A template that will drop into your pipeline and run the suite after every deploy to staging or production.
Threshold rules Planned
Will capture Web Vitals on every suite run, with per-page budgets such as LCP under 2.5s, INP under 200ms or no console errors. A run will fail when a budget breaks, and your pipeline will be able to halt or roll back.
Webhook API Planned
An API and webhooks that will start a suite from anything, such as custom CI runners, deployment hooks or internal tools.
Diff reports Planned
Will compare each run side by side with the last green run, so you will see what got faster and what got slower.
Slack notifications Planned
Will post failure alerts to a channel, naming the broken scenario and the failed step, with a link to the session video.
Common questions
Testing Suite, answered.
What is the Evaluat Testing Suite?
Testing Suite, now in early access, re-runs the real-browser scenarios you already built in Evaluat as post-deploy smoke checks. Each scenario runs once in its own real browser, and when a step breaks the report shows which one, with the session video, console logs and network logs. A suite is smaller and faster than a full performance test.
Will Testing Suite work with GitHub Actions or GitLab CI?
Not yet. A GitHub Action that will run your suite against a pull request's preview deployment, and a GitLab CI template, are both planned, along with a webhook API for custom runners and deployment hooks. Today you start a suite in Evaluat, on demand or as a one-off scheduled run up to four weeks ahead.
What does a failed run mean?
At least one scenario failed. Usually a required step broke or a check did not match, such as missing text or the wrong page title, and the report names the step and shows the error, the session video and the logs. A scenario that produced no result at all is marked Errored instead. Failing a run on Web Vitals budgets is planned.
Can I schedule a suite?
Yes, as a one-off run up to four weeks ahead, for example the morning a sale goes live. Results can go by email after every run or only when something fails. Repeating schedules, such as nightly runs, are planned.
What can I use today?
Performance Testing is available through assisted private access. Testing Suite is in early access. A suite re-runs the scenarios you build for Performance Testing, so nothing is thrown away. Run it on demand or on a one-off schedule, re-run only the failed scenarios, and email the results to your team. CI integrations, Web Vitals budgets and Slack alerts are planned.