AI Isn’t Replacing Engineers. It’s Changing What They Can Accomplish.
The conversation around AI in software engineering often focuses on one question: Will AI replace engineers?
In our experience, that’s the wrong question.
The bigger opportunity is how AI can help experienced engineers build better systems, automate repetitive work, and spend more time solving complex engineering problems.
At Distillery, we’ve seen this firsthand. On a recent client engagement, one of our QA engineers used Claude Code to help design, build, and continuously improve a scalable end-to-end testing system. The result wasn’t just faster development, it was a more reliable, measurable, and maintainable QA process.
Building a QA System, Not Just Writing Tests
As the sole QA engineer supporting the Attorney Share project, the challenge wasn’t simply writing automated tests. It was creating a system capable of scaling alongside the product.
Rather than manually maintaining hundreds of tests, our engineer used Claude Code to help build a complete Playwright end-to-end testing framework from the ground up. That included page models, shared fixtures, authentication, CI integration, and testing against live preview environments.
Today, the framework includes approximately 500 end-to-end tests covering roughly 25 product areas, supported by around 90 specification files, 27 page models, and dozens of shared utilities.
Claude Code Was Part of the Engineering Workflow, Not a Replacement for It
One of the biggest misconceptions about AI coding tools is that engineers simply ask them to generate code and ship the results.
That wasn’t the workflow.
Every task began with an implementation plan and clearly defined objectives. Claude Code assisted with implementation, debugging, and iterative improvements, while every change was reviewed and approved by the engineer before it was merged.
The testing framework itself served as another safeguard. Instead of relying solely on unit tests, changes were validated through end-to-end tests running against deployed preview environments to ensure they reflected real user behavior.
AI accelerated implementation, but engineering judgment remained central throughout the process.
Extending Claude Beyond Code Generation
As the framework matured, Claude Code became more than a coding assistant.
Within the QA workflow, it helped:
- Read tickets and pull requests to understand what had changed.
- Review code from multiple perspectives to identify potential issues and edge cases.
- Analyze CI logs to investigate failures and identify root causes.
- Distinguish between product bugs, test failures, and environment-related issues.
- Support QA validation before work was approved.
Rather than replacing QA activities, Claude helped automate repetitive analysis while allowing the engineer to focus on validation, decision-making, and overall system quality.
Improving Speed Without Sacrificing Reliability
AI-assisted development also enabled continuous improvements to the testing framework itself.
Two carefully reviewed refactors reduced the end-to-end suite runtime from approximately 24 minutes to about 12 minutes, cutting waiting time nearly in half for every engineering team member.
The project also introduced Smart Test Selection, which analyzes pull request changes and recommends the tests most likely to be affected. Rather than immediately skipping tests, the system operates with built-in safeguards, including confidence thresholds, drift detection, and automatic fallback to the full test suite whenever necessary.
The emphasis remained on maintaining confidence in quality while improving efficiency.
Making QA Measurable
Visibility became another key part of the solution.
Using Claude Code, our engineer also built Exolar, a custom test observability dashboard that tracks pass rates, failures, flaky tests, stack traces, and CI performance.
Because Claude can analyze information from Exolar, it can help triage failures by identifying whether an issue is likely caused by the application, a flaky test, or the testing environment itself.
Instead of relying on assumptions, the team can make decisions using measurable data about test health and system reliability.
The Results
Today, the testing system includes:
- Approximately 500 end-to-end tests
- Coverage across roughly 25 product areas
- A 94.3% pass rate
- More than 42,000 recorded test runs
- Approximately 80 flaky tests identified and tracked
- End-to-end suite runtime reduced from roughly 24 minutes to 12 minutes
While those metrics demonstrate measurable improvements, they represent something larger than faster testing.
They demonstrate how AI can help experienced engineers build systems that scale.
Choosing the Right AI for the Right Task
One detail from this project highlights an important reality of AI-assisted software engineering.
Although Claude Code powered much of the framework’s design, implementation, and ongoing improvements, it wasn’t used for every task. For one continuously running inference inside Smart Test Selection, a smaller, lower-cost model was intentionally selected because it better fit the performance and cost requirements.
That’s an important distinction.
Successful engineering teams don’t build around a single AI model. They choose the right tool for each job while keeping engineers responsible for architecture, validation, and technical decisions.
AI Is Amplifying Engineering Expertise
The story of this project isn’t that AI built a testing system.
An experienced engineer did.
Claude Code became a powerful collaborator – accelerating implementation, reducing repetitive work, surfacing insights, and helping improve software quality – but engineering expertise remained at the center of every decision.
That’s how we view AI at Distillery.
Not as a replacement for engineers, but as a way to help experienced teams accomplish more, deliver higher-quality software, and build better systems for our clients. Interested in exploring how AI-assisted engineering can accelerate your software development? Contact us to learn how we help organizations build smarter, faster, and with confidence.