Playwright

If you develop software for the web you might want to consider using an e2e testing tool like Playwright or Selenium WebDriver. Why? Because e2e tests exercise your product like a real user would.

Let's say you're developing CRM software for paying customers. How will your customers be interacting with your software? First they'll probably need to open a browser tab, navigate to your login page, and login. Then they'll need to perform different workflows like:

• Adding a new customer to the system.
• Updating an existing customer's information.
• Archiving and restoring old customer records.
• Exporting data to Excel or CSV.

Those are just a few quick examples. You can probably think of more, like:

• Searching for a customer by their name, address, or phone number.
• Generating reports based on the customer data.
• Etc.

The point is, these workflows are how your customers actually interact with your system. Why do so many bugs reach production and get discovered by a customer instead of the original developer? One reason is because too few developers really dogfood their software.

As I've said in another post, dogfooding simply means putting yourself in the end-user's shoes and using your software just like they would. If your latest code change touches the "Export data to Excel" function, then you should probably perform at least one test export to Excel before you release the code to production.

That sounds like common sense, but you'd be amazed how few developers follow this principle. They think as long as they have a deep enough understanding of the data structures and code, they can safely make changes without introducing unwanted regressions.

Some developers believe unit tests are superior to dogfooding, because if you can verify that every individual function works correctly, then why wouldn't the system as a whole work correctly? Unit tests are good, and you should definitely keep writing unit tests. You should also have a way to measure your unit test coverage, and gate your CI/CD pipelines if the coverage ever drops below your desired thresholds. I'm not saying you shouldn't write unit tests.

With that said, if a customer is going to perform action XYZ on your site, in a web browser, then having an e2e script that can perform action XYZ in a web browser seems like the ultimate form of validation. If the e2e script is able to correctly export data to Excel, then you can be fairly sure (but hardly ever 100% sure) your code changes didn't break the "Export to Excel" feature.

It's true that dogfooding doesn't really audit for things like code quality, security, scalability, idempotency, etc. These things aren't always easy to spot during UI product testing. Obviously you'll want to have processes in place to audit those areas as well. At the very least you should be running Claude's built-in code review skill. It also helps to have a completely different model audit the code. If the code was written by Model A, you can still have Model A audit the code, but you might also want to have the code audited by Model B.

I've found code audits performed by Claude or Codex to be considerably more thorough than any human's. I'm just as guilty of a "LGTM" PR approval as the next reviewer. But an AI reviewer? Good luck, an AI reviewer will scrutinize the heck out of your code.

Anyway, back to the point of this post. That very high level of code review scrutiny is why having automated tests is so important. Every time an AI reviewer like Cursor Bugbot spots a valid "High Risk" issue in your PR, the code will obviously need to be updated before the PR can be approved and merged. Turns out, an LLM (just like a human) is just as likely to introduce a regression during the PR review patch cycle as it is during earlier stages of development.

In other words, you:

1. Developed a feature with Claude Code.
2. Audited the feature's code with Claude Code.
3. Confirmed the linter, type-checker, and unit tests all passed.
4. Dogfooded the code and verified it worked.
5. Created a PR to the upstream branch.
6. The ChatGPT-based AI code reviewer (ex: Cursor Bugbot) spotted multiple valid "High Risk" issues that needed to be patched before your PR could be merged.
7. You asked Claude Code to implement patches for the valid PR feedback.
8. You pushed your patches to GitHub and Cursor Bugbot marked its earlier feedback as "Resolved".
9. Once no new issues were found, you merged the PR.

Sounds reasonable, right? Here's the issue. The patches implemented in step 7 invalidated the auditing, dogfooding, and testing done in steps 2 through 4. Why would Claude introduce regressions with its patches? Why would any developer introduce regressions? It's not like it's intentional, but it happens.

Fortunately, running things like linters, type-checkers, and unit tests is usually pretty fast. Like, they only take a few seconds to run, fast. Dogfooding on the other hand? Good luck. Even if the manual UI workflow can be completed in a couple of minutes, the tediousness of repeating the same click, focus, enter some test values, click again, etc. process over and over is almost unbearable. Not only is it time consuming, it's also repetitive, and by their very nature developers and engineers do not usually like repetitive tasks.

The result is that the above-average developers will at least repeat steps 2 and 3 following any code updates to their PR in step 7, but I suspect very few would repeat step 4 (the dogfooding step). Feel free to disagree with me or point out every situation where an end-to-end test wouldn't be feasible. I get it.

One of the unspoken goals of our profession is figuring out how to maximize velocity AND quality. Instead of "move fast and break things", what we really want to do is "move fast and NOT break things". The problem is things will invariably break, and that tends to be true whether you're moving fast or not.

Another common regression pattern is when a developer doesn't realize their latest code change will break an unknown or seemingly unrelated part of the product. This is where a suite of simple end-to-end tests earns its keep. Verify the core user workflows, like logging into the application and performing a few common application-specific actions (if the product is a CRM, then there should be an end-to-end test workflow for adding a customer), and confirm that all of the tested workflows still pass.

To conclude, a more quality-focused AI-driven development and review workflow would have 10 steps instead of our original 9:

1. Develop a feature with Claude Code.
2. Audit the feature's code with Claude Code.
3. Confirm the linter, type-checker, and unit tests all pass.
4. Dogfood the code and verify it works.
5. Create a PR to the upstream branch.
6. The AI code reviewer spots a valid "High Risk" issue that needs to be patched before your PR can be merged.
7. You ask Claude Code to implement patches for the valid PR feedback.
8. You repeat steps 2 through 4 and confirm everything still works as expected.
9. You push your patches to GitHub and the reviewer marks its earlier feedback as "Resolved".
10. Once no new issues are found, you merge the PR.

Now that step 4 needs to be repeated some number of times (based on the amount of valid feedback our PR receives), it might need to be repeated once, or literally a half-dozen to a dozen times. For large product or feature PRs, which will become increasingly common with AI generating all of the code, the back-and-forth review process with a tool like Bugbot can require dozens of code patches.

Anyone familiar with the new AI PR review tools knows that Bugbot will not identify ALL of the high risk issues in its first review. As you patch its initial findings, it will usually find more, and you'll have to submit another patch, and this process repeats itself until Bugbot is finally content. Unless a machine is repeating your end-to-end workflows for you, every one of those patches is a fresh chance for a regression to sneak through.

There are obviously other ways of trying to optimize your workflow to avoid this pitfall, but there are usually pros and cons to any approach. While managing to avoid this particular pitfall you might find you've become trapped in quicksand.