Case study
I turned an old retired laptop into a dedicated user-tester that logs into my apps with a throwaway account and drives them like a real person, fenced so it can reach only the app being tested and never anything real. In its first two runs it found bugs a code review wouldn't: two layout breaks on my portfolio site, and on another app a missing input validation plus a photo-upload error that was hiding a deeper problem. All fixed the same day.
User testing is my single biggest bottleneck. I build these apps, so when I test them I unconsciously walk the happy path I designed: the exact route that works. Real users don't. They land cold, click the wrong thing first, leave a field blank, paste something weird, reload halfway through, and open it on a phone. The bugs that matter live in those paths, and I'm the worst person to find them because I know where the landmines aren't.
I already had a box that probes my apps like an outside attacker: a security tester. But security testing and user testing are different jobs that find different bugs. The attacker box asks "can a stranger break in?" It never asks "can a normal person actually use this?" I needed the second box.
Two things shaped it: a user-tester holds a login and drives real apps, so what it can reach and what data it touches has to be bounded, and it must never become a path into anything real, especially running on old hardware.
A permanent user-tester that drives one app at a time as a real logged-in person, on the messy paths as well as the happy one, that can reach only the app under test and nothing else, and reports bugs in plain English, worst-first, with steps to reproduce.
The box. An old retired laptop, reimaged, running an AI coding agent with a single "user-test" skill and a strict brief: test one app, act as a real user, don't damage anything that isn't the test account's own data, and reproduce every bug before reporting it.
The isolation, matched to the threat. My security box is adversarial, so it's caged hard and treated as hostile. The user box isn't adversarial: it's me, logged in as a throwaway account, on test data. So instead of stripping its identity, I left it a normal trusted machine and fenced its outbound traffic to what the current test actually needs. The real isolation for user-testing is the test account and the test data; the network fence is the belt-and-suspenders on top.
The test accounts. A throwaway account per role: one that sees everything in its group (admin) and one ordinary member that sees only its own items, because those are genuinely different experiences, and a bug in the ordinary-user view is invisible to an admin.
The method: scope the app, arrive cold, work each core task the obvious way then the messy ways (empty states, bad input, reload mid-task, phone-width screen), verify every bug twice, and write one plain-English report worst-first.
The firewall taught me something the first day. I proved production was blocked and the app host was reachable, but my first cut also killed name resolution, so the agent couldn't even reach its own service to start. My "internet still works" check had used a raw IP address, which skips DNS entirely, so it missed it. The fix was a narrow carve-out for DNS. The lesson stuck: a fence has moving parts, and "it can still reach the internet" has to be tested by name, not by number.
A real browser changes everything. The first pass on my portfolio site ran without a browser connected, so it could only fetch pages and check links: fine for a static site's content, useless for how it actually looks. Once I connected a real browser it started clicking through the UI and seeing layout, and immediately caught two real layout bugs a link-checker never could: a "scroll" hint overlapping a button on the desktop hero, and the mobile menu's first link clipped behind the header. Both fixed the same day.
The second app is where it paid off. Pointed at a real app with a login, a vehicle-maintenance tracker, it logged in and worked the actual tasks: add a vehicle, log a service record, fuel, costs, parts, search, delete-with-undo. It found:
It was honest about its own limits. When its browser automation got flaky it told me plainly which "bugs" were its own tooling, not the app, and I confirmed two suspected issues were not real defects rather than "fixing" things that weren't broken. That restraint is the difference between a useful report and a noisy one.
I made the cost visible. Both test boxes run on my AI subscription, so I wired their usage into the same dashboard that already tracks every app's AI spend. Now I can see exactly how much quota testing burns, instead of guessing.