My first eval: automating feature request reviews

I built my first eval to see whether AI could identify feature requests that were ready to build. The experiment taught me how to compare model judgments, but also made me question what I was measuring, whose judgment counted as the baseline, and whether the requests were even meant to be executable.

Exploded plumbing diagram showing how the parts of a sink drain fit together.

Automate the boring stuff

My team receives dozens of feature requests every week. Pre-AI, these were all individually triaged, categorized, and reviewed. I was lucky to glance at a few of the incoming requests, much less thoroughly review, categorize, and respond to each. I used to let them pile up for months, then spend a day or two in a fever dream reviewing all of them. I’d proudly tell my team I was finally caught up. A month later, I’d be underwater again.

Some requests deserve serious product work, but many do not. Many requests are very small, unambiguous, and actionable. Dear Product Teams, you shouldn't accept easy fixes sitting in your backlog for months!

How I scored the requests

I built an eval: a repeatable test where I could give models the same requests I’d reviewed and compare their scores with mine. As part of this, I wrote a rubric to reflect my own evaluation process, then manually scored each request against three categories:

  • Clarity: Do I know what the ticket is asking for? If I can’t tell what should change, I can’t reliably judge whether we can or should build it.
  • Engineering feasibility: Could an agent reliably make the change in one pass? A small tweak to a well-understood part of the code is significantly less risky than a change that touches several systems.
  • Product intent: Instead of only answering if we could feasibly execute on a feature request, I wanted to review if we should execute on the feature. This was the hardest section to write, and likely the hardest criterion to define and score. I wrote a one-pager strategy guide for my area of the product. This included some practical context, like what features we planned to deprecate or replace within the next 6 months, and some higher-level principles for how we like to build products: deliver incremental value, slowly reveal complexity as needed, maintain consistent inheritance patterns, WYSIWYG, etc.

I added the three scores to provide an overall rank for each request. My starting goal was for an automated review to surface roughly the top 5% as candidates to build (though the actual percentage is arbitrary). If I later confirmed that those 5% were indeed great candidates for automation, I could widen the batch to 10%. If too many weak candidates passed, I could narrow it again. This approach starts off deliberately conservative, while anticipating we ship more and more features through automated workflows over time.

Feature request review tool showing a request, reviewer rationale, and scores for clarity, engineering feasibility, and product intent.
Front-end dashboard I built to quickly score feature requests and provide rationale.

What the scores showed

I did four runs. Between runs, I made changes to the rubric and context the evaluation models received, then compared their scores with mine to see what changed.

Some of the obvious fixes rose toward the top. That was encouraging, but the scores were still far enough apart that I wouldn’t rely on the shortlist yet. In the latest comparison, across five models, an individual score differed from mine by about 2.6 points on a 1–10 scale. As expected, product intent was where the models misaligned with me most.

Across all criteria, the five models’ average absolute score differences were between approximately 25% and 26% of 10.
Product intent had the largest average absolute score gap for every model, compared with clarity and engineering feasibility.

What a request leaves out

Problem vs Solution: Scoring the requests made me notice a mismatch. Our intake is designed to capture customers’ problems and deeper needs. That’s ideal for discovery, where we want to leave room for various solutions, but even a clear description of a problem can leave the solution wide open. A request can point to something worth solving without specifying a change I’d hand to an agent. That distinction helped me clarify the actual value of the automated review: finding the small set we could act on as written.

Two PMs can agree on what a request asks, but still disagree on the right solution, or whether we should automatically build it. Before treating the gap between my scores and the models’ as model error, I’d have other PMs score the same requests and see how far apart we are.

What I’d test next

In the short term, I intend to keep tuning the rubric, especially product intent, and see whether the strongest candidates rise to the top more consistently. Next, I’d like to pick a small batch of high-scoring requests, have an agent try to build them, and review the actual changes. Did the update solve the original problem? Would I keep it, rework it, or discard it? I should be able to use insights gained from actually building to improve the rubric and eval framework.

I’m also very interested in testing similar methods for different types of tickets. Bug tickets for example, seem much more deterministic in terms of expected intent. A proper bug ticket should document a feature that is not performing as it was intended, or in simple terms, is broken. The original intent of the feature should be unambiguous and clear - distinctly different from a feature request which explores a feature or tool that does not yet exist. I’m optimistic, I’d get much more aligned results with this type of evaluation.

What now?

I started this because I was tired of digging myself out of the backlog every few months, and tired of stumbling across requests with obvious fixes that should have shipped months ago. Some of the feature requests described real customer problems where we still needed to figure out what to build. My first eval gave me a way to look for the former, and the disagreement on product intent showed me why I wouldn’t hand it the whole backlog. I think there’s a lot of value in automating the obvious stuff, even if it’s only a small fraction of what comes in. Plus, as models get better and better, I want to spend my time designing processes that map to where we’re headed, not where we’ve been.

Onwards and upwards,
Abe