Stack Exchange has recently been working on improvements to the process by which you escalate issues to our attention and by which we are transparent about what we work on in kind. Our transparency reporting process is described in this post, but it is long overdue for critical changes. Importantly, we are rethinking what metrics we’ve been reporting out.
You’ve probably already seen some of the efforts meant to address the growing backlog that have been taking place in this past year. These include the Community Asks Sprint initiative, Slate’s post clarifying tag usage, and our recent rekindling of a bug duty rotation for Public Q&A (ok, that last one maybe you’ve only noticed if you’re looking very closely, ‘cause there is no accompanying Meta communication). In tandem with those, we’ve been working on a new way to report how we’re faring in responding to your requests. So, without further ado...
Introducing a new metric for measuring backlog health: P80s
The new metric we’ll be using internally to measure how we’re performing and to help surface issues that need attention is the “P80 metric.” P80 expresses the idea that “80% of all work items should be handled within X weeks,” and that most items should not sit indefinitely without a resolution. That X will vary for each of the three process tags (as defined on Slate’s post clarifying tag usage). For instance, to reflect the belief that we should triage issues in a timely manner even if it then takes us some time to reach an outcome for them, status-review has a shorter P80 target than the other two process tags.
Given the age of some of the issues currently in our backlog, it’ll take a few quarters before we meet what we consider to be reasonable thresholds for each of these tags. This means our initial goals won’t be to meet the long-term targets, but rather, the time will be spent getting the backlog under control. Our primary aim is to make sure it doesn’t keep aging and that the P80 doesn’t keep increasing — and only after that will we be in a better position to start improving our response times. In the long run, we’d like to be able to target 8 weeks (4 sprints - our first response time) for status-review, 16 weeks (8 sprints) for status-planned, and 40 weeks (20 sprints) for status-deferred. Please note that these are tentative goals, and that once we’re closer to meeting them, they might still be subject to revision depending on internal resource allocation, in the same way that the previous reporting was subject to adjusting depending on those same factors.
As an example of how this work in practice, using an 8-week target for status-review:
- Assume there are 100 posts with the tag.
- Each post has had the status-review tag for some number of days.
- Order all the posts by the number of days they’ve been in status-review.
- The 80th post on that list determines the P80 metric.
- For example, if the 80th post has been in status-review for 7 weeks, we are meeting our target. But if the 80th post has been in status-review for 9 weeks, we are not.
This metric acknowledges that some issues will simply take longer — possibly much longer - than the target time to finish. This is a normal and natural part of doing business. However, 80% of the issues pending in any given week should be less than 8 weeks old.
Note that transitioning an issue to a state it’s previously been in doesn’t reset its time in that status. As an example, suppose a post spends 2 weeks in status-planned, then 6 weeks in status-deferred. If the post is then moved back to status-planned, it will begin counting up from 2 weeks, not from 0 again, reflecting the total amount of time the issue has spent in status-planned. This prevents us from deferring handling an issue indefinitely on accident by moving it between statuses. We expect that folks will sometimes make small prioritization errors they later correct, and this should be accounted for seamlessly.
Reporting P80s
External reporting will take place on Meta, in the same post where it’s taken place historically. It will still take place once a quarter, and will be composed of an aggregate report. A report will look something like this:
| Status tag | Posts in tag | P80 last quarter | P80 current quarter | Goal (if applicable) |
|---|---|---|---|---|
| Review | xxx | xxx days | xxx days | xxx days by yyy / decreasing trend |
| Planned | xxx | xxx days | xxx days | xxx days by yyy / decreasing trend |
| Deferred | xxx | xxx days | xxx days | xxx days by yyy / decreasing trend |
Alongside this, we’ll also include:
- The past two years of P80 data for each tag in chart format.
- A list of important Meta posts we’ve addressed as a part of this process
- Escalation guidance (which generally will match the quarterly roadmap) for what should be escalated on that quarter.
Conclusion
We hope you see an improvement in this change, and that it revitalizes trust in how we handle your escalations internally. Of course, we don’t think changing what we measure, in and of itself, will solve the accumulating backlog of reports, but we do believe that you can’t control what you can’t measure. This metric provides us with clear, actionable steps to ensure success, and those steps are broadly in agreement with what we feel this process needs to meet the needs of the user base.
I should be clear about the fact that these metrics are primarily internal-facing and that we share them publicly for transparency and because we want to hold ourselves accountable to you — this means the internal reporting will be taking the form described above. As such, while I’m open to small corrections, requests for clarifications, and such, I’m much more interested in serious objections to the plan laid out in this post as it relates to the public reporting specifically.
Leaving answers instead of comments is generally easier to thread discussions, so please err on the side of doing that, even if the answers are short. Splitting your concerns to one topic per answer also makes following the discussion much easier.
Some of the details might change between now and a final version (which will be taking shape as an edit to the overall process post), but unless y’all feel I’m going completely in the wrong direction with the public reporting, this is gonna to be how the process will look in the future. Lemme know your thoughts below!
