14

Related:

Executive summary

  • Could the devs please make a weekly updatable dashboard with metrics outlined here shown in graphs and a table?

  • This information is vital for continued health of both graduated and beta sites, so it makes sense to implement it across the SE network.

  • Figures from the dashboard should not be relied upon in graduation decisions - the existing 10.0 QPD (Questions per day) threshold is a clear enough goal.

Ramblings

Last April Tim Post hinted at a site health dashboard being on the Kanban wall.

Yesterday abbyhairboat asked for ideas to re-invigorate community introspection after community Q&A surveys are phased outterminated with prejudice.

It looks like all SE communities need multifaceted, concise, objective, methodologically stable, ungameable measurements of site "health" to implement timely corrective actions. All too often I read meta questions "How are we doing as a site?" and all I read in response is opinions. There's some mighty disconnect between reality and perception and having a cartload of graphs and a table of fresh numbers without the hassle of going to SEDE and manually firing a one-off query will help bridge the disconnect.

Basically, moribund communities have to open their users' eyes and healthy sites have to stop worrying. I figured out some metrics and would be grateful to SE developers if they could bring the ideas to fruition instead of the rough and antiquated Area51 system which doesn't even deal with graduated sites.

For Stack Overflow, some of those indicators could be applied on a per-tag basis to account for fragmentation of SO.

The dashboard should be publicly accessible. Since it would not include the graduation QPD metric, people will be less likely to attempt gaming the system.

3
  • 5
    There is a lot of data that you want to gather and display. A lot of it sounds potentially useful, but I am not convinced that all of it is or that proposed thresholds are correct (for example, the "anything greater than 0 by CMs signals a disfunctional community" bit is misguided). I fear too many people would focus on numbers and numbers alone. We should be cautious about providing data for the sake of data. That aside, the goal of Abby's post was to look at things that can't be measured (e.g. subjective quality of posts). tl;dr: much more work to be done if we decide to go this route. Commented Jul 30, 2015 at 0:00
  • @AnnaLear - I'm afraid I missed the point. Totally. Sorry about that, must be senior moments here. I agree with you that Stack Exchange team does not need quantitative data becoming available to the communities. In this view I have voted to close this question. Commented Jul 30, 2015 at 2:34
  • 1
    To be clear - this isn't a bad idea in principle and you did excellent work identifying interesting metrics. We're just a ways out from throwing a bunch of them on a page. We are working on making more numbers available to high-rep users, so your proposal could be an extension of that down the road. We'll have to see what effect making more numbers public has and whether those particular ones are useful. Commented Jul 30, 2015 at 2:39

3 Answers 3

6

I think you're choosing far too many and too complex metrics. I'd personally be interested in many of them, but I don't think they would be all that useful in practice. The metrics we can actually measure easily often don't tell the whole story. The one big thing we can't measure is quality, and that is the most important metric.

I strongly dislike the current display of metrics for beta sites, and I think something closer to what you propose would be a large improvement. The big improvements would be to not show any hard thresholds and to show the change over time in metrics. The absolute values are often not that useful as they can vary a lot between sites, seeing the metrics change over time is far more useful.

I would use a set of simple metrics that indicate the size of the site and the active community:

  • traffic
  • number of questions and answers
  • answered questions percentage
  • number of active users
  • number of active users at important privilege levels (2k, 3k, 10k?)

The first two are simply measuring the volume of visitors and posts. This should have a generally upwards trend in healthy sites, or at least it shouldn't decline for long periods of time.

The number of active users is important to see if the active community is growing, a decline in active users is an indicator of a possibly serious problem with the site.

The last one would be useful to see how many users would be able to edit, close and delete posts after graduation. We already have the plain number of users at those rep levels, but I think knowing how many of those are still actively using or visiting the site would be helpful.

3
  • Aren't we already doing that in Area 51 stats? How would you define active users? Commented Jul 30, 2015 at 9:44
  • @DeerHunter The big difference is displaying the values changing over time. Growth or decline are much more important than absolute values. There is a mod-only statistic for active users, but I don't actually know what query that one uses. Commented Jul 30, 2015 at 10:14
  • 1
    Agreed on the value of seeing the trend. There are two reasons one can't use "simple" numbers: scaling and seasonal effects. Also almost agree on unobservable quality with the exception of "rich content" indicators. Think we are in agreement that a dashboard will be very helpful, even with two or three metrics at first. Commented Jul 30, 2015 at 10:20
2

When you said dashboard, this is what came to my mind; reminiscent of that SE mini-pocket-game: (hopefully I didn't miss the point)

enter image description here

Clicking it should show you a drop down with those metrics that need the most help.

E.g, 40 questions on hold (yikes); 132 reviews queued; number of edits in the last 48 hours; unanswered, zero score, first posts (may need all of the above)... HELP!

You want people to game the system, just in a good way. And they want to game it, so help them (want to) do it helpfully.

Basically it'd be lists of things that 'need attention': Show me the on-hold list, or Show me those short, 'poor answers'. – I'll fix 'em.

E.g. drop down list:

  • 47 questions on hold

  • 97 posts flagged for brevity

  • 12 in edit queue

I realize that this is basically how the review queue works, but I think the attempt should be made to make it more user assessable and obvious that community intervention can help solve these problems (or that the lack thereof is the problem).

I must admit though, that I completely ignore the 'new users first posts' list when it's at over a hundred (as does seemingly everyone else). IMO, someone who gets paid by SE can deal with all that non-sense.

2

I think that the term site health might not loved by some people as they could associate this term to the beta site evaluation. If I understand correclty the process was changed some years ago. I think that it should be fine to use nowadays this term.

Personally I like the term site health because I associate it to voluntary personal health check ups rather than a evaluation imposed with veiled purposes.

Another name migth be "citizenship level". Honestly I had this idea tumbling in my head since a while, not sure why, but I know tht it's not original, see Replace accept rate with citizenship level.


2023 state

The Sites listing, as it shows sites statistics, Users pages and the Reputation League pages might help to have an idea about the site health but they need some tweaks.

Lets take a look to Meta Stack Exchange Site Statistics as of March 22, 2023 :

Site Statistics

Missing Metrics

Some missing participation metrics that might be helpful

  • Votes Any kind of vote (Upvote, downvote, vote to close, vote to delete, etc.)
  • Comments
  • Edits
  • Flags
  • Active Chatrooms
  • Active Users By active users rather than using a reputaion score, I think that it might be more important to have being active in the site for certain ammount of dates during a time window, being active (post, comment, flag, etc.) during 10 days in last 30 days or something like that.

I'm not saying that they should be shown in the sites page, just that it might be helpful to make more eaisly available thant having to go to https://data.stackexchange.com. One way might be to have a site dashaboard page. On the sites page, it could be included a link to directly jump to it.

Missing Temporality

At this time it's not possible to see if the site health have changed over the time looking at the Sites listing as nowadays only shows "All Time" data. The Users pages have tabs to see week, month, quater and year data but it's not possible to compare with previous periods. The Reputation Leagues include features to see reputation changes, and see reputations from different periods (Curren, 2022, 2021, etc.) but it's not easy to see trends beyound the comparison with the previous period.

Data Explorer

There are queries from the data explorer that might help to have an idea but they do not really work as some attributes do not time temporal meta-data, i.e. it's not possible to compare how many active users were in month by month since the private beta to data explorer update date.

Posts with queries about the site activity, active users, etc.

Related to "diminsional modeling"

Qualitative data

As it happens in personal checkups, the qualitative data is important but it should not be used as the unique element to take decisions. For now I will no go deep in the qualitive part other than to say that the a quantitative dashboard is helpful and the qualitive part might be continue be handled in the corresponding per site meta.

Changes

Ideas, some are changes to current pages, some are related to create a dashboard. This is not a feature request, I'm just sharing some thougts.

  1. As mentioned above currently most of the aggregated data are site life time. It would be nice to be able to choose the period, i.e. week, month, quarter, year in the site statistic as could be done in the Reputation Leagues pages. This might imply a business intelligence approach ( use a different way to store the data, i.e. cubes), and might not be easy get historical data i.e. by mining the data dumps, but it will be nice that the Company start saving the new data in the proper structure / storage and in paralell work in adding data from the data dumps, probably this could be crowd sourced or might be handled through "connectors" / "plugins".

  2. Modify the Sites listing, Users pages above to allow the user to view data of multiple previous periods, 3 as minimum, i.e. Year: 2022, 2021 and 2020. Monthly: March, February and January.

  3. Add buttons / links to view the data as graphs to make it easier to see trends.

  4. Ideally allow site moderators to customize a publicly available site dashboard by turning on/off predefined scorecards, tables or graphs to help promote participation according to the current site needs.

  5. Users with access to the moderators tools should be able to see all the site scorecards, tables and graphs, including access to aggregated data, no details, about flags types and bans (post, review, etc.,) and users suspensions.

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.