Avoiding half-baked experiments: Setting clear standards for experiments and rollouts, beyond “Minimum”

These are good questions, but the answer will vary for each MVP (== minimum viable product). I completely agree that answers to these questions should be determined, or at least considered, for an MVP. Yet, each experiment is likely to yield different answers.

I'm going to largely focus on how we reach those answers, rather than what the answers are. To some extent, this answers #3 - community consultation, but focused on collaborating on a MVP definition. Phrased differently, "How can Stack collaborate with the community to answer these questions before starting development of an MVP, or in the early development stages?" Crucially, this aims to proactively spot areas of improvement for an MVP before problems arise.

MVPs have a scope defining what does and doesn't need to be included to release a product/feature (either to the general public or external testers, e.g., an opt-in group).

In theory, this is good - for teams, it provides milestones. For users (incl. beta testers), it communicates what to (and to not) expect.

One common complaint is "why isn't X done?". This is often a consequence of mismatching expectations, and can often be reframed as "I think X is necessary for <product> to meet my needs". Some of these X's are minor, like this - nice to have, but by no means essential. Others are more critical, and if ignored, risk disfunctional releases.

Ultimately, these have a common root cause - lack of consensus between internal people (Stack employees) and external people (random users, Meta folks, stakeholder groups, etc.).

This lack of consensus is visible - here's a handy example in the context of Discussions:

[...] We, the community and this project [Charcoal], have previously given our feedback as to what's needed. The company has repeatedly decided to move forward without what I'd consider to be the bare minimum. They've, to a limited extent, seen that it can get bad, but haven't been willing to do what we consider to be the minimum. So be it.

_{Chat message by Makyen from CHQ; quote adjusted for clarity.}

This is by no means the only example, and this isn't unique to Discussions. While I won't go into it on Meta, here's another example.

Adddressing the lack of consensus

The key idea above is that a lack of clarity and agreement for what's necessary in any given MVP leads to conflict and issues. This is both a people issue - folks (both from Meta and from Stack) can feel disrespected/ignored, and a product issue - something not working.

An idea: public-facing MVP proposals

This builds on my response to the What's on your mind? question. A major focus of my response there was improving communication between the community and staff members (esp. ones who don't regularly use Meta). That sentiment - that genuine, kind, and constructive conversations really matter - is relevant here.

The next time a significant (in terms of scope) feature is coming, why not publish a brief MVP proposal and ask for feedback specifically on the MVP plan before releasing the feature itself?

Not if the community likes the actual feature, but for suggestions/advice on the path to implementing it. This wouldn't need to be a big featured MSO/MSE post, creating a chatroom (or using The Meta Room and/or Tavern on the Meta) to get some suggestions could work.

Using something like chat can be significantly more approachable - you're not going to get tons of people downvoting your proposal, but rather (hopefully) a real-ish-time conversation gathering feedback to help make it a success.

Curation

cocomac's answer is right in general, but there's one absolute must for every MVP, no matter what the experiment is trying to test: No. More. Spam.

Stack Overflow's Discussions beta had a really bad spam problem. It got so bad that A proposal to stop curating/moderating Discussions until it's "fixed" eventually reached a net score of +135. At one point, I was afraid to curate Discussions in public in case someone was looking over my shoulder. There were several factors involved in the experiment's general state, but most of them fell under the umbrella of poor curation. No API endpoint for SmokeDetector, poor moderator tooling, no reputation requirements, nothing. This led to a deluge of spam, which routinely lingered for days or weeks.

Discussions was a particularly bad case, but curation as a guiding principle needs to apply to every experiment. That doesn't mean curation as strict as what we have on main Stack Exchange sites, but there must be a way to check spam waves or low-quality content floods regardless of the specific experiment. Appropriate curation and/or moderation should be a prerequisite for any MVP. If the "minimum viable product" is highly susceptible to spam waves, it's not really viable at all.

Slightly harsh viewpoint? Folks are chasing things that may not exist, and there's a certain degree of corporate Attention Deficit.

Part 1: A long time ago... in a Q&A network surprisingly near

The 'Core' products of SE had very clear goals, often borne of community requests. Its worth looking back at the three core components of the network

For SO proper before the trilogy

In the words of Jeff

There’s far too much great programming information trapped in forums, buried in online help, or hidden away in books that nobody buys any more. We’d like to unlock all that. Let’s create something that makes it easy to participate, and put it online in a form that is trivially easy to find.

Its surprising that the old rivalry continues , but stack oveflow was also described as

Stackoverflow is sort of like the anti-experts-exchange (minus the nausea-inducing sleaze and quasi-legal search engine gaming) meets wikipedia meets programming reddit.

Meta was grudgingly instituted, and sort of was built initially by community. While some aspects of that initial vision did not make it through, there was a clear idea of what meta was.

And of course chat.

With all these - SE identified a problem their target community had, looked at the solutions they had, and polished them. In many cases, the community already had their own paths laid out - whether it was the pre-meta forums, or IRC/Basecamp for chat.

Part 2 Warlords and waffles

Now, many of the failed projects were attempts to write further chapters for their own sake. Now, great people worked on many of these, and with my focus generally on the rest of the network, I might get some things wrong. Feel free to correct me.

For Documentation, the foundational document that describes it seems to be this MSO post - Warlords of Documentation.

We're starting off with a somewhat abstract problem - documentation sucks. We have an examination of why and a possible solution. That said, to an extent SO didn't take an existing solution, or formalised an informal solution, they were trying to build something new and the clarity of purpose or direction wasn't there. It was also very much SO/dev focused rather than being a product that broadly stood on its own.

As an established platform I guess more than "what new thing do we do?" the question we might want to ask is "What do our users want and how do we delight them?".

I use SE as documentation so - the other question would be "what can we do better than anyone else?"

We still do Q&A and it feels like putting that on the back burner to chase other things was and is a mistake.

Part 3: Discussions - the chat we have at home?

Before the sudden and welcome recollection we have a fairly robust chat system, Discussions was pushed as the replacement for Chat.

In a sense it was a way to try to mitigate the somewhat strict and possibly arcane seeming rules that grew organically around the main Q&A site but it is a very strange fit for a site, and network that prided itself on being low noise and the exact opposite of forums.

Unlike the other main components of the network, and maybe a little like documentation, it didn't formalise an existing informal tool, nor was it aimed at doing something 'better'.

I also felt Docs had a clearer idea of what they wanted to build.

Part 4: The Comments redux - the return of the discussion board

To me, it feels like the 'new' comment formula is trying to turn the main site - designed to showcase questions and answers into more of a traditional discussion board. There was a proposal some time back for renaming comments that was fairly popular and got lost in the shuffle. There's something there I feel needs extra emphasis in context

The basic building blocks of every SE site are the QUESTIONS and the ANSWERS. They are clearly portrayed as such, so if anyone puts "something else" in that space, it's such an obvious UI gaff, folks have no problem spotting it and fixing it — we have nearly 100% compliance.

For a UI change, the 'new' comments design... elevates a comment to the same level as the question or answer. I like parts of it but I'm not sure what it solves for an existing user (who is used to the existing UI) or a new user who is lead to believe a comment is equal to the question.

Part 6: We need to know where the river meets the sea

I have often wondered that if these products succeeded, what's the intended end goal. Sometimes it feels like the approach is to build things and hope they work, rather than how it'll integrate with existing products and communities. We might be a little too old for the excitement of testing in production, especially if it's a drastic change.

In very recent memory, the chat link on the sidebar was a 'small' change that went brilliantly, and I'm looking forward to the chat updates but these are new coats of paint on a existing, established product that's looking a little shabby. In a sense, its also somewhat community led too. Its almost like how things were isn't it?

Part 7: Who are we building for?

A fairly common argument for new functionality is to attract new users. To quote an old acquaintance, "It is good and virtuous". However, oftentimes, it feels like the 'new users' are the sort of folks who don't want to be here - they want something different. I'm all for new blood - but folks who tell others "SO is toxic cause I have bad experiences" or worse - people who believe that blindly seem the wrong people to design for. If we're making fundamental changes in design, or new products, it should be keeping the community we have now and sort of folks who contributed in the past in mind, as well as the people who'd benefit from being here. One would hope I'd not need to break the distracted boyfriend meme again.

We need to build for the future, sure but also, with significant periods of upheavals in the community - and both folks who work in the community and company being in flux, we also need to build the present and remember the past.

_{If you're looking for waffles mentioned in part 2 🧇🧇🧇}

Frankly, I hate the whole concept of minimum viable product. I don’t go into a project thinking what is the bare minimum I can do and still succeed. You start with a time period or some other constraint and think ‘What is the maximum I can deliver within these constraints?” The term ‘minimum viable product’ frames prioritizing core features in a way that makes a team focus on need vs want, which doesn’t leave a lot of room to delight your customers.

So, given that, I recommend setting a maximum development effort for an incremental step in a project and a checkpoint after each step to evaluate what was discovered and correct course.

Here’s our team's "new feature" workflow in a nutshell:

1. Time-boxed investigation to unearth technical challenges, gather information, discover prior work that can be leveraged etc. and then propose different options to move forward with different trade-offs
2. Present the results of the investigation to stakeholders and reach a consensus on which path forward to take, which might be more investigation, implementing something, or tabling the idea.
3. Get the team together and break the proposal down into steps and start story pointing them. This is basic Agile-style epic refinement. Our release cycle is 90 days, so we figure out the most functionality we can deliver as a team in that time period toward the goal.
4. Design the implementation and present it to stakeholders, clarifying requirements along the way.
5. Implement and release. Gather feedback from users on how it works and adjust the next release based on what we discover are the highest priority needs/pain points. Often our customer will want a different feature rather than more refinement on the one we just delivered.

We never plan the entire new feature in detail. We focus on getting the core feature with no embellishment to the users and let them work with it. As soon as they start using something, their priorities and requirements change.

Our team has struggled through a similarly soured relationship between us and our primary customer. The answer is, unintuitively, loosening the reins instead of laying down more constraints and, more obviously, more frequent and transparent communication.

(I may elaborate on this more later. I'm out of time now)

It's a complex topic. For example, I would be inclined to say that the company did not nearly enough experiments between 2010 and 2024, others may think there were way too many. If you value the Q&A model above all else, you probably don't want any experiments at all and will set the bar very high. I want to see the knowledge collection model adapted to a changing environment and that's why I would set the bar lower.

If we assume that at least some level of experimentation is desired, we should be willing to accept some disturbances during experiments and some additional cleanup effort afterwards. It will not always be avoidable. Disturbances of the regular operation should be reduced though to a minimum and should be communicated in a way where it's clear why it's needed. There should already be a plan for how to recover and clean up afterwards and this step should then be executed.

^{Example: Recent answer bot answers are not included in the data dump (good), have unclear content licenses (bad), and seem to remain on the sites even though the experiment failed (bad). It would have been better if all impacts of this experiment would have been cleaned up.}

Failed experiments should clean up all their impacts as much as possible.

^{Example: For 1-rep-vote experiment (which wasn't conducted but could have), one could have recorded votes from 1-rep users for a short time only (say 2 weeks), analyze their impact, then try to undo their impact as much as possible (vote reversal is available and done kind of regularly). If some residual effects (badges, caps) remained, it would be okay in my eyes; for science.}

Some disruption is a reasonable price to pay for a gain in knowledge.

Yes, experiments should sometimes be able to disable existing capabilities. Otherwise you would be severely limited in the scope of experiments that can be conducted and only additional capabilities could be experimented with. We could never try out to simplify anything for example.

From the past, I would say that the whole community, not only selected members, should be consulted during the planning phase of an experiment.

^{Example: Before the beta rollout of collectives and articles, selected members of the community were consulted. The feature wasn't very well received, wasn't very successful and slowly died in the years following. It seems a bit as if only consulting a few members, doesn't give enough significant input. It's better to gather as much feedback as possible.}

What's striking to me though is the half-bakedness of it all. Collectives or Discussions received very little changes after the initial rollout. It's almost like the company didn't want the features to succeed which looks a bit like a waste of resources.

Multiple rounds of experiments on an single subject might be necessary in order to make it a valuable feature. The company should somehow develop more stamina in that regard and try to polish planned features more before deciding if they are worth it. Maybe it needs a vision that goes beyond more immediate engagement. Where does the company see the platform in 6-8 years?

^{Example: The Staging ground worked when introduced, but took multiple rounds (although I just remember that the company gave up on it in the middle of the first round). The trending sort order also took multiple rounds of forth-and-back with the community (although it seems to not be available outside of SO, why not?). The unfriendly comments robot had at least two versions/iterations (although it's not in use anymore apparently, why not?).}

All in all a mixed picture there. Following through on their plans and optimizing new features seem to be weak points.

I think that meta Q&As are already quite effective for communication but maybe there are better ways to structure discussions about experiments. Maybe dedicated "folders" for experiments, where all Q&A related to a specific experiment are kept together? Could be realized with the tag system or something else. But in general, I'm happy with the existing framework.

Now the difficult part: minimal feature sets and success metrics.

I think each good/insightful experiment (doesn't need to be successful to be useful) should have a very, very clear problem and solution description. What is the problem, why is it so bad and how can we improve on it? It should also include a discussion of potential downsides/risks. What are the downsides? It must include a very clear metric or set of metrics that can decide on the success of the experiment. An A/B test is often preferred unless impossible to do and a discussion of and control for the novelty effect should also take place.

^{Unfortunately, often that is not done. I frequently wonder what the problem is supposed to be (example: "some members of the communities are looking for new ways to engage" might be too vague, what are they looking for?) or what the used terms mean (example: "people want to ask follow-up questions" - but what is a follow-up question and how is it different from a normal question, why can't it be explained in the text?) or what the metric will be (example: "we are looking for more engagement" - is needing to click one additional time more engagement?) or the novelty effect is not controlled (example: drawing a circle around voting triangles allegedly improves voting by a whopping 97%, but does it really?).}

There is much room to improve.

I have a hard time trying to come up with general applicable guidelines, but I think I can tell you when an experiment might work when I see it. So including the community in all stages might be a good idea and try to be as clear as possible with problem and solution descriptions.

As for the size, I guess experiments can come in all sizes (1-rep-voting would be a very small change to the software), but it makes sense to go in smaller steps and check back often, but of course sometimes you have to make a larger jump if there aren't any reasonable intermediate steps available.

And we should be prepared to see experiments failing frequently. There are probably many more ways that you can do things wrong than right, but that still doesn't mean that the current state is the best possible or even close to that. Failed experiments should not result in doubts about possible system-wide change or taint ideas. One can fail, and try again, and succeed the next time.

^{And a final example: Dedicated thank you feature. 1/6 of all comments are supposed to be thank yous, we could get rid of these somehow. The company thought that a dedicated thank you button that otherwise is useless would be a good idea for that. Downside would be competition with the upvote button. After one recent experiment (that did a couple of things at the same time, which would typically be bad practice) it concluded that there is a 10% reduction in thank you comments (which isn't much) and will create the button. The metric is fine, the feature set is questionable (a button that otherwise doesn't do anything) but compete; yet the problem is not solved while there would be much better approaches (e.g. AI assisted thank you comment detection with a note to upvote instead and an automatic scheduled comment deletion after X days, which would surely find more than 10% of thank you comments and additionally would not compete with upvotes but support them especially considering the technology is quite simple nowadays). The community advocated for this. If a far suboptimal solution is also seen as fail, then this experiment failed very early, when possible solutions were sought. Everything afterwards couldn't correct for the initial conceptional problem. Some might say that this experiment wasn't really necessary in this form.}