Timeline for answer to A/B testing of a "Trending" sort option for answers by Konrad Rudolph
Current License: CC BY-SA 4.0
Post Revisions
62 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 5, 2022 at 15:47 | comment | added | Travis J | That doesn't align with your previous statements. | |
| May 4, 2022 at 18:05 | comment | added | user400654 | When i vote solely on the content, i'm also voting to impact the ranking of answers whether i intend to or not. It is their primary function. | |
| May 4, 2022 at 18:00 | comment | added | Travis J | @KevinB - Voting is absolutely not a gaming mechanism. You shouldn't be using your vote to counteract previous votes or to artificially influence system metrics, you should be voting solely based on content. | |
| May 3, 2022 at 19:48 | comment | added | user400654 | That's what voting is for. Literally. | |
| May 3, 2022 at 19:46 | comment | added | Travis J | @KevinB - I have observed numerous cases of useful unique answers being dropped down the line as a result of a single downvote. You casting a downvote will not always "right" the wrong of the sorting order. That is needs to be fixed by vote gaming should be a huge red flag. | |
| May 3, 2022 at 19:45 | comment | added | Travis J | @KevinB - External downvotes play a significant role in posts which haven't had voting for a while. Using them as the only signal for sorting is a flawed system. Who can and cannot cast them is irrelevant. | |
| May 2, 2022 at 20:24 | comment | added | user400654 | thus far i've only seen cases of this A/B test causing poor quality posts saying effectively the same as the old ones being brought to the top. If me casting a downvote on such cases returns the order to where it belongs, that's a net positive. | |
| May 2, 2022 at 20:13 | comment | added | user400654 | External downvotes generally play a very minor role, considering the overwhelming number of users who can't even cast them, but can cast upvotes. | |
| May 2, 2022 at 20:10 | comment | added | Travis J | @KevinB - Fortunate is perhaps an incorrect word choice there. Semantics aside, even in the case where now both users must downvote each other, and in turn each post in the entire set must have each author downvote each other to reach equilibrium as you suggest, external downvotes will also play an expanded weighting. It is now possible for posts, as a result of this algorithm, to be upvoted in the hundreds historically, only to have a recent downvote kick them down below single upvote posts. While this may be unintentional, there is excessive gaming of votes already; we don't need more. | |
| May 2, 2022 at 19:08 | comment | added | user400654 | @TravisJ Fortunately, each user only has one such vote that can only go in one direction or the other. Any "damage" one can cause by casting downvotes will easily be outweighed in such "competitive" situations | |
| May 2, 2022 at 19:07 | comment | added | Travis J | This introduces a path of abuse. A heavily weighted downvote means that competitive posts can be gamed by using a downvote in order to significantly affect the other post. | |
| Apr 23, 2022 at 20:59 | comment | added | jrh | @KevinB thanks for downvoting and curating the site. It's still not a fair comparison though. On this particular site I'm 270 downvotes away from losing the ability to downvote (on other sites that margin is much much thinner and I just don't do it unless it's for a very serious reason and it'll make a difference). You're 93943 downvotes away from losing downvote power. | |
| Apr 22, 2022 at 6:15 | comment | added | Craig Hicks | "While we can encourage users to sort answers using something other than answer score — such as Newest (coming within the next few weeks)" --- Currently the "by date" option counts edits as updating the post. Sometimes that is helpful, but sometimes the edits don't add anything new so it is not helpful. I hope that a "newest date of original creation (not edit)" will exist. | |
| Apr 21, 2022 at 16:10 | comment | added | user400654 | eh, the majority of these voting rings upvote, not downvote. There's little to no benefit to such rings casting downvotes. Regardless, voting rings aren't a reason to not alter voting weights if altering voting weights will benefit the site as a whole assuming voting rings are dealt with. | |
| Apr 21, 2022 at 16:05 | comment | added | Ben Voigt | @jrh: Just one example -- meta.stackoverflow.com/a/322115/103167 And it's been happening for just about the entire lifetime of the site --meta.stackexchange.com/a/137449/135695 | |
| Apr 21, 2022 at 15:55 | comment | added | jrh | @BenVoigt that's wild, somebody would really grind rep on a sock puppet ring to do a downvote spree? I guess I underestimated the determination of trolls. Fair point then. | |
| Apr 21, 2022 at 15:53 | comment | added | Ben Voigt | @jrh: Until you find that some troll with a personal vendetta has created a bunch of throw-away accounts just to downvote their target. The moderators blow away sockpuppet voting rings on a regular basis, if fewer votes were needed then the sockpuppets would be harder to detect. | |
| Apr 21, 2022 at 15:53 | comment | added | user400654 | @jrh my rep would also be 25% higher if I didn't downvote, but I don't consider myself to be a low rep user | |
| Apr 21, 2022 at 15:48 | comment | added | jrh | @BenVoigt I agree on "not-totally-bad", but I'd rather just see downvotes increased in weight for everyone. My rep would be 25% higher if I didn't downvote. | |
| Apr 21, 2022 at 15:44 | comment | added | Ben Voigt | @jrh: Rep is a not-totally-bad proxy for the combination of expertise and trust, which is what's needed here. | |
| Apr 21, 2022 at 15:42 | comment | added | jrh | @BenVoigt to be a bit contrarian on the whole "treat gold badge downvotes differently", if I as a low rep user downvote, it means I'm willing to take a lot more significant portion of my rep than a 30ker to send a message. There was a point in time where I had to hold off downvoting because I'd downvote right out of the privilege (which is where I still sit on Meta.SE). I know what you mean about "proven expertise" (or perhaps a proven trend of FGITW quick draw to homework dump answers in 2009) but that doesn't mean there aren't any low rep experts. | |
| Apr 20, 2022 at 22:03 | comment | added | Zags | One issue with this is that good answers sometimes pick up a downvote or two during their initial lifespan (while they're still being refined or clarified) and the people who drop these downvotes often never retract them. If old downvotes count more than old upvotes, answers with a few early downvotes get doomed forever. | |
| Apr 20, 2022 at 17:32 | comment | added | computercarguy | My experience is that votes are used differently on different stack sites. On technical sites, up votes are only used when an answer or question directly helps the user, but down votes are used for not only pointing out wrong answers or disagreeing with/misunderstanding something, but also if the post doesn't adhere to the site rules. Non-tech site users seem to use up votes when in agreement with a post, so these Q&As tend to get a lot more up votes than down votes, and more up votes than tech stacks. I'd like to see actual research across stacks before assigning some multiplier to votes. | |
| Apr 19, 2022 at 8:31 | comment | added | Konrad Rudolph | @user3840170 I prefer not to publicly shame individual answers but I will try out your script and report the results for the handful of answers that I can remember off the top of my head! | |
| Apr 19, 2022 at 8:25 | comment | added | dumbass | @KonradRudolph You mentioned a number of times you can point to several instances of PBW answers under the R tag. Have you tested how my script sorts them? Can you mention them as an answer under its page? | |
| Apr 19, 2022 at 7:57 | comment | added | dumbass | @AndrasDeak--СлаваУкраїні No, I think that’s a separate issue. | |
| Apr 18, 2022 at 21:38 | comment | added | Andras Deak -- Слава Україні | It's probably covered by this answer (which is why I'm not posting another one), but I saw this gem before I realised that hashtag trending hashtag sorting was silently enabled for me. -5-scored post above 0-scored post. Very trending. | |
| Apr 18, 2022 at 11:36 | comment | added | Mike Nakis | At face value your statement sounds right, but my personal experience is different: every single time an answer of mine is down-voted, the down-voter is, of course, utterly wrong. (Whereas, needless to say, every single time I down-vote someone else's answer, that answer is baaaaaad!) | |
| Apr 15, 2022 at 22:40 | comment | added | trlkly | The thing is, the concept downvote weight concept seems to have been flawed from the beginning. They wanted downvotes to have more weight by them costing the voter a small hit in rep, but then give them equal weight on Questions/Answers and even significantly decreased weight on the poster's rep. It seems that, rather than them having more weight, the designers wanted them nerfed so people wouldn't use them. (Fortunately, I don't care about rep for either side, just about making the site better. So I downvote with abandon if I know something is actually wrong or misleading.) | |
| Apr 15, 2022 at 22:36 | comment | added | trlkly | @BenVoght Surely we want to incentivize fixing bad answers, though? And why would I care about what score someone else has? It's not like that factors into determining if the poster is correct. We're not in competition with each other over who gets the higher score. Which is good, because the scoring system is not fair when viewed through that lens. It heavily favors older join dates and past activity over current activity. I don't see any reason to care if someone gets their 2 points (or whatever) back, as long as the problem was fixed. | |
| Apr 14, 2022 at 8:02 | comment | added | Konrad Rudolph | @jxh Actually you’re right, that metric would likely be quite informative (unfortunately it’s also harder to measure accurately). Either way, all of this supports my point that a proper ranking of answers should account for these factors, using proper statistical modelling, rather than the current approach which does none of that, and which we know (from representative examples, see comment #6, as well as from extensive evidence from other sites and from prior statistical knowledge) to be flawed. | |
| Apr 13, 2022 at 22:19 | comment | added | jxh | @Braiam However, rarity of an event need not mean it has more weight. It could also be interpreted as noise. Another measure would be if we saw how many times a question was visited by distinct SO users, but received no vote whatsoever. The lack of interest in a post may be a greater metric for determining how much weight to give newer votes vs. older votes. The higher the interest, the higher the weight newer votes have. | |
| Apr 13, 2022 at 15:37 | comment | added | Braiam | @Flimm a descriptive analysis of anonymous votes vs user votes would tell you that user are less likely to cast downvotes, since the proportion of downvotes vs upvotes on anonymous users is higher (about .8 dowvotes for each upvotes) compared to the normal users (I think it was .1 downvotes for each upvote). | |
| Apr 13, 2022 at 10:27 | comment | added | Konrad Rudolph | @Flimm Yes, it almost certainly matters tremendously. Of course we won‘t know for sure until it has been tried but based on the evidence from other cases where similar sort orders are used, it has a substantial effect. — You seem to misunderstanding the point, however. It’s not about “difficulty”, it’s about motivation: people upvote haphazardly (often without really understanding whether something’s correct or good). But people downvote much more rarely, and (usually) only when something’s actually wrong/bad. This makes downvotes much more informative. | |
| Apr 13, 2022 at 10:23 | comment | added | Flimm | Does it matter? The difficulty in casting downvotes applies equally to all answers, so I don't think it matters that much to a sorting algorithm for answers. | |
| Mar 22, 2022 at 9:46 | comment | added | justhalf | On jxh vs Konrad: I think we all agree that "downvotes are cast less often, partially because of the cost, and partially because of the less amount of incorrect posts" is an empirical fact. Konrad is proposing that this empirical fact implies that "downvotes weigh more than upvote". This implication is what jxh is contesting [by saying that Konrad has a belief that downvotes inherently weigh more], and trying to get Konrad to give more justification for this [non-obvious] implication (I personally agree with jxh that we need more justification to draw that conclusion from the fact) | |
| Mar 18, 2022 at 17:20 | comment | added | Braiam | @jxh Probably what is needed in this argument is a null hypothesis that can be testable, which would lead us to a conclusion based on an objective result. | |
| Mar 16, 2022 at 15:41 | comment | added | jxh | @KonradRudolph I am not trying to misrepresent your position, so apologies. What I mean is that if you are using statistics to determine an empirical result, the result itself is the empirical fact, not the conclusion. The conclusion of "weighing heavier" would be an interpretation of the result. Treating it as fact would be a belief or a bias that it is inherent. | |
| Mar 16, 2022 at 10:02 | comment | added | dumbass | @jxh I did not write this answer, but I certainly would take that into account if it were upvotes which cost reputation to cast. For what it’s worth, in the latest published version of my script that sorts by weighed Wilson score, I removed downvote weighing on Meta sites in part for a similar reason. If the voting incentives were the opposite (high rep threshold and cost for upvotes, low threshold and no cost for downvotes), I might have even reversed the weighing! | |
| Mar 16, 2022 at 9:59 | comment | added | Konrad Rudolph | @jxh No, you keep misrepresenting my position, after I’ve repeatedly tried to make it clear. Your hypothetical scenario completely ignores my previous comments which have already answered this. One last time (after this I give up!): there’s nothing “inherent” about the situation, nor is it a “belief”. Empirically we can see that downvotes in the current situation weigh heavier. If the situation were changed, this would change. But we don’t need to speculate. We can (should!) just use the available data to calculate weights. This is bog-standard statistical modelling. | |
| Mar 16, 2022 at 9:07 | comment | added | jxh | @KonradRudolph You (and I guess many others) believe that a downvote inherently has greater value than an upvote. Suppose downvotes were free, and upvotes cost reputation. Would you still argue that downvotes inherently have greater value? Do you think upvoting costing reputation would make people upvote less often? Would that also factor into it having greater value? | |
| Mar 16, 2022 at 8:40 | comment | added | Konrad Rudolph | @jxh It is related. It’s feedback on the proposed change: I’m saying that vote decay alone is inadequate, and that the sorting algorithm first and foremost needs to fix its incorrect counting of downvotes, because both the current form and the proposed change are effectively broken without it. | |
| Mar 15, 2022 at 23:46 | comment | added | jxh | @KonradRudolph So the most highly upvoted answer is not related to the posted question, and advocates allowing downvotes to negate more than one upvote. | |
| Mar 15, 2022 at 21:27 | comment | added | Konrad Rudolph | @jxh My answer isn’t about “old” votes, that seems to be a misconception. All votes should be weighted, in addition to (or even instead of) decaying. In a way my answer is actually completely orthogonal to the issue of decay over time. | |
| Mar 15, 2022 at 21:15 | comment | added | Ben Voigt | @jxh: Also, "downvoters rarely change their vote, even if the post is improved to address any reasonable criticism" is often intentional. I don't reward a FGITW user for copying from the correct answers into his. | |
| Mar 15, 2022 at 21:12 | comment | added | Ben Voigt | @jxh: Revisions after the vote are different from the passage of time. AFAICT this question and Konrad's answer deal only with the passage of time. Upvotes lose value over time due to changes to the tools used to construct the answer (new library functions, new compiler features). Those factors don't make downvotes less useful. | |
| Mar 15, 2022 at 21:09 | comment | added | jxh | @BenVoigt The badness was already accounted for in the original vote. If the post was changed, generating newer upvotes, then the downvotes also become less relevant. | |
| Mar 15, 2022 at 20:59 | comment | added | Ben Voigt | @jxh: I don't agree that the same logic applies to upvotes. The vote is trustworthy, but doesn't mean that the answer will remain best in the middle of a changing landscape of technology, which is the whole reason for decaying votes in the first place. A bad answer doesn't get closer to the top when the bar gets raised, a good answer does get farther. | |
| Mar 15, 2022 at 20:56 | comment | added | jxh | @BenVoigt The upvotes should also then be protected by the same logic. However, the protection of downvotes is somewhat dubious because downvoters rarely change their vote, even if the post is improved to address any reasonable criticism. | |
| Mar 15, 2022 at 20:51 | comment | added | Ben Voigt | I think the right approach is for downvotes by gold-badge holders (in any on the question) to never decay. Someone who's taken the time to earn a gold tag badge is already trusted to instantly close questions, is unlikely to be using shotgun downvotes, and generally knows when they see a fatally flawed answer that should never be followed. | |
| Mar 15, 2022 at 20:39 | comment | added | jxh | @KonradRudolph I observe that without taking trending algorithms into account, the score of one downvote negates exactly one upvote. So your trending algorithm would then allow old downvotes to negate more than one equally old upvote. And you believe it is a truism that this should be correct because people don't use their downvotes as often. And many people agree with you. | |
| Mar 15, 2022 at 19:57 | comment | added | Konrad Rudolph | @jxh That isn’t the only argument for weighing downvotes heavier but, yes, it’s one argument, which should be pretty obvious, given psychology. As such, what I find interesting is that you don’t agree with what I’d almost consider a truism. — But at any rate another (and more important) argument is purely empirical: people generally use vastly more up- than downvotes, even accounting for the number of correct and incorrect answers. Just empirically, downvotes weigh more. | |
| Mar 15, 2022 at 19:28 | comment | added | jxh | @KonradRudolph You make a point that a downvote incurs a cost. But you assume that cost justifies the downvote should weigh heavier when using a trending algorithm. And many people appear to agree. I find that interesting. | |
| Mar 15, 2022 at 19:16 | comment | added | Konrad Rudolph | @jxh If that’s how you interpret this answer let me tell you that you have completely misunderstood it. | |
| Mar 15, 2022 at 19:10 | comment | added | jxh | I find it an interesting sentiment that people would rather allow old downvotes to outstrip the value of old upvotes. | |
| Mar 14, 2022 at 11:38 | comment | added | Konrad Rudolph | @Trilarion As for “who can decide who is right”, I can off the top of my head point to several highly upvoted answers which are objectively wrong, and for which upvotes are skewed because the mistake is subtle, or because it’s a common misconception. These examples lead me to believe that downvotes should probably be weighted at least 5×, possibly as much as 10×, compared to upvotes. | |
| Mar 14, 2022 at 11:36 | comment | added | Konrad Rudolph | @Trilarion That’s specifically why you’d use modelling. Other scorings have the exact same issue — in fact, Stack Overflow has it much better: for many answers we can establish an objective correct/incorrect value (e.g. based on comments) and use a curated set of training data, whereas product rankings, Reddit upvotes etc are much more subjective. | |
| Mar 14, 2022 at 11:13 | comment | added | NoDataDumpNoContribution | But wouldn't you need another independent measure of content quality to gauge the weight of up/downvotes? Some people might say there are sufficiently many downvotes given, others might say something else. Who can decide who is right? | |
| Mar 14, 2022 at 10:49 | comment | added | Konrad Rudolph | @Trilarion I actually don’t think it would be all that difficult, using statistical modelling. We can analyse the proportion of downvotes that users cast in general, and compare the distribution to that on specific answers. This allows assigning weights or incorporating it into something like a Wilson score for each answer. | |
| Mar 14, 2022 at 9:51 | comment | added | NoDataDumpNoContribution | "downvotes weigh much heavier due to being used a lot less" It can't be all crap on SO. Hopefully, the amount of better content on SO (at least when weighted by visits) outweighs the amount of worse content. It would be difficult to estimate how much more weight a downvote should get compared to an upvote. Doesn't mean I'm not for it. | |
| Mar 13, 2022 at 20:44 | comment | added | dumbass | I wonder how vote distribution over time factors into this. I would guess that most upvotes tend to be more clustered around the time of the post’s creation (especially for popular questions), whereas downvotes are more evenly distributed. If so, then perhaps this doesn’t need that much correcting after all, since the decay will affect upvotes more than it does downvotes. | |
| Mar 13, 2022 at 9:54 | history | answered | Konrad Rudolph | CC BY-SA 4.0 |
