19
$\begingroup$

I have been wondering: does the quality of an answer and its upvotes always correlate? It should, but it obviously does not. Obviously, because not everybody gets all aspects of every answer. Differences in style and taste influence voting behaviour, too. But I have the impression that there is more at work.

I have observed that those answers I put the most thought and effort into are often those with the least reaction. I am sure the effect is not restricted to my own answers, but those I have a good overview of, so please bear with me.

Let's look at some recent examples.

On the other hand, we have:

Obviously, I am in no position to judge the quality of my own answers, but I can't shed the feeling that putting in significant effort -- that is answering questions I don't immediately know the answer to -- is not worth my time if I only consider the SE reward system (I learn something, too, but that's not the point here). There are no negative comments, so I have to assume that the answers have no obvious problems. They just don't get any attention.

The latter answers partially share some common characteristics.

  • The questions were not new when I answered (see also the answers of our answer meetup, few of which got significant feedback).
  • The questions are sometimes harder or more obscure than average.
  • The answers are long and sometimes technical.

Are these obervations sufficient to explain the voting difference? Do people not look at new answers to old questions? Do they not read long and/or involved answers? Are shorter answers always better?

Maybe my answers are bad. If so, why don't I get comments so I can improve them and do better next time?

I'd like to understand what happens here. If my worry is warranted, this community does not reward those posts we arguably should treasure the most, namely those that require thought and time. Instead we reward those that stick to their comfort zone and answer the "easy" questions¹. In other words, we do not encourage people to tackle the harder questions (which are responsible for most unanswered questions) or those that require more text to answer.

What are your thoughts? Is there a problem, or am I overreacting? If there is a problem, how can we solve it?


  1. Note that I don't judge those that do. They make good contributions and their answers are typically high-quality because they know their topics inside out.
$\endgroup$
  • $\begingroup$ I think new answers are not promoted prominently enough by the platform. You only see the question pushed to the top (either on the main site or on the feed), but after a while of visiting the same question with the same answers because of some edits, you don't click on questions more than once. $\endgroup$ – Raphael Jul 28 '12 at 15:51
  • 2
    $\begingroup$ If more effort means longer answers then I don't find it surprising at all. IMHO, most users don't read an answer which is more than a page/two paragraphs long unless they are really interested in the question, and generally a conciseness answer is preferred to a non-concise one. And one can argue that non-conciseness of an answer is partly due to lack of effort to make it shorter. $\endgroup$ – Kaveh Jul 28 '12 at 22:21
  • 1
    $\begingroup$ @Kaveh: True, but some questions just need more complex and thus longer answers (given a fixed target level of detail). Note that more than one of the linked answers is the only answer, so there is no issue of competition involved. By the way, "more effort" also includes research, tinkering or creating an image (that shortens the text), all of which increases the response time. $\endgroup$ – Raphael Jul 29 '12 at 13:49
  • $\begingroup$ One thing that can help is providing a short outline/summery at the top. Still even with a short summery a longer answer may get less up-votes (e.g. I typically don't like up-voting an answer unless I have read most/all of it). $\endgroup$ – Kaveh Jul 29 '12 at 20:16
  • $\begingroup$ @Kaveh: Definitely, I don't vote differently (although I do read most answers completely at this point). The question is, how can we make people read (good) answers even though they are long so their authors get feedback? Should we shoot down (that is, break up) questions that ask for long answers (maybe involuntarily) in order to break everything in digestible parts? (Something in that proposal repulses me. On a computer science site, it should not be necessary to artificially keep things in attention-span size.) $\endgroup$ – Raphael Jul 29 '12 at 21:40
  • 1
    $\begingroup$ Funny, I was just wondering about this as well. Here's perhaps another example. I think you have a valid point here. However, people can use the system how they see fit, i.e. read what they want and vote what and how they want. I think maybe the system should reward people more for voting: give reputation for voting and/or (more) badges for it. Forcing anyone to vote is probably not a good idea. Although I don't know if rewarding makes people just vote in a way blindly... $\endgroup$ – Juho Jul 30 '12 at 12:38
  • 1
    $\begingroup$ I agree that often short concise answers tend to receive upvotes. It might also be helpful if someone has commented on the answer with something like "this is great". It might make people upvote more. If you don't understand the answer fully but still it seems like effort has been put into it, it's in a way a confirmation for you that not everyone has just blindly upvoted the answer. This is perhaps especially so if the comment is from someone you respect and think is knowledgeable. $\endgroup$ – Juho Jul 30 '12 at 12:57
  • $\begingroup$ @Juho I have thought about rewarding voting, but as you say, you don't want to encourage blind voting. Lauding great answers in the comments might be a good idea. It is abuse of the comment function but if it helps the site, I'm all for it. I know that if I read a positive comment by a proven expert, I am more likely to upvote even if I have not understood every detail of the question. $\endgroup$ – Raphael Jul 30 '12 at 13:50
  • $\begingroup$ imho, have seen this same trend in my own posts, & this is a very real issue with stackexchange and collective intelligence. think there are some possible technological solutions eg variations on bounties, tip jars etc, but stackexchange wont chg until its officially recognized.... its also partly a cultural problem.... guidelines on "how to vote" might be helpful.... its esp an issue on highly technical forums eg tcs.se, cs.se, mathoverflow, etc.... $\endgroup$ – vzn Oct 3 '12 at 22:59
17
$\begingroup$

You're worried that there is no incentive to write good, difficult answers to non-trivial questions. By "incentive" here you mean reputation points. However, I think the real incentives are (1) helping people, (2) honing your technical skills. If you're here just for the popularity contest, then perhaps you should only address the low-hanging fruit.

Personally, while I enjoy the popularity-contest aspect, I think that my largest benefit from this site is the challenging questions that show up now and then. Easier questions — mostly homework questions, as far as I can tell — get mostly hints from me now. It used to be frustrating that answers that required some thought and ingenuity were not being rewarded, but nowadays I feel that you're really rewarding yourself by answering them, and reputation is just a way to get you addicted to the site.

If you find yourself regularly spending hours solving questions and writing extremely nice answers, then perhaps you shouldn't. Or perhaps you should understand that doing that, you're not only training your problem-solving skills, but also perfecting your writing skills. You are doing all of it for your own benefit, regardless of how much "official" reputation you gain.

$\endgroup$
  • $\begingroup$ 1) How do I gauge whether my effort was effective, i.e. my post helped people and is well written, without any feedback? 2) It is great if people work for the greater good, but most probably won't. If we don't reward effort -- especially such that teaches the answerer little -- in the "game", we miss out on potentially good posts. $\endgroup$ – Raphael Mar 22 '13 at 12:46
  • $\begingroup$ 1) If your post gets many upvotes compared to similarly technical posts, it must have been useful. 2) If you feel unrewarded and it makes you feel unhappy, just don't do it. $\endgroup$ – Yuval Filmus Mar 22 '13 at 15:16
  • $\begingroup$ agree that the voting system should try to reflect what the group really considers "important" and not merely "low hanging fruit", but not sure how to accomplish that. its a complex question related to trying to get individual alignment with group voting goals & how the voting system [which is highly subjective] works etc.... $\endgroup$ – vzn Mar 22 '13 at 15:22
  • 3
    $\begingroup$ @YuvalFilmus: 1) My observation is that they don't, because none does (some popular examples aside). 2) Ego-centrism aside, don't we want to make more people feel happy about investing work here? That's what this is about, not whether I personally feel rewarded. $\endgroup$ – Raphael Mar 23 '13 at 15:44
9
$\begingroup$

This could, perhaps, be related to a well-known phenomenon. I don't remember the exact name, but the idea is simple enough. Ask a room of people what color to paint a shed, and everyone will chip in an answer; if the answer has already been suggested, it stands to reason that many people will upvote the answer they would have otherwise given. Ask the same group of people how best to interpret certain allusions in Dante in the context of medieval Christian literature, and it is likely that many fewer people will have an opinion, much less an answer, and therefore no answer - no matter how good an answer it may be - will be likely to receive as much attention.

Your trivial, easy-to-digest answers to simple questions were treated well, and your thorough, complex answers to more delicate questions were treated badly. I think that, as it is, the bounty system begins to address some of the underlying injustice inherent in all this, although imperfectly. That is, in order to receive a greater reward for a more obscure answer, you would currently wait until someone offered a bounty to compensate for your effort. I don't know whether there's any better, feasible solution. Some possibilities:

  • An automated bounty system, so that questions which remain open and up-voted, without an accepted answer, begin to accumulate larger and larger bounties over time. Pro: eventually, every question will have a bounty large enough to attract the attention of experts. Con: the community would possibly need to be more vigilant in policing posts to avoid clutter and abuse.
  • A system (possibly integrated with the flag system) whereby a few high-rep users could vote to award a bounty, after the fact, to a particularly deserving answer. Pro: good answers that don't receive due attention can be brought to the attention of people able to compensate the answerer. Con: increased burden on high-rep users to review answers.

I don't particularly like either of these, but I think that, ultimately, a modification to the bounty system could further help mitigate the problem you're seeing.

$\endgroup$
  • $\begingroup$ I actually suggested a tip jar on meta.SE but it received no love. $\endgroup$ – Raphael Jul 30 '12 at 19:49
  • 3
    $\begingroup$ It's specifically a bike shed. A common topic on Meta.SO. $\endgroup$ – Gilles 'SO- stop being evil' Jul 31 '12 at 19:18
  • $\begingroup$ @Gilles: Too bad that thread does not offer a solution. Apparently, SE is not concerned about "low-hanging fruit". $\endgroup$ – Raphael Aug 5 '12 at 11:40
  • 1
    $\begingroup$ dont like this answer exactly because it focuses on subjective questions, "what color to paint a shed". also the bounty system is cool and could go in all kinds of new directions. one feature I really dont like about the bounty system is that people lose their points even if nobody answers. never really did understand that! it is more of a gambling system that a quality answer will show up at the risk of lost points. think a more fair bounty system would not sacrifice points even with no answers! $\endgroup$ – vzn Mar 22 '13 at 15:26
6
$\begingroup$

The problem is that the up/down votes on questions/answers are quite subjective. If I see something with which I'm familiar (probably homework scale) I'll be in a position to see if the answer is right or wrong, brilliant or plain dumb plodding. I'll feel confortable voting (and the answer probably took a few minutes to compose). For unfamiliar, complex questions in my area of expertise I'll try to check them out and vote on them, but naturally the number of people in position to do so is much less, so less votes either way. Most of the more intricate questions/answers I'll just have to pass over.

Some measure of "effort spent" doesn't sound too bad, but is it "effort spent" or "intelligent effort" what we want to reward here? I'd contend it's the second. And effort isn't really measurable in any sensible way either.

As the saying goes, SE measurement of post worth stinks. It just stinks less than the alternatives ;-)

$\endgroup$
5
$\begingroup$

I do not see any real problem here.

The upvotes indicate (roughly) how many people have read the answer and found it useful. This is the feedback that I want to get, and this is what the system provides.

Of course few people will read long, technical answers. But I think it is nice to know whether it is 2 or 10 in a particular case.

There is no need to artificially inflate the upvote numbers. That would make the site less useful: you would then waste your time writing a long, technical answer and the system would mislead you to think that many people have read it. The current situation is fine: you get the real statistics and you can use this feedback to decide whether you want to write long answers in the future.

If a high-rep user feels sorry for someone who has written a good answer that has received too little attention, they can always award a bounty.

$\endgroup$
  • $\begingroup$ See, that's the problem: users might decide to not invest the effort in the future, which makes the site a worse place. Nobody is talking about artificially inflating votes, but finding ways to make more people find and read such answers, or to provide other ways to reward the answerers. Regarding bounties, that's just an awkward process to reward somebody. Sadly, other proposals don't get much love. $\endgroup$ – Raphael Jul 30 '12 at 20:05
  • 1
    $\begingroup$ @Raphael, if you want to have people read a post share it. There are several serious problems with your proposal. IMHO, in general the system works best when reputation is considered a side issue not the main one. If someone is here mainly to earn reputation that would be way more troubling for me than the current short comings. So I think treating reputation as currency is the wrong attitude. $\endgroup$ – Kaveh Jul 31 '12 at 7:38
  • 1
    $\begingroup$ Well-written long answer also get good amount of up votes, the problem is mainly with long answers which are not well-written where a reader asks oneself why should I spent time and energy to read this long answer and doesn't see any reason to do so. $\endgroup$ – Kaveh Jul 31 '12 at 7:40
  • $\begingroup$ @Kaveh: You mistake my intention. I do not propose to stress reputation more than necessary. Note that SE fundamentally relies on its reward mechanisms, though. I am far more concerned with a lack of feedback, be it votes or comments. My problem is that long/involved answers or answers to old questions often don't get any feedback. If my long answer is not well-written, a comment saying so gives me the opportunity to improve it. $\endgroup$ – Raphael Aug 5 '12 at 11:34
1
$\begingroup$

this meta question just newly popped up after yuval's answer. will throw in a opinion here after about 1 yr of heavy use of stackexchange sites & my subjective experience & noting/thinking about reactions. think this is a very important question that cuts to the heart of stackexchange, and the modern evolving field of "collective intelligence". there is no real guidelines ever posted on stackexchange, anywhere, for voting. moreover even if it was posted, who knows whether people conform to it. this gives huge power & freedom to the audience but also there is some major mystery there.

If my worry is warranted, this community does not reward those posts we arguably should treasure the most, namely those that require thought and time. Instead we reward those that stick to their comfort zone and answer the "easy" questions.

imho this is exactly correct, but yes, its probably a feature of stackexchange voting system mixed with human psychology. yes, would like to analyze more what answers get upvotes and which get no response or downvotes. its a excellent area for further analysis. lacking any serious quantitative analysis, we have our subjective experience. would agree—mine fits in with yours.

part of the problem is that very few people can judge very difficult questions and answers. there is only a small audience for these questions/answers. in science, sometimes the most important questions have a very narrow audience. this is the "leading edge" of research which is highly abstract, technical, and which only a few can grasp or understand. so probably people are refraining from voting out of not knowing whether the answer is good or not. there are possibly very good answers to more difficult questions but very few people can realize that based on their background.

people do not seem to "reward" votes based on how much knowledge or research they think went into the answer, or how hard the question is. they seem to vote based on whether they think its a "good" answer ie "correct". generally this also requires it be "succinct". it takes too much mental effort to evaluate complex answers. yes, stackexchange is oriented around snap answers. the basic voting system pushes it in that direction. its part of the architecture.

another point is that I suspect a lot of voting is oriented around peoples offline reputations or background. people who participate under pseudonyms are possibly at a disadvantage to "real people". also people who are elite in a field probably easily get more votes for their participation with less resistance or pause from the audience. this is not nec. a bad thing, we probably want to encourage "high rep" people (judge by non stackexchange "real reputation/credibility") to participate the most.

basically, stackexchange is only one possible architecture for collective intelligence systems. its relatively young-- less than 5yrs old. there are some other experiments, but the whole field is very young, its not much older than "Web 2.0" technology. if people want different emphasis/focus [which I myself strongly feel at times] then other directions will have to be explored by new pioneers.

one idea I had once was a separate voting bucket or axis where people could try to estimate how "hard" a question is. an answer on a very hard question might be valued more. or something like that.

stackexchange does have some flexibility however. it would be very interesting to attempt to start a new group on area 51 that might have different rationale from the beginning, ie encourage different voting patterns to somehow officially lean toward some other criteria than the conventional one. but it takes a very large group of people to start a new stackexchange.

$\endgroup$
  • 7
    $\begingroup$ Please fix your Shift key and your ' key. Proper use of English would make your prose a lot more readable. I gave up against this wall of text. $\endgroup$ – Gilles 'SO- stop being evil' Mar 22 '13 at 21:30
  • 1
    $\begingroup$ "I suspect a lot of voting is oriented around peoples offline reputations or background" - I hope not. Can you give examples? $\endgroup$ – Yuval Filmus Mar 30 '13 at 3:31
  • $\begingroup$ @yuval it seems people prob trust academics (phd students, phds) more on cs & cstheory based on profile info. as stated, this is prob ok. it would be difficult to prove this—cant even think of an objective way to measure it at all. actually it would be rather difficult to prove that voting is related to any particular trend at all. stackexchange recently advertised for a datamining position, maybe this role could shed some insight into voting patterns etc. $\endgroup$ – vzn Mar 30 '13 at 14:58
  • $\begingroup$ @vzn In my experience, reading a person's post is sufficient to guess at their background. Do I upvote posts by researchers and theoreticians more easily? Probably, because they speak my language. Do I upvote posts by people of whom I have read dozens of good posts more easily? Probably. That is not ideal, but neither intentional, nor discriminating (since I only use the posts, not any secondary information), nor can it be prevented. $\endgroup$ – Raphael Apr 3 '13 at 11:40
1
$\begingroup$

Run periodic "hunt old questions without answers" drives? Encourage such activities more? (Yes, there are a couple of interesting badges for unearthing old questions, perhaps there should be more?)

$\endgroup$
  • 1
    $\begingroup$ We have had two events of this kind (1, 2), with little response. $\endgroup$ – Raphael Mar 28 '13 at 9:51
  • $\begingroup$ the question seemed more about questions with answers that arent voted as well as they should be, not so much unanswered questions... [indeed another issue] $\endgroup$ – vzn Mar 28 '13 at 15:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .