15
$\begingroup$

Compare CS.SE to Academia.SE, which is more or less at the same state as CS.SE: about a year in beta, ~1500 visits/day, 3300 user (CS.SE has 5k users)

Look at the question on the front page of academia, and the vote they have. You'll see many 5+ questions, some 15+ and some 0-3 questions.

Now look at the CS front page: almost all the questions are 0-4, few are 5+, and 10+ are really rare.

Answer's voting behaves the same, more or less (that is, high votes on Academia, low votes on CS).


Why do we have such a relatively low voting habits? What should we do to amend this, if at all?

$\endgroup$
  • 7
    $\begingroup$ Academia is an unfair comparison; softer sites tend to have highter voting than others because questions and answers are more approachable. How do we do in comparison to, say, Mathematics and Physics? $\endgroup$ – Raphael Mar 19 '13 at 8:30
  • 1
    $\begingroup$ @Raphael It is possible that for science-topics the voting behavior is worse than it is for more soft subjects (for sure, the amount of answers will be lower, since there is only one correct answer, rather than many opinions..), though it's hard to compare to a graduated site. The voting there are low indeed. The voting in TCS, on the other hand, is strange: either very low or very high. $\endgroup$ – Ran G. Mar 19 '13 at 16:20
  • 7
    $\begingroup$ @RanG. Theoretical Computer Science has the second-highest average number of votes per post among graduated sites (#1 is Skeptics). I don't have figures for Computer Science because it's in beta so it's not on Stack Exchange Data Explorer, I'll see what I can do with the API. $\endgroup$ – Gilles 'SO- stop being evil' Mar 19 '13 at 20:36
  • $\begingroup$ I think this is an interesting question. I remember the question was also raised before somewhere, and I remember talking about it to some extent with @Raphael. Perhaps that question was more about "what kind of questions/answers receive upvotes". But I don't know if there's really much one can do: people (down)vote if they want to (or don't) for whatever reason. $\endgroup$ – Juho Mar 20 '13 at 0:38
  • 2
    $\begingroup$ We certainly don't want vote-inflation! ;) $\endgroup$ – Nicholas Mancuso Mar 20 '13 at 4:41
  • $\begingroup$ @Juho Do you mean this one? $\endgroup$ – Raphael Mar 20 '13 at 10:45
  • $\begingroup$ @Raphael Yes, that was the one :) $\endgroup$ – Juho Mar 20 '13 at 15:21
  • $\begingroup$ @Gilles wow, that's interesting! I wouldn't have guessed that for cstheory, but I guess it makes sense since the site has a lot of passionate users but not that many questions. $\endgroup$ – Artem Kaznatcheev Mar 21 '13 at 20:02
  • $\begingroup$ @gilles where did you get that statistic about high cstheory voting? can you link to it? I think clearly cstheory has both low participation and low votes compared to other se sites. or maybe it has very high voting but a very large part is DOWNVOTING, reflecting its elitism.... $\endgroup$ – vzn Mar 22 '13 at 15:31
  • 3
    $\begingroup$ @vzn I posted the numbers in my answer, with up/down votes broken down (but not q/a). CSTheory have above average but not record-breaking downvotes, and more upvotes per post than every other site but one. I link to the query in my answer, I'm aware it doesn't work now (something about the db schema must have changed). $\endgroup$ – Gilles 'SO- stop being evil' Mar 22 '13 at 15:41
15
$\begingroup$

Here are the figures for a few sites, from this Data Explorer query (thanks to rofl for coming up with a query that works). This only takes into account posts that weren't deleted at the time the data was obtained. The columns up and down are the average number of upvotes and downvotes per post. The sites are sorted by average total number of votes per post; I selected the top 6 (excluding per-site meta), the science sites and a few more.

                up    down
skeptics       11.83  1.65
meta            7.87  1.54
cstheory        8.01  0.79
workplace       7.56  1.12
programmers     7.64  0.80
matheducators   6.96  0.17
earthscience    6.37  0.42
mathoverflow    5.94  0.37
tex             5.79  0.06
cogsci          4.83  0.51
astronomy       4.60  0.34
scicomp         4.16  0.12
linguistics     4.04  0.23
cs              3.72  0.33
biology         3.73  0.27
crypto          3.64  0.28
physics         3.11  0.38
stats           3.00  0.12
chemistry       2.66  0.23
math            2.48  0.16
stackoverflow   2.15  0.21

Linguistics happens to be the median graduated site for the average number of votes per post.

Observation: among the top-voting sites, CSTheory is stands out as the one “hard” site among more “fuzzy” sites. TeX comes far behind. CS is middling when it comes to science sites.

My interpretation:

  • CSTheory posts tend to be highly upvoted because they typically involve significant effort, more so than on other sites. Posts tend not to be a quick read, which makes the reader more involved and more prone to upvoting. Furthermore, the site is highly specialized and subject areas are easily delimited, so questions easily reach an expert audience. I suspect, but haven't verified, that the relatively high downvote ratio is due to non-research-level questions.
  • Posts that are easy to understand appeal to a broad audience, it's a very common observation that they tend to attract a wide audience and hence a lot of vote. This is why the high-voting sites tend to be about “fuzzy” topics where one can get involved with little background: Skeptics (where answers often involve significant effort due to the stringent citation requirement, but are easy to read and understand), Scifi, RPG, Programmers, Academia….
  • Math Educators and Earth Sciences are in early beta. There's a lot of voting during the first couple of weeks which skews the results.
  • I can't explain why TeX is above the pack for technical sites. It also has the lowest down/up-vote ratio in the graduated network. This may be a genuine case of a well-developed culture of upvoting.
  • Science sites are fairly broad. This is especially true for computer science which does not have a solid body of knowledge that is shared by everyone (unlike more established sciences, especially math). Science sites also attract a large number of homework-type questions which, while easy and widely understood, don't attract a lot of enthusiasm from many avid users. Nevertheless, science sites reach a median level of voting among the Stack Exchange network.

The comparison with Academia isn't fair: it's like complaining that apples are too small and comparing them with grapefruit. The voting rate is not particularly low on CS. Of course I encourage visitors to upvote whenever they see a useful post (and downvote the bad ones), but I don't see that we have a problem.

$\endgroup$
  • 2
    $\begingroup$ Wow! thanks for the data and the interesting insights! $\endgroup$ – Ran G. Mar 20 '13 at 2:31
  • 1
    $\begingroup$ Seconded, both Ran's comment and Gilles' conclusion. Regarding TeX - LaTeX, in my experience they have extraordinarily good answers, which may explain the large numbers of votes. But they also have a very comfortable culture over there. $\endgroup$ – Raphael Mar 20 '13 at 10:49
  • $\begingroup$ Is it possible to get information about how users vote? Votes per posts are interesting, but strike me as less relevant than how many users vote how often. $\endgroup$ – Raphael Mar 20 '13 at 10:49
  • 3
    $\begingroup$ @Raphael You can get the total number of upvotes and downvotes cast by each user from SEDE and from the API. But I don't know how this figure is useful, because it's an absolute figure. You can't know whether a user who's voted 100 times has read 100 posts or 10000. $\endgroup$ – Gilles 'SO- stop being evil' Mar 20 '13 at 10:58
  • $\begingroup$ hmmm.... cstheory posts seem to fall into a pattern of many low-voted or down-voted posts, and a few highly-upvoted posts that receive a lot of attn which may be skewing the average.... think the average statistic may be misleading.... another pattern may be that there are not as many questions (just a trickle) to vote on on cstheory and the audience is fairly large. think some other analysis of the statistics may be more meaningful, but not sure exactly what right now... just ran the query, it doesnt seem to run for me....?? $\endgroup$ – vzn Mar 22 '13 at 15:40
9
$\begingroup$

The reason I personally don't upvote most of the questions is that they are homework exercises, and often the OP just wants us to solve the exercise for them. No homework question can find its solution in Academia.SE.

$\endgroup$
  • 1
    $\begingroup$ Well, if it comes to that, the reason I don't upvote some answers is that they only provide a tiny hint and not a genuine answer. Maybe we both should be downvoting more? $\endgroup$ – Gilles 'SO- stop being evil' Mar 21 '13 at 20:05
  • 1
    $\begingroup$ I personally perceive downvoting as very insulting, that's why instead of downvoting I simply refrain from upvoting. $\endgroup$ – Yuval Filmus Mar 21 '13 at 21:18
  • 5
    $\begingroup$ As it says in the tooltip, downvoting means you don't consider the post useful. Nothing more. $\endgroup$ – Gilles 'SO- stop being evil' Mar 21 '13 at 21:20
  • 3
    $\begingroup$ @Gilles If you provide a full (reference) answer to every homework question, you'll only attract more of such questions. I prefer giving hints as a comment and encouraging the OP to answer the question her/himself once the period for handing in the answer has expired. $\endgroup$ – frafl Mar 24 '13 at 12:09
  • 2
    $\begingroup$ @YuvalFilmus an explained downvote is more helpful than 20 omitted upvotes. Nevertheless I usually don't downvote unless a question or answer, of which I think that it is totally unacceptable, got upvoted. $\endgroup$ – frafl Mar 24 '13 at 12:14
7
$\begingroup$

I think the discussion is the wrong way round. The low number of votes per contribution is a symptom, not a problem itself.

(TL;DR:) In short, we need to encourage more people to contribute actively, and as a prerequisite, we also need to encourage existing active users to vote more.


First, as Yuval Filmus points out the number of votes per question depends on the nature of the site and the kinds of questions. A site like math.SE has an enormous number of throwaway questions and one would expect low votes per questions. A site like Cstheory.SE has relatively few but well-motivated questions which spark answers that often border on new research, so one would expect a higher concentration of votes per question.

I think the issue being highlighted is that this site has a poorly functioning vote economy.

It is difficult to build up a large active user base if new contributors are not rewarded for those contributions. For better or worse, SE uses reputation as a measure of how much contributions are valued by the community.

Every person participating chooses to vote or not vote on any content. Given the penalties for negative votes, those are infrequent and can often be ignored. Therefore a non-vote on a question or answer partially means "I don't think this is useful, or it is not clear", in contrast with an upvote. Unfortunately in the SE system a non-vote also means "I don't have the time or energy to assess this contribution".

So the message that a SE site with a low number of votes per contribution gives out is: the group of regular contributors as a whole doesn't have enough time or energy for assessment of contributions. This signal is the problem, not the number of votes itself.

Let's check the hypothesis that the regular contributors individually don't have enough time or energy to assess contributions.

One somewhat robust metric is the net reputation effect of individual users. Ignoring downvotes, which are discouraged and relatively infrequent, someone who has made $a$ upvotes for answers and $q$ upvotes for questions has contributed roughly $5(q+2a)$ reputation points into the reputation economy. Now have a look at the public voting behaviour of the top users of this and similar sites. As far as I can tell, this site seem to be doing OK on this measure, for the top users.

Here is a snapshot of the top five users at some related sites. Q is the number of votes cast for questions, A is the number of votes cast for answers, V is the number of upvotes, and ~rep-in is the estimated number of reputation points injected by the user into the site, computed as $\lfloor 5(Q+2A)(V/(Q+A)) \rfloor$. (The estimate has to be used because the information on the number of upvotes for questions and upvotes for answers is not public, so this estimate assumes that the ratio for questions vs. answers remains the same for upvotes as it is overall.)

math.SE:
rep-out      Q    A    V ~rep-in
 168633   3878 1404 5273   33373
 162504   1433 1820 3253   25365
 159384   1694 4059 5626   47976
 120727   1799 3961 4970   41938
 107325   1550 1995 1758   13736

cstheory.SE:
rep-out      Q    A    V ~rep-in
  24293    360  567  879    7083
  19648   2360 3368 5560   44146 (moderator)
  14239    262  372  626    4966
  14032    743 1078 1505   11979
  13882    525  682 1194    9343

MO 1.0: (question upvotes weigh the same as answers)
rep-out ~rep-in
  76918   47440
  56369   88800
  44174   68970
  39968   51320
  39097   60880

CS:
rep-out      Q    A    V ~rep-in
  16482    408  562  968    7644
  15681   1042 1417 2373   18702 (moderator)
  10649    714  938 1625   12738
   8522   1419 1197 2462   17942 (moderator)
   7225    277  327  589    4539

Cstheory.SE is a site with similar topic coverage, and arguably even more niche questions and answers, yet has a similar contribution ratio for the top users. MathOverflow has very high contribution ratios in its current incarnation (though the 2.0 site will drop because of SE's insistence on a "questions are half as significant as answers" philosophy). Math.SE (with those tens of thousands of homework questions) seems similar to CS.SE: the top users are busy contributing to the site, but also find some time to assess contributions of others.

Based on this sample of the top users, I don't think the user ratios at CS are necessarily a problem.

What does seem to be a problem is that the active group at CS is too small. Many of the people generating activity on the site do so by asking questions in a drive-by manner. Math.SE can cope with that because they also have a large group of active users in among the buzzing of the flies. However, at CS.SE I think a lack of active users has caused the attention economy to deflate.

I did some quick manual checks and my random sample of people in the 100-1000 reputation range seems to confirm this feeling. Recall that in a deflation-driven economy, people with some assets have an incentive to not increase what others have, because their own assets will keep increasing in value over time. The top users of an SE site presumably don't really care about reputation since they are too busy contributing content, but other contributors would rationally stand to gain from hoarding their votes, waiting for others to vote for their content while not voting for others.

To check this deflationary hypothesis, more data analysis is needed, and a weighting that takes into account different numbers of people registered as a site grows, and some analysis of changes over time.

We need to keep encouraging people to use the site actively so as to build up a large active user community. We also need to encourage users to spend a larger fraction of the 40 votes per day that they are allocated by the system, so that the economy does not go into a deflationary spiral. The latter may seem unrelated, but it actually influences the former quite strongly.

Some SE questions and answers do get a lot of attention and votes, for instance because they appear on Reddit or HN, because they are controversial, because they require less technical background, or sometimes just because they were written by someone well-known. However, the SE system does not indicate these reasons explicitly. One simply sees that some contributions have a lot of votes, while others don't. Since votes are the only form of signalling the system provides, if there are few upvotes on a contribution, this sends a signal that the contribution is not valued. A new user may stop contributing if they don't feel valued. So to build a large active user community (which is also a prerequisite for the site to move out of beta, where it has been for nearly 18 months) existing contributors need to be encouraged to vote more. We should especially nudge people who are active contributors, yet who have cast close to zero upvotes.

$\endgroup$
  • 1
    $\begingroup$ Thanks for this detailed contribution! Just a few comments. a) "Every person participating chooses to vote or not vote on any content." -- given they even saw the content. Have to answer on that. b) If I compare our rep-in values with the other sites which have been around for much, much longer, they look quite good. c) "the active group at CS is too small" -- can you provide numbers showing that, e.g. quickly falling how rep-in rank curve? d) We have been having many questions of disputable quality (e.g. obvious homework) which quite a few people boycott. $\endgroup$ – Raphael Jun 22 '13 at 15:37
  • 2
    $\begingroup$ A partly formed thought; I wonder if part of the problem is that, broadly speaking, the people answering the questions on cs are much more capable than the level of the questions requires, and hence are disinclined to contribute questions (the questions they have would more likely be appropriate for cstheory) and the people asking the questions are not yet able to answer the sort of questions they're posting, so would have trouble participating further. So we have people who are happy to help people learn, but don't need help themselves (within scope) so the number of regularly active... $\endgroup$ – Luke Mathieson Jun 26 '13 at 5:06
  • $\begingroup$ ... contributors is difficult to grow. Of course the counterpoint is math, which is obviously very active. However it has a much much broader appeal to the "public", giving it a broader base from which to recruit. Perhaps. $\endgroup$ – Luke Mathieson Jun 26 '13 at 5:08
  • $\begingroup$ @LukeMathieson All of research-level CS is within scope here, but only little on Theoretical Computer Science. Sadly, though, we don't seem to be able to utilize the breadth of our scope. I think we have a chicken-egg problem here. $\endgroup$ – Raphael Jul 5 '13 at 14:39
3
$\begingroup$

One trouble with voting is that you can only vote on what you see and read. In my experience, several factors can cause a contribution to rarely be read, and thus not voted on whatever the quality.

  1. Late answer
  2. Late edit of a previous subpar post
  3. Bad question title
  4. Bad tagging
  5. Bad language/formatting

The first two can hardly be avoided without changes to the platform.

The latter three, however, are serious problems which not only influence the perceived quality of our site as a resource, but also how the community works. Badly tagged questions are not read by the right people (which may filter their question stream by tags). Questions with bad titles are not visited even though the right people see them. Badly written or formatted posts are hard to read, so less masochist readers may stop reading early, or even incomprehensible.

I used to read and edit every single question (and most answers) up until October last year, trying to establish useful conventions in this regard. Since then, I have had less time for SE and only take care of posts that stick out as being particularly badly tagged, titled or formatted. It is amazing how many generic and/or misleading titles, incomplete or wrong tagging and horribly formatted posts there are -- and how few people seem to care.

Lucky for us, these issues can be solved by community moderation. Everybody can help getting contributions the exposure they deserve:

  • Edit titles to be less generic, snappier better related to the question, more interesting.
  • Retag questions that are poorly tagged to use our "category" tags and at least one specific tag (if applicable). Rule of thumb: every question should have at least one tag from the first tag page (if applicable).
  • Edit posts to make them more readable. In particular, introduce basic formatting (lists, math, code go a long way!) and fix crass language issues. Note that which flavors of English you can deal with can depend on your mother tongue, so not everybody will be able to fix every post!
$\endgroup$
1
$\begingroup$

I tend to upvote where I can and where I feel it is necessary (I can't downvote as yet, not enough rep).

Having said that - I upvote when I feel the question is interesting and could, in my opinion, generate several good answers. The same goes for answers - if they are well thought out and interesting, I will upvote as a matter of course. I nearly always accept an answer to any of my questions (assuming they actually answer the question).

With homework questions, I like to give a few hints and resource links and let them go for it (they'll learn more when they 'join the dots' themselves - that's the teacher in me speaking). I am very happy to field homework like questions, in this way, if I can.

I downvote when the question is far too vague or a 'do my homework based question', but also leave a comment asking for clarification - most of the time though, I will leave a comment asking the questioner what they have tried and where they have come unstuck.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .