1
$\begingroup$

As far as I understand it, the policy regarding list questions is an ongoing discussion (see here and here). The idea is that we should built up a community first, and then discuss how to handle this issue. I agree with this, but I also think that there should be guidelines for the transition phase. Right now the policy seems inconsistent.

My personal opinion is that we should handle list questions as liberal as possible for now.

I had a brief look at mathoverflow, math.SE and cstheory.SE (see for example here). Here it seems okay to have list questions at some extend. Indeed, list questions receive usually a high number of upvotes.

I would like to propose the following: (I extended the following items, some of the comments might refer to the frist version of the question.)

  1. List questions are allowed, if they are cs-related, and the question is a real question. In particular, they should follow these guidelines.
  2. All list questions are tagged. This might be a big-list or subjective tag.
  3. All list questions should be community wikis. (Not so sure about this.)
  4. If a list question turns into a open discussion or a long list of items it will be closed.
  5. If the community has grown, we review our policy.

Discussions are welcome.

$\endgroup$
  • $\begingroup$ Loook at this question -- which is arguably a list question -- and note the quality of most of the answers. Few describe the book the mention or even state whether they just found or explicitly recommend it. $\endgroup$ – Raphael Jul 26 '12 at 5:53
  • $\begingroup$ @Raphael: I agree that the question you mentioned is not the best list-question example. However I would blame it on the question (which was a little diffuse) not so much on the answers. Actually, one of the answers pointed me to a book I wasn't aware of. So this "sub-par answer" was at least for me very valuable. $\endgroup$ – A.Schulz Jul 26 '12 at 7:00
  • $\begingroup$ I think the question was pretty clear. However, the real problem is: can we find objective criteria that allow us to separate good from bad list questions? I struggle with this. $\endgroup$ – Raphael Jul 26 '12 at 8:38
  • $\begingroup$ @Raphael: I agree. That's why I proposed to give every question a fair chance and close it, if the answers turn to item. In order to distinguish them, they should be tagged (maybe 'subjective' is better than 'big-list'). $\endgroup$ – A.Schulz Jul 26 '12 at 8:50
  • $\begingroup$ Note that an answer can be: allow all questions, but delete bad answers. If we have that as a policy, I am happy to press "delete" on every item-style answer, but not without. We have a working-policy not to have meta tags, by the way. $\endgroup$ – Raphael Jul 26 '12 at 9:02
  • $\begingroup$ By the way, in my experience separating question/problem statement from proposal as question/answer-pair is usually a good strategy in order to better interpret votes. People might appreciate that you bring this up but disagree with your proposal. $\endgroup$ – Raphael Jul 26 '12 at 9:17
  • $\begingroup$ I like big-list questions in some cases. think it works well on cstheory, many good examples there. also, short answers as cited by raphael should be discouraged but are not nec a "bad" thing because the voting system will help discriminate them. "can we find objective criteria that allow us to separate good from bad list questions?" voting $\endgroup$ – vzn Mar 25 '13 at 22:41
1
$\begingroup$

We can take a step back and try to look at the question of "list questions" a little more broadly. What is the difference between a "list question" and a non-"list question"?

First off, it's not that there are multiple good/correct/up-voted answers, and it's not that older answers accumulate more up-votes. That's all true (at least potentially, in theory) of any non-trivial question. By itself, it is not enough to distinguish "list questions" from non-"list questions".

Second, it's not that list questions are inherently (bad) subjective. Non-"list questions" can be (bad) subjective, and so can "list questions"; "list questions" can also be objective or (good) subjective.

I feel that that where "list questions" most reliably fail is in the interplay between these two factors. Either the question is too broad or hard to understand, or the answers are (bad) subjective - in the sense that the answerer doesn't bother to explain and motivate answers properly, since this is somehow not expected of answers to "list questions". The former problem is already handled by other means, and the latter can be handled by rules already governing the quality of answers, so long as we make sure to change (and enforce) expectations.

I propose that we allow "list questions" with no special treatment, but enforce the following rules as rigorously as we would on any non-"list question":

  1. The question must ask for a well-defined, sufficiently-scoped list of items.
  2. Each answer must provide objective or (good) subjective argument for the proposed list item(s)

Especially with regard to item 2: simply providing an item as the answer wouldn't cut it, and answers that are clearly (bad) subjective are either removed or converted to comments. With regard to 1, if the question isn't salvageable, it and all the answers are killed with extreme prejudice and without looking back.

$\endgroup$
0
$\begingroup$

The problem with “list questions” — questions like “What are some interesting results about this object?” and “What books are there about this subject?” — is that they do not call for answers, they call for items. The Stack Exchange platform is not well-suited for lists.

Lists don't tend to generate discussion. Usually, the outcome of a list question is:

  • either only a few answers, which makes the answers as a whole woefully incomplete, so that even all the answers taken together do not make a good answer to the question;
  • or many answers, sorted in a way that doesn't cater to relevance, with no indication of why one would prefer this or that answer, with many duplicates, etc.

Voting on items tends to privilege whatever was posted first, and secondly the most popular items. It practically never highlights the best items (whatever best can mean in context) or the most relevant ones.

All list questions should be community wikis.

Making the question community wiki doesn't change anything to the quality of the answers. On this topic, please read the future of community wiki.

List questions sometimes work on Math Overflow and CSTheory because there's a major entry barrier: only a select few like-minded people participate on the site. This significantly reduces the bad, poorly sourced content. The lists of items might not be complete or rated but at least most of the items tend to be relevant. Experience shows that this isn't the case on less selective sites.

Math.SE might be able to afford a few duds now and then, but CS.SE doesn't have the established user base and traffic volume to accommodate a bunch of low-quality questions.

My personal opinion is that we should handle list questions as liberal as possible for now.

Ouch, no. The proper response to expected train wrecks is not to watch them accummulate and cordon off the perimeter and send excuses to the families when they happen. Rather, we should prevent the train wrecks in the first place.

If the community has grown, we discuss this question again.

No, on the contrary, quality can only go down. The time to eliminate bad questions is now, not after the site has failed because there were too few actual questions amid a sea of lists.

Questions on Stack Exchange must be answerable. That means that there is the expectation that one answer will be a sufficient resource on the question. If every answer has one item, that doesn't happen. For example: do not ask “what is the best book about X?” or “I want to read a book about X, what are the possibilities?”, because they call for items. On the other hand, a question like “What collection of books and articles would make a good background for a semester course about X?” can be a good question; different answers might give different reading lists, sometimes mentioning the same book but in a different set and with different rationales. Then each answer is useful on its own.

In a nutshell: don't ask for a book, ask for a curriculum.

$\endgroup$
  • $\begingroup$ See this latest question as an example for poor (if well-meaning) answers on a list question. $\endgroup$ – Raphael Jul 26 '12 at 5:56
  • 1
    $\begingroup$ Okay, I got your point. But isn't “What collection of books and articles would make a good background for a semester course about X?” a list question as well? How to handle questions like "I want to learn more about subject X, which resources are helpful?". I think many people have these kind of questions. Sure you can phrase it one way or the other. Also, If you say "The time to eliminate bad questions is now ..." then there should be a clear and visible policy about list questions soon be available, because it is handled on the different SE sites differently. $\endgroup$ – A.Schulz Jul 26 '12 at 6:52
  • 1
    $\begingroup$ @A.Schulz The key point is that the question must call for answers that contain a sufficient collection of items, not for answers that contain a single item. For recommendation-type questions, there are guidelines (linked from the FAQ) such as “long, not short, answers” and “answers that explain “why” and “how”” and “more than just mindless social fun”. Other than that, the wisdom against list questions arose from experience, and isn't directly reflected in the generic FAQ. $\endgroup$ – Gilles Jul 26 '12 at 7:28
  • 1
    $\begingroup$ @Gilles: Thanks for the link, which I missed. Maybe there is too much confusion what a list question is. Isn't it a feature of SE that you will get a list of answers? And isn't it the case that a combination of answers (let say a problem was solved differently) might be more enlightening than just a single answer? (I understand that it is not helpful to ask for items.) So my impression is, that the core of the problem is more how to handle subjective questions. And this was indeed covered nicely in the blog post. $\endgroup$ – A.Schulz Jul 26 '12 at 7:55
  • $\begingroup$ Gilles, please remember that not all of us (if anyone) has your experience with budding SE sites. What you take for granted, we can not know. Also, evidence suggests that what works depends heavily on the kind of clientele the respective site has so your experience may or may not carry over. $\endgroup$ – Raphael Jul 26 '12 at 9:14
  • $\begingroup$ @A.Schulz The subjective guidelines were intended to be about subjective questions. I don't know of a similar post about list questions; I've tried writing a couple (drawing on experience from many sites, including Science Fiction & Fantasy and Stack Overflow) but I've never been able to fully articulate what works (not much) and what doesn't and why it doesn't. I know it isn't immediately obvious that list questions don't work in the Q&A format; it took a couple of years for the SO community to realize that poll questions weren't doing anything useful. $\endgroup$ – Gilles Jul 26 '12 at 10:08
  • $\begingroup$ @Raphael My primary experience of list questions is with Science Fiction & Fantasy (where we had a lot of them initially), but Stack Overflow has had the same problems, only on SO list questions (including polls, which are a subtype) are a small proportion of the site that one can ignore (but isn't a good representation of the site to good users). CS is closer in audience to SO than to CSTheory. CS is even closer to Math, and I know Math does a lot of things differently; I don't think they've had any success with big lists, but they don't tend to care, and they have a lot of other stuff to compensate. $\endgroup$ – Gilles Jul 26 '12 at 10:12
0
$\begingroup$

The problem with list questions is less with the questions themselves, but with the answers they encourage. We have little effective means to prevent bad answers (between downvoting as mild measure and outright deletion as hammer, there is nothing). Therefore it is easier to close questions that promote (maybe only) bad answers than to try and make the community downvote bad answers. Note that we have been allowing questions -- often inherently list questions! -- which are reasonably scoped.

So, the real question here is:

What to do with questions that explicitly or implicitly ask for flat lists of items?

First, why are lists of items a problem? Gilles has already outlined why they don't work well with the SE platform: usually all answers are of similar objetive value, ranking them makes no sense. As a collection of knowledge, wikis (for instance tag wikis) work better. Furthermore, such lists usually become -- once they converge towards completeness -- impossible to read, because there is no structure.

Is it possible to overcome these issues? Sure. What papers should everyone read? on cstheory.SE asks for semi-objective recommendations and many (of the top) answers give reasons for why you should read the respective paper. What books should everyone read? on cstheory.SE works well because there is one aggregated answer per category, thus the list is structured¹.

Consider on the other hand Beautiful results in TCS on cstheory.SE; this is a completely unstructured, purely subjective blobb of things. Look at Hidden Features of Java which is nice to skim through for entertainment, but a useless reference due to redundancy and lack of structure. What should every programmer know about web development? has escalated in rambling, subjective lists of lists, often without any justification.

What do all these questions have in common? They are broad, and often ask for subjective statements². Which of them do work (as question on SE)? Those where a relatively small group of experts has come together and carefully shaped the list in a useful form.

Therefore, I think we can reasonably allow list questions, but we need a policy that governs aggregation and structuring of the answers. As mentioned in ¹, this was tried in the early days of cs.SE with little success due to lack of consensus and cooperation. I tried to formulate a policy³ but it was downvoted (without comment).

So, bottom line: I have no qualms with list questions as long as they are kept tidy. In order to do so, we need a clear policy (maybe even mandate!) to do so. So, no list questions until we have a policy. I propose my sketch form back then again, although it might need some polishing if we get down to it.


  1. I tried something similar here but the effort was rejected.
  2. To an extent, that is true for any answer: even if I write a proof, the method and style I use is highly subjective. So there is no clear-cut criterion. "How do I best write down an induction proof?" is subjective, but narrow enough to produce good, comparable answers.
  3. Since we have mods now, users do no longer need to delete their own answers; we can do that. In the long run, the community should maintain such questions on its own.
$\endgroup$
  • $\begingroup$ The lack of cooperation is something you should expect. It's part of why list questions tend not to work: there isn't a large enough proportion of the community who cares. This was particularly visible on the ill-fated Literature site, with book recommendations: the few successful book rec questions were the ones that had a good question and where people went out of their way to write really good answers and bad answers didn't get in early. The CSTheory papers question is better than most list questions, but it's still primarily a contest as to who posted first with a bonus for popularity. $\endgroup$ – Gilles Jul 26 '12 at 10:02
  • $\begingroup$ Although I mostly agree with you. I think deleting answers might be harsh (especially in beta) and we will most likely lose new users, which simply do not know about the policy. That's why I was thinking that closing or protecting the question might be a good idea in this case. Of course we should use comments, to suggest improvements to item-like answers. $\endgroup$ – A.Schulz Jul 26 '12 at 14:13
  • $\begingroup$ @Gilles I was/am not surprised by lack of cooperation (although disappointed) but it has to be said. We can always enforce a strict policy; whether people keep liking the site is the quesiton. $\endgroup$ – Raphael Jul 26 '12 at 14:20
  • $\begingroup$ @A.Schulz We can always inform users and ask them to adhere to the policy before deleting with mod powers. Protecting is not a solution; you can not comment on protected questions, those are shut down for good. $\endgroup$ – Raphael Jul 26 '12 at 14:21
  • $\begingroup$ @A.Schulz Deleting bad content drives the site quality up. Leaving bad content around may flatter the people who contribute solely for the sake of having a large number next to their name, but it drives off the best contributors. $\endgroup$ – Gilles Jul 26 '12 at 21:04
  • $\begingroup$ you just said that you have a "deletion hammer" against answers you dont like. you also say you prefer to close questions before you get a lot of downvotes on bad answers. but how can you know all the answers will be downvoted apriori? imho this is clearly moderator bias/overreach. it appears that you want to preempt the populist voting mechanisms of the site. ie you dont fully believe in/trust/embrace the collective intelligence of the users. $\endgroup$ – vzn Sep 28 '12 at 18:41
  • $\begingroup$ I think you misread. We rather close than have lots of upvotes on bad answers. And by "we" I mean the "founding core community". We have to prevent that people joining the party late in numbers destroy the site (e.g. make it a dump for questions closed on Stack Overflow). "Collective intelligence" proves to be a myth time and again; Stack Exchange clearly favors the opinion of valuable members (measured by rep) over that of newcomers. This is clearly designed to prevent trolling. $\endgroup$ – Raphael Sep 29 '12 at 11:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .