EDITED: I added a second criterion below.
There are some questions which are clearly off-topic. There are also some questions that are borderline. For a borderline question between math and cs, I would use the criteria:
Is this question much more likely to get a good answer in math.SE than in cs.SE?
Is the answer to this question likely to be useful in the future to users of cs.SE?
If the first criterion is "yes" and the second is "no", then I think it should be moved.
While in some sense, this is a fairly pragmatic criterion (in that it leads to the OP getting a good answer relatively quickly), I think it also is a fairly good gauge of whether it's actually on-topic. Suppose we accept the rather tautological definition "computer science is what computer scientists do." Using this definition, a question which people on math.SE can answer much more quickly and satisfactorily than those on cs.SE, and which other computer scientists are not going to be interested in, is clearly mathematics rather than computer science, because mathematicians can do it, while computer scientists can only do it with difficulty and are not that interested in doing it.
For the question which set off this discussion, it had to wait a week before it got answered in cs.SE. There were already several duplicate and near-duplicate questions in math.SE. So by that criterion, I would consider it off-topic. There are some mathematics questions which are just as likely to be answered by computer scientists as by mathematicians; these we should leave open.
If a question is borderline between computer science and an area which doesn't have a good stackexchange site, I'd be in favor of leaving it open.