Badly designed polls and surveys

A social network specialized in young people performed a few years ago a poll among its members to find out their habits of connection. By the way they asked a few questions to check their knowledge:

·         Do you know how much is a gigabyte of data? 63% answered yes.

·         Do you know how many photos fit in one gigabyte? It should be understood that the question refers to photos taken with a mobile phone, for otherwise the question does not make sense, as the answer depends on the dimensions and resolution of the photos. The authors of the poll assumed that a typical photo takes 70 kilobytes, which means that 14,900 of these photos would fit in a gigabyte (1,048,576 kilobytes). The correct answer they expected was “around 10,000.” (The other options offered were “around 5,000” “around 2,500” and “around 1,000”). Only 10% of those who took the poll gave the expected answer.

·         Do you know how many YouTube videos fit in a gigabyte? This question is clearly absurd, since the answer depends on the size of a YouTube video, which depends on its duration. The researchers estimated that an average YouTube video contains between 1.5 and 3 megabytes, so they expected “around 500” as the correct answer. (The other options were “around 200” “around 100” and “around 50”). Only 7% of respondents gave the expected answer.

From the previous answers, the pollsters concluded that young people do not know the size of a gigabyte of data, although they think they do. Apart from the fact that the survey is poorly designed, as the questions are ambiguous, to understand the results we need to know the answer to the following question:

·         Were the respondents told that a typical photo in the second question takes 70 kB, and that a typical video in the third question take about 2 MB? If they weren’t told, they were not given enough information to answer the questions, and the conclusion of the survey should be: young people don’t know the size of a typical photo or an average YouTube video. If they were given that information, the conclusion should be completely different: young people don’t know how to divide. In both cases, the conclusion drawn by the pollsters is wrong.

In fact, the correct response to such poorly designed questions should have been: I don't know. But today almost nobody gives that answer, because most of us believe that we know everything.


Thematic Thread on Statistics: Previous Next

Manuel Alfonseca

Why polls and surveys fail

Polls and opinion surveys often predict results that never happen. Is there a scientific reason that can explain it? I think so. The problem could be that the mathematical theories behind the polls are misapplied.

A branch of statistics is called sample theory. It was invented to solve the problem of estimating whether the products of a factory are well made or defective, without having to analyze them one by one, which would be too costly.

Let us say, for example, that a factory produces one million screws a day. In theory they should be checked one by one, but since that is impossible, only one part is analyzed. Which part? This is what sample theory tries to solve.

Suppose we analyze just 2000 screws, and find that one of them is defective (0.05%). Can we extend this result to the million screws and assert that in that population there will be approximately 500 defective screws?

Surveys and statistics: opinions and facts

Henry Whitehead (1825-1896)

We tend to confuse the majority opinion with the truth. This is wrong, as expressed by Henry Whitehead:

Never fear forming a minority of one; majorities are usually wrong.

But sometimes the consequences drawn from public opinion are even more misleading than the opinion itself. In an article entitled Are we xenophobic? published in a Spanish high-diffusion newspaper on March 17, 2011, the author discussed the result of an official survey:

...Citizens believe that immigrants receive from the state...

lots more (30.8%) or just more (38.7%) than they contribute.

And he drew from this result the following comment:

Any sociological diagnosis would understand these figures
as a breeding ground for a reactionary and xenophobic social culture.
Saying the opposite is tantamount to denying a consistent reality.