Badly designed polls and surveys

A social network specialized in young people performed a few years ago a poll among its members to find out their habits of connection. By the way they asked a few questions to check their knowledge:

·         Do you know how much is a gigabyte of data? 63% answered yes.

·         Do you know how many photos fit in one gigabyte? It should be understood that the question refers to photos taken with a mobile phone, for otherwise the question does not make sense, as the answer depends on the dimensions and resolution of the photos. The authors of the poll assumed that a typical photo takes 70 kilobytes, which means that 14,900 of these photos would fit in a gigabyte (1,048,576 kilobytes). The correct answer they expected was “around 10,000.” (The other options offered were “around 5,000” “around 2,500” and “around 1,000”). Only 10% of those who took the poll gave the expected answer.

·         Do you know how many YouTube videos fit in a gigabyte? This question is clearly absurd, since the answer depends on the size of a YouTube video, which depends on its duration. The researchers estimated that an average YouTube video contains between 1.5 and 3 megabytes, so they expected “around 500” as the correct answer. (The other options were “around 200” “around 100” and “around 50”). Only 7% of respondents gave the expected answer.

From the previous answers, the pollsters concluded that young people do not know the size of a gigabyte of data, although they think they do. Apart from the fact that the survey is poorly designed, as the questions are ambiguous, to understand the results we need to know the answer to the following question:

·         Were the respondents told that a typical photo in the second question takes 70 kB, and that a typical video in the third question take about 2 MB? If they weren’t told, they were not given enough information to answer the questions, and the conclusion of the survey should be: young people don’t know the size of a typical photo or an average YouTube video. If they were given that information, the conclusion should be completely different: young people don’t know how to divide. In both cases, the conclusion drawn by the pollsters is wrong.

In fact, the correct response to such poorly designed questions should have been: I don't know. But today almost nobody gives that answer, because most of us believe that we know everything.


Thematic Thread on Statistics: Previous Next

Manuel Alfonseca

Why polls and surveys fail

Polls and opinion surveys often predict results that never happen. Is there a scientific reason that can explain it? I think so. The problem could be that the mathematical theories behind the polls are misapplied.

A branch of statistics is called sample theory. It was invented to solve the problem of estimating whether the products of a factory are well made or defective, without having to analyze them one by one, which would be too costly.

Let us say, for example, that a factory produces one million screws a day. In theory they should be checked one by one, but since that is impossible, only one part is analyzed. Which part? This is what sample theory tries to solve.

Suppose we analyze just 2000 screws, and find that one of them is defective (0.05%). Can we extend this result to the million screws and assert that in that population there will be approximately 500 defective screws?

On-line bullying

In an article by the press agency Europa Press published on June 28 2012, which refers to a study performed by Microsoft among youths in the age range 8 to 17, it is stated that 37% of Spanish youths suffer on-line bullying through the Internet. This looks like a high figure, but it may depend on how bullying is defined.

Reading the article, it appears that 17 per cent of the polled declares having been addressed in an unfriendly way, 13 per cent have been targets of mockery and 19 per cent have felt insulted. Also, 24% of the youth confess that they bully other people.

Neither in the Europa Press article, nor in the summary of the Microsoft study, is there a definition of unfriendly behavior and the other forms of bullying. It appears that the youths who answered the poll just considered it thus.