On data collection

Excerpt from the book "How to measure anything" by D.W. Hubbard, 2007, John Wiley & Sons.

Before we measure, we should ask the following questions:

1. What is the decision that is supposed to support?
2. What really is the thing being measured?
3. Why does this thing matter to the decision being asked?
4. What do you know about it now?
5. What is the value to measuring it further?

 Types of observation biases:

• Expectancy bias: Seeing what we want to see
• Selection bias: Even when attempting randomness in samples, we can get unintentional nonrandomness
• Observer bias: (or the Hawthrone effect): Subjects change their behaviors when they are aware that they are being observed.

Notes on data sampling: When you have a lot of uncertainty, a few samples greatly reduce it.

Strategies to avoid response bias:

1. Keep the question precise and short
2. Avoid loaded terms, e.g., ones with positive or negative connotation, like "liberal"
3. Avoid leading question, i.e., it tells the respondent which particular answer is expected.
4. Avoid compounded question.
5. Reverse questions to avoid response set bias, i.e., tendency of respondents to answer questions (scales) in a particular direction regardless of the content.

Biases in human judgement:

• Anchoring (cognitive bias): once respondents are given some numerical values, they tend to fixate on them.
• Halo/horns effect: if people favor or disfavor one alternative, they are more likely to interpret additional subsequent information in a way that supports their conclusion.
• Bandwagon bias: follow the herd.
• Emerging preferences: once people begin to prefer one alternative, they will actually change their preferences about additional information in a way that supports the earlier decision.