Member-only story
The hidden universe of data analysis
Analytical decisions affect estimates when working from identical data.
When analysing data, what decisions do we make? Given a problem to solve, there are many dividing paths we could take. Those decisions could affect outputs of our analysis, such as estimates and intervals.
A pre-print analysis gave the same data and question to 73 research teams. The data involved six questions from multi-country International Survey Programme. The teams also had measures of the stock and flow of immigrants to each country. The hypothesis was whether immigration reduces support for social policies among the public.
These 162 researchers then had to catalogue their decisions, submitting 1,261 models. Also, there were 89 substantive conclusions. This is higher than the number of teams. There were two different ways of measuring the test variable. Of the teams, 16 thought these different measurements were two different hypotheses. As such, those teams submitted two conclusions each.
There was a large variation in estimates by each research team. Weighted by models per team, 25% suggested negative effects. In contrast, 17% suggested positive effects — opposite to the hypothesis. A similar pattern comes when considering the substantive conclusions. Of the 89 conclusions, 23…
