Item non response in surveys among adolescents in Iceland

Abstract

Several school-based surveys among Icelandic adolescents are conducted regularly. The topics of these surveys include mental and physical well-being, substance use, academic work, school environment, leisure and sports activities, media and internet use and family environment. Results from these surveys play a significant role in the making of educational and youth policies in Iceland.

The questionnaires used in these surveys are quite long and sometimes involve demanding response tasks. As a result, it is quite possible that item missing data might be missing systematically, leading to substantially biased findings, especially if statistical analyses that adjust for missingness are not used.

In the present study we look at the proportion of item missing data in three large surveys among Icelandic adolescents, European School Survey Project on Alcohol and Other Drugs (ESPAD), Health Behaviour in School-aged Children (HBSC), and the student questionnaire from the Programme for International Student Assessment (PISA). Analyses of the ESPAD data were restricted to respondents that reported their gender and we used the raw data as it appears before logical substitution and the removal of extreme respondents (n=2609). Analyses of the HBSC data were restricted to respondents that reported their class and their gender as either male or female (n=6946). Analyses of the PISA student questionnaire were restricted to those students who did not receive the simplified une heure test booklet (n=3230). Proportions of missing data were analysed graphically, both by item location and question format, using a novel type of diagnostic plots.

We examined two types of missing data. We looked at breakoff rates, where participants completely drop out of the survey and stop responding altogether. Secondly, we examined the item specific non response rates, where participants skip items but respond to items that appear later.

All three surveys had a substantial proportion of item missing data but this proportion varies greatly by item position, format and substance. The ESPAD questionnaire appears to be rather too long for the target population and the PISA student questionnaire is far too long. In HBSC there is a distinct association between question format and item missingness, especially among younger respondents. The other two surveys consist mostly of grids and single answer questions, resulting in insufficient question format variability to separate the effect of item format from that of location. For most of the duration of the survey the breakoff rate was substantially lower in HBSC than in the other two surveys. This could be because participants who dislike or have difficulty with question grids can skip over them and find other question formats that agree with them later on. However, there is also more variation in the substance of questions in HBSC and this could account for some or all of the difference. The overall rate of item missingness (due to both breakoff and item specific non response) is not much lower in HBSC than in the other surveys, once questionnaire length has been accounted for.

Breakoff and item specific non response rates differ visibly across gender in all surveys and class in HBSC. Item specific non response is more common among boys in all surveys. Boys also have higher breakoff rates in HBSC and ESPAD but not in PISA, presumably because skipping more items made them less likely to run out of time. In the ESPAD data responses to questions on smoking and absenteeism (that appear early in the questionnaire) are clearly associated with both breakoff rates and item specific missing rates later in the questionnaire.

We conclude that item missing data in all surveys is far from being missing completely at random and has the potential to bias results substantially.

Finally, we discuss reasons for making questionnaires in surveys among adolescents as long as they are and whether this is truly necessary. We discuss options for reducing participants’ response burden and/or mitigating the biasing effect of item missing data.