Panel mixed-mode effects: does switching modes in probability-based online panels influence measurement error?

Abstract

Online probability-based panels often apply two or more data collection modes to cover online and offline populations, and to collect data from onliners who do not respond online in time to contribute to a given wave. As a result, offline/ online status can change during the life of the panel for some individuals, which can improve response rates and representativeness, but may cause increased measurement error. In this study, we use Life in Australia™ survey data and online panel paradata to identify respondents who switched modes; almost 4% of the whole panel was interviewed using both online and offline modes in the first 2-years, and almost one-third of those 4% switched mode more than once. We selected all repeated substantive survey items, identified any relevant changes in responses that could be explained with mode effects, and determined the effect of mode switching on changes to answers, controlling for panel conditioning, panel fatigue and sociodemographic characteristics of panellists. This study identified a limited number of panel mode effects from panellists switching modes of data collection over time. We found evidence of recency and some social desirability, and established that measurement error may be more common when the proportion of mode switchers is higher. Moreover, panel conditioning had an effect on the frequency of changing answers; respondents provided more stable answers if they were more conditioned. We conclude that combining mode effects with panel conditioning, as well as an increasing representation bias over time, may lead to less accurate estimations in longitudinal surveys. Show more