Approaches to dealing with survey errors in online panel research

Abstract

Survey research is a relatively young field, and online surveys including online panel surveys are now routinely used for collecting survey data. We distinguish between different types of online panels, and this thesis is focused on both probability-based and nonprobability-based general population panels. To increase the quality of online panels in the era of nonresponse, more methodological research is needed, and that is the focus of the research in this thesis. To investigate approaches to dealing with survey errors, the Total Survey Error paradigm as a conceptual framework is applied, and both errors of representation and errors of measurement are the subject of this research. One of the contributions of this thesis is a review and discussion of a range of data sources and methodology which can be used in the study of survey errors. The other theoretical and practical contributions, presented within three groups, are related to the investigation of individual types of survey errors in online panel research. First, worldwide probability-based online panels are identified, and their methodological approaches to recruitment and data collection reviewed and compared as part of a meta-analysis. The study shows high levels of heterogeneity in both recruitment rates and recruitment solutions, as well as explains variability of recruitment rates. The other studies on errors of representation present evidence on how online panel paradata can be effectively transformed and used to identify about three in four nonrespondents in a subsequent panel wave, and answer the question of why people participate in online panel surveys while presenting evidence on how social-psychological theories can explain survey participation in a longitudinal design. Second, two studies focus on measurement error in probability-based online panel research due to mixing modes. The study on measurement mode effects shows how measurement error is present in the case of a lack of measurement equivalence between modes, and presents evidence on how applying matching methods (like coarsened exact matching) quite effectively controls for self-selection bias due to non-random assignment of online panellists to modes. The study on individual-level measurement mode effects presents a newly identified source of measurement error in online panel survey, that is, panel measurement mode effects. It also conceptualizes and showcases how panel conditioning can be a factor of two measurement aspects. These results are later related to a trade-off between representation (undercoverage) and measurement bias. Third, the thesis studies two cost- and time-efficient approaches to online data collection - nonprobability online panels and a fairly new combination of random digit dialing, text message invitations, and web-push methodology. The study on nonprobability panels, which are generally considered as less accurate but cheaper than probability-based panels, investigates post-survey adjustment methodology to improve inference in nonprobability samples. It presents evidence on how accuracy can be improved under different external data access scenarios. The study on a new approach to online survey data collection shows very low response rates, and outlines effective solutions to increase response (such as advance SMS and reminders). It also presents evidence on the fairly high accuracy of the proposed approach, which seems to be feasible for continuing recruitment to a probability-based online panel. In the final section of the thesis, the cost dimension of online survey research is discussed, the requirement of collecting data from the offline population in probability-based online panel research from different perspectives is challenged, and the theoretical contributions of this research are explained in more detail.