Draft, pilot, reflect, revise…

Today, 21 June, 2021, my students embark on Stage 3 of their MSc research projects, collecting data, analysing data and all the rest that goes with it. I will reiterate my previous advice to them on the importance of a data requirements table (Saunders et al., 2016, p. 714). This mapping exercise helps ensure that the questions they propose to ask of their participants in a survey or interview will contribute to answering the research question of interest to their project.

Offering advice that I act on myself I will emphasise the importance of conducting a pilot, a “small-scale study to test a questionnaire, interview checklist or observation schedule, to minimise the likelihood of respondents having problems in answering the question and of data recording problems as well as to allow some assessment of the questions’ validity and the reliability of the data that will be collected” (Saunders et al., 2016, p. 723). Let me share my most recent experience of a pilot survey…

My PhD research project aims to explore intergenerational relationships from the perspective of older adults in Ireland, specifically how and the extent to which such relationships support the development of digital literacy skills through informal and non-formal lifelong learning. This is a qualitative case study and I will collect data through documentary research, an online survey and interviews (more about documentary research and interviews another time). I’m a big fan of online surveys, not just for quantitative data but as a means of collecting qualitative data through, for example, open-ended questions and Likert scales. Chang uses the term indirect interview to “solve a problem of distance or schedule conflict” and “can be conducted through multiple media, such as phone, email, letter, questionnaire, and Internet.” (Chang, 2008, p. 105). Of course there are disadvantages to a non-face-to-face approach but that’s a topic for another day.

For previous research studies the pilots I undertook were small, perhaps two or three people who fit the demographic of my intended participants. The goal was to ensure that the questions were easily understood and could be answered without difficulty. This time however, the pilot needed to be on a larger scale and I was thrilled to be able to access a panel through my university in the required age range who might assist with piloting the survey. I felt that 30 responses would be a good indication of what changes the survey might need before being issued to the sample of interest to the study. I sent my request to the centre that coordinates panel feedback with an explanatory note as follows:

I am Sandra Flynn, a PhD Researcher at the Department of Educational Research at Lancaster University. Stemming from experiences of older adults during the pandemic restrictions of the last year, my research is interested in intergenerational learning exchanges between young people and older adults. My study is focussed on the digital skills of older adults in Ireland and your assistance in piloting the survey for me will help ensure its quality. It should take no more than 10-15 minutes of your time if you choose to participate. I am interested in your experience of completing the survey and whether the questions make sense. The final question allows you to enter any feedback you have from your experience of completing the survey. Any information you provide is only for the purposes of improving the quality of the survey and all responses will be destroyed within one month of the survey close date. 

The pilot survey remained open for one week and yielded 46 responses. The data from this small sample proved to be really interesting in terms of the respondents’ experiences of non-formal learning, whether classroom-based or online. While I cannot use this data for my study that focusses on Ireland, all the responses are really interesting. Of the 32 who responded to the final question, for the most part, the feedback was constructive and actionable. It was clear that I had made some unconscious assumptions and I should be careful about my researcher bias (oops). I took some time to reflect further, consider what I should change and what I should not, keeping my data requirements table close to hand since this would also need to be revised (now you see why I mentioned the table at the outset of this post ;-))

My takeaways? Conducting a pilot is hugely valuable, it improves the quality of the data collection instrument that follows and is well worth the time and effort. Don’t take off with your research without it!

Until next time, Sandra

References

Chang, H. (2008). Autoethnography as method (First ed). Walnut Creek, Calif.: Left Coast Press.

Saunders, M., Lewis, P., & Thornhill, A. (2016). Research methods for business students (Seventh ed.). New York: Pearson Education.