Lessons learned from a pilot focus group with colleagues

A few months ago, before conducting focus groups with nurses and assistant nurses at the Uppsala University Hospital for my first field study within my PhD – investigating the effects of IT use for patient care management on nurses and their work – my co-author and I set up a pilot study with a few of our colleagues. My goals with this endeavor were to:

  • Check whether we would be getting the kind of data we needed from the discussion (did our questions trigger the kind of answers we were looking for?);
  • Identify “glitches” in the design of the discussion;
  • Get feedback on the study design in order to be able to improve on it;
  • Get to practice my new role as the discussion moderator, which I hoped would lead me to feel more at ease and more “fluent” during the real event.

Of course, we needed to adjust the topic of the discussion to the specific context of the pilot in order for it to be meaningful for our participants. We also shortened the duration of the activity in comparison with the “real” study design (from 90 to 60 minutes), and made sure to include some time at the end to discuss the study design in itself. On the D-day, 3 of our colleagues thus gathered for an hour to discuss recent experiences with some of the IT systems they used at and for work, as well as to reflect on the consequences those experiences had for them and their work.

Now that we have performed 2 focus groups in a “real” setting, here are the key lessons – in terms of what it enabled us to do / see and where it was misleading – I am taking with me about conducting such a pilot focus group with colleagues:

  1. It enabled us to ensure the validity of the study – to make sure that we actually were getting the kind of data we were looking for.
    Although the topic of the discussion was not the same as in the “real” study, we were able to see whether the questions I was asking the participants to discuss led them to mention the kind of aspects we were interested in.
  2. It made it possible for us to identify some unanticipated issues in the flow of the discussion.
    For example, I realized that one particular idea we had had in order to get the discussion rolling was not working well. We were able to come up with something better for the real study – in part also thanks to the feedback we got from my colleagues at the end of the discussion.
  3. It brought to light some divergences in perceptions between my co-author and I.
    My co-author being in charge of taking notes during the discussion (and the notes being made visible to the participants in real-time), I realized for instance that I had expected a different kind of notes from the discussion than what she was writing. Becoming of aware of these differences in expectations enabled us to discuss them openly – and to come to an agreement in regard to how things should be done during the real event.
  4. We received very valuable feedback from our participants / colleagues.
    Having “experienced” our study, our colleagues were able to use their own experience with data collection in order to reflect on it critically and give us constructive feedback as well as suggestions on how we could improve on our design. This for example led us to move from a paper-based note-taking technique to the adoption of a digital note-taking tool for the “real” focus groups.
  5. Work on those follow-up questions – Academics are not (necessarily) representative of the broader population (or the population targeted by the study).
    Although the conclusions we drew from the pilot workshop about the validity and design of our study were mostly supported by our subsequent experiences in the real study setting, something that I had overlooked is that academics (and maybe especially colleagues) should not be considered as representative of the population targeted by the study. Academics are used to analyze and reflect on their practice as well as to put those reflections into words – which might have led to their finding it easier to express their views on the topic at hand. This is not necessarily the case for other populations, which means that getting academics to talk about a topic will not mean that your study participants will be as talkative. Seeing how easily my colleagues had become engaged in the discussion, I expected for the discussion to go as smoothly in the real setting and did not spend enough time thinking of possible follow-up questions to re-launch the discussion, which was a mistake.