Demos.science Education is an experiential learning platform that aims to enhance our understanding of how randomized trials work, giving us a better idea of how evidence-based answers should be obtained.
This is a pilot study to showcase Demos.science Education as an experiential learning platform that engages and educates the public on randomized trials. It is not intended to be a rigorous study on the impact of music on concentration.
If you choose to participate in the pilot study, you will have an opportunity to learn how a randomized trial works, experience what it means to participate in a randomized study, and acquire a better understanding of key terms and concepts used in the experimental research process.
The pilot study is open to persons 18 years of age and older. Participating in the experiment will require approximately 5 minutes of your time, and you can jump straight in here, or spend a few minutes reading through this page first, familiarizing yourself with how randomized trials work and why they are so important.
We will run through what a well-run trial should look like, drawing on the CONSORT Statement, which sets out how randomized trials should be reported in medical research – but don't worry, this is a much abbreviated and simplified walkthrough of the main ideas. To illustrate the different points, we will use the pilot study on the effects of different music types on concentration, working our way from early conceptualization to reporting the final conclusions. You will also have the opportunity to take part as a study participant so you can familiarize yourself with the study process from different standpoints.
As a starting point, we need to explain the background and rationale for the trial, and its general outline.
The context for this study is the COVID-19 pandemic, which resulted in many of us being asked to work from home, where we faced many more distractions affecting our productivity. What tools can we use to enhance our productivity and concentration? Music has been touted to improve concentration and focus. But before we rush out to buy the latest active noise-cancelling headphones, how can we find out if listening to music actually does enhance concentration and focus? And if so, what type of music might be most helpful?
The first thing we need to do is determine if previous studies have already provided an evidence-based answer to our question. A research study should aim to generate new knowledge, building on existing evidence. This is particularly important in studies that might pose risks to the participants. Conducting a study when previous studies have already conclusively answered the research question may be unnecessary and unethical.
Here, we find that previous studies have examined the effects of background music in the workplace, but these have often included very specific groups of individuals, like young males, or members of certain professions. Some studies have shown that music preference, age and gender can change how music affects concentration. After reviewing other studies, we might conclude that there is a need for a further study that includes a broader representation of working adults, and more evidence is needed to find out if different music types affect concentration differently. So we can narrow our research question down to:
Which music types are best at improving concentration?
The research question describes the overall question the study is addressing. For randomized trials (and other experimental studies) it is often more useful to have a specific and testable hypothesis – a statement setting out what the experiment will either confirm or refute. In this case, we are unsure which music genre will enhance concentration compared to not listening to music. So, we will frame our ‘non-directional’ hypothesis as:
Listening to different music genres will affect concentration differently.
To keep things simple, we will limit our investigations to two music genres - classical and techno.
We now need to select our study design and devise a testing method. The study design and testing method should enable us to prove or disprove the hypothesis: listening to different types of music affects concentration differently.
To allow us to test our hypothesis, we will need to compare participants’ performance across three study groups:
But how are participants allocated to each group? Letting participants choose what music they want to listen to and comparing their performance in the attention task can completely undermine the validity of the study’s conclusions! A much better design is a randomized trial, where participants are allocated to different study groups at random.
In this instance our study design will be a ‘multi-arm randomized controlled trial’. (It’s ‘multi-armed’ as we have two intervention groups listening to different music types and a ‘control’ group listening to white noise.)
We now need to decide what the eligibility criteria are – who will be included in the study? Importantly, our eligibility criteria should aim to ensure that the study participants reflect the population where we expect the intervention will be used. If our study participants are different to the population of interest, our results might not be generalizable to this population. For example, if all our study participants are classically trained musicians (perhaps because we recruited all our participants from a music school), our findings from the trial might not apply to non-musicians. Another major concern is that we exclude people who might be at higher risk if they participate in the study. So, we might need to devise exclusion criteria to avoid exposing them to unnecessary risk. In this pilot study, we are interested in the impact of music on concentration for adults working remotely. Our population of interest is working adults. So, our inclusion criteria will be:
We now describe the trial interventions. In medical research this might involve a new treatment or vaccine. A detailed description of the intervention is critical: it might be the case that a drug doesn’t work at a specific dose in one study, but would work very well at a higher dose in another study. By stating exactly what the intervention is we can also ensure reproducibility in research – the ability to repeat the experiment and obtain similar findings.
In this trial, we have two intervention groups and a control group:
Play a sample of the classical music
Play a sample of the techno music
Play a sample of the white noise
We now need a tool that will allow us to measure concentration – an attention task. Here, we use a standardized test, known as the Sustained Attention to Response Test (SART). The SART is a computer-generated test that measures individual performance in attention tasks. Participants must tap the spacebar every time a number flashes on the screen, but refrain from tapping when a specific number - "3" - appears. For this study we developed an open-source three-minute version of the test.
Generally speaking – the more participants we have in a study the higher the probability of our experiment proving (or disproving) the hypothesis correctly. On the other hand there are good reasons to keep the number of participants to a minimum, such as the logistical requirements and high costs for running studies. Importantly, if there are any risks for participants it would be unethical to recruit more participants than needed to generate robust findings. So to ensure our study yields a ‘scientifically valid’ or ‘statistically significant’ result, we need to calculate the minimum number of participants needed – our sample size – for each experimental group in our study.
In our pilot study, we estimate that we will need 202 participants in total.
As mentioned, randomization of participants into study groups is critical to avoiding bias and confounding in the findings. How to ensure true randomization is actually much more challenging than you might expect! Recruiting into different groups on alternative days for instance might seem like a simple approach but can open the door to confounding factors or even ‘gaming’ the system (participants choosing to enroll on days when they will be allocated to a preferred treatment).
In our study we’re making use of an open source randomization tool we developed, whereby participants are redirected to one of three different webpages where they will listen to the different music genres (or white noise) while completing the SART.
Ideally, the trial would be designed so that neither participants nor researchers have any knowledge of which study group participants are placed in until the trial itself is completed. The ‘unblinding’ occurs only when the data have been analyzed.
In our pilot study it is impossible to blind the participants, as they will know which group they are in. Because there are no researchers involved in their recruitment or assessment to be blinded either, the study is considered ‘unblinded’
We are however keeping the results partially concealed until we recruit enough participants to meet the sample size. Why? Because if participants see that, for example, those listening to techno are performing much better than those listening to classical music, this could affect their own performance in the experiment.
We must have a clear plan for analyzing the data we collect. A trial should have a single ‘primary’ outcome. Before we begin the experiment, we should determine which statistical test we will use to decide if the hypothesis is correct. If there are multiple outcomes and we use different or unsuitable statistical tests to compare them, we might find differences between study groups just by chance. If we report these "findings" people might believe the intervention has effects that it doesn't actually have.
In our study we will compare the average (or ‘mean’) SART error rates between the three study groups. The full statistical plan is available (not for the faint-hearted!)
Before enrolling participants in a study, the researchers must obtain the free, voluntary and informed consent of each prospective participant.
In the pilot study, participants will have an opportunity to experience a simplified informed consent process. Participants will be presented with an overview of the study that sets out key points at a quick glance. Participants will then be provided with a link to a downloadable detailed Study Information Sheet. Participants will be asked to provide free, voluntary and informed consent by clicking on three consent boxes, confirming: (1) they understand what it means to participate in the study, and voluntarily agree to participate; (2) they understand and acknowledge the risks involved in participating in the study; (3) they are 18 years of age or older.
Prior to recruiting participants and conducting the study, we need to obtain ethical and regulatory approval from local authorities. For a study to be viewed as ethical, it must abide by four guiding principles:
The ethical review process will vary depending on the nature of the experiment, the participants involved and the context in which the study takes place.
This study complies with ethical standards and codes of conduct of the University of Oxford. This pilot study is conducted for educational purposes to teach participants about randomized trials, with minimal risk to participants and no directly identifiable information being collected. After consulting an Ethics Committee at the University of Oxford it was agreed that no review would be necessary.
You can now take part as a participant in the randomized trial. This is completely voluntary and the data collected is confidential.
After running the experiment, we now need to compare the data between the intervention groups and the control group. Go ahead and see the live results of the study so far!
With our results in hand, we can now go back to our research question and hypothesis, and discuss how we might interpret the findings. What were the strengths and weaknesses (or limitations) of our study design? How generalizable do we think these results are to a real life, non-experimental setting? And what about generalizing to populations that differ from those in our study, such as children? How do our results compare with other studies? We might also suggest new research questions. Can we consider whether personal music preference will affect the extent to which music improves concentration? Is listening to music more beneficial in some types of tasks than others?
Researchers writing up their conclusions will often be very cautious in stating the limitations of their studies, avoiding bold statements such as ‘Our study definitively proves that techno music improves concentration!’. This cautiousness is all too often abandoned in popular media when they report on research findings.
Thank you for joining us in the Demos.science Education Pilot Study! We hope this experience has been informative as to how randomized trials are conducted and how they can generate evidence-based answers to everyday research questions. Once we have enough participants we will run the full (unblinded) analysis and write up the conclusions. If you would like to be kept informed of this and other Demos.science news please sign up on our website.
Your feedback will be very helpful for us in deciding whether to continue developing this educational component of the Demos.science platform.
Share this page:
This project was supported by a Wellcome Trust funded Public Engagement Bursary at the Mahidol-Oxford Tropical Medicine Research Unit (MORU).