Understanding learning in Massive Open Online Courses is difficult, partly because methods and instruments for data collection are under-developed. Published work tends to be from data samples that are too small or skewed. Dropout rates in MOOCs mean that the learners who participate in studies show persistence, therefore samples are not representative. Even if this problem can be reduced, the environment is distributed and varied, so there are multiple variables making data difficult to analyse.
Published research shows that the research methods used tend to analyse registration, participation, retention & progression, completion or assessment data, but none of these conventional course metrics measure learning in a MOOC. The reason is rooted in learners’ motivations for participating in a MOOC. Some learners want to complete the course and gain the certificate, others want to learn specific knowledge or gain skills. Learners in a MOOC don’t have direct instruction from a tutor. The courses have a self-guided format that requires learners to regulate their own learning. Therefore we should expect learner behaviour to vary.
This is particularly true for learners who are working in an area related to the MOOC. Professionals already have (at least baseline) qualifications and tend to have learned how to self-regulated and are motivated. Employers are interested in MOOCs for professional training and learning, yet little is known about how people learn in these open, online environments.
Colin Milligan and I were funded by the Bill and Melinda Gates Foundation MOOC Research Initiative to examine how professionals learn in MOOCs. Data was gathered and analysed with help from Lou McGill, Paige Mustain and Nina Hood. We used self-regulated learning theory as a framework to examine cognitive, affective and behavioural factors influencing learners. These factors range from motivation, interest, confidence and goals to interest-enhancement, learning strategies, critical thinking, help-seeking, self-reflection and self-evaluation. http://www.gcu.ac.uk/academy/pl-mooc/
We wanted to test the hypothesis that learners with high self-regulation have different cognitive, affective and behavioural responses to learning in a MOOC than those displaying low self-regulation. Self-regulated learning is not a ‘learning style’. Rather, it’s a response to a learning situation. Many of the cognitive, affective and behavioural factors can be transformed towards good self-regulation ability. Some factors are relatively easy to influence (for example help-seeking or learning strategies) while others (for example confidence) are more difficult. Nevertheless there is opportunity here to gain insight from both learning and teaching perspectives.
We studied in detail how people learned in two MOOCs: Introduction to Datascience (Coursera and the University of Washington) and Fundamentals of Clinical Trials (edX and Harvard University).
Introduction to Datascience was run by the University of Washington from September to December 2014 using the Coursera platform. There were 40,000 registered learners, though only around one-tenth of these users were active within the MOOC. The research design involved quantitative data gathering using a psychometric survey instrument, designed to measure learners’ perceived self regulation on the course. The MOOC participants were contacted via a course announcement in week 2 (of 8). 280 individuals completed the survey. Participants were asked whether they would agree to a follow-up interview. The quantitative data was combined with qualitative data gathered through semi-structured interviews with 35 participants.
Introduction to Datascience was run by the University of Washington from September to December 2014 using the Coursera platform. There were 40,000 registered learners, though only around one-tenth of these users were active within the MOOC. The research design involved quantitative data gathering using a psychometric survey instrument, designed to measure learners’ perceived self regulation on the course. The MOOC participants were contacted via a course announcement in week 2 (of 8). 280 individuals completed the survey. Participants were asked whether they would agree to a follow-up interview. The quantitative data was combined with qualitative data gathered through semi-structured interviews with 35 participants.
Learners who perceived themselves as exhibiting low self-regulation tended to be focused on completing the course and gaining the certificate. By contrast, highly self-regulated learners wanted tended to link their participation in the MOOC to work performance or personal interest:
“The most important factor… is not even how much I learn, but how big the impact of my work can be to the outside world” (HSRL, 119)
This motivation impacted learners’ goal-setting, self-evaluation and self-satisfaction. Highly self-regulated learners tended to link learning goals with work and were more strategic about where they focus effort:
“The way to approach it [learning] is to follow what interests me and not worry too much about trying to keep a complete overview of the area… I plan to complete all of the assignments[but] I won’t be too worried if I don’t.”(HSRL, 428)
We found evidence that highly self-regulated learners self-evaluate against their own benchmarks and identify progress in relation to intended aims. Learner satisfaction may be high because the learners are self-evaluating their progress against their own goals and ambitions:
“I’m… very satisfied with what I did during the course and what I’ve got out it at the end.” (HSRL, 247)
“Well now I’m feeling more powerful, I can do some things, I am confident in finding solutions for problems that are too big for me right now” (HSRL, 670).
By contrast low self-regulators are focussed on following the instructional pathway and completing the course. Self-evaluation is more difficult for these learners because they are trying to evaluate against externally enforced benchmarks set by instructional designers. This situation impacts on self-satisfaction. When asked about self-evaluation, low self regulators responded:
“It’s hard for me to gauge how much I’ve understood something… sometimes we have a blindness about it ourselves” (LSRL, 236)
“Yeah that’s a difficult question because I don’t perceive my own learning” (LSRL, 396).
These factors impact on learners’ task-strategies (the ways learners approach learning and performance by reducing a task to its essential components and reorganizing these parts meaningfully). Learners who perceive their learning as low-self-regulated try to carry out all (or most) of the MOOC activities, in contrast to high self-regulators who are more strategic about where they focus effort. When asked about whether and how he followed the course pathway, one high self-regulated learner responded:
“Carefully curated parts. So not as a whole, I’m going to be picking through what nuggets are of use to me in particular contexts” (HSRL 505).
Fundamentals of Clinical Trials was one of the first Harvard University MOOCs. It was developed by the Harvard Medical School, Harvard School of Public Health and Harvard Catalyst and ran on the edX platform from November 2013 until April 2014. The MOOC had
24,000 registered learners. The research design used the same method and instruments as used in the Introduction to Datascience study. Learners were invited to participate in the research via a course announcement in week 5 (of 14). 350 learners responded and completed the psychometric Questionnaire. 30 participated in semi-structured interviews. These learners were located in various countries around the world, with 5 in the US and 6 in India (see Figure 1):
In this course we observed a shift in the motivations of those who viewed themselves as highly self-regulated, with both the high and low self-regulated groups being motivated to gain a Harvard certificate. Although both high and low self-regulated groups had the same overall motivation, their approach to goal-setting and learning strategies was different.
Similar to the Introduction to Datascience Course, low self-regulators tended to follow the course ‘pathway’ set out by the instructional designers:
“I do download the study material which is provided by the course website, but while I watch the video I do not have a habit of making notes and I am a person who is organised in a mess. So even if I make a note I don’t recollect and read those notes.” (LSRL, 295)
“I’ve tried to go through the questions first and then go back and review the text to see…and that forces me to kind of focus on the topics a little bit more as opposed to if I go to the lecture and then try to do the questions I find myself zoning out during it.” (LSRL, 360)
However the high self-regulators tended to be more strategic about where they focused their effort:
“I don’t put too much effort into what I’m learning, but this course – looking at the videos I get to take my time to understand. Sometimes I watch the video twice, which has really helped me to have a better understanding when I’m learning.” (HSRL, 284)
In summary we have evidence that in both courses learners who are already qualified and self-regulated tend to follow parts of a MOOC that helps them solve problems. Conventional course metrics, such as progression & retention or completion rates, are poor measures of potential learning gains for professional learners in MOOCs. We need a rethink of certification, completion and measures of success in MOOCs.
For further discussion see the panel discussion led by Yishay Mor and Laia Canals at the #OER15 Conference tinyurl.com/q7ormnl and also littlebylittlejohn/mooc-measures