2.2. ASPIRE Methodology
A rigorous design and development approach for materials is important: a clear, simple, and consistent conceptual model will increase the usability of a system
[40] and support widespread uptake and use of digital resources. Therefore, development and testing of the RLO was undertaken using ASPIRE methodology (
Figure 1) which is a well-used and validated tool for RLO development
[35] and an approach that is suggested to fit optimally with requirements for designing high quality digital training in healthcare
[41]. ASPIRE methodology uses participatory co-design principles and is centred on developing a ‘community of practice’
[42] of experts and potential future users who work together at each stage of the process, to identify learning needs and create content supported by instructional designers and multi-media developers.
Figure 1. ASPIRE methodology for CoVE package development.
ASPIRE has a six-step process
[35,43][35][43] including: (1) establishing the aims of the RLO, (2) storyboarding, (3) populating/production, (4) integration, (5) release, and (6) evaluation, which we aligned to the New World Kirkpatrick Model
[36,37,44][36][37][44]. ‘Aims’ refers to the need to have a clear focus for the resource. This includes the topic area to be covered or learning goal, and the characteristics of the target group of learners. ‘Storyboarding’ is where stakeholders come together to work creatively on ideas for the content and design of the resource using storyboards. ‘Populate and Produce’ is where the ideas are translated into media components ready to be ‘integrated’ together using a suitable platform such as HTML5. ‘Release’ relates to how the resource will be made available to learners via a virtual learning environment (VLE), repository, or website, for example, and how will it be promoted. The final stage, ‘evaluate’ is determining the efficacy of the learning resource in a real learning situation.
Two synchronous consultations with members of the target audience identified the support need and established the key aim for the RLO (Step 1, 2 groups:
n = 48). Asynchronous storyboarding was used to co-construct the content, ordering, presentation, and interactive elements (Step 2:
n = 14). The project team populated the content template and produced relevant graphics and media (Step 3,
n = 3). This was integrated into a RLO template through a technical development process. Both content (specification) and technical (media) development were undertaken by the project team (
n = 3). Two-stage peer review of content and technical presentation was undertaken (Step 3:
n = 23). The final RLO was uploaded to HELM Open (
https://www.nottingham.ac.uk/helmopen/ (accessed 17 December 2021)) and released in May 2021, with evaluation data collected via an embedded survey, and post-training qualitative interviews. The process is shown in
Figure 1, and details for each step are described below.
2.2.1. Step 1: Establishing the Aims
Two synchronous group consultations were undertaken in February 2020 with healthcare professionals (Group 1) and healthcare students (Group 2) in the UK, aligned with scheduled public health education and training sessions. The purpose was to establish the topic area to be covered and the learning outcomes, and the characteristics of the target audience. The focus of these consultations was to explore participants’ views towards, and knowledge of, the COVID-19 vaccine and to discuss barriers and challenges in communicating with patients and clients about vaccination with intention to encourage vaccine uptake. The consultations were led by a health psychologist and health educator, who delivered a 20-min introductory presentation on ‘Public health and vaccines’, followed by a 40-min group discussion. The two sessions were held remotely using Microsoft Teams (Redmond, WA, USA).
Group 1 included 28 nurses, who had been registered between 2 and 40 years (86% female, 32% delivering vaccinations; 25% from ethnic minority groups). Group 2 included 20 healthcare students aged 18–42 years (55% female; 10% delivering vaccines as registered nurses, 65% from ethnic minority groups). Ethnic minority groups were purposely over-sampled due to the variations in the prevalence of vaccine hesitancy, since vaccine uptake (for COVID-19 and previous national vaccination programmes in the UK) has been lower in areas with a higher proportion of minority ethnic groups. Additionally, ethnic minority HCWs are less likely to take up vaccination themselves
[4,18,19][4][18][19]. The views of ethnic minorities were therefore essential in this study. The proportion of attendees from ethnic minority groups was higher than the proportion of minority groups in the general UK population (13%)
[45] and the UK National Health Service (NHS) (22.1%)
[46] and was higher in Group 2 than the proportion of ‘non-White only’ people within the general US population (33.7%)
[47] and US healthcare workforce (35.6%)
[48].
The key points for each discussion group were:
-
Would you have the COVID-19 vaccine if it was offered to you?
-
Would you encourage others to have the COVID-19 vaccine? Why?
-
How confident do you feel in your ability to communicate with patients, clients, or the general public about the COVID-19 vaccine?
-
Are there any barriers and challenges to effective communication about the vaccine?
The vast majority of participants were highly positive towards vaccination and believed that it was important for healthcare professionals to encourage uptake of the COVID-19 vaccine, and other vaccines in general. Many participants indicated that they would take (or had taken) the vaccine themselves and would encourage (or had) their families to vaccinate. However, 25% (5/20) of the healthcare students and 14% (4/28) of the healthcare professionals reported that they would not personally take the vaccine or advise family members to do so. Of the 9 participants who were hesitant to vaccinate, 8 were from ethnic minority groups and they highlighted the speed of development of COVID-19 vaccines, concerns about contracting COVID-19 from the vaccine, and discussed rumours about the vaccine’s purpose and possible side effects circulating within their community groups. These views dominated conversations and led to some group members who were initially positive about the vaccine being unable to respond or doubting their initial response or own knowledge. Most of the healthcare students and many of the healthcare professionals reported having low confidence in their ability to describe key facts relating to the COVID-19 vaccine and to respond to questions from others.
Many participants (particularly those who were less knowledgeable about the vaccine) felt that access to evidence-based information would increase their knowledge and confidence to promote uptake of the COVID-19 vaccine and improve their ability to discuss the vaccine with, and answer questions from others (Figure 2). It was perceived that the volume of online information in public-facing websites was overwhelming and needed to be more digestible. It was unanimously agreed that a digital resource, such as an e-learning package, would be the most appropriate format.
Figure 2. Core outcome of Step 1.
Based on the group discussions and expertise within the project team, the agreed aim of the e-learning resource was to ‘increase understanding about the COVID-19 vaccine and provide a resource that will help healthcare professionals and healthcare students to explain to patients and clients why vaccine uptake is important for individual and societal health’.
2.2.2. Step 2: Storyboarding
In this step, the content for the RLO was drafted, through a process called ‘storyboarding’. A rapid storyboarding exercise was undertaken over a period of one week, with a group of 14 healthcare professionals and members of the public, to establish the key messages, content, and design for the RLO. Due to the urgency of the COVID-19 pandemic situation, the storyboarding was asynchronous (conducted virtually, using prepared resources and without real-time facilitator interaction). This was to ensure that all participants could contribute within a short timescale, with the overall aim of developing a timely and high-quality output that would be of genuine value to health and social care organisations during a global pandemic situation. Individuals were purposively selected via professional networks to ensure participants represented the views of those with knowledge of vaccination programme delivery in different contexts and settings, vaccination uptake and decision-making; the group included 2 medical doctors, 2 health psychologists, 5 nurses, 2 occupational health specialists, and 3 members of the general public. This group constituted an expert ‘community of practice’ to assist in refining the storyboard.
The questions put to the group were:
-
What are the major areas to be covered?
-
What is the best sequence and structure for the material?
-
What do you want the users to be doing at each stage of the process?
-
How will users assess whether they have achieved the learning goal?
The storyboarding activities resulted in a final contents list (Figure 3) and framework for the resource specification, with agreement on ordering, presentation, and the use of interactive elements. The agreed learning outcome was to understand the importance of the COVID-19 vaccine for individual and societal health. Although RLOs are designed to address a single learning outcome, it was agreed that the release of the resource during a global pandemic meant that additional information was needed to ensure that learners fully understood the need to promote adherence to behavioural measures concurrently with vaccine uptake, and to carefully consider mechanisms for promoting vaccine uptake. Therefore, the agreed key message of the RLO content was that ‘a COVID-19 vaccine, when used in combination with current public health measures such as physical distancing, face masks, respiratory etiquette, and hand hygiene has the potential to reduce the significant burden of COVID-19′. It was proposed that general information about the value of vaccines was required to set the context prior to presenting materials on COVID-19 and the COVID-19 vaccine. It was also agreed that the content should communicate that the evidence situation is evolving with relation to COVID-19 and the vaccines. Following the storyboarding exercise, the initial full content draft was co-created by the project team. The project team consisted of a health psychologist, an occupational health nurse, and a learning technologist.
Figure 3. Final RLO contents list resulting from asynchronous storyboarding.
2.2.3. Step 3: Populate and Produce
The content template was then populated by the project team. Interactive images, information buttons, a quiz, and reflection through a feedback survey were included. Pedagogical design principles for multimedia learning were adopted from Wharrad and colleagues
[41] to translate ideas into media components (
Table 1).
Table 1. Mapping of design principles to RLO design feature.
The technical presentation of the RLO allowed for the user to adapt the type of media used to deliver the content, to allow reduction in cognitive load, to maximise accessibility and to improve learning experience. For example, the learner might switch text and audio on or off, and control media elements, by pausing video or slowing down the audio narration. A user can therefore decide how the information is delivered and so the RLO is adaptable to different contexts and devices.
Once the proposed RLO was complete, an international peer review panel of 23 experts was established. Panel members were purposively selected via professional networks to ensure participants represented a range of health and social care disciplines, levels of seniority, and settings. They had expertise in health and medical education, public health strategy, virology, biology, medicine, nursing and allied health, pharmacy, health psychology, sociology, and occupational health. Reviewers included COVID-19 vaccinators and experts in digital health communications and design. Reviewers were from seven countries (United Kingdom, United States of America, Pakistan, Jordan, Turkey, Thailand, and Malawi) to establish the relevance and appropriateness of content across a range of cultures and geographical regions. The review panel completed standard Stage 1 specification review forms accessed from HELM Open (
Supplementary File S1).
2.2.4. Step 4: Integration
The media components of the RLO were integrated using a bespoke, accessible, and user-friendly HTML5 template which embraces a mobile-first design philosophy. The template ensures the best possible user experience whatever device is being used to access the resource. This technical development stage was undertaken by a learning technologist from the Health E-Learning and Media Team (HELM) at the University of Nottingham working together with the project team. To evaluate this, the expert review panel completed standard Stage 2 media review forms accessed from HELM Open (
Supplementary File S2). Taking a pragmatic approach in the context of a pandemic, expert peer review of both specification and media aspects was undertaken concurrently, and the final resource was also tested for understandability and functionality with 5 members of the general public. Iterative review of the resource by all project team members continued throughout the process. The key revisions and overall findings from the peer review process are shown in
Figure 4.
Figure 4. Co-design through expert and lay peer review.
The outcome of content and technical development is shown in Figure 5 (screen examples). The final version of the RLO included audio narration and allowed users to download a certificate of completion.
Figure 5. Screen examples from the final developed RLO.
2.2.5. Step 5: Release
The final RLO was uploaded to HELM Open, a repository of over 200 freely available RLOs at the University of Nottingham. It was released as an open access resource on 3rd June 2021 at the following URL:
https://www.nottingham.ac.uk/helmopen/rlos/practice-learning/public-health/CoVE/, version 1.0, accessed on 17 December 2021) and made available to users by circulating through professional networks and social media.
2.2.6. Step 6: Evaluation
Evaluation aligned with the four levels of the New World Kirkpatrick Model
[36,37[36][37][44],
44], which is a widely used approach to analysing and evaluating the results of training and educational programs. The aim of the evaluation was to explore the perceived influence of CoVE training for health and social care professionals, through:
(i) Level 1: determining user reaction. This is reflected in the degree to which participants found CoVE training favourable, engaging and relevant to them, and/or their job role.
(ii) Level 2: establishing new learning. This is reflected in the degree to which participants acquired knowledge (‘I know it’), skills (‘I can do it right now’), attitude (‘I believe promoting the vaccine is worthwhile’), confidence (‘I think I can promote the vaccine’), and commitment (‘I intend to promote the vaccine’) based on participation in CoVE training.
(iii) Level 3: describing knowledge transfer/behaviour. This is reflected in the degree to which participants applied what they learned from the CoVE package in their job role or daily lives (behaviour change). Required drivers for behaviour change include any processes and systems that reinforced, encouraged, or rewarded promotion of the COVID-19 vaccine.
(iv) Level 4: exploring results and impact. This is reflected in the degree to which targeted outcomes occurred as a result of CoVE training. Leading indicators for this impact included observations that critical behaviours were on track to create a positive impact on desired results.
Table 2 shows a mapping of data collection approaches for each sub-component within the four Kirkpatrick levels. The evaluation took place over an 8-week period July-Sept 2021. Data were collected using pre (2 items) and post (14 items) survey questions embedded within the e-learning package, and a post-exposure interview. Survey items (
Supplementary File S3) were adapted from the ‘Evaluation Toolkit for Reusable Learning Objects and deployment of e-Learning Resources’
[49]. Post-exposure semi-structured interviews were conducted by an independent researcher who had not been involved in the RLO design or development. Interview participants were recruited through health and social care professional networks and promotional mailings. Female participants were purposely over-sampled. This was to reflect the gender balance in health and social care (70% female, across 104 countries
[50]). Potential participants were provided with a link to Jisc Online survey, where they could access the participant information sheet, and provide online consent to take part. Interviews took place shortly after participants had completed the training package (within four weeks) lasted between 12 and 36 min (average 18 min) and followed a topic guide (
Supplementary File S4). They were conducted remotely by telephone or Microsoft Teams, audio-recorded with consent, and were fully transcribed. Analysis followed principles of framework analysis
[51] to allow insights from the interview data to be mapped directly to the Kirkpatrick Evaluation Framework
[36,37][36][37]. On the basis of the researchers’ prior experience in the development and evaluation of digital training packages, and qualitative samples in published evaluations of RLOs or participants’ views towards them
[52,53,54,55][52][53][54][55] (
n = 6–15), we estimated that recruitment of 12–15 interview participants would achieve sufficient information power to map data to the pre-defined criteria of the framework and meet the study objectives. The study protocol was considered exempt from full research ethics review by the University of Nottingham Faculty of Medicine and Health Sciences Research Ethics Committee in July 2021 (Ref: FMHS 310-0721).
Table 2. Measurement aligned with the New World Kirkpatrick Evaluation Framework.