Context

I conducted this research project while working as a learning experience designer intern at Health Catalyst, a company of over 1000 employees. This was a fully remote role during the COVID-19 pandemic.

I conducted research on the experience of using a novel learning platform. This platform used brain science and psychology to generate metacognition and enhance learning and retention. The platform asked questions of learners first, before giving them the content. It also dynamically repeated questions that learners struggled on to ensure they understood the material.

The platform was developed to enhance learning, but there were worries that learners would react negatively or would not engage with it as much as a traditional learning platform. I wanted to test these assumptions and find out what learners really felt about the platform.

Role

I joined the team as a learning experience designer. My responsibilities were to take into consideration learner experience, motivation, and best practices for ensuring learning transfer to the job. At the start, I led workshops that explained the design thinking process and the importance of user research. I discovered that I could help out our team by conducting user research on our current offerings so we could find out more about our users as well as discover any improvements to make. I used these to make recommendations to the team as well as create prototypes of improvements.

Timeline

The project took place over one and a half months. Initially, the research project was to be conducted on customers, but the rollout of the learning experiences was not until the end of the year. I decided to conduct research on our internal team members who had taken our learning experiences.

Research Statement and Goals

My objectives for the research project were as follows:

• Understand how and why learners go through our learning experiences, including what messaging they had heard

• Learn of any pain points encountered throughout the learning experience process and improvements they might make

• Discover what learners do after the learning experience and how the learning experience affects their job performance

• Uncover other pathways to achieving learning goals

Research Methodology

For this study, I decided to conduct one-on-one generative research interviews. This method enabled me to dig deeper into understanding our users and foster a strong sense of empathy. Each interview lasted thirty minutes and was recorded for analysis. Questions were intentionally left open-ended to allow participants to speak openly, but the questions were in line with the research goals.

Recruitment Criteria and Process

Participants were randomly selected from a list of team members who had completed at least two learning modules. They had been asked to complete the learning experiences as part of a pilot program. In total, 6 participants were interviewed and used to analyze data. One other participant was interviewed, but data was excluded because of his involvement in the development of several of the modules.

Sample questions asked

• Think about the time you spent going through the learning experience. Walk me through the entire process, starting with your motivation for taking the course.

• Describe any struggles you had going through the course.

  • Was anything particularly frustrating?

  • How did you feel?

  • What did you do to overcome your struggle?

• How has the course impacted your job/life?

  • Where have you used this material in your job/life?

• What did you do immediately after completing the course?

Analysis and Synthesis Process

Each interview was recorded over Zoom. I then used the recording and transcription to take notes using Miro. I then coded the notes and assigned them common tags based on their content (e.g., positive, negative, suggestion, motivation etc.)

I grouped notes according to their assigned tags using an affinity diagramming plugin called Clusterizer for Miro. Using Clusterizer I was able to create a visually organized analysis of my notes, making it easy to see what themes emerged.

Additionally, I used the user research tool Dovetail to find and pull relevant quotes from each of the interviews. I compiled into highlights that I shared with the team and found quotes that were used in the research report.

I used both my notes and the recordings to develop seven main insights that I shared with my team.

Research Outputs

I developed a research report detailing my findings to share with the team. In the report, I included screenshots of the affinity diagram maps, relevant quotes, and links to highlights from the interview. I explained my seven main insights and gave my recommendations to our development teams. The report was well received by my team.

I facilitated a presentation and discussion with our team over Zoom. I explained my process and showed the main insights I gathered. I played audio clips from my interviews with learners to highlight points and bring their voices into the conversation. Throughout the presentation, I invited team members to share their thoughts and opinions. It resulted in some great insights and a lot of empathy for the learners. The chief learning officer, a member of the leadership team at Health Catalyst, was particularly engaged and receptive to the team member feedback. I received a lot of great feedback including this amazing message which I received from a team member.

"Hey Nate. I wanted to let you know that yesterday's presentation was awesome! I also really liked how you included the audio snippets. It was super engaging and informative. Way to go!"

Prototypes and Designs

The main focus of this project was on the research, but I wanted to go beyond and create some solutions to address the learner's needs. This was challenging because we created our learning experiences using a platform that was developed by an external company. I had to focus on things that our team could achieve and not things that required external development. Because of this, there were constraints to what I could propose or change.

Change in messaging

One thing I discovered after researching our learning experiences was that the platform put learners into a performance orientation. Since they were being assessed on their knowledge people felt anxiety and a strong desire to perform well. However, organizational psychology research studies have shown that a learning orientation is beneficial for training transfer. In order to promote a learning orientation, I used error management training principles to adjust messaging that popped up on incorrect answers to ease anxiety and foster learning. I changed the wording from reprimanding phrases to ones that were more forgiving and encouraging (e.g., It's okay to make errors you're here to learn!)

The company that developed the platform loved the changes in messaging so much that they even rolled them out for the whole platform!

Takeaway documents

While participants enjoyed the learning, many mentioned they would have liked to have some materials to reference after the training and while on the job. To address this, I developed a simple template for summarizing the information found within a course. I was inspired by the chapter summaries at the end of textbooks. I worked with the subject matter experts and learning engineers to create these reference materials.

Onboarding tutorial

I also found through my research that the onboarding could be improved. I decided to make a quick onboarding prototype that would help learners new to the platform orient themselves. This would relieve some of the anxiety and put them into a learning orientation. I created the prototype using Figma. As a placeholder, it is going to be inserted at the start of the learning experience. However, the goal is to develop it into the beginning of the learning experiences.

Impact

Overall, my work was very well received within our team. Several of my designs and recommendations have been implemented or continue to be implemented. I changed key aspects of a platform that is used for thousands of learners.

However, the most important part of my work has been building awareness and concern for the learner. Through my research, workshops, and designs, I have been able to spread empathy for our learners and make their success central to everything we do.

Next Steps and Recommendations

This study had many limitations and was not meant to be an exhaustive study on our learners or the platform. Interviews were voluntary and the sample was from learners who completed the modules within Health Catalyst, which may not be representative of all learners within Health Catalyst and was not representative of customers. In order to answer more questions, I suggested further research on the topic to uncover other insights.

Further research should answer questions such as:

• How effective are the learning experiences at imparting knowledge?

• How does learner's initial ability affect their experience?

• How are customer learners different than our client learners?

Based on answers to these questions, prototypes and further changes could be created to enhance the learner experience.