MyLab Operations Management educator study explores use of Learning Catalytics at Ohio State University

EDUCATOR STUDY

MyLab Operations Management educator study explores use of Learning Catalytics at Ohio State University

Key Findings

  • Students who earned higher than average Learning Catalytics scores also earned substantially higher average homework, quiz, and exam scores, and the difference is statistically significant for all three assessments.
  • Data indicate a strong correlation between MyLab homework assignments and MyLab quiz scores.
  • 98 percent of students responding to an end-of-semester survey agreed that the use of Learning Catalytics in their recitation class helped them understand and apply the chapter contents to problems, mini-cases, and other real-world examples.

School name
Ohio State University, Columbus, OH

Course name
Introduction to Operations Management

Course format
Flipped

Course materials
MyLab Operations Management and Learning Catalytics with Operations Management: Processes and Supply Chains by Krajewski, Ritzman, and Malhotra

Timeframe
Fall 2016

Educator
Steve Lundregan, Instructor and Mi Kyong Newsom, Assistant Professor

Results reported by
Candace Cooney, Pearson Customer Outcomes Analytics Manager

Setting

  • Locale: large, urban, four-year public institution founded in 1870
  • Enrollment: more than 59,000 students in Columbus, over 66,000 total
  • Undergraduates: 77 percent
  • First-year retention rate: 94 percent
  • Six-year graduation rate: 84 percent
  • Student-faculty ratio: 19:1
  • Class size: 72 percent of first-year classes have fewer than 50 students
  • Gender: 50 percent male, 50 percent female
  • Total minority: 19 percent

About the Course

Introduction to Operations Management, a course taken by approximately 2,000 students annually, is a one-semester, three-credit course required of all business students, also offered as an elective for other majors. Operations Management is designed to provide students with a broad understanding of how effective operations and supply chain management contributes to the competitiveness and survival of manufacturing, service, and non-profit organizations. Students apply selected concepts, tools, and methods to address strategic and tactical operational challenges including the ability to analyze, control, and improve critical processes responsible for efficiently making and delivering goods and services. The course explicitly recognizes that the operations and supply chain function is embedded within an organization and also within a supply chain and, therefore, adopts a ‘systems’ perspective in presenting issues, problems, and decision tools.

Upon satisfactory completion of this course, students should be able to:

  • Identify different components of the operations and supply chain function in different organizations (e.g. manufacturing or service, non-profit or for-profit, private or public);
  • Recognize operational problems and decisions and how they relate to and affect other processes within the organization and across value chains; and
  • Apply, analyze, and evaluate various concepts, methods, and tools to improve operational performance by structuring, analyzing, and solving complex problems facing operations and supply chain leaders and manager.

Challenges and Goals

Ohio State began using MyLab™ Operations Management in 2012 as part of a flipped classroom. MyLab enabled Lundregan, Newsom, and their colleagues to remove most of the in-class lecturing so they could focus on problem solving and experiential exercises instead. Students are expected to complete background reading and simple case study analysis prior to lecture to acquire a basic understanding of the concepts, tools, and methods outlined in each chapter. The implementation of MyLab was helping to flip the class, but students were not as engaged in class as they had hoped, nor were they performing as well as expected. To Lundregan and Newsom, it felt more like student compliance as opposed to student commitment. They believed MyLab was providing a great out-of-class experience, but they were hearing from students that they missed lecture and were having trouble making the connection on their own between the content in MyLab and the experiential work in class. Additionally, attendance was down, necessitating re-evaluation of the course set-up.

In Fall 2016, Lundregan chose to restructure his class and pilot Learning Catalytics™, intending to increase student engagement and achievement of learning outcomes through active learning, resulting in a more authentic flipping of the classroom. The goal of the pilot is to use student feedback to redesign and create a better course. Teaching assistants have been hired to collect anecdotal feedback from students, and Lundregan intends to share and report on the process to his students. The pilot will extend to two sections for Spring 2017 with a goal of rolling out Learning Catalytics to all students in Fall 2017.

Implementation

MyLab Operations Management is required; the program is used primarily by students working at home on a personal computer. Students use MyLab for understanding content, homework assignments, quizzing, and additional practice. Goals for assigning work in MyLab are to introduce new concepts, provide homework and practice opportunities, and to help students assess their own understanding of the course material. The expectation is that students will spend approximately two hours per week working in MyLab, which may include reading the eText and completing assignments. Based on results of a voluntary, end-of-semester survey of Lundregan’s sections (89 percent response rate), 49 percent of students said they spent more than two hours using MyLab, while 51 percent of students indicated they spent 1–2 hours using MyLab each week.

In the restructured flipped class, students can expect a weekly cadence of activity:

  • Before lecture – read the assigned textbook and case study content
  • Lecture – attend and participate in one weekly, 80-minute lecture
  • After lecture – complete assigned MyLab homework, MyLab simulation, and MyLab quiz
  • Recitation – attend and participate in one weekly, 80-minute recitation

Lecture is spent exploring concepts, tools, and methods through slide-supported lectures that reinforce the pre-lecture reading assignments, discussion of real-world events, case studies and video materials, as well as the use of Learning Catalytics. Recitation time is spent applying the concepts, tools, and methods through demonstrations of the methods and problems, individual and team-based problem solving using Learning Catalytics, and hands-on simulations.

LEARNING CATALYTICS

Learning Catalytics (LC) use is multi-functional—attendance, participation, and performance. Missing the collaborative in-class experience, Lundregan’s restructured flipped classroom allows for a return to lecturing in his large class meeting, while incorporating LC questions to help both he and his students identify if they are understanding the concepts. Lundregan’s active learning recitation class now engages his students in sharing knowledge and demonstrating the acquired understanding of that knowledge.

  • Lecture: the retooled lecture uses LC to involve his students in the learning process by breaking the experiential learning into smaller pieces. Lecture begins with ‘why’ before ‘what’, where Lundregan discusses a brief case study or vignette about the topic with lecture on the chapter content following. His slide-supported lectures reinforce and supplement the textbook and case study readings, and subsequent discussions of real-world events and cases offer a connection to the business world. Two-way engagement results as students participate by responding to questions while Lundregan lectures, so that his class operates in real-time with the LC responses driving the remaining time in class. LC is more participatory in lecture, used as a guide by Lundregan for content areas that require further explanation and development.
  • Recitation: students explore the chapter content further in recitation where they apply and evaluate what they are learning, demonstrating an understanding through problem solving, case analysis, hands-on simulations, and other experiential assignments. LC is used both for individual responses and team problem solving to score performance in recitation, with students making the connection between content and real-world application. 20 percent of the LC score is based on a student’s individual performance and 80 percent of the score is team-based. The two lowest LC scores are dropped from final LC grade calculation.

On the end-of-semester survey, students shared the following:

  • 85 percent of respondents agreed that the use of Learning Catalytics in lecture helped them identify chapter content they did not completely understand.
  • 98 percent of respondents agreed that the use of Learning Catalytics in recitation helped them apply the chapter content to problems, mini-cases, and other real-world examples.

Student comments about the use of Learning Catalytics included:

  • “I loved doing the problems with Learning Catalytics in recitation because it really helped me to understand what I was doing and helped me prepare for the problems on the exams.”
  • Learning Catalytics provided an opportunity to reinforce the subject matter discussed in class and more importantly gave us the ability to work in groups to help solve problems together, thereby enhancing our communication skills as well.”
  • “I thought Learning Catalytics was a good way to check in during lecture and recitation to help us review.”
  • “[I liked] the ‘live’ aspect of LC, being able to answer questions as the lecture material was presented to me, compared to having homework questions due several days after and trying to recall the information. The application in lecture helped me learn the material quicker; I did not have to go back and review as much as I do in other courses that do not enforce the material in class with LC.”
  • “I liked that it made me think about the material and test my knowledge. Makes it easier to remember important concepts on that day’s lecture.”

MYLAB ASSIGNMENTS

Because instructors only meet with students once a week for an 80-minute lecture, Lundregan and Newsom create the MyLab course using a variety of assignments to foster learning outside the classroom.

  • MyLab homework problems: 12 assignments comprised of approximately 1–2 multi-part questions are scheduled. Students have five attempts at completion and learning aids are turned on. On the end-of-semester survey, 80 percent of students reported that they ‘always’ or ‘usually’ used the learning aids when they were unable to start or finish an exercise in MyLab. Homework is due each Thursday by midnight, the same day as recitation. The two lowest homework scores are dropped in calculation of the final grade. Prior to lecture, Lundregan consults the MyLab gradebook to identify content students are struggling with on homework assignments, so he can address any misunderstandings in class.
  • MyLab quizzes: 10 quizzes are scheduled, comprised of 4–5, multiple-choice questions designed to gauge how well students understand the assigned materials. Students have just one attempt at completion and quizzes are timed to 30 minutes, due on Thursday by midnight. The two lowest quiz scores are dropped in calculation of the final grade.
  • OM simulations: These self-contained simulations help students employ critical thinking and analysis to make operations management decisions in realistic business contexts, gaining exposure to how OM works in the real world. Simulations take approximately 20–40 minutes to complete, students have two attempts at each simulation, and the highest score is recorded for a grade. It is recommended that students complete their first attempt at the simulation prior to lecture so they can participate in discussion and analysis of their approach and results, then complete their final attempt after recitation so they can implement any necessary changes or corrections.

Although students were not required to complete the MyLab assignments before attending lecture, many did so. On the end-of-semester survey, 86 percent of students agreed that completing MyLab assignments before attending class helped them better understand the chapter material discussed during lecture.

Additionally, 86 percent of survey respondents reported that the simulations in MyLab were a real-world application of the chapter material that allowed them to gain hands-on experience with decision making in OM. Student comments about the simulations include:

  • “Simulations were what I liked most because it connected [what we were learning] to the real world where we are going to use this knowledge.”
  • “The simulations were very fun and thought provoking!”
  • “The simulations were very realistic and helped me to have a better understanding of how this course relates to the world outside of class.”
  • “The simulations were actually really cool and pretty fun and taught me about the content.”

Other course assessments include a paper-and-pencil mid-term exam and a comprehensive final exam. The mid-term is comprised of 35 multiple-choice questions and students have 80 minutes for completion. The final exam is comprised of 50 multiple-choice questions and students have 108 minutes for completion. Instructors have created their own internal bank of test questions over time, which includes some Pearson test bank questions, as well.

EMPLOYABILITY/21ST CENTURY SKILLS

Lundregan’s use of simulation exercises and Learning Catalytics in MyLab Operations Management is helping to arm his students with 21st century employability skills. There is a growing body of knowledge about what soft skills are needed to create work opportunities for college graduates today, but these skills themselves can be a challenge for students to develop. The majority of hiring managers place significant value on proficiency in these soft skills, yet less than 38 percent of employers say college graduates are prepared in these areas. The MyLab framework, which includes simulation, critical thinking case studies, and written communication, moves away from memorization and content definition and is instead organized to help learners understand, identify, and develop these core soft skills:

  • Communication and teamwork
  • Decision-making
  • Problem-solving and analytical skills
  • Knowledge application

Assessments

  • 30% Comprehensive final exam
  • 20% Mid-term exam
  • 20% Learning Catalytics assignments
  • 10% MyLab homework assignments (12)
  • 10% In-class participation
  • 5% MyLab quizzes (10)
  • 5% Course participation

Results and Data

Figure 1 is a correlation graph; correlations do not imply causation but instead measure the strength of a relationship between two variables, where r is the correlation coefficient. The closer the positive r value is to 1.0, the stronger the correlation. The corresponding p-value measures the statistical significance/strength of this evidence (the correlation), where a p-value <.05 shows the existence of a positive correlation between these two variables.

  • A strong positive correlation exists between average MyLab homework scores and average MyLab quiz scores where r=.59 and p<.05.

For students, the formative MyLab homework is intended to help them identify where they are in terms of successfully completing the summative assessments; it appears that performance on these assignments could be a leading indicator of course success (additional research is needed to develop and test this concept further).

Correlation between average quiz scores and average homework scores

Figure 1. Correlation between Average Quiz Scores and Average MyLab Homework Scores, Fall 2016 (n=245)

Students were divided into two groups based on the average Learning Catalytics score. Students who scored above average earned higher homework, quiz and exam scores than students who scored lower than the LC average (figure 2). It is important to note that the LC scoring is at least partially participation weighted as described in the Implementation section above.

  • Students who earned higher than average Learning Catalytics scores also earned average homework scores 12 percentage points higher than students who had Learning Catalytic scores below average.
  • Students who earned higher than average Learning Catalytics scores also earned average quiz scores five percentage points higher than students who had Learning Catalytic scores below average.
  • Students who earned higher than average Learning Catalytics scores also earned average exam scores four percentage points higher than students who had Learning Catalytic scores below average.
  • 72 percent of students completed all MyLab assignments (n=176)

A t-test, which measures whether the means of two groups are statistically different (in this study, the two groups are students who scored above and students who scored below the average Learning Catalytics score), was used to compare the scores of all three assessments. In each case, the t-test results of p<0.05 indicate that the increases are statistically significant.

Relationship between Learning Catalytics scores and homework, quiz, and exam scores

Figure 2. Relationship between Learning Catalytics Scores and Average Homework, Quiz, and Exam Scores, Fall 2016 (n=245)

Figure 3 shows a strong linear relationship between the final course grade distribution per average MyLab homework assignment score. Students who earned higher MyLab scores generally earned higher final course grades, indicating that success on MyLab homework assignments may be an early indicator of course success, although a more rigorous study would be needed to explore this concept further.

Relationship between MyLab homework score and final course grade

Figure 3. Relationship between MyLab Homework Scores and Final Course Grades, Fall 2016 (n=245)

The Student Experience

Responses from the Fall 2016 end-of-semester, voluntary survey of Lundregan’s students indicate that the majority of responding students recognize the value of MyLab Operations Management.

  • 95 percent of respondents strongly agree or agree that their understanding of the course material increased as a result of using MyLab.
  • 95 percent of respondents strongly agree or agree that use of MyLab positively impacted their exam scores.
  • 95 percent of respondents strongly agree or agree that they would recommend MyLab to another student.
  • 92 percent of respondents strongly agree or agree that the use of MyLab provided additional resources that helped them learn more than they would have from traditional paper-and-pencil homework.

Student survey responses to the question, “What did you like most about MyLab?” include:

  • “I liked that MyLab enhanced my classroom learning by providing enriching homework assignments and simulations. If I did not know how to approach a homework problem, the ‘Help Me Solve This’ option walked me through how to do it, and I learned quite a bit from that feature.”
  • “I liked that it reinforced material we learned in class, or if we hadn’t learned it already it gave us a chance to try it out before going into class.”
  • “I liked the View an Example capability [learning aid]. It was extremely helpful in showing me how to solve problems and gave me a better understanding of each topic.”
  • “[I liked] seeing if my answer was correct or incorrect, it was like a mini-quiz with immediate results, with the instructor right there for me to ask questions, if necessary.”
  • “[I liked] the detailed explanation in Help Me Solve This [learning aid] when I have no clue how to solve a homework problem.”

Conclusion

Using MyLab to flip the class initially provided Lundregan and Newsom’s students with a beneficial out-of-class experience, but they found that by itself, it was not providing the in-class experience they were expecting. Students self-identified that they were not making the connection between concepts and application, and the department sought a digital course component that would increase student engagement in lecture while encouraging active learning and a true flip of the classroom. Accountable for what he wanted to accomplish, Lundregan chose to pilot Learning Catalytics in Fall 2016 to increase student participation and help his students bridge the gap between understanding content and real-world application of the content. Using LC in both lecture and recitation, but employing different tactics, students reported overwhelmingly on the end-of-semester survey that LC helped them better understand the course material and helped them apply that content to cases and simulations with a real-world perspective. Additionally, data show that students who earned higher average LC scores also earned higher average quiz and exam scores.

Lundregan, Newsom, and their colleagues plan to assess their implementation of LC at the end of Spring 2017, with a goal of having all students using LC in Fall 2017 classes.