MyLab Economics educator study evaluates homework, exam, and Learning Catalytics scores at University of Wisconsin – Milwaukee
- Data indicate a strong, positive correlation between MyLab Pre-Assignment homework scores and MyLab Problem Set scores.
- Students earning MyLab Problem Set scores above average earned higher average exam and final course grades than students who scored below average on the MyLab Problem Sets.
- 81 percent on an end-of-semester survey agreed that in-class Learning Catalytics questions helped them identify misconceptions they may have had about lecture content.
University of Wisconsin – Milwaukee, Milwaukee, WI
Principles of Macroeconomics
Face to face, flipped
MyLab Economics and Learning Catalytics with Foundations of Macroeconomics by Bade and Parkin
Rebecca Neumann, Associate Professor
Results reported by
Candace Cooney, Pearson Customer Outcomes Analytics Manager
- Locale: large, urban, public, four-year research university located near Lake Michigan
- Enrollment: more than 26,000 students
- Undergraduate: 82 percent
- Residency: 84 percent from state of Wisconsin
- Full-time students: 73 percent
- Four-year graduation rate: 14 percent
- Employment two years after graduation rate: 94 percent
- Faculty-student ratio: 18:1
- Class size: 44 percent with 20 or fewer students
- Gender: 53 percent female
- Diversity: 32 percent total minority
About the Course
Rebecca Neumann has been teaching for approximately 20 years, full time at University of Wisconsin – Milwaukee (UWM) since 2000, where she has been teaching the Principles of Macroeconomics course. Principles of Macroeconomics is a one-semester, three-credit course which enrolls about 1200 students per semester across multiple sections. Neumann’s mass-lecture section typically enrolls 300–400 students. The course is required of Economics majors and all students working toward acceptance to the School of Business; it is also taken by students as a general education requirement or elective. Principles of Macroeconomics is a companion course with Principles of Microeconomics, and the courses together introduce students to the economic way of thinking. Topics include national income accounting, the use of fiscal and monetary intervention to address the problems of inflation, unemployment, economic stability, and the study of international aspects of economic policy.
Challenges and Goals
Neumann began using MyLab™ Economics in 2010; previously, teaching assistants were tasked with grading homework. Given the variables that can be introduced by hand grading, Neumann adopted the MyLab to ensure that homework grading was reliable and consistent for each student. Additionally, she expected that requiring homework for a grade in MyLab would ensure her students would complete this work, which would translate into course success.
To increase interaction in her large lecture sections, in Fall 2016, Neumann supplemented her course assignments with Learning Catalytics™, a personal response system that she expected would encourage students to think and answer questions during lecture, for a real-time classroom experience. Her objective was to engage students in their own learning and make them accountable for understanding course content before leaving class and attempting homework assignments.
Also in Fall 2016, Neumann adopted a key component of the flipped classroom when she modified her course to include new pre-class assignments in MyLab. She anticipated that lecture might be more meaningful if her students had familiarity with the chapter content before attending class and might allow for more active-learning opportunities during lecture.
MyLab Economics is required; the program is used primarily by students working at home or at school on a personal or school computer. Students use MyLab for understanding content, homework assignments, and additional practice. Neumann’s goals for assigning work in MyLab are to introduce new concepts, provide homework and practice opportunities, and to help students assess their own understanding of the course material and track their progress. As the course instructor, her role is to assign content and homework in MyLab and provide remote support to students using the program at home.
Neumann anticipates that students will spend at least two hours per week working in MyLab. Her students confirmed this on a voluntary, end-of-semester Fall 2016 survey (63 percent response rate) — 58 percent of students said they spent more than 2 hours per week working in MyLab Economics, while an additional 42 percent of students said they spent at least 1–2 hours working in the program.
Class meets for lecture twice a week for 50 minutes along with a weekly discussion section led by a graduate student teaching assistant. Neumann videotapes her lecture which she makes available to students for lecture review or to make up a missed class. At the beginning of the semester, Neumann follows a MyLab best practice, inviting her Pearson representative to class to conduct a first-day-of-class orientation to the MyLab and Learning Catalytics (resources can be found at the Pearson First Day of Class website). Course assignments and assessments include:
Learning Catalytics (LC), a bring-your-own-device personal response system, complements lecture instruction by using real-time questions and answers to capture student feedback to content questions Neumann poses. A range of data is captured, allowing her to assess student understanding instantly. Generally, Neumann offers two to three LC questions per lecture. Often, she will ask a question at the beginning of class to either review content from the previous lecture or to introduce new chapter material to be discussed that day to frame the lecture. For some classes, Neumann used a warm-up question unrelated to chapter content that allowed students to simply enter the LC session so their devices were ready for questions during class. An additional question during lecture might address a challenging topic to identify if her students are following the discussion while another question might be used as a review question for an upcoming exam. Some LC questions are discussed by students in small groups before each must submit their own response.
Neumann evaluates the full spectrum of student responses after class to identify groups of incorrect responses, which she uses to review and correct before moving on to new material. Scoring the LC questions is currently based on participation versus accuracy of response. The lowest six scores are dropped and an average is calculated. If the student LC average is higher than the student score on exam 3, students earn the full 10 percent. If the LC average is lower than the exam 3 score, the student earns 0 percent. Students earning 0 percent for LC are not penalized however, as their final exam score will count for 10 percent more than standard grading. Essentially, the LC score does not count against a student but it may help them if they routinely participate. This grading scheme allowed students to opt out of using LC. However, most students chose to participate knowing that the LC component would likely impact their grade positively.
Even though the LC score is for participation, Neumann finds that students are paying attention and engaged in class, and the feedback she receives from the LC data output is helpful to her in identifying weaknesses. Students participating in the end-of-semester survey concur:
- 81 percent of students strongly agreed or agreed that the in-class Learning Catalytics questions helped their understanding of lecture content or helped them identify misconceptions they may have had about the material.
- 83 percent of students strongly agreed or agreed that Learning Catalytics made lecture more interactive and helped them focus on the material being covered.
- 80 percent of students recommended that Neumann continue to use Learning Catalytics in her course.
Student comments about Learning Catalytics include:
- “For me, Learning Catalytics questions were a good indicator of what material I needed supplementation in and where I was standing in terms of the rest of the class. It helped highlight what I needed to work on.”
- “Learning Catalytics encouraged coming to class.”
- “LC was a great way to keep me involved in class discussions.”
- “LC was beneficial because it not only provides instant feedback as to how students understand the material but also lets students compare how they understand things [to other students].”
- “It kept me focused on what my teacher was teaching.”
Neumann’s course is categorized by six main topics covering two to three chapters each, with the following required assignments in MyLab:
- Pre-class assignments: to encourage students to come to class familiar with the lecture topic and prepared to learn, 10 pre-class homeworks comprised of three to four concept-oriented, multiple-choice and true-false questions are assigned. Students have three attempts at assignment completion and the four lowest scores are dropped in calculating a final score.
- Problem sets: In order to help her students understand economics, Neumann assigns problem sets as weekly homework so they practice the skills and apply the material learned in lecture. Each set is comprised of approximately 15-20 questions with a mix of short concept questions and longer, multi-part problems. Students have two attempts at completion for each question and the two lowest scores are dropped in calculating a final score. Learning aids are turned on during these homework assignments. Discussion sections are dedicated to working through problems, including homework, so students have ample options to aid comprehension of the concept before the homework due date.
Pre-class assignments helped Neumann begin the process of flipping her class. Requiring students to be responsible for their own learning prior to lecture allowed her to focus more on the challenging chapter topics and problem-solving in class.
- 78 percent of students on the end-of-semester survey agreed that the MyLab pre-class assignments helped them get familiar with chapter content and prepare for the upcoming lecture.
- 79 percent of students said they would recommend that Neumann continue to use the MyLab pre-class assignments.
Student feedback from the survey suggests that the MyLab pre-class assignments achieved their intended goal:
- “[The pre-class assignments] got me to remember important information for my understanding of the lecture, and proved to be an incentive for me to do my readings.”
- “They were very helpful in introducing key concepts before the lecture actually started, 3 or 4 questions to get the class thinking was perfect in my opinion.”
- “[The pre-class assignments] gave me a preview of what we were learning so I could understand better in class.”
Neumann contends that it is hard to pass her exams without completing the MyLab homework and her students concur. 89 percent of students on the end-of-semester survey strongly agreed or agreed that use of MyLab Economics positively impacted their exam scores.
Neumann encourages her students to use the optional Study Plan and Dynamic Study Modules as preparation for class or review before exams. Although voluntary, students on the end-of-semester survey found these tools very helpful:
- 39 percent of students used the Study Plan on their own for practice or to help identify chapter material they were struggling with.
- 43 percent of students used the Dynamic Study Modules on their own as exam review or to identify chapter material they were struggling with.
Two mid-terms and a final exam comprise the summative course assessments. Mid-terms are timed at 50 minutes and the final exam is timed at two hours, with each exam covering about one-third of the course material. Each exam includes 33 questions of varying type: multiple-choice, true-false, short answer, and graphing and application questions (that are typically worth double points). No make-up exams are given, but students with a valid or prearranged absence may have the weight of the final exam increased if a mid-term is missed.
- 50% Midterm exams (2)
- 25% Final exam
- 10% Learning Catalytics questions
- 10% MyLab problem sets (8)
- 5% MyLab pre-class assignments (10)
Results and Data
Figure 1 is a correlation graph; correlations do not imply causation but instead measure the strength of a relationship between two variables, where r is the correlation coefficient. The closer the r-value is to 1.0, the stronger the correlation. The corresponding p-value measures the statistical significance/strength of this evidence (the correlation), where a p-value <.05 shows the existence of a positive correlation between these two variables.
- A very strong positive correlation exists between average MyLab pre-class assignment scores and average MyLab problem set scores, where r=.75 and p<.05.
The correlation of average MyLab problem set scores to average exam scores was moderate, where r=.32. The purpose of homework is practice and learning, and students have two attempts for each question, which may be a contributing factor to the low correlation. This is identified in the average scores:
Average MyLab problem set score: 90 percent
Average exam score: 71 percent
Even so, students on the end-of-semester survey identified the MyLab problem sets as a critical factor in their course success:
- 96 percent of students strongly agreed or agreed that the problem sets in MyLab helped them understand the chapter material and prepare for exams.
- 95 percent of students said they would recommend that their instructor continue to assign the problem sets in MyLab.
For students, the formative MyLab homework is intended to help them identify where they are in terms of successfully completing the summative tests. Neumann believes the MyLab assignments have a considerable impact on exam scores, noting that students who do the MyLab homework assignments on time and have their misunderstandings cleared up in the discussion section do quite well in the course overall.
Correlation between MyLab pre-class assignment and MyLab problem set score
Figure 1. Correlation between Average MyLab Pre-class Assignment Score and MyLab Problem Set Score, Fall 2016 (n=290)
Although Learning Catalytics was a participation grade only, figure 2 identifies that students earning higher average exam grades also earned higher Learning Catalytics grades. Students commented on the end-of-semester survey about the role LC played in their learning and exam success:
- “It helped to answer questions and get them right, but when I got them wrong my instructor was able to explain why the answer I chose was not the right one.”
- “The LC questions went well with what we were learning and helped me get a better understanding of the material.”
- “Having that interaction in class made it much more possible to understand the material.”
Additionally, figure 3 looks at average MyLab problem set scores, average exam scores and average final course scores before Learning Catalytics was introduced into Neumann’s course in Spring 2015 and after implementation of Learning Catalytics in Fall 2016. Problem set scores remained static but exam scores improved four percentage points and final course scores improved three percentage points.
Relationship between Learning Catalytics score and exam scores
Figure 2. Relationship between Average Exam Grade and Average Learning Catalytics Score, Fall 2016 (n=290)
Problem set, exam, and final course scores before and after implementation of Learning Catalytics
Figure 3. Average Problem Set, Average Exam and Average Final Course Grades before Implementation of Learning Catalytics, Spring 2015 (n=266) and after Implementation of Learning Catalytics, Fall 2016 (n=290)
Students were divided into two groups based on the average score on MyLab problem sets. Students who scored higher than the average problem set score earned higher average exam scores and final course grades than students who scored lower than average on the MyLab problem sets (figure 4).
- Average MyLab problem set score: 90 percent
- Students earning higher average problem set scores had average exam scores nine percentage points higher than students who earned scores lower than the average and final course grades 14 percentage points higher than students who earned lower than average scores.
- 76 percent of students earned a MyLab problem set score higher than average (n=223)
Relationship between problem set scores and exam and final course grades
Figure 4. Relationship between MyLab Problem Set Scores and Average Exam and Final Course Grades, Fall 2016 (n=290)
The Student Experience
Responses from the Fall 2016 end-of-semester, voluntary survey of Neumann’s students indicate that the majority of responding students recognize the value of MyLab Economics.
- 94 percent of students strongly agree or agree that their understanding of the course material increased as a result of using MyLab.
- 89 percent of students strongly agree or agree that the use of MyLab positively impacted their quiz and exam scores.
- 91 percent of students strongly agree or agree that they would recommend MyLab to another student.
Student survey responses to the question, “What did you like most about MyLab Economics?” include:
- “MyLab was the difference in my grade. It really helped me understand the material fully and I would recommend it to any student if they aren’t already using it.”
- “I liked the problems with graphs. It was a great way to learn despite having challenges; since the program doesn’t get frustrated when I fail, I can continue to work on problems to my heart’s content.”
- “I really liked how the problem sets prepped me for the exams. While going through the problem sets, I realized what I knew and what I didn’t know.”
- “[MyLab provided] reinforcement on material and it helped you when you got things wrong without giving away the answer.”
- “I enjoyed the Dynamic Study Modules, the problem sets going over in-depth examples, and other tools to succeed being available with one click.”
- “I liked the problem sets because they helped me to engage with the material.”
In Fall 2016, Neumann added two different interactive components to her use of MyLab Economics. First, she incorporated a crucial component of the flipped class teaching model when she modified her course to include new Pre-class assignments in MyLab. She expected her students would come to class with some background knowledge of the lecture content, allowing for great class discussion and interaction. As noted above, students overwhelmingly agreed that these assignments helped them prepare for and participate more in lecture. Next, to increase interaction in her large lecture sections she supplemented her course assignments with Learning Catalytics questions, another addition to the course welcomed by her students. “[The Learning Catalytics questions] made lecture more interactive” and [The Learning Catalytics questions] keep me engaged in class” were some of the frequent comments echoed by many of Neumann’s students on the end-of-semester survey. Although these questions were graded on participation only, data showed an increase in average exam and final course grades after implementation. Neumann plans to continue to work with and adapt usage of these new components in MyLab Economics in future classes.