MasteringEngineering educator study measures impact of a flipped classroom redesign at University of Texas El Paso
- In this study, average final exam scores and course success rates were higher after implementation of MasteringEngineering than before implementation.
- After the frequency of assigned MasteringEngineering homework increased, the correlation between the homework and exam average was stronger.
- According to the instructor, adding Mastering enabled him to flip the classroom and to better monitor student performance during the semester.
University of Texas at El Paso, El Paso, TX
Face to face and flipped
MasteringEngineering; Engineering Mechanics: Statics by Hibbeler
Spring 2014, Fall 2014, and Spring 2015
Calvin Stewart, Assistant Professor
Results reported by
Betsy Nixon, Pearson Customer Outcomes Analytics Manager
The University of Texas at El Paso (UTEP)’s location along the US–Mexico border makes it one of the world’s largest binational educational communities, serving more than 23,000 undergraduate and graduate students. In 2014, 62 percent of students were enrolled full-time, and 80 percent identified as Hispanic. In addition, because of the school’s proximity to a large military base, there are a large number of veteran students, as well as many first-generation students.
About the Course
Assistant Professor Calvin Stewart has been with the mechanical engineering department at UTEP since joining the school in 2013. He completed his Ph.D. in mechanical engineering from the University of Central Florida.
The face-to-face Statics course for mechanical engineering majors has a prerequisite of Calculus, and it is recommended that students have taken Physics I. Students who successfully complete the course understand the principles that govern the behavior of rigid-body mechanical engineering systems in static equilibrium. Specifically, they are able to:
- Identify an engineering problem appropriate for engineering mechanics analysis.
- Draw a free-body diagram and identify all forces and moments acting on an object at rest.
- Represent force and moment systems with equivalent systems.
- Perform an analysis to identify all forces and moments acting internally or externally on an object.
- Determine geometric properties of one-, two-, and three-dimensional objects.
Challenges and Goals
According to Stewart, mechanical engineering students encounter certain bottleneck courses where failure rates are high, and a low number of students succeed in obtaining the expected learning outcomes. He believes that possible reasons for this include a lack of pre-college preparation and motivation, learning style and cultural bias, and family obligations. Other factors that may contribute to this issue are the quality of instruction, class size, and student–instructor interaction.
The Department of Education measures factors that determine the risk of a student dropping out: delayed enrollment in higher education after high school, part-time attendance, independent-student status, having dependents, being a single parent, working full-time while enrolled, and not having a high school diploma. When compared to white students, Hispanic and black students are at risk in more categories. This suggests that innovation in education is needed to improve the outcomes of underrepresented groups in engineering. In addition, for the sophomore-level Statics course, national statistics show a high failure rate, something that also has been observed at UTEP. Stewart notes that in Spring 2014, 46 percent of students in a large-enrollment, traditionally-taught Statics course at the school did not pass (N>160).
In an attempt to address these issues, Stewart has been analyzing student performance in the face-to-face Statics course for mechanical engineering majors. He redesigned his course with the goal of improving student retention and course outcomes. As part of the redesign, which started in Fall 2014, Stewart implemented MasteringEngineering as a way to facilitate flipping the classroom and accomplishing his course goals.
Stewart began teaching Statics in Spring 2014 as a traditional face-to-face lecture course with paper-and-pencil homework. He added MasteringEngineering in Fall 2014, while continuing to teach in a more traditional format but adding assignments that required students to prepare prior to class. In Spring 2015, he moved to a fully flipped format by implementing a project-based, data-driven curriculum using homework scores, time spent on homework, and problem-difficulty data—all of which is found in the diagnostic view in MasteringEngineering—to help guide activities, homework, and class discussions.
Spring 2014 (before the use of MasteringEngineering)
Stewart encouraged students to solve all the problems in the textbook, although only assigned homework problems were handed in for weekly grading, and late homework was not accepted. All problems had to contain a free-body diagram (FBD) and enough detail for the grader to understand the solution.
Fall 2014 (with MasteringEngineering in a traditional format)
Stewart changed the course format by requiring students to watch lecture videos and read the book before attending class. Class began with a clicker quiz on the required video and reading. Several example problems were then solved in class, and students were assigned group problem-solving activities. Success in this type of environment required that students prepare extensively before attending each lecture. Students were told that the course required approximately 10 after-class hours each week.
Students were assigned one required MasteringEngineering homework assignment each week; they were told it would take approximately four or more hours to complete. Late homework was not accepted. Students needed to keep a paper record of the solutions to homework problems. Fifteen percent of the homework grade came from a single homework problem that was collected in class. Homework was to be completed using the Exam Problem Structure. Assignments were not timed, and included tutorials and problems. Questions were randomized and pooled to encourage students to do their own work.
The five lowest in-class clicker quiz grades were dropped. Exams were administered via paper and pencil, and students were allowed to bring a formula sheet to the exam. To receive full credit, solutions had to conform to the following structure:
- Knowns/unknowns. List the given parameters, list the parameters you must find.
- Free-body Diagram (FBD). Neatly draw an FBD that includes arrows with arrowheads, dimensions, and all of the parameters needed to solve the problem.
- When appropriate, include the parameters needed to solve the problem.
- List any assumptions made, as well as the equations needed to solve the problem.
- Give necessary details so calculations can be easily followed. Answers without steps are not accepted.
- Label each equation with a number (1), (2), (3), etc.
- Include units and box the final answers.
- Disorganized, incomplete, and/or copied work is penalized.
Spring 2015 (with Mastering Engineering in a flipped classroom format)
Stewart further redesigned the course by moving to a fully flipped format and changing the use of MasteringEngineering homework. Students were told that the course required approximately seven after-class hours each week. They were expected to learn Statics theory by watching YouTube lecture videos and reading the book before class; class time was spent solving problems from the book and completing homework assignments.
In the prior semester, when students had one weekly MasteringEngineering assignment, Stewart noticed that many waited until the last minute, sometimes within an hour of the time due, and often would not complete the assignment. He believed that some students forgot the concepts from the beginning of each week and struggled with the homework related to those concepts. For Spring 2015, he assigned one homework for every class period in which students met.
Students were told that each assignment would take approximately two hours to complete; they were required to keep a paper record of the solution to be used for self-study. As in the prior semesters, homework was to be completed using the Exam Problem Structure, problems were randomized and pooled, and no late homework was accepted.
Quizzes were completed in MasteringEngineering and were due the night before each class. Stewart created custom, multiple-choice problems using MasteringEngineering’s editing tools. To do well on a quiz, students needed to watch the assigned YouTube video, read the textbook, and understand the theory.
Exams were administered using paper and pencil. Students were allowed to bring the same type of information and use the same problem solution structure as in Fall 2014. Exam content for all semesters included assessment content that was developed by Stewart, from the Pearson problems, and from national tests. Exams were structured in the same way as the homework, using the same problem-solving method taught in class. One-third of the test was drawing FBDs. The test included three challenge problems (worth 90 percent) and several short-answer questions (worth 10 percent).
Results and Data
An analysis was done of the data from the three semesters in the study period to understand the relationship between students’ use of MasteringEngineering, the flipped classroom format, and student learning and course outcomes. Figure 1 shows that after redesign using MasteringEngineering, student success rates were higher.
Each semester, exams were administered differently, although exam content remained consistent. In Spring 2014, the lowest midterm exam (of three) was dropped; for Spring 2015, the final exam was optional. In Fall 2014, all three exams and the final were required and none were dropped. Because of these differences, the study was designed to compare data from exams that were administered in the same way.
The final exam for Spring 2014 without MasteringEngineering and Fall 2014 with MasteringEngineering was required for all students. Figure 2 shows a statistically significant increase in final exam scores for students using MasteringEngineering (M=67%; SD=30%; N=105) compared to students in the traditional class without MasteringEngineering (M=54%; SD=29%; N=109) with p<0.05.
Figure 1. Success Rates, Spring 2014 (n=109); Fall 2014 (n=105); Spring 2015 (n=133)
Average final exam scores without and with use of MasteringEngineering
Figure 2. Average Final Exam Scores, Spring 2014 (n=109); Fall 2014 (n=105); Err Bars = Stand Err, *p<.05
Since MasteringEngineering homework was used in both Fall 2014 and Spring 2015, but the implementation of the homework changed, an analysis was done of the correlation between MasteringEngineering homework scores and the average of the three midterm exam scores for the two semesters. The final exam for Spring 2015 was optional, so final exam scores were not included in this analysis.
In Fall 2014, there was a moderate positive correlation between MasteringEngineering homework and average exam scores where r=0.57 when there was one weekly assignment (Figure 3). In Spring 2015, when MasteringEngineering homework was assigned for each class period and students were engaging with content more frequently, there was an increase in correlation values, resulting in a stronger correlation of r=0.70 (Figure 4).
Correlation between MasteringEngineering and exams spring 2014
Figure 3. Correlation between MasteringEngineering Score and Average Score of Exams 1–3, Fall 2014 (n=105)
Correlation between MasteringEngineering and exams spring 2015
Figure 4. Correlation between MasteringEngineering Score and Average Score of Exams 1–3, Spring 2015 (n=133)
The Student Experience
In Spring 2015, students were surveyed about their use of MasteringEngineering. Their comments included the following:
- “I liked that when reviewing for exams I could go back to the [MasteringEngineering] homework and choose the review section. This made it very easy to review for exams.”
- “MasteringEngineering is user friendly. One can just log in and get to work.”
- “It allows multiple tries in case you mess up and need to revise your calculations to get a better understanding.”
- “The ability to receive feedback after inputting an answer was probably what I liked the most about MasteringEngineering. This would allow me to go back to review the process I was doing.”
- “[MasteringEngineering] explains theory and problem solutions with hints. Also, you could go back anytime and check the homework.”
Stewart believes that technology-based teaching can improve class performance, particularly in large-enrollment courses. He observes that different types of course activities enable students with varied learning backgrounds and styles to more readily understand and apply engineering principles. Moving forward, he plans more research and analysis, including collecting student performance data and using that information for remediation on a regular basis during the semester, and evaluating semester data to adjust course content and activities from semester to semester. His goal is to continue to improve student learning in order to enhance the opportunities for his students to move forward in their programs and successfully complete their educational goals.
Read a success story from one of Professor Stewart’s students: An engineering student aspires to be a Disney Imagineer, published November 2016 on Pearson’s Teaching & Learning Blog