Mastering Engineering educator study examines impact of pre-lecture assignments on exam scores at Vanderbilt University
- Average exam scores increased after Mastering Engineering pre-lecture tutorial homework assignments were added to the curriculum.
Vanderbilt University, Nashville, TN
Flipped, face to face
Mastering Engineering with Engineering Mechanics: Statics by R. C. Hibbeler
Fall 2013–Fall 2014
Lori Troxel, Associate Professor
Results reported by
Betsy Nixon, Pearson Customer Outcomes Analytics Manager
Vanderbilt University is a private research university serving approximately 6,500 undergraduates and 5,300 graduate and professional students. The majority of students attend full time, and approximately 65 percent receive some type of financial aid.1 The School of Engineering was started in 1886 and enrolls approximately 1,300 students. Bachelor of engineering degree programs are offered in biomedical engineering, chemical engineering, civil engineering, computer engineering, electrical engineering, and mechanical engineering and are accredited by the Engineering Accreditation Commission of the Accreditation Board for Engineering and Technology (ABET).2
About the Course
Lori Troxel is an associate professor of the practice of civil and environmental engineering with teaching responsibilities in the area of structural engineering and sustainable infrastructure. She has completed the ExCEEd3 teaching course and implemented many innovative teaching strategies. Troxel has taught Statics for 10 years.
Statics is a three-credit, introductory course required in the civil engineering curriculum. It is taught only in a face-to-face format. The course presents civil engineering students with the basics of engineering mechanics, including applications to systems of forces in two and three dimensions (particles and rigid bodies), resultants, equivalent systems, and equilibrium, vector notation, introduction to shear and moment diagrams, moments of inertia, friction, and three-dimensional representation. Calculus II is a corequisite for the course. The primary learning objective of the course is to develop problem-solving skills as applied to engineering mechanics problems. Additional outcomes are related to the following ABET program outcomes:
- Graduates will demonstrate an ability to apply knowledge of mathematics, science, and engineering.
- Graduates will demonstrate an ability to identify, formulate, and solve engineering problems.4
Challenges and Goals
Because problem solving is a key skill that engineering students need to succeed in both the course and the program, it is critical that students can independently work problems. Solutions manuals are readily accessible, and when used in the correct way can be beneficial to the learning process. However, Troxel believes it’s not unusual for students to attempt to find solutions for homework in lieu of working the problems themselves.
Troxel sought a way to administer homework assignments that would minimize the use of solutions manuals for copying homework answers and not doing the work. She also was interested in moving toward a more active class with the goal of flipping the classroom. She observed that students didn’t seem prepared to work problems or discuss content in class when they were assigned reading from the textbook. When students weren’t prepared with a basic understanding of concepts, the type of activities or problem solving that could be done during class time was affected. Troxel implemented Mastering™ Engineering about five years prior to the study to address these issues.
In the first year, she required that all homework be due online; nothing was due by paper and pencil. Troxel had concerns that many students were getting the answers without fully working the solutions. Troxel stopped using Mastering at the year’s end and went back to all paper-and-pencil homework. A year later, Troxel realized she was still facing the original issues. She re-implemented Mastering, but this time required online homework with paper-and-pencil solutions. She believed the class benefited from the Mastering homework features, including tutorials, hints, and feedback that facilitated learning, and with written solutions, she could ensure that students were completing the correct steps to arrive at the answers.
Mastering also enabled her to move toward flipping the classroom. By using the tutorials to help students learn those concepts outside of class, time previously spent covering basic concepts in lecture could then be used for in-class problem-solving activities. Given these challenges and goals, Troxel engaged in this study to begin to test and measure the relationship between engagement in pre-lecture activities with online materials and exam performance. To begin to measure the ways Troxel’s students engaged in this type of pre-lecture activity, she collected data related to Mastering assignments that she believed would be aligned to the learning outcomes of the course.
Troxel’s goal for homework was for students to learn how to set up and solve a problem, an essential skill for this course. She taught them the following steps to the process:
- Think about what is being asked.
- Understand what is known.
- Determine what can be found out.
Up to and including Fall 2013, Mastering homework was done after lecture with some required paper-and-pencil problems. The course was redesigned for Fall 2014, moving toward a flipped classroom with Mastering homework assigned both pre- and post-lecture.
Following are the Fall 2014 course components:
Mastering tutorial and coaching problems. Pre-lecture assignments designed to familiarize students with basic concepts before a topic was covered in class. No deduction was made for use of hints, and no points were awarded for not using hints. Since the goal was for preparation and learning, not assessment, assignments were for either zero points (practice) or a small number of points.
Mastering problem-solving homework. Post-lecture, end-of-chapter problems that were usually randomized. Written solutions were required to be done on engineering paper and turned in following the format provided by the instructor. The format for written problems had to include a problem statement, a sketch, the given problem information, a goal, and the solution steps. Homework was not timed, students were allowed multiple attempts, and the two lowest scores were dropped. Late homework was not accepted. Default Mastering settings were left in place for scoring. This process was in place for both semesters in the study.
Class participation. Similar to her course before redesign, students were expected to actively participate in problem-solving activities during class. The participation grade was based both on the instructor’s observations and on answers to specific problems.
Projects. Consistent with her course design before Fall 2014, projects were open-ended problems with real-world examples and applications. Students were required to use engineering paper and the problem-solving algorithm provided by the instructor.
Notebooks. Students were required to use notebooks to take notes in class, collect graded project reports, and take tests. Notebooks were collected at the end of the semester.
Exams. Three paper-and-pencil tests and one comprehensive final exam were administered. No makeup tests were allowed. Students had to pass the final exam in order to pass the class. If a student scored less than 60 percent on the final exam, then the final exam grade was used as the grade for the course. Exams comprise 10 percent short-answer questions and 90 percent problems.
Extra credit. Students could earn five bonus points for each Engineering Society general meeting attended (not officer meetings). The extra-credit points were added to the total points for Mastering and the written solutions.
- 30% Exams (three)
- 20% Mastering Engineering homework and written solutions (two lowest dropped)
- 20% Projects
- 15% Final comprehensive exam
- 10% Class participation
- 5% Notebook
Results and Data
An analysis of results from Fall 2013 and Fall 2014 compared exam scores. The same number of exams was given, and although exam content was not identical, Troxel maintains that the level of question difficulty was comparable. In Fall 2014, Mastering pre-lecture tutorial homework was added after exam 1. Students in Fall 2014 had pre-lecture assignments for the chapters covering exams 2 and 3 and the final. Only post-lecture homework was assigned for the first unit, which was also the case in Fall 2013.
Figure 1 shows a comparison of exam scores by semester. For Fall 2013 and Fall 2014, the average score was the same on exam 1 when only Mastering post-lecture homework was given. After MasteringEngineering pre-lecture homework was added, results show that the average exam scores for Fall 2014 were higher than the comparable exam scores for Fall 2013 without pre-lecture tutorial homework.
Average exam scores
Figure 1. Average Exam Scores, Fall 2013 (n=50) and Fall 2014 (n=88); Err Bars = Stand Err; *p<.05 (Note: In Fall 2014, pre-lecture assignments were added after exam 1.)
For exam 2, students in Fall 2014 (M = 84%; SD = 10%; N = 88) had higher scores than students in Fall 2013 (M = 81%; SD = 15%; N = 50), but it was not statistically significantly higher, with p=0.11, with a one tailed t-test assuming unequal variance.
In Fall 2014, scores for both exam 3 (M = 90%; SD = 8%; N = 88) and the final exam (M = 89%; SD = 7%; N = 88) were statistically significantly higher than scores on both the Fall 2013 exam 3 (M = 86%; SD = 13%; N = 50) and the final exam (M = 85%; SD = 14%; N = 50), with p<0.05 with the one-tailed t-test assuming unequal variance.
The study’s findings do not account for the unmeasured influence of variables that can impact student performance, such as motivation and study skills. However, based on the course performance of Troxel’s students, those who took the redesigned course and were assigned Mastering pre-lecture homework had higher exam averages than students who took the course before the redesign and were assigned only post-lecture homework. Further research is needed to test what the initial data seems to suggest is a relationship between assigning Mastering pre-lecture assignments and exam performance.
The Student Experience
A 2011 National Study of Student Engagement survey found that engineering students tended to study, on average, five hours more than their counterparts studying social science or business. This disparity in study habits might not reflect a more demanding workload, but rather a difference in the type of studying required.5 Because students often lack problem-solving skills or need practice to reinforce and develop those skills for the type of work done in engineering, problem-solving homework is a key component in Statics. Troxel found that after the course change to using both pre- and post-lecture Mastering assignments, students came to class better prepared to do problem-solving activities. She also observed that students asked questions, which showed a better understanding of basic concepts and a deeper level of thinking, and that they seemed to be more engaged and get more out of class discussion.
Between 2010 and 2014, every engineering occupation added jobs—a statistic that indicates the demand for quality graduates in the engineering field.6 In order to enter the workforce, students must first succeed in the introductory Statics course, which requires developing problem-solving skills, gaining an understanding of the concepts, and practicing problems. Students in this study performed better on exams, which were 90 percent problems, after Mastering pre-lecture homework was implemented in the redesigned course, thereby affording them more time in class for active problem solving. This change enabled Troxel to use class time to better address misconceptions, answer specific questions, and focus on enhancing students’ problem-solving abilities. Students completed additional practice on post-lecture homework, giving them multiple opportunities to develop the skills needed to succeed in the course and move forward in their programs.
Based on the course performance of Troxel’s students, those who took the redesigned course and were assigned MasteringEngineering pre-lecture homework had higher exam averages than students who took the course before the redesign and were assigned only post-lecture homework.