MasteringEngineering Statics educator study investigates student performance in a hybrid format at Tennessee Technological University
- During the period of study, the enrollment increases were primarily due to students having no available ACT score, thus placing them in an unknown risk category as described in the study.
- Instructors found that MasteringEngineering resources allowed them to replicate the recitation experience outside the classroom when it was convenient for students, which also meant additional faculty resources and time were not required.
- Students taking the hybrid course format provided positive feedback on their experience with MasteringEngineering.
- The instructor recommended communicating to students early in the semester about the value and importance of doing the MasteringEngineering homework, taking advantage of its different resources, and using good problem-solving techniques.
Tennessee Technological University, Cookeville, TN
Traditional and hybrid
MasteringEngineering; Engineering Mechanics: Statics by Hibbeler
Elizabeth Hutchins, Adjunct Instructor
Results reported by
Betsy Nixon, Pearson Customer Outcomes Analytics Manager
- Enrollment: 10,314 with 10 percent graduate students
- Type: public institution founded in 1915
- Demographics: 44 percent female; 56 percent male
- Area: rural with 74 percent of students living off campus
- Graduation rate: 53 percent (six-year rate; 2009 cohort)
- Financial aid: almost two-thirds of all students receive some form of financial aid
- History: from 1916 to 1924, the university offered courses only at the high school and junior college levels. By 1929, the State Board of Education had authorized a complete college program.
About the Course
Elizabeth Hutchins is an academic advisor and an adjunct instructor in the College of Engineering at Tennessee Technological University (TTU). She has taught for 20 years, with the last ten years being at TTU. She teaches both Statics and Mechanics of Materials.
In Spring and Fall 2014, Hutchins taught the Statics course, an introduction to the basic principles and applications of rigid bodies in static equilibrium. Statics is a primary foundational course for the Civil and Mechanical Engineering majors and must be successfully completed to continue in the engineering program. The course covers vector algebra, resultants, equilibrium, friction, centroids, moment of inertia, trusses, machines and frames, beam shear, and moments. Upon completion, students should understand the basic principles of mechanics and have developed a reasonable proficiency in applying these principles to solving problems.
The instructional outcomes for Statics are for students to be able to:
- Perform vector mathematics and apply this knowledge to problems involving forces on objects;
- Describe mathematically real-world objects using free-body diagrams;
- Apply the equations of equilibrium to objects (i.e., summation of forces and moments);
- Model determinate objects as a system of applied forces, support reactions, and use the equations of equilibrium to determine the unknown reactions;
- Determine the location of the centroid of an object;
- Determine the moment of inertia of an object about various axes;
- Determine the axial forces in determinate trusses;
- Determine shear force and bending moment at various locations along the length of beams resisting transverse loads; and
- Determine the friction forces on objects and demonstrate whether an object subjected to applied and frictional forces is in equilibrium.
Engineering programs at TTU are accredited by the Accreditation Board for Engineering and Technology (ABET). The ABET Criterion level 3 student outcomes addressed in this course are:
- An ability to apply knowledge of mathematics, science, and engineering
- An ability to identify, formulate, and solve engineering problems
- An ability to use the techniques, skills, and modern engineering tools necessary for engineering practice
Challenges and Goals
Background: The Tennessee Public Agenda for Higher Education focuses on increasing statewide educational attainment by implementing Tennessee’s Complete College Tennessee Act (CCTA) (2010), which emphasizes the importance of the state’s educational system for leveraging economic development. In addition, the state of Tennessee’s Drive to 55 Alliance engages private sector partners, leaders, and nonprofit organizations to support the state’s initiative of 55 percent of Tennesseans equipped with a college/associate degree or certificate by the year 2025. To work towards these goals, the current emphasis is on integration of higher education and cost-effective instructional delivery systems.
In 2013, The Tennessee Board of Regents (TBR) funded an initiative that solicited course revitalization proposals that demonstrate a cost-effective approach to student success in high-enrollment gateway courses. Engineering faculty at TTU participated in that initiative and designed and conducted a study in 2014 implementing a hybrid model for two gateway engineering mechanics courses, Statics and Mechanics of Materials, both required for the Civil and Mechanical Engineering majors.
Issue: TTU faculty who teach these courses generally believe that the most effective way to learn the material is by practicing and understanding how to solve problems. At TTU, the course has historically been taught with a conventional lecture format, along with extensive individual instruction to supplement the lecture. This traditional format tends to be labor intensive for faculty and therefore difficult to scale to larger sections. As part of the 2014 TBR study, TTU developed a hybrid model comprised of a traditional, in-class lecture with online homework and digital resources for learning outside of class. The hypothesis was that a successful implementation of the hybrid model would mean that faculty would be able to administer a more efficient and effective course as follows:
- A more efficient course was defined as the same level of required faculty resources (instruction and grading time) for an increased number of students in the class.
- A more effective course was defined as demonstrating improved student understanding and retention of the material.
For this preliminary study, TTU considered the minimum acceptable course results to be overall performance at the same level as prior years when only the traditional format was offered, so there would be no negative impacts from the change.
Process: TTU had been using the Hibbeler textbooks and adopted MasteringEngineering in Spring 2014 for the online resources for the following reasons:
- The learning aides in MasteringEngineering would facilitate the hybrid delivery format.
- MasteringEngineering textbook problems that had randomly-assigned parameters could help control collaboration.
- Selected Mastering tutorial problems included context-sensitive hints for students who have common errors in their solutions. The hints were developed by experienced subject-matter experts. Because a student could access an annotated solution with more detail than the examples in a typical textbook, this was viewed as a tool to facilitate student learning outside of class and office hours.
- Homework would be automatically graded and provided immediate feedback to the student, which meant that a course could have higher enrollments, but the workload to faculty for grading homework was not increased.
- Faculty believed that MasteringEngineering’s asynchronous student-learning environment could be used with instructor-guided assignments or as an independent study aide, thereby facilitating both types of instruction and resources needed for the hybrid portion of the course.
- Faculty believed that, in many respects, MasteringEngineering could replicate a recitation period, but without the scheduling constraints, and would provide resources whenever it was convenient for students.
The results of this research study were published in 2015 in the paper, An Academic Program Assessment Methodology to Leverage Drive to 55 in the CCTA Integrated Higher Education Environment, by David Huddleston, David Elizandro, Jane Liu, Guillermo Ramirez, and Elizabeth Hutchins, and the results were presented at the American Society for Engineering Education (ASEE) conference in July, 2016. This educator case study will summarize and report on the Statics course findings, and a separate educator case study will feature the Mechanics of Materials results.
The ACT’s annual STEM report for 2016 stated that nearly half (48 percent) of the 2.1 million 2016 American high school graduates who took the ACT test expressed an interest in STEM majors or careers, but only 26 percent of those graduates met or surpassed the ACT College Readiness Benchmark in STEM. In order to better understand student readiness and course performance, the 2014 TTU preliminary study was designed as an initial review of the course data. The long-term goal was to develop an experimental design to collect data to better understand the impact of the hybrid model on student learning. TTU utilized ACT scores to categorize students in groups to facilitate evaluating student performance. The ACT requirements to enroll in the engineering program at TTU were as follows:
- Incoming freshmen from high school must have a high school GPA of 3.0 or a 20 ACT composite with a 22 ACT Math sub-score.
- For freshmen 21 years of age or older, COMPASS exam scores are used in lieu of the ACT requirements.
- International students are required to have a high school diploma, demonstrated language proficiency, and an ACT Math sub-score (usually via the ACT COMPASS) of 19.
- Transfer student requirements are a 2.0 composite GPA, a 2.0 GPA in the last full-time semester, and a C or higher in a pre-calculus mathematics course.
For Spring and Fall 2014, both hybrid and traditional formats were offered in Statics, and data were collected. Students self-selected into either course format. Two hybrid Statics sections were taught in the Spring and Fall of 2014 with enrollments of 81 and 77 students (approximately 44 percent of total enrolled Statics students). Spring 2015 data were not available at the time of analysis and are not included. However, during that period, 77 of 155 (50 percent) enrolled Statics students were taking a hybrid format.
While each instructor developed their own course content, faculty collaborated on content covered, pace, and selection of homework for both courses. Implementation for each format was as follows:
Traditional: Students in the traditional sections received the same in-class lecture as the hybrid sections, but they did not have access to the MasteringEngineering resources, including automatic scoring, hints and feedback, and additional study resources. All homework was done via paper and pencil, completed on engineering paper, and was to follow a GIVEN, REQUIRED, and SOLUTION type format. Diagrams were to be drawn neatly using a straight edge. The homework was handed in for instructor grading.
Hybrid: The hybrid classes were a blend of in-class lecture, electronic instruction, and MasteringEngineering homework and resources that included tutorial aides. Students in the hybrid sections received the same in-class lecture as the traditional sections, which included example problems similar to the MasteringEngineering assigned homework problems. While homework was assigned using MasteringEngineering, solutions for homework had to be completed on engineering paper and follow a GIVEN, REQUIRED, and SOLUTION type format. This provided students the same problem-solving practice as traditional students, but then allowed them to input their answer in Mastering and know immediately if the calculations were correct.
The MasteringEngineering homework utilized both the end-of-section and tutorial problems. The assignments were not timed, and students were allowed multiple attempts since the goal was to practice and develop problem-solving skills. Some assignments were due before lecture, but most were due after in-class lecture. The instructors used macroscopic performance data from the Mastering analytics to determine specific areas that needed extra review and would cover those areas during class time.
- 60% Tests (three)
- 20% Comprehensive final exam
- 10% Homework (MasteringEngineering or paper/pencil)
- 10% Written homework, class work, and quizzes
Results and Data
To evaluate the results of the study and better understand student performance, TTU analyzed course scores based on ACT scores. For the analysis, the student population was categorized in four mutually-exclusive groups based on ACT scores as follows:
- Category 1: Students with ACT scores ≥25. Most engineering schools view these as core students who are adequately prepared to begin engineering degree coursework.
- Category 2: Students with ACT scores between 22–24. These are regional mission specific students who, with mentoring, should be able to complete engineering degree requirements.
- Category 3: Students with ACT scores <22. These are believed to be at-risk students who may have problems mastering a college of engineering curriculum.
- Category 4: Unknown-risk students. These tend to be transfer students who are not required to submit ACT test scores and international students without an ACT score.
Because a portion of the student population repeats a course, data represent the number of students enrolled, including duplicates, for the period of study. Figure 1 presents enrollment numbers for 2009–2014. The increase in the number of students enrolled in 2014 is indicative of an improvement in course efficiency, one of the parameters of the study. The focus of TTU’s analysis was then on course effectiveness. During 2014 and continuing into 2015 (2015 data not available at time of analysis), increases in course enrollment were primarily the result of increases in unknown-risk students (category 4). For the calendar years 2009–2014, the number of students in the unknown-risk category increased from 10 to 30 percent of the total enrollment.
Figure 2 shows the partial results presented in the TTU study. Data in figure 2 represent course grades for the academic years 2013 (all traditional sections) and 2014 (mix of traditional and hybrid sections) by ACT category. Data are available for 2009–2014, but due to space limitations, only two years are presented in this study. All results can be found online in the full version of the paper at the ASEE site.
The results show the following:
Category 1 (ACT ≥25): Data indicate no overall performance changes for student success rates for the Category 1 group with the total A/B/C students remaining essentially the same.
Category 2 (ACT 22–24): The A/B/C rate increased from 2013 to 2014 by approximately 11 percentage points. The results are similar to the 2011 and 2012 data, but further study would need to be done to more fully understand the findings.
Category 3 (ACT <22): Overall Statics grade distributions for category 3 are visibly different from the first two categories and reflect a bias to lower course grades. It is important to note that in both calendar years 2013 (traditional format) and 2014 (traditional and hybrid formats), there were increases in the percent of Cs and D/F/Ws in Statics, so the trend towards lower course scores was increasing for this group before the hybrid model was implemented. However, the student success rate, which includes As, Bs, and Cs, there was an increase for 2014 over 2013. Further study would need to be done to better understand the findings.
Category 4 (unknown-risk): Grade distributions for this category are also visibly different from students in categories 1 and 2. However, the distributions are similar to category 3, at-risk students. The combination of these two categories (3 and 4) appear to have a bimodal distribution for course effectiveness. Although annual percentages vary, approximately 40 percent or more of the students in categories 3 and 4 earned a grade below C.
The analysis conducted by TTU showed that with increased enrollments and after implementing the hybrid course format with MasteringEngineering, students in the first two categories continued to perform at the same level or slightly higher. Since grades for category 3 and 4 students continue to be problematic in the revised courses, effectiveness seems to be independent of the course format. However, the study reports that class effectiveness seems to be dependent on academic profile, and that “…it is important to note that variation in student performance data is also the result of systemic issues in higher education that introduce a major source of noise in course effectiveness measurements and complicates a meaningful experimental design.”1
Since the study was designed as an initial review, further data collection and analysis will continue in an attempt to better understand the impact of moving from a traditional lecture format with paper and pencil homework to a hybrid model with lecture and MasteringEngineering interactive homework and learning resources. TTU faculty have identified some best practices that they recommend to others implementing Mastering into their course:
- Build homework around MasteringEngineering videos to provide students with a visual resource.
- Early in the semester during lecture, emphasize the coaching tools in MasteringEngineering, and encourage students to use the different resources available.
- Continue to encourage students to develop their study skills, note-taking skills, and organizational skills.
- Communicate to students that MasteringEngineering provides resources to help all learning styles.
Statics enrollment by ACT score
Figure 1. Statics Enrollment, Hybrid and Traditional, 2009–2014; 2009 (n=~195); 2010 (n=~240); 2011 (n=~260); 2012 (n=~335); 2013 (n=~305); 2014 (n=~365)
Statics course grade by ACT score category
Figure 2. Statics Course Grade Grouped by ACT Score Category, 2013–2014
The Student Experience
As part of the TBR funded study, the faculty at TTU collected feedback from students in various formats including student surveys and individual student reviews of MasteringEngineering. TTU faculty concluded, “Based on student feedback, the electronic study feature is one of the most effective blended features of PME [MasteringEngineering].”2 One student stated in his written evaluation, “Mastering Engineering is the most technologically advanced tutorial and homework system. It tutors engineering students individually while providing instructors with rich teaching diagnostics.”
In addition, during the Spring 2014 semester, Hutchins’ hybrid Statics students were surveyed and asked what they liked about MasteringEngineering. While the overall response rate was low, some comments from students included the following:
- “I enjoyed its simplicity of use. I also enjoyed the fact that you could rework all of the problems again after they were due for practice. It presented some really great study material for the exams.”
- “It was very user friendly.”
- “Allowing me to complete practice problems, and receiving instant feedback.”
- “It was easy to understand. When I didn’t get a problem the study center part of the website was incredibly helpful.”
- “The study areas and the videos are VERY helpful.”
The analysis of the effectiveness of a hybrid course delivery using MasteringEngineering at TTU indicates that more students enrolled in the course, and there were no apparent adverse effects on student performance with the hybrid format for core and regional mission students (categories 1 and 2). However, the analysis identified critical issues that exceeded the scope of the original project by confirming that almost half of the at-risk and unknown-risk students had difficulty in these courses. As a result of the analysis of the data, it became apparent to faculty that, “improving class effectiveness (student success) and efficiency (leveraging faculty effort) is dependent on the academic profile of students, which is dependent on the higher education environment created by the CCTA, as well as the institutional mission and resources to sustain the mission.”3
Instructors also solicited student feedback from students using MasteringEngineering as part of the study. Student feedback was positive and provided insight into the educational benefit of the resources available for students to use and learn outside of class that is not available with paper and pencil homework. TTU plans to continue to offer both a hybrid and traditional delivery option. Most of the changes to the hybrid course following the study have been minor refinements. Hutchins will continue using Mastering and stated, “MasteringEngineering offers learning tools that serve many students quite well based on their learning style. Other students respond better to more traditional methods. Having both available within a hybrid format seems to provide learning opportunities that can reach a diverse audience effectively.”
1 An Academic Program Assessment Methodology to Leverage Drive to 55 in the CCTA Integrated Higher Education Environment, Huddleston, David; Elizandro, David; Liu, Jane; Ramirez, Guillermo; and Hutchins, Elizabeth (2015)