MasteringOceanography educator study evaluates addition of required online homework at Old Dominion University
- Despite large enrollments, the instructor was able to add required homework because of MasteringOceanography’s automatic grading and feedback, which allowed her to monitor student performance on a regular basis and help students evaluate their own understanding.
- Data indicate that the group of students who attempted a higher number of MasteringOceanography homework assignments had higher test averages than students who attempted less homework.
- Data indicate that the group of students who had MasteringOceanography average scores higher than the class mean had higher test averages than those who had Mastering scores below the mean.
- The instructor recommends continually evaluating the amount and type of Mastering homework assigned and making adjustments as needed to provide beneficial resources and enhance learning.
Old Dominion University, Norfolk, VA
Face to face
MasteringOceanography; Essentials of Oceanography by Trujillo and Thurman
Shannon Wells, Lecturer
Results reported by
Betsy Nixon and Traci Simons, Pearson Customer Outcomes Analytics Managers
- Location: Metropolitan
- Type: Research institution
- Enrollment: 24,672 Total; 20,101 undergraduate and 4,571 graduate
- Financial aid: 83 percent of first-time, full-time students received some sort of financial aid in 2013–2014; 34 percent of first-time, full-time students received Pell grants
- Graduation Rate: 25 percent (4 years) for the 2006 cohort of first-time, full-time degree/certificate-seeking undergraduates; 49 percent (6 years); and 54 percent (8 years)
- Ethnicity: 46 percent Caucasian; 27 percent African American; and 7 percent Hispanic/Latino
- Retention rate: 82 percent of Fall 2014 freshmen also enrolled in Fall 2015
About the Course
Shannon Wells has been a faculty member in the Ocean, Earth and Atmospheric Science department at Old Dominion University (ODU) since 2011. She teaches four sections of Introductory Oceanography yearly with approximately 1,000 students, and the sections are always filled. In addition, she teaches Environmental Geology, and Earth Science and Historical Geology.
The Introductory Oceanography course is a four-credit course with three lecture hours and two lab hours. It is an introductory course emphasizing the geology, chemistry, physics and biology of the ocean basins. The lab component emphasizes the practice of basic scientific methods. Knowledge of the metric system, scientific notation, ratio and proportion, and graphing is required. Students also take a required field trip as part of the course.
There are 23 lab sections each semester, and all are taught by TAs. Students who take this course are non-science majors working to fulfill a natural science requirement. Enrollment is approximately 500 in the Fall and 500 in the Spring with two sections each semester. Due to the timing of registration, one section tends to consist primarily of freshmen, while the other section is populated with primarily upper class students.
Challenges and Goals
Because of the large enrollment, Wells was not able to assign regular graded homework. With her lecturing responsibilities, individually graded homework was not manageable, and there was no TA to assist with the grading task.
Wells’ sections were typically split: primarily freshmen in one and upperclassmen in the other. She felt that the performance for the section dominated by freshmen tended to fall off as the semester progressed. With no required homework assigned, there was no way to monitor progress throughout the semester. For students, no formal homework meant that they typically didn’t know where they were struggling or what topics they needed to study more until after their first test.
When Wells adopted MasteringOceanography in 2013, her goal was to:
- give students more immediate feedback;
- use homework assignments to reinforce what was presented in the class; and
- better monitor student understanding and progress during the semester.
Spring 2016 course work for Introductory Oceanography consisted of four tests, four MasteringOceanography homework units, and a laboratory grade.
Tests: Four tests were administered with each test covering the material from lecture since the prior test, and the same tests were administered in all sections. The final was cumulative. Tests were administered via Blackboard, and students had to have the Respondus Lockdown Browser installed on their computer in order to take the tests. Each test was available during a specific date range, was timed, and had to be completed within a 90-minute period.
MasteringOceanography (MO): Students were expected to spend on average 2–3 hours per week reading and working in MO. There were 19 MO homework assignments which were grouped by unit topic into four units. Each assignment opened after the lecture on the applicable topic with the goal of reinforcing what students were hearing in class and reading in the textbook. MO homework assignments consisted of reading quiz questions, concept checks for quick multiple-choice questions, and give-it-some-thought and visualizing questions from the Mastering items. MO assignments were not timed, and students had one attempt. They received penalties for using available hints and credit for not using hints.
Attendance: Students were encouraged to attend all classes and received an attendance bonus at the end of the semester if they attended a majority of class sessions. Attendance for lab was mandatory. Students who missed three or more labs failed the course.
Wells reports that her MO implementation has taken a lot of tweaking and says, “When I first started using it, I got excited about all the question options and accidentally overloaded the homework, so instead of reinforcing lecture, it became a burden to students.” To rectify this, Wells has been more discriminatory about the number and types of questions she assigns for homework. In addition, she no longer assigns the extra resources that she had included before, such as some of the videos. She said that while they were interesting for students, she was concerned that students felt overwhelmed. The study resources in MO are still available to students who choose to explore the additional study material on their own.
Wells appreciates that she can use the Mastering diagnostics to track her students’ performance. She tends to look at time spent and homework scores to see how students are doing, often looking at hint usage. “If a large majority of my students are losing points because of hints, then it’s a red flag for me that they may need more help with that topic.” She can then review it in upcoming classes.
In general, Wells reviews the MO diagnostics after the first test and also performs a question analysis on the test questions, which are generally questions she has written. She looks at the average time on Mastering assignments to see if it is in line with the national MO statistics. However, while Wells sees the benefit of comparing her class’ performance to national averages, her main concern is whether her students are understanding the concepts she’s teaching.
- 45% Tests (three)
- 25% Laboratory
- 15% Final test
- 15% MasteringOceanography homework
Results and Data
In an effort to better understand performance in each of the Spring 2016 sections, an initial analysis compared section 1, primarily populated with upperclassmen, to section 2, dominated by freshmen. The results showed that the sections had the same test 1 average score. However, at test 2, section 1 had a higher average, and that continued through test 3 and the final. Wells believes that the performance in the section predominantly populated by freshmen tends to fall off during the semester, and the results for this semester confirmed that. Because of the difference in test scores, the following analyses were first done on each section separately, and then on the combined semester data. The results showed the same outcome for each section, along with the combined section data. Due to space limitations, the results for the combined data for both sections are presented in this study.
Participation and performance on the MO homework were evaluated. The first analysis was to group students based on the number of Mastering assignments attempted, with the second analysis grouping students by their Mastering homework average score. The mean number of Mastering assignments skipped, considered to be an assignment with a score of zero, was two; the average Mastering homework score was 83 percent. These parameters were used to group students and compare test averages.
Figure 1 presents mean test scores for two groups: (1) students who attempted 90 percent or more of the assigned homework (i.e. skipped less than the mean of two skipped Mastering assignments); and (2) students who skipped two or more Mastering assignments, meaning they attempted less than 90 percent of the homework. Those who did more Mastering homework had significantly higher test averages than students who attempted less assignments, and the difference was significant with p<.05.
The second analysis grouped students based on their Mastering homework averages. Figure 2 compares (1) the group of students with a Mastering average score above the MO average of 83 percent; and (2) the group of students whose MO average was at or below 83 percent. Students who scored higher than the mean had significantly higher test averages with p<.05.
Test average based on MasteringOceanography participation – combined sections
Figure 1. Students Who Attempted All/Skipped <2 Mastering Assignments (n=329); Students Who Skipped ≥2 Mastering Assignments (n=153); Err bars=Stan Err, p<.05
Test average based on MasteringOceanography performance – combined sections
Figure 2. MasteringOceanography Score >83% (n=302); MasteringOceanography Score ≤83% (n=180); Err bars=Stan Err, p<.05
To evaluate the progression of students during the semester, an additional analysis was done grouping students using their test 1 score as a baseline. The method of analysis was selected with the intent to explore the impact of homework completion on assessment performance as the course progressed.
The median score for test 1 was 72 percent. Students were first grouped either (1) below the median of 72 percent for test 1, or (2) at or above the median. These groups were identified as High Test 1 (HT1) or Low Test 1 (LT1). The High Test 1 and the Low Test 1 groups were further divided based on the number of Mastering assignments attempted during the semester, either (1) attempted all or skipped one Mastering assignment, identified as High Homework (HHW); or (2) skipped two or more Mastering assignments, identified as Low Homework (LHW).
The four groups in the analysis were as follows:
- HT1/HHW – High Test 1 (above median test 1) and High Homework (attempted all or skipped less than two assignments)
- HT1/LHW – High Test 1 (above median test 1) and Low Homework (skipped two or more assignments)
- LT1/HHW – Low Test 1 (at or below median test 1) and High Homework (attempted all or skipped less than two assignments)
- LT1/LHW – Low Test 1 (at or below median test 1) and Low Homework (skipped two or more assignments)
Figure 3 shows the following results:
- The HT1 (HT1/HHW and HT1/LHW) groups had equivalent test 1 scores.
- The two HT1 groups diverged at test 2, and the HT1/HHW group earned a six percentage point higher final test average than the HT1/LHW group. The difference was statistically significant (p<.01).
- The HT1 groups had a 17 percentage point higher test 1 average than the LT1/HHW group. However, the gap between the HT1/LHW and the LT1/HHW groups closed at test 2, and the average scores for those two groups were statistically equivalent at tests 2 and 3, ending with the same final test average score.
- The LT1 groups had statistically equivalent test 1 scores (p=.06).
- The LT1/HHW group final test average was five percentage points higher than the LT1/LHW, and the difference was statistically significant (p<.05).
Test average based on first test and MasteringOceanography participation
Figure 3. HT1/HHW (n=167); HT1/LHW (n=59); LT1/HHW (n=162); LT1/LHW (n=94)
- Students in the two High Test 1 groups started out with the same test 1 score, but the group that did more Mastering homework earned higher subsequent test averages.
- Students in the Low Test 1 group that did more Mastering homework started out much lower than the High Test 1 groups, but the gap closed with the High Test 1 group that did less Mastering homework for the subsequent tests.
- Students in the Low Test 1 group that did more Mastering homework started out equivalent with the Low Test 1 group that did less Mastering homework, but they ended with a significantly higher final test average than that group.
Overall, the groups of students who did more Mastering homework tended to do better in the course than the equivalent group of students who did less Mastering homework.
The Student Experience
While students are able to provide feedback about MasteringOceanography on course evaluations, Wells reports that many do not participate because the evaluations are optional. However, two comments from evaluations that stood out to Wells and helped her understand how students use Mastering and what features they like include:
- “The online homework helped me the most. The videos, animations, graphs, and diagrams succinctly summarized key concepts from the text and lectures.”
- “The online homework really helped me, personally. It was a good mix of questions that required you to read/know the more important parts of the chapters and questions that required you to apply the concepts learned in lab and class discussion by using critical thinking skills.”
Wells feels that incorporating MasteringOceanography into her course has helped her accomplish the goals she set at adoption: students receive immediate feedback through MO assignments; student evaluations provide examples of how students view the MO homework, and Wells is able to monitor how her students are doing before the first test by using the Mastering diagnostics. In addition, Wells plans to share the homework completion data with her students to show the importance of homework completion to course success.
Wells says that MasteringOceanography has helped her fine-tune what she asks of students and what she presents in class to enhance learning. Upon first implementing MasteringOceanography, she says she quickly learned how over-zealous she was, but has adjusted her expectations. She hears good feedback, and states that, “MasteringOceanography has helped me to better understand student learning.” For instructors looking to implement MasteringOceanography, Wells advises, “Instructors need to decide what is presented to students, because if it’s too much and kids can’t handle it, it won’t help. Don’t be afraid to recognize that, and tweak Mastering to fit your students’ needs and capabilities.”