Mastering-AP-University-Texas-Arlington

EDUCATOR STUDY

Mastering A&P educator study analyzes quizzing and use of optional resources at University of Texas at Arlington

Key Findings

  • Students who had higher than average Mastering quizzes tended to have significantly higher final exam averages than students with at or below average quiz scores.
  • On an end-of-semester student survey, a majority of respondents gave positive feedback about Dynamic Study Modules in Mastering. As a result, the instructor will require this activity next semester.
  • As a new user, the instructor gathered student feedback and evaluated results to understand the student experience and used that information to make informed decisions about Mastering implementation changes for future semesters.

School name
University of Texas at Arlington, Arlington, TX

Course name
Anatomy and Physiology I

Course format
Face to face

Course materials
Modified Mastering A&P for Human Anatomy & Physiology by Maireb & Hoehn

Timeframe
Spring 2018

Educator
Xavier Aranda, Lecturer

Results reported by
Betsy Nixon, Pearson Results Manager

Setting

  • Type: Four-year public research university
  • Enrollment: 29,606 undergraduate; 12,327 graduate (Spring 2017)
  • Ethnicity: 36.5% White; 24.2% Hispanic; 15.8% African American; 9.6% Asian; 10.9% International
  • Setting: Urban

About the Course

Lecturer Xavier Aranda started at the University of Texas at Arlington (UTA) teaching a Forensic Biology Laboratory part-time from 2010 through 2013 while working full-time at the Texas Department of Public Safety Crime Laboratory. He was hired full-time at UTA in 2014 and currently teaches Anatomy and Physiology Lecture and Laboratory, Introduction to Forensic Laboratory, Forensic Biology Lecture and Laboratory, and Genetic Methods Laboratory.

Anatomy and Physiology I (A&P I) is a four-credit lecture and lab, and the first in a two-course sequence which introduces the basic anatomical terms of the human body, reviews cellular and molecular concepts to be applied at the systematic level, and covers the structure and function of the integumentary, skeletal, muscular, and nervous systems. It is primarily taken by students in the fields of sport activities, medical technology, and pre-nursing. Students who wish to apply to the nursing program must complete A&P I and II with a B or higher to be competitive for admission.

Challenges and Goals

The A&P courses are important for students since they must not only pass them, but must do well to continue in subsequent courses and their desired program of study. Aranda finds that students often struggle with knowing how to study, learning and applying course concepts, and managing their time. He feels they often have trouble staying on track during the semester, which can impact their performance in the course.

His course structure is designed with a consistent schedule throughout the semester; students see concepts multiple times before an exam and have multiple resources available to help them practice and learn outside of class. Aranda acknowledges that students may have a tendency to work together or google answers outside of class, but he has implemented the homework keeping this in mind.

Modified Mastering™ A&P (MA&P) was piloted in his A&P I course in Summer 2017 and adopted for the Fall 2017 semester. During Spring 2018, Aranda surveyed students and evaluated his course results to gain insight into the impact on learning, discern changes in critical thinking, and help guide implementation changes to enhance learning and course success.

Implementation

A&P I had the following components in Spring 2018:

Exams: Three paper-and-pencil exams comprised of 100 multiple-choice questions from the Pearson test bank were administered in class.

Final exam: A two-part paper-and-pencil final exam comprised of 100 multiple-choice questions from the Pearson test bank and 70 questions written by the instructor or from other sources was administered in class. Aranda feels that using questions from different sources can help prevent cheating on the exam.

MA&P: Fifteen chapter quizzes and two module quizzes were completed outside of class time in MA&P.

  • Chapter quizzes contained 10 questions selected primarily from the reading questions with a few test bank questions. After Aranda lectured on a topic, students had one week to complete the corresponding quiz. On an end-of-semester survey, 87% of respondents said they tended to do the Mastering quizzes by themselves, but since students sometimes work together, quiz questions were randomized so answers can’t be easily shared.
  • Module quizzes covered approximately five chapters each and contained mainly questions from the test bank. The first module quiz contained 15 randomized questions, and the second contained 25 randomized questions. Students had one week to complete the quizzes.

Research has shown quizzing to be an effective way to improve learning. A study conducted by Rebecca Orr and Shellene Foster from Collin College entitled Increasing Student Success Using Online Quizzing in Introductory (Majors) Biology and published in CBE-Life Sciences Education, found that “Analysis of exam grades earned by those taking 100% of pre-exam quizzes indicates that not only does this group have a significantly higher exam average than the group of students who took 0% of the pre-exam quizzes, but they also have a significantly higher exam average than the class average. Through detailed, statistical analysis, the benefit of quizzing is demonstrated to be significant for students of diverse academic abilities. Pre-exam quizzing using an online homework platform is an effective way to increase student performance on exams and allows class time to be utilized for teaching activities.”1

An end-of-semester survey was administered to Aranda’s A&P I Spring 2018 students (90% response rate). Table 1 shows the responses to two questions in which students were asked to select strongly agree, agree, disagree, or strongly disagree in response to a statement. The responses indicate that the MA&P quizzes helped students, and they found them to be a valuable activity in the course.

Survey statement Percent answering strongly agree or agree
The Mastering quizzes helped me do better on exams. 94%
Doing the Mastering quizzes helped me understand what I know and what I need to study more. 92%

Table 1. Student Survey Results, Spring 2018 (n=121)

Since Aranda had been using MA&P for less than a year, he wanted to become more familiar with the different resources available and gather feedback from students to find out which resources they utilized and what helped them the most. During the semester, Aranda informed students about the Dynamic Study Modules (DSMs) and the Mastering study area to encourage them to use those for practice and learning, but he did not require the DSMs as homework. On the end-of-semester survey, 71% of students reported using the DSMs, and 43% reported using the Mastering study area, while 17% reported only doing the assigned Mastering quizzes. Based on these results, Aranda set up some DSM assignments early in the Summer 2018 session, and students requested that he continue to provide those. Because of the positive student feedback, he will add required DSM assignments beginning Fall 2018.

Assessments

  • 35% Lab
  • 30% Lecture exams (3)
  • 18% Mastering quizzes and other assignments
  • 17% Comprehensive final exam

Results and Data

To better understand the impact of using Mastering in A&P I, Aranda evaluated his course results from Spring 2018. The MA&P quizzes were designed to evaluate student understanding in preparation for the high-stakes exams. As stated previously, research has shown that quizzing can be beneficial to learning. Robert Bjork’s research asserts that retrieval of stored information acts as a memory modifier, and that using tests as learning events creates “desirable difficulties that enhance learning.”2 For Aranda’s Spring 2018 students, the average score for the MA&P quizzes was 90%. An analysis was done grouping students based on their quiz average which showed that the group of students who scored above average on the quizzes had a significantly higher final exam average (p<.05) than the group of students who scored at or below the MA&P quiz average (figure 1).

Final exam average based on Modified Mastering A&P quiz average

Figure 1. MA&P Quiz Average >90% (n=86); MA&P Quiz Average ≤90% (n=49), Spring 2018

To gain additional insight into student progress, an analysis was done grouping students based on their performance on the first semester exam in conjunction with their MA&P quiz performance. The analysis first grouped students based on the exam 1 average of 77%. Students were grouped above the exam 1 mean (High Exam 1) or at or below the mean (Low Exam 1). Those two groups were each then divided based on the MA&P quiz average of 90%. The four groups in the analysis are as follows:

  • HE1/HHW = High Exam 1 (>77%) / High MA&P Quiz Homework (>90%)
  • HE1/LHW = High Exam 1 (>77%) / Low MA&P Quiz Homework (≤90%)
  • LE1/HHW = Low Exam 1 (≤77%) / High MA&P Quiz Homework (>90%)
  • LE1/LHW = Low Exam 1 (≤77%) / Low MA&P Quiz Homework (≤90%)

The findings of the analysis are as follows with significance determined at p<.05 (figure 2):

  • When compared to each other, the High Exam 1 (HE1) groups were two percentage points apart on exam 1, and it was not significant. Their final exam average was five percentage points apart, with the group of students who had a higher MA&P quiz average (blue line) having a significantly higher final exam average than the group of students with a lower MA&P quiz average (yellow line).
  • When compared to each other, the Low Exam 1 (LE1) groups had the same score on exam 1. They were seven percentage points apart on the final exam, with the group of students who had a higher MA&P quiz average (red line) having a significantly higher final exam average than the group of students with a lower MA&P quiz average (teal line).
  • The HE1/LHW (yellow line) and the LE1/HHW (red line) groups were 15 percentage points apart on exam 1. They were eight percentage points apart on the final exam. While the results on exam 1 were significant and remained that way on the final exam, the gap was reduced.
  • The LE1 students who scored above average on the MA&P quizzes (red line) was the only group that had a higher final exam average than their respective exam 1 average. The other three groups had a final exam average one to four points lower than their first exam average.

Individual exam score averages based on exam 1 performance and Modified Mastering A&P quiz homework average

Figure 2. HE1/HHW (n=50); HE1/LHW (n=15); LE1/HHW (n=36); LE1/LHW (n=33)

Overall, the results show a trend for students who did well on the MA&P quizzes to perform better on the final exam than the equivalent students who had lower quiz scores.

The Student Experience

On the end-of-semester survey with 121 out of 134 students responding, 91% of students agreed with the statement, “I would recommend Mastering to another student as a good resource for the course.” Students were also asked, “What did you like most about Mastering?” Individual answers indicate that students found the Mastering study resources and Dynamic Study Modules helpful to them, even though these were not required assignments. Responses included:

About MA&P:

  • “I like how it helps me practice the material and shows me what I need to study more.”
  • “I like Mastering because it helped me realize what I needed to study and tested me to see if I really understood what I had learned.”
  • “I liked that I was able to access it online, especially the textbook so that I didn’t have to carry a book around.”

About Dynamic Study Modules:

  • “I found the Dynamic Study Modules to be the most useful resource for studying.”
  • “It [Mastering] was very informative and geared towards exactly what I needed to know and learn. I really enjoyed the Dynamic Study Modules.”
  • “I liked the study area and the interactive pictures and models. The Dynamic Study Modules helped me study by giving me explanations on the questions I get wrong.”

About study resources:

  • “The study area was probably my favorite because it has multiple quizzes and labeling diagrams.”
  • “It has plenty of diagrams, practice quizzes, and flashcards that help you study for the exam.”
  • “I liked all the features that it offered, and I had plenty of resources to use for studying.”

Conclusion

Aranda piloted Modified Mastering A&P in Summer 2017. He first implemented MA&P quizzes to help students prepare for the exams. The course results show that the groups of students who did better on the MA&P quizzes did better on the final exam than the comparable group who had lower MA&P scores. After receiving student feedback, he found that the majority of students reported using the optional Dynamic Study Modules to help them practice and learn, and many students also reported utilizing the additional Mastering study resources. In addition, 91% of the respondents said they would recommend Mastering as a good resource for the course.

Prior to Summer 2018, Dynamic Study Modules were available to students, but not assigned as required homework. Based on student feedback, Aranda started to assign DSMs as required homework in Summer 2018, and nearly all of the students in the course requested he continue posting them for homework. He plans to continue using both the DSMs for homework to help students practice and learn, and the MA&P quizzes to help them assess their understanding prior to exams. He is also considering adding Learning Catalytics to his course in the future and will look at ways to incorporate it. In addition, the department will begin to use Mastering A&P for online labs starting Fall 2018.

Best practices

When Aranda was asked what recommendations he would have for other instructors starting out with MA&P, he shared the following best practices:

  • Explore the program and resources before using it for class.
  • Review quizzes in the “student view” before assigning. A Pearson best practice is to create a separate student account and review the entire course as a student to understand what your students will see.
  • If assigning quizzes for completion outside class, use the reading chapter quiz questions and sparingly use the test bank questions to minimize cheating opportunities.
  • Use the test bank questions for the printed exam, but include some questions from a different test bank or instructor-written questions.
  • Explain to the students that there will be a delay in posting grades to Blackboard from Modified Mastering. Use the Mastering guides when starting the course. The Mastering quick start guide, implementation guide, and other training resources for instructors are available online and are designed to help instructors get started.

 

________________________________________________

1Rebecca Orr and Shellene Foster, “Increasing Student Success Using Online Quizzing in Introductory (Majors) Biology,” CBE—Life Sciences Education, Vol. 12, No. 3 https://www.lifescied.org/doi/full/10.1187/cbe.12-10-0183

2Ibid