Revel educator study measures homework, exam, and final course scores at North Carolina State University

EDUCATOR STUDY

Revel educator study measures homework, exam, and final course scores at North Carolina State University

Key Findings

  • Students earning higher Revel scores also earned substantially higher average homework, average exam, and final course scores.
  • Data show that average homework, exam, and final course scores improved after implementation of Revel.
  • A strong majority of responding students on an end-of-semester survey (94 percent) agreed Revel’s “read a little, do a little” approach helped them learn and remember chapter content better than traditional print textbooks.

School name
North Carolina State University, Raleigh, NC

Course name
Introduction to Agricultural & Resource Economics

Course format
Face to face

Course materials
Revel with InMicroeconomics by Hubbard and O’Brien

Timeframe
Spring 2011Fall 2016

Educator
Dr. Jonathan Phillips, Lecturer

Results reported by
Candace Cooney, Pearson Customer Outcomes Analytics Manager

Setting

  • Locale: large, urban, four-year public research university, one of the three corners of the Research Triangle
  • Enrollment: over 34,000 students
  • Student-faculty ratio: 14:1
  • Five-year graduation rate: 71 percent (2015)
  • Classes with fewer than 20 students: 35 percent
  • Gender: 55 percent male
  • Age: 39 percent of students are 23 or older
  • Diversity: 27 percent total diversity
  • Financial aid: 47 percent of students receive need based aid

About the Course

Jonathan Phillips has been teaching for approximately 17 years at NCSU and has taught Introduction to Agricultural and Resource Economics for 16 years. This one-semester, three-credit course enrolls about 150 students each fall, at least half of whom are Agricultural Business majors; students in the College of Management do not take this course.

General course learning goals include:

  • understanding the basic concepts of demand, supply and elasticity;
  • learning about marginal concepts and applying them to everyday activities;
  • understanding the impact of policy when the news discusses interest rates; and
  • becoming economically educated to form opinions on government policies.

Challenges and Goals

Prior to Fall 2016, Phillips had not been using any digital component in his Economics course; his traditional approach to teaching included paper homework and short-answer exams. He had developed his own homework questions which mirrored his lectures, and he didn’t want to settle for choosing questions in the online homework program that might be similar to his lecture content but not exactly what he wanted. However, he remained on the lookout for a digital system that could potentially provide his students with an enhanced experience. When presented with the opportunity to pilot Revel, he was intrigued for two reasons: Revel was a textbook replacement, and, the reading, media and assignments were integrated. This integration of dynamic media with reading and assessment was key to his decision to pilot the program. Students could read about a new concept, view a simulated graph explaining the concept, watch a brief video clip providing a real-world example, while completing assignments and assessments interspersed frequently for immediate feedback. The goal was better content understanding and retention, metrics that Phillips gauges by student interactions during office hours. Phillips chose to pilot Revel in Fall 2016 with his two sections.

Implementation

Revel is required; the program is used primarily by students working at home on a personal computer. Students use Revel for reading and understanding chapter content, reviewing concept simulations and videos, and completing homework assignments. Phillip’s goal is for Revel to completely replace the printed textbook while introducing new concepts, providing homework and practice opportunities, helping students assess their own understanding of the course material and tracking their progress. As the course instructor, his role is to assign content and homework in Revel, and provide support and remote monitoring to students using the program at home.

Phillips anticipates that students will spend at least 1–2 hours per week working in Revel, including time spent reading, viewing simulations and videos, and completing homework assignments. Phillip’s students confirmed this on a voluntary, end-of-semester Fall 2016 survey (37 percent response rate)64 percent of students said they spent more than two hours per week working on activities in Revel.

Students had access to the following Revel options for each chapter:

  • Reading: students can read pre-built chapter content with embedded, gradable, multiple-choice or draw-graph exercises/concept checks called Try It. Feedback and learning aids are available once completed.
  • Practice Set: students may choose to complete practice exercises tied to select chapter-opening learning objectives. These interactive worked-out problems show students how to solve economic problems by breaking them down step-by-step, promoting mastery of the content.
  • Homework: students can answer approximately 35 multiple choice and media questions covering most chapter topics.
  • Chapter quiz: quizzes generally consist of 30 questions allowing students to check their understanding of chapter content.
  • Chapter test: a comprehensive, 25-question test completes the chapter assessment options.

Although Revel was required, Phillips and his students were piloting the program and testing out the capabilities, so flexibility in grading and earning points was necessary. Students could earn up to 60 points toward their final course grade, dependent upon the total percentage of assignments they completed. Readings and Try It assignments were required, along with end-of-chapter homework assignments consisting of 15-20 questions each. Students are allowed multiple attempts at completion to ensure they are learning from their mistakes and understanding the content. They work through the chapter at their own pace, with firm due dates for submitting assignments.

On the Fall 2016 end-of-semester survey, students reported success using various Revel tools.

32 percent of students reported using the note taking/highlighting feature.

  • “I used the notes/highlights feature to highlight areas I had questions about, summarize key terms, and revisited my notes and highlights as study guides throughout the semester.”

50 percent of students strongly agreed or agreed that the media options in Revel (self test-tables, timelines, image galleries, animations) were engaging.

  • “The media was very useful in explaining why things happened in different scenarios. Just words could have been used to explain these things, but they would have been less effective.”
  • “The interactive graphs in Revel helped prepare me for drawing graphs on exams.”

72 percent of students said the Revel format [read-a-little, do-a-little] helped them prepare for exams.

  • “Revel helped me study and prepare for my exams by providing practice problems and unit summaries that were easy to access and revisit when I studied.”
  • “Assignments really drill the concepts into your head, so studying wasn’t that bad.”

Four pencil-and-paper homework assignments are required outside of Revel, ranging from 12 to 20 questions, with assignments becoming more challenging as the semester progresses. Each homework assignment is expected to take students about an hour to complete and is intended to identify the student’s level of understanding after reading and working in Revel. Students may work together on homework assignments, but each student is required to submit their own work. No late work is accepted. Additionally, students take four exams comprised of problems and applied, short-answer questions, with each exam increasing in expectations and decreasing in the number of questions. The final exam may have just five to ten questions requiring students to put ideas and concepts together, giving the theory real world relevance.

Assessments

  • 84% Exams (four)
  • 10% Homework assignments (four)
  • 6% Revel assignments

Results and Data

Median Revel score was 81 percent. Figure 1 illustrates the relationship between Revel score and average homework, exam and final course grades.

  • Students earning a Revel score above the median earned average homework grades seven percentage points higher than students who had Revel scores below the median.
  • Students earning a Revel score above the median earned average exam grades six percentage points higher than students who had Revel scores below the median.
  • Students earning a Revel score above the median earned average final course grades eight percentage points higher than students who had Revel scores below the median.

Relationship between Revel score and homework, exam, and final course grades

Figure 1. Relationship between Revel Score and Average Homework, Exam and Final Course Grade, Fall 2016 (n=145)

In figure 2, Revel completion rates are analyzed to determine if a relationship exists between Revel completion and average homework and average test grades. Students were placed into two groups based on how many points they earned for Revel completion; students who earned all 60 points by completing at least 60 percent of Revel assignments earned higher average homework and test grades than students who did not achieve completion of at least 60 percent of Revel assignments.

  • Average Revel score: 72 percent
  • Students who earned full Revel completion points had an average homework score eight percentage points higher and an average exam score six percentage points higher than students who did not earn all Revel points.
  • 78 percent of students achieved full Revel completion points.

Relationship between Revel completion and homework and exam scores

Figure 2. Relationship Between Revel Completion and Average Homework and Exam Scores, Fall 2016 (n=145)

After piloting Revel in Fall 2016, average homework, test and final course grades increased over average grades from Spring 2011, 2012 and 2013; Phillips did not teach this course in 2014 and 2015 (figure 3).

Average scores before and after implementation of Revel

Figure 3. Average Homework, Test and Final Course Grades Before Implementation of Revel, Spring 2011, Spring 2012, Spring 2013 (n=135), and After Implementation of Revel, Fall 2016 (n=145)

Correlation analysis was explored, looking at Revel score and the average homework score. Correlations do not imply causation but instead measure the strength of a relationship between two variables, where r is the correlation coefficient. The closer the r value is to 1.0, the stronger the correlation. The correlation between Revel score and average homework score was r=.36, which is considered weak; Phillips believes this may be due to the stage of Revel development and his scoring decision. Because the product was still in beta-stage, Phillips had to adjust Revel scoring because of start-up and log-in issues, system lock-ups and other unforeseen technical glitches. Further analysis when Revel has been fully vetted and adopted is needed to develop and test this concept further.

Correlation between Revel score and homework score

Figure 4. Correlation between Average Revel Score and Average Homework Score, Fall 2016 (n=145)

The Student Experience

Responses from the Fall 2016 end-of-semester, voluntary survey of Phillip’s students indicate that the majority of responding students believe in the value Revel offers:

  • 95 percent of students strongly agree or agree that the “read a little, do a little” approach of Revel helped them learn and retain the material better than traditional textbooks.
  • 70 percent of students strongly agree or agree Revel provided additional resources that helped them learn more than they would have from more traditional pencil and paper homework.
  • 83 percent of students strongly agree or agree that their understanding of the course material increased as a result of using Revel.
  • 81 percent of students strongly agree or agree that the use of Revel positively impacted their quiz and exam scores.

Students offered the following comments about using Revel:

  • “Revel helped me to understand the main topics we were discussing, and the concept checks along the way made sure that I wasn’t simply skimming through the text. I was able to immediately test whether or not I understood what I had just read. I wish other classes such as my math courses and bio/chem courses would use Revel to teach new topics we are learning so we can focus on reviewing and better understanding them in the classroom.”
  • “It is nice having reading prior to class to help give you a feel for the topic.”
  • “Revel was useful in explaining information, and the media made concepts much easier to understand.”
  • “It’s a great program because it has real world examples to help you can understand concepts.”

Conclusion

Phillips decision to pilot Revel was predicated on the idea that Revel was a textbook replacement, not a digital homework enhancement to the course. He crafts his own homework assignments based on his lectures and Revel allowed him to continue using those assignments, while offering his students course content in an interactive format. He anticipated that the Revel format would lead to improved understanding of course content, which would in turn lead to fewer students requiring help outside the classroom. Phillips measures what his students struggle with based on their questions during office hours, and he found that students did not ask as many questions during the Revel pilot, particularly for challenging topics like Elasticity. Also, Phillips’ students shared anecdotally that the reading and interaction combination of Revel was having an impact on their content understanding and retention. This was repeated often by his students in the end-of-semester survey, highlighted by student comments such as this: “Revel gave additional material from lecture that was helpful in studying and reviewing for a test” and “The definitions and videos gave me clarity on the lecture material.” Phillips plans to evaluate his usage of Revel and may make changes to the implementation and scoring of Revel next fall.