REVEL educator study assesses quiz, exam, and final course grades at Central Michigan University
- Data for this course indicate that after implementation of Revel average exam scores increased 11 percentage points from the previous semester.
- Data show a strong positive correlation between average Revel scores and both average quiz scores and final course grades.
- Students showing mastery of course material by earning an A/B/C average quiz grade had Revel scores 12 percentage points higher than students earning a D/F average quiz grade.
Central Michigan University, Mt. Pleasant, MI
Principles of Computer Programming
Revel for Introduction to Java Programming by Liang
Spring 2015 and Fall 2015
Patrick Seeling, Associate Professor
- Locale: four-year, rural, public institution approximately 150 miles northwest of Detroit
- Enrollment: approximately 27,000
- Undergraduates: 80 percent
- In-state students: 95 percent
- Student-faculty ratio: 21:1
- Classroom size: 33 percent with fewer than 20 students
- Full-time retention rate: 76 percent
- Four-year graduation rate: 20 percent
- Gender: 56 percent female
- Ethnicity/race: 11 percent minority
About the Course
Associate Professor Patrick Seeling has been teaching since January 2008, arriving at Central Michigan University in 2011 and has been teaching Introduction to Programming since Fall 2014. Introduction to Programming is a one-semester, three-credit course enrolling over 350 students each year. Most students who take this course are earning a major or minor in computer science or engineering, but it is open to all students as the course can also fulfill a general education requirement. The course focuses on algorithm development and problem-solving methods, as well as the design and development of computer programs in a high-level programming language. Upon successful completion of this course, students will be able to:
- Describe the underlying principles of quantitative and algorithmic problem solving methodologies
- Apply algorithmic skills to a variety of problems amenable to solution or simulation by computer
- Write well documented computer programs in a high level programming language
- Apply computational skills and scientific principles to problems involving real world data
Challenges and Goals
In 2011, instructors at Central Michigan discussed the need for a more hands-on approach to the Introduction to Computer Programming course. They wanted students to do more programming, knowing that actual programming practice is the only way for them to become proficient. In the past this had been a challenge because they lacked the time necessary to create and grade individual programming assignments outside of those already being implemented in the course.
Looking for a digital program that might act as a ‘computer lab outside of the computer lab’ which would also identify student errors and provide feedback for correction led them to MyProgrammingLab. MyProgrammingLab was adopted as a departmental decision for Fall 2011 and was implemented by all instructors. The structured homework kept students on task, provided the additional hands-on coding work they needed, and the program also provided personalized feedback in the form of error messages that include both computer and plain English interpretations of causes for the incorrect answer.
In Fall 2015, Seeling decided to pilot Revel, which was new, in place of MyProgrammingLab; the one year subscription life offered a definite value to his students, and he was interested in seeing how he might turn the positive experience he had with MyProgrammingLab into a similarly positive experience with Revel.
Seeling’s use of Revel is required; the program is used both at home and in class, using a personal computer. Students use the program for reviewing new concepts and content understanding, homework assignments, practice, and exam preparation. Seeling’s goal for assigning homework in Revel is to give students additional practice with coding, after getting exposure to the content in lecture. Additionally, Revel helps his students assess their own understanding of the course material and track their progress, using the performance dashboard. As the course instructor, Seeling’s role is to introduce new concepts in lecture, reinforce coding issues that students struggle with on Revel assignments in lab, and assign content and homework in Revel.
The course is offered on a pass/fail basis for individually graded items; therefore, students don’t earn traditional grades, but instead earn points for Revel assignments, lab-based programming assignments, Blackboard quizzes and exams, with all of these assessments requiring competency levels in order to earn the points.
In his flipped classroom, Seeling’s students participate in three active learning sessions per week; in order for students to have an effective classroom experience, it is essential that they come to class having completed their Revel assignments. His students get their first exposure to the chapter material outside of class, working on their own, and then attend class to get the deeper learning around the more challenging topics. Reading assignments have a due date listed on the course calendar; students are expected to review the material, take notes and be prepared to discuss it in class. Additionally, each topic covered in class has associated exercises in Revel that must be completed by the assignment due date. Seeling indicates that students need very little support to get started in Revel; at the beginning of the semester, he gives them general registration information and guidelines for usage, then lets them figure out how to use the combination of narrative, Interactives and homework on their own.
Seeling refers to lectures as ‘every day is lab day’; sessions are completely hands-on, with students working either together in groups or alone if desired, to complete coding and programming assignments. Seeling and his teaching assistant circulate the room, watching and working with students as the need arises. Students working in pairs or groups are still responsible for submitting their own work, but discussion and collaboration is strongly encouraged. In order for this flipped environment to work, students are expected to bring their own computing device to lecture each day and come prepared to begin working on new material by completing the reading and assignments in Revel.
As part of each Revel assignment, Seeling’s students use the VideoNotes in Revel, which are narrated step-by-step video tutorials that show how to solve problems from design through coding. Students also have the option to view Animations which step them through the code line by line, showing exactly what is happening in the program. In addition, students are expected to use the Interactives in Revel, which allow them to practice what they have just read and learned; the Interactives enable students to run and modify live code with the goal of retaining key programming concepts. These learning tools are interspersed throughout Revel, encouraging students to complete assignments immediately after reading and learning about them.
On a voluntary Fall 2015 end-of-semester survey of Seeling’s students (37 percent response rate), students responded to how useful they found these course components to be:
- 85 percent of students agreed or strongly agreed that the VideoNotes in Revel were useful in showing them how to solve problems completely, from design through coding.
- 62 percent of students agreed or strongly agreed that the Animations in Revel that stepped line by line through code, showing what was happening in the program, helped them better understand how to replicate the coding on their own.
- 64 percent of students agreed or strongly agreed that the use of Interactives in Revel (LiveExamples) helped them practice and modify running live code for retention of key programming concepts.
- 93 percent of students agreed or strongly agreed that the ‘Read a Little, Do a Little’ approach of Revel, where reading is mixed with Interactives and quizzing, was an effective way to learn the course material.
Students are given one Revel programming assignment approximately every other week. They have unlimited attempts at assignment completion; Seeling’s goal is for students to learn the material and get it correct on homework, so they can replicate that success on quizzes and exams. Students must earn at least 75 percent on each exercise by the due date to pass that assignment. Based on this competency level, points are earned on an all or nothing basis.
Eight quizzes are administered through Blackboard; quizzes are multiple choice and questions are taken from the Pearson test bank. The 10-question quizzes are open book and timed at 30 minutes; questions are randomized and pooled, and students have two attempts at each quiz. The highest score is used to record the final grade—students must score 6 of 10 points to earn a passing grade for the quiz.
The Revel assignments, programming/coding assignments and Blackboard quizzes create a strong collection of formative assessments prior to the summative exams. There is one midterm exam and a cumulative final exam, each taken in a computer lab on campus. Each exam has three parts which are graded individually; students have just one attempt at each question or exercise. Each part has a separate pass/fail grade which translates to approximately 70 percent achievement and exams are timed to 50 minutes.
- Part 1: 10–12 general-theory multiple choice questions from the Pearson testbank; questions are randomized and pooled
- Part 2: 10–12 multiple choice computer coding and program analysis questions; students are given short code snippets to read, understand and correct
- Part 3: One short code writing assignment.
Seeling expects his students will spend at least 2 hours for every hour spent in class working on course materials, and at least three hours per week working in Revel, recognizing that some students will spend more time working through the interactive narrative, the video tutorials and the animations than others. On a voluntary Fall 2015 end-of-semester survey of Seeling’s students (37 percent response rate), 60 percent of students responded that they spent at least 2–4 hours working in Revel each week, and another 20 percent of students indicated they spent more than four hours.
- 8 points Programming assignments
- 8 points Blackboard quizzes
- 8 points Revel assignments
- 3 points Midterm exam
- 3 points Final exam
Results and Data
Figures 1 and 2 are correlation graphs; correlations do not imply causation but instead measure the strength of a relationship between two variables, where r is the correlation coefficient. The closer the r value is to 1.0, the stronger the correlation. The corresponding p-value measures the statistical significance/strength of this evidence (the correlation), where a p-value <.01 shows the existence of a positive correlation between these two variables:
- A strong positive correlation exists between average Revel grades and average final course grades, where r=.67 and p<.01 (Revel homework contributes 27 percent to the final course grade, influencing this relationship)
- A strong positive correlation exists between average Revel grades and average quiz grades, where r=.50 and p<.01
For students, the formative Revel assignment grades are intended to help them identify where they are in terms of successfully completing the more summative midterm and final exams; it appears that performance on these assignments could be a leading indicator of exam success (additional research is needed to develop and test this concept further). As a best practice, Revel assignment grades are intended to help Seeling identify students early on who are struggling and might be at risk of poor overall course performance.
After required Revel homework assignments were added to this course in Fall 2015, data show that average exam grades increased by 11 percentage points, improving from a failing grade average (59%) to a passing grade average (70%), when compared to Spring 2015, and final course grades improved one percentage point (figure 3). A t-test, which measures whether the means of two groups are statistically different (average exam grades for Spring 2015 and Fall 2015 in this study), was used to compare the average exam grades. Results of the t-test show that students in Fall 2015 (mean = 70%) scored higher on average exam grades than students in Spring 2015 (mean = 59%), where t(118) = -2.14 and p<0.05, indicating that this increase was statistically significant (table 1).
Figure 4 looks at the relationship of the average quiz grade distribution per average Revel homework score. Students showing mastery of course material by earning an A, B or C average on quizzes had average Revel grades 12 percentage points higher than students who earned a D or F average quiz grade.
- Students earning an average quiz grade of A or B scored an average of 96% on the Revel homework assignments.
- Students earning a final course grade of F scored an average of 75% on the Revel homework assignments.
Correlation between average Revel grade and final course grade
Figure 1. Correlation between Average Revel Grade and Final Course Grade, Fall 2015 (n=41)
Correlation between average Revel grade and average quiz grade
Figure 2. Correlation between Average Revel Grade and Average Quiz Grade, Fall 2015 (n=41)
Comparison of average exam scores and average final course grade before implementation of Revel and after implementation of Revel
Figure 3. Comparison of Average Exam Scores and Average Final Course Grade, Before Implementation of Revel, Spring 2015 (n=79) and After Implementation of Revel, Fall 2015 (n=41)
Two-sample t-Test assuming equal variances
Table 1. Two-Sample t-Test Assuming Equal Variances, Spring 2015 Exam Average (n=79) and Fall 2015 Exam Average (n=41)
Relationship between average Revel score and average quiz letter grades
Figure 4. Relationship between Average Revel Score and Average Quiz Letter Grades, Fall 2015 (n=41)
The Student Experience
Responses from the end-of-semester voluntary survey of Seeling’s students (37 percent response rate) indicate that the majority of responding students recognize the value of Revel for Programming:
- 80 percent of students agree or strongly agree that they learned more using Revel than they would have from a traditional printed textbook.
- 93 percent of students agree or strongly agree that the ‘Read a Little, Do a Little’ approach of Revel, where reading is mixed with interactives and quizzing, was an effective way to learn the course material.
- 93 percent of students agree or strongly agree that their understanding of the course material increased as a result of using Revel
- 80 percent of students agree or strongly agree that use of Revel positively impacted their quiz and exam scores.
Student survey responses to the question “What did you like most about Revel?” include:
- “It was by far the best e-book I’ve used. It was easy to navigate and the content was direct and well-worded.”
- “The videos and live coding examples were very helpful.”
- “I liked the fact that the book read straightforward. There weren’t any non-essential details, everything contained in the chapters was important.”
- “I liked the video examples and the practice questions between the different sections.”
In the flipped classroom model, students take responsibility for their learning outside the classroom so that critical thinking and conceptual learning can take place inside the classroom. In order for Seeling’s flipped lab environment to be successful, students must complete the pre-lecture work so they can begin hands-on work with design and coding immediately. Additionally, students must bring their own computer or other device in order to complete their own work; at times this led to students watching or listening instead of creating and producing code, but ultimately learning took place as students learned and shared with one another. Seeling intends to continue with Revel in the future, potentially making some modifications to the course design and structure.