Using Analytics for Continuous Course Improvement
An increasing number of postsecondary educators are using data to understand students’ comprehension and mastery of course content. But Robert Pribush, PhD, who teaches General Chemistry I and II at Butler University in Indiana has taken it one step further and has used data to continually improve his course. Even though his students’ scores are on par with national standardized test scores, over the past several years he has used student course data to study different aspects of their performance and has been able to determine certain behavioral patterns. Using this information he has changed his pedagogy to improve student outcomes.
We caught up with Dr. Pribush to ask him some questions about his pedagogy and how he used data to improve his course.
1. Why did you decide to try to raise student achievement?
I have always tried to attain the highest level of student mastery in my courses relative to normed national measures as one measure of my students’ ability to compete on a national level. Ever since I began teaching I have used the American Chemical Society (ACS) Examinations Institute’s nationally-standardized exams as a consistent way of comparing my classes from year to year. Although my students consistently performed significantly above national averages, in 2007 my classes seemed to have peaked in the average percentile performance on the ACS exam. Because I was using MasteringChemistry, an online homework program, I decided to see if my daily monitoring of scores and adjusting my classroom presentations in response to their performance would indeed raise my average student mastery as measured by the ACS exam.
2. What did you think were the most important issues to focus upon to help students improve their learning?
The average student that I encounter has not been challenged in high school. Most have achieved high grades with minimal study and without demonstrating conceptual understanding. Too much of what my average student has learned has been through memorization and not understanding. In addition to teaching towards content mastery, a primary focus of mine has been to teach students better study habits and work ethic.
I have tried to accomplish this by having a one-hour or less homework assignment containing ten or fewer problems for every textbook section covered. Each assignment is coded so the student knows where to find the material in the textbook. For example, Assignment 13.6 is related to section 6 in Chapter 13 of the textbook. I want students to effectively use their textbook. To ensure that each student is actively engaged in learning topics covered in class, the deadline for the completion of each assignment is two days after the material is introduced in class. This gives the student one day to discover what they do not understand and ask questions to allow completion of the assignment the second day. Keeping students on task is a vital component of enhancing student mastery of learning outcomes.
3. You seem to value the ability to monitor student progress throughout the semester, why is this so important to you?
Monitoring class and individual performance statistics, especially assignment grades, time-on-task, and problem difficulty level relative to database norms, all of which are easily accomplished with my online gradebook, are important to me. I try to attain a 90% mastery level for every assignment and every student. Students who fall below that mark tend not to earn an A in the course. If the class average is below 90%, I will alter class presentations to further review those topics.
Monitoring individual student time-on-task is particularly revealing. Students who are struggling in the course tend to spend much more time doing homework than high achieving students. Together with assignment grades, time-on-task data helps me identify students in need of additional help. Time-on-task monitoring is also an easy way to detect students who are not doing their own work!
My “office hours” with students are often done in the evening when students are doing their homework. Seeing answers students submit to problems makes it for me to determine how to efficiently help them better understand the concepts and calculations involved in solving the problems.
4. By being able to know where your students are struggling and adjusting your pedagogy, how has this impacted students?
By adjusting my pedagogy in response to what I observe from monitoring student homework, students realize that I am aware of their misconceptions and problem-solving difficulties and care enough to further help them in class and out of class to better master the material. I become a mentor as well as a professor. Seeing that I sincerely care about each student is a powerful motivating factor.
5. What are some of the biggest changes you have seen in students as a result of knowing their misconceptions, their weak areas, and their strong areas?
For each exam I give the students an outline that is outcomes based. For example, “The student should be able to perform calculations involving interconversion of concentration units”. The student is instructed to match this learning outcome to homework assignments as a way of discovering their strengths and weaknesses prior to taking the exam. I strive to have each student self-assess to better prepare for each exam. The result of this exercise is that students become more active learners, and assume personal responsibility for their learning.
6. By having student performance data so readily available, how do you think it has helped you improve your course structure and teaching practices from year to year?
I have never taught a course the same way two years in a row. While my basic course outline is well established, the way I deliver material, the examples I use to demonstrate concepts and problem-solving approaches, and the frequency that I have students problem solve in class are based on how I observe students performing on their MasteringChemisty online homework assignments. Having students become more active learners in a mentoring environment has led to higher achievement levels than many students thought possible. Students appreciate the effectiveness of the online homework program and the close professor/mentor-student relationships that can be achieved when students observe the professor caring enough to monitor their performance and work with them to effect higher levels of achievement. Establishing good student attitude toward learning and pedagogy used is critical to the success of a course. Students have to believe that a professor’s approach and methodology is effective.
For more detailed information about how Dr. Pribush redesigned his courses, read the full case study.
STEM Magazine has published the following article by Dr. Pribush: Educator Uses Analytics for Continuous Course Improvement and improved Learner Outcomes in Chemistry.
Dr. Pribush welcomes your comments below, or if you prefer to contact him directly you can reach him at 317-940-9406 or firstname.lastname@example.org.