MasteringMicrobiology educator study reports on implementation of a pre- and post-test at Shoreline Community College
- Students who scored less than 50 percent on the MasteringMicrobiology pre-test had a pre-test average score of 34 percent, an average increase of 26 percentage points on the post-test, and a final exam average of 75 percent. Students who scored 50 percent or higher on the pre-test had an average pre-test score of 58 percent, an average increase of 17 percentage points on the post-test, and a final exam average of 82 percent.
- Correlations between pre-test and final exam scores were weak, but correlations between post-test and final exam scores were strong.
- The instructor identified best practices for implementing a pre- and post-test to collect diagnostic data which included communicating with students so they understand how the results will be used.
Shoreline Community College, Shoreline, WA
Flipped and face to face
Modified MasteringMicrobiology; Microbiology with Diseases by Body System by Bauman
Fall 2015–Winter 2016
Judy Penn, Professor
Results reported by
Betsy Nixon, Pearson Customer Outcomes Analytics Manager
- Enrollment: 6,115 with 55 percent attending full-time and 45 percent part-time
- Established: 1964; operates under the regulations of the State Board for Community and Technical Colleges
- System: Quarters
- Student Profile: 53 percent of students are 17–24 years old
- Retention: 62 percent for full-time students and 63 percent for part-time students
About the Course
Professor Judy Penn has taught for more than 25 years at Shoreline Community College. Penn has an interest in both teaching a flipped classroom and in promoting online instruction and resources. Penn teaches the Microbiology course, taken primarily by nursing and allied health majors. Students generally are a mix of returning adult students and recent high school graduates, representing a wide range of ages, cultures, skills, and life experiences.
The introductory Microbiology course is a survey of microorganisms with a focus on healthcare applications. The emphasis is on disease process, microbial control and immunology. Laboratory techniques include isolation and identification of bacteria. This is a five-credit lecture and lab course. At the end of the course, students should be able to:
- Summarize the early history of microbiology, noting especially the major contributors to the development of the germ theory of disease and the problems presented by the belief in the theory of spontaneous generation. Describe the major developments in technology and microscopy leading to our current understanding of the nature of microorganisms and immunology.
- Demonstrate a comprehensive understanding of the structure, growth processes, metabolism and genetics of prokaryotic and eukaryotic microorganisms, as well as viruses. Use this knowledge to explain the mechanisms of action of antibiotics and other antimicrobial agents, as well as the implications of microbial activities in industry and the ecosystem.
- Understand and demonstrate the techniques involved in the culture of bacteria, including the properties and applications of various media and diagnostic reagents. Demonstrate aseptic technique. Understand the importance of those procedures in prevention of nosocomial diseases in a healthcare setting.
- Demonstrate a working knowledge of basic biotechnology procedures.
- Select appropriate, efficient strategies for isolation and identification of an unknown bacterium. Correctly interpret the results of these procedures. Write a scientific report that describes procedures & results and makes appropriate conclusions.
- Describe how microorganisms cause disease, as well as how the innate and adaptive defense mechanisms of the human body combat disease agents (includes basic principles of immunology, and immunological procedures).
- Demonstrate basic knowledge of the etiologic agents, methods of transmission, distinguishing signs/symptoms, and treatment strategies for the major diseases of the human body.
Challenges and Goals
Penn has used MasteringMicrobiology for several years and has continued to evaluate and adjust her implementation to address course and student learning needs. In 2014, she published an educator case study after redesigning her course to incorporate more active learning, moving from a mostly traditional lecture format to a fully-flipped format. She used Mastering to facilitate pre-lecture homework to prepare students for in-class activities, and she saw assessment scores and As and Bs in the course increase.
Penn has constantly tried to stay informed about research in Microbiology education, including staying updated with improved learning technologies so she could design the course to both teach the required concepts and to enhance critical thinking skills. A few years ago, Penn was interested in developing a course pre- and post-test for Microbiology so she could better track learning outcomes. She partnered with two instructors at other colleges to develop questions based on the American Society of Microbiology (ASM) learning objectives. Penn started administering the pre- and post-tests in her course to test the questions. She believed that the pre-test would be useful to understand what her students already knew coming into the course, and the post-test would help her evaluate improvement with their general knowledge of ASM objectives. While the pre- and post-test provided additional information, Penn said it is too short to fully assess those skills, but it is a way to gather additional data to evaluate learning.
Penn started using MasteringMicrobiology in Fall 2009, class testing the beta version because she believed it provided resources that would facilitate accomplishing her course goals. It provided a way to ensure students were prepared for class by having them do an activity outside of class in Mastering, and the resulting diagnostics enabled her to better understand student comprehension of the materials prior to class meetings. Before adopting MasteringMicrobiology, Penn had been giving a paper-and-pencil pop quiz or other short assessment in class to try to get real-time feedback. After adopting Mastering, the pre-lecture homework allowed her to identify student comprehension issues and misconceptions prior to class, so she could use lecture time to review the problem areas and spend more time on active learning.
Beginning with the Winter 2013 quarter, Penn redesigned the Microbiology course to a fully-flipped classroom. Before each class, students received the information needed for class by reading, viewing a video lecture, and completing a MasteringMicrobiology assignment. During class time, students participated in group activities, such as working on application questions including case studies, pairing up with a student to practice explaining a process, and making models or diagrams that illustrate concepts. Penn monitored the activities and answered questions or provided feedback as needed to guide the learning.
In 2014, Penn began working with two other instructors to develop a pre- and post-test based on the ASM outcomes. These questions have now been added to the MasteringMicrobiology platform and are available as a “Book/Source” within MasteringMicrobiology for all titles (ASM General Microbiology Learning Outcomes Questions). Penn began administering a pre- and post-test to her students in Fall 2015. When Penn developed her course pre- and post-tests from the available questions, she said she chose the questions which best fit her course outcomes. She reviewed the pre-test results to better understand what her students knew coming into the course, and said she finds that occasionally there is a pre-test item that most students already seem to know so she can minimize time spent on that concept during the term. The post-test results provided information on student learning and identified any trends where particular learning objectives were not being met. That helped her determine where she may need to make changes with concept coverage in future terms.
Penn’s course components for the period of study included:
Lecture assignments: The score for lecture assignments included the MasteringMicrobiology (MMB) homework assignments and the activities done in class. All MMB assignments were due at 7 a.m. on the due date. Each assignment item was typically 1–2 points, with a total of about 60 points available; however, students earned a maximum of 50 points. Because of the excess points available, no make-ups or late submissions were permitted. Penn told students not to stress about the point total, but rather to use these exercises as a way to check progress in learning. She recommended to students that they answer the MMB questions without looking at the book and their notes first, and if they didn’t get a question right, they could try again since they had multiple attempts.
Quizzes: There were eight in-class paper-and-pencil quizzes, with the lowest score dropped. At the beginning of each unit, students saw a learning objective document that covered lecture, lab, and a set of diseases to help them study. The quizzes included objective (multiple choice, true-false, matching), practicum, and short-answer questions. Students had an opportunity to earn up to 10 extra credit points for the course, which was added to their quiz score.
Final exam: There was a required comprehensive final exam covering all learning outcomes (LO) from the quarter. The LOs for the final exam were separate, and each required students to integrate concepts from several units. For example, students would be required to connect the structure of bacterial cells with anti-microbial agents or with pathogenicity. The final was a two hour, paper-and-pencil exam administered in class. The same question types as those used in the quizzes were used for the final exam.
Laboratory assignments: Lab assignments included a pre-lab assignment, due before each lab, which required students to:
- Read an exercise, prepare a flow diagram of the procedure, and a place to record results in a lab notebook.
- Take a MMB pre-lab quiz due at 7 a.m. on the day of lab.
No late flow diagrams or pre-lab quizzes were accepted. However, the lowest score for a 5-point pre-lab item was dropped, thus accommodating one late day. In addition, Penn provided optional MMB questions to help students review concepts after each lab.
MasteringMicrobiology pre-test: The MMB pre-test was required to be completed within the first three days of the quarter. It was a timed 20-minute test with only one attempt allowed, and contained 22 questions. The test was timed because Penn didn’t want students looking up answers since it’s designed to be a check of the knowledge they have coming into the course. Penn did not expect students to know much about Microbiology entering the course, so she did not expect high scores. The primary goal of the pre-test was to obtain a baseline score to evaluate learning during the course. The questions she selected were based on her course outcomes. Once the pre-test was completed, students were not able to see the correct answers, thus eliminating the possibility of using it as an answer-key for the post-test. Penn did not tell students in advance that the post-test would be the same as the pre-test.
MasteringMicrobiology post-test: The MMB post-test contained the same questions as the pre-test. The post-test was due a few days before the final exam. It was timed and formatted the same as the pre-test, and the results were primarily for Penn to compare to the pre-test scores. Students could also use it as a review tool for the final because they saw topics on which they might need more review.
There were also two optional, not-for-credit MasteringMicrobiology assignments. Those included a practice quiz (objective questions to practice for the graded quiz), and lab study questions (study questions to help test their understanding of course content and the applications from lab) that were provided after lab.
For other instructors looking to implement a MMB pre- and post-test, Penn recommends the following best practices:
- Limit the number of questions and time spent on these tests, particularly the pre-test so as not to overwhelm students.
- Explain the rationale for the pre-test thoroughly. Penn says when she first started using the pre-test, students would come to class saying they’d taken the test and felt stupid after not knowing most or any of the answers. To avoid that type of discouragement at the start of the semester, she advised talking with the students before administering the pre-test.
- Provide a reward of a few bonus points for students who increase their pre- to post-test scores by a certain percentage. It’s an incentive for them to actually think about their post-test answers, rather than just click any answer to get it done.
- Select questions which match what you are teaching in the course. Edit questions, if necessary, to make sure they fit your course outcomes and content.
- Analyze the data from the pre- and post-tests. Penn says it’s easy to skip this with all of the obligations instructors have, “so put this on your to-do list for the end of the quarter [semester], during that lull just before the final exam has to be given and graded.”
- 55% Quizzes (8 with the lowest dropped)
- 25% Lab (lab notebook, paper, and MasteringMicrobiology pre-lab quizzes)
- 10% Lecture (class activities and MasteringMicrobiology homework assignments)
- 10% Comprehensive final exam
Results and Data
Penn’s earlier results were published in an educator case study in 2015 which covered 2012–2014. In 2016, she evaluated the results from the Fall 2015 and Winter 2016 quarters when she implemented the pre- and post-tests. This analysis focused on the findings from the pre- and post-tests. The Fall and Winter data were combined for the initial analysis, and results showed:
- The average increase from the pre- to post-test was 22 percentage points.
- Students who scored less than 50 percent on the pre-test had an average increase of 26 percentage points on the post-test and a final exam average of 75 percent.
- Students who scored 50 percent or higher on the pre-test had an average increase of 17 percentage points on the post-test and a final exam average of 82 percent. (One student who did not take the post-test was not included in this analysis.)
The diagnostic pre- and post-tests were developed to evaluate student understanding of the ASM learning objectives and required Microbiology concepts, and Penn’s final exam is written to evaluate student understanding of those same concepts. Figures 1 and 2 show the correlations by quarter for the pre-test to final exam scores (orange) and the correlations for the post-test to final exam scores (blue). Correlations do not imply causation but instead measure the strength of a relationship between two variables, where r is the correlation coefficient. The closer the r value is to 1.0, the stronger the correlation. The corresponding p-value measures the statistical significance/strength of this evidence (the correlation), where a p-value <.05 shows the existence of a positive correlation between these two variables.
- For Fall 2015, there is a weak correlation between the pre-test and the final exam average, where r=.25 and p=.182.
- For Fall 2015, there is a strong correlation between the post-test and the final exam average, where r=.62 and p<.001.
- For Winter 2016, there is a weak correlation between the pre-test and the final exam average, where r=.36 and p=.108.
- For Winter 2016, there is a strong correlation between the post-test and the final exam average, where r=.75 and p<.0001.
The weak pre-test correlations provided evidence that these students did not come into the course with an understanding of the concepts the course covers. The increase in pre- to post-test scores noted above, along with the strong post-test to final exam score correlations provided evidence that students developed a better understanding and comprehension of the concepts during the course. This information helped Penn understand the level of learning that took place and allowed her to monitor how well the outcomes were being addressed.
Penn felt that, while there was a strong correlation between the post-test and the final exam,she would like to see student scores on the post-test reach a higher range. However, she feels that the performance on the post-test could be, in part, due to the fact that it is timed, since some students do not do well with timed assessments. In addition, Penn believes that having such a small number of questions on the post-test, typically only one question per learning outcome, meant the assessment wasn’t as accurate in testing their full knowledge as one that would ask more than one question for each outcome.
Correlation between MasteringMicrobiology pre-test and post-test scores to final exam scores showing the semester change, fall 2015
Figure 1. Correlation between MasteringMicrobiology Pre-test and Post-test Scores to Final Exam Scores Showing the Semester Change, Fall 2015 (n=29)
Correlation between MasteringMicrobiology pre-test and post-test scores to final exam scores showing the semester change, winter 2016
Figure 2. Correlation between MasteringMicrobiology Pre-test and Post-test Scores to Final Exam Scores Showing the Semester Change, Winter 2016 (n=21)
The Student Experience
Penn’s students were surveyed and asked about their experience using MasteringMicrobiology in the Fall 2015 quarter. 100% of students who completed the course did the survey. When asked how Mastering impacted their learning, they gave the following comments:
- “Easy to use, lots of tools besides the assignments.”
- “Having concise videos to explain topics was helpful.”
- “It helped me review the material, and the animations clarified confusing concepts. A helpful website to have.”
- “It saved me time. In most classes I have to make up my own study methods, extra reviews etc. Then, I don’t know if what I am doing is 100% accurate because I don’t have a rubric checked by the professor.”
- “These study materials helped me effectively review without quadruple checking that the quiz I made myself is accurate.”
- “It kept me staying on track with what we were learning in class.”
In addition, when asked (with 10 being extremely likely and 1 being not very likely at all) to indicate how likely they would be to recommend Mastering to another student, the average answer 8.4.
The paper, Using Data to Improve Student Achievement, states that, “As lifelong learners, teachers recognize that their professional practice continues to evolve as they reflect and act on new information. If teachers have information that helps them confidently identify the root of educational challenges and track progress, they can more readily develop action plans that will have a positive impact on their students.”1 Penn understands the importance of remaining current with Microbiology topics and educational issues and using technology to help her accomplish her goals in the course. MasteringMicrobiology is a tool she has used for several years to provide resources and graded assignments in her course. However, she wanted to develop a diagnostic test to gather data to assess students pre- and post-semester, so she worked with other instructors to develop questions which Pearson has since added to Mastering so all users have access. Penn now administers a pre- and post-test, and uses that data, along with all other scores from her students, to better understand their course performance and achievement of learning outcomes which helps her determine if changes need to be made moving forward.
1 van Barneveld, Christina, Using Data to Improve Student Performance (2008)