Current state of game-based assessment
What is the vision of game-based assessment?
Games can provide assessment without tests. We can use players’ interactions in these engaging environments as evidence for what students know and can do. Rather than tell a student, “stop your learning activity and take this test,” we can gather data as they are immersed in a learning game that gives us the same or more information about student progress.
Some examples from my work include SimCityEDU (assessing systems thinking), Mars Generation One: Argubot Academy (assessing argumentation), and Alice in Arealand (assessing understanding of the concept of area). Val Shute at Florida State University has an excellent example in Physics Playground (assessing qualitative physics) and the Educational Gaming Environments group at TERC has been doing nice work with a game called Impulse (measuring implicit science learning).
Why games for assessment?
- Games are already part of the natural activity of students. The latest Pew research tells us that 97% of students aged 12-18 play some kind of digital game. This is a natural activity for them, we won’t interrupt their lives to collect data.
- Games allow us to present real-world contexts. This increases the likelihood that inferences about students gained from the data will apply outside the classroom.
- Games allow us to assess skills we otherwise couldn’t with a traditional multiple-choice assessment. Assessment of things like creativity, collaboration, and persistence are possible using data from game play.
- Games provide streams of data that allow us to assess process as well as final product. We can understand how a student came to an answer, not just the end result. This allows us to provide better feedback to the student and gives us a better understanding of how students progress.
- Games share fundamental processes with assessment. Essentially this same process holds true for assessments and games (in assessment it is called the 4 process model. The system presents a problem, the player/student produces a work product in response, we then evaluate this product based on rules and assign a score. We accumulate these scores into an overall score, and then use that information to present the next task. It is the same thing in games and assessment!
What are the most appropriate uses of game-based assessments?
The most appropriate use of game-based assessment is in formative situations where the game is both a learning and assessment tool. Games have a number of challenges as stand-alone assessments (see below) that are overcome when they function in a more formative classroom environment integrated with instruction.
That word “integrated” is key. There are a number of nice games out there that never get any traction in classrooms because it is a huge task to ask teachers to go out and find good games and determine for themselves how to integrate a given game into their curriculum. Rather, we need to find ways to create curricula that have games embedded in them for the particular topics for which games are well-suited. Those topics may be things that are difficult to teach or big ideas that need to be emphasized, for example.
How can game-based assessments be incorporated into large-scale and/or high stakes assessments?
It is going to be very difficult to incorporate games into a large scale assessment situation in which a broad range of content must be assessed in a very limited amount of time, as is the current conceptualization of most large-scale assessment programs like NAEP. If we broaden our view of assessment to gathering data from learner’s interactions with digital environments on a more ongoing basis, it would be possible to incorporate game-based assessments into a large-scale assessment program.
However, if we then want to add high stakes by attaching key decisions like graduation or promotion to the results, there are even more concerns. The validity and reliability of game-based assessments has not been established to the extent that the results should be used as a sole basis for any of these decisions. Of course, the question remains as to whether any single assessment should be the sole basis for such a decision, but that is a different argument!
What kinds of insights into student thinking do game-based assessments provide?
My work has focused on aligning evidence from games to learning progressions. Evidence can be extracted from the rich log file data resulting from students’ interactions with a game that can be aggregated to provide estimates of student mastery on stages of a learning progression. I have found learning progressions to be a helpful frame for modeling the movement from less to more sophisticated levels of understanding. The concept of stages in a learning progression also aligns well to the idea of levels in a game. Games can then be designed so that learners make meaningful choices aligned to the levels or stages of the progression, including misconceptions association with particular levels.
What are some of the limitations for game-based assessments?
- Time – good games are systems and it takes time to learn the system. As noted above, many of our current assessment paradigms require us to assess large amounts of content quickly. Games are not efficient means of assessment. In many cases, asking a multiple choice question is much quicker. The promise of games is that the engagement they encourage and the situations they can present will provide better evidence for constructs, particularly those that are difficult to measure in traditional ways. However, to make use of this, we are going to need to reconceptualize assessment as something that can happen over time as we collect data from ongoing interactions.
- Coverage – Games are generally good at going deep with a narrow range of content. It is difficult for them to efficiently cover a broad range of content.
- Psychometrics – Many traditional methods developed to evaluate the reliability and validity of assessments do not fit game-based assessments. While research is underway on modifications of existing conceptualizations and methods, these are not widely tested and certainly not widely accepted. This means while we can provide evidence that can be used to make low-stakes decisions such as how to group students for a particular lesson (not an ongoing basis), using these measures to make decisions with long term consequences should be done with caution.
Are there advantages of using game-based assessments over simulation-based assessments?
Game-based assessment provides activity designed from the beginning with engagement in mind. Elements from narrative to difficulty to the core activity loop are designed to maintain student attention and motivation. There is a research-base suggesting that assessments in which students are more engaged and motivated result in more valid inference. This is the major thing that game-based assessment can offer over simulation-based assessment.
About the Author
Kristen DiCerbo, PhD, was a principal research scientist for the Center for Learning Science & Technology within Pearson’s Research & Innovation Network. Dr. DiCerbo’s research program centered on digital technologies in learning and assessment, particularly on the use of data generated from interactions to inform instructional decisions. She has conducted qualitative and quantitative investigations of games and simulations, particularly focusing on the identification and accumulation of evidence. She previously worked as an educational researcher at Cisco and as a school psychologist. She holds doctorate and master’s degrees in Educational Psychology from Arizona State University.