Most online study tools use rigid multiple-choice, matching, or short-answer quizzes to provide feedback for which questions you answered “right” or “wrong”.  But what if your readiness for a test (or other type of performance) could be better measured using varying degrees of right and wrong for each concept?  What if a study tool could let you determine your level of readiness for each exam concept, by simply trusting you to answer the question: “How well did you know this?”

Such self-assessment is precisely the principle behind Brainscape’s application of the metacognitive concept known as Judgment of Learning, or JOL.  Asking learners to rate their JOLs on an item-by-item basis is an extremely valuable instrument in education, as it helps learners more accurately identify their strengths and weaknesses while helping educators focus on content areas in which their students are least comfortable (Black & William, 1998). It is also a critical instrument in educational software algorithms that use confidence-based repetition.

Over the past few decades, researchers have used various experimental formats to ask learners to rate their JOLs: either a series of confidence options (e.g. 1-5), a sliding confidence scale (0-100%), or a simple inquiry into whether the learner wishes to study the question again in the future (yes/no).  Researchers have also differed greatly in terms of how such JOLs are incorporated into a question list: either asking for the user’s self-assessment before each answer is revealed, after each answer is revealed, or after all answers in an entire list have been revealed.  The important common thread is that each of these types of experiments engages the learner’s metacognitive processing.

Brainscape uses a bold application of such metacognitive processing as its JOLs actually replace, rather than supplement, computer-provided feedback.  For example, if Brainscape asks the learner “What is the capital of Canada?”, the user does not actually have to type or select an answer but simply reveals the back side of a virtual flashcard.  The true self-assessment comes when the user rates her own confidence on each flashcard, which Brainscape’s algorithm then uses to determine how soon the flashcard should be displayed again.   (Low-confidence items are displayed more frequently until the user reports higher confidence.)

This shift of the assessment burden from the computer to the learner begs three important questions: (1) How do we know that the learner will accurately assess her knowledge? (2) How do we know the learner won’t just start revealing answers without genuinely thinking about them beforehand? and (3) What happens if a learner incorrectly reports high confidence in an item they actually don’t know very well?

In an answer to the first question, the Brainscape team reminds skeptics that adults are actually quite accurate in assessing their JOLs to begin with (Son, 2004; Dunlosky & Nelson, 1994; Lovelace, 1984).  Considering that Brainscape exists as a personal study tool, not an external assessment mechanism, Brainscape users have no motivation to “cheat” as they might have on an actual test or graded assignment.  Brainscape instead helps users manage their own knowledge and learn to become more honest with themselves.

To the second question, we remind readers that the alternatives to the flashcard approach are sub-optimal for time-pressed learners who are seeking a convenient and effective study experience.  Multiple-choice and matching questions are generally considered inferior from a cognitive standpoint since they merely test recognition rather than engaging active recall (Pashler et al., 2007), while short-answer and fill-in-the-blank questions are not conducive to quick study sessions as they involve time-consuming typing effort (particularly on the mobile phone), which is especially bothersome for users who already know the answer confidently.  Brainscape chooses to present questions as simple flip/reveal flashcards in order to increase the number of repetitions that can be achieved in a given span of time.  Research shows that the number of memory retrieval attempts, and the optimal spacing between them, are among the most important determinants of the strength of the memory encoding (Squire, 1992; Nelson & Leonesio, 1988; Janiszewski, Noel, & Sawyer, 2003).  We believe that the benefit of increased exposures outweighs the costs of potential “zone-out” – especially when the learner is already motivated to concentrate on learning the topic at hand.

Finally, regarding the possibility of overconfident JOLs, we should consider this a benefit rather than a risk of Brainscape’s approach, as the eventual correction of misjudged JOLs can often yield better retention benefits than if the confidence was never misjudged in the first place.  Butterfield & Metcalfe (2006) show that people are actually more likely to remember a corrected wrong answer if they had previously exuded high confidence that their submitted answer was correct and subsequently learned that it was wrong.  According to this logic, if a Brainscape user fails to recall a target displaying a previously high confidence ranking, she is likely to devote more mental energies to correcting the error.  Barrick & Hall (2004) show that such error corrections are even more beneficial when items are spaced out rather than massed together.

In fact, in a spaced or expanding study environment such as Brainscape, even a systematic display of overconfidence is unlikely to hinder the user’s progress.  While Meeter & Nelson (2003) demonstrate that a systematic confidence bias has no effect on the relative proportions of items in each JOL category, Pashler et al. (2007) confirm that “the cost of overshooting the right spacing is consistently found to be much smaller than the cost of having very short spacing” (p. 5).  Brainscape is therefore rather immune to users’ potentially poor study skills – and can actually improve your metacognitive skills over time.

Whatever software or technique you are using to help manage your study, applying self-assessment practices will consistently improve your performance on subsequent tests of your knowledge.  Remember to trust yourself, because you are the best judge of how well you know something!

References

Bahrick, H., & Hall, L. (2004). The importance of retrieval failures to long-term retention: A metacognitive explanation of the spacing effect.   Journal of Memory and Language, 52(4), 566-577.

Black, P., & Wiliam, D. (1998). Inside the black box: Raising standards through classroom assessment. London: King’s College.

Butterfield, B., & Metcalfe, J (2006). The correction of errors committed with high confidence. Metacognition and Learning, 1(1), 69-84.

Dunlosky, J., & Nelson, T. O. (1994). Does the sensitivity of judgments of learning (JOLs) to the effects of various study activities depend on when the JOLs occur? Journal of Memory and Language, 33, 545– 565.

Janiszewski, C., Noel, H., & Sawyer, A.G. (2003). A meta-analysis of the spacing effect in verbal learning: Implications for research on advertising repetition and consumer memory. Journal of Consumer Research, 30, 138–149.

Lovelace, E. (1984). Metamemory: Monitoring future recallability during study. Journal of Experimental Psychology: Learning, Memory, and Cognition, 10(4), 756-766.

Meeter, M., & Nelson, T. (2003). Multiple study trials and judgments of learning. Acta Psychologica, 13(2), 123-132.

Nelson, T. O., & Leonesio, R. J. (1988). Allocation of self-paced study time and the “labor-in-vain effect.” Journal of Experimental Psychology: Learning, Memory, and Cognition, 14, 676–686.

Pashler, H., Bain, P., Bottge, B., Graesser, A., Koedinger, K., McDaniel, M., & Metcalfe, J. (2007). Organizing instruction and study to improve student learning. Institute for Educational Sciences practice guide, U.S. Department of Education.

Son, L. (2004). Spacing one’s study: Evidence for a metacognitive control strategy. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30(3), 601-604.

Squire, L.R. (1992). Memory and the hippocampus: A synthesis from findings with monkeys and humans. Psychological Reviews, 99(2), 195-231.