pattern #11 - Test

Testing Activity

Definition

Testing is a procedure for critical evaluation, meant to determine the presence, and quality of something while correcting misconceptions, reinforce key knowledge and providing learners with an opportunity to self-assess their knowledge.

  • VALUE Improve Retention
  • COMPLEXITY High
  • Design Effort High
  • Share

Overview

Designing sound testing activities is a challenge. Some would even argue that is more difficult to test something than it is to teach it.
If done right, testing activities will not only measure learner progress, but also reinforce key knowledge, correct common misconceptions, guide learners in the learning process, build confidence in own knowledge, motivate to perform better, reduce fear of formal exams and provide trainers with actionable data on performance.
In his book E-Learning by Design, William Horton defines 8 components for testing items. While these are not mandatory for each testing activity, they should be considered when designing for testing.

Testing Activity Components
Test Components

I know I am breaking the Spatial Contiguitiy Principle, from Mayer’s Multimedia Principle, I have presented the details sepparately due to the lack of space for this much text.

# Name Description
1 Stem A statement that needs to be completed or answered by one of several given choices, and typically includes:
• Lead-in. A statement that provides context or background information for the question you are asking;
• Question. The statement that needs to be answered;
• Instruction. A statement that provides learners with a clear indication on how to answer to the question;
2 Distracters The wrong answers; When writing distracters, make sure you (i) avoid distracters that are clear wrong answers, (ii) provide enough distracters to avoid guess answers and (iii) do not create distracters that are too close to the key as your goal is not to trick learners into giving the wrong answer
3 Key The correct answer
3 Confirmatory feedback Which basically means marking correct answer with green and wrong answers with red; most testing items provide this functionality
4 Formative feedback Providing additional information to help them better understand the correct answer;
5 Question Score Shows the score for the existing activity;
6 Progress/Navigation Shows the current item and the total number of questions to be answered; enables learners to browse through questions;
7 Action Buttons Enable learners to check their answer and go to the next activity; It is important to allow learners to press the Check/Next button even if they have not provided an answer as you do not want your learners to guess-answer when they do not know the correct answer.
8 Reference Materials Various types of resources (such as photos, audios, videos, PDFs) that can be used to add more validity to the testing item.

The concepts of validity and reliability are central to a sound design of testing activities. While reliability (the degree to which a testing item produces consistent results over time) is not a concern for most digital testing activities, as they are close-ended, validity (how well a testing item measures what is supposed to measure) needs to be well embedded into the design of any testing activity.
A classic example bad practice when it comes to testing validity is testing just the memory rather than the actual outcome. Let’s take a course on appraising staff performance, where the global learning objective is to help managers provide critical feedback to staff members, but the assessment activity only assesses key theoretical aspects. While knowledge is central to skill development, the goal of the course is a skill that is not actually tested i.e. ability to provide critical feedback to staff members. This is a validity issue.
A good start when designing assessment activities is to ask yourself questions such as:
— Which are the most common mistakes students make when applying new knowledge and skills?
— Which are they key concepts or procedures that need reinforcement?
— Which are the typical decisions that need to be taken in a real-world activity?
— Which are the typical activities where learners will need to apply?
— Which are the caveats in applying new knowledge and skills that learners need to be aware of?
— What is the wording that students would most probably encounter in real-world?

EVIDENCE-BASED RESEARCH

In his great book Make It Stick, Peter C Brown makes a great case for using testing activities for more than just a simple dipstick and try to use it as a tool for learning. The great benefit of testing activities is that by recalling information, it strengthen the neural connections that hold that information making it easier to retrieve in the future.
In a study performed with a middle school science class, part of the material was complemented with a set of low-stakes quizzes with feedback while another part was not quizzed. In a test a month later, students managed an A- on average, while for the part that was not quizzes, they received an C+.

Testing Activity Components
Ebbinghaus’ forgetting curve


It is a known fact that we forget most of the new information in less than a day, something highlighted in Ebbinghaus’ forgetting curve (ditch Dale’s Cone of Learning, as it mainly includes anecdotal data).
Therefore, central to improving the way we learn is stopping the process of forgetting. The testing effect, also known as retrieval testing, is one way to effectively achieve this.

Testing Activity Components
Retrieval Testing


In a study where students had to read a story that named 60 concrete objects, students who where tested immediately remembered 53% of the objects and 39% a week later. However, those who were not tested initially, only recalled 28% a week later. Another group who took three tests immediately after reading the story, remembered 53% of the objects a week later, similar with the first test. They were basically immunized against forgetting.


In another study, students who read a passage and then took a test asking them to recall what they have read retained 50% more than students who had not been tested. A study that even reached the mainstream media, through a New York Times article.
There are also nuances on applying testing as retrieval practice. Studies show that providing learners with formative feedback that better explains the wrong answer is correlated with better retention rates. However, providing delayed feedback is essential, as immediate feedback can become like training wheels on a bike i.e. when you remove them, they act as a gap that needs to be overcome. One example of delayed feedback is allowing learners to search for the correct answers only at the end of the test.
The same study also highlights that open-ended testing activities, such as short-answer or essay writing, are more effective than close-ended activities, such as single-choice questions.

Worked Example

The above test example is a proposal for a private hospital, to support the yearly training of doctors. While it may be less known, doctors tend to get worse at providing a sound prognosis over time. That is because of their experience, which exposes them only to the most common situations and makes it harder for them to identify symptoms of a rare disease, which they may have not encountered in years. The testing activity is built around daily situations.
Focus on performance
Questions are built around performance rather than content, as the goal of the eLearning module is to address key mistakes done in everyday practice.
Don’t force an answer
The target answer can check the correct answer without submitting an answer. While the majority of testing activities require an answer as a basic usability heuristic, this means that learners are forced to guess-answer which in turn may generate false positive in testing results.


similar patterns

Explore patterns that serve similar goals