Effectiveness Assessment
General Description of Experiment
The experiment consisted of 2 short (about a couple of paragraphs) articles, each summarizing a short scientific experiment, and displayed through the WebLenses Portal environment. The first article has the WebLenses associated with it, and they can be turned on for the treatment group, or off for the control group. The second article doesn't have any WebLenses associated with it, and is used to measure transfer of learning (from the first case/ article).Two groups of subjects are established: a control group and a treatment group. A subject in the control group is shown the first article with the WebLenses turned off, and then asked to answer some questions and reason about the article. Paper-based reference material (identical to the one available through the online WebLenses) is available to the subject.
In the treatment group, the subject reads the same article through the WebLenses Portal (identical to the control's) but the WebLenses are turned on, and allow just-in-time, on-demand, context-sensitive access to the reference material (identical to the one presented in paper form to the control group).
What is tested in the case of the first article is whether with just-in-time, on-demand, context-sensitive presentation and visualization of the same "facts", the treatment group can demonstrate a more critical, statistics-based analysis, and more informed reasoning about "the same story".
In the second article, the experiment is looking at whether working with the WebLenses on the first article, will lead the subjects in the treatment group to more critically examine the second article, even if WebLenses are not provided in this case. In other words, can carefully chosen WebLenses help subjects learn to be more critical, differentiating, and thoughtful about what is presented to them in the article.
Design
The experiment has a 2-by-2 design (see below), with a control/treatment, and WebLenses exposure/Transfer assessment. There were four subjects in each group.Each subject was randomly assigned to either the control or the treatment group, and each one participated in the experiment individually (not as part of a group). There was no pre-defined order of subjects taking the experiment, and it was purely based on convenience and subjects' availability.
| Control (paper reference material) | Treatment (WebLenses) | |
| Reference material available | ||
| Retention/Transfer |
Results
The results show the performance of the control group vs. the treatment group:
As can be seen, the initial performance of the treatment subjects
(having access to WebLenses, as opposed to paper-based reference
material) was higher (.75 vs. .5 correct answers, out of 1.0).
Both groups suffered from a drop in performance during the
retention/transfer phase (Control dropping from .5 to .42, and
Treatment from .75 to .67), but in this phase too, the treatment group
performed better (.67 vs. .42).
Due to the small sample size (4 in each group) and the small number of articles (2 for each group), it is impossible to draw general conclusions, but it seems that:
- With WebLenses, subjects spend more time engaging the reference material
- Subjects compared more, side by side, cases when using WebLenses
- They seem to recall and transfer more from the case where they had access to WebLenses to the case where they don't