GroupvsIndividual Web Accessibility Evaluations: Effects with Novice Evaluators

Giorgio Brajnik · Markel Vigo · Yeliz Yeşilada · Simon Harper

19 citations0/0 files storedInteracting with Computers

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Digital Accessibility for Disabilities Usability and User Interface Design Web Data Mining and Analysis

Abstract

We present an experiment comparing performance of 20 novice evaluators of accessibility carrying out Web Content Accessibility Guidelines 2.0 conformance reviews working individually to performance obtained when they work in teams of two. They were asked to first carry out an individual assessment of a web page. Later on, they were matched randomly to constitute a group of two and they were asked to revise their initial assessment and to produce a group assessment of the same page. Results indicate that significant differences were found for sensitivity (inversely related to false negatives: +8%) and agreement (when measured in terms of the majority view: +10%). Members of groups exhibited strong agreement on the evaluation results among them and with the group outcome. Other measures of validity and reliability are not significantly affected by group work. Practical implications of these findings are that, for example, when it is important to reduce the false-negative rate, then employing a group of two people is more useful than having individuals carrying out the assessment. Openings for future research include further explorations of whether similar results hold for groups larger than two or what is the effect of mixing people with different accessibility background. When novice accessibility evaluators work in groups, their ability to identify all the true problems increases (by 8%). Likewise, reliability of group evaluations increases (by 10%). Individual or group evaluations can be considered as equivalent methods with respect to false positives (if differences up to 8% in correctness are tolerated). Individual or group evaluations can be considered as equivalent methods with respect to overall effectiveness (if differences up to 11% in F -measure are tolerated).

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 19 citations exist upstream.

References

1982Group versus individual performance: Are N + 1 heads better than one?2012Guidelines are only half of the story 2006Web Accessibility: Web Standards and Regulatory Compliance 2011Automatic web accessibility metrics: Where we are and where we can go 2013Exploring perceptions of web accessibility: a survey approach 2011The Expertise Effect on Web Accessibility Evaluation Methods 2008A survey on the accessibility awareness of people involved in web development projects in Brazil 2008Web Accessibility Evaluation

Cited by

2018Inclusion in the Third Wave: Access to Experience 2022Accessibility and usability evaluation of university websites in Afghanistan: a comparison between public and private universities 2021AET tool for hybrid accessibility evaluation 2023Accessibility of Mobile Apps for Visually Impaired Users: Problems Encountered by User Evaluation, Inspections and Automated Tools 2024Accessibility Inspections Using the Web Content Accessibility Guidelines by Novice Evaluators: an Experience Report

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.