Crowdsourcing a mixed systematic review on a complex topic and a heterogeneous population: Lessons learned

Abstract

Systematic reviews (SR) typically focus on well-defined topics and homogeneous populations. In contrast, conducting a SR on patients with complex care needs (PCCN) presents two main challenges: a novel ill-defined topic and a heterogeneous population. This commentary summarizes the lessons learned from our experience using crowdsourcing to meet these challenges. PCCN often suffer from combinations of multiple issues (medical, interactional, sociocultural, psychological, related to healthcare services use), which can lead to difficult decision-making involving different stakeholders (themselves, caregivers, practitioners). The objective of our SR was to systematically assess decisional needs of PCCN reported in the literature. With the collaboration of librarians, we identified 8616 potentially relevant studies in five databases. A team of 20 crowdreviewers were trained and participated in the selection of 156 relevant studies, using specialised online software. The main benefit of crowdsourcing was the diversity of crowdreviewers’ viewpoints which helped us establish an intersubjective knowledge classification of idiosyncratic concepts related to PCCN and their decisional needs. In line with other crowdsourced reviews, our experience confirms that crowdsourcing can be useful in SR with a large number of studies on ill-defined domains and with heterogeneous populations.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 12 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.