Automatic rating of therapist facilitative interpersonal skills in text: A natural language processing application

Abstract

Background: While message-based therapy has been shown to be effective in treating a range of mood disorders, it is critical to ensure that providers are meeting a consistently high standard of care over this medium. One recently developed measure of messaging quality-The Facilitative Interpersonal Skills Task for Text (FIS-T)-provides estimates of therapists' demonstrated ability to convey psychotherapy's common factors (e.g., hopefulness, warmth, persuasiveness) over text. However, the FIS-T's scoring procedure relies on trained human coders to manually code responses, thereby rendering the FIS-T an unscalable quality control tool for large messaging therapy platforms. Objective: In the present study, researchers developed two algorithms to automatically score therapist performance on the FIS-T task. Methods: The FIS-T was administered to 978 messaging therapists, whose responses were then manually scored by a trained team of raters. Two machine learning algorithms were then trained on task-taker messages and coder scores: a support vector regressor (SVR) and a transformer-based neural network (DistilBERT). Results: = 0.588) in human-derived ratings and realized a prediction mean absolute error of 0.134 on a 1-5 scale. Conclusions: Algorithms can be effectively used to ensure that digital providers meet a consistently high standard of interactions in the course of messaging therapy. Natural language processing can be applied to develop new quality assurance systems in message-based digital psychotherapy.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 5 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.