Effective Computer Access with Plan-Based Intelligent Screen Reader

Abstract

The intelligent screen reader system observes a user¡¯s keystrokes when he or she is performing tasks, and infers an underlying plan structure. The system then generates an optimized script that allows the user to perform the same task more efficiently in the future. The intelligent screen reader system is particularly beneficial for visually impaired computer users.The JAWS for Windows screen reader was chosen for this prototype, since the JAWS scripting language provides flexibility to users for customizing their tasks within a specific application. The JAWS macro recorder, developed by Freedom Scientific Inc., records all the user¡¯s actions. Automated Synthesis of Plan Recognition Networks (ASPRN), developed by Intelligent Reasoning Systems Inc., takes a plan or a script representation as input and outputs a specially-constructed belief network that supports plan recognition. The Script Generation Interface, developed by the University of Pittsburgh, allows users to modify the actions recorded by the JAWS macro recorder and generate an optimized script. Development of the SGI involved integrating the SGI with the JAWS macro recorder, integrating the SGI with ASPRN, and implementing the SGI user interface.Preliminary usability testing involving the intelligent screen reader system was conducted with visually impaired users. Within limited actions and tasks, the results are satisfactory. Users felt the SGI was easy to learn and operate, and that it was efficient to create useful scripts with the intelligent screen reader system. Future work includes adding more plans to the plan library and creating a tutorial system.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 0 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.