Screen Recognition: Creating Accessibility Metadata for Mobile Applications using View Type Detection

Abstract

Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sufficient metadata for accessibility features to work as expected. In this paper, we utilize the relevant data information that can be obtained from the terminal to detect the type of each View in the display interface, and we can also perform finer division based on this information. In order to achieve this method, we collected the feature information of View under multiple scenarios in the normal use of the three mainstream and representative apps, including Today's Headlines, Tiktok and WeChat, and manually calibrated the type of View to establish a database containing more than 40000 pieces of data. By utilizing the layout features, structural features, and attribute features of View to learn and predict the type of View, a detection system has been implemented that can accurately predict the type of unknown View. This system can accurately predict the type of an unknown View, with a prediction accuracy of over 95% for known apps and relatively accurate predictions for unknown apps.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 3 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.