Wearable guidance for using everyday touchscreens independently.
Our team studied public material from Molly Burke, a content creator, author, speaker, and disability advocate who shares her experiences of blindness. We reviewed her website, talks, podcasts, and more than 30 social media posts to understand the accessibility barriers she describes in everyday life.
This analysis draws on published accounts rather than a direct interview. We used those accounts to identify recurring needs and review where existing tools help or leave gaps.
A recurring theme in our analysis was that a task becomes harder when a product, service, or environment assumes everyone can see. An accessible menu or tactile product label can make the same task possible without another person’s help.
Molly’s goals include working, shopping, traveling, exercising, and getting ready on her own terms. As a creator, she also needs to use social platforms, collaborate with brands, and attend events.
Across the material we reviewed, accessible design supported both practical independence and participation in the activities she enjoys. Frustration often arose when an otherwise manageable task required assistance because information or controls were presented only visually.
Flat controls on payment terminals, treadmills, elevators, and appliances offer few tactile cues. Without accessible audio or physical feedback, an everyday task can require another person’s help.
User need Know which control is selected and receive clear confirmation of an action.
The research describes rideshare refusals when traveling with a guide dog. Reporting an incident afterward does not resolve the immediate disruption of being unable to reach a destination.
User need Travel with a mobility aid without sacrificing independence or relying on a sighted companion.
Finding objects and understanding a new space can require extra assistance. When a guide dog is unavailable, familiar routines become harder to navigate.
User need Receive useful environmental information that complements existing mobility tools.
Molly uses VoiceOver and keyboard shortcuts for work and everyday tasks. Unlabeled controls and inaccessible apps or websites interrupt that workflow, even when she knows her assistive tools well.
User need Access information and controls through the tools she already uses.
Inconsistent tactile labels make it harder to distinguish products, read menus, and organize belongings. Memorizing locations works only while items stay where expected.
User need Identify items and information without repeatedly asking another person or relying only on memory.
We reviewed tools addressing payment feedback, transportation, visual assistance, digital access, and product identification. These comparisons summarize the team’s research, rather than hands-on usability testing.
Across the five needs, the shared problem was extra effort caused by inaccessible information, inconsistent feedback, and dependence on someone else’s availability.
Provide reliable information and clear feedback that work with existing assistive tools, reduce extra effort, and give people more control over everyday tasks.
We narrowed our project pitch to one need: operating flat touchscreens that lack tactile or audio feedback. The goal is to help a user locate a control, deliberately press it, and understand whether the action worked.
These fictional profiles and storyboards explore how the concept might fit different routines, abilities, and environments. They are design assumptions, not interview participants or evidence of tested outcomes.
Content creator & accessibility advocate
At the gymMaya is blind and uses a screen reader, voice commands, and a white cane. She enjoys working out, but flat treadmill controls make starting or adjusting a session dependent on someone else.
She/her · New York City · B.A. Communications · Middle class. Blind due to retinitis pigmentosa.
Find the treadmill controls, start a workout, and adjust settings without repeatedly asking gym staff for help.
Flat touchscreens offer neither tactile boundaries nor audio confirmation. Asking for help works, but limits when and how she can exercise independently.
Context, barriers, and how the proposed system changes the situation.
Maya reaches a treadmill. Its flat screen gives her no tactile landmarks.
She cannot tell where Start is or whether a press registered.
The proposed ring and camera provide guidance so she can operate the controls herself.
Start a workout and adjust the intended settings independently.
LIMITS & ASSUMPTIONSRequires a clear camera view. RingSense would not replace the treadmill’s emergency stop controls.
Fictional profile informed by the team’s secondary research on inaccessible touchscreens.
Our pitch proposes RingSense: a haptic ring, a camera on compatible smart glasses or a chest-mounted phone, and a smartphone app. In the revised proposal, the default interaction is automated: computer vision and optical character recognition (OCR) in the app would interpret a supported touchscreen and track the fingertip. The ring would deliver haptic cues, and the app would speak through text-to-speech. A live human assistant would be an optional, explicitly requested fallback.
Computer vision, text recognition, and finger tracking would identify the screen state, target control, and fingertip position.
Bluetooth commands would trigger directional vibrations, proximity pulses, and an alignment cue.
After the user physically presses the screen, the app would check its response and announce the updated state.
RingSense proposes an AI-assisted perception system with rule-based guidance. The camera supplies images; the smartphone app interprets them and decides which cues to send; the ring vibrates. The ring does not capture images or understand the interface itself.
A camera on compatible glasses or a chest-mounted phone would provide a view of both. Computer vision would locate the screen, track the fingertip, and estimate their positions in the same screen coordinate system.
The initial scope would use a small set of prepared interface layouts with labeled controls and known screen states. OCR would read visible text and values, such as the current treadmill speed. Unrecognized layouts or states would require setup or assistance rather than an assumed button location.
Speech recognition would turn a request into text. For the initial prototype, supported commands would map to predefined tasks and labeled controls. A large language model (LLM) is not required for this baseline. An LLM could later help interpret more varied requests, but that is an open design option, not a chosen or implemented component.
The app would compare the tracked fingertip with the target’s bounds to calculate direction and proximity. A rule-based mapping would translate those measurements into Bluetooth vibration commands. The ring would indicate a proposed direction, pulse faster near the target, and signal steady alignment.
The user would press the touchscreen. The app would inspect subsequent frames for the expected visible change, such as a new speed value, and use text-to-speech to report what it detected. If the screen were obscured or the change could not be verified, it would report uncertainty instead of announcing success.
Routine instructions and confirmations would be generated by the app and spoken using text-to-speech. No remote person would watch the camera feed or provide the normal guidance.
The user could request a live assistant if automated guidance could not continue. The app would announce the switch, ask permission to share the camera view, and identify the live voice as a person. Automated directional cues would pause during the handoff. A service integration remains proposed.
Microsoft’s Seeing AI demonstrates camera-based text recognition and spoken presentation. RingSense would draw on that interaction pattern to read control labels and screen values. We are not proposing to copy its code or assume access to its recognition models. Microsoft: Seeing AI
VizLens combines crowd-labeled interface images with computer vision to describe controls beneath a user’s finger. RingSense would draw on the labeled-layout and fingertip-relative feedback approach, then explore directional ring cues and post-press state confirmation. Its initial prepared layouts would not require live crowd workers during each interaction; how new layouts are labeled remains an open question. VizLens research paper
These are proposed adaptations to evaluate. No recognition model, LLM provider, human-assistance partnership, or working end-to-end implementation is established yet.
The pitch uses a goal of setting a treadmill to 3.5 mph to illustrate the interaction. This walkthrough illustrates the proposed interaction flow.
The user asks to set the treadmill speed to 3.5 mph. The app would match a prepared treadmill layout, read the current speed with OCR, and map the supported request to the labeled speed control.
Directional vibrations would suggest where to move the finger. Pulses would speed up nearer the target and slow down when moving away.
Two pulses would signal steady alignment, accompanied by the app’s synthetic voice saying “aligned with speed up.”
The user would make the physical press. The app would use OCR to check the displayed speed after the press. Its synthetic voice would announce the detected value, or say it could not verify the change.
A ring button would repeat or pause instructions. If tracking were lost, guidance would pause and announce that the finger could not be located. If requested, a separate, announced handoff would connect a live human assistant after permission to share the camera view.
| State | Feedback |
|---|---|
| Direction | Vibration on the corresponding side of the ring. |
| Approaching the target | Progressively faster pulses. |
| Moving away | Slower pulses, with spoken guidance available. |
| Steady alignment | Two pulses and an alignment announcement. |
| After a press | Text-to-speech confirmation only after the expected visible change is detected; otherwise report uncertainty. |
| Finger not detected | Pause guidance and announce the tracking loss. |
The intent is to make movement guidance available through touch while using speech for context and confirmation. The cues still need to be evaluated for comfort, distinguishability, and accuracy.
The proposed mobile app covers three areas: hardware connectivity, an active guidance view with finger tracking, and customizable device settings. The pitch calls for VoiceOver and TalkBack support and spoken output through the phone or compatible glasses.
A connection to a service such as Be My Eyes is a proposed integration, not an existing RingSense partnership.
Prototype coming soon. Still exploring.