Augmented reality for the blind - hardware and software combination

This is the background, or outline page relating to the audio demonstration I initially recorded for the blind android users podcast episode 276 podcast - here's the youtube streaming direct section link, and you can check out my own posting thereof directly via my own youtube channel - @BlindZA - Augmented Reality for the Blind? There's an app for that: The vOICe for Android!

Overall, this relates to my relatively long-term usage of the vOICe for android, but the majority of the content here specifically relates to making use of an external camera - in this context, specifically a pair of camera glasses that I acquired via amazon.com - but you should be able to set up a similar form of functionality using any camera that counts as a driverless USB webcam - more on that lower down.

For a more detailed outline of the software and how to interact with it, you can find a full explanation of the functionality it offers via the details page - Augmented Reality for the Blind? There's an app for that - on the seeingWithSound.com project page.

In terms of basic self-training processes, and teaching yourself how to interpret and work with the vOICe in different contexts, you can also check out the self-training guidelines Peter Meijer has compiled - Learn to see - The vOICe Training Manual - would recommend scanning through this but then focus on some of the basics before writing it off as too involved/advanced.

In terms of real-world usage and the benefits it can provide to a 100% blind individual, here are a couple of relatively simple examples of some of my own explanations of forms of virtual sight-seeing I carried out using it:

• Coastal sight-seeing
• Festive season exhibition

For a variety of even more diverse contexts that then also include working with the vOICe at times, you can also check out the specific category of blog posts by a long-term user of the vOICe - Pranav Lal: TechEsoterica.com - Sensory Substitution - the vOICe

In terms of working with the android version of the software, you can use the built-in camera of your android device, and my most common usage for the last few years in the outside world has been working with a smaller android handset, sort of wearing it strapped to the back of my left hand.

This does, again though, bring up the same-old, same-old point that it's not the best thing for a blind/VI individual to use a hand to carry around their phone, or even if they wore it on a lanyard around their necks, this is not going to be the safest/most convenient arrangement in all environments/locations.

Peter Meijer has however also implemented/included the ability to make use of external cameras via a specific form of interface - if you go into options, camera, at the bottom of that screen, there's an edit field to allow you to provide the http streaming URL for an alternative camera source, and it's sort of based on the principle of real-time streaming protocol.

Hence, with help from sources on the seeingWithSound mailing list, I sourced the following camera glasses off amazon.com, and these are what made use of during the small form of demonstration in the audio recording.

Using the model number off amazon, I found this online manual, which is where was able to read up about changing modes, etc. - to confirm, I used beMyEyes A.I. text recognition to scan a page or two of the pamphlet that was inside the box, and I recognised some specific phrase combinations, so am 99% sure it's the correct manual, although graphical diagrams, etc. won't be of much use to me directly: https://m.media-amazon.com/images/I/71FHHHblBZL.pdf.

Otherwise, if a person can't immediately source glasses with an embedded camera, then at the very least a relatively small/inconspicuous webcam might also be feasible, as long as it offered something along the lines of 1080p full-high definition (FHD) clarity/resolution video quality, a decent field-of-view and focal-length, and it must be prepared/able to act as a standard driverless webcam in terms of streaming camera input frame-by-frame directly.

For example, here's another camera I sourced for testing and trying out this setup, and it is physically roundabout 2cm by 2cm in terms of height and width, and maybe 5cm long in terms of length of the camera itself, and has a 2 meter long USB lead hard-wired to the camera, with a standard USB-A adapter on the other end - Mini security USB camera. So, while the lead might be slightly thicker than you'd prefer since it is more or less a standard USB lead, this is somewhat cheaper and you could attach it to your own set of glasses, mount it on a form of headgear that suited you, and so on.

Any standard USB-A lead used to connect these forms of cameras to your android handset will most likely require a form of on-the-go adapter which, specifically, offers data transmission since most of them that you find off-the-shelf are purely designed to work as forms of power-transmission adapters, so make sure you acquire a full-on data transmission OTG adapter.

I am also looking into sourcing something like a group of young people who are interested in electronics and/or robotics, and they might be able to source standard USB webcams with a relatively small physical form-factor, and strip them down and re-package them for this form of usage by blind/VI individuals, for a smaller price and further convenience in terms of local sourcing.

In terms of it rendering the camera view/input to the vOICe, since that requires a live streaming URL, it seems like the only current pieces of software that can offer this natively on the android platform are either USB camera (free), or USB camera (pro) - the pro version is still not perfect in terms of UI-accessibility, but it's quite a bit cleaner, and what they both offer is the ability to fire up automatically and provide a local streaming URL where the vOICe can pick up live camera input, and you should also be able to just make use of that URL in something like chrome web browser to have camera input rendering live on-screen, or else some other endoscope/inspection camera software might also help with that.

In terms of configuration specific to either version of that software, the only options can really specifically recommend you look into, for now, are to go into their settings via the more options button, settings and scroll through items to make sure MJPG bypass..." option is enabled, make sure "Save latest device settings" option is enabled, and in initial list, go into user and password options and make sure they are both empty fields - your camera will only be available to the device you connect it to so you don't need to enable authentication and this relates to the streaming URL you will make use of in the vOICe itself.

Another setting that is probably enabled by default, but which you might want to confirm is under settings, select general, and firstly make sure "start IP camera after connection" option is toggled on - think should be by default.

Besides all of that, once I connect either of my current USB camera hardware devices, this software now comes up, and if it was a new device, it might ask me to select it first, before then running as-is, and I just hit the back button, select the option to have it run in the background, and it reverts to the notification shade where I can then bring it up again and exit it for real when I am done with making use of it's functionality.

Otherwise, in terms of the slightly convoluted interface of this streaming software, which you will only really need to work with occasionally, I have also gotten another android developer who themselves are visually-impaired looking into producing a simpler/cleaner bit of software to implement the real-time streaming process in such a way that blind/VI users could work with it more easily.

Besides just getting the camera to make itself available as a visual input source for the vOICe, in the demo you will also hear me making use of both Google accessibility suite's screenreader software and google gemini, which will only really run on slightly newer handsets, and this relates to the fact that if you switch to live mode inside it, you can then turn on live screen sharing, where gemini can then interact with you via voice conversation relating to what's appearing on-screen - which literally lets a blind/VI individual ask gemini additional questions about their surroundings, while the vOICe specifically allows blind/VI individuals to work with what I refer to as virtual vision by seeing-with-sound.

What I will generally do is connect the camera glasses to my handset via the USB lead, and the USB camera software is assigned as default software for the event triggered by attaching the external camera, and will thus automatically launch, when I then dismiss it to run in the background. I then bring up the vOICe, and under it's options, camera category, I have it set to the 4th option, marked as IP camera, and below that I set the text value of the streaming URL to http://localhost:8081/video and that then tells it to make use of the input from the external camera being streamed via the third-party software. This setting will carry across sessions so you do not need to set it each time you bring up the vOICe.

In terms of working with the google version of accessibility suite/talkback on a Samsung handset, which relates to one little bit of additional functionality I work with at times, john dyer of blindAndroidUsers recorded the following bit of a tutorial - Google TalkBack Install and Setup on a Samsung Phone or Tablet (2025 Version) - or else, the people at accessibleAndroid also have a textual page relating to this software replacement - Uninstall Samsung TalkBack and install Google TalkBack - up to you.

You can just use the vOICe as-is on it's own, but if you wish to make use of talkback's ability to describe screen contents, you will need to activate camera colour-preview mode to make the vOICe render the camera snapshots on-screen in a live/realistic mode frame-by-frame.

The way you do this is that, once the vOICe is running, you would be required to do a long swipe left-to-right, and if talkback screen reader is actively running you would need to perform this gesture with two fingers instead of one, but I myself have assigned a four-finger double-tap-and-hold gesture to ask talkback to then ignore, or pass-through the next gesture to the system, and this allows me to perform that gesture and then just do a long swipe left-to-right using a single finger, when the vOICe will then announce that change of camera colour-preview status.

Another thing Peter Meijer has done is that, if the vOICe picks up that you are running gemini live screen-sharing mode, then it will activate camera colour-preview mode automatically.

In terms of activating live screen sharing quickly and easily you can place the gemini-specific widget on your home screen to immediately launch things like live screen-sharing mode, with no other interaction besides permission approval required.

I can then listen to the vOICe via a headset/earphones, interpreting it's output as virtual-vision myself, and if I think there's something of interest or would like additional confirmation or details, I have set macro droid - device automation software to react to me then shaking the handset, and it virtually triggers the talkback gesture I have assigned to have it then perform a full-screen description, pause 8 seconds so it will be read out to me, and then dismisses it by performing activation of the back system-activity.

In terms of permissions and/or allowing it to run as an accessibility service, macro droid will require you having allowed both it's primary service labeled as just macro droid, and it's macro droid UI interaction service to be enabled in order for it to be able to perform virtual gestures - you will find those under system settings, accessibility, installed apps.

That macro droid macro which includes a constraint that the vOICe must be running in the foreground, and that I have already assigned the two-part single finger talkback gesture of swipe left and flick up to then describe on-screen contents, and which is triggered by me just shaking the handset can be downloaded/imported from here - double-check it's contents first if you are not sure: vOICe_Describe.macro

If you have otherwise activated gemini live screen-sharing, you can just talk to gemini and ask it about what's now appearing/changing on-screen interactively - the only real issues with that are that gemini live mode will only be available on newer devices, and, yes, gemini's recognition engines run in the cloud, so you will, at times, experience a bit of lag, but not too bad at all most of the time.

Lastly, apparently the glasses look modern, stylish, and maybe slightly cyber-punkish, but they are actually not too heavy, are definitely comfortable, and feel pretty well-made - I am happy all-round - thanks, Peter for all your help in figuring out this combination!