Content
# Visionbridge AI Glasses
Visionbridge AI Glasses is an ESP32-based firmware project for wearable AI hardware. It turns a small camera, microphone, speaker, display, and wireless module into a voice-driven assistant that can see, listen, speak, and connect to cloud AI services.
The project is designed for experimental AI glasses and compact embedded assistants. It combines real-time audio interaction, camera capture, image explanation, device-side MCP tools, and support for multiple ESP32 board families.
## What It Does
- Captures voice input from an onboard microphone.
- Streams audio through WebSocket or MQTT/UDP communication paths.
- Supports wake-word and audio processing with ESP-SR.
- Encodes and decodes voice data with OPUS.
- Captures images from supported ESP32 camera boards.
- Sends images to a vision endpoint for explanation or question answering.
- Displays status, text, and emoji feedback on OLED or LCD screens.
- Supports battery reporting and power management on compatible boards.
- Exposes device controls through MCP tools, including speaker, LED, GPIO, servo, and camera actions.
- Supports Wi-Fi and selected cellular module workflows.
## Why This Exists
Visionbridge is a bridge between physical vision and AI reasoning. The goal is to make a small wearable device that can answer questions about the environment, respond conversationally, and control local hardware without needing a phone screen as the main interface.
For AI glasses, the most important flows are:
1. Wake or activate the assistant.
2. Ask a voice question.
3. Capture a photo when visual context is needed.
4. Send the image and question to an AI vision service.
5. Speak or display the response.
## Project Structure
```text
main/
application.* Main device application flow
mcp_server.* Device-side MCP tools and camera actions
protocols/ WebSocket, MQTT, and shared protocol logic
audio/ Audio codecs, processing, and wake-word support
display/ OLED, LCD, LVGL, image, GIF, and emoji display code
led/ LED drivers and effects
boards/ Board-specific pin maps and hardware drivers
docs/
websocket.md WebSocket protocol notes
mqtt-udp.md MQTT and UDP protocol notes
mcp-protocol.md MCP protocol interaction flow
mcp-usage.md MCP device-control examples
partitions/
v1/, v2/ ESP-IDF partition tables
scripts/
build helpers, audio tools, image converters, and asset utilities
```
## Hardware Targets
The firmware supports many ESP32-based boards and can be adapted to custom AI glasses hardware. The most relevant targets for glasses-style builds are ESP32-S3 boards with:
- Camera support
- PSRAM
- Microphone input
- Speaker or amplifier output
- Wi-Fi
- Optional display
- Optional battery management
Camera-capable board examples in this tree include LilyGO T-CameraPlus S3, M5Stack CoreS3, SenseCAP Watcher, ESP32-S3 camera boards, and several Waveshare or custom S3 boards.
For a custom glasses PCB, start by copying the closest board under `main/boards/`, then update:
- `config.h` for GPIO assignments
- `config.json` for board metadata
- Board initialization code for camera, audio, display, and power rails
- `main/Kconfig.projbuild` if a new board selection option is needed
- `main/CMakeLists.txt` if a new board source folder is added
## Software Requirements
- ESP-IDF 5.4 or newer
- Git
- Python dependencies required by ESP-IDF
- A supported ESP32 target, usually `esp32s3` for camera-based builds
Optional tools:
- VS Code or Cursor with the ESP-IDF extension
- Serial monitor for provisioning and debugging
- Audio/image conversion scripts from `scripts/`
## Build
Set up ESP-IDF first, then build from the repository root.
```bash
idf.py set-target esp32s3
idf.py menuconfig
idf.py build
```
In `menuconfig`, choose the board type, language, protocol settings, asset settings, and any hardware-specific options needed for your build.
## Flash
Connect the device over USB, then flash and monitor:
```bash
idf.py -p COMx flash monitor
```
On Linux or macOS, replace `COMx` with the serial device, for example:
```bash
idf.py -p /dev/ttyUSB0 flash monitor
```
## Configuration Notes
The firmware stores runtime settings such as Wi-Fi credentials, protocol tokens, server addresses, and vision endpoint data through the device settings flow. Avoid committing personal `sdkconfig`, generated build files, local `.env` files, or private credentials.
The repository intentionally ignores common local artifacts:
- `build/`
- `managed_components/`
- `sdkconfig`
- `sdkconfig.old`
- `.env`
- `.vscode/`
- `.DS_Store`
- `._*`
## Vision Flow
The camera flow is implemented around the device-side MCP server:
- `main/mcp_server.cc` registers camera tools.
- `main/boards/common/camera.h` defines the camera interface.
- `main/boards/common/esp32_camera.*` implements ESP32 camera capture and image explanation.
- `main/boards/sensecap-watcher/sscma_camera.*` handles the SenseCAP camera path.
When a vision endpoint is configured, the assistant can capture a frame, attach the user question, and send it to the server for explanation.
## Protocols
Visionbridge supports two main communication modes:
- WebSocket for realtime assistant sessions.
- MQTT plus UDP for message signaling and audio transport.
See:
- [WebSocket protocol](docs/websocket.md)
- [MQTT/UDP protocol](docs/mqtt-udp.md)
- [MCP protocol](docs/mcp-protocol.md)
- [MCP usage examples](docs/mcp-usage.md)
## Security And Privacy
AI glasses can capture sensitive audio and images. Treat this firmware as a development platform and review these areas before public deployment:
- Do not hardcode production secrets in firmware.
- Do not log Wi-Fi passwords or private tokens in production builds.
- Use HTTPS or another protected transport for image explanation endpoints.
- Make recording and camera capture behavior obvious to the wearer and nearby people.
- Store only the minimum data needed for operation.
## Development Tips
- Start with a known working board configuration before adding custom hardware.
- Keep `sdkconfig.defaults` under version control, but keep generated `sdkconfig` local.
- Rebuild after changing board selections or target chips.
- Use serial logs to verify camera, display, audio codec, and network initialization.
- Keep board-specific code isolated under `main/boards/<board-name>/`.
## License
This project is released under the MIT License. See [LICENSE](LICENSE).
## Acknowledgements
Visionbridge AI Glasses builds on the ESP32 embedded AI ecosystem, including ESP-IDF, ESP-SR, LVGL, ESP camera support, OPUS audio components, and the XiaoZhi ESP32 project structure.
