LinguaLens
A personal experiment.
Point your camera, get a live translation.
LinguaLens teaches vocabulary through your phone’s camera. Point it at everyday objects (a cup, a chair, a phone) and it draws a box around each one and shows its name in French, Spanish, German, Japanese or Italian. Tap a word to hear it spoken.
Words you discover are collected in a vocabulary panel, and a quiz mode hides the labels until you tap them, so you can test yourself.
- 01
The camera feeds a model in your browser
TensorFlow.js runs a pre-trained object-detection model (COCO-SSD) directly on your device. Camera frames are not uploaded anywhere.
- 02
Detections are cleaned up
Overlapping boxes are filtered, object proportions are checked, and a label only appears after the same object is seen several times, which stops flickering and many wrong guesses.
- 03
Words come from a dictionary
The detected object name is looked up in the app’s own dictionary of translations for the five languages.
- 04
Hear it, save it, quiz yourself
The browser’s speech synthesis pronounces the word, and what you have discovered is kept in local storage. A snapshot button saves the camera view with its boxes.
A hosted copy needs HTTPS for camera access, and the first load needs internet to download the model.