Semantic Search

Semantic search is a way of searching that understands what you're describing in a sentence and finds matching content, even when videos or documents have no tags or titles pre-assigned by humans.

Traditional search is like a librarian who finds books only by looking at titles and keywords written on index cards. If the word "stairs" isn't written on the card, a scene showing stairs can never be found. Semantic search is like a librarian who has actually watched and listened to the video itself. Even if you describe it in a sentence — stairs, people running, a goal shot from left to right in the rain — it looks across the visuals, dialogue, and background sounds to pinpoint the moment that matches the meaning.

The key is that it doesn't match words literally one by one; instead, it first grasps the intent of the question, then figures out on its own where in the content the relevant clues are. This means that even with vast amounts of untagged material, a single sentence can pinpoint the desired scene, and when combined with existing tag-based search, it can narrow down the target first for faster, more accurate results.

Example from an article: Semantic search, powered by Twelve Labs' multimodal embedding model Marengo, can find scenes in a broadcast archive using nothing but sentence queries like "stairs," "people running," or "voting" — with zero human-made tags. Common misconception corrected: unlike keyword search, it doesn't require the exact same words to appear in the material; it can find matches even when the wording differs, as long as the meaning aligns.

You can get a feel for it by packing a complex condition into a single sentence search. Example prompt: "A goal shot from left to right in the rain by a player in a white shirt against a team in yellow." Try mixing people, setting, action, and situation into one sentence like this, and compare whether results come up even without tags — that's where the difference from traditional keyword search shows.