Computer Vision and Vision-Language Models

At the core of Orn’s intelligence stack lies the integration of Computer Vision (CV) Models and Vision-Language Models (VLMs). These two components work in tandem: CV models operate at the pixel level, extracting objects, motions, and environmental details, while VLMs interpret these outputs into structured narratives that capture meaning, intent, and task compliance. Together, they enable Orn to move from simple detection to true understanding and evaluation of user-submitted videos.

Computer Vision Models form the perception backbone of Orn. They specialize in identifying and localizing objects, tracking hand and body poses, and segmenting actions within video frames. CV models confirm that a video was recorded in first-person perspective by analyzing camera motion and hand visibility, ensuring it reflects authentic egocentric capture. They also detect the key items involved in a task — knives, fruit, golf clubs, wheelchairs — and track their interaction with the user over time. Beyond objects, CV models map out the temporal structure of each submission, splitting it into start, middle, and end phases so that Orn can confirm the task was performed in full. These capabilities provide the raw ingredients for higher-order interpretation.

Vision-Language Models build on this perception layer by aligning visual features with natural-language representations. Instead of simply detecting that a knife and an apple are present, VLMs describe the sequence of events: “The user picks up a knife, slices the apple into pieces, and places them into a bowl.” By translating visual activity into text-like outputs, VLMs give Orn the ability to assess tasks semantically — was the correct sequence followed, were all required steps completed, and does the outcome match expectations?

This dual-layer approach powers Orn’s Task Scoring. Submissions are not judged arbitrarily but benchmarked against reference libraries of expert demonstrations curated for each activity type. In a sushi-making task, for example, CV models detect the rice, seaweed, and rolling action, while VLMs generate a narrative of the process. Orn then compares this narrative against the professional gold standard: did the user place the rice correctly, roll tightly, and cut evenly? The closer a submission aligns to expert patterns, the higher its score. This ensures that evaluation reflects how well a task was performed, not just whether it was attempted.

These capabilities also feed directly into Skill Trees. CV models classify tasks into domains such as kitchen, sports, or outdoor activities, while VLMs enrich them with semantic attributes like dexterity, endurance, or creativity. Together, they build multidimensional profiles of user ability, mapping both what types of tasks a contributor can perform and how effectively they execute them. This data powers task allocation, gating, and progression within EgoPlay.

Finally, CV and VLM integration enhances human-in-the-loop review and dataset quality. Reviewers no longer need to watch full clips; they can validate AI-generated summaries and labels, dramatically reducing overhead. Meanwhile, downstream consumers such as robotics companies benefit from datasets that contain both precise object annotations and rich narrative descriptions, providing a more complete training signal for embodied AI systems.

By combining Computer Vision and Vision-Language Models, Orn creates a pipeline that is both perceptive and interpretive. CV models provide the eyes, detecting the fine-grained details of user activity, while VLMs provide the language, translating these details into structured, meaningful evaluations. This synergy ensures that every submission is not only labeled but also understood, scored, and contextualized — forming the intelligence core of Orn’s architecture.

All thresholds, parameters, and detection methods described are subject to continuous refinement as technology advances and as the requirements of the ecosystem evolve.