EMX (Emotional Matrix) is an expressive multimodal interface for generating lifelike conversational behavior and facial expressions for humanoid robots. Built during my time at K-Scale Labs, EMX orchestrates real-time perception and generation across audio, visual, and behavioral modalities.

The system integrates Google MediaPipe for facial and hand landmark extraction, enabling robots to interpret user gestures and expressions in real time. For affective understanding, it incorporates Hume AI and emotion2vec to derive fine-grained emotional embeddings from audio, allowing robots to infer user intent and modulate expressions accordingly.

EMX uses OpenAI's Realtime Voice API to generate low-latency audio responses, enabling natural, interruption-capable dialogue. The entire pipeline runs with minimal latency, making it suitable for real-time human-robot interaction scenarios.