Faicey Integration - Modular UI/UX Expression System

Overview

Faicey is now integrated into the mindX agency system, enabling the creation of personalized interface expressions (faces) for agents. Faicey combines prompt, agent, dataset, model, and persona to generate modular, customizable UI/UX systems that adapt to each agent's identity.

What is Faicey?

Faicey is a UI/UX AIML modular response system that creates personalized interface expressions for AI agents. It's conceptualized as a way to give agents a "face" - a modular, customizable interface that reflects their persona and capabilities.

Reference: https://github.com/faicey

Related Projects:

mlodular - Modular machine learning components

AUTOMINDx - Autonomous mind systems

Faicey Principles

User-Friendliness: Intuitive interface with minimal learning curve

Customization Options: Toggle buttons, drag-and-drop modules

Real-Time Feedback: Immediate, actionable responses

Seamless AI Integration: Adaptable to new language models

Modular, Scalable, Fast: Easy module addition/removal

Multi-Modal, Multi-Model: Support various input/output modalities

Integration Architecture

FaiceyAgent

The FaiceyAgent (agents/faicey_agent.py) is responsible for:

Expression Generation: Creates Faicey expressions from agent personas

Module Management: Manages modular UI components

Customization: Handles user customization options

Real-Time Feedback: Configures feedback systems

Expression Emergence

A Faicey expression emerges from the combination of:

Prompt: Agent instructions and system prompts

Agent: Agent identity, capabilities, and configuration

Dataset: Knowledge base and training data information

Model: LLM configuration and model capabilities

Persona: Cognitive identity, behavioral traits, communication style

Default Modules

The system includes default UI modules:

text_input: Text input field for user queries

text_output: Text output area for agent responses

agent_status: Display agent status and metrics

persona_display: Show current persona information

model_info: Display model configuration and capabilities

real_time_feedback: Real-time progress and feedback indicators

API Endpoints

Create Expression

POST /faicey/expressions
Content-Type: application/json

{
  "agent_id": "my_agent",
  "persona_id": "expert_persona_123",
  "prompt": "You are a helpful assistant",
  "agent_config": {...},
  "dataset_info": {...},
  "model_config": {...}
}

List Expressions

GET /faicey/expressions?agent_id=my_agent

Get Expression

GET /faicey/expressions/{expression_id}

Get Expression for Agent

GET /faicey/expressions/agent/{agent_id}

Update Expression

PUT /faicey/expressions/{expression_id}
Content-Type: application/json

{
  "ui_modules": [...],
  "customization_options": {...}
}

Export UI Config

GET /faicey/expressions/{expression_id}/ui-config

Returns a UI configuration object ready for frontend consumption.

Usage Example

from agents.faicey_agent import FaiceyAgent
from agents.memory_agent import MemoryAgent
from agents.persona_agent import PersonaAgent
from utils.config import Config

Initialize
memory_agent = MemoryAgent()
persona_agent = PersonaAgent(agent_id="persona_manager", memory_agent=memory_agent)
faicey_agent = FaiceyAgent(
    agent_id="faicey_agent",
    memory_agent=memory_agent,
    persona_agent=persona_agent
)

Create expression from persona
result = await faicey_agent.create_expression_from_persona(
    agent_id="my_agent",
    persona_id="expert_persona_123",
    prompt="You are an expert assistant",
    agent_config={"capabilities": ["reasoning", "code_generation"]},
    model_config={"provider": "gemini", "model": "gemini-pro"}
)

Get UI configuration for frontend
ui_config = await faicey_agent.export_expression_ui_config(result["expression_id"])

Storage

Faicey expressions are stored in:

Registry: data/faicey/faicey_registry.json

Expressions: data/faicey/expressions/{expression_id}.json

Modules: data/faicey/modules/module_registry.json

Integration with Agency System

Faicey integrates with the mindX agency system through:

PersonaAgent: Provides persona information for expression generation

MemoryAgent: Stores expression metadata and usage statistics

BDIAgent: Inherits BDI capabilities for goal-oriented expression design

Agency Registry: Expressions are linked to agents in the agency registry

Skills and Rendering

Skills System

Faicey expressions include a skills system that tracks agent capabilities:

Capability Skills: Extracted from agent configuration

Expertise Skills: Derived from persona expertise areas

Rendering Skills: Three.js and wireframe rendering capabilities

Model Skills: LLM integration capabilities

Each skill has:

skill_id: Unique identifier

name: Skill name

category: Skill category (capability, expertise, rendering, model)

description: Skill description

level: Proficiency level (1-10)

enabled: Whether the skill is active

config: Skill-specific configuration

Three.js Wireframe Rendering

Faicey includes Three.js integration for 3D wireframe rendering:

Configuration:

Scene setup with fog and background

Perspective camera with configurable FOV

WebGL renderer with antialiasing and shadows

OrbitControls for interactive navigation

Ambient and directional lighting

Wireframe Features:

Line-based wireframe meshes

Vertex visualization

Edge rendering

Customizable colors and line widths

Support for Box, Sphere, Plane, and custom geometries

Usage:

import FaiceyThreeJSRenderer from './components/FaiceyThreeJS';

const renderer = new FaiceyThreeJSRenderer(containerElement, threejsConfig);
// Create wireframe shapes
renderer.createWireframeBox(1, wireframeConfig);
renderer.createWireframeSphere(1, 32, wireframeConfig);
renderer.createWireframePlane(2, 2, wireframeConfig);

Wireframe Configuration:

{
  "enabled": true,
  "line_width": 1,
  "wireframe_color": "#00a8ff",
  "show_vertices": true,
  "show_edges": true,
  "vertex_size": 0.05,
  "material": {
    "type": "LineBasicMaterial",
    "color": "#00a8ff",
    "transparent": true,
    "opacity": 0.8
  }
}

Advanced Three.js Features

Faicey includes support for advanced Three.js examples from the official Three.js documentation:

Decals (webgl_decals)

Project decals onto 3D surfaces:

const decal = renderer.createDecal(
    targetMesh,
    position,
    rotation,
    scale,
    texture
);

Reference: https://threejs.org/examples/#webgl_decals

Bumpmap Materials (webgl_materials_bumpmap)

Bump mapping and normal mapping:

const material = renderer.createBumpmapMaterial(
    texture,
    bumpMap,
    normalMap,
    { bump_scale: 1.0, normal_scale: { x: 1, y: 1 } }
);

Reference: https://threejs.org/examples/#webgl_materials_bumpmap

PCD Point Cloud Loader (webgl_loader_pcd)

Load and render PCD point cloud files:

const pointCloud = await renderer.loadPCD('path/to/file.pcd', {
    point_size: 1.0,
    point_color: '#00a8ff'
});

Reference: https://threejs.org/examples/#webgl_loader_pcd

Fat Wireframe Lines (webgl_lines_fat_wireframe)

Thick wireframe lines:

const fatWireframe = renderer.createFatWireframe(geometry, {
    line_width: 5.0,
    line_color: '#00a8ff'
});

Reference: https://threejs.org/examples/#webgl_lines_fat_wireframe

Wireframe Materials (webgl_materials_wireframe)

Advanced wireframe material rendering:

const wireframeMesh = renderer.createWireframeMeshWithMaterial(geometry, {
    wireframe_color: '#00a8ff',
    wireframe_linewidth: 2
});

Reference: https://threejs.org/examples/#webgl_materials_wireframe

Video/Webcam Materials (webgl_materials_video_webcam)

Video texture and webcam material support:

// Video texture
const videoTexture = renderer.createVideoTexture(videoElement);
const material = renderer.createVideoMaterial(videoTexture);

// Webcam texture
const webcamTexture = await renderer.createWebcamTexture({
    webcam_constraints: { video: true, audio: false }
});

Reference: https://threejs.org/examples/#webgl_materials_video_webcam

WebGPU Morph Targets (webgpu_morphtargets_face)

WebGPU morph targets for facial animation:

const morphMesh = await renderer.createMorphTargetMesh(
    geometry,
    morphTargets,
    { morph_influence: 1.0 }
);

// Update morph target
renderer.updateMorphTargetInfluence(morphMesh, targetIndex, influence);

Reference: https://threejs.org/examples/#webgpu_morphtargets_face

Speech Inflection System

The speech inflection system provides complete facial animation for speaking, listening, and seeing modes using WebGPU morph targets.

Features

Text-to-Speech Animation: Converts text input to phonemes and animates mouth shapes (visemes)

Phoneme-to-Viseme Mapping: Comprehensive mapping from English alphabet and phonemes to visual mouth shapes

Eye Movements: Animated eye tracking, blinking, and directional movements

Eyebrow Expressions: Dynamic eyebrow positions for emotional expression

Ear Animations: Ears perk up during listening mode

Audio Synchronization: Synchronizes morph target animations with audio playback

Multi-Alphabet Support: Extensible system for multiple alphabets and tone systems

Morph Target Categories

Mouth (Visemes) - 20+ targets

Vowels: A, E, I, O, U, AW, AY, EY, OY

Consonants: M/B/P, F/V, TH, L, W/Q, S/Z, SH/ZH, CH/J, R, N/D/T, K/G, H

Silence: SIL (neutral mouth)

Eyes - 9+ targets

Open, Closed, Blink, Look-left, Look-right, Look-up, Look-down, Squint, Wide-open

Eyebrows - 7+ targets

Neutral, Raised, Furrowed, Relaxed, Left-raised, Right-raised, Both-raised

Ears - 4+ targets

Neutral, Perked-up, Slightly-raised, Relaxed

Usage

import FaiceyThreeJSRenderer from './components/FaiceyThreeJS';
import FaiceySpeechInflection from './components/FaiceySpeechInflection';

// Create morph target mesh
const morphMesh = renderer.createMorphTargetMesh(geometry, morphTargets);
// Initialize speech inflection system
const speechSystem = await renderer.initializeSpeechInflection(morphMesh, {
    alphabet: 'english',
    viseme_blend_duration: 0.1,
    eye_blink_interval: 3.0
});
// Speaking mode
await speechSystem.startSpeaking("Hello, I am an AI agent", audioUrl);
// Listening mode
speechSystem.startListening();
// Stop
speechSystem.stopSpeaking();
speechSystem.stopListening();

Configuration

{
  "speech_inflection": {
    "enabled": true,
    "alphabet": "english",
    "tone_system": null,
    "viseme_blend_duration": 0.1,
    "eye_blink_interval": 3.0,
    "listening_ear_animation": true,
    "speaking_eye_tracking": true,
    "features": [
      "text_to_speech_animation",
      "phoneme_to_viseme_mapping",
      "eye_movements",
      "eyebrow_expressions",
      "ear_animations",
      "listening_mode",
      "speaking_mode",
      "audio_synchronization"
    ]
  }
}

Animation Modes

Speaking Mode

Text input → Phoneme extraction

Phoneme → Viseme mapping

Generate animation timeline

Animate mouth shapes synchronized with audio

Eye movements follow speech rhythm

Subtle eyebrow movements for expression

Listening Mode

Ears perk up (morph target animation)

Eyes focus forward (attention)

Eyebrows slightly raised (engagement)

Return to neutral when listening ends

Idle Mode

Slow breathing animation

Random blinks

Subtle eye movements

Neutral expressions

Phoneme-to-Viseme Mapping

The system uses comprehensive phoneme-to-viseme mappings stored in data/faicey/phoneme_viseme_map.json:

English Alphabet: Full mapping of vowels and consonants to visemes

Multi-Alphabet Support: Extensible for other languages

Tone Systems: Support for tonal languages (Chinese, Thai, Vietnamese)

Blending Rules: Smooth transitions between visemes

Morph Target Definitions

Morph targets are defined in data/faicey/morph_target_definitions.json:

Influence Ranges: Each target has valid influence range [0.0, 1.0]

Blending Rules: Rules for combining multiple morph targets

Animation Timing: Transition durations and timing parameters

Future Enhancements

Frontend UI components for rendering Faicey expressions

Integration with mlodular for enhanced ML capabilities

Multi-modal input/output support (voice, images, etc.)

Dynamic module loading and plugin system

Real-time expression updates based on agent state changes

Advanced 3D visualizations for agent knowledge graphs

Interactive wireframe manipulation tools

API Endpoints for Speech Inflection

Start Speaking

POST /faicey/speech/speak
Content-Type: application/json

{
  "text": "Hello, I am an AI agent",
  "audio_url": "https://example.com/audio.mp3",
  "alphabet": "english",
  "tone_system": null
}

Start Listening

POST /faicey/speech/listen

Activates listening mode with ear and eye animations.

Stop Speech/Listening

POST /faicey/speech/stop

Returns to idle mode.

Data Files

Phoneme-to-Viseme Mapping

File: data/faicey/phoneme_viseme_map.json

Contains comprehensive mappings:

English alphabet phonemes to visemes

Multi-alphabet support structure

Tone system definitions (Chinese, Thai, Vietnamese)

Viseme blending rules

Word boundary handling

Morph Target Definitions

File: data/faicey/morph_target_definitions.json

Contains:

Complete morph target definitions for mouth, eyes, eyebrows, ears

Influence ranges and default values

Blending rules for combining targets

Animation timing parameters

Implementation Details

Text-to-Phoneme Conversion

The system converts text to phonemes using:

Character-by-character analysis

Two-character phoneme detection (TH, SH, CH, etc.)

Vowel and consonant classification

Space/punctuation handling (silence)

Phoneme-to-Viseme Mapping

Each phoneme maps to a viseme (visual mouth shape):

Vowels → Open mouth shapes (A, E, I, O, U)

Consonants → Specific mouth/tongue positions

Silence → Neutral closed mouth

Animation Timeline

The system generates an animation timeline with:

Viseme events with timing

Eye movement events

Blending between visemes

Audio synchronization points

Real-Time Animation

The animation loop:

Updates current time based on audio playback

Finds current viseme from timeline

Blends between current and next viseme

Updates eye movements and blinking

Handles mode transitions

References

Faicey GitHub

mlodular GitHub

AUTOMINDx GitHub

Three.js WebGPU Morph Targets

---