How I Built a 14B Parameter Avatar System That Runs at 20 FPS

The Challenge: Real-Time Generative Avatars at Scale

When Sarathi Studio took on the LiveAvatar project, the brief was deceptively simple: generate photorealistic talking-head avatars from a single reference image, driven by audio input, at 20 frames per second. The model powering this was Alibaba's WanS2V-14B -- a 14-billion parameter video generation model that, out of the box, takes about 45 seconds to produce a single 2-second clip. We needed to make it run 900x faster.

This post documents how we got there: the hardware decisions, the model surgery, the streaming architecture, and the painful lessons about GPU memory management that no paper will teach you.

Why WanS2V-14B?

We evaluated several video generation architectures before settling on WanS2V. The alternatives -- Stable Video Diffusion, AnimateDiff, SadTalker -- either lacked the lip-sync fidelity we needed or produced artifacts around jaw boundaries that were unacceptable for professional use cases. WanS2V-14B, despite its size, produced the most natural mouth movements and preserved identity consistency across long sequences.

The model uses a hybrid architecture: a frozen image encoder (based on a CLIP ViT-H variant) feeds into a temporal diffusion transformer with 14 billion parameters. The diffusion process runs in a compressed latent space, but even so, the compute requirements are enormous.

Hardware: 5x H800 GPUs with Tensor Parallelism

Our inference cluster runs on 5 NVIDIA H800 GPUs (80GB HBM3 each) connected via NVLink. The model does not fit on a single GPU -- even in FP16, the parameters alone consume approximately 28GB, and the KV cache and intermediate activations push total memory requirements past 120GB during inference.

We use tensor parallelism across 4 GPUs for the diffusion transformer, with the 5th GPU dedicated to the image encoder and audio feature extraction pipeline. The parallelism strategy splits attention heads across GPUs rather than splitting layers, which minimizes inter-GPU communication during the attention computation.

# Tensor parallel configuration for WanS2V-14B
tp_config = {
    "tensor_parallel_size": 4,
    "pipeline_parallel_size": 1,
    "partition_strategy": "attention_heads",
    "communication_backend": "nccl",
    "encoder_device": "cuda:4",  # dedicated GPU for encoder
    "dtype": "bfloat16",
    "max_batch_size": 1,  # single-stream real-time
    "kv_cache_dtype": "fp8_e4m3",  # quantized KV cache
}

The KV cache quantization to FP8 was critical. It reduced cache memory by 50% with negligible quality impact, freeing enough headroom for the temporal sliding window approach described below.

LoRA Fine-Tuning for Domain-Specific Quality

The base WanS2V model generates excellent general video, but talking-head avatars have specific requirements: precise lip synchronization, stable head pose, and consistent skin texture across frames. We fine-tuned with LoRA (rank 64, alpha 128) on a curated dataset of 12,000 talking-head clips.

The fine-tuning targeted three specific attention layers in the temporal transformer -- the cross-attention layers that attend to audio features, and the first and last self-attention layers in the temporal stack. Freezing the spatial attention layers preserved the model's image quality while allowing the temporal dynamics to specialize.

# LoRA configuration for talking-head specialization
lora_config = {
    "target_modules": [
        "temporal_cross_attn.q_proj",
        "temporal_cross_attn.v_proj",
        "temporal_self_attn_0.q_proj",
        "temporal_self_attn_0.v_proj",
        "temporal_self_attn_last.q_proj",
        "temporal_self_attn_last.v_proj",
    ],
    "r": 64,
    "lora_alpha": 128,
    "lora_dropout": 0.05,
    "training_steps": 15000,
    "learning_rate": 2e-5,
    "warmup_ratio": 0.05,
}

Training ran for 38 hours on 8x A100 GPUs. The resulting LoRA adapter adds only 180MB to the model but improved lip-sync accuracy (measured by LSE-D score) from 7.8 to 9.2.

The Streaming Pipeline: From 45 Seconds to 50ms

The real engineering challenge was not model quality -- it was latency. A standard diffusion model runs 20-50 denoising steps per frame. At 14B parameters, each step takes roughly 90ms on our 4-GPU setup. That is 1.8 seconds per frame -- nowhere near 20 FPS.

We attacked this from three directions:

1. Temporal Sliding Window with Cached Denoising

Instead of generating each frame independently, we maintain a sliding window of 8 frames and only fully denoise the newest frame. Previous frames contribute their intermediate latents as conditioning context, and we reuse 70% of the computation from the previous window step.

2. Reduced Denoising Steps with Distilled Scheduler

We distilled the original 50-step DDPM scheduler into a 4-step consistency model scheduler. This required a separate distillation training run (about 20 hours on 4x A100s) but reduced per-frame denoising from 50 steps to 4 steps -- a 12.5x speedup.

3. Speculative Frame Generation

While the current frame is being finalized, we speculatively begin generating the next frame using predicted audio features from a lightweight lookahead model. If the prediction is accurate (which it is roughly 85% of the time), the next frame is already partially computed when its audio arrives.

class StreamingAvatarPipeline:
    def __init__(self, model, scheduler, window_size=8):
        self.model = model
        self.scheduler = scheduler  # 4-step consistency scheduler
        self.window_size = window_size
        self.frame_cache = LatentCache(maxlen=window_size)
        self.speculative_engine = SpeculativeGenerator(lookahead_ms=100)

    async def generate_frame(self, audio_chunk, reference_image_latent):
        # Get cached context from previous frames
        context = self.frame_cache.get_context()

        # Run 4-step denoising with temporal conditioning
        latent = torch.randn_like(reference_image_latent)
        for step in self.scheduler.timesteps:
            latent = self.model.denoise_step(
                latent, step, audio_chunk,
                reference_image_latent, context
            )

        # Decode to pixel space
        frame = self.model.decode(latent)
        self.frame_cache.push(latent)

        # Start speculative generation for next frame
        self.speculative_engine.begin(self.frame_cache)

        return frame

Combined, these three optimizations brought per-frame latency to approximately 48ms -- just under our 50ms budget for 20 FPS.

Memory Management: The Silent Killer

The most frustrating bugs were all memory-related. GPU memory fragmentation would cause OOM errors after 10-15 minutes of continuous generation, even though peak usage was well within limits. We solved this with three techniques:

• Pre-allocated memory pools: We reserve fixed-size tensors at startup and reuse them across frames, eliminating dynamic allocation entirely during inference.
• Aggressive cache eviction: The sliding window cache uses a strict FIFO policy with pre-allocated slots. No dynamic resizing.
• Periodic defragmentation: Every 1000 frames (roughly every 50 seconds), we run a lightweight defragmentation pass that consolidates fragmented allocations. This adds a single dropped frame but prevents the gradual memory leak that otherwise crashes the system.

Deployment and Monitoring

The system runs behind a WebSocket API that accepts audio chunks and returns JPEG-encoded frames. We chose WebSockets over WebRTC for the initial deployment because the one-way video stream (server to client) did not need WebRTC's bidirectional negotiation complexity.

Monitoring tracks four critical metrics: frame latency p99, GPU memory utilization per device, lip-sync accuracy score (computed on a sampled frame every 5 seconds), and thermal throttling events. The H800 GPUs sustain boost clocks reliably at 70-75C, but we have seen thermal throttling cause latency spikes above 80C in our early deployments before we improved the cooling configuration.

Results and What Comes Next

The production system generates avatars at 20 FPS with a p99 latency of 52ms, serving up to 8 concurrent sessions on a single 5-GPU node. Identity preservation (measured by face embedding cosine similarity) averages 0.94 across 10-minute sessions.

The next challenge is reducing the hardware requirements. We are experimenting with INT4 quantization of the diffusion transformer, which preliminary results suggest could cut the GPU count from 5 to 3 while maintaining acceptable quality. We are also exploring distillation into a smaller 3B parameter model specifically optimized for the talking-head use case. If successful, that could bring the system to a single GPU -- making it accessible for individual creators, not just studio deployments.