Introduction
Creating vertical video assets at scale is a pain point for many media companies. Manual reframing of 16:9 footage into 9:16, 4:5, or 1:1 is slow, expensive, and inconsistent. Netflix recently shared their approach to automate this process for their Clips feed and marketing campaigns. In this deep dive, we analyze their pipeline and extract lessons you can apply to your own video processing workflows.
Source: Netflix Technology Blog
The Problem with Center Cropping
Simply cropping the center of a 16:9 video to 9:16 often cuts out key subjects, especially in scenes with dynamic blocking or multiple speakers. Netflix's solution treats resizing as a tracking and temporal optimization problem, using a "virtual camera" that intelligently follows the action.
Pipeline Overview
The system processes each video in three stages, generating all three aspect ratios in a single pass to avoid redundant computation:
- Shot Detection and Splitting – Segments video into atomic shots using a neural network, then splits shots further based on active speaker detection.
- Focus Region Discovery – Combines face detection and visual saliency maps to identify the most important region per frame, ranking faces by active speaker status and size.
- Temporal Reframing – Decides how the crop window moves over time, using heuristics to lock static crops for low motion and smooth dynamic tracking for high motion.
This architecture ensures that crops feel intentional and cinematic, not shaky or random.
Key Technical Insights
Active Speaker Detection Matters
In dialogue scenes, simply focusing on the largest face fails when multiple characters speak. Netflix uses an internal face detector combined with LipFlap to identify who is speaking, then logically splits a continuous shot into sub-shots. This allows the virtual camera to cut between speakers as the conversation flows.
Temporal Smoothing is Critical
Frame-by-frame tracking of the focus region produces jittery results. Netflix applies heuristics: for small displacement, lock a static crop; for significant motion, compute a dynamic window and smooth its path. This mimics professional camera operation.
Integration with Editing Tools
The algorithm is embedded into Netflix's internal Clips tool, giving editors a unified interface to review and adjust crops. For external partners, the resizing parameters can be exported as XML and loaded into Adobe Premiere, enabling agencies to start from algorithmic framing decisions rather than manual work.
Limitations and Considerations
- Computational Cost: Running face detection and saliency models per frame requires significant GPU resources. Netflix mitigates this by processing all aspect ratios in one pass, but smaller teams may need to optimize further.
- Animated Content: While the system handles both live-action and animated titles, extreme stylized content may still require manual tweaks.
- Creative Control: Automation can't replace human judgment for complex scenes; the tool is designed to assist, not fully replace, editors.
Next Steps for Learners
If you're building a similar system, start by experimenting with open-source shot detection models (e.g., PySceneDetect) and face detection libraries (e.g., MediaPipe). For temporal smoothing, consider Kalman filters or simple moving averages. And always keep a human-in-the-loop for quality control.
Related reading: Why Data Security Must Follow the Data, Not the Tool and Evaluating AI-Generated Customer Journeys with Structural CDP Metrics.

Code Example: Simplified Shot Detection and Focus Region Tracking
Below is a Python snippet illustrating a basic shot detection and focus region tracking pipeline using OpenCV and PySceneDetect. This is a simplified version of the concepts Netflix employs.
# Import necessary libraries
import cv2
from scenedetect import VideoManager, SceneManager
from scenedetect.detectors import ContentDetector
# Initialize video manager and scene manager
video_manager = VideoManager(['input_video.mp4'])
scene_manager = SceneManager()
scene_manager.add_detector(ContentDetector(threshold=30.0))
# Detect shots
video_manager.start()
scene_manager.detect_scenes(frame_source=video_manager)
shot_list = scene_manager.get_scene_list()
print(f"Detected {len(shot_list)} shots.")
# Process each shot: find focus region (e.g., largest face)
for i, (start, end) in enumerate(shot_list):
cap = cv2.VideoCapture('input_video.mp4')
cap.set(cv2.CAP_PROP_POS_FRAMES, start.get_frames())
ret, frame = cap.read()
if ret:
# Convert to grayscale for face detection
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')
faces = face_cascade.detectMultiScale(gray, 1.1, 4)
if len(faces) > 0:
# Select largest face as focus region
x, y, w, h = max(faces, key=lambda f: f[2]*f[3])
print(f"Shot {i}: Focus region at ({x},{y}) size ({w},{h})")
else:
print(f"Shot {i}: No face detected, using center crop.")
cap.release()
# Note: For production, you'd apply temporal smoothing and generate crops.
This example demonstrates shot segmentation and per-shot focus region detection. Netflix's system adds active speaker detection and temporal optimization on top.

Comparison: Netflix's Approach vs. Adobe Premiere Auto Reframe
| Feature | Netflix's System | Adobe Premiere Auto Reframe |
|---|---|---|
| Shot detection | Neural network + active speaker sub-shots | Scene detection based on content changes |
| Focus region | Face detection + saliency + speaker ranking | Motion tracking and face detection |
| Temporal smoothing | Heuristic-based (static/dynamic windows) | Keyframe-based smoothing |
| Multi-ratio support | 9:16, 4:5, 1:1 in one pass | One ratio at a time |
| Integration | Internal Clips tool + XML export | Native within Premiere |
| Output quality | Outperforms Auto Reframe on internal eval | Good for simple scenes |
Netflix's system is tailored for high-volume, high-quality output, while Auto Reframe is more accessible for individual editors. The XML export bridges the gap by allowing external partners to leverage Netflix's algorithm.

Conclusion
Netflix's multi-ratio video resizing pipeline is a masterclass in combining computer vision with practical tooling. By treating reframing as a temporal optimization problem and integrating with existing editor workflows, they've scaled vertical video production without sacrificing quality. For developers building similar systems, the key takeaways are: use active speaker detection for dialogue scenes, apply temporal smoothing to avoid jitter, and always provide a human-in-the-loop for adjustments.
Further Reading: