StreamY
Go · FFmpeg · PostgreSQL · Docker
High-performance video streaming backend in Go featuring HLS transcoding, chunked delivery, adaptive streaming, and distributed media processing
Demo
Watch at 1.5x — transcoding and final speeds are highlighted in the terminal, and seek forward (at 1:02) to see the video player in action!
For the full optimization journey with before/after speeds, see the optimization thread.
Overview
StreamY is a backend-focused video processing and streaming system written in Go, built to understand how media pipelines work — with the eventual goal of distributing workloads across multiple machines.
An uploaded video is probed for metadata, dynamically planned into renditions, encoded and segmented into HLS, and stored in MinIO. The API handles uploads while a separate background worker runs the FFmpeg pipeline, so long transcoding jobs never block requests. The current focus is single-machine processing and optimization; distributed scheduling, caching, and CDN delivery are later stages.
Key Highlights & Core Contributions
- HLS Transcoding Pipeline: Complete FFmpeg-based pipeline generating HLS playlists and multi-resolution segments (1080p down to 360p) from uploads.
- Dynamic Rendition Planning: Probes each source with ffprobe and plans only the renditions that make sense for it, preserving aspect ratio.
- Decoupled Worker Architecture: Background worker claims videos and runs the media pipeline independently from the API server.
- Concurrent Upload Engine: Worker-pool based MinIO uploads that turned the biggest post-GPU bottleneck into a non-issue.
Performance
Test workload: ~36.5 MB, 34.7s 1080p30 H.264 source on a Ryzen 5 5600H + RTX 3050.
| Pipeline | Total time | Main change |
|---|---|---|
| CPU encoding | ~105s | Initial implementation |
| GPU encoding | ~50.2s | H.264 NVENC |
| Concurrent uploads | ~19.1s | Concurrent MinIO uploads |
| Current best | ~9.1s | Sequential encoding + concurrent uploads + p4 preset |
That's roughly an 11.5× reduction end to end. The most surprising finding: the bottleneck wasn't encoding but uploads — the original sequential upload path took ~41.7s, concurrent uploads cut it to ~1.06s (a ~39× reduction). Conversely, parallel NVENC encoding hurt overall time on this GPU (contention), so renditions encode sequentially while I/O-bound uploads run concurrently.
Design Direction
StreamY is intentionally a V1 single-machine pipeline — a measured baseline (9.1s) to compare against. The next stage is distributed processing: a job queue fanning media jobs out to multiple GPU workers backed by shared storage, then caching, CDN/edge delivery, scheduling, and fault tolerance.