Video element rendering in chromium nwjs

Video rendering in chromium/nwjs:

There are three major components to Chromium’s video implementation:

  • Pipeline
    • Chromium’s implementation of a media playback engine
    • Handles asynchronous operations like audio/video demuxing decoding, synchronization and resource fetching
    • Pipeline works like a state machine to perform asynchronous initialization, pausing, seeking and playing.
  • FFmpeg
    • Open source library used for Demuxing(container format parsing) and audio/video decoding
    • implements parallel frame-level decoding for many popular codecs.
  • WebKit
    • Implements the HTML and Javascript bindings ( to communicate with video element using javascript)
    • Handles rendering the user agent controls (Creation of media player UI)
    • Provides WebMediaPlayer interface for port-specific implementations of a media playback engine

###Pipeline:###

The pipeline is a pull-based media playback engine that abstracts each step of media playback into 6 different filters.

1. data source
2. demuxing 
3. audio decoding
4. video decoding 
5. audio rendering
6. video rendering

The pipeline manages the lifetime of the filters and exposes a simple thread-safe interface to clients. The filters are connected together to form a filter graph. All 6 filters are running in different thread to provide responsive media playback in browser. Caching of data is implemented at each filter level. Pipeline is completely pull-based and relies on the system clock (vsync) to drive playback. As the system clock requests additional data, the video renderer requests decoded video data from the video decoder, which requests encoded buffers from the demuxer, which reads from the data source, and so on.

State machine of pipeline:

alt text

Working of filters in pipeline

alt text

Caching of data is implemented at each filter level.

alt text

Pipeline and ffmpeg

alt text

Following are filters names and their respective implementation in chromium code.

  • Data source BufferedDataSource
    • A data source capable of loading URLs and buffering the data using an in-memory sliding window.
    • This class ask network layer to fetch data.
    • BufferedDataSource::intermediate_read_buffer_ variable stores data downloaded from network.
  • Demuxing FFmpegDemuxer/FFmpegDemuxerStream/DemuxerStream
    • Demuxer initialises ffmpeg using FFmpegGlue class.
    • Demuxer use buffers downloaded by data source and attempt to recognize the container by looking at the first few bytes
    • Demuxer determines number of streams , type of streams(audio/video) and check whether video/audio codec configuration supported
    • Create audio/video ffmegDemuxerStream and put ffmpeg::AVStream in it.
    • DemuxerStream and a list of Decoders and provides decoded output to Audio/VideoRendererImpl
  • Audio Decoder FFmpegAudioDecoder
    • FFmpegAudioDecoder is wrapper over audio actual decoders provided by ffmpeg.
    • On the basis of audio codec information we determine using demuxer, FFmpegAudioDecoder initialise respective ffmpeg decoder.
    • FFmpegAudioDecoder ask ffmpeg to decode frames present inside encoded ffmpegDemuxerStreams::AudioStreams.
    • Decoded frames generated by FFmpegAudioDecoder are consumed by AudioRendererImpl.
  • Video Decoder FFmpegVideoDecoder
    • FFmpegVideoDecoder is wrapper over actual video decoders provided by ffmpeg.
    • On the basis of video codec information we determine using demuxer, FFmpegVideoDecoder initialise respective ffmpeg decoder.
    • FFmpegVideoDecoder ask ffmpeg to decode frames present inside encoded ffmpegDemuxerStreams::VideoStreams.
    • FFmpegVideoDecoder allocates memory for decoded data.(CPU and GPU based on SW/HW decoder)
    • Decoded frames(videoFrame) generated by FFmpegVideoDecoder are consumed by VideoRendererImpl.
  • Audio Rendering AudioRendererImpl
    • It is responsible for initialization of audio decoder.
    • AudioRendererImpl talks to an AudioRendererAlgorithm that takes care of queueing audio data and stretching/shrinking audio data.
    • It will provide decode audio frames to sound card. Video Rendering VideoRendererImpl
    • It is responsible for initialization of video decoder.
    • VideoRendererImpl creates its own thread for the sole purpose of timing frame presentation.
    • It handles reading from the VideoFrameStream and stores the results in a queue of decoded frames and executing a callback to notify compositing/painting when a frame is ready for rendering.
    • ready_outputs_ : variable stores decoded data

Other important classes:

  • WebMediaPlayerImpl
    • Runs in three thread:
      • Media Thread
      • Renderer Thread
      • Compositor thread.
    • The canonical implementation of blink::WebMediaPlayer that’s backed by Pipeline.
    • Handles normal resource loading, Media Source, and Encrypted Media.
  • FFmpegURLProtocol It maintain state of your seek and read position while interacting with network using bufferDataSource.
  • DecoderBufferQueue Maintains a queue of DecoderBuffers in increasing timestamp order.

Chromium-ffmpeg implementation known facts:

  • Wrappers written by chromium to make playback responsive are tightly couple with ffmpeg. (They are written considering ffmpeg as underlying media framework and they are not generic)
  • FFmpeg works on buffers provided by networks(bufferDataSource)
  • FFmpeg can be decomposed to run in threaded implementation for demuxing, decoding and rendering.
  • Caching of data is implemented at each filter level.
  • HTML Media Elements like audio, video and Web audio are depends on chromium-ffmpeg integration. This Media elements uses underlying playback engine (pipeline) and media framework (Ffmpeg).
  • Ffmpeg supports almost all pixel format but Chromium pipeline supports only famous pixel formats.

References: - https://www.chromium.org/developers/design-documents/video

Written on February 6, 2016