RTP Header Extension in Flutter

Question

I am building an app with Flutter and struggling to implement a RTP Header Extension.

I am currently sending a video stream from my app to a server and would like to add metadata (a dict with several ids) to each frame via the flutter_webrtc package. Each frame metadata has different values.

I didn't find any easy way to achieve this, thus I was thinking of using the RTP Header Extension functionnality by inserting these metadata in the Header Extension but I can't figure how to do it in Flutter, as it doesn't seem to be natively supported by any Flutter package.

I think I would need to modify the WebRTC package at a lower level in order to be able to achieve this but I don't know how deep I should go.

Is there a simpler way to go? If I need to modify the package's files, how can I achieve my goal?

Thanks in advance!

Answer 1

Sending single video frame requires to use multiple RTP packets, how you will split metadata between them? Also sending video stream is not reliable and server may not be able to receive/decode all video frames sent by the app because of lost RTP packets, bandwidth limitations and other network conditions.

Would be much easier to use webrtc DataChannel and send your metadata together with FrameNumber(index) and/or timestamp. Server will receive it separately and render over received video.