Update version to 0.1-rc.4

Totally rewritten MSE player
Support codecs negotiation for MSE
2025-10-05 00:12:48 +08:00 · 2022-11-24 02:00:31 +03:00 · 2022-11-24 01:59:48 +03:00 · 2022-11-23 21:45:10 +03:00 · 2022-11-23 20:35:11 +03:00 · 2022-11-23 20:34:06 +03:00
24 changed files with 726 additions and 289 deletions
--- a/README.md
+++ b/README.md
@@ -9,6 +9,7 @@ Ultimate camera streaming application with support RTSP, WebRTC, HomeKit, FFmpeg
 - streaming from [RTSP](#source-rtsp), [RTMP](#source-rtmp), [MJPEG](#source-ffmpeg), [HLS/HTTP](#source-ffmpeg), [USB Cameras](#source-ffmpeg-device) and [other sources](#module-streams)
 - streaming to [RTSP](#module-rtsp), [WebRTC](#module-webrtc), [MSE/MP4](#module-mp4) or [MJPEG](#module-mjpeg)
 - first project in the World with support streaming from [HomeKit Cameras](#source-homekit)
+- first project in the World with support H265 for WebRTC in browser (only Safari)
 - on the fly transcoding for unsupported codecs via [FFmpeg](#source-ffmpeg)
 - multi-source 2-way [codecs negotiation](#codecs-negotiation)
   - mixing tracks from different sources to single stream
@@ -26,35 +27,6 @@ Ultimate camera streaming application with support RTSP, WebRTC, HomeKit, FFmpeg
 - [MediaSoup](https://mediasoup.org/) framework routing idea
 - HomeKit Accessory Protocol from [@brutella](https://github.com/brutella/hap)

-## Codecs negotiation
-
-For example, you want to watch RTSP-stream from [Dahua IPC-K42](https://www.dahuasecurity.com/fr/products/All-Products/Network-Cameras/Wireless-Series/Wi-Fi-Series/4MP/IPC-K42) camera in your Chrome browser.
-
- this camera support 2-way audio standard **ONVIF Profile T**
- this camera support codecs **H264, H265** for send video, and you select `H264` in camera settings
- this camera support codecs **AAC, PCMU, PCMA** for send audio (from mic), and you select `AAC/16000` in camera settings
- this camera support codecs **AAC, PCMU, PCMA** for receive audio (to speaker), you don't need to select them
- your browser support codecs **H264, VP8, VP9, AV1** for receive video, you don't need to select them
- your browser support codecs **OPUS, PCMU, PCMA** for send and receive audio, you don't need to select them
- you can't get camera audio directly, because its audio codecs doesn't match with your browser codecs 
-   - so you decide to use transcoding via FFmpeg and add this setting to config YAML file
-   - you have chosen `OPUS/48000/2` codec, because it is higher quality than the `PCMU/8000` or `PCMA/8000`
-
-Now you have stream with two sources - **RTSP and FFmpeg**:
-
-```yaml
-streams:
-  dahua:
-    - rtsp://admin:password@192.168.1.123/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif
-    - ffmpeg:rtsp://admin:password@192.168.1.123/cam/realmonitor?channel=1&subtype=0#audio=opus
-```
-
-**go2rtc** automatically match codecs for you browser and all your stream sources. This called **multi-source 2-way codecs negotiation**. And this is one of the main features of this app.
-
-![](assets/codecs.svg)
-
-**PS.** You can select `PCMU` or `PCMA` codec in camera setting and don't use transcoding at all. Or you can select `AAC` codec for main stream and `PCMU` codec for second stream and add both RTSP to YAML config, this also will work fine.
-
 ## Fast start

 1. Download [binary](#go2rtc-binary) or use [Docker](#go2rtc-docker) or [Home Assistant Add-on](#go2rtc-home-assistant-add-on)
@@ -207,24 +179,27 @@ streams:
  hls: ffmpeg:https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/gear5/prog_index.m3u8#video=copy

  # [MJPEG] video will be transcoded to H264
-  mjpeg: ffmpeg:http://185.97.122.128/cgi-bin/faststream.jpg?stream=half&fps=15#video=h264
+  mjpeg: ffmpeg:http://185.97.122.128/cgi-bin/faststream.jpg#video=h264

  # [RTSP] video with rotation, should be transcoded, so select H264
-  rotate: ffmpeg:rtsp://rtsp:12345678@192.168.1.123/av_stream/ch0#raw=-vf transpose=1#video=h264
+  rotate: ffmpeg:rtsp://rtsp:12345678@192.168.1.123/av_stream/ch0#video=h264#rotate=90
 ```

-All trascoding formats has [built-in templates](https://github.com/AlexxIT/go2rtc/blob/master/cmd/ffmpeg/ffmpeg.go): `h264`, `h264/ultra`, `h264/high`, `h265`, `opus`, `pcmu`, `pcmu/16000`, `pcmu/48000`, `pcma`, `pcma/16000`, `pcma/48000`, `aac/16000`.
+All trascoding formats has [built-in templates](https://github.com/AlexxIT/go2rtc/blob/master/cmd/ffmpeg/ffmpeg.go): `h264`, `h264/ultra`, `h264/high`, `h265`, `opus`, `pcmu`, `pcmu/16000`, `pcmu/48000`, `pcma`, `pcma/16000`, `pcma/48000`, `aac`, `aac/16000`.

 But you can override them via YAML config. You can also add your own formats to config and use them with source params.

 ```yaml
 ffmpeg:
  bin: ffmpeg                                        # path to ffmpeg binary
-  h264: "-codec:v libx264 -g 30 -preset superfast -tune zerolatency -profile main -level 4.1"
+  h264: "-codec:v libx264 -g:v 30 -preset:v superfast -tune:v zerolatency -profile:v main -level:v 4.1"
  mycodec: "-any args that support ffmpeg..."
 ```

-Also you can use `raw` param for any additional FFmpeg arguments. As example for video rotation (`#raw=-vf transpose=1`). Remember that rotation is not possible without transcoding, so add supported codec as second param (`#video=h264`).
+- You can use `video` and `audio` params multiple times (ex. `#video=copy#audio=copy#audio=pcmu`)
+- You can use go2rtc stream name as ffmpeg input (ex. `ffmpeg:camera1#video=h264`)
+- You can use `rotate` params with `90`, `180`, `270` or `-90` values, important with transcoding (ex. `#video=h264#rotate=90`)
+- You can use `raw` param for any additional FFmpeg arguments (ex. `#raw=-vf transpose=1`).

 #### Source: FFmpeg Device

@@ -279,6 +254,24 @@ You can pair device with go2rtc on the HomeKit page. If you can't see your devic

 If you see a device but it does not have a pair button - it is paired to some ecosystem (Apple Home, Home Assistant, HomeBridge etc). You need to delete device from that ecosystem, and it will be available for pairing. If you cannot unpair device, you will have to reset it.

+**Important:**
+
+- HomeKit audio uses very non-standard **AAC-ELD** codec with very non-standard params and specification violation
+- Audio can be transcoded by [ffmpeg](#source-ffmpeg) source with `#async` option
+- Audio can be played by `ffplay` with `-use_wallclock_as_timestamps 1 -async 1` options
+- Audio can't be played in `VLC` and probably any other player
+
+Recommended settings for using HomeKit Camera with WebRTC, MSE, MP4, RTSP:
+
+```
+streams:
+  aqara_g3:
+    - hass:Camera-Hub-G3-AB12
+    - ffmpeg:aqara_g3#audio=aac#audio=opus#async
+```
+
+RTSP link with "normal" audio for any player: `rtsp://192.168.1.123:8554/aqara_g3?video&audio=aac`
+
 **This source is in active development!** Tested only with [Aqara Camera Hub G3](https://www.aqara.com/eu/product/camera-hub-g3) (both EU and CN versions).

 #### Source: Ivideon
@@ -320,9 +313,10 @@ The HTTP API is the main part for interacting with the application. Default addr

 ```yaml
 api:
-  listen: ":1984"  # HTTP API port ("" - disabled)
-  base_path: ""    # API prefix for serve on suburl
-  static_dir: ""   # folder for static files (custom web interface)
+  listen: ":1984"    # HTTP API port ("" - disabled)
+  base_path: "/rtc"  # API prefix for serve on suburl (/api => /rtc/api)
+  static_dir: "www"  # folder for static files (custom web interface)
+  origin: "*"        # allow CORS requests (only * supported)
 ```

 **PS. go2rtc** doesn't provide HTTPS or password protection. Use [Nginx](https://nginx.org/) or [Ngrok](#module-ngrok) or [Home Assistant Add-on](#go2rtc-home-assistant-add-on) for this tasks.
@@ -557,18 +551,17 @@ PS. Additionally WebRTC opens a lot of random UDP ports for transmit encrypted m

 Device | WebRTC | MSE | MP4
 -------|--------|-----|----
-*latency*       | best        | medium      | bad
-Desktop Chrome  | H264        | H264, H265* | H264, H265*
-Desktop Safari  | H264, H265* | H264        | no
-Desktop Edge    | H264        | H264, H265* | H264, H265*
-Desktop Firefox | H264        | H264        | H264
-Desktop Opera   | no          | H264        | H264
-iPhone Safari   | H264, H265* | no          | no
-iPad Safari     | H264, H265* | H264        | no
-Android Chrome  | H264        | H264        | H264
-masOS Hass App  | no          | no          | no
+*latency*           | best        | medium      | bad
+Desktop Chrome 107+ | H264        | H264, H265* | H264, H265*
+Desktop Safari      | H264, H265* | H264, H265  | **no!**
+Desktop Edge        | H264        | H264, H265* | H264, H265*
+Desktop Firefox     | H264        | H264        | H264
+iPad Safari 13+     | H264, H265* | H264, H265  | **no!**
+iPhone Safari 13+   | H264, H265* | **no!**     | **no!**
+Android Chrome 107+ | H264        | H264, H265* | H264
+masOS Hass App      | no          | no          | no

- Chrome H265: [read this](https://github.com/StaZhu/enable-chromium-hevc-hardware-decoding)
+- Chrome H265: [read this](https://chromestatus.com/feature/5186511939567616) and [read this](https://github.com/StaZhu/enable-chromium-hevc-hardware-decoding)
 - Edge H265: [read this](https://www.reddit.com/r/MicrosoftEdge/comments/v9iw8k/enable_hevc_support_in_edge/)
 - Desktop Safari H265: Menu > Develop > Experimental > WebRTC H265
 - iOS Safari H265: Settings > Safari > Advanced > Experimental > WebRTC H265
@@ -578,6 +571,41 @@ masOS Hass App  | no          | no          | no
 - WebRTC audio codecs: `PCMU/8000`, `PCMA/8000`, `OPUS/48000/2`
 - MSE/MP4 audio codecs: `AAC`

+**Apple devices**
+
+- all Apple devices don't support MP4 stream (they only support progressive loading of static files)
+- iPhones don't support MSE technology because it competes with the HLS technology, invented by Apple
+- HLS is the worst technology for **live** streaming, it still exists only because of iPhones
+
+## Codecs negotiation
+
+For example, you want to watch RTSP-stream from [Dahua IPC-K42](https://www.dahuasecurity.com/fr/products/All-Products/Network-Cameras/Wireless-Series/Wi-Fi-Series/4MP/IPC-K42) camera in your Chrome browser.
+
+- this camera support 2-way audio standard **ONVIF Profile T**
+- this camera support codecs **H264, H265** for send video, and you select `H264` in camera settings
+- this camera support codecs **AAC, PCMU, PCMA** for send audio (from mic), and you select `AAC/16000` in camera settings
+- this camera support codecs **AAC, PCMU, PCMA** for receive audio (to speaker), you don't need to select them
+- your browser support codecs **H264, VP8, VP9, AV1** for receive video, you don't need to select them
+- your browser support codecs **OPUS, PCMU, PCMA** for send and receive audio, you don't need to select them
+- you can't get camera audio directly, because its audio codecs doesn't match with your browser codecs
+  - so you decide to use transcoding via FFmpeg and add this setting to config YAML file
+  - you have chosen `OPUS/48000/2` codec, because it is higher quality than the `PCMU/8000` or `PCMA/8000`
+
+Now you have stream with two sources - **RTSP and FFmpeg**:
+
+```yaml
+streams:
+  dahua:
+    - rtsp://admin:password@192.168.1.123/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif
+    - ffmpeg:rtsp://admin:password@192.168.1.123/cam/realmonitor?channel=1&subtype=0#audio=opus
+```
+
+**go2rtc** automatically match codecs for you browser and all your stream sources. This called **multi-source 2-way codecs negotiation**. And this is one of the main features of this app.
+
+![](assets/codecs.svg)
+
+**PS.** You can select `PCMU` or `PCMA` codec in camera setting and don't use transcoding at all. Or you can select `AAC` codec for main stream and `PCMU` codec for second stream and add both RTSP to YAML config, this also will work fine.
+
 ## TIPS

 **Using apps for low RTSP delay**
--- a/cmd/api/api.go
+++ b/cmd/api/api.go
@@ -17,6 +17,7 @@ func Init() {
 			Listen    string `yaml:"listen"`
 			BasePath  string `yaml:"base_path"`
 			StaticDir string `yaml:"static_dir"`
+			Origin    string `yaml:"origin"`
 		} `yaml:"api"`
 	}

@@ -34,7 +35,7 @@ func Init() {
 	log = app.GetLogger("api")

 	initStatic(cfg.Mod.StaticDir)
-	initWS()
+	initWS(cfg.Mod.Origin)

 	HandleFunc("api/streams", streamsHandler)
 	HandleFunc("api/ws", apiWS)
@@ -48,16 +49,18 @@ func Init() {

 	log.Info().Str("addr", cfg.Mod.Listen).Msg("[api] listen")

+	s := http.Server{}
+	s.Handler = http.DefaultServeMux
+
+	if log.Trace().Enabled() {
+		s.Handler = middlewareLog(s.Handler)
+	}
+
+	if cfg.Mod.Origin == "*" {
+		s.Handler = middlewareCORS(s.Handler)
+	}
+
 	go func() {
-		s := http.Server{}
-
-		if log.Trace().Enabled() {
-			s.Handler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-				log.Trace().Stringer("url", r.URL).Msgf("[api] %s", r.Method)
-				http.DefaultServeMux.ServeHTTP(w, r)
-			})
-		}
-
 		if err = s.Serve(listener); err != nil {
 			log.Fatal().Err(err).Msg("[api] serve")
 		}
@@ -83,6 +86,21 @@ var basePath string
 var log zerolog.Logger
 var wsHandlers = make(map[string]WSHandler)

+func middlewareLog(next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		log.Trace().Msgf("[api] %s %s", r.Method, r.URL)
+		next.ServeHTTP(w, r)
+	})
+}
+
+func middlewareCORS(next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Access-Control-Allow-Origin", "*")
+		w.Header().Set("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE, OPTIONS")
+		next.ServeHTTP(w, r)
+	})
+}
+
 func streamsHandler(w http.ResponseWriter, r *http.Request) {
 	src := r.URL.Query().Get("src")
 	name := r.URL.Query().Get("name")
--- a/cmd/api/ws.go
+++ b/cmd/api/ws.go
@@ -4,35 +4,19 @@ import (
 	"github.com/AlexxIT/go2rtc/pkg/streamer"
 	"github.com/gorilla/websocket"
 	"net/http"
-	"net/url"
-	"strings"
 	"sync"
 )

-func initWS() {
+func initWS(origin string) {
 	wsUp = &websocket.Upgrader{
 		ReadBufferSize:  1024,
 		WriteBufferSize: 512000,
 	}
-	wsUp.CheckOrigin = func(r *http.Request) bool {
-		origin := r.Header["Origin"]
-		if len(origin) == 0 {
+
+	if origin == "*" {
+		wsUp.CheckOrigin = func(r *http.Request) bool {
 			return true
 		}
-		o, err := url.Parse(origin[0])
-		if err != nil {
-			return false
-		}
-		if o.Host == r.Host {
-			return true
-		}
-		log.Trace().Msgf("[api.ws] origin: %s, host: %s", o.Host, r.Host)
-		// some users change Nginx external port using Docker port
-		// so origin will be with a port and host without
-		if i := strings.IndexByte(o.Host, ':'); i > 0 {
-			return o.Host[:i] == r.Host
-		}
-		return false
 	}
 }

--- a/cmd/app/app.go
+++ b/cmd/app/app.go
@@ -10,7 +10,7 @@ import (
 	"runtime"
 )

-var Version = "0.1-rc.3"
+var Version = "0.1-rc.4"
 var UserAgent = "go2rtc/" + Version

 func Init() {
--- a/cmd/ffmpeg/ffmpeg.go
+++ b/cmd/ffmpeg/ffmpeg.go
@@ -38,7 +38,7 @@ func Init() {
 		"h264":       "-c:v libx264 -g:v 30 -preset:v superfast -tune:v zerolatency -profile:v main -level:v 4.1 -pix_fmt:v yuv420p",
 		"h264/ultra": "-c:v libx264 -g:v 30 -preset:v ultrafast -tune:v zerolatency",
 		"h264/high":  "-c:v libx264 -g:v 30 -preset:v superfast -tune:v zerolatency",
-		"h265":       "-c:v libx265 -g:v 30 -preset:v ultrafast -tune:v zerolatency",
+		"h265":       "-c:v libx265 -g:v 30 -preset:v superfast -tune:v zerolatency -profile:v main -level:v 5.1 -pix_fmt:v yuv420p",
 		"mjpeg":      "-c:v mjpeg -force_duplicated_matrix:v 1 -huffman:v 0 -pix_fmt:v yuvj420p",
 		"opus":       "-c:a libopus -ar:a 48000 -ac:a 2",
 		"pcmu":       "-c:a pcm_mulaw -ar:a 8000 -ac:a 1",
@@ -146,7 +146,7 @@ func Init() {
 				s += " -vn"
 			case 1:
 				if len(query["audio"]) > 1 {
-					s += " -map 0:v:0"
+					s += " -map 0:v:0?"
 				}
 				for _, video := range query["video"] {
 					if video == "copy" {
@@ -158,9 +158,9 @@ func Init() {
 			default:
 				for i, video := range query["video"] {
 					if video == "copy" {
-						s += " -map 0:v:0 -c:v:" + strconv.Itoa(i) + " copy"
+						s += " -map 0:v:0? -c:v:" + strconv.Itoa(i) + " copy"
 					} else {
-						s += " -map 0:v:0 " + strings.ReplaceAll(tpl[video], ":v ", ":v:"+strconv.Itoa(i)+" ")
+						s += " -map 0:v:0? " + strings.ReplaceAll(tpl[video], ":v ", ":v:"+strconv.Itoa(i)+" ")
 					}
 				}
 			}
@@ -170,7 +170,7 @@ func Init() {
 				s += " -an"
 			case 1:
 				if len(query["video"]) > 1 {
-					s += " -map 0:a:0"
+					s += " -map 0:a:0?"
 				}
 				for _, audio := range query["audio"] {
 					if audio == "copy" {
@@ -182,9 +182,9 @@ func Init() {
 			default:
 				for i, audio := range query["audio"] {
 					if audio == "copy" {
-						s += " -map 0:a:0 -c:a:" + strconv.Itoa(i) + " copy"
+						s += " -map 0:a:0? -c:a:" + strconv.Itoa(i) + " copy"
 					} else {
-						s += " -map 0:a:0 " + strings.ReplaceAll(tpl[audio], ":a ", ":a:"+strconv.Itoa(i)+" ")
+						s += " -map 0:a:0? " + strings.ReplaceAll(tpl[audio], ":a ", ":a:"+strconv.Itoa(i)+" ")
 					}
 				}
 			}
--- a/cmd/mp4/mp4.go
+++ b/cmd/mp4/mp4.go
@@ -63,12 +63,12 @@ func handlerKeyframe(w http.ResponseWriter, r *http.Request) {
 }

 func handlerMP4(w http.ResponseWriter, r *http.Request) {
+	log.Trace().Msgf("[api.mp4] %s %+v", r.Method, r.Header)
+
 	if isChromeFirst(w, r) || isSafari(w, r) {
 		return
 	}

-	log.Trace().Msgf("[api.mp4] %+v", r)
-
 	src := r.URL.Query().Get("src")
 	stream := streams.GetOrNew(src)
 	if stream == nil {
--- a/cmd/mp4/mse.go
+++ b/cmd/mp4/mse.go
@@ -5,6 +5,7 @@ import (
 	"github.com/AlexxIT/go2rtc/cmd/streams"
 	"github.com/AlexxIT/go2rtc/pkg/mp4"
 	"github.com/AlexxIT/go2rtc/pkg/streamer"
+	"strings"
 )

 const MsgTypeMSE = "mse" // fMP4
@@ -22,6 +23,10 @@ func handlerWS(ctx *api.Context, msg *streamer.Message) {
 	cons.UserAgent = ctx.Request.UserAgent()
 	cons.RemoteAddr = ctx.Request.RemoteAddr

+	if codecs, ok := msg.Value.(string); ok {
+		cons.Medias = parseMedias(codecs)
+	}
+
 	cons.Listen(func(msg interface{}) {
 		if data, ok := msg.([]byte); ok {
 			for len(data) > packetSize {
@@ -55,3 +60,42 @@ func handlerWS(ctx *api.Context, msg *streamer.Message) {

 	cons.Start()
 }
+
+func parseMedias(codecs string) (medias []*streamer.Media) {
+	var videos []*streamer.Codec
+	var audios []*streamer.Codec
+
+	for _, name := range strings.Split(codecs, ",") {
+		switch name {
+		case "avc1.640029":
+			codec := &streamer.Codec{Name: streamer.CodecH264}
+			videos = append(videos, codec)
+		case "hvc1.1.6.L153.B0":
+			codec := &streamer.Codec{Name: streamer.CodecH265}
+			videos = append(videos, codec)
+		case "mp4a.40.2":
+			codec := &streamer.Codec{Name: streamer.CodecAAC}
+			audios = append(audios, codec)
+		}
+	}
+
+	if videos != nil {
+		media := &streamer.Media{
+			Kind:      streamer.KindVideo,
+			Direction: streamer.DirectionRecvonly,
+			Codecs:    videos,
+		}
+		medias = append(medias, media)
+	}
+
+	if audios != nil {
+		media := &streamer.Media{
+			Kind:      streamer.KindAudio,
+			Direction: streamer.DirectionRecvonly,
+			Codecs:    audios,
+		}
+		medias = append(medias, media)
+	}
+
+	return
+}
--- a/cmd/rtsp/rtsp.go
+++ b/cmd/rtsp/rtsp.go
@@ -140,6 +140,11 @@ func tcpHandler(c net.Conn) {

 		switch msg {
 		case rtsp.MethodDescribe:
+			if len(conn.URL.Path) == 0 {
+				log.Warn().Msg("[rtsp] server empty URL on DESCRIBE")
+				return
+			}
+
 			name = conn.URL.Path[1:]

 			stream := streams.Get(name)
@@ -161,6 +166,11 @@ func tcpHandler(c net.Conn) {
 			}

 		case rtsp.MethodAnnounce:
+			if len(conn.URL.Path) == 0 {
+				log.Warn().Msg("[rtsp] server empty URL on ANNOUNCE")
+				return
+			}
+
 			name = conn.URL.Path[1:]

 			stream := streams.Get(name)
--- a/cmd/webrtc/candidates.go
+++ b/cmd/webrtc/candidates.go
@@ -13,7 +13,27 @@ func AddCandidate(address string) {
 	candidates = append(candidates, address)
 }

-func addCanditates(answer string) (string, error) {
+func asyncCandidates(ctx *api.Context) {
+	for _, address := range candidates {
+		address, err := webrtc.LookupIP(address)
+		if err != nil {
+			log.Warn().Err(err).Caller().Send()
+			continue
+		}
+
+		cand, err := webrtc.NewCandidate(address)
+		if err != nil {
+			log.Warn().Err(err).Caller().Send()
+			continue
+		}
+
+		log.Trace().Str("candidate", cand).Msg("[webrtc] config")
+
+		ctx.Write(&streamer.Message{Type: webrtc.MsgTypeCandidate, Value: cand})
+	}
+}
+
+func syncCanditates(answer string) (string, error) {
 	if len(candidates) == 0 {
 		return answer, nil
 	}
--- a/cmd/webrtc/webrtc.go
+++ b/cmd/webrtc/webrtc.go
@@ -33,7 +33,7 @@ func Init() {
 	address := cfg.Mod.Listen
 	pionAPI, err := webrtc.NewAPI(address)
 	if pionAPI == nil {
-		log.Error().Err(err).Msg("[webrtc] init API")
+		log.Error().Err(err).Caller().Msg("webrtc.NewAPI")
 		return
 	}

@@ -55,7 +55,7 @@ func Init() {

 	candidates = cfg.Mod.Candidates

-	api.HandleWS(webrtc.MsgTypeOffer, offerHandler)
+	api.HandleWS(webrtc.MsgTypeOffer, asyncHandler)
 	api.HandleWS(webrtc.MsgTypeCandidate, candidateHandler)

 	api.HandleFunc("api/webrtc", syncHandler)
@@ -66,7 +66,7 @@ var log zerolog.Logger

 var NewPConn func() (*pion.PeerConnection, error)

-func offerHandler(ctx *api.Context, msg *streamer.Message) {
+func asyncHandler(ctx *api.Context, msg *streamer.Message) {
 	src := ctx.Request.URL.Query().Get("src")
 	stream := streams.Get(src)
 	if stream == nil {
@@ -81,7 +81,7 @@ func offerHandler(ctx *api.Context, msg *streamer.Message) {
 	conn := new(webrtc.Conn)
 	conn.Conn, err = NewPConn()
 	if err != nil {
-		log.Error().Err(err).Msg("[webrtc] new conn")
+		log.Error().Err(err).Caller().Msg("NewPConn")
 		return
 	}

@@ -104,14 +104,14 @@ func offerHandler(ctx *api.Context, msg *streamer.Message) {
 	log.Trace().Msgf("[webrtc] offer:\n%s", offer)

 	if err = conn.SetOffer(offer); err != nil {
-		log.Warn().Err(err).Msg("[api.webrtc] set offer")
+		log.Warn().Err(err).Caller().Msg("conn.SetOffer")
 		ctx.Error(err)
 		return
 	}

 	// 2. AddConsumer, so we get new tracks
 	if err = stream.AddConsumer(conn); err != nil {
-		log.Warn().Err(err).Msg("[api.webrtc] add consumer")
+		log.Warn().Err(err).Caller().Msg("stream.AddConsumer")
 		_ = conn.Conn.Close()
 		ctx.Error(err)
 		return
@@ -120,25 +120,20 @@ func offerHandler(ctx *api.Context, msg *streamer.Message) {
 	conn.Init()

 	// exchange sdp without waiting all candidates
-	//answer, err := conn.ExchangeSDP(offer, false)
-	//answer, err := conn.GetAnswer()
-	answer, err := conn.GetCompleteAnswer()
-	if err == nil {
-		answer, err = addCanditates(answer)
-	}
+	answer, err := conn.GetAnswer()
 	log.Trace().Msgf("[webrtc] answer\n%s", answer)

 	if err != nil {
-		log.Error().Err(err).Msg("[webrtc] get answer")
+		log.Error().Err(err).Caller().Msg("conn.GetAnswer")
 		ctx.Error(err)
 		return
 	}

-	ctx.Write(&streamer.Message{
-		Type: webrtc.MsgTypeAnswer, Value: answer,
-	})
-
 	ctx.Consumer = conn
+
+	ctx.Write(&streamer.Message{Type: webrtc.MsgTypeAnswer, Value: answer})
+
+	asyncCandidates(ctx)
 }

 func syncHandler(w http.ResponseWriter, r *http.Request) {
@@ -151,19 +146,19 @@ func syncHandler(w http.ResponseWriter, r *http.Request) {
 	// get offer
 	offer, err := ioutil.ReadAll(r.Body)
 	if err != nil {
-		log.Error().Err(err).Caller().Send()
+		log.Error().Err(err).Caller().Msg("ioutil.ReadAll")
 		return
 	}

 	answer, err := ExchangeSDP(stream, string(offer), r.UserAgent())
 	if err != nil {
-		log.Error().Err(err).Caller().Send()
+		log.Error().Err(err).Caller().Msg("ExchangeSDP")
 		return
 	}

 	// send SDP to client
 	if _, err = w.Write([]byte(answer)); err != nil {
-		log.Error().Err(err).Caller().Send()
+		log.Error().Err(err).Caller().Msg("w.Write")
 	}
 }

@@ -174,7 +169,7 @@ func ExchangeSDP(
 	conn := new(webrtc.Conn)
 	conn.Conn, err = NewPConn()
 	if err != nil {
-		log.Error().Err(err).Msg("[webrtc] new conn")
+		log.Error().Err(err).Caller().Msg("NewPConn")
 		return
 	}

@@ -192,13 +187,13 @@ func ExchangeSDP(
 	log.Trace().Msgf("[webrtc] offer:\n%s", offer)

 	if err = conn.SetOffer(offer); err != nil {
-		log.Warn().Err(err).Msg("[api.webrtc] set offer")
+		log.Warn().Err(err).Caller().Msg("conn.SetOffer")
 		return
 	}

 	// 2. AddConsumer, so we get new tracks
 	if err = stream.AddConsumer(conn); err != nil {
-		log.Warn().Err(err).Msg("[api.webrtc] add consumer")
+		log.Warn().Err(err).Caller().Msg("stream.AddConsumer")
 		_ = conn.Conn.Close()
 		return
 	}
@@ -209,12 +204,12 @@ func ExchangeSDP(
 	//answer, err := conn.ExchangeSDP(offer, false)
 	answer, err = conn.GetCompleteAnswer()
 	if err == nil {
-		answer, err = addCanditates(answer)
+		answer, err = syncCanditates(answer)
 	}
 	log.Trace().Msgf("[webrtc] answer\n%s", answer)

 	if err != nil {
-		log.Error().Err(err).Msg("[webrtc] get answer")
+		log.Error().Err(err).Caller().Msg("conn.GetCompleteAnswer")
 	}

 	return
--- a/pkg/fake/producer.go
+++ b/pkg/fake/producer.go
@@ -24,7 +24,7 @@ func (p *Producer) GetTrack(media *streamer.Media, codec *streamer.Codec) *strea
 		panic("you shall not pass!")
 	}

-	track := &streamer.Track{Codec: codec, Direction: media.Direction}
+	track := streamer.NewTrack(codec, media.Direction)

 	switch media.Direction {
 	case streamer.DirectionSendonly:
--- a/pkg/h265/README.md
+++ b/pkg/h265/README.md
@@ -1,3 +1,4 @@
 ## Useful links

 - https://datatracker.ietf.org/doc/html/rfc7798
+- [Add initial support for WebRTC HEVC](https://trac.webkit.org/changeset/259452/webkit)
--- a/pkg/h265/helper.go
+++ b/pkg/h265/helper.go
@@ -2,19 +2,57 @@ package h265

 import (
 	"encoding/base64"
+	"encoding/binary"
 	"github.com/AlexxIT/go2rtc/pkg/streamer"
 )

 const (
-	NALUnitTypeIFrame = 19
+	NALUTypePFrame  = 1
+	NALUTypeIFrame  = 19
+	NALUTypeIFrame2 = 20
+	NALUTypeIFrame3 = 21
+	NALUTypeVPS     = 32
+	NALUTypeSPS     = 33
+	NALUTypePPS     = 34
+	NALUTypeFU      = 49
 )

-func NALUnitType(b []byte) byte {
-	return b[4] >> 1
+func NALUType(b []byte) byte {
+	return (b[4] >> 1) & 0x3F
 }

 func IsKeyframe(b []byte) bool {
-	return NALUnitType(b) == NALUnitTypeIFrame
+	for {
+		switch NALUType(b) {
+		case NALUTypePFrame:
+			return false
+		case NALUTypeIFrame, NALUTypeIFrame2, NALUTypeIFrame3:
+			return true
+		}
+
+		size := int(binary.BigEndian.Uint32(b)) + 4
+		if size < len(b) {
+			b = b[size:]
+			continue
+		} else {
+			return false
+		}
+	}
+}
+
+func Types(data []byte) []byte {
+	var types []byte
+	for {
+		types = append(types, NALUType(data))
+
+		size := 4 + int(binary.BigEndian.Uint32(data))
+		if size < len(data) {
+			data = data[size:]
+		} else {
+			break
+		}
+	}
+	return types
 }

 func GetParameterSet(fmtp string) (vps, sps, pps []byte) {
--- a/pkg/h265/rtp.go
+++ b/pkg/h265/rtp.go
@@ -1,6 +1,7 @@
 package h265

 import (
+	"encoding/binary"
 	"github.com/AlexxIT/go2rtc/pkg/h264"
 	"github.com/AlexxIT/go2rtc/pkg/streamer"
 	"github.com/deepch/vdk/codec/h265parser"
@@ -8,77 +9,59 @@ import (
 )

 func RTPDepay(track *streamer.Track) streamer.WrapperFunc {
-	vps, sps, pps := GetParameterSet(track.Codec.FmtpLine)
+	//vps, sps, pps := GetParameterSet(track.Codec.FmtpLine)
+	//ps := h264.EncodeAVC(vps, sps, pps)

-	var buffer []byte
+	buf := make([]byte, 0, 512*1024) // 512K
+	var nuStart int

 	return func(push streamer.WriterFunc) streamer.WriterFunc {
 		return func(packet *rtp.Packet) error {
-			nut := (packet.Payload[0] >> 1) & 0x3f
-			//fmt.Printf(
-			//	"[RTP] codec: %s, nalu: %2d, size: %6d, ts: %10d, pt: %2d, ssrc: %d, seq: %d\n",
-			//	track.Codec.Name, nut, len(packet.Payload), packet.Timestamp,
-			//	packet.PayloadType, packet.SSRC, packet.SequenceNumber,
-			//)
+			data := packet.Payload
+			nuType := (data[0] >> 1) & 0x3F
+			//log.Printf("[RTP] codec: %s, nalu: %2d, size: %6d, ts: %10d, pt: %2d, ssrc: %d, seq: %d, %v", track.Codec.Name, nuType, len(packet.Payload), packet.Timestamp, packet.PayloadType, packet.SSRC, packet.SequenceNumber, packet.Marker)

-			switch nut {
-			case h265parser.NAL_UNIT_UNSPECIFIED_49:
-				data := packet.Payload
+			if nuType == NALUTypeFU {
 				switch data[2] >> 6 {
 				case 2: // begin
-					buffer = []byte{
-						(data[0] & 0x81) | (data[2] & 0x3f << 1), data[1],
-					}
-					buffer = append(buffer, data[3:]...)
+					nuType = data[2] & 0x3F
+
+					// push PS data before keyframe
+					//if len(buf) == 0 && nuType >= 19 && nuType <= 21 {
+					//	buf = append(buf, ps...)
+					//}
+
+					nuStart = len(buf)
+					buf = append(buf, 0, 0, 0, 0) // NAL unit size
+					buf = append(buf, (data[0]&0x81)|(nuType<<1), data[1])
+					buf = append(buf, data[3:]...)
 					return nil
 				case 0: // continue
-					buffer = append(buffer, data[3:]...)
+					buf = append(buf, data[3:]...)
 					return nil
 				case 1: // end
-					packet.Payload = append(buffer, data[3:]...)
+					buf = append(buf, data[3:]...)
+					binary.BigEndian.PutUint32(buf[nuStart:], uint32(len(buf)-nuStart-4))
 				}
-			case h265parser.NAL_UNIT_VPS:
-				vps = packet.Payload
-				return nil
-			case h265parser.NAL_UNIT_SPS:
-				sps = packet.Payload
-				return nil
-			case h265parser.NAL_UNIT_PPS:
-				pps = packet.Payload
-				return nil
-			default:
-				//panic("not implemented")
+			} else {
+				nuStart = len(buf)
+				buf = append(buf, 0, 0, 0, 0) // NAL unit size
+				buf = append(buf, data...)
+				binary.BigEndian.PutUint32(buf[nuStart:], uint32(len(data)))
 			}

-			var clone rtp.Packet
-
-			nut = (packet.Payload[0] >> 1) & 0x3f
-			if nut >= h265parser.NAL_UNIT_CODED_SLICE_BLA_W_LP && nut <= h265parser.NAL_UNIT_CODED_SLICE_CRA {
-				clone = *packet
-				clone.Version = h264.RTPPacketVersionAVC
-				clone.Payload = h264.EncodeAVC(vps)
-				if err := push(&clone); err != nil {
-					return err
-				}
-
-				clone = *packet
-				clone.Version = h264.RTPPacketVersionAVC
-				clone.Payload = h264.EncodeAVC(sps)
-				if err := push(&clone); err != nil {
-					return err
-				}
-
-				clone = *packet
-				clone.Version = h264.RTPPacketVersionAVC
-				clone.Payload = h264.EncodeAVC(pps)
-				if err := push(&clone); err != nil {
-					return err
-				}
+			// collect all NAL Units for Access Unit
+			if !packet.Marker {
+				return nil
 			}

-			clone = *packet
+			//log.Printf("[HEVC] %v, len: %d", Types(buf), len(buf))
+
+			clone := *packet
 			clone.Version = h264.RTPPacketVersionAVC
-			clone.Payload = h264.EncodeAVC(packet.Payload)
+			clone.Payload = buf
+
+			buf = buf[:0]

 			return push(&clone)
 		}
@@ -106,13 +89,13 @@ func SafariPay(mtu uint16) streamer.WrapperFunc {

 			var start byte

-			nut := (data[4] >> 1) & 0b111111
+			nuType := (data[4] >> 1) & 0b111111
 			//fmt.Printf("[H265] nut: %2d, size: %6d, data: %16x\n", nut, len(data), data[4:20])
 			switch {
-			case nut >= h265parser.NAL_UNIT_VPS && nut <= h265parser.NAL_UNIT_PPS:
+			case nuType >= h265parser.NAL_UNIT_VPS && nuType <= h265parser.NAL_UNIT_PPS:
 				buffer = append(buffer, data...)
 				return nil
-			case nut >= h265parser.NAL_UNIT_CODED_SLICE_BLA_W_LP && nut <= h265parser.NAL_UNIT_CODED_SLICE_CRA:
+			case nuType >= h265parser.NAL_UNIT_CODED_SLICE_BLA_W_LP && nuType <= h265parser.NAL_UNIT_CODED_SLICE_CRA:
 				buffer = append([]byte{3}, buffer...)
 				data = append(buffer, data...)
 				start = 1
--- a/pkg/homekit/client.go
+++ b/pkg/homekit/client.go
@@ -75,7 +75,7 @@ func (c *Client) GetTrack(media *streamer.Media, codec *streamer.Codec) *streame
 		}
 	}

-	track := &streamer.Track{Codec: codec, Direction: media.Direction}
+	track := streamer.NewTrack(codec, media.Direction)
 	c.tracks = append(c.tracks, track)
 	return track
 }
--- a/pkg/ivideon/client.go
+++ b/pkg/ivideon/client.go
@@ -192,10 +192,7 @@ func (c *Client) getTracks() error {
 				}
 				c.medias = append(c.medias, media)

-				track := &streamer.Track{
-					Direction: streamer.DirectionSendonly,
-					Codec:     codec,
-				}
+				track := streamer.NewTrack(codec, streamer.DirectionSendonly)
 				c.tracks[msg.TrackID] = track

 			case "mp4a": // mp4a.40.2
--- a/pkg/mp4/consumer.go
+++ b/pkg/mp4/consumer.go
@@ -12,6 +12,7 @@ import (
 type Consumer struct {
 	streamer.Element

+	Medias     []*streamer.Media
 	UserAgent  string
 	RemoteAddr string

@@ -23,6 +24,11 @@ type Consumer struct {
 }

 func (c *Consumer) GetMedias() []*streamer.Media {
+	if c.Medias != nil {
+		return c.Medias
+	}
+
+	// default medias
 	return []*streamer.Media{
 		{
 			Kind:      streamer.KindVideo,
--- a/pkg/mp4/muxer.go
+++ b/pkg/mp4/muxer.go
@@ -3,7 +3,6 @@ package mp4
 import (
 	"encoding/binary"
 	"encoding/hex"
-	"fmt"
 	"github.com/AlexxIT/go2rtc/pkg/h264"
 	"github.com/AlexxIT/go2rtc/pkg/h265"
 	"github.com/AlexxIT/go2rtc/pkg/streamer"
@@ -36,8 +35,9 @@ func (m *Muxer) MimeType(codecs []*streamer.Codec) string {
 		case streamer.CodecH264:
 			s += "avc1." + h264.GetProfileLevelID(codec.FmtpLine)
 		case streamer.CodecH265:
-			// +Safari +Chrome +Edge -iOS15 -Android13
-			s += "hvc1.1.6.L93.B0" // hev1.1.6.L93.B0
+			// H.265 profile=main level=5.1
+			// hvc1 - supported in Safari, hev1 - doesn't, both supported in Chrome
+			s += "hvc1.1.6.L153.B0"
 		case streamer.CodecAAC:
 			s += "mp4a.40.2"
 		}
@@ -97,7 +97,10 @@ func (m *Muxer) GetInit(codecs []*streamer.Codec) ([]byte, error) {
 		case streamer.CodecH265:
 			vps, sps, pps := h265.GetParameterSet(codec.FmtpLine)
 			if sps == nil {
-				return nil, fmt.Errorf("empty SPS: %#v", codec)
+				// some dummy SPS and PPS not a problem
+				vps = []byte{0x40, 0x01, 0x0c, 0x01, 0xff, 0xff, 0x01, 0x40, 0x00, 0x00, 0x03, 0x00, 0x00, 0x03, 0x00, 0x00, 0x03, 0x00, 0x00, 0x03, 0x00, 0x99, 0xac, 0x09}
+				sps = []byte{0x42, 0x01, 0x01, 0x01, 0x40, 0x00, 0x00, 0x03, 0x00, 0x00, 0x03, 0x00, 0x00, 0x03, 0x00, 0x00, 0x03, 0x00, 0x99, 0xa0, 0x01, 0x40, 0x20, 0x05, 0xa1, 0xfe, 0x5a, 0xee, 0x46, 0xc1, 0xae, 0x55, 0x04}
+				pps = []byte{0x44, 0x01, 0xc0, 0x73, 0xc0, 0x4c, 0x90}
 			}

 			codecData, err := h265parser.NewCodecDataFromVPSAndSPSAndPPS(vps, sps, pps)
--- a/pkg/rtmp/client.go
+++ b/pkg/rtmp/client.go
@@ -83,9 +83,7 @@ func (c *Client) Dial() (err error) {
 			}
 			c.medias = append(c.medias, media)

-			track := &streamer.Track{
-				Codec: codec, Direction: media.Direction,
-			}
+			track := streamer.NewTrack(codec, media.Direction)
 			c.tracks = append(c.tracks, track)

 		case av.AAC:
@@ -108,9 +106,7 @@ func (c *Client) Dial() (err error) {
 			}
 			c.medias = append(c.medias, media)

-			track := &streamer.Track{
-				Codec: codec, Direction: media.Direction,
-			}
+			track := streamer.NewTrack(codec, media.Direction)
 			c.tracks = append(c.tracks, track)

 		default:
--- a/pkg/rtsp/conn.go
+++ b/pkg/rtsp/conn.go
@@ -429,9 +429,7 @@ func (c *Conn) SetupMedia(
 		return nil, err
 	}

-	track := &streamer.Track{
-		Codec: codec, Direction: media.Direction,
-	}
+	track := streamer.NewTrack(codec, media.Direction)

 	switch track.Direction {
 	case streamer.DirectionSendonly:
@@ -519,9 +517,7 @@ func (c *Conn) Accept() error {
 			// TODO: fix someday...
 			c.channels = map[byte]*streamer.Track{}
 			for i, media := range c.Medias {
-				track := &streamer.Track{
-					Codec: media.Codecs[0], Direction: media.Direction,
-				}
+				track := streamer.NewTrack(media.Codecs[0], media.Direction)
 				c.tracks = append(c.tracks, track)
 				c.channels[byte(i<<1)] = track
 			}
--- a/pkg/streamer/track.go
+++ b/pkg/streamer/track.go
@@ -13,12 +13,18 @@ type Track struct {
 	Codec     *Codec
 	Direction string
 	sink      map[*Track]WriterFunc
-	sinkMu    sync.RWMutex
+	sinkMu    *sync.RWMutex
+}
+
+func NewTrack(codec *Codec, direction string) *Track {
+	return &Track{Codec: codec, Direction: direction, sinkMu: new(sync.RWMutex)}
 }

 func (t *Track) String() string {
 	s := t.Codec.String()
+	t.sinkMu.RLock()
 	s += fmt.Sprintf(", sinks=%d", len(t.sink))
+	t.sinkMu.RUnlock()
 	return s
 }

@@ -38,14 +44,12 @@ func (t *Track) Bind(w WriterFunc) *Track {
 		t.sink = map[*Track]WriterFunc{}
 	}

-	clone := &Track{
-		Codec: t.Codec, Direction: t.Direction, sink: t.sink,
-	}
-	t.sink[clone] = w
+	clone := *t
+	t.sink[&clone] = w

 	t.sinkMu.Unlock()

-	return clone
+	return &clone
 }

 func (t *Track) Unbind() {
@@ -55,7 +59,9 @@ func (t *Track) Unbind() {
 }

 func (t *Track) GetSink(from *Track) {
+	t.sinkMu.Lock()
 	t.sink = from.sink
+	t.sinkMu.Unlock()
 }

 func (t *Track) HasSink() bool {
--- a/pkg/webrtc/conn.go
+++ b/pkg/webrtc/conn.go
@@ -1,10 +1,8 @@
 package webrtc

 import (
-	"fmt"
 	"github.com/AlexxIT/go2rtc/pkg/streamer"
 	"github.com/pion/webrtc/v3"
-	"sort"
 )

 const (
@@ -59,7 +57,7 @@ func (c *Conn) Init() {
 			}
 		}

-		fmt.Printf("TODO: webrtc ontrack %+v\n", remote)
+		//fmt.Printf("TODO: webrtc ontrack %+v\n", remote)
 	})

 	// OK connection:
@@ -100,12 +98,23 @@ func (c *Conn) SetOffer(offer string) (err error) {
 		return
 	}
 	rawSDP := []byte(c.Conn.RemoteDescription().SDP)
-	c.medias, err = streamer.UnmarshalSDP(rawSDP)
+	medias, err := streamer.UnmarshalSDP(rawSDP)
+	if err != nil {
+		return
+	}

 	// sort medias, so video will always be before audio
-	sort.Slice(c.medias, func(i, j int) bool {
-		return c.medias[i].Kind == streamer.KindVideo
-	})
+	// and ignore application media from Hass default lovelace card
+	for _, media := range medias {
+		if media.Kind == streamer.KindVideo {
+			c.medias = append(c.medias, media)
+		}
+	}
+	for _, media := range medias {
+		if media.Kind == streamer.KindAudio {
+			c.medias = append(c.medias, media)
+		}
+	}

 	return
 }
--- a/www/mse.html
+++ b/www/mse.html
@@ -3,97 +3,26 @@
 <head>
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>go2rtc - MSE</title>
+    <script src="video-rtc.js"></script>
    <style>
        body {
+            background: black;
            margin: 0;
            padding: 0;
        }

-        html, body {
+        html, body, video {
            height: 100%;
            width: 100%;
        }
-
-        #video {
-            width: 100%;
-            height: 100%;
-            background: black;
-        }
    </style>
 </head>
 <body>
-<!-- muted is important for autoplay -->
-<video id="video" autoplay controls playsinline muted></video>
 <script>
-    // support api_path
-    const baseUrl = location.origin + location.pathname.substr(
-        0, location.pathname.lastIndexOf("/")
-    );
-    const video = document.querySelector('#video');
-
-    function init() {
-        let mediaSource, sourceBuffer, queueBuffer = [];
-
-        const ws = new WebSocket(`ws${baseUrl.substr(4)}/api/ws${location.search}`);
-        ws.binaryType = "arraybuffer";
-
-        ws.onopen = () => {
-            mediaSource = new MediaSource();
-            video.src = URL.createObjectURL(mediaSource);
-            mediaSource.onsourceopen = () => {
-                mediaSource.onsourceopen = null;
-                URL.revokeObjectURL(video.src);
-                ws.send(JSON.stringify({"type": "mse"}));
-            };
-        };
-
-        ws.onmessage = ev => {
-            if (typeof ev.data === 'string') {
-                const data = JSON.parse(ev.data);
-                console.debug("ws.onmessage", data);
-
-                if (data.type === "mse") {
-                    sourceBuffer = mediaSource.addSourceBuffer(data.value);
-                    sourceBuffer.mode = "segments"; // segments or sequence
-                    sourceBuffer.onupdateend = () => {
-                        if (!sourceBuffer.updating && queueBuffer.length > 0) {
-                            try {
-                                sourceBuffer.appendBuffer(queueBuffer.shift());
-                            } catch (e) {
-                                // console.warn(e);
-                            }
-                        }
-                    }
-                }
-            } else if (sourceBuffer.updating || queueBuffer.length > 0) {
-                queueBuffer.push(ev.data);
-            } else {
-                try {
-                    sourceBuffer.appendBuffer(ev.data);
-                } catch (e) {
-                    // console.warn(e);
-                }
-            }
-
-            if (video.seekable.length > 0) {
-                const delay = video.seekable.end(video.seekable.length - 1) - video.currentTime;
-                if (delay < 1) {
-                    video.playbackRate = 1;
-                } else if (delay > 10) {
-                    video.playbackRate = 10;
-                } else if (delay > 2) {
-                    video.playbackRate = Math.floor(delay);
-                }
-            }
-        }
-
-        video.onpause = () => {
-            ws.close();
-            setTimeout(init, 0);
-        }
-    }
-
-    init();
+    const url = new URL("api/ws" + location.search, location.href);
+    const video = document.createElement("video-rtc");
+    video.src = url.toString();
+    document.body.appendChild(video);
 </script>
 </body>
 </html>
--- a/www/video-rtc.js
+++ b/www/video-rtc.js
@@ -0,0 +1,374 @@
+/**
+ * Common function for processing MSE and MSE2 data.
+ * @param ms MediaSource
+ */
+function MediaSourceHandler(ms) {
+    let sb, qb = [];
+
+    return ev => {
+        if (typeof ev.data === "string") {
+            const msg = JSON.parse(ev.data);
+            if (msg.type === "mse") {
+                if (!MediaSource.isTypeSupported(msg.value)) {
+                    console.warn("Not supported: " + msg.value)
+                    return;
+                }
+
+                sb = ms.addSourceBuffer(msg.value);
+                sb.mode = "segments"; // segments or sequence
+                sb.addEventListener("updateend", () => {
+                    if (!sb.updating && qb.length > 0) {
+                        try {
+                            sb.appendBuffer(qb.shift());
+                        } catch (e) {
+                            // console.warn(e);
+                        }
+                    }
+                });
+            }
+        } else if (sb.updating || qb.length > 0) {
+            qb.push(ev.data);
+            // console.debug("buffer:", qb.length);
+        } else {
+            try {
+                sb.appendBuffer(ev.data);
+            } catch (e) {
+                // console.warn(e);
+            }
+        }
+    }
+}
+
+/**
+ * Dedicated Worker Handler for MSE2 https://chromestatus.com/feature/5177263249162240
+ */
+if (typeof importScripts == "function") {
+    // protect below code (class VideoRTC) from fail inside Worker
+    HTMLElement = Object;
+    customElements = {define: Function()};
+
+    const ms = new MediaSource();
+    ms.addEventListener("sourceopen", ev => {
+        postMessage({type: ev.type});
+    }, {once: true});
+
+    onmessage = MediaSourceHandler(ms);
+
+    postMessage({type: "handle", value: ms.handle}, [ms.handle]);
+}
+
+/**
+ * Video player for MSE and WebRTC connections.
+ *
+ * All modern web technologies are supported in almost any browser except Apple Safari.
+ *
+ * Support:
+ * - RTCPeerConnection for Safari iOS 11.0+
+ * - IntersectionObserver for Safari iOS 12.2+
+ * - MediaSource in Workers for Chrome 108+
+ *
+ * Doesn't support:
+ * - MediaSource for Safari iOS all
+ * - Customized built-in elements (extends HTMLVideoElement) because all Safari
+ */
+class VideoRTC extends HTMLElement {
+    DISCONNECT_TIMEOUT = 5000;
+    RECONNECT_TIMEOUT = 30000;
+
+    CODECS = [
+        "avc1.640029",      // H.264 high 4.1 (Chromecast 1st and 2nd Gen)
+        "avc1.64002A",      // H.264 high 4.2 (Chromecast 3rd Gen)
+        "hvc1.1.6.L153.B0", // H.265 main 5.1 (Chromecast Ultra)
+        "mp4a.40.2",        // AAC LC
+        "mp4a.40.5",        // AAC HE
+        "mp4a.69",          // MP3
+        "mp4a.6B",          // MP3
+    ];
+
+    /**
+     * Enable MediaSource in Workers mode.
+     * @type {boolean}
+     */
+    MSE2 = true;
+
+    /**
+     * Run stream when not displayed on the screen. Default `false`.
+     * @type {boolean}
+     */
+    background = false;
+
+    /**
+     * Run stream only when player in the viewport. Stop when user scroll out player.
+     * Value is percentage of visibility from `0` (not visible) to `1` (full visible).
+     * Default `0` - disable;
+     * @type {number}
+     */
+    intersectionThreshold = 0;
+
+    /**
+     * Run stream only when browser page on the screen. Stop when user change browser
+     * tab or minimise browser windows.
+     * @type {boolean}
+     */
+    visibilityCheck = true;
+
+    /**
+     * @type {HTMLVideoElement}
+     */
+    video = null;
+
+    /**
+     * @type {RTCPeerConnection}
+     */
+    pc = null;
+
+    /**
+     * @type {WebSocket}
+     */
+    ws = null;
+
+    /**
+     * Internal WebSocket connection state. Values: CLOSED, CONNECTING, OPEN
+     * @type {number}
+     */
+    wsState = WebSocket.CLOSED;
+
+    /**
+     * Internal WebSocket URL.
+     * @type {string}
+     */
+    wsURL = "";
+
+    /**
+     * Internal disconnect TimeoutID.
+     * @type {number}
+     */
+    disconnectTimeout = 0;
+
+    /**
+     * Internal reconnect TimeoutID.
+     * @type {number}
+     */
+    reconnectTimeout = 0;
+
+    constructor() {
+        super();
+
+        console.debug("this.constructor");
+
+        this.video = document.createElement("video");
+        this.video.controls = true;
+        this.video.playsInline = true;
+    }
+
+    /** public properties **/
+
+    /**
+     * Set video source (WebSocket URL). Support relative path.
+     * @param value
+     */
+    set src(value) {
+        if (value.startsWith("/")) {
+            value = "ws" + location.origin.substr(4) + value;
+        } else if (value.startsWith("http")) {
+            value = "ws" + value.substr(4);
+        }
+
+        this.wsURL = value;
+
+        if (this.isConnected) this.connectedCallback();
+    }
+
+    /**
+     * Play video. Support automute when autoplay blocked.
+     * https://developer.chrome.com/blog/autoplay/
+     */
+    play() {
+        this.video.play().catch(er => {
+            if (er.name === "NotAllowedError" && !this.video.muted) {
+                this.video.muted = true;
+                this.video.play();
+            }
+        });
+    }
+
+    get codecs() {
+        return this.CODECS.filter(value => {
+            return MediaSource.isTypeSupported(`video/mp4; codecs="${value}"`);
+        }).join();
+    }
+
+    /**
+     * `CustomElement`. Invoked each time the custom element is appended into a
+     * document-connected element.
+     */
+    connectedCallback() {
+        console.debug("this.connectedCallback", this.wsState);
+        if (this.disconnectTimeout) {
+            clearTimeout(this.disconnectTimeout);
+            this.disconnectTimeout = 0;
+        }
+
+        // because video autopause on disconnected from DOM
+        const seek = this.video.seekable;
+        if (seek.length > 0) {
+            this.video.currentTime = seek.end(seek.length - 1);
+            this.play();
+        }
+
+        if (!this.wsURL || this.wsState !== WebSocket.CLOSED) return;
+
+        // CLOSED => CONNECTING
+        this.wsState = WebSocket.CONNECTING;
+
+        this.internalInit();
+        this.internalConnect();
+    }
+
+    /**
+     * `CustomElement`. Invoked each time the custom element is disconnected from the
+     * document's DOM.
+     */
+    disconnectedCallback() {
+        console.debug("this.disconnectedCallback", this.wsState);
+        if (this.background || this.disconnectTimeout ||
+            this.wsState === WebSocket.CLOSED) return;
+
+        this.disconnectTimeout = setTimeout(() => {
+            if (this.reconnectTimeout) {
+                clearTimeout(this.reconnectTimeout);
+                this.reconnectTimeout = 0;
+            }
+
+            this.disconnectTimeout = 0;
+            // CONNECTING, OPEN => CLOSED
+            this.wsState = WebSocket.CLOSED;
+
+            if (this.ws) {
+                this.ws.close();
+                this.ws = null;
+            }
+        }, this.DISCONNECT_TIMEOUT);
+    }
+
+    internalInit() {
+        if (this.childElementCount) return;
+
+        this.appendChild(this.video);
+
+        if (this.background) return;
+
+        if ("hidden" in document && this.visibilityCheck) {
+            document.addEventListener("visibilitychange", () => {
+                if (document.hidden) {
+                    this.disconnectedCallback();
+                } else if (this.isConnected) {
+                    this.connectedCallback();
+                }
+            })
+        }
+
+        if ("IntersectionObserver" in window && this.intersectionThreshold) {
+            const observer = new IntersectionObserver(entries => {
+                entries.forEach(entry => {
+                    if (!entry.isIntersecting) {
+                        this.disconnectedCallback();
+                    } else if (this.isConnected) {
+                        this.connectedCallback();
+                    }
+                });
+            }, {threshold: this.intersectionThreshold});
+            observer.observe(this);
+        }
+    }
+
+    internalConnect() {
+        if (this.wsState !== WebSocket.CONNECTING) return;
+        if (this.ws) throw "connect with non null WebSocket";
+
+        const ts = Date.now();
+
+        this.ws = new WebSocket(this.wsURL);
+        this.ws.binaryType = "arraybuffer";
+
+        this.ws.addEventListener("open", () => {
+            console.debug("ws.open", this.wsState);
+            if (this.wsState !== WebSocket.CONNECTING) return;
+
+            // CONNECTING => OPEN
+            this.wsState = WebSocket.OPEN;
+        });
+        this.ws.addEventListener("close", () => {
+            console.debug("ws.close", this.wsState);
+            if (this.wsState === WebSocket.CLOSED) return;
+
+            // CONNECTING, OPEN => CONNECTING
+            this.wsState = WebSocket.CONNECTING;
+            this.ws = null;
+
+            // reconnect no more than once every X seconds
+            const delay = Math.max(this.RECONNECT_TIMEOUT - (Date.now() - ts), 0);
+
+            this.reconnectTimeout = setTimeout(() => {
+                this.reconnectTimeout = 0;
+                this.internalConnect();
+            }, delay);
+        });
+
+        if ("MediaSource" in window && this.MSE2) {
+            if (MediaSource.canConstructInDedicatedWorker) {
+                this.internalMSE2();
+            } else {
+                this.internalMSE();
+            }
+        }
+
+        // TODO: this.internalRTC();
+    }
+
+    internalMSE() {
+        console.debug("this.internalMSE");
+        this.ws.addEventListener("open", () => {
+            const ms = new MediaSource();
+            ms.addEventListener("sourceopen", () => {
+                URL.revokeObjectURL(this.video.src);
+                this.ws.send(JSON.stringify({type: "mse", value: this.codecs}));
+            }, {once: true});
+
+            this.video.src = URL.createObjectURL(ms);
+            this.play();
+
+            this.ws.addEventListener("message", MediaSourceHandler(ms));
+        });
+    }
+
+    internalMSE2() {
+        console.debug("this.internalMSE2");
+        const worker = new Worker("video-rtc.js");
+        worker.addEventListener("message", ev => {
+            if (ev.data.type === "handle") {
+                this.video.srcObject = ev.data.value;
+                this.play();
+            } else if (ev.data.type === "sourceopen") {
+                this.ws.send(JSON.stringify({type: "mse", value: this.codecs}));
+            }
+        });
+
+        this.ws.addEventListener("message", ev => {
+            if (typeof ev.data === "string") {
+                worker.postMessage(ev.data);
+            } else {
+                worker.postMessage(ev.data, [ev.data]);
+            }
+        });
+        this.ws.addEventListener("close", () => {
+            worker.terminate();
+        });
+    }
+
+    internalRTC() {
+        if (!("RTCPeerConnection" in window)) return; // macOS Desktop app
+    }
+}
+
+customElements.define("video-rtc", VideoRTC);
Author	SHA1	Message	Date
Alexey Khit	fae59c7992	Update version to 0.1-rc.4	2022-11-24 02:00:31 +03:00
Alexey Khit	aff34f1d21	Totally rewritten MSE player	2022-11-24 01:59:48 +03:00
Alexey Khit	65e7efa775	Support codecs negotiation for MSE	2022-11-23 21:45:10 +03:00
Alexey Khit	3c3e9d282b	Update about H265 support in readme	2022-11-23 20:35:11 +03:00
Alexey Khit	bd51069086	Add support CORS for API	2022-11-23 20:34:06 +03:00
Alexey Khit	1ddf7f1a6c	Change trace log for stream.mp4	2022-11-23 12:57:11 +03:00
Alexey Khit	0e281e36d3	Fix race (concurency) for Track	2022-11-22 20:03:36 +03:00
Alexey Khit	3d6472cfb1	Update H265 preset for FFmpeg	2022-11-22 17:22:54 +03:00
Alexey Khit	7c31fa2ffd	Fix empty SPS for MSE H265	2022-11-22 17:22:26 +03:00
Alexey Khit	0ed9d2410a	Fix H265 support for MSE in Safari	2022-11-22 17:21:58 +03:00
Alexey Khit	1c89e7945e	Remove printf for webrtc ontrack	2022-11-18 09:13:24 +03:00
Alexey Khit	48635ae341	Add two locks for Track	2022-11-18 09:12:48 +03:00
Alexey Khit	fdb316910f	Fix WebRTC async connection	2022-11-16 11:26:56 +03:00
Alexey Khit	e29f2594fa	Fix multiple transcoding when track not exists	2022-11-15 16:16:22 +03:00
Alexey Khit	c3da7584b0	Add check on RTSP server requers with empty url path	2022-11-14 19:06:43 +03:00
Alexey Khit	1e247cba92	Igroner app media for WebRTC from Hass	2022-11-14 17:26:59 +03:00
Alexey Khit	01631d9eb0	Update readme	2022-11-14 14:57:43 +03:00