Bring back x11 adapter for Linux build only

Original x11 adapter uses ~40% less CPU usage than kbinani/screenshot. Ideally, we should migrate 100% to kbinani/screenshot. But, the CPU usage difference is substantial. In the future, we should look deeper into kbinani/screenshot and try to improve it.
Revert "Create codeql-analysis.yml"
2025-09-27 04:46:10 +08:00 · 2020-11-23 20:36:17 -05:00 · 2020-11-21 14:38:31 -08:00 · 2020-11-21 14:11:35 -08:00 · 2020-11-21 14:11:00 -08:00 · 2020-11-21 14:07:32 -08:00
19 changed files with 431 additions and 92 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -1 +0,0 @@
-*.gif filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -13,11 +13,7 @@
 </p>
 <br>

-[MediaDevices](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices) provides access to connected media input devices like cameras and microphones, as well as screen sharing. It can also be used to encode your video/audio stream to various codec selections. 
-
-The focus of the project has been to seek out a **simple** and **elegant design** for writing media pipelines.
-
-![](img/demo.gif)
+`mediadevices` provides access to media input devices like cameras, microphones, and screen capture. It can also be used to encode your video/audio stream to various codec selections. `mediadevices` abstracts away the complexities of interacting with things like hardware and codecs allowing you to focus on building appilcations, interacting only with an amazingly simple, easy, and elegant API!

 ## Install

@@ -81,6 +77,7 @@ func main() {
 * [Face Detection](/examples/facedetection) - Use a machine learning algorithm to detect faces in a camera stream
 * [RTP Stream](examples/rtp) - Capture camera stream, encode it in H264/VP8/VP9, and send it to a RTP server
 * [HTTP Broadcast](/examples/http) - Broadcast camera stream through HTTP with MJPEG
+* [Archive](/examples/archive) - Archive H264 encoded video stream from a camera

 ## Available Media Inputs

@@ -88,10 +85,51 @@ func main() {
 | :--------: | :---: | :-: | :-----: |
 |   Camera   |  ✔️   | ✔️  |   ✔️    |
 | Microphone |  ✔️   | ✔️  |   ✔️    |
-|   Screen   |  ✔️   | ✖️  |   ✖️    |
+|   Screen   |  ✔️   | ✔️  |   ✔️    |
+
+By default, there's no media input registered. This decision was made to allow you to pay what you need. Therefore, you need to import the associated packages for the media inputs. For example, if you want to use a camera, you need to import the camera package as a side effect:
+
+```go
+import (
+	...
+	_ "github.com/pion/mediadevices/pkg/driver/camera"
+)
+```

 ## Available Codecs

+In order to encode your video/audio, `mediadevices` needs to know what codecs that you want to use and their parameters. To do this, you need to import the associated packages for the codecs, and add them to the codec selector that you'll pass to `GetUserMedia`:
+
+```go
+package main
+
+import (
+	"github.com/pion/mediadevices"
+	"github.com/pion/mediadevices/pkg/codec/x264"      // This is required to use H264 video encoder
+	_ "github.com/pion/mediadevices/pkg/driver/camera" // This is required to register camera adapter
+)
+
+func main() {
+	// configure codec specific parameters
+	x264Params, _ := x264.NewParams()
+	x264Params.Preset = x264.PresetMedium
+	x264Params.BitRate = 1_000_000 // 1mbps
+
+	codecSelector := mediadevices.NewCodecSelector(
+		mediadevices.WithVideoEncoders(&x264Params),
+	)
+	
+	mediaStream, _ := mediadevices.GetUserMedia(mediadevices.MediaStreamConstraints{
+		Video: func(c *mediadevices.MediaTrackConstraints) {},
+		Codec: codecSelector, // let GetUsermedia know available codecs
+	})
+}
+```
+
+Since `mediadevices` doesn't implement the video/audio codecs, it needs to call the codec libraries from the system through cgo. Therefore, you're required to install the codec libraries before you can use them in `mediadevices`. In the next section, it shows a list of available codecs, where the packages are defined (documentation linked), and installation instructions.
+
+Note: we do not provide recommendations on choosing one codec or another as it is very complex and can be subjective.
+
 ### Video Codecs

 #### x264
@@ -106,8 +144,7 @@ A free software library and application for encoding video streams into the H.26
 A framework to enable H264 hardware encoding for Raspberry Pi or boards that use VideoCore GPUs.

 * Package: [github.com/pion/mediadevices/pkg/codec/mmal](https://pkg.go.dev/github.com/pion/mediadevices/pkg/codec/mmal)
-* Installation:
-  * Raspbian: `export PKG_CONFIG_PATH=/opt/vc/lib/pkgconfig`
+* Installation: no installation needed, mmal should come built in Raspberry Pi devices

 #### openh264
 A codec library which supports H.264 encoding and decoding. It is suitable for use in real time applications.
@@ -143,28 +180,28 @@ A totally open, royalty-free, highly versatile audio codec.

 ## Benchmark

-Result as of Nov 4, 2020 with Go 1.14 on a Raspberry pi 3, `mediadevices` can produce a **720p at 30 fps with <500ms latency video**.  
+Result as of Nov 4, 2020 with Go 1.14 on a Raspberry pi 3, `mediadevices` can produce video, encode, send across network, and decode at **720p, 30 fps with < 500 ms latency**.  

-The test was taken by capturing a camera stream, decode raw frames, encode the video stream with mmal to H264, and send the stream through Webrtc.
+The test was taken by capturing a camera stream, decoding the raw frames, encoding the video stream with mmal, and sending the stream through Webrtc.

 ## FAQ

 ### Failed to find the best driver that fits the constraints

-`mediadevices` provides an automated driver discovery through `GetUserMedia` and `GetDisplayMedia`. In an oversimplified explanation, the discovery algorithm as followed:
+`mediadevices` provides an automated driver discovery through `GetUserMedia` and `GetDisplayMedia`. The driver discover algorithm works something like:

 1. Open all registered drivers
 2. Get all properties (property describes what a driver is capable of, e.g. resolution, frame rate, etc.) from opened drivers
 3. Find the best property that meets the criteria

 So, when `mediadevices` returns `failed to find the best driver that fits the constraints` error, one of the following conditions might have occured:
-* In your program, the driver has never been imported as a side effect, e.g. `import _ github.com/pion/mediadevices/pkg/driver/camera`
-* Your constraint is too strict that there's no driver can fullfil your requirements. In this case, you can try to turn up the debug level by specifying the following environment variable: `export PION_LOG_DEBUG=all`
-* Your driver is not supported/implemented. In this case, you can either wait for the maintainers to implement it. Or, you can implement it yourself and register it through `RegisterDriverAdapter`
+* Driver was not imported as a side effect in your program, e.g. `import _ github.com/pion/mediadevices/pkg/driver/camera`
+* Your constraint is too strict that there's no driver can fullfil your requirements. In this case, you can try to turn up the debug level by specifying the following environment variable: `export PION_LOG_DEBUG=all` to see what was too strict and tune that.
+* Your driver is not supported/implemented. In this case, you can either let us know (file an issue) and wait for the maintainers to implement it. Or, you can implement it yourself and register it through `RegisterDriverAdapter`

 ### Failed to find vpx/x264/mmal/opus codecs

-Since `mediadevices` uses cgo to access video/audio codecs, it needs to find these libraries from the system. To do that, `mediadevices` uses [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/) for library discovery.
+Since `mediadevices` uses cgo to access video/audio codecs, it needs to find these libraries from the system. To accomplish this, [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/) is used for library discovery.

 If you see the following error message at compile time:
 ```
--- a/examples/archive/README.md
+++ b/examples/archive/README.md
@@ -0,0 +1,38 @@
+## Instructions
+
+### Install required codecs
+
+In this example, we'll be using x264 as our video codec. Therefore, we need to make sure that these codecs are installed within our system. 
+
+Installation steps:
+
+* [x264](https://github.com/pion/mediadevices#x264)
+
+### Download archive examplee
+
+```
+git clone https://github.com/pion/mediadevices.git
+```
+
+### Run archive example
+
+Run `cd mediadevices/examples/archive && go build && ./archive recorded.h264`
+
+To stop recording, press `Ctrl+c` or send a SIGINT signal.
+
+### Playback recorded video
+
+Install GStreamer and run:
+```
+gst-launch-1.0 playbin uri=file://${PWD}/recorded.h264
+```
+
+Or run VLC media plyer:
+```
+vlc recorded.h264
+```
+
+A video should start playing in your GStreamer or VLC window.
+
+Congrats, you have used pion-MediaDevices! Now start building something cool
+
--- a/examples/archive/archive
+++ b/examples/archive/archive
--- a/examples/archive/main.go
+++ b/examples/archive/main.go
@@ -0,0 +1,82 @@
+package main
+
+import (
+	"fmt"
+	"image"
+	"io"
+	"os"
+	"os/signal"
+	"syscall"
+
+	"github.com/pion/mediadevices"
+	"github.com/pion/mediadevices/pkg/codec/x264"      // This is required to use H264 video encoder
+	_ "github.com/pion/mediadevices/pkg/driver/camera" // This is required to register camera adapter
+	"github.com/pion/mediadevices/pkg/frame"
+	"github.com/pion/mediadevices/pkg/io/video"
+	"github.com/pion/mediadevices/pkg/prop"
+)
+
+func must(err error) {
+	if err != nil {
+		panic(err)
+	}
+}
+
+func main() {
+	if len(os.Args) != 2 {
+		fmt.Printf("usage: %s <path/to/file.h264>\n", os.Args[0])
+		return
+	}
+	dest := os.Args[1]
+
+	sigs := make(chan os.Signal, 1)
+	signal.Notify(sigs, syscall.SIGINT)
+
+	x264Params, err := x264.NewParams()
+	must(err)
+	x264Params.Preset = x264.PresetMedium
+	x264Params.BitRate = 1_000_000 // 1mbps
+
+	codecSelector := mediadevices.NewCodecSelector(
+		mediadevices.WithVideoEncoders(&x264Params),
+	)
+
+	mediaStream, err := mediadevices.GetUserMedia(mediadevices.MediaStreamConstraints{
+		Video: func(c *mediadevices.MediaTrackConstraints) {
+			c.FrameFormat = prop.FrameFormat(frame.FormatYUY2)
+			c.Width = prop.Int(640)
+			c.Height = prop.Int(480)
+		},
+		Codec: codecSelector,
+	})
+	must(err)
+
+	videoTrack := mediaStream.GetVideoTracks()[0].(*mediadevices.VideoTrack)
+	defer videoTrack.Close()
+
+	videoTrack.Transform(video.TransformFunc(func(r video.Reader) video.Reader {
+		return video.ReaderFunc(func() (img image.Image, release func(), err error) {
+			// we send io.EOF signal to the encoder reader to stop reading. Therefore, io.Copy
+			// will finish its execution and the program will finish
+			select {
+			case <-sigs:
+				return nil, func() {}, io.EOF
+			default:
+			}
+
+			return r.Read()
+		})
+	}))
+
+	reader, err := videoTrack.NewEncodedReader(x264Params.RTPCodec().Name)
+	must(err)
+	defer reader.Close()
+
+	out, err := os.Create(dest)
+	must(err)
+
+	fmt.Println("Recording... Press Ctrl+c to stop")
+	_, err = io.Copy(out, reader)
+	must(err)
+	fmt.Println("Your video has been recorded to", dest)
+}
--- a/examples/rtp/README.md
+++ b/examples/rtp/README.md
@@ -2,17 +2,16 @@

 ### Install required codecs

-In this example, we'll be using x264 and opus as our video and audio codecs. Therefore, we need to make sure that these codecs are installed within our system. 
+In this example, we'll be using x264 as our video codec. Therefore, we need to make sure that these codecs are installed within our system. 

 Installation steps:

 * [x264](https://github.com/pion/mediadevices#x264)
-* [opus](https://github.com/pion/mediadevices#opus)

-### Download rtpexample
+### Download rtp example

 ```
-go get github.com/pion/mediadevices/examples/rtp
+git clone https://github.com/pion/mediadevices.git
 ```

 ### Listen RTP
@@ -30,7 +29,7 @@ vlc ./h264.sdp

 ### Run rtp

-Run `rtp localhost:5000`
+Run `cd mediadevices/examples/archive && go build && ./rtp localhost:5000`

 A video should start playing in your GStreamer or VLC window.
 It's not WebRTC, but pure RTP.
--- a/examples/webrtc/README.md
+++ b/examples/webrtc/README.md
@@ -39,4 +39,4 @@ Copy the text that `./webrtc` just emitted and copy into second text area

 A video should start playing in your browser above the input boxes, and will continue playing until you close the application.

-Congrats, you have used pion-WebRTC! Now start building something cool
+Congrats, you have used pion-MediaDevices! Now start building something cool
--- a/go.mod
+++ b/go.mod
@@ -3,11 +3,15 @@ module github.com/pion/mediadevices
 go 1.13

 require (
+	github.com/BurntSushi/xgb v0.0.0-20201008132610-5f9e7b3c49cd // indirect
 	github.com/blackjack/webcam v0.0.0-20200313125108-10ed912a8539
-	github.com/gen2brain/malgo v0.10.19
+	github.com/gen2brain/malgo v0.10.25
+	github.com/gen2brain/shm v0.0.0-20200228170931-49f9650110c5 // indirect
+	github.com/kbinani/screenshot v0.0.0-20191211154542-3a185f1ce18f
 	github.com/lherman-cs/opus v0.0.2
+	github.com/lxn/win v0.0.0-20201111105847-2a20daff6a55 // indirect
 	github.com/pion/logging v0.2.2
-	github.com/pion/rtp v1.6.0
+	github.com/pion/rtp v1.6.1
 	github.com/pion/webrtc/v2 v2.2.26
 	github.com/satori/go.uuid v1.2.0
 	golang.org/x/image v0.0.0-20200927104501-e162460cd6b5
--- a/go.sum
+++ b/go.sum
@@ -1,3 +1,5 @@
+github.com/BurntSushi/xgb v0.0.0-20201008132610-5f9e7b3c49cd h1:u7K2oMFMd8APDV3fM1j2rO3U/XJf1g1qC3DDTKou8iM=
+github.com/BurntSushi/xgb v0.0.0-20201008132610-5f9e7b3c49cd/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
 github.com/blackjack/webcam v0.0.0-20200313125108-10ed912a8539 h1:1aIqYfg9s9RETAJHGfVKZW4ok0b22p4QTwk8MsdRtPs=
 github.com/blackjack/webcam v0.0.0-20200313125108-10ed912a8539/go.mod h1:G0X+rEqYPWSq0dG8OMf8M446MtKytzpPjgS3HbdOJZ4=
 github.com/cheekybits/genny v1.0.0 h1:uGGa4nei+j20rOSeDeP5Of12XVm7TGUd4dJA9RDitfE=
@@ -7,8 +9,10 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
-github.com/gen2brain/malgo v0.10.19 h1:IUVF6WdVV7Txt47Kx2ajz0rWQ0MU0zO+tbcKmhva7l8=
-github.com/gen2brain/malgo v0.10.19/go.mod h1:zHSUNZAXfCeNsZou0RtQ6Zk7gDYLIcKOrUWtAdksnEs=
+github.com/gen2brain/malgo v0.10.25 h1:VRiYTBmBeHTCXD0wCg7XyLi6lJJBqND/XVmSEyrGkGc=
+github.com/gen2brain/malgo v0.10.25/go.mod h1:zHSUNZAXfCeNsZou0RtQ6Zk7gDYLIcKOrUWtAdksnEs=
+github.com/gen2brain/shm v0.0.0-20200228170931-49f9650110c5 h1:Y5Q2mEwfzjMt5+3u70Gtw93ZOu2UuPeeeTBDntF7FoY=
+github.com/gen2brain/shm v0.0.0-20200228170931-49f9650110c5/go.mod h1:uF6rMu/1nvu+5DpiRLwusA6xB8zlkNoGzKn8lmYONUo=
 github.com/golang/mock v1.2.0 h1:28o5sBqPkBsMGnC6b4MvE2TzSr5/AT4c/1fLqVGIwlk=
 github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
 github.com/golang/protobuf v1.2.0 h1:P3YflyNX/ehuJFLhxviNdFxQPkGK5cDcApsge1SqnvM=
@@ -17,6 +21,8 @@ github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
 github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI=
 github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
+github.com/kbinani/screenshot v0.0.0-20191211154542-3a185f1ce18f h1:5hWo+DzJQSOBl6X+TDac0SPWffRonuRJ2///OYtYRT8=
+github.com/kbinani/screenshot v0.0.0-20191211154542-3a185f1ce18f/go.mod h1:f8GY5V3lRzakvEyr49P7hHRYoHtPr8zvj/7JodCoRzw=
 github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
 github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
@@ -26,6 +32,8 @@ github.com/lherman-cs/opus v0.0.2 h1:fE9Du3NKXDBztqvoTd6P2y9eJ9vgIHahGK8yQostnhA
 github.com/lherman-cs/opus v0.0.2/go.mod h1:v9KQvlDYMuvlwniumBVMlrB0VHQvyTgxNvaXjPmTmps=
 github.com/lucas-clemente/quic-go v0.7.1-0.20190401152353-907071221cf9 h1:tbuodUh2vuhOVZAdW3NEUvosFHUMJwUNl7jk/VSEiwc=
 github.com/lucas-clemente/quic-go v0.7.1-0.20190401152353-907071221cf9/go.mod h1:PpMmPfPKO9nKJ/psF49ESTAGQSdfXxlg1otPbEB2nOw=
+github.com/lxn/win v0.0.0-20201111105847-2a20daff6a55 h1:4BxFx5XCtXc+nFtXDGDW+Uu5sPtsAbvPh6RObj3fG9o=
+github.com/lxn/win v0.0.0-20201111105847-2a20daff6a55/go.mod h1:KxxjdtRkfNoYDCUP5ryK7XJJNTnpC8atvtmTheChOtk=
 github.com/marten-seemann/qtls v0.2.3 h1:0yWJ43C62LsZt08vuQJDK1uC1czUc3FJeCLPoNAI4vA=
 github.com/marten-seemann/qtls v0.2.3/go.mod h1:xzjG7avBwGGbdZ8dTGxlBnLArsVKLvwmjgmPuiQEcYk=
 github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
@@ -53,6 +61,8 @@ github.com/pion/rtcp v1.2.3 h1:2wrhKnqgSz91Q5nzYTO07mQXztYPtxL8a0XOss4rJqA=
 github.com/pion/rtcp v1.2.3/go.mod h1:zGhIv0RPRF0Z1Wiij22pUt5W/c9fevqSzT4jje/oK7I=
 github.com/pion/rtp v1.6.0 h1:4Ssnl/T5W2LzxHj9ssYpGVEQh3YYhQFNVmSWO88MMwk=
 github.com/pion/rtp v1.6.0/go.mod h1:QgfogHsMBVE/RFNno467U/KBqfUywEH+HK+0rtnwsdI=
+github.com/pion/rtp v1.6.1 h1:2Y2elcVBrahYnHKN2X7rMHX/r1R4TEBMP1LaVu/wNhk=
+github.com/pion/rtp v1.6.1/go.mod h1:bDb5n+BFZxXx0Ea7E5qe+klMuqiBrP+w8XSjiWtCUko=
 github.com/pion/sctp v1.7.10 h1:o3p3/hZB5Cx12RMGyWmItevJtZ6o2cpuxaw6GOS4x+8=
 github.com/pion/sctp v1.7.10/go.mod h1:EhpTUQu1/lcK3xI+eriS6/96fWetHGCvBi9MSsnaBN0=
 github.com/pion/sdp/v2 v2.4.0 h1:luUtaETR5x2KNNpvEMv/r4Y+/kzImzbz4Lm1z8eQNQI=
@@ -109,6 +119,7 @@ golang.org/x/sys v0.0.0-20190228124157-a34e9553db1e/go.mod h1:STP8DvDyc/dI5b8T5h
 golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200724161237-0e2f3a69832c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20201018230417-eeed37f84f13/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20201029080932-201ba4db2418 h1:HlFl4V6pEMziuLXyRkm5BIYq1y1GAbb02pRlWvI54OM=
 golang.org/x/sys v0.0.0-20201029080932-201ba4db2418/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
--- a/img/demo.gif
+++ b/img/demo.gif
--- a/ioreader.go
+++ b/ioreader.go
@@ -0,0 +1,14 @@
+package mediadevices
+
+type encodedReadCloserImpl struct {
+	readFn  func([]byte) (int, error)
+	closeFn func() error
+}
+
+func (r *encodedReadCloserImpl) Read(b []byte) (int, error) {
+	return r.readFn(b)
+}
+
+func (r *encodedReadCloserImpl) Close() error {
+	return r.closeFn()
+}
--- a/mediastream_test.go
+++ b/mediastream_test.go
@@ -1,6 +1,7 @@
 package mediadevices

 import (
+	"io"
 	"testing"

 	"github.com/pion/webrtc/v2"
@@ -37,6 +38,10 @@ func (track *mockMediaStreamTrack) NewRTPReader(codecName string, mtu int) (RTPR
 	return nil, nil
 }

+func (track *mockMediaStreamTrack) NewEncodedReader(codecName string) (io.ReadCloser, error) {
+	return nil, nil
+}
+
 func TestMediaStreamFilters(t *testing.T) {
 	audioTracks := []Track{
 		&mockMediaStreamTrack{AudioInput},
--- a/pkg/codec/mmal/mmal.go
+++ b/pkg/codec/mmal/mmal.go
@@ -3,7 +3,8 @@
 // Reference: https://github.com/raspberrypi/userland/tree/master/interface/mmal
 package mmal

-// #cgo pkg-config: mmal
+// #cgo CFLAGS: -I/opt/vc/include
+// #cgo LDFLAGS: -L/opt/vc/lib -lmmal -lmmal_core -lmmal_util -lmmal_vc_client -lbcm_host -lvcsm -lvcos
 // #include "bridge.h"
 import "C"
 import (
--- a/pkg/codec/vpx/vpx.go
+++ b/pkg/codec/vpx/vpx.go
@@ -237,10 +237,19 @@ func (e *encoder) Read() ([]byte, func(), error) {
 		e.raw.d_w, e.raw.d_h = C.uint(width), C.uint(height)
 	}

+	duration := t - e.tLastFrame
+	// VPX doesn't allow 0 duration. If 0 is given, vpx_codec_encode will fail with VPX_CODEC_INVALID_PARAM.
+	// 0 duration is possible because mediadevices first gets the frame meta data by reading from the source,
+	// and consequently the codec will read the first frame from the buffer. This means the first frame won't
+	// have a pause to the second frame, which means if the delay is <1 ms (vpx duration resolution), duration
+	// is going to be 0.
+	if duration == 0 {
+		duration = 1
+	}
 	var flags int
 	if ec := C.encode_wrapper(
 		e.codec, e.raw,
-		C.long(t-e.tStart), C.ulong(t-e.tLastFrame), C.long(flags), C.ulong(e.deadline),
+		C.long(t-e.tStart), C.ulong(duration), C.long(flags), C.ulong(e.deadline),
 		(*C.uchar)(&yuvImg.Y[0]), (*C.uchar)(&yuvImg.Cb[0]), (*C.uchar)(&yuvImg.Cr[0]),
 	); ec != C.VPX_CODEC_OK {
 		return nil, func() {}, fmt.Errorf("vpx_codec_encode failed (%d)", ec)
--- a/pkg/codec/x264/bridge.h
+++ b/pkg/codec/x264/bridge.h
@@ -16,7 +16,7 @@ typedef struct Slice {

 typedef struct Encoder {
  x264_t *h;
-  x264_picture_t pic_in, pic_out;
+  x264_picture_t pic_in;
  x264_param_t param;
 } Encoder;

@@ -52,15 +52,21 @@ Encoder *enc_new(x264_param_t param, char *preset, int *rc) {
    goto fail;
  }

-  if (x264_picture_alloc(&e->pic_in, param.i_csp, param.i_width, param.i_height) < 0) {
+  x264_picture_t pic_in;
+  if (x264_picture_alloc(&pic_in, param.i_csp, param.i_width, param.i_height) < 0) {
    *rc = ERR_ALLOC_PICTURE;
    goto fail;
  }

+  // FIXME: we use x264_picture_alloc to set the metadata only, we don't need the allocated memory
+  //        to store the frame. Since we free the frame memory here, we don't need to call
+  //        x264_picture_clean later.
+  e->pic_in = pic_in;
+  x264_picture_clean(&pic_in);
+
  e->h = x264_encoder_open(&e->param);
  if (!e->h) {
    *rc = ERR_OPEN_ENGINE;
-    x264_picture_clean(&e->pic_in);
    goto fail;
  }

@@ -75,24 +81,24 @@ Slice enc_encode(Encoder *e, uint8_t *y, uint8_t *cb, uint8_t *cr, int *rc) {
  x264_nal_t *nal;
  int i_nal;

+  x264_picture_t pic_out;
  e->pic_in.img.plane[0] = y;
  e->pic_in.img.plane[1] = cb;
  e->pic_in.img.plane[2] = cr;

-  int frame_size = x264_encoder_encode(e->h, &nal, &i_nal, &e->pic_in, &e->pic_out);
+  int frame_size = x264_encoder_encode(e->h, &nal, &i_nal, &e->pic_in, &pic_out);
  Slice s = {.data_len = frame_size};
  if (frame_size <= 0) {
    *rc = ERR_ENCODE;
    return s;
  }

-  e->pic_in.i_pts++;
+  // e->pic_in.i_pts++;
  s.data = nal->p_payload;
  return s;
 }

 void enc_close(Encoder *e, int *rc) {
  x264_encoder_close(e->h);
-  x264_picture_clean(&e->pic_in);
  free(e);
 }
--- a/pkg/driver/camera/camera_linux.go
+++ b/pkg/driver/camera/camera_linux.go
@@ -21,6 +21,7 @@ import (

 const (
 	maxEmptyFrameCount = 5
+	prioritizedDevice  = "video0"
 )

 var (
@@ -94,9 +95,14 @@ func init() {

 			discovered[reallink] = struct{}{}
 			cam := newCamera(device)
+			priority := driver.PriorityNormal
+			if label == prioritizedDevice {
+				priority = driver.PriorityHigh
+			}
 			driver.GetManager().Register(cam, driver.Info{
 				Label:      label,
 				DeviceType: driver.Camera,
+				Priority:   priority,
 			})
 		}
 	}
--- a/pkg/driver/microphone/microphone.go
+++ b/pkg/driver/microphone/microphone.go
@@ -37,12 +37,6 @@ type microphone struct {

 func init() {
 	var err error
-	/*
-		backends := []malgo.Backend{
-			malgo.BackendPulseaudio,
-			malgo.BackendAlsa,
-		}
-	*/
 	ctx, err = malgo.InitContext(nil, malgo.ContextConfig{}, func(message string) {
 		logger.Debugf("%v\n", message)
 	})
@@ -56,11 +50,18 @@ func init() {
 	}

 	for _, device := range devices {
-		// TODO: Detect default device and prioritize it
-		driver.GetManager().Register(newMicrophone(device), driver.Info{
-			Label:      device.ID.String(),
-			DeviceType: driver.Microphone,
-		})
+		info, err := ctx.DeviceInfo(malgo.Capture, device.ID, malgo.Shared)
+		if err == nil {
+			priority := driver.PriorityNormal
+			if info.IsDefault > 0 {
+				priority = driver.PriorityHigh
+			}
+			driver.GetManager().Register(newMicrophone(info), driver.Info{
+				Label:      device.ID.String(),
+				DeviceType: driver.Microphone,
+				Priority:   priority,
+			})
+		}
 	}

 	// Decide which endian
@@ -133,7 +134,7 @@ func (m *microphone) AudioRecord(inputProp prop.Media) (audio.Reader, error) {
 		return nil, err
 	}

-	return audio.ReaderFunc(func() (wave.Audio, func(), error) {
+	var reader audio.Reader = audio.ReaderFunc(func() (wave.Audio, func(), error) {
 		chunk, ok := <-m.chunkChan
 		if !ok {
 			device.Stop()
@@ -145,7 +146,12 @@ func (m *microphone) AudioRecord(inputProp prop.Media) (audio.Reader, error) {
 		// FIXME: the decoder should also fill this information
 		decodedChunk.(*wave.Float32Interleaved).Size.SamplingRate = inputProp.SampleRate
 		return decodedChunk, func() {}, err
-	}), nil
+	})
+
+	// FIXME: The current audio detection and audio encoder can only work with a static latency. Since the latency from the driver
+	//        can fluctuate, we need to stabilize it. Maybe there's a better way for doing this?
+	reader = audio.NewBuffer(int(inputProp.Latency.Seconds() * float64(inputProp.SampleRate)))(reader)
+	return reader, nil
 }

 func (m *microphone) Properties() []prop.Media {
@@ -159,46 +165,36 @@ func (m *microphone) Properties() []prop.Media {
 	}

 	for ch := m.MinChannels; ch <= m.MaxChannels; ch++ {
-		for sampleRate := m.MinSampleRate; sampleRate <= m.MaxSampleRate; sampleRate += sampleRateStep {
-			for i := 0; i < int(m.FormatCount); i++ {
-				format := m.Formats[i]
+		// FIXME: Currently support 48kHz only. We need to implement a resampler first.
+		// for sampleRate := m.MinSampleRate; sampleRate <= m.MaxSampleRate; sampleRate += sampleRateStep {
+		sampleRate := 48000
+		for i := 0; i < int(m.FormatCount); i++ {
+			format := m.Formats[i]

-				supportedProp := prop.Media{
-					Audio: prop.Audio{
-						ChannelCount: int(ch),
-						SampleRate:   int(sampleRate),
-						IsBigEndian:  isBigEndian,
-						// miniaudio only supports interleaved at the moment
-						IsInterleaved: true,
-					},
-				}
-
-				switch malgo.FormatType(format) {
-				case malgo.FormatF32:
-					supportedProp.SampleSize = 4
-					supportedProp.IsFloat = true
-				case malgo.FormatS16:
-					supportedProp.SampleSize = 2
-					supportedProp.IsFloat = false
-				}
-
-				supportedProps = append(supportedProps, supportedProp)
+			supportedProp := prop.Media{
+				Audio: prop.Audio{
+					ChannelCount: int(ch),
+					SampleRate:   int(sampleRate),
+					IsBigEndian:  isBigEndian,
+					// miniaudio only supports interleaved at the moment
+					IsInterleaved: true,
+					// FIXME: should change this to a less discrete value
+					Latency: time.Millisecond * 20,
+				},
 			}
-		}
-	}

-	// FIXME: remove this hardcoded value. Malgo doesn't support "ma_context_get_device_info" API yet. The above iterations
-	//        will always return nothing as of now
-	supportedProps = append(supportedProps, prop.Media{
-		Audio: prop.Audio{
-			Latency:       time.Millisecond * 20,
-			ChannelCount:  1,
-			SampleRate:    48000,
-			SampleSize:    4,
-			IsFloat:       true,
-			IsBigEndian:   isBigEndian,
-			IsInterleaved: true,
-		},
-	})
+			switch malgo.FormatType(format) {
+			case malgo.FormatF32:
+				supportedProp.SampleSize = 4
+				supportedProp.IsFloat = true
+			case malgo.FormatS16:
+				supportedProp.SampleSize = 2
+				supportedProp.IsFloat = false
+			}
+
+			supportedProps = append(supportedProps, supportedProp)
+		}
+		// }
+	}
 	return supportedProps
 }
--- a/pkg/driver/screen/screen.go
+++ b/pkg/driver/screen/screen.go
@@ -1 +1,81 @@
+// +build !linux
+
 package screen
+
+import (
+	"fmt"
+	"image"
+	"io"
+
+	"github.com/kbinani/screenshot"
+	"github.com/pion/mediadevices/pkg/driver"
+	"github.com/pion/mediadevices/pkg/frame"
+	"github.com/pion/mediadevices/pkg/io/video"
+	"github.com/pion/mediadevices/pkg/prop"
+)
+
+type screen struct {
+	displayIndex int
+	doneCh       chan struct{}
+}
+
+func init() {
+	activeDisplays := screenshot.NumActiveDisplays()
+	for i := 0; i < activeDisplays; i++ {
+		priority := driver.PriorityNormal
+		if i == 0 {
+			priority = driver.PriorityHigh
+		}
+
+		s := newScreen(i)
+		driver.GetManager().Register(s, driver.Info{
+			Label:      fmt.Sprint(i),
+			DeviceType: driver.Screen,
+			Priority:   priority,
+		})
+	}
+}
+
+func newScreen(displayIndex int) *screen {
+	s := screen{
+		displayIndex: displayIndex,
+	}
+	return &s
+}
+
+func (s *screen) Open() error {
+	s.doneCh = make(chan struct{})
+	return nil
+}
+
+func (s *screen) Close() error {
+	close(s.doneCh)
+	return nil
+}
+
+func (s *screen) VideoRecord(selectedProp prop.Media) (video.Reader, error) {
+	r := video.ReaderFunc(func() (img image.Image, release func(), err error) {
+		select {
+		case <-s.doneCh:
+			return nil, nil, io.EOF
+		default:
+		}
+
+		img, err = screenshot.CaptureDisplay(s.displayIndex)
+		release = func() {}
+		return
+	})
+	return r, nil
+}
+
+func (s *screen) Properties() []prop.Media {
+	resolution := screenshot.GetDisplayBounds(s.displayIndex)
+	supportedProp := prop.Media{
+		Video: prop.Video{
+			Width:       resolution.Dx(),
+			Height:      resolution.Dy(),
+			FrameFormat: frame.FormatRGBA,
+		},
+	}
+	return []prop.Media{supportedProp}
+}
--- a/track.go
+++ b/track.go
@@ -3,6 +3,7 @@ package mediadevices
 import (
 	"errors"
 	"image"
+	"io"
 	"math/rand"
 	"sync"

@@ -57,6 +58,8 @@ type Track interface {
 	// NewRTPReader creates a new reader from the source. The reader will encode the source, and packetize
 	// the encoded data in RTP format with given mtu size.
 	NewRTPReader(codecName string, mtu int) (RTPReadCloser, error)
+	// NewEncodedReader creates a new Go standard io.ReadCloser that reads the encoded data in codecName format
+	NewEncodedReader(codecName string) (io.ReadCloser, error)
 }

 type baseTrack struct {
@@ -182,6 +185,31 @@ func (track *baseTrack) unbind(pc *webrtc.PeerConnection) error {
 	return nil
 }

+func (track *baseTrack) newEncodedReader(reader codec.ReadCloser) (io.ReadCloser, error) {
+	var encoded []byte
+	release := func() {}
+	return &encodedReadCloserImpl{
+		readFn: func(b []byte) (int, error) {
+			var err error
+
+			if len(encoded) == 0 {
+				release()
+				encoded, release, err = reader.Read()
+				if err != nil {
+					reader.Close()
+					track.onError(err)
+					return 0, err
+				}
+			}
+
+			n := copy(b, encoded)
+			encoded = encoded[n:]
+			return n, nil
+		},
+		closeFn: reader.Close,
+	}, nil
+}
+
 func newTrackFromDriver(d driver.Driver, constraints MediaTrackConstraints, selector *CodecSelector) (Track, error) {
 	if err := d.Open(); err != nil {
 		return nil, err
@@ -298,6 +326,21 @@ func (track *VideoTrack) NewRTPReader(codecName string, mtu int) (RTPReadCloser,
 	}, nil
 }

+func (track *VideoTrack) NewEncodedReader(codecName string) (io.ReadCloser, error) {
+	reader := track.NewReader(false)
+	inputProp, err := detectCurrentVideoProp(track.Broadcaster)
+	if err != nil {
+		return nil, err
+	}
+
+	encodedReader, _, err := track.selector.selectVideoCodecByNames(reader, inputProp, codecName)
+	if err != nil {
+		return nil, err
+	}
+
+	return track.newEncodedReader(encodedReader)
+}
+
 // AudioTrack is a specific track type that contains audio source which allows multiple readers to access, and
 // manipulate.
 type AudioTrack struct {
@@ -336,9 +379,6 @@ func newAudioTrackFromDriver(d driver.Driver, recorder driver.AudioRecorder, con
 		return nil, err
 	}

-	// FIXME: The current audio detection and audio encoder can only work with a static latency. Since the latency from the driver
-	//        can fluctuate, we need to stabilize it. Maybe there's a better way for doing this?
-	reader = audio.NewBuffer(int(constraints.selectedMedia.Latency.Seconds() * float64(constraints.selectedMedia.SampleRate)))(reader)
 	return newAudioTrackFromReader(d, reader, selector), nil
 }

@@ -380,7 +420,7 @@ func (track *AudioTrack) NewRTPReader(codecName string, mtu int) (RTPReadCloser,
 		return nil, err
 	}

-	sample := newVideoSampler(selectedCodec.ClockRate)
+	sample := newAudioSampler(selectedCodec.ClockRate, inputProp.Latency)

 	// FIXME: not sure the best way to get unique ssrc. We probably should have a global keeper that can generate a random ID and does book keeping?
 	packetizer := rtp.NewPacketizer(mtu, selectedCodec.PayloadType, rand.Uint32(), selectedCodec.Payloader, rtp.NewRandomSequencer(), selectedCodec.ClockRate)
@@ -402,3 +442,18 @@ func (track *AudioTrack) NewRTPReader(codecName string, mtu int) (RTPReadCloser,
 		closeFn: encodedReader.Close,
 	}, nil
 }
+
+func (track *AudioTrack) NewEncodedReader(codecName string) (io.ReadCloser, error) {
+	reader := track.NewReader(false)
+	inputProp, err := detectCurrentAudioProp(track.Broadcaster)
+	if err != nil {
+		return nil, err
+	}
+
+	encodedReader, _, err := track.selector.selectAudioCodecByNames(reader, inputProp, codecName)
+	if err != nil {
+		return nil, err
+	}
+
+	return track.newEncodedReader(encodedReader)
+}
Author	SHA1	Message	Date
Lukas Herman	5d5001d0b4	Bring back x11 adapter for Linux build only Original x11 adapter uses ~40% less CPU usage than kbinani/screenshot. Ideally, we should migrate 100% to kbinani/screenshot. But, the CPU usage difference is substantial. In the future, we should look deeper into kbinani/screenshot and try to improve it.	2020-11-23 20:36:17 -05:00
Lukas Herman	c734c53b00	Revert "Create codeql-analysis.yml" This reverts commit `ce032411a7`.	2020-11-21 14:38:31 -08:00
Lukas Herman	92ac89a620	Update README.md	2020-11-21 14:11:35 -08:00
Lukas Herman	ce032411a7	Create codeql-analysis.yml	2020-11-21 14:11:00 -08:00
Lukas Herman	3ea7120130	Remove some hardcoded microphone metadata Changes: * Use malgo.DeviceInfo to get the microphone metadata * Move static buffering to the driver layer	2020-11-21 14:07:32 -08:00
Lukas Herman	282d0a4fb4	Migrate to kbinani/screenshot for screen adapter https://github.com/kbinani/screenshot is a library to capture desktop screen. It is cgo free for Windows, Linux, FreeBSD, OpenBSD, NetBSD, and Solaris.	2020-11-21 13:40:35 -08:00
Lukas Herman	356eee19ce	Remove pkg-config requirement for mmal	2020-11-21 20:57:26 +00:00
Lukas Herman	02d0cd3f44	Prioritize /dev/video0 on Linux camera	2020-11-21 12:44:42 -08:00
Lukas Herman	a8dbecff30	Fix crash in x264 encoder close	2020-11-21 11:42:21 -08:00
Lukas Herman	273457b370	Remove git LFS	2020-11-17 11:41:09 -08:00
Lukas Herman	9846a7eb67	Remove demo.gif to avoid big storage usage	2020-11-17 11:37:28 -08:00
Renovate Bot	b2e72af884	Update module pion/rtp to v1.6.1 Generated by Renovate Bot	2020-11-16 21:14:08 -08:00
Renovate Bot	b8dd3811ac	Update module gen2brain/malgo to v0.10.24 Generated by Renovate Bot	2020-11-16 08:38:46 -08:00
Lukas Herman	c1958b62a2	Update README.md	2020-11-09 23:21:10 -08:00
Lukas Herman	ea90f86abd	Update README.md	2020-11-09 23:19:50 -08:00
Lukas Herman	716da16e4a	Add NewEncodedReeader API Changes: * Add NewEncodedReeader method to Track interface * Add video archival example	2020-11-09 23:17:48 -08:00
Lukas Herman	1550a68003	Fix incorrect audio sampler	2020-11-09 22:24:42 -08:00
Lukas Herman	d65170dfe3	Fix wrong duration in vpx due to buffer usage VPX doesn't allow 0 duration. If 0 is given, vpx_codec_encode will fail with VPX_CODEC_INVALID_PARAM. 0 duration is possible because mediadevices first gets the frame meta data by reading from the source, and consequently the codec will read the first frame from the buffer. This means the first frame won't have a pause to the second frame, which means if the delay is <1 ms (vpx duration resolution), duration is going to be 0.	2020-11-09 22:16:23 -08:00
Lukas Herman	4057524bf0	Update README	2020-11-08 14:16:26 -08:00