From db5af107b735297961035bb1a3622ac2df1a8c8a Mon Sep 17 00:00:00 2001 From: Jan Stabenow Date: Fri, 20 Jan 2023 18:19:41 +0100 Subject: [PATCH 01/26] Mod description and added the online docs --- README.md | 1045 +++-------------------------------------------------- 1 file changed, 45 insertions(+), 1000 deletions(-) diff --git a/README.md b/README.md index 0c1baf20..5015cf9e 100644 --- a/README.md +++ b/README.md @@ -2,30 +2,54 @@ The cloud-native audio/video processing API. -[![License: MIT](https://img.shields.io/badge/License-Apache%202.0-brightgreen.svg)](<[https://opensource.org/licenses/MI](https://www.apache.org/licenses/LICENSE-2.0)>) +[![License: Apache2](https://img.shields.io/badge/License-Apache%202.0-brightgreen.svg)](<[https://opensource.org/licenses/MI](https://www.apache.org/licenses/LICENSE-2.0)>) [![CodeQL](https://github.com/datarhei/core/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/datarhei/core/actions/workflows/codeql-analysis.yml) [![tests](https://github.com/datarhei/core/actions/workflows/go-tests.yml/badge.svg)](https://github.com/datarhei/core/actions/workflows/go-tests.yml) [![codecov](https://codecov.io/gh/datarhei/core/branch/main/graph/badge.svg?token=90YMPZRAFK)](https://codecov.io/gh/datarhei/core) [![Go Report Card](https://goreportcard.com/badge/github.com/datarhei/core)](https://goreportcard.com/report/github.com/datarhei/core) [![PkgGoDev](https://pkg.go.dev/badge/github.com/datarhei/core)](https://pkg.go.dev/github.com/datarhei/core) +[![Gitbook](https://img.shields.io/badge/GitBook-quick%20start-green)](https://docs.datarhei.com/core/guides/beginner) -datarhei Core is management for FFmpeg processes without development effort. It is a central interface for mapping AV processes, is responsible for design and management, and provides all necessary interfaces to access the video content. The included control for FFmpeg can keep all used functions reliable and executable without the need for software developers to take care of it. In addition, process and resource limitation for all FFmpeg processes protects the host system from application overload. The overall system gives access to current process values (CPU, RAM) and complete control of system resources and loads with statistical access to process data and current and historical logs. +The datarhei Core is a process management solution for FFmpeg that offers a range of interfaces for media content, including HTTP, RTMP, SRT, and storage options. It is optimized for use in virtual environments such as Docker. It has been implemented in various contexts, from small-scale applications like Restreamer to large-scale, multi-instance frameworks spanning multiple locations, such as dedicated servers, cloud instances, and single-board computers. The datarhei Core stands out from traditional media servers by emphasizing FFmpeg and its capabilities rather than focusing on media conversion. -## Features +## Objectives of development -- Unrestricted FFmpeg process management -- Optimized for long-running tasks -- In-Memory- and Disk-Filesystem for media assets -- HTTP/S, RTMP/S and SRT services -- Let's Encrypt for HTTPS and RTMPS -- HLS/DASH Session tracking with bandwidth and current viewer limiters -- Multiple resource limiters and monitoring -- FFmpeg progress data -- Metrics incl. Prometheus support -- Logging and debugging for FFmpeg processes with history -- Multiple auth. by JWT and Auth0 -- 100% JSON REST API (Swagger documented) -- GraphQL for metrics, process, and progress data +The objectives of development are: + +* Unhindered use of FFmpeg processes +* Portability of FFmpeg, including management across development and production environments +* Scalability of FFmpeg-based applications through the ability to offload processes to additional instances +* Streamlining of media product development by focusing on features and design. + +## What issues have been resolved thus far? + +### Process management + +* Run multiple processes via API +* Unrestricted FFmpeg commands in process configuration. +* Error detection and recovery (e.g., FFmpeg stalls, dumps) +* Referencing for process chaining (pipelines) +* Placeholders for storage, RTMP, and SRT usage (automatic credentials management and URL resolution) +* Logs (access to current stdout/stderr) +* Log history (configurable log history, e.g., for error analysis) +* Resource limitation (max. CPU and MEMORY usage per process) +* Statistics (like FFmpeg progress per input and output, CPU and MEMORY, state, uptime) +* Input verification (like FFprobe) +* Metadata (option to store additional information like a title) + +### Media delivery + +* Configurable file systems (in-memory, disk-mount, S3) +* HTTP/S, RTMP/S, and SRT services, including Let's Encrypt +* Bandwidth and session limiting for HLS/MPEG DASH sessions (protects restreams from congestion) +* Viewer session API and logging + +### Misc + +* HTTP REST and GraphQL API +* Swagger documentation +* Metrics incl. Prometheus support (also detects POSIX and cgroups resources) +* Docker images for fast setup of development environments up to the integration of cloud resources ## Quick start @@ -47,993 +71,14 @@ docker run --name core -d \ 3. Log in with Swagger Authorize > Basic authorization > Username: admin, Password: secret -## Docker images - -Native (linux/amd64,linux/arm64,linux/arm/v7) - -- datarhei/base:core-alpine-latest -- datarhei/base:core-ubuntu-latest - -Bundle with FFmpeg (linux/amd64,linux/arm64,linux/arm/v7) - -- datarhei/core:latest - -Bundle with FFmpeg for Raspberry Pi (linux/arm/v7) - -- datarhei/core:rpi-latest - -Bundle with FFmpeg for Nvidia Cuda (linux/amd64) - -- datarhei/core:cuda-latest - -Bundle with FFmpeg for Intel VAAPI (linux/amd64) - -- datarhei/core:vaapi-latest - ## Documentation -## Environment variables +Documentation is available on [docs.datarhei.com/core](https://docs.datarhei.com/core). -The environment variables can be set in the file `.env`, e.g. - -``` -CORE_API_AUTH_USERNAME=admin -CORE_API_AUTH_PASSWORD=datarhei -... -``` - -You can also provide them on the command line, whatever you prefer. If the same environment variable is set -in the `.env` file and on the command line, the one set on the command line will overrule the one from the `.env` file. - -The currently known environment variables (but not all will be respected) are: - -| Name | Default | Description | -| ----------------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| CORE_CONFIGFILE | (not set) | Path to a config file. The following environment variables will override the respective values in the config file. | -| CORE_ADDRESS | `:8080` | HTTP listening address. | -| CORE_LOG_LEVEL | `info` | silent, error, warn, info, debug. | -| CORE_LOG_TOPICS | (not set) | List of topics to log (comma separated) | -| CORE_LOG_MAXLINES | `1000` | Number of latest log lines to keep in memory. | -| CORE_DB_DIR | `.` | Directory for holding the operational data. This directory must exist. | -| CORE_HOST_NAME | (not set) | Set to the domain name of the host this instance is running on. | -| CORE_HOST_AUTO | `true` | Enable detection of public IP addresses. | -| CORE_API_READ_ONLY | `false` | Allow only ready only access to the API | -| CORE_API_ACCESS_HTTP_ALLOW | (not set) | Comma separated list of IP ranges in CIDR notation (HTTP traffic), e.g. `127.0.0.1/32,::1/128`. | -| CORE_API_ACCESS_HTTP_BLOCK | (not set) | Comma separated list of IP ranges in CIDR notation (HTTP traffic), e.g. `127.0.0.1/32,::1/128`. | -| CORE_API_ACCESS_HTTPS_ALLOW | (not set) | Comma separated list of IP ranges in CIDR notation (HTTPS traffic), e.g. `127.0.0.1/32,::1/128`. | -| CORE_API_ACCESS_HTTPS_BLOCK | (not set) | Comma separated list of IP ranges in CIDR notation (HTTPS traffic), e.g. `127.0.0.1/32,::1/128`. | -| CORE_API_AUTH_ENABLE | `true` | Set to `false` to disable auth for all clients. | -| CORE_API_AUTH_DISABLE_LOCALHOST | `false` | Set to `true` to disable auth for clients from localhost. | -| CORE_API_AUTH_USERNAME | (required) | Username for auth. | -| CORE_API_AUTH_PASSWORD | (required) | Password for auth. | -| CORE_API_AUTH_JWT_SECRET | (not set) | A secret for en- and decrypting the JWT. If not set, a secret will be generated. | -| CORE_API_AUTH_AUTH0_ENABLE | `false` | Enable Auth0. | -| CORE_API_AUTH_AUTH0_TENANTS | (not set) | List of base64 encoded Auth0 tenant JSON objects (comma-separated). The tenant JSON object is defined as `{"domain":string,"audience":string,"users":array of strings}` | -| CORE_TLS_ADDRESS | `:8181` | Port to listen on for HTTPS requests. | -| CORE_TLS_ENABLE | `false` | Set to `true` to enable TLS support. | -| CORE_TLS_AUTO | `false` | Set to `true` to enable automatic retrieval of a Let's Encrypt certificate. Requires `CORE_TLS_ENABLE` to be `true` and `CORE_HOST_NAME` to be set with `CORE_HOST_AUTO` to `false`. | -| CORE_TLS_CERTFILE | (not set) | TLS certificate file in PEM format. | -| CORE_TLS_KEYFILE | (not set) | TLS key file in PEM format. | -| CORE_STORAGE_DISK_DIR | `.` | A directory that will be exposed by HTTP on /. This directory must exist. | -| CORE_STORAGE_DISK_MAXSIZEMBYTES | `0` | Max. allowed megabytes for `CORE_STORAGE_DISK_DIR`. | -| CORE_STORAGE_DISK_CACHE_ENABLE | `true` | Enable cache for files from `CORE_STORAGE_DISK_DIR`. | -| CORE_STORAGE_DISK_CACHE_MAXSIZEMBYTES | `0` | Max. allowed cache size, 0 for unlimited. | -| CORE_STORAGE_DISK_CACHE_TTLSECONDS | `300` | Seconds to keep files in cache. | -| CORE_STORAGE_DISK_CACHE_MAXFILESIZEMBYTES | `1` | Max. file size to put in cache. | -| CORE_STORAGE_DISK_CACHE_TYPES_ALLOW | (not set) | List of file extensions to cache (space-separated, e.g. ".html .js"), empty for all. | -| CORE_STORAGE_DISK_CACHE_TYPES_BLOCK | (not set) | List of file extensions not to cache (space-separated, e.g. ".m3u8 .mpd"), empty for none. | -| CORE_STORAGE_MEMORY_AUTH_ENABLE | `true` | Enable basic auth for PUT,POST, and DELETE on /memfs. | -| CORE_STORAGE_MEMORY_AUTH_USERNAME | (not set) | Username for Basic-Auth of `/memfs`. Required if auth is enabled. | -| CORE_STORAGE_MEMORY_AUTH_PASSWORD | (not set) | Password for Basic-Auth of `/memfs`. Required if auth is enabled. | -| CORE_STORAGE_MEMORY_MAXSIZEMBYTES | `0` | Max. allowed megabytes for `/memfs`. Any value <= 0 means unlimited. | -| CORE_STORAGE_MEMORY_PURGE | `false` | Set to `true` to remove the oldest entries if the `/memfs` is full. | -| CORE_STORAGE_COCORE_ORIGINS | `*` | List of allowed CORS origins (comma separated). Will be used for `/` and `/memfs`. | -| CORE_STORAGE_MIMETYPES_FILE | `mime.types` | Path to file with MIME type definitions. | -| CORE_RTMP_ENABLE | `false` | Enable RTMP server. | -| CORE_RTMP_ENABLE_TLS | `false` | Enable RTMP over TLS (RTMPS). Requires `CORE_TLS_ENABLE` to be `true`. | -| CORE_RTMP_ADDRESS | `:1935` | RTMP server listen address. | -| CORE_RTMP_ADDRESS_TLS | `:1936` | RTMPS server listen address. | -| CORE_RTMP_APP | `/` | RTMP app for publishing. | -| CORE_RTMP_TOKEN | (not set) | RTMP token for publishing and playing. The token is the value of the URL query parameter `token`. | -| CORE_SRT_ENABLE | `false` | Enable SRT server. | -| CORE_SRT_ADDRESS | `:6000` | SRT server listen address. | -| CORE_SRT_PASSPHRASE | (not set) | SRT passphrase. | -| CORE_SRT_TOKEN | (not set) | SRT token for publishing and playing. The token is the value of the URL query parameter `token`. | -| CORE_SRT_LOG_ENABLE | `false` | Enable SRT server logging. | -| CORE_SRT_LOG_TOPICS | (not set) | List topics to log from SRT server. See https://github.com/datarhei/gosrt#logging. | -| CORE_FFMPEG_BINARY | `ffmpeg` | Path to FFmpeg binary. | -| CORE_FFMPEG_MAXPROCESSES | `0` | Max. allowed simultaneously running FFmpeg instances. Any value <= 0 means unlimited. | -| CORE_FFMPEG_ACCESS_INPUT_ALLOW | (not set) | List of pattern for allowed input URI (space-separated), leave emtpy to allow any. | -| CORE_FFMPEG_ACCESS_INPUT_BLOCK | (not set) | List of pattern for blocked input URI (space-separated), leave emtpy to block none. | -| CORE_FFMPEG_ACCESS_OUTPUT_ALLOW | (not set) | List of pattern for allowed output URI (space-separated), leave emtpy to allow any. | -| CORE_FFMPEG_ACCESS_OUTPUT_BLOCK | (not set) | List of pattern for blocked output URI (space-separated), leave emtpy to block none. | -| CORE_FFMPEG_LOG_MAXLINES | `50` | Number of latest log lines to keep for each process. | -| CORE_FFMPEG_LOG_MAXHISTORY | `3` | Number of latest logs to keep for each process. | -| CORE_PLAYOUT_ENABLE | `false` | Enable playout API where available | -| CORE_PLAYOUT_MINPORT | `0` | Min. port a playout server per input can run on. | -| CORE_PLAYOUT_MAXPORT | `0` | Max. port a playout server per input can run on. | -| CORE_DEBUG_PROFILING | `false` | Set to `true` to enable profiling endpoint on `/profiling`. | -| CORE_DEBUG_FORCEGC | `0` | Number of seconds between forcing GC to return memory to the OS. Use in conjuction with `GODEBUG=madvdontneed=1`. Any value <= 0 means not to force GC. | -| CORE_METRICS_ENABLE | `false` | Enable collecting historic metrics data. | -| CORE_METRICS_ENABLE_PROMETHEUS | `false` | Enable prometheus endpoint /metrics. | -| CORE_METRICS_RANGE_SECONDS | `300` | Seconds to keep history metric data. | -| CORE_METRICS_INTERVAL_SECONDS | `2` | Interval for collecting metrics. | -| CORE_SESSIONS_ENABLE | `false` | Enable HLS statistics for `/memfs`. | -| CORE_SESSIONS_IP_IGNORELIST | (not set) | Comma separated list of IP ranges in CIDR notation, e.g. `127.0.0.1/32,::1/128`. | -| CORE_SESSIONS_SESSION_TIMEOUT_SEC | `30` | Timeout of a session in seconds. | -| CORE_SESSIONS_PERSIST | `false` | Whether to persist the session history. Will be stored in `CORE_DB_DIR`. | -| CORE_SESSIONS_MAXBITRATE_MBIT | `0` | Max. allowed outgoing bitrate in mbit/s, 0 for unlimited. | -| CORE_SESSIONS_MAXSESSIONS | `0` | Max. allowed number of simultaneous sessions, 0 for unlimited. | -| CORE_ROUTER_BLOCKED_PREFIXES | `/api` | List of path prefixes that can't be routed. | -| CORE_ROUTER_ROUTES | (not set) | List of route mappings of the form [from]:[to], e.g. `/foo:/bar`. Leave empty for no routings. | -| CORE_ROUTER_UI_PATH | (not set) | Path to directory with files for a UI. It will be mounted to `/ui` and uses `index.html` as default index page. | - -## Config - -The minimum config file has to look like this: - -``` -{ - "version": 1 -} -``` - -All other values will be filled with default values and persisted on disk. The entire default config file: - -``` -{ - "version": 3, - "id": "[will be generated if not given]", - "name": "[will be generated if not given]", - "address": ":8080", - "log": { - "level": "info", - "topics": [], - "max_lines": 1000 - }, - "db": { - "dir": "./config" - }, - "host": { - "name": [], - "auto": true - }, - "api": { - "read_only": false, - "access": { - "http": { - "allow": [], - "block": [] - }, - "https": { - "allow": [], - "block": [] - } - }, - "auth": { - "enable": true, - "disable_localhost": false, - "username": "", - "password": "", - "jwt": { - "secret": "" - }, - "auth0": { - "enable": false, - "tenants": [] - } - } - }, - "tls": { - "address": ":8181", - "enable": false, - "auto": false, - "cert_file": "", - "key_file": "" - }, - "storage": { - "disk": { - "dir": "./data", - "max_size_mbytes": 0, - "cache": { - "enable": true, - "max_size_mbytes": 0, - "ttl_seconds": 300, - "max_file_size_mbytes": 1, - "types": { - "allow": [], - "block": [] - } - } - }, - "memory": { - "auth": { - "enable": true, - "username": "admin", - "password": "vxbx0ViqfA75P1KCyw" - }, - "max_size_mbytes": 0, - "purge": false - }, - "cors": { - "origins": [ - "*" - ] - }, - "mimetypes_file": "mime.types" - }, - "rtmp": { - "enable": false, - "enable_tls": false, - "address": ":1935", - "address_tls": ":1936", - "app": "/", - "token": "" - }, - "srt": { - "enable": false, - "address": ":6000", - "passphrase": "", - "token": "", - "log": { - "enable": false, - "topics": [], - } - }, - "ffmpeg": { - "binary": "ffmpeg", - "max_processes": 0, - "access": { - "input": { - "allow": [], - "block": [] - }, - "output": { - "allow": [], - "block": [] - } - }, - "log": { - "max_lines": 50, - "max_history": 3 - } - }, - "playout": { - "enable": false, - "min_port": 0, - "max_port": 0 - }, - "debug": { - "profiling": false, - "force_gc": 0 - }, - "stats": { - "enable": true, - "ip_ignorelist": [ - "127.0.0.1/32", - "::1/128" - ], - "session_timeout_sec": 30, - "persist": false, - "persist_interval_sec": 300, - "max_bitrate_mbit": 0, - "max_sessions": 0 - }, - "service": { - "enable": false, - "token": "", - "url": "https://service.datarhei.com" - }, - "router": { - "blocked_prefixes": [ - "/api" - ], - "routes": {} - } -} -``` - -If you don't provide a path to a config file, the default config will be used, and nothing will be persisted to the disk. Default values can be overruled by environment variables. - -## TLS / HTTPS - -Enable TLS / HTTPS support by setting `CORE_TLS_ENABLE=true` and provide the certificate file and key file in PEM format by setting the environment variables `CORE_TLS_CERTFILE` and `CORE_TLS_KEYFILE` accordingly. If a certificate authority signs the certificate, the certificate file should be the concatenation of the server's certificate, any intermediates, and the CA's certificate. - -If TLS with given certificates is enabled, an HTTP server listening on `CORE_ADDRESS` (address) will be additionally started. This server provides access to the same memory filesystem as the HTTPS server (including limits and authorization), but its access is restricted to localhost only. - -### Let's Encrypt - -If you want to use automatic certificates from Let's Encrypt, set the environment variable `CORE_TLS_AUTO` to `true.` To work, the -environment variables `CORE_TLS_ENABLE` have to be `true,` and `CORE_HOST_NAME` has to be set to the host this host will be reachable. Otherwise, the ACME challenge will not work. The environment variables `CORE_TLS_CERTFILE` and `CORE_TLS_KEYFILE` will be ignored. - -If automatic TLS is enabled, the HTTP server (CORE_ADDRESS, resp. address) must listen on port 80. It is required to automatically acquire the certificate (serving the `HTTP-01` challenge). As a further requirement, `CORE_HOST_NAME` (host.name) must be set because it is used a the canonical name for the certificate. - -The obtained certificates will be stored in `CORE_DB_DIR/cert` to be available after a restart. - -The obtained certificates will be stored in `CORE_DB_DIR/cert` to be available after a restart. - -### Self-Signed certificates - -To create a self-signed certificate and key file pair, run this command and provide a reasonable value for the Common Name (CN). The CN is the fully qualified name of the host the instance is running on (e.g., `localhost`). You can also use an IP address or a wildcard name, e.g., `*.example.com`. - -RSA SSL certificate - -```sh -openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out cert.pem -subj '/CN=localhost' -``` - -ECDSA SSL certificate - -```sh -openssl ecparam -name secp521r1 -genkey -out key.pem -openssl req -new -x509 -key key.pem -out cert.pem -days 365 -subj '/CN=localhost' -``` - -Call `openssl ecparam -list_curves` to see all available supported curves listed. - -## Access Control - -To control who has access to the API, a list of allowed IPs can be defined. This list is provided at startup with the environment variables `CORE_API_ACCESS_HTTP_BLOCK` and `CORE_API_ACCESS_HTTP_ALLOW.` This is a comma-separated list of IPs in CIDR notation, -e.g. `127.0.0.1/32,::1/128`. If the list is empty, then all IPs are allowed. If the list contains any invalid IP range, the server -will refuse to start. This can be separately defined for the HTTP and HTTPS server if you have TLS enabled with the environment variables `CORE_API_ACCESS_HTTPS_BLOCK` and `CORE_API_ACCESS_HTTPS_ALLOW.` - -## Input/Output Control - -To control where FFmpeg can read and where FFmpeg can write, you can define a pattern that matches the -input addresses or the output addresses. These patterns are regular expressions that can be provided at startup with the -environment variables `CORE_FFMPEG_ACCESS_INPUT` and `CORE_FFMPEG_ACCESS_OUTPUT.` The expressions need to be space-separated, e.g. -`HTTPS?:// RTSP:// RTMP://`. If one of the lists is empty, then no restriction on input, resp. The output will be applied. - -Independently of the value of `CORE_FFMPEG_ACCESS_OUTPUT` there's a check that verifies that output can only be written to the specified `CORE_STORAGE_DISK_DIR` and works as follows: If the address has a protocol specifier other than `file:,` then no further checks will be applied. If the protocol is `file:` or no protocol specifier is given, the address is assumed to be a path that is checked against the path shown in `CORE_STORAGE_DISK_DIR.` - -It will be rejected if the address is outside the `CORE_STORAGE_DISK_DIR` directory. Otherwise, the protocol `file:` will be prepended. If you give some expressions for `CORE_FFMPEG_ACCESS_OUTPUT,` you should also allow `file:.` - -Special cases are the output addresses `-` (which will be rewritten to `pipe:`), and `/dev/null` (which will be allowed even though it's outside of `CORE_STORAGE_DISK_DIR`). - -If you set a value for `CORE_STORAGE_DISK_CACHE_MAXSIZEMBYTES`, which is larger than `0`, it will be interpreted as max—allowed megabytes for the `CORE_STORAGE_DISK_DIR.` As soon as the limit is reached, all processes that have outputs writing to `CORE_STORAGE_DISK_DIR` will be stopped. You are responsible for cleaning up the directory and restarting these processes. - -## RTMP - -The datarhei Core includes a simple RTMP server for publishing and playing streams. Set the environment variable `CORE_RTMP_ENABLE` to `true` to enable the RTMP server. It is listening on `CORE_RTMP_ADDRESS`. Use `CORE_RTMP_APP` to limit the app a stream can be published on, e.g. `/live` to require URLs to start with `/live`. To prevent anybody can publish streams, set `CORE_RTMP_TOKEN` to a secret only known to the publishers and subscribers. The token has to be put in the query of the stream URL, e.g. `/live/stream?token=...`. - -For additionaly enabling the RTMPS server, set the config variable `rtmp.enable_tls` or environment variable `CORE_RTMP_ENABLE_TLS` to `true`. This requires `tls.enable` or `CORE_TLS_ENABLE` to be set to to `true`. Use `rtmp.address_tls` or `CORE_RTMP_ADDRESS_TLS` to set the listen address for the RTMPS server. - -| Method | Path | Description | -| ------ | ------------ | ------------------------------------- | -| GET | /api/v3/rtmp | List all currently published streams. | - -## SRT - -The datarhei Core includes a simple SRT server for publishing and playing streams. Set the environment variable `CORE_SRT_ENABLE` to `true` to enable the SRT server. It is listening on `CORE_SRT_ADDRESS`. - -The `streamid` is formatted according to Appendix B of the [SRT specs](https://datatracker.ietf.org/doc/html/draft-sharabayko-srt#appendix-B). The following keys are supported: - -| Key | Descriptions | -| ------- | ----------------------------------------------------------------------------------------------------------------- | -| `m` | The connection mode, either `publish` for publishing a stream or `request` for subscribing to a published stream. | -| `r` | Name of the resource. | -| `token` | A token to prevent anybody to publish or subscribe to a stream. This is set with `CORE_SRT_TOKEN`. | - -An example publishing streamid: `#!:m=publish,r=12345,token=foobar`. - -With your SRT client, connect to the SRT server always in `caller` mode, e.g. `srt://127.0.0.1:6000?mode=caller&streamid=#!:m=publish,r=12345,token=foobar&passphrase=foobarfoobar&transmode=live`. - -Via the API you can gather statistics of the currently connected SRT clients. - -| Method | Path | Description | -| ------ | ----------- | ------------------------------------- | -| GET | /api/v3/srt | List all currently published streams. | - -## Playout - -FFmpeg processes with a `avstream:` (or `playout:`) input stream can expose an HTTP API to control the playout of that stream. With -`CORE_PLAYOUT_ENABLE` you enable exposing this API. The API is only exposed to `localhost` and is transparently connected to the datarhei Core API. You have to provide a port range (`CORE_PLAYOUT_MINPORT` and `CORE_PLAYOUT_MAXPORT`) where datarhei/core can use ports to assign it to the playout API. - -| Method | Path | Description | -| -------- | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| GET | /api/v3/process/:id/playout/:inputid/status | Retrieve the current status as JSON. | -| GET | /api/v3/process/:id/playout/:inputid/keyframe/\*name | Retrieve the last deliverd key frame from the input stream as JPEG (if `name` has the ending `.jpg`) or PNG (if `name` has the ending `.png`). | -| GET | /api/v3/process/:id/playout/:inputid/errorframe/encode | Immediately encode the error frame to a GOP. Will only have an effect if the last key frame is currently in a loop. | -| PUT/POST | /api/v3/process/:id/playout/:inputid/errorframe/\*name | Upload any image or video media that can be decoded and will be used to replace the key frame loop. If the key frame is currently in a loop, it will be repaced immediately. Otherwise, it will be used the next time the key frame is in a loop. The body of the request is the media file. | -| PUT | /api/v3/process/:id/playout/:inputid/stream | Replace the current stream. The body of the request is the URL of the new stream. | - -## MIME Types - -The file with the MIME types has one MIME type per line followed by a list of file extensions (including the "."). - -``` -text/plain .txt -text/html .htm .html -... -``` - -## Memory Filesystem - -AA very simple in-memory filesystem is available. The uploaded data is stored in a map, where the path used to upload the file -is used as the key. Use the `POST` or `PUT` method with the proper direction for uploading a file. The body of the request contains the contents of the file. No particular encoding or `Content-Type` is required. The file can then be downloaded from the same path. - -| Method | Path | Description | -| ------ | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| POST | /memfs/\*path | Upload a file to the memory filesystem. The filename is `path` which can contain slashes. If there's already a file with the same `path`, it will be overwritten. | -| PUT | /memfs/\*path | Same as POST. | -| GET | /memfs/\*path | Download the file stored under `path`. The MIME types are applied based on the extension in the `path`. | -| DELETE | /memfs/\*path | Delete the file stored under `path`. | -| POST | /api/v3/memfs/\*path | Upload a file to the memory filesystem. | -| PUT | /api/v3/memfs/\*path | Same as POST. | -| GET | /api/v3/memfs/\*path | Download the file stored under `path`. | -| PATCH | /api/v3/memfs/\*path | Create a link to a file. The body contains the path to that file. | -| DELETE | /api/v3/memfs/\*path | Delete the file stored under `path`. | -| GET | /api/v3/memfs | List all files that are currently stored in the in-memory filesystem. | - -Use these endpoints to, e.g., store HLS chunks and .m3u8 files (in contrast to an actual disk or a ramdisk): - -``` -ffmpeg -f lavfi -re -i testsrc2=size=640x480:rate=25 -c:v libx264 -preset:v ultrafast -r 25 -g 50 -f hls -start_number 0 -hls_time 2 -hls_list_size 6 -hls_flags delete_segments+temp_file+append_list -method PUT -hls_segment_filename http://localhost:8080/memfs/foobar_%04d.ts -y http://localhost:8080/memfs/foobar.m3u8 -``` - -Then you can play it generally with, e.g., `ffplay http://localhost:3000/memfs/foobar.m3u8`. - -Use the environment variables `CORE_STORAGE_MEMORY_AUTH_USERNAME` and `CORE_STORAGE_MEMORY_AUTH_PASSWORD` to protect the `/memfs` with Basic-Auth. Basic-Auth will only be enabled -if both environment variables are set to non-empty values. The `GET /memfs/:path` will not be protected with Basic-Auth. - -Use the environment variable `CORE_STORAGE_MEMORY_MAXSIZEMBYTES` to limit the amount of data that is allowed to be stored. The value is interpreted as megabytes. Use a value equal to or smaller than `0` not to impose any limits. A `507 Insufficient Storage` will be returned if you hit the limit. - -Listing all currently stored files is done by calling `/v3/memfs` with the credentials set by the environment variables `CORE_API_AUTH_USERNAME` and `CORE_API_AUTH_PASSWORD`. -It also accepts the query parameter `sort` (`name,` `size,` or `lastmod`) and `order` (`asc` or `desc`). If a valid value for `sort` is given, the results are sorted in ascending order. - -## Routes - -All contents in `CORE_STORAGE_DISK_DIR` are served from `/.` If you want to redirect some paths to an existing file, you can add static routes in `router.routes` by providing a direct mapping, e.g. - -``` -router: { - routes: { - "/foo.txt": "/bar.txt", - } -} -``` - -The paths have to start with a `/.` Alternatively, you can serve whole directories from another root than `CORE_STORAGE_DISK_DIR.` Use a `/*` at the end of a path as key and a path on the filesystem as the target, e.g. - -``` -router: { - routes: { - "/ui/*": "/path/to/ui", - } -} -``` - -If you use a relative path as target, then it will be added to the current working directory. - -## API - -Check the detailed API description on `/api/swagger/index.html`. - -### Login / Auth - -With auth enabled, you have to retrieve a JWT/OAuth token before you can access the `/v3/` API calls. - -| Method | Path | Description | -| ------ | --------------------- | ---------------------------------------------- | -| POST | /api/login | Retrieve a token to access the API. | -| GET | /api/v3/refresh_token | Retrieve a fresh token with a new expiry date. | - -For the login you have to send - -``` -{ - "username": "...", - "password": "..." -} -``` - -The `username` and the `password` are set by the environment variables `CORE_API_AUTH_USERNAME` and `CORE_API_AUTH_PASSWORD`. - -On successful login, the response looks like this: - -``` -{ - "expire": "2019-01-18T19:55:55+01:00", - "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NDc4Mzc3NTUsImlkIjpudWxsLCJvcmlnX2lhdCI6MTU0NzgzNDE1NX0.ZcrpD4oRBqG3wUrfnh1DOVpXdUT7dvUnvetKFEVRKKc" -} -``` - -Use the `token` in all subsequent calls to the `/api/v3/` endpoints, e.g. - -``` -http http://localhost:8080/api/v3/process "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NDc4Mzc3NTUsImlkIjpudWxsLCJvcmlnX2lhdCI6MTU0NzgzNDE1NX0.ZcrpD4oRBqG3wUrfnh1DOVpXdUT7dvUnvetKFEVRKKc" -``` - -### Config - -| Method | Path | Description | -| ------ | --------------------- | -------------------------------------------------------------------------------------------------------------------- | -| GET | /api/v3/config | Retrieve the current config without the override values from the environment variables. | -| GET | /api/v3/config/active | Retrieve the current config with the override values from the environment variables are taken into account. | -| PUT | /api/v3/config | Store a new config. Only some values are respected, and the new config will only be used after a restart. | -| GET | /api/v3/config/reload | Reload the config. The config will be re-read and validated from the store. It will cause a restart of all services. | - -When retrieving the config via the API, critical values (such as passwords) will be disguised if not required otherwise. - -### Process - -With the process API call, you can manage different FFmpeg processes. A process is defined as: - -``` -{ - "id": "SomeId", - "reference": "SomeId", - "type": "ffmpeg", - "input": [ - { - "id": "inputid", - "address": "rtsp://1.2.3.4/stream.sdp", - "options": [ - ... list of input options ... - ] - }, - ... list of inputs ... - ], - "output": [ - { - "id": "outputid", - "address": "rtmp://rtmp.youtube.com/live2/...", - "options": [ - ... list of output options ... - ], - "cleanup": [{ - "pattern": "(memfs|diskfs):...", - "max_files: "number, - "max_file_age_seconds": "number", - "purge_on_delete: "(true|false)" - }] - }, - ... list of outputs ... - ], - "options": [ - ... list of global options ... - ], - "reconnect": (true|false), - "reconnect_delay_seconds": 10, - "autostart": (true|false), - "stale_timeout_seconds": 30 -} -``` - -The input, output, and global options are interpreted as command-line options for FFmpeg. - -#### Process Cleanup - -With the optional array of cleanup rules for each output, it is possible to define rules for removing files from the -memory filesystem or disk. Each rule consists of a glob pattern and a max. allowed number of files matching that pattern or -permitted maximum age for the files matching that pattern. The pattern starts with either `memfs:` or `diskfs:` depending on -which filesystem this rule is designated to. Then a [glob pattern](https://pkg.go.dev/path/filepath#Match) follows to -identify the files. If `max_files` is set to a number > 0, then the oldest files from the matching files will be deleted if -the list of matching files is longer than that number. If `max_file_age_seconds` is set to a number > 0, then all files -that are older than this number of seconds from the matching files will be deleted. If `purge_on_delete` is set to `true`, -then all matching files will be deleted when the process is deleted. - -The API calls are - -| Method | Path | Description | -| ------ | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| POST | /api/v3/process | Adds a process. Overwriting an existing ID will result in an error. | -| GET | /api/v3/process | Retrieve a list of all known processes. Use the query parameter `ids` to list (comma separated) the IDs of the process you want to be part of the response. If the list is empty, all processes will be listed. Use the query parameter `filter` to list (comma separated) the wanted details per process (`config`, `state`, `log`). If the list is empty, all details will be included. | -| GET | /api/v3/process/:id | Retreive the details of a process including the config, state, and logs. Use the query parameter `filter` to list (comma separated) the wanted details per process (`config`, `state`, `log`). If the list is empty, all details will be included. | -| PUT | /api/v3/process/:id | Replaces the process with a new config. | -| GET | /api/v3/process/:id/config | Retrieve the config of a process as it was provided. | -| GET | /api/v3/process/:id/state | Retrieve the current state of a process. This includes the progress data if the process is running. | -| GET | /api/v3/process/:id/report | Retrieve the report and logs of a process. | -| GET | /api/v3/process/:id/debug | Retrieve an anonymized version of the details of a process. | -| DELETE | /api/v3/process/:id | Remove a specific process. Only possible if the process is not running. | -| PUT | /api/v3/process/:id/command | Send a command to a process. | -| GET | /api/v3/process/:id/data | Get all arbitrary JSON data that is stored with this process. | -| GET | /api/v3/process/:id/data/:key | Get arbitrary JSON data that is stored under the key `key.` | -| PUT | /api/v3/process/:id/data/:key | Store aribtrary JSON data under the key `key.` If the data is `null,` the key will be removed. | - -### Commands - -A command is defined as: - -``` -{ - "command": ("start"|"stop"|"restart"|"reload") -} -``` - -| Command | Description | -| --------- | ---------------------------------------------------------------------------------------------- | -| `start` | Start the process. If the process is already started, this won't have any effect. | -| `stop` | Stop the process. If the process is already stopped, this won't have any effect. | -| `restart` | Restart the process. If the process is not running, this won't have any effect. | -| `reload` | Reload the process. If the process was running, the reloaded process will start automatically. | - -### Placeholder - -Currently supported placeholders are: - -| Placeholder | Description | Location | -| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | -| `{diskfs}` | Will be replaced by the provided `CORE_STORAGE_DISK_DIR`. | `options`, `input.address`, `input.options`, `output.address`, `output.options` | -| `{memfs}` | Will be replace by the base URL of the MemFS. | `input.address`, `input.options`, `output.address`, `output.options` | -| `{processid}` | Will be replaced by the ID of the process. | `input.id`, `input.address`, `input.options`, `output.id`, `output.address`, `output.options`, `output.cleanup.pattern` | -| `{reference}` | Will be replaced by the reference of the process | `input.id`, `input.address`, `input.options`, `output.id`, `output.address`, `output.options`, `output.cleanup.pattern` | -| `{inputid}` | Will be replaced by the ID of the input. | `input.address`, `input.options` | -| `{outputid}` | Will be replaced by the ID of the output. | `output.address`, `output.options`, `output.cleanup.pattern` | -| `{rtmp}` | Will be replaced by the internal address of the RTMP server. Requires parameter `name` (name of the stream). | `input.address`, `output.address` | -| `{srt}` | Will be replaced by the internal address of the SRT server. Requires parameter `name` (name of the stream) and `mode` (either `publish` or `request`). | `input.address`, `output.address` | - -Before replacing the placeholders in the process config, all references (see below) will be resolved. - -If the value that gets filled in on the place of the placeholder needs escaping, you can define the character to be escaped in the placeholder by adding it to the placeholder name and prefix it with a `^`. -E.g. escape all `:` in the value (`http://example.com:8080`) for `{memfs}` placeholder, write `{memfs^:}`. It will then be replaced by `http\://example.com\:8080`. The escape character is always `\`. In -case there are `\` in the value, they will also get escaped. If the placeholder doesn't imply escaping, the value will be uses as-is. - -Add parameters to a placeholder by appending a comma separated list of key/values, e.g. `{placeholder,key1=value1,key2=value2}`. This can be combined with escaping. - -### References - -The input address of a process may contain a reference to the output of another process. It has the form `#[processid]:output=[id]`. - -A reference starts with a `#` followed by the process ID it refers to, followed by a `:.` Then comes `output,` followed by a `=.` -and the ID of the output. - -## FFmpeg - -### Statistics - -This repository contains a patch for the FFmpeg program to provide detailed progress information. With this patch, FFmpeg will output -the progress information in a JSON string that contains the data for each input and output stream individually. The JSON output is enabled -by default. It can be enabled with the global `-jsonstats` switch on the command line. Use the `-stats` switch -on the command line for the standard progress output. - -The Docker image that you can build with the provided Dockerfile includes the patched version of FFmpeg for your convenience. - -Example output with `-stats`: - -``` -frame= 143 fps= 25 q=-1.0 Lsize= 941kB time=00:00:05.68 bitrate=1357.0kbits/s speed=0.995x -``` - -Example output with `-jsonstats`: - -``` -JSONProgress:{"inputs":[{"id":0, "stream":0, "type":"video", "codec":"rawvideo", "coder":"rawvideo", "pix_fmt":"rgb24", "frame":188, "fps":24.95, "width":1280, "height":720, "size_kb":507600, "bitrate_kbps":552960.0},{"id":1, "stream":0, "type":"audio", "codec":"pcm_u8", "coder":"pcm_u8", "frame":314, "sampling_hz":44100, "layout":"stereo", "size_kb":628, "bitrate_kbps":705.6}], "outputs":[{"id":0, "stream":0, "type":"video", "codec":"h264", "coder":"libx264", "pix_fmt":"yuv420p", "frame":188, "fps":24.95, "q":-1.0, "width":1280, "height":720, "size_kb":1247, "bitrate_kbps":1365.6},{"id":0, "stream":1, "type":"audio", "codec":"aac", "coder":"aac", "frame":315, "sampling_hz":44100, "layout":"stereo", "size_kb":2, "bitrate_kbps":2.1}], "frame":188, "fps":24.95, "q":-1.0, "size_kb":1249, "bitrate_kbps":1367.7, "time":"0h0m7.48s", "speed":0.993, "dup":0, "drop":0} -``` - -The same output but nicely formatted: - -```json -{ - "bitrate_kbps": 1367.7, - "drop": 0, - "dup": 0, - "fps": 24.95, - "frame": 188, - "inputs": [ - { - "bitrate_kbps": 552960.0, - "codec": "rawvideo", - "coder": "rawvideo", - "fps": 24.95, - "frame": 188, - "height": 720, - "id": 0, - "pix_fmt": "rgb24", - "size_kb": 507600, - "stream": 0, - "type": "video", - "width": 1280 - }, - { - "bitrate_kbps": 705.6, - "codec": "pcm_u8", - "coder": "pcm_u8", - "frame": 314, - "id": 1, - "layout": "stereo", - "sampling_hz": 44100, - "size_kb": 628, - "stream": 0, - "type": "audio" - } - ], - "outputs": [ - { - "bitrate_kbps": 1365.6, - "codec": "h264", - "coder": "libx264", - "fps": 24.95, - "frame": 188, - "height": 720, - "id": 0, - "pix_fmt": "yuv420p", - "q": -1.0, - "size_kb": 1247, - "stream": 0, - "type": "video", - "width": 1280 - }, - { - "bitrate_kbps": 2.1, - "codec": "aac", - "coder": "aac", - "frame": 315, - "id": 0, - "layout": "stereo", - "sampling_hz": 44100, - "size_kb": 2, - "stream": 1, - "type": "audio" - } - ], - "q": -1.0, - "size_kb": 1249, - "speed": 0.993, - "time": "0h0m7.48s" -} -``` - -### Resilient Streaming - -Prepend the input source with `avstream:`, e.g. `... -i avstream:rtsp://1.2.3.4/stream.sdp ...`. It will reconnect to the stream if it breaks and repeats the last known intraframe until new data from the input stream is available. - -## Example - -Start `core` with the proper environment variables. Create a `.env` file or provide them on the command line. For this example, please use the following command line: - -``` -env CORE_API_AUTH_USERNAME=admin CORE_API_AUTH_PASSWORD=datarhei CORE_LOGLEVEL=debug CORE_STORAGE_DISK_DIR=./data ./core -``` - -Also, make sure that the directory `./data` exists. Otherwise, the state will not be stored and will be lost after a restart of -datarhei/core and the FFmpeg process will not be able to write the files. - -In this example, we will add a fake video and audio source. The video will be encoded with H264, and the audio will be encoded with AAC. The output will be an m3u8 stream. - -To talk to the API, we use the program [httpie](https://httpie.org/). - -First, we create a JSON file with the process definition (e.g. `testsrc.json`): - -```json -{ - "id": "testsrc", - "type": "ffmpeg", - "options": ["-loglevel", "info", "-err_detect", "ignore_err"], - "input": [ - { - "address": "testsrc=size=1280x720:rate=25", - "id": "video", - "options": ["-f", "lavfi", "-re"] - }, - { - "address": "anullsrc=r=44100:cl=stereo", - "id": "audio", - "options": ["-f", "lavfi"] - } - ], - "output": [ - { - "address": "http://127.0.0.1:8080/memfs/{processid}_{outputid}.m3u8", - "id": "hls", - "options": [ - "-codec:v", - "libx264", - "-preset:v", - "ultrafast", - "-r", - "25", - "-g", - "50", - "-pix_fmt", - "yuv420p", - "-b:v", - "1024k", - "-codec:a", - "aac", - "-b:a", - "64k", - "-hls_time", - "2", - "-hls_list_size", - "10", - "-hls_flags", - "delete_segments+temp_file+append_list", - "-hls_segment_filename", - "http://127.0.0.1:8080/memfs/{processid}_{outputid}_%04d.ts" - ] - } - ], - "reconnect": true, - "reconnect_delay_seconds": 10, - "stale_timeout_seconds": 10 -} -``` - -and POST it to the API: - -``` -http POST http://localhost:8080/v3/process < testsrc.json -``` - -Then check if it is there (as provided) - -``` -http http://localhost:8080/v3/process/testsrc -``` - -For the advanced, create another JSON file (e.g. `dump.json`): - -```json -{ - "id": "dump", - "type": "ffmpeg", - "options": ["-loglevel", "info", "-err_detect", "ignore_err"], - "input": [ - { - "address": "#testsrc:output=hls", - "id": "video", - "options": [] - } - ], - "output": [ - { - "address": "{diskfs}/{processid}.mp4", - "id": "hls", - "options": ["-codec", "copy", "-y"] - } - ], - "reconnect": true, - "reconnect_delay_seconds": 10, - "stale_timeout_seconds": 10 -} -``` - -and POST it to the API: - -``` -http POST http://localhost:8080/v3/process < dump.json -``` - -Then check if it is there (as provided) - -``` -http http://localhost:8080/v3/process/dump -``` - -Let's start the `testsrc` process - -``` -http PUT http://localhost:8080/v3/process/testsrc/command command=start -``` - -Now we can observe the progress of the process - -``` -http http://localhost:8080/v3/process/testsrc -``` - -or the log of the process - -``` -http http://localhost:8080/v3/process/testsrc/log -``` - -If you want to change the video bitrate, edit the `testsrc.json` file accordingly and replace the process: - -``` -http PUT http://localhost:8080/v3/process/testsrc < testsrc.json -``` - -It will stop the process, replace the config, and restart it. - -Now open, e.g., VLC, and load the stream `http://localhost:8080/memfs/testsrc_hls.m3u8`. - -This is enough; let's stop it - -``` -http PUT http://localhost:8080/v3/process/testsrc/command command=stop -``` - -and check its progress again - -``` -HTTP http://localhost:8080/v3/process/testsrc -``` - -Delete the process - -``` -HTTP DELETE http://localhost:8080/v3/process/testsrc -``` - -## Metrics - -Metrics for the processes and other aspects are provided for a Prometheus scraper on `/metrics.` - -Currently, available metrics are: - -| Metric | Type | Dimensions | Description | -| --------------------- | ------- | --------------------------------------------------------- | ----------------------------------------------- | -| ffmpeg_process | gauge | `core`, `process`, `name` | General stats per process. | -| ffmpeg_process_io | gauge | `core`, `process`, `type`, `id`, `index`, `media`, `name` | Stats per input and output of a process. | -| mem_limit_bytes | gauge | `core` | Total available memory in bytes. | -| mem_free_bytes | gauge | `core` | Free memory in bytes. | -| net_rx_bytes | gauge | `core`, `interface` | Number of received bytes by interface. | -| net_tx_bytes | gauge | `core`, `interface` | Number of sent bytes by interface. | -| cpus_system_time_secs | gauge | `core`, `cpu` | System time per CPU in seconds. | -| cpus_user_time_secs | gauge | `core`, `cpu` | User time per CPU in seconds. | -| cpus_idle_time_secs | gauge | `core`, `cpu` | Idle time per CPU in seconds. | -| session_total | counter | `core`, `collector` | Total number of sessions by collector. | -| session_active | gauge | `core`, `collector` | Current number of active sessions by collector. | -| session_rx_bytes | counter | `core`, `collector` | Total received bytes by collector. | -| session_tx_bytes | counter | `core`, `collector` | Total sent bytes by collector. | - -## Profiling - -Profiling information is available under `/profiling.` Set the environment variable `CORE_DEBUG_PROFILING=true` to make this endpoint -available. If authentication is enabled, you have to provide the token in the header. - -## Development - -### Requirement - -- Go v1.18+ ([Download here](https://golang.org/dl/)) - -### Build - -Clone the repository and build the binary - -``` -git clone git@github.com:datarhei/core.git -cd core -make -``` - -After the build process, the binary is available as `core` - -For more build options, run `make help.` - -### Cross Compile - -If you want to run the binary on a different operating system and/or architecture, you create the appropriate binary by simply setting some -environment variables, e.g. - -``` -env GOOS=linux GOARCH=arm go build -o core-linux-arm -env GOOS=linux GOARCH=arm64 go build -o core-linux-arm64 -env GOOS=freebsd GOARCH=amd64 go build -o core-freebsd-amd64 -env GOOS=windows GOARCH=amd64 go build -o core-windows-amd64 -env GOOS=macos GOARCH=amd64 go build -o core-macos-amd64 -... -``` - -### Docker - -Build the Docker image and run it to try out the API - -``` -docker build -t core . -docker run -it --rm -v ${PWD}/data:/core/data -p 8080:8080 core -``` - -### API Documentation - -The documentation of the API is available on `/api/swagger/index.html.` - -To generate the API documentation from the code, use [swag](https://github.com/swaggo/swag). - -``` -go install github.com/swaggo/swag (unless already installed) -make swagger -``` - -### Code style - -The source code is formatted with `go fmt`, or simply run `make fmt`. Static analysis of the source code is done with `staticcheck` -(see [staticcheck](https://staticcheck.io/docs/)), or simply run `make lint`. - -Before committing changes, you should run `make commit` to ensure that the source code is in shape. +- [Quick start](https://docs.datarhei.com/core/guides/beginner) +- [Installation](https://docs.datarhei.com/core/installation) +- [Configuration](https://docs.datarhei.com/core/configuration) +- [Coding](https://docs.datarhei.com/core/development/coding) ## License From 7b77c5fa7620c59f2fab4a55b00df771f113bf3c Mon Sep 17 00:00:00 2001 From: Jan Stabenow Date: Fri, 20 Jan 2023 18:38:37 +0100 Subject: [PATCH 02/26] Add media-core image --- README.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 5015cf9e..27d81a47 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Core -The cloud-native audio/video processing API. +![dsdsds](https://github.com/datarhei/misc/blob/main/img/media-core.png?raw=true) [![License: Apache2](https://img.shields.io/badge/License-Apache%202.0-brightgreen.svg)](<[https://opensource.org/licenses/MI](https://www.apache.org/licenses/LICENSE-2.0)>) [![CodeQL](https://github.com/datarhei/core/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/datarhei/core/actions/workflows/codeql-analysis.yml) @@ -51,6 +51,13 @@ The objectives of development are: * Metrics incl. Prometheus support (also detects POSIX and cgroups resources) * Docker images for fast setup of development environments up to the integration of cloud resources +## Docker images + +- datarhei/core:latest (AMD64, ARM64, ARMv7) +- datarhei/core:cuda-latest (Nvidia CUDA 11.7.1, AMD64) +- datarhei/core:rpi-latest (Raspberry Pi / OMX/V4L2-M2M, AMD64/ARMv7) +- datarhei/core:vaapi-latest (Intel VAAPI, AMD64) + ## Quick start 1. Run the Docker image From baf1c3391a583c38731fd0ebb634038d5e41a374 Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Mon, 3 Apr 2023 21:21:02 +0200 Subject: [PATCH 03/26] Deprecate ENV names that do not correspond to JSON name --- config/config.go | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/config/config.go b/config/config.go index b6eab656..33d9492b 100644 --- a/config/config.go +++ b/config/config.go @@ -150,7 +150,7 @@ func (d *Config) init() { // Log d.vars.Register(value.NewString(&d.Log.Level, "info"), "log.level", "CORE_LOG_LEVEL", nil, "Loglevel: silent, error, warn, info, debug", false, false) d.vars.Register(value.NewStringList(&d.Log.Topics, []string{}, ","), "log.topics", "CORE_LOG_TOPICS", nil, "Show only selected log topics", false, false) - d.vars.Register(value.NewInt(&d.Log.MaxLines, 1000), "log.max_lines", "CORE_LOG_MAXLINES", nil, "Number of latest log lines to keep in memory", false, false) + d.vars.Register(value.NewInt(&d.Log.MaxLines, 1000), "log.max_lines", "CORE_LOG_MAX_LINES", []string{"CORE_LOG_MAXLINES"}, "Number of latest log lines to keep in memory", false, false) // DB d.vars.Register(value.NewMustDir(&d.DB.Dir, "./config", d.fs), "db.dir", "CORE_DB_DIR", nil, "Directory for holding the operational data", false, false) @@ -182,19 +182,19 @@ func (d *Config) init() { d.vars.Register(value.NewBool(&d.TLS.Enable, false), "tls.enable", "CORE_TLS_ENABLE", nil, "Enable HTTPS", false, false) d.vars.Register(value.NewBool(&d.TLS.Auto, false), "tls.auto", "CORE_TLS_AUTO", nil, "Enable Let's Encrypt certificate", false, false) d.vars.Register(value.NewEmail(&d.TLS.Email, "cert@datarhei.com"), "tls.email", "CORE_TLS_EMAIL", nil, "Email for Let's Encrypt registration", false, false) - d.vars.Register(value.NewFile(&d.TLS.CertFile, "", d.fs), "tls.cert_file", "CORE_TLS_CERTFILE", nil, "Path to certificate file in PEM format", false, false) - d.vars.Register(value.NewFile(&d.TLS.KeyFile, "", d.fs), "tls.key_file", "CORE_TLS_KEYFILE", nil, "Path to key file in PEM format", false, false) + d.vars.Register(value.NewFile(&d.TLS.CertFile, "", d.fs), "tls.cert_file", "CORE_TLS_CERT_FILE", []string{"CORE_TLS_CERTFILE"}, "Path to certificate file in PEM format", false, false) + d.vars.Register(value.NewFile(&d.TLS.KeyFile, "", d.fs), "tls.key_file", "CORE_TLS_KEY_FILE", []string{"CORE_TLS_KEYFILE"}, "Path to key file in PEM format", false, false) // Storage d.vars.Register(value.NewFile(&d.Storage.MimeTypes, "./mime.types", d.fs), "storage.mimetypes_file", "CORE_STORAGE_MIMETYPES_FILE", []string{"CORE_MIMETYPES_FILE"}, "Path to file with mime-types", false, false) // Storage (Disk) d.vars.Register(value.NewMustDir(&d.Storage.Disk.Dir, "./data", d.fs), "storage.disk.dir", "CORE_STORAGE_DISK_DIR", nil, "Directory on disk, exposed on /", false, false) - d.vars.Register(value.NewInt64(&d.Storage.Disk.Size, 0), "storage.disk.max_size_mbytes", "CORE_STORAGE_DISK_MAXSIZEMBYTES", nil, "Max. allowed megabytes for storage.disk.dir, 0 for unlimited", false, false) + d.vars.Register(value.NewInt64(&d.Storage.Disk.Size, 0), "storage.disk.max_size_mbytes", "CORE_STORAGE_DISK_MAX_SIZE_MBYTES", []string{"CORE_STORAGE_DISK_MAXSIZEMBYTES"}, "Max. allowed megabytes for storage.disk.dir, 0 for unlimited", false, false) d.vars.Register(value.NewBool(&d.Storage.Disk.Cache.Enable, true), "storage.disk.cache.enable", "CORE_STORAGE_DISK_CACHE_ENABLE", nil, "Enable cache for /", false, false) - d.vars.Register(value.NewUint64(&d.Storage.Disk.Cache.Size, 0), "storage.disk.cache.max_size_mbytes", "CORE_STORAGE_DISK_CACHE_MAXSIZEMBYTES", nil, "Max. allowed cache size, 0 for unlimited", false, false) - d.vars.Register(value.NewInt64(&d.Storage.Disk.Cache.TTL, 300), "storage.disk.cache.ttl_seconds", "CORE_STORAGE_DISK_CACHE_TTLSECONDS", nil, "Seconds to keep files in cache", false, false) - d.vars.Register(value.NewUint64(&d.Storage.Disk.Cache.FileSize, 1), "storage.disk.cache.max_file_size_mbytes", "CORE_STORAGE_DISK_CACHE_MAXFILESIZEMBYTES", nil, "Max. file size to put in cache", false, false) + d.vars.Register(value.NewUint64(&d.Storage.Disk.Cache.Size, 0), "storage.disk.cache.max_size_mbytes", "CORE_STORAGE_DISK_CACHE_MAX_SIZE_MBYTES", []string{"CORE_STORAGE_DISK_CACHE_MAXSIZEMBYTES"}, "Max. allowed cache size, 0 for unlimited", false, false) + d.vars.Register(value.NewInt64(&d.Storage.Disk.Cache.TTL, 300), "storage.disk.cache.ttl_seconds", "CORE_STORAGE_DISK_CACHE_TTL_SECONDS", []string{"CORE_STORAGE_DISK_CACHE_TTLSECONDS"}, "Seconds to keep files in cache", false, false) + d.vars.Register(value.NewUint64(&d.Storage.Disk.Cache.FileSize, 1), "storage.disk.cache.max_file_size_mbytes", "CORE_STORAGE_DISK_CACHE_MAX_FILE_SIZE_MBYTES", []string{"CORE_STORAGE_DISK_CACHE_MAXFILESIZEMBYTES"}, "Max. file size to put in cache", false, false) d.vars.Register(value.NewStringList(&d.Storage.Disk.Cache.Types.Allow, []string{}, " "), "storage.disk.cache.type.allow", "CORE_STORAGE_DISK_CACHE_TYPES_ALLOW", []string{"CORE_STORAGE_DISK_CACHE_TYPES"}, "File extensions to cache, empty for all", false, false) d.vars.Register(value.NewStringList(&d.Storage.Disk.Cache.Types.Block, []string{".m3u8", ".mpd"}, " "), "storage.disk.cache.type.block", "CORE_STORAGE_DISK_CACHE_TYPES_BLOCK", nil, "File extensions not to cache, empty for none", false, false) @@ -202,7 +202,7 @@ func (d *Config) init() { d.vars.Register(value.NewBool(&d.Storage.Memory.Auth.Enable, true), "storage.memory.auth.enable", "CORE_STORAGE_MEMORY_AUTH_ENABLE", nil, "Enable basic auth for PUT,POST, and DELETE on /memfs", false, false) d.vars.Register(value.NewString(&d.Storage.Memory.Auth.Username, "admin"), "storage.memory.auth.username", "CORE_STORAGE_MEMORY_AUTH_USERNAME", nil, "Username for Basic-Auth of /memfs", false, false) d.vars.Register(value.NewString(&d.Storage.Memory.Auth.Password, rand.StringAlphanumeric(18)), "storage.memory.auth.password", "CORE_STORAGE_MEMORY_AUTH_PASSWORD", nil, "Password for Basic-Auth of /memfs", false, true) - d.vars.Register(value.NewInt64(&d.Storage.Memory.Size, 0), "storage.memory.max_size_mbytes", "CORE_STORAGE_MEMORY_MAXSIZEMBYTES", nil, "Max. allowed megabytes for /memfs, 0 for unlimited", false, false) + d.vars.Register(value.NewInt64(&d.Storage.Memory.Size, 0), "storage.memory.max_size_mbytes", "CORE_STORAGE_MEMORY_MAX_SIZE_MBYTES", []string{"CORE_STORAGE_MEMORY_MAXSIZEMBYTES"}, "Max. allowed megabytes for /memfs, 0 for unlimited", false, false) d.vars.Register(value.NewBool(&d.Storage.Memory.Purge, false), "storage.memory.purge", "CORE_STORAGE_MEMORY_PURGE", nil, "Automatically remove the oldest files if /memfs is full", false, false) // Storage (S3) @@ -234,17 +234,17 @@ func (d *Config) init() { d.vars.Register(value.NewStringList(&d.FFmpeg.Access.Input.Block, []string{}, " "), "ffmpeg.access.input.block", "CORE_FFMPEG_ACCESS_INPUT_BLOCK", nil, "List of blocked expression to match against the input addresses", false, false) d.vars.Register(value.NewStringList(&d.FFmpeg.Access.Output.Allow, []string{}, " "), "ffmpeg.access.output.allow", "CORE_FFMPEG_ACCESS_OUTPUT_ALLOW", nil, "List of allowed expression to match against the output addresses", false, false) d.vars.Register(value.NewStringList(&d.FFmpeg.Access.Output.Block, []string{}, " "), "ffmpeg.access.output.block", "CORE_FFMPEG_ACCESS_OUTPUT_BLOCK", nil, "List of blocked expression to match against the output addresses", false, false) - d.vars.Register(value.NewInt(&d.FFmpeg.Log.MaxLines, 50), "ffmpeg.log.max_lines", "CORE_FFMPEG_LOG_MAXLINES", nil, "Number of latest log lines to keep for each process", false, false) - d.vars.Register(value.NewInt(&d.FFmpeg.Log.MaxHistory, 3), "ffmpeg.log.max_history", "CORE_FFMPEG_LOG_MAXHISTORY", nil, "Number of latest logs to keep for each process", false, false) + d.vars.Register(value.NewInt(&d.FFmpeg.Log.MaxLines, 50), "ffmpeg.log.max_lines", "CORE_FFMPEG_LOG_MAX_LINES", []string{"CORE_FFMPEG_LOG_MAXLINES"}, "Number of latest log lines to keep for each process", false, false) + d.vars.Register(value.NewInt(&d.FFmpeg.Log.MaxHistory, 3), "ffmpeg.log.max_history", "CORE_FFMPEG_LOG_MAX_HISTORY", []string{"CORE_FFMPEG_LOG_MAXHISTORY"}, "Number of latest logs to keep for each process", false, false) // Playout d.vars.Register(value.NewBool(&d.Playout.Enable, false), "playout.enable", "CORE_PLAYOUT_ENABLE", nil, "Enable playout proxy where available", false, false) - d.vars.Register(value.NewPort(&d.Playout.MinPort, 0), "playout.min_port", "CORE_PLAYOUT_MINPORT", nil, "Min. playout server port", false, false) - d.vars.Register(value.NewPort(&d.Playout.MaxPort, 0), "playout.max_port", "CORE_PLAYOUT_MAXPORT", nil, "Max. playout server port", false, false) + d.vars.Register(value.NewPort(&d.Playout.MinPort, 0), "playout.min_port", "CORE_PLAYOUT_MIN_PORT", []string{"CORE_PLAYOUT_MINPORT"}, "Min. playout server port", false, false) + d.vars.Register(value.NewPort(&d.Playout.MaxPort, 0), "playout.max_port", "CORE_PLAYOUT_MAX_PORT", []string{"CORE_PLAYOUT_MAXPORT"}, "Max. playout server port", false, false) // Debug d.vars.Register(value.NewBool(&d.Debug.Profiling, false), "debug.profiling", "CORE_DEBUG_PROFILING", nil, "Enable profiling endpoint on /profiling", false, false) - d.vars.Register(value.NewInt(&d.Debug.ForceGC, 0), "debug.force_gc", "CORE_DEBUG_FORCEGC", nil, "Number of seconds between forcing GC to return memory to the OS", false, false) + d.vars.Register(value.NewInt(&d.Debug.ForceGC, 0), "debug.force_gc", "CORE_DEBUG_FORCE_GC", []string{"CORE_DEBUG_FORCEGC"}, "Number of seconds between forcing GC to return memory to the OS", false, false) d.vars.Register(value.NewInt64(&d.Debug.MemoryLimit, 0), "debug.memory_limit_mbytes", "CORE_DEBUG_MEMORY_LIMIT_MBYTES", nil, "Impose a soft memory limit for the core, in megabytes", false, false) // Metrics @@ -260,7 +260,7 @@ func (d *Config) init() { d.vars.Register(value.NewBool(&d.Sessions.Persist, false), "sessions.persist", "CORE_SESSIONS_PERSIST", nil, "Whether to persist session history. Will be stored as sessions.json in db.dir", false, false) d.vars.Register(value.NewInt(&d.Sessions.PersistInterval, 300), "sessions.persist_interval_sec", "CORE_SESSIONS_PERSIST_INTERVAL_SEC", nil, "Interval in seconds in which to persist the current session history", false, false) d.vars.Register(value.NewUint64(&d.Sessions.MaxBitrate, 0), "sessions.max_bitrate_mbit", "CORE_SESSIONS_MAXBITRATE_MBIT", nil, "Max. allowed outgoing bitrate in mbit/s, 0 for unlimited", false, false) - d.vars.Register(value.NewUint64(&d.Sessions.MaxSessions, 0), "sessions.max_sessions", "CORE_SESSIONS_MAXSESSIONS", nil, "Max. allowed number of simultaneous sessions, 0 for unlimited", false, false) + d.vars.Register(value.NewUint64(&d.Sessions.MaxSessions, 0), "sessions.max_sessions", "CORE_SESSIONS_MAX_SESSIONS", []string{"CORE_SESSIONS_MAXSESSIONS"}, "Max. allowed number of simultaneous sessions, 0 for unlimited", false, false) // Service d.vars.Register(value.NewBool(&d.Service.Enable, false), "service.enable", "CORE_SERVICE_ENABLE", nil, "Enable connecting to the Restreamer Service", false, false) From 7e9e6fce8df5e0cceede230d840b3894b808397c Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Tue, 4 Apr 2023 20:44:57 +0200 Subject: [PATCH 04/26] Add number of keyframes and extradata size to process progress data --- docs/docs.go | 9 ++++++ docs/swagger.json | 9 ++++++ docs/swagger.yaml | 7 +++++ ffmpeg/parse/parser.go | 26 +++++++++++++++-- ffmpeg/parse/stats.go | 10 +++---- ffmpeg/parse/types.go | 62 +++++++++++++++++++++++++--------------- http/api/progress.go | 28 ++++++++++-------- restream/app/avstream.go | 14 ++++----- restream/app/progress.go | 46 +++++++++++++++-------------- 9 files changed, 139 insertions(+), 72 deletions(-) diff --git a/docs/docs.go b/docs/docs.go index 484bd9ca..5f51006e 100644 --- a/docs/docs.go +++ b/docs/docs.go @@ -3342,6 +3342,11 @@ const docTemplate = `{ "coder": { "type": "string" }, + "extradata_size_bytes": { + "description": "bytes", + "type": "integer", + "format": "uint64" + }, "format": { "type": "string" }, @@ -3364,6 +3369,10 @@ const docTemplate = `{ "type": "integer", "format": "uint64" }, + "keyframe": { + "type": "integer", + "format": "uint64" + }, "layout": { "type": "string" }, diff --git a/docs/swagger.json b/docs/swagger.json index a76838f6..426d1075 100644 --- a/docs/swagger.json +++ b/docs/swagger.json @@ -3335,6 +3335,11 @@ "coder": { "type": "string" }, + "extradata_size_bytes": { + "description": "bytes", + "type": "integer", + "format": "uint64" + }, "format": { "type": "string" }, @@ -3357,6 +3362,10 @@ "type": "integer", "format": "uint64" }, + "keyframe": { + "type": "integer", + "format": "uint64" + }, "layout": { "type": "string" }, diff --git a/docs/swagger.yaml b/docs/swagger.yaml index 43e9479f..10ddcaaa 100644 --- a/docs/swagger.yaml +++ b/docs/swagger.yaml @@ -902,6 +902,10 @@ definitions: type: string coder: type: string + extradata_size_bytes: + description: bytes + format: uint64 + type: integer format: type: string fps: @@ -918,6 +922,9 @@ definitions: description: General format: uint64 type: integer + keyframe: + format: uint64 + type: integer layout: type: string packet: diff --git a/ffmpeg/parse/parser.go b/ffmpeg/parse/parser.go index 0adb53d2..edf0ca03 100644 --- a/ffmpeg/parse/parser.go +++ b/ffmpeg/parse/parser.go @@ -356,7 +356,7 @@ func (p *parser) Parse(line string) uint64 { if p.collector.IsCollectableIP(p.process.input[i].IP) { p.collector.Activate("") - p.collector.Ingress("", int64(p.stats.input[i].diff.size)*1024) + p.collector.Ingress("", int64(p.stats.input[i].diff.size)) } } } @@ -373,7 +373,7 @@ func (p *parser) Parse(line string) uint64 { if p.collector.IsCollectableIP(p.process.output[i].IP) { p.collector.Activate("") - p.collector.Egress("", int64(p.stats.output[i].diff.size)*1024) + p.collector.Egress("", int64(p.stats.output[i].diff.size)) } } } @@ -410,7 +410,7 @@ func (p *parser) parseDefaultProgress(line string) error { if matches = p.re.size.FindStringSubmatch(line); matches != nil { if x, err := strconv.ParseUint(matches[1], 10, 64); err == nil { - p.progress.ffmpeg.Size = x + p.progress.ffmpeg.Size = x * 1024 } } @@ -485,6 +485,26 @@ func (p *parser) parseFFmpegProgress(line string) error { return fmt.Errorf("output length mismatch (have: %d, want: %d)", len(progress.Output), len(p.process.output)) } + if progress.Size == 0 { + progress.Size = progress.SizeKB * 1024 + } + + for i, io := range progress.Input { + if io.Size == 0 { + io.Size = io.SizeKB * 1024 + } + + progress.Input[i].Size = io.Size + } + + for i, io := range progress.Output { + if io.Size == 0 { + io.Size = io.SizeKB * 1024 + } + + progress.Output[i].Size = io.Size + } + p.progress.ffmpeg = progress return nil diff --git a/ffmpeg/parse/stats.go b/ffmpeg/parse/stats.go index df7e714b..d36ecb82 100644 --- a/ffmpeg/parse/stats.go +++ b/ffmpeg/parse/stats.go @@ -1,11 +1,11 @@ package parse type statsData struct { - frame uint64 - packet uint64 - size uint64 // kbytes - dup uint64 - drop uint64 + frame uint64 // counter + packet uint64 // counter + size uint64 // bytes + dup uint64 // counter + drop uint64 // counter } type stats struct { diff --git a/ffmpeg/parse/types.go b/ffmpeg/parse/types.go index bf031fb0..aa0c33e3 100644 --- a/ffmpeg/parse/types.go +++ b/ffmpeg/parse/types.go @@ -44,9 +44,9 @@ func (d *Duration) UnmarshalJSON(b []byte) error { type ffmpegAVstreamIO struct { State string `json:"state"` - Packet uint64 `json:"packet"` + Packet uint64 `json:"packet"` // counter Time uint64 `json:"time"` - Size uint64 `json:"size_kb"` + Size uint64 `json:"size_kb"` // kbytes } func (avio *ffmpegAVstreamIO) export() app.AVstreamIO { @@ -54,7 +54,7 @@ func (avio *ffmpegAVstreamIO) export() app.AVstreamIO { State: avio.State, Packet: avio.Packet, Time: avio.Time, - Size: avio.Size, + Size: avio.Size * 1024, } } @@ -91,14 +91,17 @@ func (av *ffmpegAVstream) export() *app.AVstream { type ffmpegProgressIO struct { // common - Index uint64 `json:"index"` - Stream uint64 `json:"stream"` - Size uint64 `json:"size_kb"` // kbytes - Bitrate float64 `json:"-"` // kbit/s - Frame uint64 `json:"frame"` - Packet uint64 `json:"packet"` - FPS float64 `json:"-"` - PPS float64 `json:"-"` + Index uint64 `json:"index"` + Stream uint64 `json:"stream"` + SizeKB uint64 `json:"size_kb"` // kbytes + Size uint64 `json:"size_bytes"` // bytes + Bitrate float64 `json:"-"` // bit/s + Frame uint64 `json:"frame"` // counter + Keyframe uint64 `json:"keyframe"` // counter + Packet uint64 `json:"packet"` // counter + Extradata uint64 `json:"extradata_size_bytes"` // bytes + FPS float64 `json:"-"` // rate, frames per second + PPS float64 `json:"-"` // rate, packets per second // video Quantizer float64 `json:"q"` @@ -108,28 +111,36 @@ func (io *ffmpegProgressIO) exportTo(progress *app.ProgressIO) { progress.Index = io.Index progress.Stream = io.Stream progress.Frame = io.Frame + progress.Keyframe = io.Keyframe progress.Packet = io.Packet progress.FPS = io.FPS progress.PPS = io.PPS progress.Quantizer = io.Quantizer - progress.Size = io.Size * 1024 - progress.Bitrate = io.Bitrate * 1024 + progress.Bitrate = io.Bitrate + progress.Extradata = io.Extradata + + if io.Size == 0 { + progress.Size = io.SizeKB * 1024 + } else { + progress.Size = io.Size + } } type ffmpegProgress struct { Input []ffmpegProgressIO `json:"inputs"` Output []ffmpegProgressIO `json:"outputs"` - Frame uint64 `json:"frame"` - Packet uint64 `json:"packet"` - FPS float64 `json:"-"` - PPS float64 `json:"-"` + Frame uint64 `json:"frame"` // counter + Packet uint64 `json:"packet"` // counter + FPS float64 `json:"-"` // rate, frames per second + PPS float64 `json:"-"` // rate, packets per second Quantizer float64 `json:"q"` - Size uint64 `json:"size_kb"` // kbytes - Bitrate float64 `json:"-"` // kbit/s + SizeKB uint64 `json:"size_kb"` // kbytes + Size uint64 `json:"size_bytes"` // bytes + Bitrate float64 `json:"-"` // bit/s Time Duration `json:"time"` Speed float64 `json:"speed"` - Drop uint64 `json:"drop"` - Dup uint64 `json:"dup"` + Drop uint64 `json:"drop"` // counter + Dup uint64 `json:"dup"` // counter } func (p *ffmpegProgress) exportTo(progress *app.Progress) { @@ -138,13 +149,18 @@ func (p *ffmpegProgress) exportTo(progress *app.Progress) { progress.FPS = p.FPS progress.PPS = p.PPS progress.Quantizer = p.Quantizer - progress.Size = p.Size * 1024 progress.Time = p.Time.Seconds() - progress.Bitrate = p.Bitrate * 1024 + progress.Bitrate = p.Bitrate progress.Speed = p.Speed progress.Drop = p.Drop progress.Dup = p.Dup + if p.Size == 0 { + progress.Size = p.SizeKB * 1024 + } else { + progress.Size = p.Size + } + for i := range p.Input { if len(progress.Input) <= i { break diff --git a/http/api/progress.go b/http/api/progress.go index a402d55a..1bf22c59 100644 --- a/http/api/progress.go +++ b/http/api/progress.go @@ -13,18 +13,20 @@ type ProgressIO struct { Address string `json:"address" jsonschema:"minLength=1"` // General - Index uint64 `json:"index" format:"uint64"` - Stream uint64 `json:"stream" format:"uint64"` - Format string `json:"format"` - Type string `json:"type"` - Codec string `json:"codec"` - Coder string `json:"coder"` - Frame uint64 `json:"frame" format:"uint64"` - FPS json.Number `json:"fps" swaggertype:"number" jsonschema:"type=number"` - Packet uint64 `json:"packet" format:"uint64"` - PPS json.Number `json:"pps" swaggertype:"number" jsonschema:"type=number"` - Size uint64 `json:"size_kb" format:"uint64"` // kbytes - Bitrate json.Number `json:"bitrate_kbit" swaggertype:"number" jsonschema:"type=number"` // kbit/s + Index uint64 `json:"index" format:"uint64"` + Stream uint64 `json:"stream" format:"uint64"` + Format string `json:"format"` + Type string `json:"type"` + Codec string `json:"codec"` + Coder string `json:"coder"` + Frame uint64 `json:"frame" format:"uint64"` + Keyframe uint64 `json:"keyframe" format:"uint64"` + FPS json.Number `json:"fps" swaggertype:"number" jsonschema:"type=number"` + Packet uint64 `json:"packet" format:"uint64"` + PPS json.Number `json:"pps" swaggertype:"number" jsonschema:"type=number"` + Size uint64 `json:"size_kb" format:"uint64"` // kbytes + Bitrate json.Number `json:"bitrate_kbit" swaggertype:"number" jsonschema:"type=number"` // kbit/s + Extradata uint64 `json:"extradata_size_bytes" format:"uint64"` // bytes // Video Pixfmt string `json:"pix_fmt,omitempty"` @@ -56,11 +58,13 @@ func (i *ProgressIO) Unmarshal(io *app.ProgressIO) { i.Codec = io.Codec i.Coder = io.Coder i.Frame = io.Frame + i.Keyframe = io.Keyframe i.FPS = json.Number(fmt.Sprintf("%.3f", io.FPS)) i.Packet = io.Packet i.PPS = json.Number(fmt.Sprintf("%.3f", io.PPS)) i.Size = io.Size / 1024 i.Bitrate = json.Number(fmt.Sprintf("%.3f", io.Bitrate/1024)) + i.Extradata = io.Extradata i.Pixfmt = io.Pixfmt i.Quantizer = json.Number(fmt.Sprintf("%.3f", io.Quantizer)) i.Width = io.Width diff --git a/restream/app/avstream.go b/restream/app/avstream.go index fcfb8ded..70cf9634 100644 --- a/restream/app/avstream.go +++ b/restream/app/avstream.go @@ -2,19 +2,19 @@ package app type AVstreamIO struct { State string - Packet uint64 + Packet uint64 // counter Time uint64 - Size uint64 + Size uint64 // bytes } type AVstream struct { Input AVstreamIO Output AVstreamIO - Aqueue uint64 - Queue uint64 - Dup uint64 - Drop uint64 - Enc uint64 + Aqueue uint64 // gauge + Queue uint64 // gauge + Dup uint64 // counter + Drop uint64 // counter + Enc uint64 // counter Looping bool Duplicating bool GOP string diff --git a/restream/app/progress.go b/restream/app/progress.go index 7d081d39..c9f1fcd5 100644 --- a/restream/app/progress.go +++ b/restream/app/progress.go @@ -5,18 +5,20 @@ type ProgressIO struct { Address string // General - Index uint64 - Stream uint64 - Format string - Type string - Codec string - Coder string - Frame uint64 - FPS float64 - Packet uint64 - PPS float64 - Size uint64 // bytes - Bitrate float64 // bit/s + Index uint64 + Stream uint64 + Format string + Type string + Codec string + Coder string + Frame uint64 // counter + Keyframe uint64 // counter + FPS float64 // rate, frames per second + Packet uint64 // counter + PPS float64 // rate, packets per second + Size uint64 // bytes + Bitrate float64 // bit/s + Extradata uint64 // bytes // Video Pixfmt string @@ -36,15 +38,15 @@ type ProgressIO struct { type Progress struct { Input []ProgressIO Output []ProgressIO - Frame uint64 - Packet uint64 - FPS float64 - PPS float64 - Quantizer float64 - Size uint64 // bytes - Time float64 + Frame uint64 // counter + Packet uint64 // counter + FPS float64 // rate, frames per second + PPS float64 // rate, packets per second + Quantizer float64 // gauge + Size uint64 // bytes + Time float64 // seconds with fractions Bitrate float64 // bit/s - Speed float64 - Drop uint64 - Dup uint64 + Speed float64 // gauge + Drop uint64 // counter + Dup uint64 // counter } From 0e73a0fdf3b0384d08f8e19492966007d55afbb0 Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Tue, 11 Apr 2023 14:52:27 +0200 Subject: [PATCH 05/26] Clarify metric descriptions --- monitor/ffmpeg.go | 2 +- monitor/restream.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/monitor/ffmpeg.go b/monitor/ffmpeg.go index a447901a..7d76d430 100644 --- a/monitor/ffmpeg.go +++ b/monitor/ffmpeg.go @@ -17,7 +17,7 @@ func NewFFmpegCollector(f ffmpeg.FFmpeg) metric.Collector { ffmpeg: f, } - c.processDescr = metric.NewDesc("ffmpeg_process", "State of the ffmpeg process", []string{"state"}) + c.processDescr = metric.NewDesc("ffmpeg_process", "Accumulated state changes of all ffmpeg processes", []string{"state"}) return c } diff --git a/monitor/restream.go b/monitor/restream.go index cfd069f4..c83e17ff 100644 --- a/monitor/restream.go +++ b/monitor/restream.go @@ -25,7 +25,7 @@ func NewRestreamCollector(r restream.Restreamer) metric.Collector { c.restreamProcessDescr = metric.NewDesc("restream_process", "Current process values by name", []string{"processid", "state", "order", "name"}) c.restreamProcessStatesDescr = metric.NewDesc("restream_process_states", "Current process state", []string{"processid", "state"}) c.restreamProcessIODescr = metric.NewDesc("restream_io", "Current process IO values by name", []string{"processid", "type", "id", "address", "index", "stream", "media", "name"}) - c.restreamStatesDescr = metric.NewDesc("restream_state", "Summarized process states", []string{"state"}) + c.restreamStatesDescr = metric.NewDesc("restream_state", "Summarized current process states", []string{"state"}) return c } From aef1b7c9a2d77b5ac008c0053d978b35c2c62492 Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Tue, 11 Apr 2023 15:04:31 +0200 Subject: [PATCH 06/26] Fix #10 --- go.mod | 7 +- go.sum | 15 +- psutil/psutil.go | 5 + .../shirou/gopsutil/v3/cpu/cpu_aix_nocgo.go | 22 +- .../shirou/gopsutil/v3/cpu/cpu_darwin.go | 15 +- .../shirou/gopsutil/v3/cpu/cpu_linux.go | 30 +- .../shirou/gopsutil/v3/cpu/cpu_windows.go | 14 +- .../shirou/gopsutil/v3/disk/disk.go | 2 + .../shirou/gopsutil/v3/disk/disk_aix_nocgo.go | 14 +- .../shirou/gopsutil/v3/disk/disk_darwin.go | 8 +- .../shirou/gopsutil/v3/disk/disk_freebsd.go | 3 +- .../shirou/gopsutil/v3/disk/disk_linux.go | 2 +- .../shirou/gopsutil/v3/disk/disk_windows.go | 224 +++++------ .../gopsutil/v3/internal/common/binary.go | 1 + .../gopsutil/v3/internal/common/common.go | 11 +- .../v3/internal/common/common_linux.go | 5 + .../v3/internal/common/common_windows.go | 7 +- .../shirou/gopsutil/v3/mem/mem_aix_nocgo.go | 2 +- .../shirou/gopsutil/v3/mem/mem_freebsd.go | 3 +- .../shirou/gopsutil/v3/mem/mem_openbsd.go | 1 + .../shirou/gopsutil/v3/net/net_aix_cgo.go | 12 +- .../shirou/gopsutil/v3/net/net_windows.go | 3 +- .../gopsutil/v3/process/process_freebsd.go | 8 +- .../v3/process/process_windows_32bit.go | 3 +- .../shoenig/go-m1cpu/.golangci.yaml | 12 + vendor/github.com/shoenig/go-m1cpu/LICENSE | 363 ++++++++++++++++++ vendor/github.com/shoenig/go-m1cpu/Makefile | 12 + vendor/github.com/shoenig/go-m1cpu/README.md | 66 ++++ vendor/github.com/shoenig/go-m1cpu/cpu.go | 208 ++++++++++ .../shoenig/go-m1cpu/incompatible.go | 53 +++ .../stretchr/testify/assert/assertions.go | 78 ++-- vendor/golang.org/x/sys/cpu/hwcap_linux.go | 15 + vendor/golang.org/x/sys/cpu/runtime_auxv.go | 16 + .../x/sys/cpu/runtime_auxv_go121.go | 19 + vendor/golang.org/x/sys/execabs/execabs.go | 2 +- .../golang.org/x/sys/execabs/execabs_go118.go | 6 + .../golang.org/x/sys/execabs/execabs_go119.go | 4 + vendor/golang.org/x/sys/unix/ioctl.go | 17 +- vendor/golang.org/x/sys/unix/ioctl_zos.go | 8 +- vendor/golang.org/x/sys/unix/ptrace_darwin.go | 6 + vendor/golang.org/x/sys/unix/ptrace_ios.go | 6 + vendor/golang.org/x/sys/unix/syscall_aix.go | 5 +- vendor/golang.org/x/sys/unix/syscall_bsd.go | 3 +- .../golang.org/x/sys/unix/syscall_darwin.go | 12 +- .../x/sys/unix/syscall_darwin_amd64.go | 1 + .../x/sys/unix/syscall_darwin_arm64.go | 1 + .../x/sys/unix/syscall_dragonfly.go | 1 + .../golang.org/x/sys/unix/syscall_freebsd.go | 43 ++- .../x/sys/unix/syscall_freebsd_386.go | 17 +- .../x/sys/unix/syscall_freebsd_amd64.go | 17 +- .../x/sys/unix/syscall_freebsd_arm.go | 15 +- .../x/sys/unix/syscall_freebsd_arm64.go | 15 +- .../x/sys/unix/syscall_freebsd_riscv64.go | 15 +- vendor/golang.org/x/sys/unix/syscall_hurd.go | 8 + vendor/golang.org/x/sys/unix/syscall_linux.go | 36 +- .../golang.org/x/sys/unix/syscall_netbsd.go | 5 +- .../golang.org/x/sys/unix/syscall_openbsd.go | 1 + .../golang.org/x/sys/unix/syscall_solaris.go | 21 +- .../x/sys/unix/syscall_zos_s390x.go | 4 +- vendor/golang.org/x/sys/unix/zerrors_linux.go | 10 +- .../x/sys/unix/zptrace_armnn_linux.go | 8 +- .../x/sys/unix/zptrace_linux_arm64.go | 4 +- .../x/sys/unix/zptrace_mipsnn_linux.go | 8 +- .../x/sys/unix/zptrace_mipsnnle_linux.go | 8 +- .../x/sys/unix/zptrace_x86_linux.go | 8 +- .../golang.org/x/sys/unix/zsyscall_aix_ppc.go | 10 + .../x/sys/unix/zsyscall_aix_ppc64.go | 10 + .../x/sys/unix/zsyscall_aix_ppc64_gc.go | 7 + .../x/sys/unix/zsyscall_aix_ppc64_gccgo.go | 8 + .../x/sys/unix/zsyscall_darwin_amd64.go | 16 + .../x/sys/unix/zsyscall_darwin_arm64.go | 16 + .../x/sys/unix/zsyscall_dragonfly_amd64.go | 10 + .../x/sys/unix/zsyscall_freebsd_386.go | 20 + .../x/sys/unix/zsyscall_freebsd_amd64.go | 20 + .../x/sys/unix/zsyscall_freebsd_arm.go | 20 + .../x/sys/unix/zsyscall_freebsd_arm64.go | 20 + .../x/sys/unix/zsyscall_freebsd_riscv64.go | 20 + .../golang.org/x/sys/unix/zsyscall_linux.go | 10 + .../x/sys/unix/zsyscall_netbsd_386.go | 10 + .../x/sys/unix/zsyscall_netbsd_amd64.go | 10 + .../x/sys/unix/zsyscall_netbsd_arm.go | 10 + .../x/sys/unix/zsyscall_netbsd_arm64.go | 10 + .../x/sys/unix/zsyscall_openbsd_386.go | 8 + .../x/sys/unix/zsyscall_openbsd_amd64.go | 8 + .../x/sys/unix/zsyscall_openbsd_arm.go | 8 + .../x/sys/unix/zsyscall_openbsd_arm64.go | 8 + .../x/sys/unix/zsyscall_openbsd_mips64.go | 8 + .../x/sys/unix/zsyscall_openbsd_ppc64.go | 8 + .../x/sys/unix/zsyscall_openbsd_riscv64.go | 8 + .../x/sys/unix/zsyscall_solaris_amd64.go | 11 + .../x/sys/unix/zsyscall_zos_s390x.go | 10 + .../x/sys/unix/ztypes_freebsd_386.go | 2 +- .../x/sys/unix/ztypes_freebsd_amd64.go | 2 +- .../x/sys/unix/ztypes_freebsd_arm.go | 2 +- .../x/sys/unix/ztypes_freebsd_arm64.go | 2 +- .../x/sys/unix/ztypes_freebsd_riscv64.go | 2 +- vendor/golang.org/x/sys/unix/ztypes_linux.go | 140 +++++-- .../golang.org/x/sys/unix/ztypes_linux_386.go | 2 +- .../x/sys/unix/ztypes_linux_amd64.go | 2 +- .../golang.org/x/sys/unix/ztypes_linux_arm.go | 2 +- .../x/sys/unix/ztypes_linux_arm64.go | 2 +- .../x/sys/unix/ztypes_linux_loong64.go | 2 +- .../x/sys/unix/ztypes_linux_mips.go | 2 +- .../x/sys/unix/ztypes_linux_mips64.go | 2 +- .../x/sys/unix/ztypes_linux_mips64le.go | 2 +- .../x/sys/unix/ztypes_linux_mipsle.go | 2 +- .../golang.org/x/sys/unix/ztypes_linux_ppc.go | 2 +- .../x/sys/unix/ztypes_linux_ppc64.go | 2 +- .../x/sys/unix/ztypes_linux_ppc64le.go | 2 +- .../x/sys/unix/ztypes_linux_riscv64.go | 2 +- .../x/sys/unix/ztypes_linux_s390x.go | 2 +- .../x/sys/unix/ztypes_linux_sparc64.go | 2 +- .../x/sys/windows/syscall_windows.go | 6 +- .../golang.org/x/sys/windows/types_windows.go | 85 ++++ .../x/sys/windows/zsyscall_windows.go | 27 ++ vendor/modules.txt | 9 +- 116 files changed, 1731 insertions(+), 454 deletions(-) create mode 100644 vendor/github.com/shoenig/go-m1cpu/.golangci.yaml create mode 100644 vendor/github.com/shoenig/go-m1cpu/LICENSE create mode 100644 vendor/github.com/shoenig/go-m1cpu/Makefile create mode 100644 vendor/github.com/shoenig/go-m1cpu/README.md create mode 100644 vendor/github.com/shoenig/go-m1cpu/cpu.go create mode 100644 vendor/github.com/shoenig/go-m1cpu/incompatible.go create mode 100644 vendor/golang.org/x/sys/cpu/runtime_auxv.go create mode 100644 vendor/golang.org/x/sys/cpu/runtime_auxv_go121.go diff --git a/go.mod b/go.mod index e13a2ff6..454d2310 100644 --- a/go.mod +++ b/go.mod @@ -21,8 +21,8 @@ require ( github.com/minio/minio-go/v7 v7.0.47 github.com/prep/average v0.0.0-20200506183628-d26c465f48c3 github.com/prometheus/client_golang v1.14.0 - github.com/shirou/gopsutil/v3 v3.22.11 - github.com/stretchr/testify v1.8.1 + github.com/shirou/gopsutil/v3 v3.23.3 + github.com/stretchr/testify v1.8.2 github.com/swaggo/echo-swagger v1.3.5 github.com/swaggo/swag v1.8.7 github.com/vektah/gqlparser/v2 v2.5.1 @@ -80,6 +80,7 @@ require ( github.com/rogpeppe/go-internal v1.8.1 // indirect github.com/rs/xid v1.4.0 // indirect github.com/russross/blackfriday/v2 v2.1.0 // indirect + github.com/shoenig/go-m1cpu v0.1.4 // indirect github.com/sirupsen/logrus v1.9.0 // indirect github.com/swaggo/files v0.0.0-20220728132757-551d4a08d97a // indirect github.com/tklauser/go-sysconf v0.3.11 // indirect @@ -95,7 +96,7 @@ require ( go.uber.org/goleak v1.1.12 // indirect go.uber.org/multierr v1.9.0 // indirect golang.org/x/crypto v0.5.0 // indirect - golang.org/x/sys v0.5.0 // indirect + golang.org/x/sys v0.6.0 // indirect golang.org/x/text v0.7.0 // indirect golang.org/x/time v0.3.0 // indirect golang.org/x/tools v0.4.0 // indirect diff --git a/go.sum b/go.sum index 788ba6ff..dd05bdd5 100644 --- a/go.sum +++ b/go.sum @@ -204,8 +204,12 @@ github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/sergi/go-diff v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0= github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM= -github.com/shirou/gopsutil/v3 v3.22.11 h1:kxsPKS+Eeo+VnEQ2XCaGJepeP6KY53QoRTETx3+1ndM= -github.com/shirou/gopsutil/v3 v3.22.11/go.mod h1:xl0EeL4vXJ+hQMAGN8B9VFpxukEMA0XdevQOe5MZ1oY= +github.com/shirou/gopsutil/v3 v3.23.3 h1:Syt5vVZXUDXPEXpIBt5ziWsJ4LdSAAxF4l/xZeQgSEE= +github.com/shirou/gopsutil/v3 v3.23.3/go.mod h1:lSBNN6t3+D6W5e5nXTxc8KIMMVxAcS+6IJlffjRRlMU= +github.com/shoenig/go-m1cpu v0.1.4 h1:SZPIgRM2sEF9NJy50mRHu9PKGwxyyTTJIWvCtgVbozs= +github.com/shoenig/go-m1cpu v0.1.4/go.mod h1:Wwvst4LR89UxjeFtLRMrpgRiyY4xPsejnVZym39dbAQ= +github.com/shoenig/test v0.6.3 h1:GVXWJFk9PiOjN0KoJ7VrJGH6uLPnqxR7/fe3HUPfE0c= +github.com/shoenig/test v0.6.3/go.mod h1:byHiCGXqrVaflBLAMq/srcZIHynQPQgeyvkvXnjqq0k= github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc= github.com/sirupsen/logrus v1.9.0 h1:trlNQbNUG3OdDrDil03MCb1H2o9nJ1x4/5LYw7byDE0= github.com/sirupsen/logrus v1.9.0/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= @@ -221,8 +225,9 @@ github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= -github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= +github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8= +github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/swaggo/echo-swagger v1.3.5 h1:kCx1wvX5AKhjI6Ykt48l3PTsfL9UD40ZROOx/tYzWyY= github.com/swaggo/echo-swagger v1.3.5/go.mod h1:3IMHd2Z8KftdWFEEjGmv6QpWj370LwMCOfovuh7vF34= github.com/swaggo/files v0.0.0-20220728132757-551d4a08d97a h1:kAe4YSu0O0UFn1DowNo2MY5p6xzqtJ/wQ7LZynSvGaY= @@ -332,8 +337,8 @@ golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU= -golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ= +golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= diff --git a/psutil/psutil.go b/psutil/psutil.go index b2a33a4f..16b306bd 100644 --- a/psutil/psutil.go +++ b/psutil/psutil.go @@ -2,6 +2,7 @@ package psutil import ( "context" + "errors" "fmt" "io" "io/fs" @@ -284,6 +285,10 @@ func (u *util) cpuTimes() (*cpuTimesStat, error) { return nil, err } + if len(times) == 0 { + return nil, errors.New("cpu.Times() returned an empty slice") + } + s := &cpuTimesStat{ total: cpuTotal(×[0]), system: times[0].System, diff --git a/vendor/github.com/shirou/gopsutil/v3/cpu/cpu_aix_nocgo.go b/vendor/github.com/shirou/gopsutil/v3/cpu/cpu_aix_nocgo.go index d158000e..1a291532 100644 --- a/vendor/github.com/shirou/gopsutil/v3/cpu/cpu_aix_nocgo.go +++ b/vendor/github.com/shirou/gopsutil/v3/cpu/cpu_aix_nocgo.go @@ -6,8 +6,8 @@ package cpu import ( "context" "regexp" - "strings" "strconv" + "strings" "github.com/shirou/gopsutil/v3/internal/common" ) @@ -28,19 +28,19 @@ func TimesWithContext(ctx context.Context, percpu bool) ([]TimesStat, error) { } ret := TimesStat{CPU: "cpu-total"} - h := whiteSpaces.Split(lines[len(lines)-3], -1) // headers - v := whiteSpaces.Split(lines[len(lines)-2], -1) // values + h := whiteSpaces.Split(lines[len(lines)-3], -1) // headers + v := whiteSpaces.Split(lines[len(lines)-2], -1) // values for i, header := range h { if t, err := strconv.ParseFloat(v[i], 64); err == nil { switch header { - case `%usr`: - ret.User = t - case `%sys`: - ret.System = t - case `%wio`: - ret.Iowait = t - case `%idle`: - ret.Idle = t + case `%usr`: + ret.User = t + case `%sys`: + ret.System = t + case `%wio`: + ret.Iowait = t + case `%idle`: + ret.Idle = t } } } diff --git a/vendor/github.com/shirou/gopsutil/v3/cpu/cpu_darwin.go b/vendor/github.com/shirou/gopsutil/v3/cpu/cpu_darwin.go index 7acb258d..41f395e5 100644 --- a/vendor/github.com/shirou/gopsutil/v3/cpu/cpu_darwin.go +++ b/vendor/github.com/shirou/gopsutil/v3/cpu/cpu_darwin.go @@ -8,6 +8,7 @@ import ( "strconv" "strings" + "github.com/shoenig/go-m1cpu" "github.com/tklauser/go-sysconf" "golang.org/x/sys/unix" ) @@ -85,11 +86,15 @@ func InfoWithContext(ctx context.Context) ([]InfoStat, error) { c.CacheSize = int32(cacheSize) c.VendorID, _ = unix.Sysctl("machdep.cpu.vendor") - // Use the rated frequency of the CPU. This is a static value and does not - // account for low power or Turbo Boost modes. - cpuFrequency, err := unix.SysctlUint64("hw.cpufrequency") - if err == nil { - c.Mhz = float64(cpuFrequency) / 1000000.0 + if m1cpu.IsAppleSilicon() { + c.Mhz = float64(m1cpu.PCoreHz() / 1_000_000) + } else { + // Use the rated frequency of the CPU. This is a static value and does not + // account for low power or Turbo Boost modes. + cpuFrequency, err := unix.SysctlUint64("hw.cpufrequency") + if err == nil { + c.Mhz = float64(cpuFrequency) / 1000000.0 + } } return append(ret, c), nil diff --git a/vendor/github.com/shirou/gopsutil/v3/cpu/cpu_linux.go b/vendor/github.com/shirou/gopsutil/v3/cpu/cpu_linux.go index 44bf1ce4..962d3430 100644 --- a/vendor/github.com/shirou/gopsutil/v3/cpu/cpu_linux.go +++ b/vendor/github.com/shirou/gopsutil/v3/cpu/cpu_linux.go @@ -190,7 +190,7 @@ func InfoWithContext(ctx context.Context) ([]InfoStat, error) { switch key { case "Processor": processorName = value - case "processor": + case "processor", "cpu number": if c.CPU >= 0 { finishCPUInfo(&c) ret = append(ret, c) @@ -203,6 +203,9 @@ func InfoWithContext(ctx context.Context) ([]InfoStat, error) { c.CPU = int32(t) case "vendorId", "vendor_id": c.VendorID = value + if strings.Contains(value, "S390") { + processorName = "S390" + } case "CPU implementer": if v, err := strconv.ParseUint(value, 0, 8); err == nil { switch v { @@ -242,7 +245,18 @@ func InfoWithContext(ctx context.Context) ([]InfoStat, error) { c.Family = value case "model", "CPU part": c.Model = value - case "model name", "cpu": + // if CPU is arm based, model name is found via model number. refer to: arch/arm64/kernel/cpuinfo.c + if c.VendorID == "ARM" { + if v, err := strconv.ParseUint(c.Model, 0, 16); err == nil { + modelName, exist := armModelToModelName[v] + if exist { + c.ModelName = modelName + } else { + c.ModelName = "Undefined" + } + } + } + case "Model Name", "model name", "cpu": c.ModelName = value if strings.Contains(value, "POWER8") || strings.Contains(value, "POWER7") { @@ -262,7 +276,7 @@ func InfoWithContext(ctx context.Context) ([]InfoStat, error) { return ret, err } c.Stepping = int32(t) - case "cpu MHz", "clock": + case "cpu MHz", "clock", "cpu MHz dynamic": // treat this as the fallback value, thus we ignore error if t, err := strconv.ParseFloat(strings.Replace(value, "MHz", "", 1), 64); err == nil { c.Mhz = t @@ -285,16 +299,6 @@ func InfoWithContext(ctx context.Context) ([]InfoStat, error) { c.Microcode = value } } - if c.VendorID == "ARM" && c.ModelName == "" { - if v, err := strconv.ParseUint(c.Model, 0, 16); err == nil { - modelName, exist := armModelToModelName[v] - if exist { - c.ModelName = modelName - } else { - c.ModelName = "Undefined" - } - } - } if c.CPU >= 0 { finishCPUInfo(&c) ret = append(ret, c) diff --git a/vendor/github.com/shirou/gopsutil/v3/cpu/cpu_windows.go b/vendor/github.com/shirou/gopsutil/v3/cpu/cpu_windows.go index d1a0e4cd..e10612fd 100644 --- a/vendor/github.com/shirou/gopsutil/v3/cpu/cpu_windows.go +++ b/vendor/github.com/shirou/gopsutil/v3/cpu/cpu_windows.go @@ -14,8 +14,7 @@ import ( ) var ( - procGetActiveProcessorCount = common.Modkernel32.NewProc("GetActiveProcessorCount") - procGetNativeSystemInfo = common.Modkernel32.NewProc("GetNativeSystemInfo") + procGetNativeSystemInfo = common.Modkernel32.NewProc("GetNativeSystemInfo") ) type win32_Processor struct { @@ -204,15 +203,12 @@ type systemInfo struct { func CountsWithContext(ctx context.Context, logical bool) (int, error) { if logical { // https://github.com/giampaolo/psutil/blob/d01a9eaa35a8aadf6c519839e987a49d8be2d891/psutil/_psutil_windows.c#L97 - err := procGetActiveProcessorCount.Find() - if err == nil { // Win7+ - ret, _, _ := procGetActiveProcessorCount.Call(uintptr(0xffff)) // ALL_PROCESSOR_GROUPS is 0xffff according to Rust's winapi lib https://docs.rs/winapi/*/x86_64-pc-windows-msvc/src/winapi/shared/ntdef.rs.html#120 - if ret != 0 { - return int(ret), nil - } + ret := windows.GetActiveProcessorCount(windows.ALL_PROCESSOR_GROUPS) + if ret != 0 { + return int(ret), nil } var systemInfo systemInfo - _, _, err = procGetNativeSystemInfo.Call(uintptr(unsafe.Pointer(&systemInfo))) + _, _, err := procGetNativeSystemInfo.Call(uintptr(unsafe.Pointer(&systemInfo))) if systemInfo.dwNumberOfProcessors == 0 { return 0, err } diff --git a/vendor/github.com/shirou/gopsutil/v3/disk/disk.go b/vendor/github.com/shirou/gopsutil/v3/disk/disk.go index dd4cc1d5..0d4b2534 100644 --- a/vendor/github.com/shirou/gopsutil/v3/disk/disk.go +++ b/vendor/github.com/shirou/gopsutil/v3/disk/disk.go @@ -9,6 +9,8 @@ import ( var invoke common.Invoker = common.Invoke{} +type Warnings = common.Warnings + type UsageStat struct { Path string `json:"path"` Fstype string `json:"fstype"` diff --git a/vendor/github.com/shirou/gopsutil/v3/disk/disk_aix_nocgo.go b/vendor/github.com/shirou/gopsutil/v3/disk/disk_aix_nocgo.go index 190a2d90..4f93c752 100644 --- a/vendor/github.com/shirou/gopsutil/v3/disk/disk_aix_nocgo.go +++ b/vendor/github.com/shirou/gopsutil/v3/disk/disk_aix_nocgo.go @@ -8,8 +8,8 @@ import ( "regexp" "strings" - "golang.org/x/sys/unix" "github.com/shirou/gopsutil/v3/internal/common" + "golang.org/x/sys/unix" ) var whiteSpaces = regexp.MustCompile(`\s+`) @@ -17,11 +17,11 @@ var startBlank = regexp.MustCompile(`^\s+`) var ignoreFSType = map[string]bool{"procfs": true} var FSType = map[int]string{ - 0: "jfs2", 1: "namefs", 2: "nfs", 3: "jfs", 5: "cdrom", 6: "proc", - 16: "special-fs", 17: "cache-fs", 18: "nfs3", 19: "automount-fs", 20: "pool-fs", 32: "vxfs", - 33: "veritas-fs", 34: "udfs", 35: "nfs4", 36: "nfs4-pseudo", 37: "smbfs", 38: "mcr-pseudofs", - 39: "ahafs", 40: "sterm-nfs", 41: "asmfs", - } + 0: "jfs2", 1: "namefs", 2: "nfs", 3: "jfs", 5: "cdrom", 6: "proc", + 16: "special-fs", 17: "cache-fs", 18: "nfs3", 19: "automount-fs", 20: "pool-fs", 32: "vxfs", + 33: "veritas-fs", 34: "udfs", 35: "nfs4", 36: "nfs4-pseudo", 37: "smbfs", 38: "mcr-pseudofs", + 39: "ahafs", 40: "sterm-nfs", 41: "asmfs", +} func PartitionsWithContext(ctx context.Context, all bool) ([]PartitionStat, error) { var ret []PartitionStat @@ -42,7 +42,7 @@ func PartitionsWithContext(ctx context.Context, all bool) ([]PartitionStat, erro start := 0 finished := false for pos, ch := range lines[1] { - if ch == ' ' && ! finished { + if ch == ' ' && !finished { name := strings.TrimSpace(lines[0][start:pos]) colidx[name] = idx finished = true diff --git a/vendor/github.com/shirou/gopsutil/v3/disk/disk_darwin.go b/vendor/github.com/shirou/gopsutil/v3/disk/disk_darwin.go index 0877b761..933cb045 100644 --- a/vendor/github.com/shirou/gopsutil/v3/disk/disk_darwin.go +++ b/vendor/github.com/shirou/gopsutil/v3/disk/disk_darwin.go @@ -20,9 +20,15 @@ func PartitionsWithContext(ctx context.Context, all bool) ([]PartitionStat, erro return ret, err } fs := make([]unix.Statfs_t, count) - if _, err = unix.Getfsstat(fs, unix.MNT_WAIT); err != nil { + count, err = unix.Getfsstat(fs, unix.MNT_WAIT) + if err != nil { return ret, err } + // On 10.14, and possibly other OS versions, the actual count may + // be less than from the first call. Truncate to the returned count + // to prevent accessing uninitialized entries. + // https://github.com/shirou/gopsutil/issues/1390 + fs = fs[:count] for _, stat := range fs { opts := []string{"rw"} if stat.Flags&unix.MNT_RDONLY != 0 { diff --git a/vendor/github.com/shirou/gopsutil/v3/disk/disk_freebsd.go b/vendor/github.com/shirou/gopsutil/v3/disk/disk_freebsd.go index 753ce9ac..9b53106c 100644 --- a/vendor/github.com/shirou/gopsutil/v3/disk/disk_freebsd.go +++ b/vendor/github.com/shirou/gopsutil/v3/disk/disk_freebsd.go @@ -12,9 +12,8 @@ import ( "strconv" "strings" - "golang.org/x/sys/unix" - "github.com/shirou/gopsutil/v3/internal/common" + "golang.org/x/sys/unix" ) // PartitionsWithContext returns disk partition. diff --git a/vendor/github.com/shirou/gopsutil/v3/disk/disk_linux.go b/vendor/github.com/shirou/gopsutil/v3/disk/disk_linux.go index 3911af9c..b6a3adcf 100644 --- a/vendor/github.com/shirou/gopsutil/v3/disk/disk_linux.go +++ b/vendor/github.com/shirou/gopsutil/v3/disk/disk_linux.go @@ -341,7 +341,7 @@ func PartitionsWithContext(ctx context.Context, all bool) ([]PartitionStat, erro } if strings.HasPrefix(d.Device, "/dev/mapper/") { - devpath, err := filepath.EvalSymlinks(common.HostDev(strings.Replace(d.Device, "/dev", "", -1))) + devpath, err := filepath.EvalSymlinks(common.HostDev(strings.Replace(d.Device, "/dev", "", 1))) if err == nil { d.Device = devpath } diff --git a/vendor/github.com/shirou/gopsutil/v3/disk/disk_windows.go b/vendor/github.com/shirou/gopsutil/v3/disk/disk_windows.go index 825ce3b5..8a1a28d6 100644 --- a/vendor/github.com/shirou/gopsutil/v3/disk/disk_windows.go +++ b/vendor/github.com/shirou/gopsutil/v3/disk/disk_windows.go @@ -6,20 +6,16 @@ package disk import ( "bytes" "context" - "errors" "fmt" - "strconv" + "sync" "syscall" "unsafe" - "golang.org/x/sys/windows" - "github.com/shirou/gopsutil/v3/internal/common" + "golang.org/x/sys/windows" "golang.org/x/sys/windows/registry" ) -type Warnings = common.Warnings - var ( procGetDiskFreeSpaceExW = common.Modkernel32.NewProc("GetDiskFreeSpaceExW") procGetLogicalDriveStringsW = common.Modkernel32.NewProc("GetLogicalDriveStringsW") @@ -84,44 +80,101 @@ func UsageWithContext(ctx context.Context, path string) (*UsageStat, error) { return ret, nil } +// PartitionsWithContext returns disk partitions. +// Since GetVolumeInformation doesn't have a timeout, this method uses context to set deadline by users. func PartitionsWithContext(ctx context.Context, all bool) ([]PartitionStat, error) { warnings := Warnings{ Verbose: true, } var ret []PartitionStat + retChan := make(chan []PartitionStat) + errChan := make(chan error) lpBuffer := make([]byte, 254) - diskret, _, err := procGetLogicalDriveStringsW.Call( - uintptr(len(lpBuffer)), - uintptr(unsafe.Pointer(&lpBuffer[0]))) - if diskret == 0 { - return ret, err - } - for _, v := range lpBuffer { - if v >= 65 && v <= 90 { - i, err := getVolumeInformation(string(v), &warnings) - if err != nil && !errors.Is(err, errDeviceNotReady) && !errors.Is(err, errInvalidDriveType) { - continue - } - opts := []string{"rw"} - if i.FileSystemFlags&fileReadOnlyVolume != 0 { - opts = []string{"ro"} - } - if i.FileSystemFlags&fileFileCompression != 0 { - opts = append(opts, "compress") - } + var waitgrp sync.WaitGroup + waitgrp.Add(1) + defer waitgrp.Done() - path := string(v) + ":" - d := PartitionStat{ - Mountpoint: path, - Device: path, - Fstype: i.FileSystemName, - Opts: opts, - } - ret = append(ret, d) + f := func() { + defer func() { + waitgrp.Wait() + // fires when this func and the outside func finishes. + close(errChan) + close(retChan) + }() + + diskret, _, err := procGetLogicalDriveStringsW.Call( + uintptr(len(lpBuffer)), + uintptr(unsafe.Pointer(&lpBuffer[0]))) + if diskret == 0 { + errChan <- err + return } + for _, v := range lpBuffer { + if v >= 65 && v <= 90 { + path := string(v) + ":" + typepath, _ := windows.UTF16PtrFromString(path) + typeret, _, _ := procGetDriveType.Call(uintptr(unsafe.Pointer(typepath))) + if typeret == 0 { + err := windows.GetLastError() + warnings.Add(err) + continue + } + // 2: DRIVE_REMOVABLE 3: DRIVE_FIXED 4: DRIVE_REMOTE 5: DRIVE_CDROM + + if typeret == 2 || typeret == 3 || typeret == 4 || typeret == 5 { + lpVolumeNameBuffer := make([]byte, 256) + lpVolumeSerialNumber := int64(0) + lpMaximumComponentLength := int64(0) + lpFileSystemFlags := int64(0) + lpFileSystemNameBuffer := make([]byte, 256) + volpath, _ := windows.UTF16PtrFromString(string(v) + ":/") + driveret, _, err := procGetVolumeInformation.Call( + uintptr(unsafe.Pointer(volpath)), + uintptr(unsafe.Pointer(&lpVolumeNameBuffer[0])), + uintptr(len(lpVolumeNameBuffer)), + uintptr(unsafe.Pointer(&lpVolumeSerialNumber)), + uintptr(unsafe.Pointer(&lpMaximumComponentLength)), + uintptr(unsafe.Pointer(&lpFileSystemFlags)), + uintptr(unsafe.Pointer(&lpFileSystemNameBuffer[0])), + uintptr(len(lpFileSystemNameBuffer))) + if driveret == 0 { + if typeret == 5 || typeret == 2 { + continue // device is not ready will happen if there is no disk in the drive + } + warnings.Add(err) + continue + } + opts := []string{"rw"} + if lpFileSystemFlags&fileReadOnlyVolume != 0 { + opts = []string{"ro"} + } + if lpFileSystemFlags&fileFileCompression != 0 { + opts = append(opts, "compress") + } + + d := PartitionStat{ + Mountpoint: path, + Device: path, + Fstype: string(bytes.Replace(lpFileSystemNameBuffer, []byte("\x00"), []byte(""), -1)), + Opts: opts, + } + ret = append(ret, d) + } + } + } + retChan <- ret + } + + go f() + select { + case err := <-errChan: + return ret, err + case ret := <-retChan: + return ret, warnings.Reference() + case <-ctx.Done(): + return ret, ctx.Err() } - return ret, warnings.Reference() } func IOCountersWithContext(ctx context.Context, names ...string) (map[string]IOCountersStat, error) { @@ -136,8 +189,7 @@ func IOCountersWithContext(ctx context.Context, names ...string) (map[string]IOC } for _, v := range lpBuffer[:lpBufferLen] { if 'A' <= v && v <= 'Z' { - vStr := string(rune(v)) - path := vStr + ":" + path := string(rune(v)) + ":" typepath, _ := windows.UTF16PtrFromString(path) typeret := windows.GetDriveType(typepath) if typeret == 0 { @@ -162,22 +214,14 @@ func IOCountersWithContext(ctx context.Context, names ...string) (map[string]IOC if err != nil { return drivemap, err } - - i, err := getVolumeInformation(vStr, nil) - if err != nil { - return nil, err - } - drivemap[path] = IOCountersStat{ - ReadBytes: uint64(diskPerformance.BytesRead), - WriteBytes: uint64(diskPerformance.BytesWritten), - ReadCount: uint64(diskPerformance.ReadCount), - WriteCount: uint64(diskPerformance.WriteCount), - ReadTime: uint64(diskPerformance.ReadTime / 10000 / 1000), // convert to ms: https://github.com/giampaolo/psutil/issues/1012 - WriteTime: uint64(diskPerformance.WriteTime / 10000 / 1000), - Name: path, - SerialNumber: strconv.FormatInt(i.SerialNumber, 10), - Label: i.Label, + ReadBytes: uint64(diskPerformance.BytesRead), + WriteBytes: uint64(diskPerformance.BytesWritten), + ReadCount: uint64(diskPerformance.ReadCount), + WriteCount: uint64(diskPerformance.WriteCount), + ReadTime: uint64(diskPerformance.ReadTime / 10000 / 1000), // convert to ms: https://github.com/giampaolo/psutil/issues/1012 + WriteTime: uint64(diskPerformance.WriteTime / 10000 / 1000), + Name: path, } } } @@ -185,83 +229,9 @@ func IOCountersWithContext(ctx context.Context, names ...string) (map[string]IOC } func SerialNumberWithContext(ctx context.Context, name string) (string, error) { - i, err := getVolumeInformation(name, nil) - if err != nil { - return "", err - } - - return strconv.FormatInt(i.SerialNumber, 10), nil + return "", common.ErrNotImplementedError } func LabelWithContext(ctx context.Context, name string) (string, error) { - i, err := getVolumeInformation(name, nil) - if err != nil { - return "", err - } - - return i.Label, nil -} - -type volumeInformation struct { - Label string - SerialNumber int64 - MaximumComponentLength int64 - FileSystemFlags int64 - FileSystemName string -} - -var ( - errDeviceNotReady = errors.New("device not ready") - errInvalidDriveType = errors.New("invalid drive type specified") -) - -// getVolumeInformation returns all the information gathered from GetVolumeInformationW -func getVolumeInformation(name string, warnings *common.Warnings) (*volumeInformation, error) { - path := name + ":" - typepath, _ := windows.UTF16PtrFromString(path) - typeret, _, _ := procGetDriveType.Call(uintptr(unsafe.Pointer(typepath))) - if typeret == 0 { - err := windows.GetLastError() - if warnings != nil { - warnings.Add(err) - } - return nil, err - } - - if typeret == windows.DRIVE_REMOVABLE || typeret == windows.DRIVE_FIXED || typeret == windows.DRIVE_REMOTE || typeret == windows.DRIVE_CDROM { - lpVolumeNameBuffer := make([]byte, 256) - lpVolumeSerialNumber := int64(0) - lpMaximumComponentLength := int64(0) - lpFileSystemFlags := int64(0) - lpFileSystemNameBuffer := make([]byte, 256) - volpath, _ := windows.UTF16PtrFromString(name + ":/") - driveret, _, err := procGetVolumeInformation.Call( - uintptr(unsafe.Pointer(volpath)), - uintptr(unsafe.Pointer(&lpVolumeNameBuffer[0])), - uintptr(len(lpVolumeNameBuffer)), - uintptr(unsafe.Pointer(&lpVolumeSerialNumber)), - uintptr(unsafe.Pointer(&lpMaximumComponentLength)), - uintptr(unsafe.Pointer(&lpFileSystemFlags)), - uintptr(unsafe.Pointer(&lpFileSystemNameBuffer[0])), - uintptr(len(lpFileSystemNameBuffer))) - if driveret == 0 { - if typeret == 5 || typeret == 2 { - return nil, errDeviceNotReady // device is not ready will happen if there is no disk in the drive - } - if warnings != nil { - warnings.Add(err) - } - return nil, err - } - - return &volumeInformation{ - Label: string(bytes.Replace(lpVolumeNameBuffer, []byte("\x00"), []byte(""), -1)), - SerialNumber: lpVolumeSerialNumber, - MaximumComponentLength: lpMaximumComponentLength, - FileSystemFlags: lpFileSystemFlags, - FileSystemName: string(bytes.Replace(lpFileSystemNameBuffer, []byte("\x00"), []byte(""), -1)), - }, nil - } - - return nil, errInvalidDriveType + return "", common.ErrNotImplementedError } diff --git a/vendor/github.com/shirou/gopsutil/v3/internal/common/binary.go b/vendor/github.com/shirou/gopsutil/v3/internal/common/binary.go index 446c3597..5e8d43db 100644 --- a/vendor/github.com/shirou/gopsutil/v3/internal/common/binary.go +++ b/vendor/github.com/shirou/gopsutil/v3/internal/common/binary.go @@ -21,6 +21,7 @@ package common // high-performance serialization, especially for large data structures, // should look at more advanced solutions such as the encoding/gob // package or protocol buffers. + import ( "errors" "io" diff --git a/vendor/github.com/shirou/gopsutil/v3/internal/common/common.go b/vendor/github.com/shirou/gopsutil/v3/internal/common/common.go index db93b671..c1e96ca7 100644 --- a/vendor/github.com/shirou/gopsutil/v3/internal/common/common.go +++ b/vendor/github.com/shirou/gopsutil/v3/internal/common/common.go @@ -6,6 +6,7 @@ package common // - linux (amd64, arm) // - freebsd (amd64) // - windows (amd64) + import ( "bufio" "bytes" @@ -364,14 +365,8 @@ func HostDev(combineWith ...string) string { return GetEnv("HOST_DEV", "/dev", combineWith...) } -// MockEnv set environment variable and return revert function. -// MockEnv should be used testing only. -func MockEnv(key string, value string) func() { - original := os.Getenv(key) - os.Setenv(key, value) - return func() { - os.Setenv(key, original) - } +func HostRoot(combineWith ...string) string { + return GetEnv("HOST_ROOT", "/", combineWith...) } // getSysctrlEnv sets LC_ALL=C in a list of env vars for use when running diff --git a/vendor/github.com/shirou/gopsutil/v3/internal/common/common_linux.go b/vendor/github.com/shirou/gopsutil/v3/internal/common/common_linux.go index 3b4de8e5..fa6373b5 100644 --- a/vendor/github.com/shirou/gopsutil/v3/internal/common/common_linux.go +++ b/vendor/github.com/shirou/gopsutil/v3/internal/common/common_linux.go @@ -259,6 +259,11 @@ func VirtualizationWithContext(ctx context.Context) (string, string, error) { } } + if PathExists(HostRoot(".dockerenv")) { + system = "docker" + role = "guest" + } + // before returning for the first time, cache the system and role cachedVirtOnce.Do(func() { cachedVirtMutex.Lock() diff --git a/vendor/github.com/shirou/gopsutil/v3/internal/common/common_windows.go b/vendor/github.com/shirou/gopsutil/v3/internal/common/common_windows.go index 295b70bf..301b2315 100644 --- a/vendor/github.com/shirou/gopsutil/v3/internal/common/common_windows.go +++ b/vendor/github.com/shirou/gopsutil/v3/internal/common/common_windows.go @@ -218,9 +218,12 @@ func WMIQueryWithContext(ctx context.Context, query string, dst interface{}, con } // Convert paths using native DOS format like: -// "\Device\HarddiskVolume1\Windows\systemew\file.txt" +// +// "\Device\HarddiskVolume1\Windows\systemew\file.txt" +// // into: -// "C:\Windows\systemew\file.txt" +// +// "C:\Windows\systemew\file.txt" func ConvertDOSPath(p string) string { rawDrive := strings.Join(strings.Split(p, `\`)[:3], `\`) diff --git a/vendor/github.com/shirou/gopsutil/v3/mem/mem_aix_nocgo.go b/vendor/github.com/shirou/gopsutil/v3/mem/mem_aix_nocgo.go index 09ffd8ed..fc9e4922 100644 --- a/vendor/github.com/shirou/gopsutil/v3/mem/mem_aix_nocgo.go +++ b/vendor/github.com/shirou/gopsutil/v3/mem/mem_aix_nocgo.go @@ -71,7 +71,7 @@ func callSVMon(ctx context.Context) (*VirtualMemoryStat, *SwapMemoryStat, error) swap.Total = t * pagesize } if t, err := strconv.ParseUint(p[3], 10, 64); err == nil { - swap.Free = swap.Total - t * pagesize + swap.Free = swap.Total - t*pagesize } } break diff --git a/vendor/github.com/shirou/gopsutil/v3/mem/mem_freebsd.go b/vendor/github.com/shirou/gopsutil/v3/mem/mem_freebsd.go index 44543ef7..9a56785b 100644 --- a/vendor/github.com/shirou/gopsutil/v3/mem/mem_freebsd.go +++ b/vendor/github.com/shirou/gopsutil/v3/mem/mem_freebsd.go @@ -8,9 +8,8 @@ import ( "errors" "unsafe" - "golang.org/x/sys/unix" - "github.com/shirou/gopsutil/v3/internal/common" + "golang.org/x/sys/unix" ) func VirtualMemory() (*VirtualMemoryStat, error) { diff --git a/vendor/github.com/shirou/gopsutil/v3/mem/mem_openbsd.go b/vendor/github.com/shirou/gopsutil/v3/mem/mem_openbsd.go index 97644923..e37d5abe 100644 --- a/vendor/github.com/shirou/gopsutil/v3/mem/mem_openbsd.go +++ b/vendor/github.com/shirou/gopsutil/v3/mem/mem_openbsd.go @@ -9,6 +9,7 @@ import ( "encoding/binary" "errors" "fmt" + "github.com/shirou/gopsutil/v3/internal/common" "golang.org/x/sys/unix" ) diff --git a/vendor/github.com/shirou/gopsutil/v3/net/net_aix_cgo.go b/vendor/github.com/shirou/gopsutil/v3/net/net_aix_cgo.go index 8cf8c914..8c34f881 100644 --- a/vendor/github.com/shirou/gopsutil/v3/net/net_aix_cgo.go +++ b/vendor/github.com/shirou/gopsutil/v3/net/net_aix_cgo.go @@ -18,14 +18,14 @@ func IOCountersWithContext(ctx context.Context, pernic bool) ([]IOCountersStat, iocounters := make([]IOCountersStat, 0, len(ifs)) for _, netif := range ifs { n := IOCountersStat{ - Name: netif.Name, - BytesSent: uint64(netif.OBytes), - BytesRecv: uint64(netif.IBytes), + Name: netif.Name, + BytesSent: uint64(netif.OBytes), + BytesRecv: uint64(netif.IBytes), PacketsSent: uint64(netif.OPackets), PacketsRecv: uint64(netif.IPackets), - Errin: uint64(netif.OErrors), - Errout: uint64(netif.IErrors), - Dropout: uint64(netif.XmitDrops), + Errin: uint64(netif.OErrors), + Errout: uint64(netif.IErrors), + Dropout: uint64(netif.XmitDrops), } iocounters = append(iocounters, n) } diff --git a/vendor/github.com/shirou/gopsutil/v3/net/net_windows.go b/vendor/github.com/shirou/gopsutil/v3/net/net_windows.go index 731c8f97..68b26bdc 100644 --- a/vendor/github.com/shirou/gopsutil/v3/net/net_windows.go +++ b/vendor/github.com/shirou/gopsutil/v3/net/net_windows.go @@ -211,7 +211,8 @@ func IOCountersByFileWithContext(ctx context.Context, pernic bool, filename stri // Return a list of network connections // Available kind: -// reference to netConnectionKindMap +// +// reference to netConnectionKindMap func Connections(kind string) ([]ConnectionStat, error) { return ConnectionsWithContext(context.Background(), kind) } diff --git a/vendor/github.com/shirou/gopsutil/v3/process/process_freebsd.go b/vendor/github.com/shirou/gopsutil/v3/process/process_freebsd.go index 779f8126..a123ccf9 100644 --- a/vendor/github.com/shirou/gopsutil/v3/process/process_freebsd.go +++ b/vendor/github.com/shirou/gopsutil/v3/process/process_freebsd.go @@ -69,7 +69,13 @@ func (p *Process) CwdWithContext(ctx context.Context) (string, error) { } func (p *Process) ExeWithContext(ctx context.Context) (string, error) { - return "", common.ErrNotImplementedError + mib := []int32{CTLKern, KernProc, KernProcPathname, p.Pid} + buf, _, err := common.CallSyscall(mib) + if err != nil { + return "", err + } + + return strings.Trim(string(buf), "\x00"), nil } func (p *Process) CmdlineWithContext(ctx context.Context) (string, error) { diff --git a/vendor/github.com/shirou/gopsutil/v3/process/process_windows_32bit.go b/vendor/github.com/shirou/gopsutil/v3/process/process_windows_32bit.go index 982287d9..db4d4533 100644 --- a/vendor/github.com/shirou/gopsutil/v3/process/process_windows_32bit.go +++ b/vendor/github.com/shirou/gopsutil/v3/process/process_windows_32bit.go @@ -8,9 +8,8 @@ import ( "syscall" "unsafe" - "golang.org/x/sys/windows" - "github.com/shirou/gopsutil/v3/internal/common" + "golang.org/x/sys/windows" ) type PROCESS_MEMORY_COUNTERS struct { diff --git a/vendor/github.com/shoenig/go-m1cpu/.golangci.yaml b/vendor/github.com/shoenig/go-m1cpu/.golangci.yaml new file mode 100644 index 00000000..dc6fefb9 --- /dev/null +++ b/vendor/github.com/shoenig/go-m1cpu/.golangci.yaml @@ -0,0 +1,12 @@ +run: + timeout: 5m +linters: + enable: + - gofmt + - errcheck + - errname + - errorlint + - bodyclose + - durationcheck + - whitespace + diff --git a/vendor/github.com/shoenig/go-m1cpu/LICENSE b/vendor/github.com/shoenig/go-m1cpu/LICENSE new file mode 100644 index 00000000..e87a115e --- /dev/null +++ b/vendor/github.com/shoenig/go-m1cpu/LICENSE @@ -0,0 +1,363 @@ +Mozilla Public License, version 2.0 + +1. Definitions + +1.1. "Contributor" + + means each individual or legal entity that creates, contributes to the + creation of, or owns Covered Software. + +1.2. "Contributor Version" + + means the combination of the Contributions of others (if any) used by a + Contributor and that particular Contributor's Contribution. + +1.3. "Contribution" + + means Covered Software of a particular Contributor. + +1.4. "Covered Software" + + means Source Code Form to which the initial Contributor has attached the + notice in Exhibit A, the Executable Form of such Source Code Form, and + Modifications of such Source Code Form, in each case including portions + thereof. + +1.5. "Incompatible With Secondary Licenses" + means + + a. that the initial Contributor has attached the notice described in + Exhibit B to the Covered Software; or + + b. that the Covered Software was made available under the terms of + version 1.1 or earlier of the License, but not also under the terms of + a Secondary License. + +1.6. "Executable Form" + + means any form of the work other than Source Code Form. + +1.7. "Larger Work" + + means a work that combines Covered Software with other material, in a + separate file or files, that is not Covered Software. + +1.8. "License" + + means this document. + +1.9. "Licensable" + + means having the right to grant, to the maximum extent possible, whether + at the time of the initial grant or subsequently, any and all of the + rights conveyed by this License. + +1.10. "Modifications" + + means any of the following: + + a. any file in Source Code Form that results from an addition to, + deletion from, or modification of the contents of Covered Software; or + + b. any new file in Source Code Form that contains any Covered Software. + +1.11. "Patent Claims" of a Contributor + + means any patent claim(s), including without limitation, method, + process, and apparatus claims, in any patent Licensable by such + Contributor that would be infringed, but for the grant of the License, + by the making, using, selling, offering for sale, having made, import, + or transfer of either its Contributions or its Contributor Version. + +1.12. "Secondary License" + + means either the GNU General Public License, Version 2.0, the GNU Lesser + General Public License, Version 2.1, the GNU Affero General Public + License, Version 3.0, or any later versions of those licenses. + +1.13. "Source Code Form" + + means the form of the work preferred for making modifications. + +1.14. "You" (or "Your") + + means an individual or a legal entity exercising rights under this + License. For legal entities, "You" includes any entity that controls, is + controlled by, or is under common control with You. For purposes of this + definition, "control" means (a) the power, direct or indirect, to cause + the direction or management of such entity, whether by contract or + otherwise, or (b) ownership of more than fifty percent (50%) of the + outstanding shares or beneficial ownership of such entity. + + +2. License Grants and Conditions + +2.1. Grants + + Each Contributor hereby grants You a world-wide, royalty-free, + non-exclusive license: + + a. under intellectual property rights (other than patent or trademark) + Licensable by such Contributor to use, reproduce, make available, + modify, display, perform, distribute, and otherwise exploit its + Contributions, either on an unmodified basis, with Modifications, or + as part of a Larger Work; and + + b. under Patent Claims of such Contributor to make, use, sell, offer for + sale, have made, import, and otherwise transfer either its + Contributions or its Contributor Version. + +2.2. Effective Date + + The licenses granted in Section 2.1 with respect to any Contribution + become effective for each Contribution on the date the Contributor first + distributes such Contribution. + +2.3. Limitations on Grant Scope + + The licenses granted in this Section 2 are the only rights granted under + this License. No additional rights or licenses will be implied from the + distribution or licensing of Covered Software under this License. + Notwithstanding Section 2.1(b) above, no patent license is granted by a + Contributor: + + a. for any code that a Contributor has removed from Covered Software; or + + b. for infringements caused by: (i) Your and any other third party's + modifications of Covered Software, or (ii) the combination of its + Contributions with other software (except as part of its Contributor + Version); or + + c. under Patent Claims infringed by Covered Software in the absence of + its Contributions. + + This License does not grant any rights in the trademarks, service marks, + or logos of any Contributor (except as may be necessary to comply with + the notice requirements in Section 3.4). + +2.4. Subsequent Licenses + + No Contributor makes additional grants as a result of Your choice to + distribute the Covered Software under a subsequent version of this + License (see Section 10.2) or under the terms of a Secondary License (if + permitted under the terms of Section 3.3). + +2.5. Representation + + Each Contributor represents that the Contributor believes its + Contributions are its original creation(s) or it has sufficient rights to + grant the rights to its Contributions conveyed by this License. + +2.6. Fair Use + + This License is not intended to limit any rights You have under + applicable copyright doctrines of fair use, fair dealing, or other + equivalents. + +2.7. Conditions + + Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in + Section 2.1. + + +3. Responsibilities + +3.1. Distribution of Source Form + + All distribution of Covered Software in Source Code Form, including any + Modifications that You create or to which You contribute, must be under + the terms of this License. You must inform recipients that the Source + Code Form of the Covered Software is governed by the terms of this + License, and how they can obtain a copy of this License. You may not + attempt to alter or restrict the recipients' rights in the Source Code + Form. + +3.2. Distribution of Executable Form + + If You distribute Covered Software in Executable Form then: + + a. such Covered Software must also be made available in Source Code Form, + as described in Section 3.1, and You must inform recipients of the + Executable Form how they can obtain a copy of such Source Code Form by + reasonable means in a timely manner, at a charge no more than the cost + of distribution to the recipient; and + + b. You may distribute such Executable Form under the terms of this + License, or sublicense it under different terms, provided that the + license for the Executable Form does not attempt to limit or alter the + recipients' rights in the Source Code Form under this License. + +3.3. Distribution of a Larger Work + + You may create and distribute a Larger Work under terms of Your choice, + provided that You also comply with the requirements of this License for + the Covered Software. If the Larger Work is a combination of Covered + Software with a work governed by one or more Secondary Licenses, and the + Covered Software is not Incompatible With Secondary Licenses, this + License permits You to additionally distribute such Covered Software + under the terms of such Secondary License(s), so that the recipient of + the Larger Work may, at their option, further distribute the Covered + Software under the terms of either this License or such Secondary + License(s). + +3.4. Notices + + You may not remove or alter the substance of any license notices + (including copyright notices, patent notices, disclaimers of warranty, or + limitations of liability) contained within the Source Code Form of the + Covered Software, except that You may alter any license notices to the + extent required to remedy known factual inaccuracies. + +3.5. Application of Additional Terms + + You may choose to offer, and to charge a fee for, warranty, support, + indemnity or liability obligations to one or more recipients of Covered + Software. However, You may do so only on Your own behalf, and not on + behalf of any Contributor. You must make it absolutely clear that any + such warranty, support, indemnity, or liability obligation is offered by + You alone, and You hereby agree to indemnify every Contributor for any + liability incurred by such Contributor as a result of warranty, support, + indemnity or liability terms You offer. You may include additional + disclaimers of warranty and limitations of liability specific to any + jurisdiction. + +4. Inability to Comply Due to Statute or Regulation + + If it is impossible for You to comply with any of the terms of this License + with respect to some or all of the Covered Software due to statute, + judicial order, or regulation then You must: (a) comply with the terms of + this License to the maximum extent possible; and (b) describe the + limitations and the code they affect. Such description must be placed in a + text file included with all distributions of the Covered Software under + this License. Except to the extent prohibited by statute or regulation, + such description must be sufficiently detailed for a recipient of ordinary + skill to be able to understand it. + +5. Termination + +5.1. The rights granted under this License will terminate automatically if You + fail to comply with any of its terms. However, if You become compliant, + then the rights granted under this License from a particular Contributor + are reinstated (a) provisionally, unless and until such Contributor + explicitly and finally terminates Your grants, and (b) on an ongoing + basis, if such Contributor fails to notify You of the non-compliance by + some reasonable means prior to 60 days after You have come back into + compliance. Moreover, Your grants from a particular Contributor are + reinstated on an ongoing basis if such Contributor notifies You of the + non-compliance by some reasonable means, this is the first time You have + received notice of non-compliance with this License from such + Contributor, and You become compliant prior to 30 days after Your receipt + of the notice. + +5.2. If You initiate litigation against any entity by asserting a patent + infringement claim (excluding declaratory judgment actions, + counter-claims, and cross-claims) alleging that a Contributor Version + directly or indirectly infringes any patent, then the rights granted to + You by any and all Contributors for the Covered Software under Section + 2.1 of this License shall terminate. + +5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user + license agreements (excluding distributors and resellers) which have been + validly granted by You or Your distributors under this License prior to + termination shall survive termination. + +6. Disclaimer of Warranty + + Covered Software is provided under this License on an "as is" basis, + without warranty of any kind, either expressed, implied, or statutory, + including, without limitation, warranties that the Covered Software is free + of defects, merchantable, fit for a particular purpose or non-infringing. + The entire risk as to the quality and performance of the Covered Software + is with You. Should any Covered Software prove defective in any respect, + You (not any Contributor) assume the cost of any necessary servicing, + repair, or correction. This disclaimer of warranty constitutes an essential + part of this License. No use of any Covered Software is authorized under + this License except under this disclaimer. + +7. Limitation of Liability + + Under no circumstances and under no legal theory, whether tort (including + negligence), contract, or otherwise, shall any Contributor, or anyone who + distributes Covered Software as permitted above, be liable to You for any + direct, indirect, special, incidental, or consequential damages of any + character including, without limitation, damages for lost profits, loss of + goodwill, work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses, even if such party shall have been + informed of the possibility of such damages. This limitation of liability + shall not apply to liability for death or personal injury resulting from + such party's negligence to the extent applicable law prohibits such + limitation. Some jurisdictions do not allow the exclusion or limitation of + incidental or consequential damages, so this exclusion and limitation may + not apply to You. + +8. Litigation + + Any litigation relating to this License may be brought only in the courts + of a jurisdiction where the defendant maintains its principal place of + business and such litigation shall be governed by laws of that + jurisdiction, without reference to its conflict-of-law provisions. Nothing + in this Section shall prevent a party's ability to bring cross-claims or + counter-claims. + +9. Miscellaneous + + This License represents the complete agreement concerning the subject + matter hereof. If any provision of this License is held to be + unenforceable, such provision shall be reformed only to the extent + necessary to make it enforceable. Any law or regulation which provides that + the language of a contract shall be construed against the drafter shall not + be used to construe this License against a Contributor. + + +10. Versions of the License + +10.1. New Versions + + Mozilla Foundation is the license steward. Except as provided in Section + 10.3, no one other than the license steward has the right to modify or + publish new versions of this License. Each version will be given a + distinguishing version number. + +10.2. Effect of New Versions + + You may distribute the Covered Software under the terms of the version + of the License under which You originally received the Covered Software, + or under the terms of any subsequent version published by the license + steward. + +10.3. Modified Versions + + If you create software not governed by this License, and you want to + create a new license for such software, you may create and use a + modified version of this License if you rename the license and remove + any references to the name of the license steward (except to note that + such modified license differs from this License). + +10.4. Distributing Source Code Form that is Incompatible With Secondary + Licenses If You choose to distribute Source Code Form that is + Incompatible With Secondary Licenses under the terms of this version of + the License, the notice described in Exhibit B of this License must be + attached. + +Exhibit A - Source Code Form License Notice + + This Source Code Form is subject to the + terms of the Mozilla Public License, v. + 2.0. If a copy of the MPL was not + distributed with this file, You can + obtain one at + http://mozilla.org/MPL/2.0/. + +If it is not possible or desirable to put the notice in a particular file, +then You may include the notice in a location (such as a LICENSE file in a +relevant directory) where a recipient would be likely to look for such a +notice. + +You may add additional accurate notices of copyright ownership. + +Exhibit B - "Incompatible With Secondary Licenses" Notice + + This Source Code Form is "Incompatible + With Secondary Licenses", as defined by + the Mozilla Public License, v. 2.0. + diff --git a/vendor/github.com/shoenig/go-m1cpu/Makefile b/vendor/github.com/shoenig/go-m1cpu/Makefile new file mode 100644 index 00000000..28d78639 --- /dev/null +++ b/vendor/github.com/shoenig/go-m1cpu/Makefile @@ -0,0 +1,12 @@ +SHELL = bash + +default: test + +.PHONY: test +test: + @echo "--> Running Tests ..." + @go test -v -race ./... + +vet: + @echo "--> Vet Go sources ..." + @go vet ./... diff --git a/vendor/github.com/shoenig/go-m1cpu/README.md b/vendor/github.com/shoenig/go-m1cpu/README.md new file mode 100644 index 00000000..399657ac --- /dev/null +++ b/vendor/github.com/shoenig/go-m1cpu/README.md @@ -0,0 +1,66 @@ +# m1cpu + +[![Go Reference](https://pkg.go.dev/badge/github.com/shoenig/go-m1cpu.svg)](https://pkg.go.dev/github.com/shoenig/go-m1cpu) +[![MPL License](https://img.shields.io/github/license/shoenig/go-m1cpu?color=g&style=flat-square)](https://github.com/shoenig/go-m1cpu/blob/main/LICENSE) +[![Run CI Tests](https://github.com/shoenig/go-m1cpu/actions/workflows/ci.yaml/badge.svg)](https://github.com/shoenig/go-m1cpu/actions/workflows/ci.yaml) + +The `go-m1cpu` module is a library for inspecting Apple Silicon CPUs in Go. + +Use the `m1cpu` Go package for looking up the CPU frequency for Apple M1 and M2 CPUs. + +# Install + +```shell +go get github.com/shoenig/go-m1cpu@latest +``` + +# CGO + +This package requires the use of [CGO](https://go.dev/blog/cgo). + +Extracting the CPU properties is done via Apple's [IOKit](https://developer.apple.com/documentation/iokit?language=objc) +framework, which is accessible only through system C libraries. + +# Example + +Simple Go program to print Apple Silicon M1/M2 CPU speeds. + +```go +package main + +import ( + "fmt" + + "github.com/shoenig/go-m1cpu" +) + +func main() { + fmt.Println("Apple Silicon", m1cpu.IsAppleSilicon()) + + fmt.Println("pCore GHz", m1cpu.PCoreGHz()) + fmt.Println("eCore GHz", m1cpu.ECoreGHz()) + + fmt.Println("pCore Hz", m1cpu.PCoreHz()) + fmt.Println("eCore Hz", m1cpu.ECoreHz()) +} +``` + +Using `go test` to print out available information. + +``` +âžś go test -v -run Show +=== RUN Test_Show + cpu_test.go:42: pCore Hz 3504000000 + cpu_test.go:43: eCore Hz 2424000000 + cpu_test.go:44: pCore GHz 3.504 + cpu_test.go:45: eCore GHz 2.424 + cpu_test.go:46: pCore count 8 + cpu_test.go:47: eCoreCount 4 + cpu_test.go:50: pCore Caches 196608 131072 16777216 + cpu_test.go:53: eCore Caches 131072 65536 4194304 +--- PASS: Test_Show (0.00s) +``` + +# License + +Open source under the [MPL](LICENSE) diff --git a/vendor/github.com/shoenig/go-m1cpu/cpu.go b/vendor/github.com/shoenig/go-m1cpu/cpu.go new file mode 100644 index 00000000..a9b8fc5e --- /dev/null +++ b/vendor/github.com/shoenig/go-m1cpu/cpu.go @@ -0,0 +1,208 @@ +//go:build darwin && arm64 && cgo + +package m1cpu + +// #cgo LDFLAGS: -framework CoreFoundation -framework IOKit +// #include +// #include +// #include +// +// #define HzToGHz(hz) ((hz) / 1000000000.0) +// +// UInt64 global_pCoreHz; +// UInt64 global_eCoreHz; +// int global_pCoreCount; +// int global_eCoreCount; +// int global_pCoreL1InstCacheSize; +// int global_eCoreL1InstCacheSize; +// int global_pCoreL1DataCacheSize; +// int global_eCoreL1DataCacheSize; +// int global_pCoreL2CacheSize; +// int global_eCoreL2CacheSize; +// char global_brand[32]; +// +// UInt64 getFrequency(CFTypeRef typeRef) { +// CFDataRef cfData = typeRef; +// +// CFIndex size = CFDataGetLength(cfData); +// UInt8 buf[size]; +// CFDataGetBytes(cfData, CFRangeMake(0, size), buf); +// +// UInt8 b1 = buf[size-5]; +// UInt8 b2 = buf[size-6]; +// UInt8 b3 = buf[size-7]; +// UInt8 b4 = buf[size-8]; +// +// UInt64 pCoreHz = 0x00000000FFFFFFFF & ((b1<<24) | (b2 << 16) | (b3 << 8) | (b4)); +// return pCoreHz; +// } +// +// int sysctl_int(const char * name) { +// int value = -1; +// size_t size = 8; +// sysctlbyname(name, &value, &size, NULL, 0); +// return value; +// } +// +// void sysctl_string(const char * name, char * dest) { +// size_t size = 32; +// sysctlbyname(name, dest, &size, NULL, 0); +// } +// +// void initialize() { +// global_pCoreCount = sysctl_int("hw.perflevel0.physicalcpu"); +// global_eCoreCount = sysctl_int("hw.perflevel1.physicalcpu"); +// global_pCoreL1InstCacheSize = sysctl_int("hw.perflevel0.l1icachesize"); +// global_eCoreL1InstCacheSize = sysctl_int("hw.perflevel1.l1icachesize"); +// global_pCoreL1DataCacheSize = sysctl_int("hw.perflevel0.l1dcachesize"); +// global_eCoreL1DataCacheSize = sysctl_int("hw.perflevel1.l1dcachesize"); +// global_pCoreL2CacheSize = sysctl_int("hw.perflevel0.l2cachesize"); +// global_eCoreL2CacheSize = sysctl_int("hw.perflevel1.l2cachesize"); +// sysctl_string("machdep.cpu.brand_string", global_brand); +// +// CFMutableDictionaryRef matching = IOServiceMatching("AppleARMIODevice"); +// io_iterator_t iter; +// IOServiceGetMatchingServices(kIOMainPortDefault, matching, &iter); +// +// const size_t bufsize = 512; +// io_object_t obj; +// while ((obj = IOIteratorNext(iter))) { +// char class[bufsize]; +// IOObjectGetClass(obj, class); +// char name[bufsize]; +// IORegistryEntryGetName(obj, name); +// +// if (strncmp(name, "pmgr", bufsize) == 0) { +// CFTypeRef pCoreRef = IORegistryEntryCreateCFProperty(obj, CFSTR("voltage-states5-sram"), kCFAllocatorDefault, 0); +// CFTypeRef eCoreRef = IORegistryEntryCreateCFProperty(obj, CFSTR("voltage-states1-sram"), kCFAllocatorDefault, 0); +// +// long long pCoreHz = getFrequency(pCoreRef); +// long long eCoreHz = getFrequency(eCoreRef); +// +// global_pCoreHz = pCoreHz; +// global_eCoreHz = eCoreHz; +// return; +// } +// } +// } +// +// UInt64 eCoreHz() { +// return global_eCoreHz; +// } +// +// UInt64 pCoreHz() { +// return global_pCoreHz; +// } +// +// Float64 eCoreGHz() { +// return HzToGHz(global_eCoreHz); +// } +// +// Float64 pCoreGHz() { +// return HzToGHz(global_pCoreHz); +// } +// +// int pCoreCount() { +// return global_pCoreCount; +// } +// +// int eCoreCount() { +// return global_eCoreCount; +// } +// +// int pCoreL1InstCacheSize() { +// return global_pCoreL1InstCacheSize; +// } +// +// int pCoreL1DataCacheSize() { +// return global_pCoreL1DataCacheSize; +// } +// +// int pCoreL2CacheSize() { +// return global_pCoreL2CacheSize; +// } +// +// int eCoreL1InstCacheSize() { +// return global_eCoreL1InstCacheSize; +// } +// +// int eCoreL1DataCacheSize() { +// return global_eCoreL1DataCacheSize; +// } +// +// int eCoreL2CacheSize() { +// return global_eCoreL2CacheSize; +// } +// +// char * modelName() { +// return global_brand; +// } +import "C" + +func init() { + C.initialize() +} + +// IsAppleSilicon returns true on this platform. +func IsAppleSilicon() bool { + return true +} + +// PCoreHZ returns the max frequency in Hertz of the P-Core of an Apple Silicon CPU. +func PCoreHz() uint64 { + return uint64(C.pCoreHz()) +} + +// ECoreHZ returns the max frequency in Hertz of the E-Core of an Apple Silicon CPU. +func ECoreHz() uint64 { + return uint64(C.eCoreHz()) +} + +// PCoreGHz returns the max frequency in Gigahertz of the P-Core of an Apple Silicon CPU. +func PCoreGHz() float64 { + return float64(C.pCoreGHz()) +} + +// ECoreGHz returns the max frequency in Gigahertz of the E-Core of an Apple Silicon CPU. +func ECoreGHz() float64 { + return float64(C.eCoreGHz()) +} + +// PCoreCount returns the number of physical P (performance) cores. +func PCoreCount() int { + return int(C.pCoreCount()) +} + +// ECoreCount returns the number of physical E (efficiency) cores. +func ECoreCount() int { + return int(C.eCoreCount()) +} + +// PCoreCacheSize returns the sizes of the P (performance) core cache sizes +// in the order of +// +// - L1 instruction cache +// - L1 data cache +// - L2 cache +func PCoreCache() (int, int, int) { + return int(C.pCoreL1InstCacheSize()), + int(C.pCoreL1DataCacheSize()), + int(C.pCoreL2CacheSize()) +} + +// ECoreCacheSize returns the sizes of the E (efficiency) core cache sizes +// in the order of +// +// - L1 instruction cache +// - L1 data cache +// - L2 cache +func ECoreCache() (int, int, int) { + return int(C.eCoreL1InstCacheSize()), + int(C.eCoreL1DataCacheSize()), + int(C.eCoreL2CacheSize()) +} + +// ModelName returns the model name of the CPU. +func ModelName() string { + return C.GoString(C.modelName()) +} diff --git a/vendor/github.com/shoenig/go-m1cpu/incompatible.go b/vendor/github.com/shoenig/go-m1cpu/incompatible.go new file mode 100644 index 00000000..d425025a --- /dev/null +++ b/vendor/github.com/shoenig/go-m1cpu/incompatible.go @@ -0,0 +1,53 @@ +//go:build !darwin || !arm64 || !cgo + +package m1cpu + +// IsAppleSilicon return false on this platform. +func IsAppleSilicon() bool { + return false +} + +// PCoreHZ requires darwin/arm64 +func PCoreHz() uint64 { + panic("m1cpu: not a darwin/arm64 system") +} + +// ECoreHZ requires darwin/arm64 +func ECoreHz() uint64 { + panic("m1cpu: not a darwin/arm64 system") +} + +// PCoreGHz requires darwin/arm64 +func PCoreGHz() float64 { + panic("m1cpu: not a darwin/arm64 system") +} + +// ECoreGHz requires darwin/arm64 +func ECoreGHz() float64 { + panic("m1cpu: not a darwin/arm64 system") +} + +// PCoreCount requires darwin/arm64 +func PCoreCount() int { + panic("m1cpu: not a darwin/arm64 system") +} + +// ECoreCount requires darwin/arm64 +func ECoreCount() int { + panic("m1cpu: not a darwin/arm64 system") +} + +// PCoreCacheSize requires darwin/arm64 +func PCoreCache() (int, int, int) { + panic("m1cpu: not a darwin/arm64 system") +} + +// ECoreCacheSize requires darwin/arm64 +func ECoreCache() (int, int, int) { + panic("m1cpu: not a darwin/arm64 system") +} + +// ModelName requires darwin/arm64 +func ModelName() string { + panic("m1cpu: not a darwin/arm64 system") +} diff --git a/vendor/github.com/stretchr/testify/assert/assertions.go b/vendor/github.com/stretchr/testify/assert/assertions.go index fa1245b1..2924cf3a 100644 --- a/vendor/github.com/stretchr/testify/assert/assertions.go +++ b/vendor/github.com/stretchr/testify/assert/assertions.go @@ -8,7 +8,6 @@ import ( "fmt" "math" "os" - "path/filepath" "reflect" "regexp" "runtime" @@ -141,12 +140,11 @@ func CallerInfo() []string { } parts := strings.Split(file, "/") - file = parts[len(parts)-1] if len(parts) > 1 { + filename := parts[len(parts)-1] dir := parts[len(parts)-2] - if (dir != "assert" && dir != "mock" && dir != "require") || file == "mock_test.go" { - path, _ := filepath.Abs(file) - callers = append(callers, fmt.Sprintf("%s:%d", path, line)) + if (dir != "assert" && dir != "mock" && dir != "require") || filename == "mock_test.go" { + callers = append(callers, fmt.Sprintf("%s:%d", file, line)) } } @@ -530,7 +528,7 @@ func isNil(object interface{}) bool { []reflect.Kind{ reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, - reflect.Ptr, reflect.Slice}, + reflect.Ptr, reflect.Slice, reflect.UnsafePointer}, kind) if isNilableKind && value.IsNil() { @@ -818,49 +816,44 @@ func Subset(t TestingT, list, subset interface{}, msgAndArgs ...interface{}) (ok return true // we consider nil to be equal to the nil set } - defer func() { - if e := recover(); e != nil { - ok = false - } - }() - listKind := reflect.TypeOf(list).Kind() - subsetKind := reflect.TypeOf(subset).Kind() - if listKind != reflect.Array && listKind != reflect.Slice && listKind != reflect.Map { return Fail(t, fmt.Sprintf("%q has an unsupported type %s", list, listKind), msgAndArgs...) } + subsetKind := reflect.TypeOf(subset).Kind() if subsetKind != reflect.Array && subsetKind != reflect.Slice && listKind != reflect.Map { return Fail(t, fmt.Sprintf("%q has an unsupported type %s", subset, subsetKind), msgAndArgs...) } - subsetValue := reflect.ValueOf(subset) if subsetKind == reflect.Map && listKind == reflect.Map { - listValue := reflect.ValueOf(list) - subsetKeys := subsetValue.MapKeys() + subsetMap := reflect.ValueOf(subset) + actualMap := reflect.ValueOf(list) - for i := 0; i < len(subsetKeys); i++ { - subsetKey := subsetKeys[i] - subsetElement := subsetValue.MapIndex(subsetKey).Interface() - listElement := listValue.MapIndex(subsetKey).Interface() + for _, k := range subsetMap.MapKeys() { + ev := subsetMap.MapIndex(k) + av := actualMap.MapIndex(k) - if !ObjectsAreEqual(subsetElement, listElement) { - return Fail(t, fmt.Sprintf("\"%s\" does not contain \"%s\"", list, subsetElement), msgAndArgs...) + if !av.IsValid() { + return Fail(t, fmt.Sprintf("%#v does not contain %#v", list, subset), msgAndArgs...) + } + if !ObjectsAreEqual(ev.Interface(), av.Interface()) { + return Fail(t, fmt.Sprintf("%#v does not contain %#v", list, subset), msgAndArgs...) } } return true } - for i := 0; i < subsetValue.Len(); i++ { - element := subsetValue.Index(i).Interface() + subsetList := reflect.ValueOf(subset) + for i := 0; i < subsetList.Len(); i++ { + element := subsetList.Index(i).Interface() ok, found := containsElement(list, element) if !ok { - return Fail(t, fmt.Sprintf("\"%s\" could not be applied builtin len()", list), msgAndArgs...) + return Fail(t, fmt.Sprintf("%#v could not be applied builtin len()", list), msgAndArgs...) } if !found { - return Fail(t, fmt.Sprintf("\"%s\" does not contain \"%s\"", list, element), msgAndArgs...) + return Fail(t, fmt.Sprintf("%#v does not contain %#v", list, element), msgAndArgs...) } } @@ -879,34 +872,28 @@ func NotSubset(t TestingT, list, subset interface{}, msgAndArgs ...interface{}) return Fail(t, "nil is the empty set which is a subset of every set", msgAndArgs...) } - defer func() { - if e := recover(); e != nil { - ok = false - } - }() - listKind := reflect.TypeOf(list).Kind() - subsetKind := reflect.TypeOf(subset).Kind() - if listKind != reflect.Array && listKind != reflect.Slice && listKind != reflect.Map { return Fail(t, fmt.Sprintf("%q has an unsupported type %s", list, listKind), msgAndArgs...) } + subsetKind := reflect.TypeOf(subset).Kind() if subsetKind != reflect.Array && subsetKind != reflect.Slice && listKind != reflect.Map { return Fail(t, fmt.Sprintf("%q has an unsupported type %s", subset, subsetKind), msgAndArgs...) } - subsetValue := reflect.ValueOf(subset) if subsetKind == reflect.Map && listKind == reflect.Map { - listValue := reflect.ValueOf(list) - subsetKeys := subsetValue.MapKeys() + subsetMap := reflect.ValueOf(subset) + actualMap := reflect.ValueOf(list) - for i := 0; i < len(subsetKeys); i++ { - subsetKey := subsetKeys[i] - subsetElement := subsetValue.MapIndex(subsetKey).Interface() - listElement := listValue.MapIndex(subsetKey).Interface() + for _, k := range subsetMap.MapKeys() { + ev := subsetMap.MapIndex(k) + av := actualMap.MapIndex(k) - if !ObjectsAreEqual(subsetElement, listElement) { + if !av.IsValid() { + return true + } + if !ObjectsAreEqual(ev.Interface(), av.Interface()) { return true } } @@ -914,8 +901,9 @@ func NotSubset(t TestingT, list, subset interface{}, msgAndArgs ...interface{}) return Fail(t, fmt.Sprintf("%q is a subset of %q", subset, list), msgAndArgs...) } - for i := 0; i < subsetValue.Len(); i++ { - element := subsetValue.Index(i).Interface() + subsetList := reflect.ValueOf(subset) + for i := 0; i < subsetList.Len(); i++ { + element := subsetList.Index(i).Interface() ok, found := containsElement(list, element) if !ok { return Fail(t, fmt.Sprintf("\"%s\" could not be applied builtin len()", list), msgAndArgs...) diff --git a/vendor/golang.org/x/sys/cpu/hwcap_linux.go b/vendor/golang.org/x/sys/cpu/hwcap_linux.go index f3baa379..1d9d91f3 100644 --- a/vendor/golang.org/x/sys/cpu/hwcap_linux.go +++ b/vendor/golang.org/x/sys/cpu/hwcap_linux.go @@ -24,6 +24,21 @@ var hwCap uint var hwCap2 uint func readHWCAP() error { + // For Go 1.21+, get auxv from the Go runtime. + if a := getAuxv(); len(a) > 0 { + for len(a) >= 2 { + tag, val := a[0], uint(a[1]) + a = a[2:] + switch tag { + case _AT_HWCAP: + hwCap = val + case _AT_HWCAP2: + hwCap2 = val + } + } + return nil + } + buf, err := ioutil.ReadFile(procAuxv) if err != nil { // e.g. on android /proc/self/auxv is not accessible, so silently diff --git a/vendor/golang.org/x/sys/cpu/runtime_auxv.go b/vendor/golang.org/x/sys/cpu/runtime_auxv.go new file mode 100644 index 00000000..5f92ac9a --- /dev/null +++ b/vendor/golang.org/x/sys/cpu/runtime_auxv.go @@ -0,0 +1,16 @@ +// Copyright 2023 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package cpu + +// getAuxvFn is non-nil on Go 1.21+ (via runtime_auxv_go121.go init) +// on platforms that use auxv. +var getAuxvFn func() []uintptr + +func getAuxv() []uintptr { + if getAuxvFn == nil { + return nil + } + return getAuxvFn() +} diff --git a/vendor/golang.org/x/sys/cpu/runtime_auxv_go121.go b/vendor/golang.org/x/sys/cpu/runtime_auxv_go121.go new file mode 100644 index 00000000..b975ea2a --- /dev/null +++ b/vendor/golang.org/x/sys/cpu/runtime_auxv_go121.go @@ -0,0 +1,19 @@ +// Copyright 2023 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +//go:build go1.21 +// +build go1.21 + +package cpu + +import ( + _ "unsafe" // for linkname +) + +//go:linkname runtime_getAuxv runtime.getAuxv +func runtime_getAuxv() []uintptr + +func init() { + getAuxvFn = runtime_getAuxv +} diff --git a/vendor/golang.org/x/sys/execabs/execabs.go b/vendor/golang.org/x/sys/execabs/execabs.go index b981cfbb..3bf40fdf 100644 --- a/vendor/golang.org/x/sys/execabs/execabs.go +++ b/vendor/golang.org/x/sys/execabs/execabs.go @@ -63,7 +63,7 @@ func LookPath(file string) (string, error) { } func fixCmd(name string, cmd *exec.Cmd) { - if filepath.Base(name) == name && !filepath.IsAbs(cmd.Path) { + if filepath.Base(name) == name && !filepath.IsAbs(cmd.Path) && !isGo119ErrFieldSet(cmd) { // exec.Command was called with a bare binary name and // exec.LookPath returned a path which is not absolute. // Set cmd.lookPathErr and clear cmd.Path so that it diff --git a/vendor/golang.org/x/sys/execabs/execabs_go118.go b/vendor/golang.org/x/sys/execabs/execabs_go118.go index 6ab5f508..2000064a 100644 --- a/vendor/golang.org/x/sys/execabs/execabs_go118.go +++ b/vendor/golang.org/x/sys/execabs/execabs_go118.go @@ -7,6 +7,12 @@ package execabs +import "os/exec" + func isGo119ErrDot(err error) bool { return false } + +func isGo119ErrFieldSet(cmd *exec.Cmd) bool { + return false +} diff --git a/vendor/golang.org/x/sys/execabs/execabs_go119.go b/vendor/golang.org/x/sys/execabs/execabs_go119.go index 46c5b525..f364b341 100644 --- a/vendor/golang.org/x/sys/execabs/execabs_go119.go +++ b/vendor/golang.org/x/sys/execabs/execabs_go119.go @@ -15,3 +15,7 @@ import ( func isGo119ErrDot(err error) bool { return errors.Is(err, exec.ErrDot) } + +func isGo119ErrFieldSet(cmd *exec.Cmd) bool { + return cmd.Err != nil +} diff --git a/vendor/golang.org/x/sys/unix/ioctl.go b/vendor/golang.org/x/sys/unix/ioctl.go index 1c51b0ec..7ce8dd40 100644 --- a/vendor/golang.org/x/sys/unix/ioctl.go +++ b/vendor/golang.org/x/sys/unix/ioctl.go @@ -8,7 +8,6 @@ package unix import ( - "runtime" "unsafe" ) @@ -27,7 +26,7 @@ func IoctlSetInt(fd int, req uint, value int) error { // passing the integer value directly. func IoctlSetPointerInt(fd int, req uint, value int) error { v := int32(value) - return ioctl(fd, req, uintptr(unsafe.Pointer(&v))) + return ioctlPtr(fd, req, unsafe.Pointer(&v)) } // IoctlSetWinsize performs an ioctl on fd with a *Winsize argument. @@ -36,9 +35,7 @@ func IoctlSetPointerInt(fd int, req uint, value int) error { func IoctlSetWinsize(fd int, req uint, value *Winsize) error { // TODO: if we get the chance, remove the req parameter and // hardcode TIOCSWINSZ. - err := ioctl(fd, req, uintptr(unsafe.Pointer(value))) - runtime.KeepAlive(value) - return err + return ioctlPtr(fd, req, unsafe.Pointer(value)) } // IoctlSetTermios performs an ioctl on fd with a *Termios. @@ -46,9 +43,7 @@ func IoctlSetWinsize(fd int, req uint, value *Winsize) error { // The req value will usually be TCSETA or TIOCSETA. func IoctlSetTermios(fd int, req uint, value *Termios) error { // TODO: if we get the chance, remove the req parameter. - err := ioctl(fd, req, uintptr(unsafe.Pointer(value))) - runtime.KeepAlive(value) - return err + return ioctlPtr(fd, req, unsafe.Pointer(value)) } // IoctlGetInt performs an ioctl operation which gets an integer value @@ -58,18 +53,18 @@ func IoctlSetTermios(fd int, req uint, value *Termios) error { // for those, IoctlRetInt should be used instead of this function. func IoctlGetInt(fd int, req uint) (int, error) { var value int - err := ioctl(fd, req, uintptr(unsafe.Pointer(&value))) + err := ioctlPtr(fd, req, unsafe.Pointer(&value)) return value, err } func IoctlGetWinsize(fd int, req uint) (*Winsize, error) { var value Winsize - err := ioctl(fd, req, uintptr(unsafe.Pointer(&value))) + err := ioctlPtr(fd, req, unsafe.Pointer(&value)) return &value, err } func IoctlGetTermios(fd int, req uint) (*Termios, error) { var value Termios - err := ioctl(fd, req, uintptr(unsafe.Pointer(&value))) + err := ioctlPtr(fd, req, unsafe.Pointer(&value)) return &value, err } diff --git a/vendor/golang.org/x/sys/unix/ioctl_zos.go b/vendor/golang.org/x/sys/unix/ioctl_zos.go index 5384e7d9..6532f09a 100644 --- a/vendor/golang.org/x/sys/unix/ioctl_zos.go +++ b/vendor/golang.org/x/sys/unix/ioctl_zos.go @@ -27,9 +27,7 @@ func IoctlSetInt(fd int, req uint, value int) error { func IoctlSetWinsize(fd int, req uint, value *Winsize) error { // TODO: if we get the chance, remove the req parameter and // hardcode TIOCSWINSZ. - err := ioctl(fd, req, uintptr(unsafe.Pointer(value))) - runtime.KeepAlive(value) - return err + return ioctlPtr(fd, req, unsafe.Pointer(value)) } // IoctlSetTermios performs an ioctl on fd with a *Termios. @@ -51,13 +49,13 @@ func IoctlSetTermios(fd int, req uint, value *Termios) error { // for those, IoctlRetInt should be used instead of this function. func IoctlGetInt(fd int, req uint) (int, error) { var value int - err := ioctl(fd, req, uintptr(unsafe.Pointer(&value))) + err := ioctlPtr(fd, req, unsafe.Pointer(&value)) return value, err } func IoctlGetWinsize(fd int, req uint) (*Winsize, error) { var value Winsize - err := ioctl(fd, req, uintptr(unsafe.Pointer(&value))) + err := ioctlPtr(fd, req, unsafe.Pointer(&value)) return &value, err } diff --git a/vendor/golang.org/x/sys/unix/ptrace_darwin.go b/vendor/golang.org/x/sys/unix/ptrace_darwin.go index 463c3eff..39dba6ca 100644 --- a/vendor/golang.org/x/sys/unix/ptrace_darwin.go +++ b/vendor/golang.org/x/sys/unix/ptrace_darwin.go @@ -7,6 +7,12 @@ package unix +import "unsafe" + func ptrace(request int, pid int, addr uintptr, data uintptr) error { return ptrace1(request, pid, addr, data) } + +func ptracePtr(request int, pid int, addr uintptr, data unsafe.Pointer) error { + return ptrace1Ptr(request, pid, addr, data) +} diff --git a/vendor/golang.org/x/sys/unix/ptrace_ios.go b/vendor/golang.org/x/sys/unix/ptrace_ios.go index ed0509a0..9ea66330 100644 --- a/vendor/golang.org/x/sys/unix/ptrace_ios.go +++ b/vendor/golang.org/x/sys/unix/ptrace_ios.go @@ -7,6 +7,12 @@ package unix +import "unsafe" + func ptrace(request int, pid int, addr uintptr, data uintptr) (err error) { return ENOTSUP } + +func ptracePtr(request int, pid int, addr uintptr, data unsafe.Pointer) (err error) { + return ENOTSUP +} diff --git a/vendor/golang.org/x/sys/unix/syscall_aix.go b/vendor/golang.org/x/sys/unix/syscall_aix.go index 2db1b51e..d9f5544c 100644 --- a/vendor/golang.org/x/sys/unix/syscall_aix.go +++ b/vendor/golang.org/x/sys/unix/syscall_aix.go @@ -292,9 +292,7 @@ func anyToSockaddr(fd int, rsa *RawSockaddrAny) (Sockaddr, error) { break } } - - bytes := (*[len(pp.Path)]byte)(unsafe.Pointer(&pp.Path[0]))[0:n] - sa.Name = string(bytes) + sa.Name = string(unsafe.Slice((*byte)(unsafe.Pointer(&pp.Path[0])), n)) return sa, nil case AF_INET: @@ -411,6 +409,7 @@ func (w WaitStatus) CoreDump() bool { return w&0x80 == 0x80 } func (w WaitStatus) TrapCause() int { return -1 } //sys ioctl(fd int, req uint, arg uintptr) (err error) +//sys ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) = ioctl // fcntl must never be called with cmd=F_DUP2FD because it doesn't work on AIX // There is no way to create a custom fcntl and to keep //sys fcntl easily, diff --git a/vendor/golang.org/x/sys/unix/syscall_bsd.go b/vendor/golang.org/x/sys/unix/syscall_bsd.go index eda42671..7705c327 100644 --- a/vendor/golang.org/x/sys/unix/syscall_bsd.go +++ b/vendor/golang.org/x/sys/unix/syscall_bsd.go @@ -245,8 +245,7 @@ func anyToSockaddr(fd int, rsa *RawSockaddrAny) (Sockaddr, error) { break } } - bytes := (*[len(pp.Path)]byte)(unsafe.Pointer(&pp.Path[0]))[0:n] - sa.Name = string(bytes) + sa.Name = string(unsafe.Slice((*byte)(unsafe.Pointer(&pp.Path[0])), n)) return sa, nil case AF_INET: diff --git a/vendor/golang.org/x/sys/unix/syscall_darwin.go b/vendor/golang.org/x/sys/unix/syscall_darwin.go index 192b071b..7064d6eb 100644 --- a/vendor/golang.org/x/sys/unix/syscall_darwin.go +++ b/vendor/golang.org/x/sys/unix/syscall_darwin.go @@ -14,7 +14,6 @@ package unix import ( "fmt" - "runtime" "syscall" "unsafe" ) @@ -376,11 +375,10 @@ func Flistxattr(fd int, dest []byte) (sz int, err error) { func Kill(pid int, signum syscall.Signal) (err error) { return kill(pid, int(signum), 1) } //sys ioctl(fd int, req uint, arg uintptr) (err error) +//sys ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) = SYS_IOCTL func IoctlCtlInfo(fd int, ctlInfo *CtlInfo) error { - err := ioctl(fd, CTLIOCGINFO, uintptr(unsafe.Pointer(ctlInfo))) - runtime.KeepAlive(ctlInfo) - return err + return ioctlPtr(fd, CTLIOCGINFO, unsafe.Pointer(ctlInfo)) } // IfreqMTU is struct ifreq used to get or set a network device's MTU. @@ -394,16 +392,14 @@ type IfreqMTU struct { func IoctlGetIfreqMTU(fd int, ifname string) (*IfreqMTU, error) { var ifreq IfreqMTU copy(ifreq.Name[:], ifname) - err := ioctl(fd, SIOCGIFMTU, uintptr(unsafe.Pointer(&ifreq))) + err := ioctlPtr(fd, SIOCGIFMTU, unsafe.Pointer(&ifreq)) return &ifreq, err } // IoctlSetIfreqMTU performs the SIOCSIFMTU ioctl operation on fd to set the MTU // of the network device specified by ifreq.Name. func IoctlSetIfreqMTU(fd int, ifreq *IfreqMTU) error { - err := ioctl(fd, SIOCSIFMTU, uintptr(unsafe.Pointer(ifreq))) - runtime.KeepAlive(ifreq) - return err + return ioctlPtr(fd, SIOCSIFMTU, unsafe.Pointer(ifreq)) } //sys sysctl(mib []_C_int, old *byte, oldlen *uintptr, new *byte, newlen uintptr) (err error) = SYS_SYSCTL diff --git a/vendor/golang.org/x/sys/unix/syscall_darwin_amd64.go b/vendor/golang.org/x/sys/unix/syscall_darwin_amd64.go index b37310ce..9fa87980 100644 --- a/vendor/golang.org/x/sys/unix/syscall_darwin_amd64.go +++ b/vendor/golang.org/x/sys/unix/syscall_darwin_amd64.go @@ -47,5 +47,6 @@ func Syscall9(num, a1, a2, a3, a4, a5, a6, a7, a8, a9 uintptr) (r1, r2 uintptr, //sys getfsstat(buf unsafe.Pointer, size uintptr, flags int) (n int, err error) = SYS_GETFSSTAT64 //sys Lstat(path string, stat *Stat_t) (err error) = SYS_LSTAT64 //sys ptrace1(request int, pid int, addr uintptr, data uintptr) (err error) = SYS_ptrace +//sys ptrace1Ptr(request int, pid int, addr unsafe.Pointer, data uintptr) (err error) = SYS_ptrace //sys Stat(path string, stat *Stat_t) (err error) = SYS_STAT64 //sys Statfs(path string, stat *Statfs_t) (err error) = SYS_STATFS64 diff --git a/vendor/golang.org/x/sys/unix/syscall_darwin_arm64.go b/vendor/golang.org/x/sys/unix/syscall_darwin_arm64.go index d51ec996..f17b8c52 100644 --- a/vendor/golang.org/x/sys/unix/syscall_darwin_arm64.go +++ b/vendor/golang.org/x/sys/unix/syscall_darwin_arm64.go @@ -47,5 +47,6 @@ func Syscall9(num, a1, a2, a3, a4, a5, a6, a7, a8, a9 uintptr) (r1, r2 uintptr, //sys getfsstat(buf unsafe.Pointer, size uintptr, flags int) (n int, err error) = SYS_GETFSSTAT //sys Lstat(path string, stat *Stat_t) (err error) //sys ptrace1(request int, pid int, addr uintptr, data uintptr) (err error) = SYS_ptrace +//sys ptrace1Ptr(request int, pid int, addr unsafe.Pointer, data uintptr) (err error) = SYS_ptrace //sys Stat(path string, stat *Stat_t) (err error) //sys Statfs(path string, stat *Statfs_t) (err error) diff --git a/vendor/golang.org/x/sys/unix/syscall_dragonfly.go b/vendor/golang.org/x/sys/unix/syscall_dragonfly.go index a41111a7..221efc26 100644 --- a/vendor/golang.org/x/sys/unix/syscall_dragonfly.go +++ b/vendor/golang.org/x/sys/unix/syscall_dragonfly.go @@ -172,6 +172,7 @@ func Getfsstat(buf []Statfs_t, flags int) (n int, err error) { } //sys ioctl(fd int, req uint, arg uintptr) (err error) +//sys ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) = SYS_IOCTL //sys sysctl(mib []_C_int, old *byte, oldlen *uintptr, new *byte, newlen uintptr) (err error) = SYS___SYSCTL diff --git a/vendor/golang.org/x/sys/unix/syscall_freebsd.go b/vendor/golang.org/x/sys/unix/syscall_freebsd.go index d50b9dc2..5bdde03e 100644 --- a/vendor/golang.org/x/sys/unix/syscall_freebsd.go +++ b/vendor/golang.org/x/sys/unix/syscall_freebsd.go @@ -161,7 +161,8 @@ func Getfsstat(buf []Statfs_t, flags int) (n int, err error) { return } -//sys ioctl(fd int, req uint, arg uintptr) (err error) +//sys ioctl(fd int, req uint, arg uintptr) (err error) = SYS_IOCTL +//sys ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) = SYS_IOCTL //sys sysctl(mib []_C_int, old *byte, oldlen *uintptr, new *byte, newlen uintptr) (err error) = SYS___SYSCTL @@ -253,6 +254,7 @@ func Sendfile(outfd int, infd int, offset *int64, count int) (written int, err e } //sys ptrace(request int, pid int, addr uintptr, data int) (err error) +//sys ptracePtr(request int, pid int, addr unsafe.Pointer, data int) (err error) = SYS_PTRACE func PtraceAttach(pid int) (err error) { return ptrace(PT_ATTACH, pid, 0, 0) @@ -267,19 +269,36 @@ func PtraceDetach(pid int) (err error) { } func PtraceGetFpRegs(pid int, fpregsout *FpReg) (err error) { - return ptrace(PT_GETFPREGS, pid, uintptr(unsafe.Pointer(fpregsout)), 0) + return ptracePtr(PT_GETFPREGS, pid, unsafe.Pointer(fpregsout), 0) } func PtraceGetRegs(pid int, regsout *Reg) (err error) { - return ptrace(PT_GETREGS, pid, uintptr(unsafe.Pointer(regsout)), 0) + return ptracePtr(PT_GETREGS, pid, unsafe.Pointer(regsout), 0) +} + +func PtraceIO(req int, pid int, offs uintptr, out []byte, countin int) (count int, err error) { + ioDesc := PtraceIoDesc{ + Op: int32(req), + Offs: offs, + } + if countin > 0 { + _ = out[:countin] // check bounds + ioDesc.Addr = &out[0] + } else if out != nil { + ioDesc.Addr = (*byte)(unsafe.Pointer(&_zero)) + } + ioDesc.SetLen(countin) + + err = ptracePtr(PT_IO, pid, unsafe.Pointer(&ioDesc), 0) + return int(ioDesc.Len), err } func PtraceLwpEvents(pid int, enable int) (err error) { return ptrace(PT_LWP_EVENTS, pid, 0, enable) } -func PtraceLwpInfo(pid int, info uintptr) (err error) { - return ptrace(PT_LWPINFO, pid, info, int(unsafe.Sizeof(PtraceLwpInfoStruct{}))) +func PtraceLwpInfo(pid int, info *PtraceLwpInfoStruct) (err error) { + return ptracePtr(PT_LWPINFO, pid, unsafe.Pointer(info), int(unsafe.Sizeof(*info))) } func PtracePeekData(pid int, addr uintptr, out []byte) (count int, err error) { @@ -299,13 +318,25 @@ func PtracePokeText(pid int, addr uintptr, data []byte) (count int, err error) { } func PtraceSetRegs(pid int, regs *Reg) (err error) { - return ptrace(PT_SETREGS, pid, uintptr(unsafe.Pointer(regs)), 0) + return ptracePtr(PT_SETREGS, pid, unsafe.Pointer(regs), 0) } func PtraceSingleStep(pid int) (err error) { return ptrace(PT_STEP, pid, 1, 0) } +func Dup3(oldfd, newfd, flags int) error { + if oldfd == newfd || flags&^O_CLOEXEC != 0 { + return EINVAL + } + how := F_DUP2FD + if flags&O_CLOEXEC != 0 { + how = F_DUP2FD_CLOEXEC + } + _, err := fcntl(oldfd, how, newfd) + return err +} + /* * Exposed directly */ diff --git a/vendor/golang.org/x/sys/unix/syscall_freebsd_386.go b/vendor/golang.org/x/sys/unix/syscall_freebsd_386.go index 6a91d471..b8da5100 100644 --- a/vendor/golang.org/x/sys/unix/syscall_freebsd_386.go +++ b/vendor/golang.org/x/sys/unix/syscall_freebsd_386.go @@ -42,6 +42,10 @@ func (cmsg *Cmsghdr) SetLen(length int) { cmsg.Len = uint32(length) } +func (d *PtraceIoDesc) SetLen(length int) { + d.Len = uint32(length) +} + func sendfile(outfd int, infd int, offset *int64, count int) (written int, err error) { var writtenOut uint64 = 0 _, _, e1 := Syscall9(SYS_SENDFILE, uintptr(infd), uintptr(outfd), uintptr(*offset), uintptr((*offset)>>32), uintptr(count), 0, uintptr(unsafe.Pointer(&writtenOut)), 0, 0) @@ -57,16 +61,5 @@ func sendfile(outfd int, infd int, offset *int64, count int) (written int, err e func Syscall9(num, a1, a2, a3, a4, a5, a6, a7, a8, a9 uintptr) (r1, r2 uintptr, err syscall.Errno) func PtraceGetFsBase(pid int, fsbase *int64) (err error) { - return ptrace(PT_GETFSBASE, pid, uintptr(unsafe.Pointer(fsbase)), 0) -} - -func PtraceIO(req int, pid int, offs uintptr, out []byte, countin int) (count int, err error) { - ioDesc := PtraceIoDesc{ - Op: int32(req), - Offs: offs, - Addr: uintptr(unsafe.Pointer(&out[0])), // TODO(#58351): this is not safe. - Len: uint32(countin), - } - err = ptrace(PT_IO, pid, uintptr(unsafe.Pointer(&ioDesc)), 0) - return int(ioDesc.Len), err + return ptracePtr(PT_GETFSBASE, pid, unsafe.Pointer(fsbase), 0) } diff --git a/vendor/golang.org/x/sys/unix/syscall_freebsd_amd64.go b/vendor/golang.org/x/sys/unix/syscall_freebsd_amd64.go index 48110a0a..47155c48 100644 --- a/vendor/golang.org/x/sys/unix/syscall_freebsd_amd64.go +++ b/vendor/golang.org/x/sys/unix/syscall_freebsd_amd64.go @@ -42,6 +42,10 @@ func (cmsg *Cmsghdr) SetLen(length int) { cmsg.Len = uint32(length) } +func (d *PtraceIoDesc) SetLen(length int) { + d.Len = uint64(length) +} + func sendfile(outfd int, infd int, offset *int64, count int) (written int, err error) { var writtenOut uint64 = 0 _, _, e1 := Syscall9(SYS_SENDFILE, uintptr(infd), uintptr(outfd), uintptr(*offset), uintptr(count), 0, uintptr(unsafe.Pointer(&writtenOut)), 0, 0, 0) @@ -57,16 +61,5 @@ func sendfile(outfd int, infd int, offset *int64, count int) (written int, err e func Syscall9(num, a1, a2, a3, a4, a5, a6, a7, a8, a9 uintptr) (r1, r2 uintptr, err syscall.Errno) func PtraceGetFsBase(pid int, fsbase *int64) (err error) { - return ptrace(PT_GETFSBASE, pid, uintptr(unsafe.Pointer(fsbase)), 0) -} - -func PtraceIO(req int, pid int, offs uintptr, out []byte, countin int) (count int, err error) { - ioDesc := PtraceIoDesc{ - Op: int32(req), - Offs: offs, - Addr: uintptr(unsafe.Pointer(&out[0])), // TODO(#58351): this is not safe. - Len: uint64(countin), - } - err = ptrace(PT_IO, pid, uintptr(unsafe.Pointer(&ioDesc)), 0) - return int(ioDesc.Len), err + return ptracePtr(PT_GETFSBASE, pid, unsafe.Pointer(fsbase), 0) } diff --git a/vendor/golang.org/x/sys/unix/syscall_freebsd_arm.go b/vendor/golang.org/x/sys/unix/syscall_freebsd_arm.go index 52f1d4b7..08932093 100644 --- a/vendor/golang.org/x/sys/unix/syscall_freebsd_arm.go +++ b/vendor/golang.org/x/sys/unix/syscall_freebsd_arm.go @@ -42,6 +42,10 @@ func (cmsg *Cmsghdr) SetLen(length int) { cmsg.Len = uint32(length) } +func (d *PtraceIoDesc) SetLen(length int) { + d.Len = uint32(length) +} + func sendfile(outfd int, infd int, offset *int64, count int) (written int, err error) { var writtenOut uint64 = 0 _, _, e1 := Syscall9(SYS_SENDFILE, uintptr(infd), uintptr(outfd), uintptr(*offset), uintptr((*offset)>>32), uintptr(count), 0, uintptr(unsafe.Pointer(&writtenOut)), 0, 0) @@ -55,14 +59,3 @@ func sendfile(outfd int, infd int, offset *int64, count int) (written int, err e } func Syscall9(num, a1, a2, a3, a4, a5, a6, a7, a8, a9 uintptr) (r1, r2 uintptr, err syscall.Errno) - -func PtraceIO(req int, pid int, offs uintptr, out []byte, countin int) (count int, err error) { - ioDesc := PtraceIoDesc{ - Op: int32(req), - Offs: offs, - Addr: uintptr(unsafe.Pointer(&out[0])), // TODO(#58351): this is not safe. - Len: uint32(countin), - } - err = ptrace(PT_IO, pid, uintptr(unsafe.Pointer(&ioDesc)), 0) - return int(ioDesc.Len), err -} diff --git a/vendor/golang.org/x/sys/unix/syscall_freebsd_arm64.go b/vendor/golang.org/x/sys/unix/syscall_freebsd_arm64.go index 5537ee4f..d151a0d0 100644 --- a/vendor/golang.org/x/sys/unix/syscall_freebsd_arm64.go +++ b/vendor/golang.org/x/sys/unix/syscall_freebsd_arm64.go @@ -42,6 +42,10 @@ func (cmsg *Cmsghdr) SetLen(length int) { cmsg.Len = uint32(length) } +func (d *PtraceIoDesc) SetLen(length int) { + d.Len = uint64(length) +} + func sendfile(outfd int, infd int, offset *int64, count int) (written int, err error) { var writtenOut uint64 = 0 _, _, e1 := Syscall9(SYS_SENDFILE, uintptr(infd), uintptr(outfd), uintptr(*offset), uintptr(count), 0, uintptr(unsafe.Pointer(&writtenOut)), 0, 0, 0) @@ -55,14 +59,3 @@ func sendfile(outfd int, infd int, offset *int64, count int) (written int, err e } func Syscall9(num, a1, a2, a3, a4, a5, a6, a7, a8, a9 uintptr) (r1, r2 uintptr, err syscall.Errno) - -func PtraceIO(req int, pid int, offs uintptr, out []byte, countin int) (count int, err error) { - ioDesc := PtraceIoDesc{ - Op: int32(req), - Offs: offs, - Addr: uintptr(unsafe.Pointer(&out[0])), // TODO(#58351): this is not safe. - Len: uint64(countin), - } - err = ptrace(PT_IO, pid, uintptr(unsafe.Pointer(&ioDesc)), 0) - return int(ioDesc.Len), err -} diff --git a/vendor/golang.org/x/sys/unix/syscall_freebsd_riscv64.go b/vendor/golang.org/x/sys/unix/syscall_freebsd_riscv64.go index 164abd5d..d5cd64b3 100644 --- a/vendor/golang.org/x/sys/unix/syscall_freebsd_riscv64.go +++ b/vendor/golang.org/x/sys/unix/syscall_freebsd_riscv64.go @@ -42,6 +42,10 @@ func (cmsg *Cmsghdr) SetLen(length int) { cmsg.Len = uint32(length) } +func (d *PtraceIoDesc) SetLen(length int) { + d.Len = uint64(length) +} + func sendfile(outfd int, infd int, offset *int64, count int) (written int, err error) { var writtenOut uint64 = 0 _, _, e1 := Syscall9(SYS_SENDFILE, uintptr(infd), uintptr(outfd), uintptr(*offset), uintptr(count), 0, uintptr(unsafe.Pointer(&writtenOut)), 0, 0, 0) @@ -55,14 +59,3 @@ func sendfile(outfd int, infd int, offset *int64, count int) (written int, err e } func Syscall9(num, a1, a2, a3, a4, a5, a6, a7, a8, a9 uintptr) (r1, r2 uintptr, err syscall.Errno) - -func PtraceIO(req int, pid int, offs uintptr, out []byte, countin int) (count int, err error) { - ioDesc := PtraceIoDesc{ - Op: int32(req), - Offs: offs, - Addr: uintptr(unsafe.Pointer(&out[0])), // TODO(#58351): this is not safe. - Len: uint64(countin), - } - err = ptrace(PT_IO, pid, uintptr(unsafe.Pointer(&ioDesc)), 0) - return int(ioDesc.Len), err -} diff --git a/vendor/golang.org/x/sys/unix/syscall_hurd.go b/vendor/golang.org/x/sys/unix/syscall_hurd.go index 4ffb6480..381fd467 100644 --- a/vendor/golang.org/x/sys/unix/syscall_hurd.go +++ b/vendor/golang.org/x/sys/unix/syscall_hurd.go @@ -20,3 +20,11 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { } return } + +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + r0, er := C.ioctl(C.int(fd), C.ulong(req), C.uintptr_t(uintptr(arg))) + if r0 == -1 && er != nil { + err = er + } + return +} diff --git a/vendor/golang.org/x/sys/unix/syscall_linux.go b/vendor/golang.org/x/sys/unix/syscall_linux.go index 5443dddd..97353315 100644 --- a/vendor/golang.org/x/sys/unix/syscall_linux.go +++ b/vendor/golang.org/x/sys/unix/syscall_linux.go @@ -1015,8 +1015,7 @@ func anyToSockaddr(fd int, rsa *RawSockaddrAny) (Sockaddr, error) { for n < len(pp.Path) && pp.Path[n] != 0 { n++ } - bytes := (*[len(pp.Path)]byte)(unsafe.Pointer(&pp.Path[0]))[0:n] - sa.Name = string(bytes) + sa.Name = string(unsafe.Slice((*byte)(unsafe.Pointer(&pp.Path[0])), n)) return sa, nil case AF_INET: @@ -1365,6 +1364,10 @@ func SetsockoptTCPRepairOpt(fd, level, opt int, o []TCPRepairOpt) (err error) { return setsockopt(fd, level, opt, unsafe.Pointer(&o[0]), uintptr(SizeofTCPRepairOpt*len(o))) } +func SetsockoptTCPMD5Sig(fd, level, opt int, s *TCPMD5Sig) error { + return setsockopt(fd, level, opt, unsafe.Pointer(s), unsafe.Sizeof(*s)) +} + // Keyctl Commands (http://man7.org/linux/man-pages/man2/keyctl.2.html) // KeyctlInt calls keyctl commands in which each argument is an int. @@ -1579,6 +1582,7 @@ func BindToDevice(fd int, device string) (err error) { } //sys ptrace(request int, pid int, addr uintptr, data uintptr) (err error) +//sys ptracePtr(request int, pid int, addr uintptr, data unsafe.Pointer) (err error) = SYS_PTRACE func ptracePeek(req int, pid int, addr uintptr, out []byte) (count int, err error) { // The peek requests are machine-size oriented, so we wrap it @@ -1596,7 +1600,7 @@ func ptracePeek(req int, pid int, addr uintptr, out []byte) (count int, err erro // boundary. n := 0 if addr%SizeofPtr != 0 { - err = ptrace(req, pid, addr-addr%SizeofPtr, uintptr(unsafe.Pointer(&buf[0]))) + err = ptracePtr(req, pid, addr-addr%SizeofPtr, unsafe.Pointer(&buf[0])) if err != nil { return 0, err } @@ -1608,7 +1612,7 @@ func ptracePeek(req int, pid int, addr uintptr, out []byte) (count int, err erro for len(out) > 0 { // We use an internal buffer to guarantee alignment. // It's not documented if this is necessary, but we're paranoid. - err = ptrace(req, pid, addr+uintptr(n), uintptr(unsafe.Pointer(&buf[0]))) + err = ptracePtr(req, pid, addr+uintptr(n), unsafe.Pointer(&buf[0])) if err != nil { return n, err } @@ -1640,7 +1644,7 @@ func ptracePoke(pokeReq int, peekReq int, pid int, addr uintptr, data []byte) (c n := 0 if addr%SizeofPtr != 0 { var buf [SizeofPtr]byte - err = ptrace(peekReq, pid, addr-addr%SizeofPtr, uintptr(unsafe.Pointer(&buf[0]))) + err = ptracePtr(peekReq, pid, addr-addr%SizeofPtr, unsafe.Pointer(&buf[0])) if err != nil { return 0, err } @@ -1667,7 +1671,7 @@ func ptracePoke(pokeReq int, peekReq int, pid int, addr uintptr, data []byte) (c // Trailing edge. if len(data) > 0 { var buf [SizeofPtr]byte - err = ptrace(peekReq, pid, addr+uintptr(n), uintptr(unsafe.Pointer(&buf[0]))) + err = ptracePtr(peekReq, pid, addr+uintptr(n), unsafe.Pointer(&buf[0])) if err != nil { return n, err } @@ -1696,11 +1700,11 @@ func PtracePokeUser(pid int, addr uintptr, data []byte) (count int, err error) { } func PtraceGetRegs(pid int, regsout *PtraceRegs) (err error) { - return ptrace(PTRACE_GETREGS, pid, 0, uintptr(unsafe.Pointer(regsout))) + return ptracePtr(PTRACE_GETREGS, pid, 0, unsafe.Pointer(regsout)) } func PtraceSetRegs(pid int, regs *PtraceRegs) (err error) { - return ptrace(PTRACE_SETREGS, pid, 0, uintptr(unsafe.Pointer(regs))) + return ptracePtr(PTRACE_SETREGS, pid, 0, unsafe.Pointer(regs)) } func PtraceSetOptions(pid int, options int) (err error) { @@ -1709,7 +1713,7 @@ func PtraceSetOptions(pid int, options int) (err error) { func PtraceGetEventMsg(pid int) (msg uint, err error) { var data _C_long - err = ptrace(PTRACE_GETEVENTMSG, pid, 0, uintptr(unsafe.Pointer(&data))) + err = ptracePtr(PTRACE_GETEVENTMSG, pid, 0, unsafe.Pointer(&data)) msg = uint(data) return } @@ -2154,6 +2158,14 @@ func isGroupMember(gid int) bool { return false } +func isCapDacOverrideSet() bool { + hdr := CapUserHeader{Version: LINUX_CAPABILITY_VERSION_3} + data := [2]CapUserData{} + err := Capget(&hdr, &data[0]) + + return err == nil && data[0].Effective&(1< 0 { diff --git a/vendor/golang.org/x/sys/unix/zsyscall_freebsd_386.go b/vendor/golang.org/x/sys/unix/zsyscall_freebsd_386.go index 77479d45..11290656 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_freebsd_386.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_freebsd_386.go @@ -388,6 +388,16 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := Syscall(SYS_IOCTL, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func sysctl(mib []_C_int, old *byte, oldlen *uintptr, new *byte, newlen uintptr) (err error) { var _p0 unsafe.Pointer if len(mib) > 0 { @@ -414,6 +424,16 @@ func ptrace(request int, pid int, addr uintptr, data int) (err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ptracePtr(request int, pid int, addr unsafe.Pointer, data int) (err error) { + _, _, e1 := Syscall6(SYS_PTRACE, uintptr(request), uintptr(pid), uintptr(addr), uintptr(data), 0, 0) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func Access(path string, mode uint32) (err error) { var _p0 *byte _p0, err = BytePtrFromString(path) diff --git a/vendor/golang.org/x/sys/unix/zsyscall_freebsd_amd64.go b/vendor/golang.org/x/sys/unix/zsyscall_freebsd_amd64.go index 2e966d4d..55f5abfe 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_freebsd_amd64.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_freebsd_amd64.go @@ -388,6 +388,16 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := Syscall(SYS_IOCTL, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func sysctl(mib []_C_int, old *byte, oldlen *uintptr, new *byte, newlen uintptr) (err error) { var _p0 unsafe.Pointer if len(mib) > 0 { @@ -414,6 +424,16 @@ func ptrace(request int, pid int, addr uintptr, data int) (err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ptracePtr(request int, pid int, addr unsafe.Pointer, data int) (err error) { + _, _, e1 := Syscall6(SYS_PTRACE, uintptr(request), uintptr(pid), uintptr(addr), uintptr(data), 0, 0) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func Access(path string, mode uint32) (err error) { var _p0 *byte _p0, err = BytePtrFromString(path) diff --git a/vendor/golang.org/x/sys/unix/zsyscall_freebsd_arm.go b/vendor/golang.org/x/sys/unix/zsyscall_freebsd_arm.go index d65a7c0f..d39651c2 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_freebsd_arm.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_freebsd_arm.go @@ -388,6 +388,16 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := Syscall(SYS_IOCTL, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func sysctl(mib []_C_int, old *byte, oldlen *uintptr, new *byte, newlen uintptr) (err error) { var _p0 unsafe.Pointer if len(mib) > 0 { @@ -414,6 +424,16 @@ func ptrace(request int, pid int, addr uintptr, data int) (err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ptracePtr(request int, pid int, addr unsafe.Pointer, data int) (err error) { + _, _, e1 := Syscall6(SYS_PTRACE, uintptr(request), uintptr(pid), uintptr(addr), uintptr(data), 0, 0) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func Access(path string, mode uint32) (err error) { var _p0 *byte _p0, err = BytePtrFromString(path) diff --git a/vendor/golang.org/x/sys/unix/zsyscall_freebsd_arm64.go b/vendor/golang.org/x/sys/unix/zsyscall_freebsd_arm64.go index 6f0b97c6..ddb74086 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_freebsd_arm64.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_freebsd_arm64.go @@ -388,6 +388,16 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := Syscall(SYS_IOCTL, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func sysctl(mib []_C_int, old *byte, oldlen *uintptr, new *byte, newlen uintptr) (err error) { var _p0 unsafe.Pointer if len(mib) > 0 { @@ -414,6 +424,16 @@ func ptrace(request int, pid int, addr uintptr, data int) (err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ptracePtr(request int, pid int, addr unsafe.Pointer, data int) (err error) { + _, _, e1 := Syscall6(SYS_PTRACE, uintptr(request), uintptr(pid), uintptr(addr), uintptr(data), 0, 0) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func Access(path string, mode uint32) (err error) { var _p0 *byte _p0, err = BytePtrFromString(path) diff --git a/vendor/golang.org/x/sys/unix/zsyscall_freebsd_riscv64.go b/vendor/golang.org/x/sys/unix/zsyscall_freebsd_riscv64.go index e1c23b52..09a53a61 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_freebsd_riscv64.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_freebsd_riscv64.go @@ -388,6 +388,16 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := Syscall(SYS_IOCTL, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func sysctl(mib []_C_int, old *byte, oldlen *uintptr, new *byte, newlen uintptr) (err error) { var _p0 unsafe.Pointer if len(mib) > 0 { @@ -414,6 +424,16 @@ func ptrace(request int, pid int, addr uintptr, data int) (err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ptracePtr(request int, pid int, addr unsafe.Pointer, data int) (err error) { + _, _, e1 := Syscall6(SYS_PTRACE, uintptr(request), uintptr(pid), uintptr(addr), uintptr(data), 0, 0) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func Access(path string, mode uint32) (err error) { var _p0 *byte _p0, err = BytePtrFromString(path) diff --git a/vendor/golang.org/x/sys/unix/zsyscall_linux.go b/vendor/golang.org/x/sys/unix/zsyscall_linux.go index 36ea3a55..430cb24d 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_linux.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_linux.go @@ -379,6 +379,16 @@ func ptrace(request int, pid int, addr uintptr, data uintptr) (err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ptracePtr(request int, pid int, addr uintptr, data unsafe.Pointer) (err error) { + _, _, e1 := Syscall6(SYS_PTRACE, uintptr(request), uintptr(pid), uintptr(addr), uintptr(data), 0, 0) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func reboot(magic1 uint, magic2 uint, cmd int, arg string) (err error) { var _p0 *byte _p0, err = BytePtrFromString(arg) diff --git a/vendor/golang.org/x/sys/unix/zsyscall_netbsd_386.go b/vendor/golang.org/x/sys/unix/zsyscall_netbsd_386.go index 79f73899..8e1d9c8f 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_netbsd_386.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_netbsd_386.go @@ -405,6 +405,16 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := Syscall(SYS_IOCTL, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func sysctl(mib []_C_int, old *byte, oldlen *uintptr, new *byte, newlen uintptr) (err error) { var _p0 unsafe.Pointer if len(mib) > 0 { diff --git a/vendor/golang.org/x/sys/unix/zsyscall_netbsd_amd64.go b/vendor/golang.org/x/sys/unix/zsyscall_netbsd_amd64.go index fb161f3a..21c69504 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_netbsd_amd64.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_netbsd_amd64.go @@ -405,6 +405,16 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := Syscall(SYS_IOCTL, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func sysctl(mib []_C_int, old *byte, oldlen *uintptr, new *byte, newlen uintptr) (err error) { var _p0 unsafe.Pointer if len(mib) > 0 { diff --git a/vendor/golang.org/x/sys/unix/zsyscall_netbsd_arm.go b/vendor/golang.org/x/sys/unix/zsyscall_netbsd_arm.go index 4c8ac993..298168f9 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_netbsd_arm.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_netbsd_arm.go @@ -405,6 +405,16 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := Syscall(SYS_IOCTL, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func sysctl(mib []_C_int, old *byte, oldlen *uintptr, new *byte, newlen uintptr) (err error) { var _p0 unsafe.Pointer if len(mib) > 0 { diff --git a/vendor/golang.org/x/sys/unix/zsyscall_netbsd_arm64.go b/vendor/golang.org/x/sys/unix/zsyscall_netbsd_arm64.go index 76dd8ec4..68b8bd49 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_netbsd_arm64.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_netbsd_arm64.go @@ -405,6 +405,16 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := Syscall(SYS_IOCTL, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func sysctl(mib []_C_int, old *byte, oldlen *uintptr, new *byte, newlen uintptr) (err error) { var _p0 unsafe.Pointer if len(mib) > 0 { diff --git a/vendor/golang.org/x/sys/unix/zsyscall_openbsd_386.go b/vendor/golang.org/x/sys/unix/zsyscall_openbsd_386.go index caeb807b..0b0f910e 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_openbsd_386.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_openbsd_386.go @@ -527,6 +527,14 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { return } +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := syscall_syscall(libc_ioctl_trampoline_addr, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + var libc_ioctl_trampoline_addr uintptr //go:cgo_import_dynamic libc_ioctl ioctl "libc.so" diff --git a/vendor/golang.org/x/sys/unix/zsyscall_openbsd_amd64.go b/vendor/golang.org/x/sys/unix/zsyscall_openbsd_amd64.go index a05e5f4f..48ff5de7 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_openbsd_amd64.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_openbsd_amd64.go @@ -527,6 +527,14 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { return } +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := syscall_syscall(libc_ioctl_trampoline_addr, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + var libc_ioctl_trampoline_addr uintptr //go:cgo_import_dynamic libc_ioctl ioctl "libc.so" diff --git a/vendor/golang.org/x/sys/unix/zsyscall_openbsd_arm.go b/vendor/golang.org/x/sys/unix/zsyscall_openbsd_arm.go index b2da8e50..2452a641 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_openbsd_arm.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_openbsd_arm.go @@ -527,6 +527,14 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { return } +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := syscall_syscall(libc_ioctl_trampoline_addr, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + var libc_ioctl_trampoline_addr uintptr //go:cgo_import_dynamic libc_ioctl ioctl "libc.so" diff --git a/vendor/golang.org/x/sys/unix/zsyscall_openbsd_arm64.go b/vendor/golang.org/x/sys/unix/zsyscall_openbsd_arm64.go index 048b2655..5e35600a 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_openbsd_arm64.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_openbsd_arm64.go @@ -527,6 +527,14 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { return } +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := syscall_syscall(libc_ioctl_trampoline_addr, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + var libc_ioctl_trampoline_addr uintptr //go:cgo_import_dynamic libc_ioctl ioctl "libc.so" diff --git a/vendor/golang.org/x/sys/unix/zsyscall_openbsd_mips64.go b/vendor/golang.org/x/sys/unix/zsyscall_openbsd_mips64.go index 6f33e37e..b04cef1a 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_openbsd_mips64.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_openbsd_mips64.go @@ -527,6 +527,14 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { return } +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := syscall_syscall(libc_ioctl_trampoline_addr, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + var libc_ioctl_trampoline_addr uintptr //go:cgo_import_dynamic libc_ioctl ioctl "libc.so" diff --git a/vendor/golang.org/x/sys/unix/zsyscall_openbsd_ppc64.go b/vendor/golang.org/x/sys/unix/zsyscall_openbsd_ppc64.go index 330cf7f7..47a07ee0 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_openbsd_ppc64.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_openbsd_ppc64.go @@ -527,6 +527,14 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { return } +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := syscall_syscall(libc_ioctl_trampoline_addr, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + var libc_ioctl_trampoline_addr uintptr //go:cgo_import_dynamic libc_ioctl ioctl "libc.so" diff --git a/vendor/golang.org/x/sys/unix/zsyscall_openbsd_riscv64.go b/vendor/golang.org/x/sys/unix/zsyscall_openbsd_riscv64.go index 5f24de0d..573378fd 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_openbsd_riscv64.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_openbsd_riscv64.go @@ -527,6 +527,14 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { return } +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := syscall_syscall(libc_ioctl_trampoline_addr, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + var libc_ioctl_trampoline_addr uintptr //go:cgo_import_dynamic libc_ioctl ioctl "libc.so" diff --git a/vendor/golang.org/x/sys/unix/zsyscall_solaris_amd64.go b/vendor/golang.org/x/sys/unix/zsyscall_solaris_amd64.go index 78d4a424..4873a1e5 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_solaris_amd64.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_solaris_amd64.go @@ -657,6 +657,17 @@ func ioctlRet(fd int, req uint, arg uintptr) (ret int, err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ioctlPtrRet(fd int, req uint, arg unsafe.Pointer) (ret int, err error) { + r0, _, e1 := sysvicall6(uintptr(unsafe.Pointer(&procioctl)), 3, uintptr(fd), uintptr(req), uintptr(arg), 0, 0, 0) + ret = int(r0) + if e1 != 0 { + err = e1 + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func poll(fds *PollFd, nfds int, timeout int) (n int, err error) { r0, _, e1 := sysvicall6(uintptr(unsafe.Pointer(&procpoll)), 3, uintptr(unsafe.Pointer(fds)), uintptr(nfds), uintptr(timeout), 0, 0, 0) n = int(r0) diff --git a/vendor/golang.org/x/sys/unix/zsyscall_zos_s390x.go b/vendor/golang.org/x/sys/unix/zsyscall_zos_s390x.go index f2079457..07bfe2ef 100644 --- a/vendor/golang.org/x/sys/unix/zsyscall_zos_s390x.go +++ b/vendor/golang.org/x/sys/unix/zsyscall_zos_s390x.go @@ -267,6 +267,16 @@ func ioctl(fd int, req uint, arg uintptr) (err error) { // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT +func ioctlPtr(fd int, req uint, arg unsafe.Pointer) (err error) { + _, _, e1 := syscall_syscall(SYS_IOCTL, uintptr(fd), uintptr(req), uintptr(arg)) + if e1 != 0 { + err = errnoErr(e1) + } + return +} + +// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT + func Access(path string, mode uint32) (err error) { var _p0 *byte _p0, err = BytePtrFromString(path) diff --git a/vendor/golang.org/x/sys/unix/ztypes_freebsd_386.go b/vendor/golang.org/x/sys/unix/ztypes_freebsd_386.go index d9c78cdc..29dc4833 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_freebsd_386.go +++ b/vendor/golang.org/x/sys/unix/ztypes_freebsd_386.go @@ -362,7 +362,7 @@ type FpExtendedPrecision struct{} type PtraceIoDesc struct { Op int32 Offs uintptr - Addr uintptr + Addr *byte Len uint32 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_freebsd_amd64.go b/vendor/golang.org/x/sys/unix/ztypes_freebsd_amd64.go index 26991b16..0a89b289 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_freebsd_amd64.go +++ b/vendor/golang.org/x/sys/unix/ztypes_freebsd_amd64.go @@ -367,7 +367,7 @@ type FpExtendedPrecision struct{} type PtraceIoDesc struct { Op int32 Offs uintptr - Addr uintptr + Addr *byte Len uint64 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_freebsd_arm.go b/vendor/golang.org/x/sys/unix/ztypes_freebsd_arm.go index f8324e7e..c8666bb1 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_freebsd_arm.go +++ b/vendor/golang.org/x/sys/unix/ztypes_freebsd_arm.go @@ -350,7 +350,7 @@ type FpExtendedPrecision struct { type PtraceIoDesc struct { Op int32 Offs uintptr - Addr uintptr + Addr *byte Len uint32 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_freebsd_arm64.go b/vendor/golang.org/x/sys/unix/ztypes_freebsd_arm64.go index 4220411f..88fb48a8 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_freebsd_arm64.go +++ b/vendor/golang.org/x/sys/unix/ztypes_freebsd_arm64.go @@ -347,7 +347,7 @@ type FpExtendedPrecision struct{} type PtraceIoDesc struct { Op int32 Offs uintptr - Addr uintptr + Addr *byte Len uint64 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_freebsd_riscv64.go b/vendor/golang.org/x/sys/unix/ztypes_freebsd_riscv64.go index 0660fd45..698dc975 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_freebsd_riscv64.go +++ b/vendor/golang.org/x/sys/unix/ztypes_freebsd_riscv64.go @@ -348,7 +348,7 @@ type FpExtendedPrecision struct{} type PtraceIoDesc struct { Op int32 Offs uintptr - Addr uintptr + Addr *byte Len uint64 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_linux.go b/vendor/golang.org/x/sys/unix/ztypes_linux.go index 7d9fc8f1..ca84727c 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_linux.go +++ b/vendor/golang.org/x/sys/unix/ztypes_linux.go @@ -456,36 +456,60 @@ type Ucred struct { } type TCPInfo struct { - State uint8 - Ca_state uint8 - Retransmits uint8 - Probes uint8 - Backoff uint8 - Options uint8 - Rto uint32 - Ato uint32 - Snd_mss uint32 - Rcv_mss uint32 - Unacked uint32 - Sacked uint32 - Lost uint32 - Retrans uint32 - Fackets uint32 - Last_data_sent uint32 - Last_ack_sent uint32 - Last_data_recv uint32 - Last_ack_recv uint32 - Pmtu uint32 - Rcv_ssthresh uint32 - Rtt uint32 - Rttvar uint32 - Snd_ssthresh uint32 - Snd_cwnd uint32 - Advmss uint32 - Reordering uint32 - Rcv_rtt uint32 - Rcv_space uint32 - Total_retrans uint32 + State uint8 + Ca_state uint8 + Retransmits uint8 + Probes uint8 + Backoff uint8 + Options uint8 + Rto uint32 + Ato uint32 + Snd_mss uint32 + Rcv_mss uint32 + Unacked uint32 + Sacked uint32 + Lost uint32 + Retrans uint32 + Fackets uint32 + Last_data_sent uint32 + Last_ack_sent uint32 + Last_data_recv uint32 + Last_ack_recv uint32 + Pmtu uint32 + Rcv_ssthresh uint32 + Rtt uint32 + Rttvar uint32 + Snd_ssthresh uint32 + Snd_cwnd uint32 + Advmss uint32 + Reordering uint32 + Rcv_rtt uint32 + Rcv_space uint32 + Total_retrans uint32 + Pacing_rate uint64 + Max_pacing_rate uint64 + Bytes_acked uint64 + Bytes_received uint64 + Segs_out uint32 + Segs_in uint32 + Notsent_bytes uint32 + Min_rtt uint32 + Data_segs_in uint32 + Data_segs_out uint32 + Delivery_rate uint64 + Busy_time uint64 + Rwnd_limited uint64 + Sndbuf_limited uint64 + Delivered uint32 + Delivered_ce uint32 + Bytes_sent uint64 + Bytes_retrans uint64 + Dsack_dups uint32 + Reord_seen uint32 + Rcv_ooopack uint32 + Snd_wnd uint32 + Rcv_wnd uint32 + Rehash uint32 } type CanFilter struct { @@ -528,7 +552,7 @@ const ( SizeofIPv6MTUInfo = 0x20 SizeofICMPv6Filter = 0x20 SizeofUcred = 0xc - SizeofTCPInfo = 0x68 + SizeofTCPInfo = 0xf0 SizeofCanFilter = 0x8 SizeofTCPRepairOpt = 0x8 ) @@ -1043,6 +1067,7 @@ const ( PerfBitCommExec = CBitFieldMaskBit24 PerfBitUseClockID = CBitFieldMaskBit25 PerfBitContextSwitch = CBitFieldMaskBit26 + PerfBitWriteBackward = CBitFieldMaskBit27 ) const ( @@ -1239,7 +1264,7 @@ type TCPMD5Sig struct { Flags uint8 Prefixlen uint8 Keylen uint16 - _ uint32 + Ifindex int32 Key [80]uint8 } @@ -1939,7 +1964,11 @@ const ( NFT_MSG_GETOBJ = 0x13 NFT_MSG_DELOBJ = 0x14 NFT_MSG_GETOBJ_RESET = 0x15 - NFT_MSG_MAX = 0x19 + NFT_MSG_NEWFLOWTABLE = 0x16 + NFT_MSG_GETFLOWTABLE = 0x17 + NFT_MSG_DELFLOWTABLE = 0x18 + NFT_MSG_GETRULE_RESET = 0x19 + NFT_MSG_MAX = 0x1a NFTA_LIST_UNSPEC = 0x0 NFTA_LIST_ELEM = 0x1 NFTA_HOOK_UNSPEC = 0x0 @@ -2443,9 +2472,11 @@ const ( SOF_TIMESTAMPING_OPT_STATS = 0x1000 SOF_TIMESTAMPING_OPT_PKTINFO = 0x2000 SOF_TIMESTAMPING_OPT_TX_SWHW = 0x4000 + SOF_TIMESTAMPING_BIND_PHC = 0x8000 + SOF_TIMESTAMPING_OPT_ID_TCP = 0x10000 - SOF_TIMESTAMPING_LAST = 0x8000 - SOF_TIMESTAMPING_MASK = 0xffff + SOF_TIMESTAMPING_LAST = 0x10000 + SOF_TIMESTAMPING_MASK = 0x1ffff SCM_TSTAMP_SND = 0x0 SCM_TSTAMP_SCHED = 0x1 @@ -3265,7 +3296,7 @@ const ( DEVLINK_ATTR_LINECARD_SUPPORTED_TYPES = 0xae DEVLINK_ATTR_NESTED_DEVLINK = 0xaf DEVLINK_ATTR_SELFTESTS = 0xb0 - DEVLINK_ATTR_MAX = 0xb0 + DEVLINK_ATTR_MAX = 0xb3 DEVLINK_DPIPE_FIELD_MAPPING_TYPE_NONE = 0x0 DEVLINK_DPIPE_FIELD_MAPPING_TYPE_IFINDEX = 0x1 DEVLINK_DPIPE_MATCH_TYPE_FIELD_EXACT = 0x0 @@ -3281,7 +3312,8 @@ const ( DEVLINK_PORT_FUNCTION_ATTR_HW_ADDR = 0x1 DEVLINK_PORT_FN_ATTR_STATE = 0x2 DEVLINK_PORT_FN_ATTR_OPSTATE = 0x3 - DEVLINK_PORT_FUNCTION_ATTR_MAX = 0x3 + DEVLINK_PORT_FN_ATTR_CAPS = 0x4 + DEVLINK_PORT_FUNCTION_ATTR_MAX = 0x4 ) type FsverityDigest struct { @@ -3572,7 +3604,8 @@ const ( ETHTOOL_MSG_MODULE_SET = 0x23 ETHTOOL_MSG_PSE_GET = 0x24 ETHTOOL_MSG_PSE_SET = 0x25 - ETHTOOL_MSG_USER_MAX = 0x25 + ETHTOOL_MSG_RSS_GET = 0x26 + ETHTOOL_MSG_USER_MAX = 0x26 ETHTOOL_MSG_KERNEL_NONE = 0x0 ETHTOOL_MSG_STRSET_GET_REPLY = 0x1 ETHTOOL_MSG_LINKINFO_GET_REPLY = 0x2 @@ -3611,7 +3644,8 @@ const ( ETHTOOL_MSG_MODULE_GET_REPLY = 0x23 ETHTOOL_MSG_MODULE_NTF = 0x24 ETHTOOL_MSG_PSE_GET_REPLY = 0x25 - ETHTOOL_MSG_KERNEL_MAX = 0x25 + ETHTOOL_MSG_RSS_GET_REPLY = 0x26 + ETHTOOL_MSG_KERNEL_MAX = 0x26 ETHTOOL_A_HEADER_UNSPEC = 0x0 ETHTOOL_A_HEADER_DEV_INDEX = 0x1 ETHTOOL_A_HEADER_DEV_NAME = 0x2 @@ -3679,7 +3713,8 @@ const ( ETHTOOL_A_LINKSTATE_SQI_MAX = 0x4 ETHTOOL_A_LINKSTATE_EXT_STATE = 0x5 ETHTOOL_A_LINKSTATE_EXT_SUBSTATE = 0x6 - ETHTOOL_A_LINKSTATE_MAX = 0x6 + ETHTOOL_A_LINKSTATE_EXT_DOWN_CNT = 0x7 + ETHTOOL_A_LINKSTATE_MAX = 0x7 ETHTOOL_A_DEBUG_UNSPEC = 0x0 ETHTOOL_A_DEBUG_HEADER = 0x1 ETHTOOL_A_DEBUG_MSGMASK = 0x2 @@ -4409,7 +4444,7 @@ const ( NL80211_ATTR_MAC_HINT = 0xc8 NL80211_ATTR_MAC_MASK = 0xd7 NL80211_ATTR_MAX_AP_ASSOC_STA = 0xca - NL80211_ATTR_MAX = 0x140 + NL80211_ATTR_MAX = 0x141 NL80211_ATTR_MAX_CRIT_PROT_DURATION = 0xb4 NL80211_ATTR_MAX_CSA_COUNTERS = 0xce NL80211_ATTR_MAX_MATCH_SETS = 0x85 @@ -4552,6 +4587,7 @@ const ( NL80211_ATTR_SUPPORT_MESH_AUTH = 0x73 NL80211_ATTR_SURVEY_INFO = 0x54 NL80211_ATTR_SURVEY_RADIO_STATS = 0xda + NL80211_ATTR_TD_BITMAP = 0x141 NL80211_ATTR_TDLS_ACTION = 0x88 NL80211_ATTR_TDLS_DIALOG_TOKEN = 0x89 NL80211_ATTR_TDLS_EXTERNAL_SETUP = 0x8c @@ -5752,3 +5788,25 @@ const ( AUDIT_NLGRP_NONE = 0x0 AUDIT_NLGRP_READLOG = 0x1 ) + +const ( + TUN_F_CSUM = 0x1 + TUN_F_TSO4 = 0x2 + TUN_F_TSO6 = 0x4 + TUN_F_TSO_ECN = 0x8 + TUN_F_UFO = 0x10 +) + +const ( + VIRTIO_NET_HDR_F_NEEDS_CSUM = 0x1 + VIRTIO_NET_HDR_F_DATA_VALID = 0x2 + VIRTIO_NET_HDR_F_RSC_INFO = 0x4 +) + +const ( + VIRTIO_NET_HDR_GSO_NONE = 0x0 + VIRTIO_NET_HDR_GSO_TCPV4 = 0x1 + VIRTIO_NET_HDR_GSO_UDP = 0x3 + VIRTIO_NET_HDR_GSO_TCPV6 = 0x4 + VIRTIO_NET_HDR_GSO_ECN = 0x80 +) diff --git a/vendor/golang.org/x/sys/unix/ztypes_linux_386.go b/vendor/golang.org/x/sys/unix/ztypes_linux_386.go index 89c516a2..4ecc1495 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_linux_386.go +++ b/vendor/golang.org/x/sys/unix/ztypes_linux_386.go @@ -414,7 +414,7 @@ const ( type SockaddrStorage struct { Family uint16 - _ [122]int8 + Data [122]byte _ uint32 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_linux_amd64.go b/vendor/golang.org/x/sys/unix/ztypes_linux_amd64.go index 62b4fb26..34fddff9 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_linux_amd64.go +++ b/vendor/golang.org/x/sys/unix/ztypes_linux_amd64.go @@ -427,7 +427,7 @@ const ( type SockaddrStorage struct { Family uint16 - _ [118]int8 + Data [118]byte _ uint64 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_linux_arm.go b/vendor/golang.org/x/sys/unix/ztypes_linux_arm.go index e86b3589..3b14a603 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_linux_arm.go +++ b/vendor/golang.org/x/sys/unix/ztypes_linux_arm.go @@ -405,7 +405,7 @@ const ( type SockaddrStorage struct { Family uint16 - _ [122]uint8 + Data [122]byte _ uint32 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_linux_arm64.go b/vendor/golang.org/x/sys/unix/ztypes_linux_arm64.go index 6c6be4c9..0517651a 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_linux_arm64.go +++ b/vendor/golang.org/x/sys/unix/ztypes_linux_arm64.go @@ -406,7 +406,7 @@ const ( type SockaddrStorage struct { Family uint16 - _ [118]int8 + Data [118]byte _ uint64 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_linux_loong64.go b/vendor/golang.org/x/sys/unix/ztypes_linux_loong64.go index 4982ea35..3b0c5181 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_linux_loong64.go +++ b/vendor/golang.org/x/sys/unix/ztypes_linux_loong64.go @@ -407,7 +407,7 @@ const ( type SockaddrStorage struct { Family uint16 - _ [118]int8 + Data [118]byte _ uint64 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_linux_mips.go b/vendor/golang.org/x/sys/unix/ztypes_linux_mips.go index 173141a6..fccdf4dd 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_linux_mips.go +++ b/vendor/golang.org/x/sys/unix/ztypes_linux_mips.go @@ -410,7 +410,7 @@ const ( type SockaddrStorage struct { Family uint16 - _ [122]int8 + Data [122]byte _ uint32 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_linux_mips64.go b/vendor/golang.org/x/sys/unix/ztypes_linux_mips64.go index 93ae4c51..500de8fc 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_linux_mips64.go +++ b/vendor/golang.org/x/sys/unix/ztypes_linux_mips64.go @@ -409,7 +409,7 @@ const ( type SockaddrStorage struct { Family uint16 - _ [118]int8 + Data [118]byte _ uint64 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_linux_mips64le.go b/vendor/golang.org/x/sys/unix/ztypes_linux_mips64le.go index 4e4e510c..d0434cd2 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_linux_mips64le.go +++ b/vendor/golang.org/x/sys/unix/ztypes_linux_mips64le.go @@ -409,7 +409,7 @@ const ( type SockaddrStorage struct { Family uint16 - _ [118]int8 + Data [118]byte _ uint64 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_linux_mipsle.go b/vendor/golang.org/x/sys/unix/ztypes_linux_mipsle.go index 3f5ba013..84206ba5 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_linux_mipsle.go +++ b/vendor/golang.org/x/sys/unix/ztypes_linux_mipsle.go @@ -410,7 +410,7 @@ const ( type SockaddrStorage struct { Family uint16 - _ [122]int8 + Data [122]byte _ uint32 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_linux_ppc.go b/vendor/golang.org/x/sys/unix/ztypes_linux_ppc.go index 71dfe7cd..ab078cf1 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_linux_ppc.go +++ b/vendor/golang.org/x/sys/unix/ztypes_linux_ppc.go @@ -417,7 +417,7 @@ const ( type SockaddrStorage struct { Family uint16 - _ [122]uint8 + Data [122]byte _ uint32 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_linux_ppc64.go b/vendor/golang.org/x/sys/unix/ztypes_linux_ppc64.go index 3a2b7f0a..42eb2c4c 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_linux_ppc64.go +++ b/vendor/golang.org/x/sys/unix/ztypes_linux_ppc64.go @@ -416,7 +416,7 @@ const ( type SockaddrStorage struct { Family uint16 - _ [118]uint8 + Data [118]byte _ uint64 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_linux_ppc64le.go b/vendor/golang.org/x/sys/unix/ztypes_linux_ppc64le.go index a52d6275..31304a4e 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_linux_ppc64le.go +++ b/vendor/golang.org/x/sys/unix/ztypes_linux_ppc64le.go @@ -416,7 +416,7 @@ const ( type SockaddrStorage struct { Family uint16 - _ [118]uint8 + Data [118]byte _ uint64 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_linux_riscv64.go b/vendor/golang.org/x/sys/unix/ztypes_linux_riscv64.go index dfc007d8..c311f961 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_linux_riscv64.go +++ b/vendor/golang.org/x/sys/unix/ztypes_linux_riscv64.go @@ -434,7 +434,7 @@ const ( type SockaddrStorage struct { Family uint16 - _ [118]uint8 + Data [118]byte _ uint64 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_linux_s390x.go b/vendor/golang.org/x/sys/unix/ztypes_linux_s390x.go index b53cb910..bba3cefa 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_linux_s390x.go +++ b/vendor/golang.org/x/sys/unix/ztypes_linux_s390x.go @@ -429,7 +429,7 @@ const ( type SockaddrStorage struct { Family uint16 - _ [118]int8 + Data [118]byte _ uint64 } diff --git a/vendor/golang.org/x/sys/unix/ztypes_linux_sparc64.go b/vendor/golang.org/x/sys/unix/ztypes_linux_sparc64.go index fe0aa354..ad8a0138 100644 --- a/vendor/golang.org/x/sys/unix/ztypes_linux_sparc64.go +++ b/vendor/golang.org/x/sys/unix/ztypes_linux_sparc64.go @@ -411,7 +411,7 @@ const ( type SockaddrStorage struct { Family uint16 - _ [118]int8 + Data [118]byte _ uint64 } diff --git a/vendor/golang.org/x/sys/windows/syscall_windows.go b/vendor/golang.org/x/sys/windows/syscall_windows.go index 41cb3c01..3723b2c2 100644 --- a/vendor/golang.org/x/sys/windows/syscall_windows.go +++ b/vendor/golang.org/x/sys/windows/syscall_windows.go @@ -824,6 +824,9 @@ const socket_error = uintptr(^uint32(0)) //sys WSAStartup(verreq uint32, data *WSAData) (sockerr error) = ws2_32.WSAStartup //sys WSACleanup() (err error) [failretval==socket_error] = ws2_32.WSACleanup //sys WSAIoctl(s Handle, iocc uint32, inbuf *byte, cbif uint32, outbuf *byte, cbob uint32, cbbr *uint32, overlapped *Overlapped, completionRoutine uintptr) (err error) [failretval==socket_error] = ws2_32.WSAIoctl +//sys WSALookupServiceBegin(querySet *WSAQUERYSET, flags uint32, handle *Handle) (err error) [failretval==socket_error] = ws2_32.WSALookupServiceBeginW +//sys WSALookupServiceNext(handle Handle, flags uint32, size *int32, querySet *WSAQUERYSET) (err error) [failretval==socket_error] = ws2_32.WSALookupServiceNextW +//sys WSALookupServiceEnd(handle Handle) (err error) [failretval==socket_error] = ws2_32.WSALookupServiceEnd //sys socket(af int32, typ int32, protocol int32) (handle Handle, err error) [failretval==InvalidHandle] = ws2_32.socket //sys sendto(s Handle, buf []byte, flags int32, to unsafe.Pointer, tolen int32) (err error) [failretval==socket_error] = ws2_32.sendto //sys recvfrom(s Handle, buf []byte, flags int32, from *RawSockaddrAny, fromlen *int32) (n int32, err error) [failretval==-1] = ws2_32.recvfrom @@ -1019,8 +1022,7 @@ func (rsa *RawSockaddrAny) Sockaddr() (Sockaddr, error) { for n < len(pp.Path) && pp.Path[n] != 0 { n++ } - bytes := (*[len(pp.Path)]byte)(unsafe.Pointer(&pp.Path[0]))[0:n] - sa.Name = string(bytes) + sa.Name = string(unsafe.Slice((*byte)(unsafe.Pointer(&pp.Path[0])), n)) return sa, nil case AF_INET: diff --git a/vendor/golang.org/x/sys/windows/types_windows.go b/vendor/golang.org/x/sys/windows/types_windows.go index 0c4add97..857acf10 100644 --- a/vendor/golang.org/x/sys/windows/types_windows.go +++ b/vendor/golang.org/x/sys/windows/types_windows.go @@ -1243,6 +1243,51 @@ const ( DnsSectionAdditional = 0x0003 ) +const ( + // flags of WSALookupService + LUP_DEEP = 0x0001 + LUP_CONTAINERS = 0x0002 + LUP_NOCONTAINERS = 0x0004 + LUP_NEAREST = 0x0008 + LUP_RETURN_NAME = 0x0010 + LUP_RETURN_TYPE = 0x0020 + LUP_RETURN_VERSION = 0x0040 + LUP_RETURN_COMMENT = 0x0080 + LUP_RETURN_ADDR = 0x0100 + LUP_RETURN_BLOB = 0x0200 + LUP_RETURN_ALIASES = 0x0400 + LUP_RETURN_QUERY_STRING = 0x0800 + LUP_RETURN_ALL = 0x0FF0 + LUP_RES_SERVICE = 0x8000 + + LUP_FLUSHCACHE = 0x1000 + LUP_FLUSHPREVIOUS = 0x2000 + + LUP_NON_AUTHORITATIVE = 0x4000 + LUP_SECURE = 0x8000 + LUP_RETURN_PREFERRED_NAMES = 0x10000 + LUP_DNS_ONLY = 0x20000 + + LUP_ADDRCONFIG = 0x100000 + LUP_DUAL_ADDR = 0x200000 + LUP_FILESERVER = 0x400000 + LUP_DISABLE_IDN_ENCODING = 0x00800000 + LUP_API_ANSI = 0x01000000 + + LUP_RESOLUTION_HANDLE = 0x80000000 +) + +const ( + // values of WSAQUERYSET's namespace + NS_ALL = 0 + NS_DNS = 12 + NS_NLA = 15 + NS_BTH = 16 + NS_EMAIL = 37 + NS_PNRPNAME = 38 + NS_PNRPCLOUD = 39 +) + type DNSSRVData struct { Target *uint16 Priority uint16 @@ -3258,3 +3303,43 @@ const ( DWMWA_TEXT_COLOR = 36 DWMWA_VISIBLE_FRAME_BORDER_THICKNESS = 37 ) + +type WSAQUERYSET struct { + Size uint32 + ServiceInstanceName *uint16 + ServiceClassId *GUID + Version *WSAVersion + Comment *uint16 + NameSpace uint32 + NSProviderId *GUID + Context *uint16 + NumberOfProtocols uint32 + AfpProtocols *AFProtocols + QueryString *uint16 + NumberOfCsAddrs uint32 + SaBuffer *CSAddrInfo + OutputFlags uint32 + Blob *BLOB +} + +type WSAVersion struct { + Version uint32 + EnumerationOfComparison int32 +} + +type AFProtocols struct { + AddressFamily int32 + Protocol int32 +} + +type CSAddrInfo struct { + LocalAddr SocketAddress + RemoteAddr SocketAddress + SocketType int32 + Protocol int32 +} + +type BLOB struct { + Size uint32 + BlobData *byte +} diff --git a/vendor/golang.org/x/sys/windows/zsyscall_windows.go b/vendor/golang.org/x/sys/windows/zsyscall_windows.go index ac60052e..6d2a2685 100644 --- a/vendor/golang.org/x/sys/windows/zsyscall_windows.go +++ b/vendor/golang.org/x/sys/windows/zsyscall_windows.go @@ -474,6 +474,9 @@ var ( procWSAEnumProtocolsW = modws2_32.NewProc("WSAEnumProtocolsW") procWSAGetOverlappedResult = modws2_32.NewProc("WSAGetOverlappedResult") procWSAIoctl = modws2_32.NewProc("WSAIoctl") + procWSALookupServiceBeginW = modws2_32.NewProc("WSALookupServiceBeginW") + procWSALookupServiceEnd = modws2_32.NewProc("WSALookupServiceEnd") + procWSALookupServiceNextW = modws2_32.NewProc("WSALookupServiceNextW") procWSARecv = modws2_32.NewProc("WSARecv") procWSARecvFrom = modws2_32.NewProc("WSARecvFrom") procWSASend = modws2_32.NewProc("WSASend") @@ -4067,6 +4070,30 @@ func WSAIoctl(s Handle, iocc uint32, inbuf *byte, cbif uint32, outbuf *byte, cbo return } +func WSALookupServiceBegin(querySet *WSAQUERYSET, flags uint32, handle *Handle) (err error) { + r1, _, e1 := syscall.Syscall(procWSALookupServiceBeginW.Addr(), 3, uintptr(unsafe.Pointer(querySet)), uintptr(flags), uintptr(unsafe.Pointer(handle))) + if r1 == socket_error { + err = errnoErr(e1) + } + return +} + +func WSALookupServiceEnd(handle Handle) (err error) { + r1, _, e1 := syscall.Syscall(procWSALookupServiceEnd.Addr(), 1, uintptr(handle), 0, 0) + if r1 == socket_error { + err = errnoErr(e1) + } + return +} + +func WSALookupServiceNext(handle Handle, flags uint32, size *int32, querySet *WSAQUERYSET) (err error) { + r1, _, e1 := syscall.Syscall6(procWSALookupServiceNextW.Addr(), 4, uintptr(handle), uintptr(flags), uintptr(unsafe.Pointer(size)), uintptr(unsafe.Pointer(querySet)), 0, 0) + if r1 == socket_error { + err = errnoErr(e1) + } + return +} + func WSARecv(s Handle, bufs *WSABuf, bufcnt uint32, recvd *uint32, flags *uint32, overlapped *Overlapped, croutine *byte) (err error) { r1, _, e1 := syscall.Syscall9(procWSARecv.Addr(), 7, uintptr(s), uintptr(unsafe.Pointer(bufs)), uintptr(bufcnt), uintptr(unsafe.Pointer(recvd)), uintptr(unsafe.Pointer(flags)), uintptr(unsafe.Pointer(overlapped)), uintptr(unsafe.Pointer(croutine)), 0, 0) if r1 == socket_error { diff --git a/vendor/modules.txt b/vendor/modules.txt index c6b3fa37..ae5f8026 100644 --- a/vendor/modules.txt +++ b/vendor/modules.txt @@ -275,7 +275,7 @@ github.com/rs/xid # github.com/russross/blackfriday/v2 v2.1.0 ## explicit github.com/russross/blackfriday/v2 -# github.com/shirou/gopsutil/v3 v3.22.11 +# github.com/shirou/gopsutil/v3 v3.23.3 ## explicit; go 1.15 github.com/shirou/gopsutil/v3/cpu github.com/shirou/gopsutil/v3/disk @@ -283,10 +283,13 @@ github.com/shirou/gopsutil/v3/internal/common github.com/shirou/gopsutil/v3/mem github.com/shirou/gopsutil/v3/net github.com/shirou/gopsutil/v3/process +# github.com/shoenig/go-m1cpu v0.1.4 +## explicit; go 1.20 +github.com/shoenig/go-m1cpu # github.com/sirupsen/logrus v1.9.0 ## explicit; go 1.13 github.com/sirupsen/logrus -# github.com/stretchr/testify v1.8.1 +# github.com/stretchr/testify v1.8.2 ## explicit; go 1.13 github.com/stretchr/testify/assert github.com/stretchr/testify/require @@ -384,7 +387,7 @@ golang.org/x/net/ipv6 golang.org/x/net/publicsuffix golang.org/x/net/webdav golang.org/x/net/webdav/internal/xml -# golang.org/x/sys v0.5.0 +# golang.org/x/sys v0.6.0 ## explicit; go 1.17 golang.org/x/sys/cpu golang.org/x/sys/execabs From a1682b7aa406633b286d2000fae73c905a69bd13 Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Thu, 13 Apr 2023 12:19:20 +0200 Subject: [PATCH 07/26] Fix parsing S3 storage definition from environment variable --- config/value/s3.go | 64 +++++++++++++++++++---------------------- config/value/s3_test.go | 53 ++++++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+), 34 deletions(-) create mode 100644 config/value/s3_test.go diff --git a/config/value/s3.go b/config/value/s3.go index a85a0838..7d80c193 100644 --- a/config/value/s3.go +++ b/config/value/s3.go @@ -3,28 +3,29 @@ package value import ( "fmt" "net/url" + "regexp" "strings" - - "golang.org/x/net/publicsuffix" ) // array of s3 storages // https://access_key_id:secret_access_id@region.endpoint/bucket?name=aaa&mount=/abc&username=xxx&password=yyy type S3Storage struct { - Name string `json:"name"` - Mountpoint string `json:"mountpoint"` - Auth struct { - Enable bool `json:"enable"` - Username string `json:"username"` - Password string `json:"password"` - } `json:"auth"` - Endpoint string `json:"endpoint"` - AccessKeyID string `json:"access_key_id"` - SecretAccessKey string `json:"secret_access_key"` - Bucket string `json:"bucket"` - Region string `json:"region"` - UseSSL bool `json:"use_ssl"` + Name string `json:"name"` + Mountpoint string `json:"mountpoint"` + Auth S3StorageAuth `json:"auth"` + Endpoint string `json:"endpoint"` + AccessKeyID string `json:"access_key_id"` + SecretAccessKey string `json:"secret_access_key"` + Bucket string `json:"bucket"` + Region string `json:"region"` + UseSSL bool `json:"use_ssl"` +} + +type S3StorageAuth struct { + Enable bool `json:"enable"` + Username string `json:"username"` + Password string `json:"password"` } func (t *S3Storage) String() string { @@ -50,7 +51,7 @@ func (t *S3Storage) String() string { v := url.Values{} v.Set("name", t.Name) - v.Set("mountpoint", t.Mountpoint) + v.Set("mount", t.Mountpoint) if t.Auth.Enable { if len(t.Auth.Username) != 0 { @@ -70,12 +71,14 @@ func (t *S3Storage) String() string { type s3StorageListValue struct { p *[]S3Storage separator string + reName *regexp.Regexp } func NewS3StorageListValue(p *[]S3Storage, val []S3Storage, separator string) *s3StorageListValue { v := &s3StorageListValue{ p: p, separator: separator, + reName: regexp.MustCompile(`^[A-Za-z0-9_-]+$`), } *p = val @@ -93,27 +96,16 @@ func (s *s3StorageListValue) Set(val string) error { t := S3Storage{ Name: u.Query().Get("name"), - Mountpoint: u.Query().Get("mountpoint"), + Mountpoint: u.Query().Get("mount"), AccessKeyID: u.User.Username(), } - hostname := u.Hostname() - port := u.Port() + password, _ := u.User.Password() + t.SecretAccessKey = password - domain, err := publicsuffix.EffectiveTLDPlusOne(hostname) - if err != nil { - return fmt.Errorf("invalid eTLD (%s): %w", hostname, err) - } - - t.Endpoint = domain - if len(port) != 0 { - t.Endpoint += ":" + port - } - - region := strings.TrimSuffix(hostname, domain) - if len(region) != 0 { - t.Region = strings.TrimSuffix(region, ".") - } + region, endpoint, _ := strings.Cut(u.Host, ".") + t.Endpoint = endpoint + t.Region = region secret, ok := u.User.Password() if ok { @@ -129,7 +121,7 @@ func (s *s3StorageListValue) Set(val string) error { if u.Query().Has("username") || u.Query().Has("password") { t.Auth.Enable = true t.Auth.Username = u.Query().Get("username") - t.Auth.Username = u.Query().Get("password") + t.Auth.Password = u.Query().Get("password") } list = append(list, t) @@ -160,6 +152,10 @@ func (s *s3StorageListValue) Validate() error { return fmt.Errorf("the name for s3 storage %d is missing", i) } + if !s.reName.MatchString(t.Name) { + return fmt.Errorf("the name for s3 storage must match the pattern %s", s.reName.String()) + } + if len(t.Mountpoint) == 0 { return fmt.Errorf("the mountpoint for s3 storage %d is missing", i) } diff --git a/config/value/s3_test.go b/config/value/s3_test.go new file mode 100644 index 00000000..79cbee9d --- /dev/null +++ b/config/value/s3_test.go @@ -0,0 +1,53 @@ +package value + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestS3Value(t *testing.T) { + filesystems := []S3Storage{} + + v := NewS3StorageListValue(&filesystems, nil, " ") + require.Equal(t, "(empty)", v.String()) + + v.Set("https://access_key_id1:secret_access_id1@region1.subdomain.endpoint1.com/bucket1?name=aaa1&mount=/abc1&username=xxx1&password=yyy1 http://access_key_id2:secret_access_id2@region2.endpoint2.com/bucket2?name=aaa2&mount=/abc2&username=xxx2&password=yyy2") + require.Equal(t, []S3Storage{ + { + Name: "aaa1", + Mountpoint: "/abc1", + Auth: S3StorageAuth{ + Enable: true, + Username: "xxx1", + Password: "yyy1", + }, + Endpoint: "subdomain.endpoint1.com", + AccessKeyID: "access_key_id1", + SecretAccessKey: "secret_access_id1", + Bucket: "bucket1", + Region: "region1", + UseSSL: true, + }, + { + Name: "aaa2", + Mountpoint: "/abc2", + Auth: S3StorageAuth{ + Enable: true, + Username: "xxx2", + Password: "yyy2", + }, + Endpoint: "endpoint2.com", + AccessKeyID: "access_key_id2", + SecretAccessKey: "secret_access_id2", + Bucket: "bucket2", + Region: "region2", + UseSSL: false, + }, + }, filesystems) + require.Equal(t, "https://access_key_id1:---@region1.subdomain.endpoint1.com/bucket1?mount=%2Fabc1&name=aaa1&password=---&username=xxx1 http://access_key_id2:---@region2.endpoint2.com/bucket2?mount=%2Fabc2&name=aaa2&password=---&username=xxx2", v.String()) + require.NoError(t, v.Validate()) + + v.Set("https://access_key_id1:secret_access_id1@region1.endpoint1.com/bucket1?name=djk*;..&mount=/abc1&username=xxx1&password=yyy1") + require.Error(t, v.Validate()) +} From d807becc8a50873ec0407a3125888092b6702c4f Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Thu, 13 Apr 2023 15:22:33 +0200 Subject: [PATCH 08/26] Add support for input framerate data from jsonstats patch --- docs/docs.go | 41 ++++++++++++++++++++++++++++------------ docs/swagger.json | 41 ++++++++++++++++++++++++++++------------ docs/swagger.yaml | 27 ++++++++++++++++++-------- ffmpeg/parse/types.go | 18 +++++++++++++----- http/api/progress.go | 38 +++++++++++++++++++++++-------------- restream/app/progress.go | 9 +++++++-- 6 files changed, 121 insertions(+), 53 deletions(-) diff --git a/docs/docs.go b/docs/docs.go index 5f51006e..41e9394e 100644 --- a/docs/docs.go +++ b/docs/docs.go @@ -3357,6 +3357,20 @@ const docTemplate = `{ "type": "integer", "format": "uint64" }, + "framerate": { + "type": "object", + "properties": { + "avg": { + "type": "number" + }, + "max": { + "type": "number" + }, + "min": { + "type": "number" + } + } + }, "height": { "type": "integer", "format": "uint64" @@ -4734,18 +4748,7 @@ const docTemplate = `{ "type": "string" }, "auth": { - "type": "object", - "properties": { - "enable": { - "type": "boolean" - }, - "password": { - "type": "string" - }, - "username": { - "type": "string" - } - } + "$ref": "#/definitions/value.S3StorageAuth" }, "bucket": { "type": "string" @@ -4769,6 +4772,20 @@ const docTemplate = `{ "type": "boolean" } } + }, + "value.S3StorageAuth": { + "type": "object", + "properties": { + "enable": { + "type": "boolean" + }, + "password": { + "type": "string" + }, + "username": { + "type": "string" + } + } } }, "securityDefinitions": { diff --git a/docs/swagger.json b/docs/swagger.json index 426d1075..2956a6ec 100644 --- a/docs/swagger.json +++ b/docs/swagger.json @@ -3350,6 +3350,20 @@ "type": "integer", "format": "uint64" }, + "framerate": { + "type": "object", + "properties": { + "avg": { + "type": "number" + }, + "max": { + "type": "number" + }, + "min": { + "type": "number" + } + } + }, "height": { "type": "integer", "format": "uint64" @@ -4727,18 +4741,7 @@ "type": "string" }, "auth": { - "type": "object", - "properties": { - "enable": { - "type": "boolean" - }, - "password": { - "type": "string" - }, - "username": { - "type": "string" - } - } + "$ref": "#/definitions/value.S3StorageAuth" }, "bucket": { "type": "string" @@ -4762,6 +4765,20 @@ "type": "boolean" } } + }, + "value.S3StorageAuth": { + "type": "object", + "properties": { + "enable": { + "type": "boolean" + }, + "password": { + "type": "string" + }, + "username": { + "type": "string" + } + } } }, "securityDefinitions": { diff --git a/docs/swagger.yaml b/docs/swagger.yaml index 10ddcaaa..2b35b2bd 100644 --- a/docs/swagger.yaml +++ b/docs/swagger.yaml @@ -913,6 +913,15 @@ definitions: frame: format: uint64 type: integer + framerate: + properties: + avg: + type: number + max: + type: number + min: + type: number + type: object height: format: uint64 type: integer @@ -1898,14 +1907,7 @@ definitions: access_key_id: type: string auth: - properties: - enable: - type: boolean - password: - type: string - username: - type: string - type: object + $ref: '#/definitions/value.S3StorageAuth' bucket: type: string endpoint: @@ -1921,6 +1923,15 @@ definitions: use_ssl: type: boolean type: object + value.S3StorageAuth: + properties: + enable: + type: boolean + password: + type: string + username: + type: string + type: object info: contact: email: hello@datarhei.com diff --git a/ffmpeg/parse/types.go b/ffmpeg/parse/types.go index aa0c33e3..1f829b5e 100644 --- a/ffmpeg/parse/types.go +++ b/ffmpeg/parse/types.go @@ -93,11 +93,16 @@ type ffmpegProgressIO struct { // common Index uint64 `json:"index"` Stream uint64 `json:"stream"` - SizeKB uint64 `json:"size_kb"` // kbytes - Size uint64 `json:"size_bytes"` // bytes - Bitrate float64 `json:"-"` // bit/s - Frame uint64 `json:"frame"` // counter - Keyframe uint64 `json:"keyframe"` // counter + SizeKB uint64 `json:"size_kb"` // kbytes + Size uint64 `json:"size_bytes"` // bytes + Bitrate float64 `json:"-"` // bit/s + Frame uint64 `json:"frame"` // counter + Keyframe uint64 `json:"keyframe"` // counter + Framerate struct { + Min float64 `json:"min"` + Max float64 `json:"max"` + Average float64 `json:"avg"` + } `json:"framerate"` Packet uint64 `json:"packet"` // counter Extradata uint64 `json:"extradata_size_bytes"` // bytes FPS float64 `json:"-"` // rate, frames per second @@ -112,6 +117,9 @@ func (io *ffmpegProgressIO) exportTo(progress *app.ProgressIO) { progress.Stream = io.Stream progress.Frame = io.Frame progress.Keyframe = io.Keyframe + progress.Framerate.Min = io.Framerate.Min + progress.Framerate.Max = io.Framerate.Max + progress.Framerate.Average = io.Framerate.Average progress.Packet = io.Packet progress.FPS = io.FPS progress.PPS = io.PPS diff --git a/http/api/progress.go b/http/api/progress.go index 1bf22c59..051eea5b 100644 --- a/http/api/progress.go +++ b/http/api/progress.go @@ -7,26 +7,33 @@ import ( "github.com/datarhei/core/v16/restream/app" ) +type ProgressIOFramerate struct { + Min json.Number `json:"min" swaggertype:"number" jsonschema:"type=number"` + Max json.Number `json:"max" swaggertype:"number" jsonschema:"type=number"` + Average json.Number `json:"avg" swaggertype:"number" jsonschema:"type=number"` +} + // ProgressIO represents the progress of an ffmpeg input or output type ProgressIO struct { ID string `json:"id" jsonschema:"minLength=1"` Address string `json:"address" jsonschema:"minLength=1"` // General - Index uint64 `json:"index" format:"uint64"` - Stream uint64 `json:"stream" format:"uint64"` - Format string `json:"format"` - Type string `json:"type"` - Codec string `json:"codec"` - Coder string `json:"coder"` - Frame uint64 `json:"frame" format:"uint64"` - Keyframe uint64 `json:"keyframe" format:"uint64"` - FPS json.Number `json:"fps" swaggertype:"number" jsonschema:"type=number"` - Packet uint64 `json:"packet" format:"uint64"` - PPS json.Number `json:"pps" swaggertype:"number" jsonschema:"type=number"` - Size uint64 `json:"size_kb" format:"uint64"` // kbytes - Bitrate json.Number `json:"bitrate_kbit" swaggertype:"number" jsonschema:"type=number"` // kbit/s - Extradata uint64 `json:"extradata_size_bytes" format:"uint64"` // bytes + Index uint64 `json:"index" format:"uint64"` + Stream uint64 `json:"stream" format:"uint64"` + Format string `json:"format"` + Type string `json:"type"` + Codec string `json:"codec"` + Coder string `json:"coder"` + Frame uint64 `json:"frame" format:"uint64"` + Keyframe uint64 `json:"keyframe" format:"uint64"` + Framerate ProgressIOFramerate `json:"framerate"` + FPS json.Number `json:"fps" swaggertype:"number" jsonschema:"type=number"` + Packet uint64 `json:"packet" format:"uint64"` + PPS json.Number `json:"pps" swaggertype:"number" jsonschema:"type=number"` + Size uint64 `json:"size_kb" format:"uint64"` // kbytes + Bitrate json.Number `json:"bitrate_kbit" swaggertype:"number" jsonschema:"type=number"` // kbit/s + Extradata uint64 `json:"extradata_size_bytes" format:"uint64"` // bytes // Video Pixfmt string `json:"pix_fmt,omitempty"` @@ -59,6 +66,9 @@ func (i *ProgressIO) Unmarshal(io *app.ProgressIO) { i.Coder = io.Coder i.Frame = io.Frame i.Keyframe = io.Keyframe + i.Framerate.Min = json.Number(fmt.Sprintf("%.3f", io.Framerate.Min)) + i.Framerate.Max = json.Number(fmt.Sprintf("%.3f", io.Framerate.Max)) + i.Framerate.Average = json.Number(fmt.Sprintf("%.3f", io.Framerate.Average)) i.FPS = json.Number(fmt.Sprintf("%.3f", io.FPS)) i.Packet = io.Packet i.PPS = json.Number(fmt.Sprintf("%.3f", io.PPS)) diff --git a/restream/app/progress.go b/restream/app/progress.go index c9f1fcd5..675bf007 100644 --- a/restream/app/progress.go +++ b/restream/app/progress.go @@ -11,8 +11,13 @@ type ProgressIO struct { Type string Codec string Coder string - Frame uint64 // counter - Keyframe uint64 // counter + Frame uint64 // counter + Keyframe uint64 // counter + Framerate struct { + Min float64 + Max float64 + Average float64 + } FPS float64 // rate, frames per second Packet uint64 // counter PPS float64 // rate, packets per second From 6ddd58a1243c3a3898562008f07bb30e6762c40e Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Mon, 24 Apr 2023 11:59:09 +0200 Subject: [PATCH 09/26] Preserve process log history when updating a process --- ffmpeg/parse/parser.go | 21 ++++++++++++++++++++ restream/restream.go | 1 + restream/restream_test.go | 41 +++++++++++++++++++++++++++++++++------ 3 files changed, 57 insertions(+), 6 deletions(-) diff --git a/ffmpeg/parse/parser.go b/ffmpeg/parse/parser.go index edf0ca03..a3c60a2b 100644 --- a/ffmpeg/parse/parser.go +++ b/ffmpeg/parse/parser.go @@ -33,6 +33,9 @@ type Parser interface { // ReportHistory returns an array of previews logs ReportHistory() []Report + + // TransferReportHistory transfers the report history to another parser + TransferReportHistory(Parser) error } // Config is the config for the Parser implementation @@ -767,3 +770,21 @@ func (p *parser) ReportHistory() []Report { return history } + +func (p *parser) TransferReportHistory(dst Parser) error { + pp, ok := dst.(*parser) + if !ok { + return fmt.Errorf("the target parser is not of the required type") + } + + p.logHistory.Do(func(l interface{}) { + if l == nil { + return + } + + pp.logHistory.Value = l + pp.logHistory = pp.logHistory.Next() + }) + + return nil +} diff --git a/restream/restream.go b/restream/restream.go index fcf38999..597a2e0a 100644 --- a/restream/restream.go +++ b/restream/restream.go @@ -867,6 +867,7 @@ func (r *restream) UpdateProcess(id string, config *app.Config) error { return ErrUnknownProcess } + task.parser.TransferReportHistory(t.parser) t.process.Order = task.process.Order if id != t.id { diff --git a/restream/restream_test.go b/restream/restream_test.go index 11b08240..a0d782b0 100644 --- a/restream/restream_test.go +++ b/restream/restream_test.go @@ -21,10 +21,11 @@ func getDummyRestreamer(portrange net.Portranger, validatorIn, validatorOut ffmp } ffmpeg, err := ffmpeg.New(ffmpeg.Config{ - Binary: binary, - Portrange: portrange, - ValidatorInput: validatorIn, - ValidatorOutput: validatorOut, + Binary: binary, + LogHistoryLength: 3, + Portrange: portrange, + ValidatorInput: validatorIn, + ValidatorOutput: validatorOut, }) if err != nil { return nil, err @@ -437,10 +438,10 @@ func TestLog(t *testing.T) { rs.AddProcess(process) _, err = rs.GetProcessLog("foobar") - require.NotEqual(t, nil, err, "shouldn't be able to get log from non-existing process") + require.Error(t, err) log, err := rs.GetProcessLog(process.ID) - require.Equal(t, nil, err, "should be able to get log from existing process") + require.NoError(t, err) require.Equal(t, 0, len(log.Prelude)) require.Equal(t, 0, len(log.Log)) @@ -461,6 +462,34 @@ func TestLog(t *testing.T) { require.NotEqual(t, 0, len(log.Log)) } +func TestLogTransfer(t *testing.T) { + rs, err := getDummyRestreamer(nil, nil, nil, nil) + require.NoError(t, err) + + process := getDummyProcess() + + err = rs.AddProcess(process) + require.NoError(t, err) + + rs.StartProcess(process.ID) + time.Sleep(3 * time.Second) + rs.StopProcess(process.ID) + + rs.StartProcess(process.ID) + rs.StopProcess(process.ID) + + log, _ := rs.GetProcessLog(process.ID) + + require.Equal(t, 1, len(log.History)) + + err = rs.UpdateProcess(process.ID, process) + require.NoError(t, err) + + log, _ = rs.GetProcessLog(process.ID) + + require.Equal(t, 1, len(log.History)) +} + func TestPlayoutNoRange(t *testing.T) { rs, err := getDummyRestreamer(nil, nil, nil, nil) require.NoError(t, err) From 317d6eb4d91ad978e03c973e6a1fe65fd082daf1 Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Mon, 24 Apr 2023 12:05:01 +0200 Subject: [PATCH 10/26] Add updated_at field in process infos --- http/api/process.go | 1 + http/handler/api/restream.go | 1 + restream/app/process.go | 2 ++ restream/restream.go | 5 +++++ restream/restream_test.go | 13 ++++++++++++- 5 files changed, 21 insertions(+), 1 deletion(-) diff --git a/http/api/process.go b/http/api/process.go index e217b455..d52d4393 100644 --- a/http/api/process.go +++ b/http/api/process.go @@ -14,6 +14,7 @@ type Process struct { Type string `json:"type" jsonschema:"enum=ffmpeg"` Reference string `json:"reference"` CreatedAt int64 `json:"created_at" jsonschema:"minimum=0" format:"int64"` + UpdatedAt int64 `json:"updated_at" jsonschema:"minimum=0" format:"int64"` Config *ProcessConfig `json:"config,omitempty"` State *ProcessState `json:"state,omitempty"` Report *ProcessReport `json:"report,omitempty"` diff --git a/http/handler/api/restream.go b/http/handler/api/restream.go index c61f363a..96a4f75c 100644 --- a/http/handler/api/restream.go +++ b/http/handler/api/restream.go @@ -544,6 +544,7 @@ func (h *RestreamHandler) getProcess(id, filterString string) (api.Process, erro Reference: process.Reference, Type: "ffmpeg", CreatedAt: process.CreatedAt, + UpdatedAt: process.UpdatedAt, } if wants["config"] { diff --git a/restream/app/process.go b/restream/app/process.go index 4ec6036a..5e301c39 100644 --- a/restream/app/process.go +++ b/restream/app/process.go @@ -106,6 +106,7 @@ type Process struct { Reference string `json:"reference"` Config *Config `json:"config"` CreatedAt int64 `json:"created_at"` + UpdatedAt int64 `json:"updated_at"` Order string `json:"order"` } @@ -115,6 +116,7 @@ func (process *Process) Clone() *Process { Reference: process.Reference, Config: process.Config.Clone(), CreatedAt: process.CreatedAt, + UpdatedAt: process.UpdatedAt, Order: process.Order, } diff --git a/restream/restream.go b/restream/restream.go index 597a2e0a..bd5e1482 100644 --- a/restream/restream.go +++ b/restream/restream.go @@ -456,6 +456,8 @@ func (r *restream) createTask(config *app.Config) (*task, error) { CreatedAt: time.Now().Unix(), } + process.UpdatedAt = process.CreatedAt + if config.Autostart { process.Order = "start" } @@ -867,6 +869,9 @@ func (r *restream) UpdateProcess(id string, config *app.Config) error { return ErrUnknownProcess } + // This would require a major version jump + //t.process.CreatedAt = task.process.CreatedAt + t.process.UpdatedAt = time.Now().Unix() task.parser.TransferReportHistory(t.parser) t.process.Order = task.process.Order diff --git a/restream/restream_test.go b/restream/restream_test.go index a0d782b0..208e30b0 100644 --- a/restream/restream_test.go +++ b/restream/restream_test.go @@ -216,6 +216,14 @@ func TestUpdateProcess(t *testing.T) { err = rs.AddProcess(process2) require.Equal(t, nil, err) + process, err := rs.GetProcess(process2.ID) + require.NoError(t, err) + + //createdAt := process.CreatedAt + updatedAt := process.UpdatedAt + + time.Sleep(2 * time.Second) + process3 := getDummyProcess() require.NotNil(t, process3) process3.ID = "process2" @@ -230,8 +238,11 @@ func TestUpdateProcess(t *testing.T) { _, err = rs.GetProcess(process1.ID) require.Error(t, err) - _, err = rs.GetProcess(process3.ID) + process, err = rs.GetProcess(process3.ID) require.NoError(t, err) + + //require.Equal(t, createdAt, process.CreatedAt) + require.NotEqual(t, updatedAt, process.UpdatedAt) } func TestGetProcess(t *testing.T) { From b3696f492db7c8d96aa8f312e47046d9a79037ea Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Mon, 24 Apr 2023 12:10:40 +0200 Subject: [PATCH 11/26] Update changelog --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index bdd254d2..5b82f401 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,10 +2,17 @@ ### Core v16.12.0 > v16.?.? +- Add updated_at field in process infos +- Add preserve process log history when updating a process +- Add support for input framerate data from jsonstats patch +- Add number of keyframes and extradata size to process progress data - Fix better naming for storage endpoint documentation - Fix freeing up S3 mounts - Fix URL validation if the path contains FFmpeg specific placeholders - Fix purging default file from HTTP cache +- Fix parsing S3 storage definition from environment variable +- Fix checking length of CPU time array ([#10](https://github.com/datarhei/core/issues/10)) +- Deprecate ENV names that do not correspond to JSON name ### Core v16.11.0 > v16.12.0 From 061542645c78dbe76f83a9e229b0cf4d59928495 Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Mon, 24 Apr 2023 12:28:42 +0200 Subject: [PATCH 12/26] Fix test --- restream/restream_test.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/restream/restream_test.go b/restream/restream_test.go index 208e30b0..c142c7fd 100644 --- a/restream/restream_test.go +++ b/restream/restream_test.go @@ -219,7 +219,7 @@ func TestUpdateProcess(t *testing.T) { process, err := rs.GetProcess(process2.ID) require.NoError(t, err) - //createdAt := process.CreatedAt + createdAt := process.CreatedAt updatedAt := process.UpdatedAt time.Sleep(2 * time.Second) @@ -241,7 +241,7 @@ func TestUpdateProcess(t *testing.T) { process, err = rs.GetProcess(process3.ID) require.NoError(t, err) - //require.Equal(t, createdAt, process.CreatedAt) + require.NotEqual(t, createdAt, process.CreatedAt) // this should be equal, but will require a major version jump require.NotEqual(t, updatedAt, process.UpdatedAt) } From b58cc8a7ee9fab3fd407fcdf477a52b8a3496062 Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Mon, 24 Apr 2023 16:09:01 +0200 Subject: [PATCH 13/26] Fix race condition --- process/process.go | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/process/process.go b/process/process.go index 4bfcb4b4..d1754822 100644 --- a/process/process.go +++ b/process/process.go @@ -593,6 +593,10 @@ func (p *process) stop(wait bool) error { if p.callbacks.onExit == nil { p.callbacks.onExit = func() { wg.Done() + + p.callbacks.lock.Lock() + defer p.callbacks.lock.Unlock() + p.callbacks.onExit = nil } } else { @@ -600,6 +604,10 @@ func (p *process) stop(wait bool) error { p.callbacks.onExit = func() { cb() wg.Done() + + p.callbacks.lock.Lock() + defer p.callbacks.lock.Unlock() + p.callbacks.onExit = cb } } From bea10cb1147597a9e4cd8073ac0cd9c26529895d Mon Sep 17 00:00:00 2001 From: Jan Stabenow Date: Tue, 25 Apr 2023 13:56:21 +0200 Subject: [PATCH 14/26] Mod bumps FFmpeg to v5.1.3 --- .github_build/Build.bundle.cuda.env | 2 +- .github_build/Build.bundle.env | 2 +- .github_build/Build.bundle.rpi.env | 2 +- .github_build/Build.bundle.vaapi.env | 2 +- CHANGELOG.md | 1 + 5 files changed, 5 insertions(+), 4 deletions(-) diff --git a/.github_build/Build.bundle.cuda.env b/.github_build/Build.bundle.cuda.env index 808b4583..be91c889 100644 --- a/.github_build/Build.bundle.cuda.env +++ b/.github_build/Build.bundle.cuda.env @@ -1,3 +1,3 @@ # CORE NVIDIA CUDA BUNDLE -FFMPEG_VERSION=5.1.2 +FFMPEG_VERSION=5.1.3 CUDA_VERSION=11.7.1 diff --git a/.github_build/Build.bundle.env b/.github_build/Build.bundle.env index 060a458e..6758f80d 100644 --- a/.github_build/Build.bundle.env +++ b/.github_build/Build.bundle.env @@ -1,2 +1,2 @@ # CORE BUNDLE -FFMPEG_VERSION=5.1.2 +FFMPEG_VERSION=5.1.3 diff --git a/.github_build/Build.bundle.rpi.env b/.github_build/Build.bundle.rpi.env index 781096cd..3dbe35a9 100644 --- a/.github_build/Build.bundle.rpi.env +++ b/.github_build/Build.bundle.rpi.env @@ -1,2 +1,2 @@ # CORE RASPBERRY-PI BUNDLE -FFMPEG_VERSION=5.1.2 +FFMPEG_VERSION=5.1.3 diff --git a/.github_build/Build.bundle.vaapi.env b/.github_build/Build.bundle.vaapi.env index 060a458e..6758f80d 100644 --- a/.github_build/Build.bundle.vaapi.env +++ b/.github_build/Build.bundle.vaapi.env @@ -1,2 +1,2 @@ # CORE BUNDLE -FFMPEG_VERSION=5.1.2 +FFMPEG_VERSION=5.1.3 diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b82f401..dce53c44 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ - Add preserve process log history when updating a process - Add support for input framerate data from jsonstats patch - Add number of keyframes and extradata size to process progress data +- Mod bumps FFmpeg to v5.1.3 (datarhei/core:tag bundles) - Fix better naming for storage endpoint documentation - Fix freeing up S3 mounts - Fix URL validation if the path contains FFmpeg specific placeholders From 9b6354ab9441a1db760cb1e916c3e5a9e888a352 Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Tue, 25 Apr 2023 15:57:17 +0200 Subject: [PATCH 15/26] Revert commit b58cc8a7ee9fab3fd407fcdf477a52b8a3496062 --- process/process.go | 6 ------ 1 file changed, 6 deletions(-) diff --git a/process/process.go b/process/process.go index d1754822..9a8637a2 100644 --- a/process/process.go +++ b/process/process.go @@ -594,9 +594,6 @@ func (p *process) stop(wait bool) error { p.callbacks.onExit = func() { wg.Done() - p.callbacks.lock.Lock() - defer p.callbacks.lock.Unlock() - p.callbacks.onExit = nil } } else { @@ -605,9 +602,6 @@ func (p *process) stop(wait bool) error { cb() wg.Done() - p.callbacks.lock.Lock() - defer p.callbacks.lock.Unlock() - p.callbacks.onExit = cb } } From a2dab2682f079d0401d18ad933feebaa845110a7 Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Wed, 26 Apr 2023 09:49:28 +0200 Subject: [PATCH 16/26] Fix not propagating process limits --- ffmpeg/ffmpeg.go | 6 ++++++ process/process.go | 42 +++++++++++++++++++-------------------- restream/restream.go | 13 ++++++++++-- restream/restream_test.go | 23 +++++++++++++++++++++ 4 files changed, 60 insertions(+), 24 deletions(-) diff --git a/ffmpeg/ffmpeg.go b/ffmpeg/ffmpeg.go index d5f79184..f58d87c1 100644 --- a/ffmpeg/ffmpeg.go +++ b/ffmpeg/ffmpeg.go @@ -32,6 +32,9 @@ type ProcessConfig struct { Reconnect bool ReconnectDelay time.Duration StaleTimeout time.Duration + LimitCPU float64 + LimitMemory uint64 + LimitDuration time.Duration Command []string Parser process.Parser Logger log.Logger @@ -117,6 +120,9 @@ func (f *ffmpeg) New(config ProcessConfig) (process.Process, error) { Reconnect: config.Reconnect, ReconnectDelay: config.ReconnectDelay, StaleTimeout: config.StaleTimeout, + LimitCPU: config.LimitCPU, + LimitMemory: config.LimitMemory, + LimitDuration: config.LimitDuration, Parser: config.Parser, Logger: config.Logger, OnStart: config.OnStart, diff --git a/process/process.go b/process/process.go index 9a8637a2..32b28ae2 100644 --- a/process/process.go +++ b/process/process.go @@ -63,26 +63,19 @@ type Config struct { // Status represents the current status of a process type Status struct { - // State is the current state of the process. See stateType for the known states. - State string - - // States is the cumulative history of states the process had. - States States - - // Order is the wanted condition of process, either "start" or "stop" - Order string - - // Duration is the time since the last change of the state - Duration time.Duration - - // Time is the time of the last change of the state - Time time.Time - - // Used CPU in percent - CPU float64 - - // Used memory in bytes - Memory uint64 + State string // State is the current state of the process. See stateType for the known states. + States States // States is the cumulative history of states the process had. + Order string // Order is the wanted condition of process, either "start" or "stop" + Duration time.Duration // Duration is the time since the last change of the state + Time time.Time // Time is the time of the last change of the state + CPU struct { + Current float64 // Used CPU in percent + Limit float64 // Limit in percent + } + Memory struct { + Current uint64 // Used memory in bytes + Limit uint64 // Limit in bytes + } } // States @@ -390,6 +383,7 @@ func (p *process) getStateString() string { // Status returns the current status of the process func (p *process) Status() Status { cpu, memory := p.limits.Current() + cpuLimit, memoryLimit := p.limits.Limits() p.state.lock.Lock() stateTime := p.state.time @@ -407,10 +401,14 @@ func (p *process) Status() Status { Order: order, Duration: time.Since(stateTime), Time: stateTime, - CPU: cpu, - Memory: memory, } + s.CPU.Current = cpu + s.CPU.Limit = cpuLimit + + s.Memory.Current = memory + s.Memory.Limit = memoryLimit + return s } diff --git a/restream/restream.go b/restream/restream.go index bd5e1482..eb014297 100644 --- a/restream/restream.go +++ b/restream/restream.go @@ -355,6 +355,9 @@ func (r *restream) load() error { Reconnect: t.config.Reconnect, ReconnectDelay: time.Duration(t.config.ReconnectDelay) * time.Second, StaleTimeout: time.Duration(t.config.StaleTimeout) * time.Second, + LimitCPU: t.config.LimitCPU, + LimitMemory: t.config.LimitMemory, + LimitDuration: time.Duration(t.config.LimitWaitFor) * time.Second, Command: t.command, Parser: t.parser, Logger: t.logger, @@ -494,6 +497,9 @@ func (r *restream) createTask(config *app.Config) (*task, error) { Reconnect: t.config.Reconnect, ReconnectDelay: time.Duration(t.config.ReconnectDelay) * time.Second, StaleTimeout: time.Duration(t.config.StaleTimeout) * time.Second, + LimitCPU: t.config.LimitCPU, + LimitMemory: t.config.LimitMemory, + LimitDuration: time.Duration(t.config.LimitWaitFor) * time.Second, Command: t.command, Parser: t.parser, Logger: t.logger, @@ -1179,6 +1185,9 @@ func (r *restream) reloadProcess(id string) error { Reconnect: t.config.Reconnect, ReconnectDelay: time.Duration(t.config.ReconnectDelay) * time.Second, StaleTimeout: time.Duration(t.config.StaleTimeout) * time.Second, + LimitCPU: t.config.LimitCPU, + LimitMemory: t.config.LimitMemory, + LimitDuration: time.Duration(t.config.LimitWaitFor) * time.Second, Command: t.command, Parser: t.parser, Logger: t.logger, @@ -1218,8 +1227,8 @@ func (r *restream) GetProcessState(id string) (*app.State, error) { state.State = status.State state.States.Marshal(status.States) state.Time = status.Time.Unix() - state.Memory = status.Memory - state.CPU = status.CPU + state.Memory = status.Memory.Current + state.CPU = status.CPU.Current state.Duration = status.Duration.Round(10 * time.Millisecond).Seconds() state.Reconnect = -1 state.Command = make([]string, len(task.command)) diff --git a/restream/restream_test.go b/restream/restream_test.go index c142c7fd..64a536a5 100644 --- a/restream/restream_test.go +++ b/restream/restream_test.go @@ -883,3 +883,26 @@ func TestReplacer(t *testing.T) { require.Equal(t, process, rs.tasks["314159265359"].config) } + +func TestProcessLimit(t *testing.T) { + rsi, err := getDummyRestreamer(nil, nil, nil, nil) + require.NoError(t, err) + + process := getDummyProcess() + process.LimitCPU = 61 + process.LimitMemory = 42 + process.Autostart = false + + err = rsi.AddProcess(process) + require.NoError(t, err) + + rs := rsi.(*restream) + + task, ok := rs.tasks[process.ID] + require.True(t, ok) + + status := task.ffmpeg.Status() + + require.Equal(t, float64(61), status.CPU.Limit) + require.Equal(t, uint64(42), status.Memory.Limit) +} From e45f80ed42833c328bb34272a45d43a162bc23a5 Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Wed, 26 Apr 2023 09:50:09 +0200 Subject: [PATCH 17/26] Fix tests --- net/url/url_test.go | 2 +- restream/fs/fs_test.go | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/net/url/url_test.go b/net/url/url_test.go index dff373b8..1ad33409 100644 --- a/net/url/url_test.go +++ b/net/url/url_test.go @@ -55,7 +55,7 @@ func TestScheme(t *testing.T) { require.False(t, r) } -func TestPars(t *testing.T) { +func TestParse(t *testing.T) { u, err := Parse("http://localhost/foobar") require.NoError(t, err) require.Equal(t, &URL{ diff --git a/restream/fs/fs_test.go b/restream/fs/fs_test.go index 0162be1d..2b208096 100644 --- a/restream/fs/fs_test.go +++ b/restream/fs/fs_test.go @@ -78,7 +78,7 @@ func TestMaxAge(t *testing.T) { require.Eventually(t, func() bool { return cleanfs.Files() == 0 - }, 5*time.Second, time.Second) + }, 10*time.Second, time.Second) cleanfs.WriteFileReader("/chunk_3.ts", strings.NewReader("chunk_3")) @@ -96,7 +96,7 @@ func TestMaxAge(t *testing.T) { require.ElementsMatch(t, []string{"/chunk_3.ts"}, names) return true - }, 2*time.Second, time.Second) + }, 5*time.Second, time.Second) cleanfs.Stop() } From 74110dae5421139ad5f824e6ae891b3563120175 Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Fri, 28 Apr 2023 17:38:36 +0200 Subject: [PATCH 18/26] Fix possible infinite loop with HLS session rewriter --- http/middleware/session/HLS.go | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/http/middleware/session/HLS.go b/http/middleware/session/HLS.go index cb0768ca..ba993366 100644 --- a/http/middleware/session/HLS.go +++ b/http/middleware/session/HLS.go @@ -7,6 +7,7 @@ import ( "io" "net/http" "net/url" + "path" urlpath "path" "path/filepath" "regexp" @@ -298,7 +299,7 @@ func (r *bodyReader) Close() error { func (r *bodyReader) getSegments(dir string) []string { segments := []string{} - // Find all segments URLS in the .m3u8 + // Find all segment URLs in the .m3u8 scanner := bufio.NewScanner(&r.buffer) for scanner.Scan() { line := scanner.Text() @@ -405,9 +406,23 @@ func (g *sessionRewriter) rewriteHLS(sessionID string, requestURL *url.URL) { q := u.Query() + loop := false + // If this is a master manifest (i.e. an m3u8 which contains references to other m3u8), then // we give each substream an own session ID if they don't have already. if strings.HasSuffix(u.Path, ".m3u8") { + // Check if we're referring to ourselves. This will cause an infinite loop + // and has to be stopped. + file := u.Path + if !strings.HasPrefix(file, "/") { + dir := path.Dir(requestURL.Path) + file = filepath.Join(dir, file) + } + + if requestURL.Path == file { + loop = true + } + q.Set("session", shortuuid.New()) isMaster = true @@ -417,7 +432,11 @@ func (g *sessionRewriter) rewriteHLS(sessionID string, requestURL *url.URL) { u.RawQuery = q.Encode() - buffer.WriteString(u.String() + "\n") + if loop { + buffer.WriteString("# m3u8 is referencing itself: " + u.String() + "\n") + } else { + buffer.WriteString(u.String() + "\n") + } } if err := scanner.Err(); err != nil { From 8e2874a456ae8345c136b18ed58bdee083450475 Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Wed, 3 May 2023 10:34:07 +0200 Subject: [PATCH 19/26] Fix exposing build system paths in logs --- Makefile | 16 ++++++++-------- log/log.go | 16 ++++++++++++---- 2 files changed, 20 insertions(+), 12 deletions(-) diff --git a/Makefile b/Makefile index e20cce00..a93319c0 100644 --- a/Makefile +++ b/Makefile @@ -15,11 +15,11 @@ init: ## build: Build core (default) build: - CGO_ENABLED=${CGO_ENABLED} GOOS=${GOOS} GOARCH=${GOARCH} go build -o core${BINSUFFIX} + CGO_ENABLED=${CGO_ENABLED} GOOS=${GOOS} GOARCH=${GOARCH} go build -o core${BINSUFFIX} -trimpath # github workflow workaround build_linux: - CGO_ENABLED=0 GOOS=linux GOARCH=${OSARCH} go build -o core + CGO_ENABLED=0 GOOS=linux GOARCH=${OSARCH} go build -o core -trimpath ## swagger: Update swagger API documentation (requires github.com/swaggo/swag) swagger: @@ -69,19 +69,19 @@ lint: ## import: Build import binary import: - cd app/import && CGO_ENABLED=${CGO_ENABLED} GOOS=${GOOS} GOARCH=${GOARCH} go build -o ../../import -ldflags="-s -w" + cd app/import && CGO_ENABLED=${CGO_ENABLED} GOOS=${GOOS} GOARCH=${GOARCH} go build -o ../../import -trimpath -ldflags="-s -w" # github workflow workaround import_linux: - cd app/import && CGO_ENABLED=0 GOOS=linux GOARCH=${OSARCH} go build -o ../../import -ldflags="-s -w" + cd app/import && CGO_ENABLED=0 GOOS=linux GOARCH=${OSARCH} go build -o ../../import -trimpath -ldflags="-s -w" ## ffmigrate: Build ffmpeg migration binary ffmigrate: - cd app/ffmigrate && CGO_ENABLED=${CGO_ENABLED} GOOS=${GOOS} GOARCH=${GOARCH} go build -o ../../ffmigrate -ldflags="-s -w" + cd app/ffmigrate && CGO_ENABLED=${CGO_ENABLED} GOOS=${GOOS} GOARCH=${GOARCH} go build -o ../../ffmigrate -trimpath -ldflags="-s -w" # github workflow workaround ffmigrate_linux: - cd app/ffmigrate && CGO_ENABLED=0 GOOS=linux GOARCH=${OSARCH} go build -o ../../ffmigrate -ldflags="-s -w" + cd app/ffmigrate && CGO_ENABLED=0 GOOS=linux GOARCH=${OSARCH} go build -o ../../ffmigrate -trimpath -ldflags="-s -w" ## coverage: Generate code coverage analysis coverage: @@ -94,11 +94,11 @@ commit: vet fmt lint test build ## release: Build a release binary of core release: - CGO_ENABLED=${CGO_ENABLED} GOOS=${GOOS} GOARCH=${GOARCH} go build -o core -ldflags="-s -w -X github.com/datarhei/core/v16/app.Commit=$(COMMIT) -X github.com/datarhei/core/v16/app.Branch=$(BRANCH) -X github.com/datarhei/core/v16/app.Build=$(BUILD)" + CGO_ENABLED=${CGO_ENABLED} GOOS=${GOOS} GOARCH=${GOARCH} go build -o core -trimpath -ldflags="-s -w -X github.com/datarhei/core/v16/app.Commit=$(COMMIT) -X github.com/datarhei/core/v16/app.Branch=$(BRANCH) -X github.com/datarhei/core/v16/app.Build=$(BUILD)" # github workflow workaround release_linux: - CGO_ENABLED=0 GOOS=linux GOARCH=${OSARCH} go build -o core -ldflags="-s -w -X github.com/datarhei/core/v16/app.Commit=$(COMMIT) -X github.com/datarhei/core/v16/app.Branch=$(BRANCH) -X github.com/datarhei/core/v16/app.Build=$(BUILD)" + CGO_ENABLED=0 GOOS=linux GOARCH=${OSARCH} go build -o core -trimpath -ldflags="-s -w -X github.com/datarhei/core/v16/app.Commit=$(COMMIT) -X github.com/datarhei/core/v16/app.Branch=$(BRANCH) -X github.com/datarhei/core/v16/app.Build=$(BUILD)" ## docker: Build standard Docker image docker: diff --git a/log/log.go b/log/log.go index 14a78e2c..d730b449 100644 --- a/log/log.go +++ b/log/log.go @@ -6,6 +6,7 @@ import ( "fmt" "reflect" "runtime" + "runtime/debug" "strings" "time" ) @@ -101,8 +102,9 @@ type Logger interface { // logger is an implementation of the Logger interface. type logger struct { - output Writer - component string + output Writer + component string + modulePath string } // New returns an implementation of the Logger interface. @@ -111,13 +113,18 @@ func New(component string) Logger { component: component, } + if info, ok := debug.ReadBuildInfo(); ok { + l.modulePath = info.Path + } + return l } func (l *logger) clone() *logger { clone := &logger{ - output: l.output, - component: l.component, + output: l.output, + component: l.component, + modulePath: l.modulePath, } return clone @@ -213,6 +220,7 @@ func (e *Event) WithComponent(component string) Logger { func (e *Event) Log(format string, args ...interface{}) { _, file, line, _ := runtime.Caller(1) + file = strings.TrimPrefix(file, e.logger.modulePath) n := e.clone() From d41469cdbf6f597502d1f581d24359b59257db9b Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Fri, 5 May 2023 10:04:18 +0200 Subject: [PATCH 20/26] Upgrade rtmp library dependency This fixes a bug in the rtmp library where an error has been left unchecked caused by a malformed app or playPath. This led to a nil value for the URL of the publish or play request. However, this URL should never be nil and accessing this URL caused a panic and finally shutting the core down, resulting in a DoS. Thanks to Johannes Frank --- go.mod | 4 +-- go.sum | 4 +-- .../datarhei/joy4/format/rtmp/rtmp.go | 29 +++++++++++++++---- vendor/modules.txt | 2 +- 4 files changed, 29 insertions(+), 10 deletions(-) diff --git a/go.mod b/go.mod index 454d2310..8f577246 100644 --- a/go.mod +++ b/go.mod @@ -8,7 +8,7 @@ require ( github.com/atrox/haikunatorgo/v2 v2.0.1 github.com/caddyserver/certmagic v0.17.2 github.com/datarhei/gosrt v0.3.1 - github.com/datarhei/joy4 v0.0.0-20220914170649-23c70d207759 + github.com/datarhei/joy4 v0.0.0-20230505074825-fde05957445a github.com/go-playground/validator/v10 v10.11.1 github.com/gobwas/glob v0.2.3 github.com/golang-jwt/jwt/v4 v4.4.3 @@ -29,7 +29,6 @@ require ( github.com/xeipuuv/gojsonschema v1.2.0 go.uber.org/zap v1.24.0 golang.org/x/mod v0.7.0 - golang.org/x/net v0.7.0 ) require ( @@ -96,6 +95,7 @@ require ( go.uber.org/goleak v1.1.12 // indirect go.uber.org/multierr v1.9.0 // indirect golang.org/x/crypto v0.5.0 // indirect + golang.org/x/net v0.7.0 // indirect golang.org/x/sys v0.6.0 // indirect golang.org/x/text v0.7.0 // indirect golang.org/x/time v0.3.0 // indirect diff --git a/go.sum b/go.sum index dd05bdd5..9742f4a3 100644 --- a/go.sum +++ b/go.sum @@ -34,8 +34,8 @@ github.com/cpuguy83/go-md2man/v2 v2.0.1/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46t github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/datarhei/gosrt v0.3.1 h1:9A75hIvnY74IUFyeguqYXh1lsGF8Qt8fjxJS2Ewr12Q= github.com/datarhei/gosrt v0.3.1/go.mod h1:M2nl2WPrawncUc1FtUBK6gZX4tpZRC7FqL8NjOdBZV0= -github.com/datarhei/joy4 v0.0.0-20220914170649-23c70d207759 h1:h8NyekuQSDvLIsZVTV172m5/RVArXkEM/cnHaUzszQU= -github.com/datarhei/joy4 v0.0.0-20220914170649-23c70d207759/go.mod h1:Jcw/6jZDQQmPx8A7INEkXmuEF7E9jjBbSTfVSLwmiQw= +github.com/datarhei/joy4 v0.0.0-20230505074825-fde05957445a h1:Tf4DSHY1xruBglr+yYP5Wct7czM86GKMYgbXH8a7OFo= +github.com/datarhei/joy4 v0.0.0-20230505074825-fde05957445a/go.mod h1:Jcw/6jZDQQmPx8A7INEkXmuEF7E9jjBbSTfVSLwmiQw= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= diff --git a/vendor/github.com/datarhei/joy4/format/rtmp/rtmp.go b/vendor/github.com/datarhei/joy4/format/rtmp/rtmp.go index e9c0d66b..8d9dc745 100644 --- a/vendor/github.com/datarhei/joy4/format/rtmp/rtmp.go +++ b/vendor/github.com/datarhei/joy4/format/rtmp/rtmp.go @@ -178,6 +178,7 @@ func (self *Server) Serve(listener net.Listener) error { if Debug { fmt.Println("rtmp: server: client closed err:", err) } + conn.Close() }() } } @@ -190,6 +191,7 @@ func (self *Server) Close() { close(self.doneChan) self.listener.Close() + self.listener = nil } const ( @@ -398,7 +400,7 @@ func getTcUrl(u *url.URL) string { return nu.String() } -func createURL(tcurl, app, play string) (u *url.URL) { +func createURL(tcurl, app, play string) (*url.URL, error) { ps := strings.Split(app+"/"+play, "/") out := []string{""} for _, s := range ps { @@ -410,7 +412,11 @@ func createURL(tcurl, app, play string) (u *url.URL) { out = append(out, "") } path := strings.Join(out, "/") - u, _ = url.ParseRequestURI(path) + + u, err := url.ParseRequestURI(path) + if err != nil { + return nil, err + } if tcurl != "" { tu, _ := url.Parse(tcurl) @@ -419,7 +425,8 @@ func createURL(tcurl, app, play string) (u *url.URL) { u.Scheme = tu.Scheme } } - return + + return u, nil } var CodecTypes = flv.CodecTypes @@ -553,7 +560,13 @@ func (self *Conn) readConnect() (err error) { return } - self.URL = createURL(tcurl, connectpath, publishpath) + u, uerr := createURL(tcurl, connectpath, publishpath) + if uerr != nil { + err = fmt.Errorf("invalid URL: %w", uerr) + return + } + + self.URL = u self.publishing = true self.reading = true self.stage++ @@ -599,7 +612,13 @@ func (self *Conn) readConnect() (err error) { return } - self.URL = createURL(tcurl, connectpath, playpath) + u, uerr := createURL(tcurl, connectpath, playpath) + if uerr != nil { + err = fmt.Errorf("invalid URL: %w", uerr) + return + } + + self.URL = u self.playing = true self.writing = true self.stage++ diff --git a/vendor/modules.txt b/vendor/modules.txt index ae5f8026..fb3247d0 100644 --- a/vendor/modules.txt +++ b/vendor/modules.txt @@ -59,7 +59,7 @@ github.com/datarhei/gosrt/internal/congestion github.com/datarhei/gosrt/internal/crypto github.com/datarhei/gosrt/internal/net github.com/datarhei/gosrt/internal/packet -# github.com/datarhei/joy4 v0.0.0-20220914170649-23c70d207759 +# github.com/datarhei/joy4 v0.0.0-20230505074825-fde05957445a ## explicit; go 1.14 github.com/datarhei/joy4/av github.com/datarhei/joy4/av/avutil From e76d140541b7a94ec9fca6c5e1dc64c4a4196e68 Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Fri, 5 May 2023 10:44:25 +0200 Subject: [PATCH 21/26] Update changelog --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index dce53c44..a92845b5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ - Fix purging default file from HTTP cache - Fix parsing S3 storage definition from environment variable - Fix checking length of CPU time array ([#10](https://github.com/datarhei/core/issues/10)) +- Fix RTMP DoS attack (thx Johannes Frank) - Deprecate ENV names that do not correspond to JSON name ### Core v16.11.0 > v16.12.0 From e613a7423f468545cb5b9d9e8a883e0a62b3454a Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Fri, 5 May 2023 10:47:32 +0200 Subject: [PATCH 22/26] Update changelog --- CHANGELOG.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index a92845b5..2d6a0985 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,9 @@ - Fix purging default file from HTTP cache - Fix parsing S3 storage definition from environment variable - Fix checking length of CPU time array ([#10](https://github.com/datarhei/core/issues/10)) +- Fix possible infinite loop with HLS session rewriter +- Fix not propagating process limits +- Fix URL validation if the path contains FFmpeg specific placeholders - Fix RTMP DoS attack (thx Johannes Frank) - Deprecate ENV names that do not correspond to JSON name From ca261a56ee10cd66d5295bb5edb0d136b27caa12 Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Fri, 5 May 2023 12:03:48 +0200 Subject: [PATCH 23/26] Add looping_runtime to avstream status --- docs/docs.go | 35 +++++++++++++++++++---------- docs/swagger.json | 35 +++++++++++++++++++---------- docs/swagger.yaml | 24 +++++++++++++------- ffmpeg/parse/types.go | 48 +++++++++++++++++++++------------------- http/api/avstream.go | 22 +++++++++--------- restream/app/avstream.go | 23 ++++++++++--------- 6 files changed, 111 insertions(+), 76 deletions(-) diff --git a/docs/docs.go b/docs/docs.go index 41e9394e..c453f3f4 100644 --- a/docs/docs.go +++ b/docs/docs.go @@ -2032,6 +2032,10 @@ const docTemplate = `{ "looping": { "type": "boolean" }, + "looping_runtime": { + "type": "integer", + "format": "uint64" + }, "output": { "$ref": "#/definitions/api.AVstreamIO" }, @@ -3044,6 +3048,10 @@ const docTemplate = `{ }, "type": { "type": "string" + }, + "updated_at": { + "type": "integer", + "format": "int64" } } }, @@ -3358,18 +3366,7 @@ const docTemplate = `{ "format": "uint64" }, "framerate": { - "type": "object", - "properties": { - "avg": { - "type": "number" - }, - "max": { - "type": "number" - }, - "min": { - "type": "number" - } - } + "$ref": "#/definitions/api.ProgressIOFramerate" }, "height": { "type": "integer", @@ -3427,6 +3424,20 @@ const docTemplate = `{ } } }, + "api.ProgressIOFramerate": { + "type": "object", + "properties": { + "avg": { + "type": "number" + }, + "max": { + "type": "number" + }, + "min": { + "type": "number" + } + } + }, "api.RTMPChannel": { "type": "object", "properties": { diff --git a/docs/swagger.json b/docs/swagger.json index 2956a6ec..8351c34c 100644 --- a/docs/swagger.json +++ b/docs/swagger.json @@ -2025,6 +2025,10 @@ "looping": { "type": "boolean" }, + "looping_runtime": { + "type": "integer", + "format": "uint64" + }, "output": { "$ref": "#/definitions/api.AVstreamIO" }, @@ -3037,6 +3041,10 @@ }, "type": { "type": "string" + }, + "updated_at": { + "type": "integer", + "format": "int64" } } }, @@ -3351,18 +3359,7 @@ "format": "uint64" }, "framerate": { - "type": "object", - "properties": { - "avg": { - "type": "number" - }, - "max": { - "type": "number" - }, - "min": { - "type": "number" - } - } + "$ref": "#/definitions/api.ProgressIOFramerate" }, "height": { "type": "integer", @@ -3420,6 +3417,20 @@ } } }, + "api.ProgressIOFramerate": { + "type": "object", + "properties": { + "avg": { + "type": "number" + }, + "max": { + "type": "number" + }, + "min": { + "type": "number" + } + } + }, "api.RTMPChannel": { "type": "object", "properties": { diff --git a/docs/swagger.yaml b/docs/swagger.yaml index 2b35b2bd..ffe87a64 100644 --- a/docs/swagger.yaml +++ b/docs/swagger.yaml @@ -22,6 +22,9 @@ definitions: $ref: '#/definitions/api.AVstreamIO' looping: type: boolean + looping_runtime: + format: uint64 + type: integer output: $ref: '#/definitions/api.AVstreamIO' queue: @@ -701,6 +704,9 @@ definitions: $ref: '#/definitions/api.ProcessState' type: type: string + updated_at: + format: int64 + type: integer type: object api.ProcessConfig: properties: @@ -914,14 +920,7 @@ definitions: format: uint64 type: integer framerate: - properties: - avg: - type: number - max: - type: number - min: - type: number - type: object + $ref: '#/definitions/api.ProgressIOFramerate' height: format: uint64 type: integer @@ -963,6 +962,15 @@ definitions: format: uint64 type: integer type: object + api.ProgressIOFramerate: + properties: + avg: + type: number + max: + type: number + min: + type: number + type: object api.RTMPChannel: properties: name: diff --git a/ffmpeg/parse/types.go b/ffmpeg/parse/types.go index 1f829b5e..45bafa3f 100644 --- a/ffmpeg/parse/types.go +++ b/ffmpeg/parse/types.go @@ -59,33 +59,35 @@ func (avio *ffmpegAVstreamIO) export() app.AVstreamIO { } type ffmpegAVstream struct { - Input ffmpegAVstreamIO `json:"input"` - Output ffmpegAVstreamIO `json:"output"` - Address string `json:"id"` - URL string `json:"url"` - Stream uint64 `json:"stream"` - Aqueue uint64 `json:"aqueue"` - Queue uint64 `json:"queue"` - Dup uint64 `json:"dup"` - Drop uint64 `json:"drop"` - Enc uint64 `json:"enc"` - Looping bool `json:"looping"` - Duplicating bool `json:"duplicating"` - GOP string `json:"gop"` + Input ffmpegAVstreamIO `json:"input"` + Output ffmpegAVstreamIO `json:"output"` + Address string `json:"id"` + URL string `json:"url"` + Stream uint64 `json:"stream"` + Aqueue uint64 `json:"aqueue"` + Queue uint64 `json:"queue"` + Dup uint64 `json:"dup"` + Drop uint64 `json:"drop"` + Enc uint64 `json:"enc"` + Looping bool `json:"looping"` + LoopingRuntime uint64 `json:"looping_runtime"` + Duplicating bool `json:"duplicating"` + GOP string `json:"gop"` } func (av *ffmpegAVstream) export() *app.AVstream { return &app.AVstream{ - Aqueue: av.Aqueue, - Queue: av.Queue, - Drop: av.Drop, - Dup: av.Dup, - Enc: av.Enc, - Looping: av.Looping, - Duplicating: av.Duplicating, - GOP: av.GOP, - Input: av.Input.export(), - Output: av.Output.export(), + Aqueue: av.Aqueue, + Queue: av.Queue, + Drop: av.Drop, + Dup: av.Dup, + Enc: av.Enc, + Looping: av.Looping, + LoopingRuntime: av.LoopingRuntime, + Duplicating: av.Duplicating, + GOP: av.GOP, + Input: av.Input.export(), + Output: av.Output.export(), } } diff --git a/http/api/avstream.go b/http/api/avstream.go index 279b3352..91c6c5e7 100644 --- a/http/api/avstream.go +++ b/http/api/avstream.go @@ -23,16 +23,17 @@ func (i *AVstreamIO) Unmarshal(io *app.AVstreamIO) { } type AVstream struct { - Input AVstreamIO `json:"input"` - Output AVstreamIO `json:"output"` - Aqueue uint64 `json:"aqueue" format:"uint64"` - Queue uint64 `json:"queue" format:"uint64"` - Dup uint64 `json:"dup" format:"uint64"` - Drop uint64 `json:"drop" format:"uint64"` - Enc uint64 `json:"enc" format:"uint64"` - Looping bool `json:"looping"` - Duplicating bool `json:"duplicating"` - GOP string `json:"gop"` + Input AVstreamIO `json:"input"` + Output AVstreamIO `json:"output"` + Aqueue uint64 `json:"aqueue" format:"uint64"` + Queue uint64 `json:"queue" format:"uint64"` + Dup uint64 `json:"dup" format:"uint64"` + Drop uint64 `json:"drop" format:"uint64"` + Enc uint64 `json:"enc" format:"uint64"` + Looping bool `json:"looping"` + LoopingRuntime uint64 `json:"looping_runtime" format:"uint64"` + Duplicating bool `json:"duplicating"` + GOP string `json:"gop"` } func (a *AVstream) Unmarshal(av *app.AVstream) { @@ -46,6 +47,7 @@ func (a *AVstream) Unmarshal(av *app.AVstream) { a.Drop = av.Drop a.Enc = av.Enc a.Looping = av.Looping + a.LoopingRuntime = av.LoopingRuntime a.Duplicating = av.Duplicating a.GOP = av.GOP diff --git a/restream/app/avstream.go b/restream/app/avstream.go index 70cf9634..988a18cb 100644 --- a/restream/app/avstream.go +++ b/restream/app/avstream.go @@ -3,19 +3,20 @@ package app type AVstreamIO struct { State string Packet uint64 // counter - Time uint64 + Time uint64 // sec Size uint64 // bytes } type AVstream struct { - Input AVstreamIO - Output AVstreamIO - Aqueue uint64 // gauge - Queue uint64 // gauge - Dup uint64 // counter - Drop uint64 // counter - Enc uint64 // counter - Looping bool - Duplicating bool - GOP string + Input AVstreamIO + Output AVstreamIO + Aqueue uint64 // gauge + Queue uint64 // gauge + Dup uint64 // counter + Drop uint64 // counter + Enc uint64 // counter + Looping bool + LoopingRuntime uint64 // sec + Duplicating bool + GOP string } From ec7d47734b4b79c4eae99f9dfd9224a8a62c4132 Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Mon, 8 May 2023 11:47:40 +0200 Subject: [PATCH 24/26] Remove double import --- http/middleware/session/HLS.go | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/http/middleware/session/HLS.go b/http/middleware/session/HLS.go index ba993366..f061e2d2 100644 --- a/http/middleware/session/HLS.go +++ b/http/middleware/session/HLS.go @@ -7,7 +7,6 @@ import ( "io" "net/http" "net/url" - "path" urlpath "path" "path/filepath" "regexp" @@ -415,7 +414,7 @@ func (g *sessionRewriter) rewriteHLS(sessionID string, requestURL *url.URL) { // and has to be stopped. file := u.Path if !strings.HasPrefix(file, "/") { - dir := path.Dir(requestURL.Path) + dir := urlpath.Dir(requestURL.Path) file = filepath.Join(dir, file) } From 8f85b7665bc5e64b4994b391e98e32b16729ca02 Mon Sep 17 00:00:00 2001 From: Jan Stabenow Date: Mon, 8 May 2023 12:24:16 +0200 Subject: [PATCH 25/26] Mod updates workflows --- .github_build/Build.alpine.env | 2 +- .github_build/Build.ubuntu.env | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github_build/Build.alpine.env b/.github_build/Build.alpine.env index ced95107..fb9fe895 100644 --- a/.github_build/Build.alpine.env +++ b/.github_build/Build.alpine.env @@ -2,4 +2,4 @@ OS_NAME=alpine OS_VERSION=3.16 GOLANG_IMAGE=golang:1.20-alpine3.16 -CORE_VERSION=16.12.0 +CORE_VERSION=16.13.0 diff --git a/.github_build/Build.ubuntu.env b/.github_build/Build.ubuntu.env index 4e02a698..7372d1c2 100644 --- a/.github_build/Build.ubuntu.env +++ b/.github_build/Build.ubuntu.env @@ -2,4 +2,4 @@ OS_NAME=ubuntu OS_VERSION=20.04 GOLANG_IMAGE=golang:1.20-alpine3.16 -CORE_VERSION=16.12.0 +CORE_VERSION=16.13.0 From e0fdc37e8d589c5ab7c3d2a9f7cae4cd52743086 Mon Sep 17 00:00:00 2001 From: Ingo Oppermann Date: Mon, 8 May 2023 12:52:14 +0200 Subject: [PATCH 26/26] Bump version to 16.13.0 --- CHANGELOG.md | 2 +- app/version.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2d6a0985..45d307b5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # Core -### Core v16.12.0 > v16.?.? +### Core v16.12.0 > v16.13.0 - Add updated_at field in process infos - Add preserve process log history when updating a process diff --git a/app/version.go b/app/version.go index ec718dfd..3278e184 100644 --- a/app/version.go +++ b/app/version.go @@ -29,7 +29,7 @@ func (v versionInfo) MinorString() string { // Version of the app var Version = versionInfo{ Major: 16, - Minor: 12, + Minor: 13, Patch: 0, }