package/mpp
1
0
Fork 0
mirror of https://github.com/nyanmisaka/mpp.git synced 2025-09-27 05:25:53 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

[doc]: add documents for MPP design

Browse Source 
Change-Id: Icaa01b6ceaa0cc828e62679ca7671f62f75f457f
Signed-off-by: Herman Chen <herman.chen@rock-chips.com>
This commit is contained in:
Herman Chen
2016-10-17 15:49:02 +08:00
parent 53602de26c
commit 771baf627b
4 changed files with 345 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files

70
doc/kernel_driver.txt Normal file
View File

@@ -0,0 +1,70 @@
Hardware kernel driver design (2016.10.17)
================================================================================
Rockchip has two sets of hardware kernel driver.
The first one is vcodec_service/vpu_service/mpp_service which is a high
performance stateless frame base hardware kernel driver. This driver supports
all available codecs that hardware can provide. This driver is used on Android/
Linux.
Here is the vcodec_service kernel driver framework diagram.
+-------------+ +-------------+ +-------------+
| client A | | client B | | client C |
+-------------+ +-------------+ +-------------+
userspace
+------------------------------------------------------------------------------+
kernel
+-------------+ +-------------+ +-------------+
| session A | | session B | | session C |
+-------------+ +-------------+ +-------------+
| | | | | |
waiting done waiting done waiting done
| | | | | |
+---+---+ +---+---+ +---+---+ +---+---+ +---+---+
| task3 | | task0 | | task1 | | task0 | | task0 |
+---+---+ +-------+ +-------+ +-------+ +---+---+
| |
+---+---+ +---+---+
| task2 | | task1 |
+-------+ +-------+
+-----------+
| service |
+---------+-----------+---------+
| | |
waiting running done
| | |
+-----+-----+ +-----+-----+ +-----+-----+
| task A2 | | task A1 | | task C0 |
+-----+-----+ +-----------+ +-----+-----+
| |
+-----+-----+ +-----+-----+
| task B1 | | task C1 |
+-----+-----+ +-----+-----+
| |
+-----+-----+ +-----+-----+
| task A3 | | task A0 |
+-----------+ +-----+-----+
|
+-----+-----+
| task B0 |
+-----------+
The principle of this design is to separate user task handling and hardware
resource management and minimize the kernel serial process time between two
hardware process operation.
The driver uses session as communication channel. Each userspace client (client)
will have a kernel session. Client will commit tasks to session. Then hardware
is managed by service (vpu_service/vcodec_service). Service will provide the
ability to process tasks in sessions.
When client commits a task to kernel the task will be set to waiting status and
link to both session waiting list and service waiting list. Then service will
get task from waiting list to running list and run. When hardware finishs a task
the task will be moved to done list and put to both service done list and
session done list. Finally client will get the finished task from session.

57
doc/mpp_buffer.txt Normal file
View File

@@ -0,0 +1,57 @@
Mpp buffer design (2016.10.12)
================================================================================
Mpp buffer is the warpper of the buffer used by hardware. Hardware usually can
not use the buffer malloc by cpu. Then we design MppBuffer for different memory
allocator on different platform. Currently it is designed for ion buffer on
Android and drm buffer on Linux. Later may support vb2_buffer in v4l2 devices.
In order to manage buffer usage in different user mpp buffer module introduces
MppBufferGroup as bufffer manager. All MppBuffer will connect to its manager.
MppBufferGroup provides allocator service and buffer reuse ability.
Different MppBufferGroup has different allocator. And besides normal malloc/free
function the allocator can also accept buffer from external file descriptor.
MppBufferGroup has two lists, unused buffer list and used buffer list. When
buffer is free buffer will not be released immediately. Buffer will be moved to
unused list for later reuse. There is a good reason for doing so. When video
resolution comes to 4K the buffer size will be above 12M. It will take a long
time to allocate buffer and generate map table for it. So reusing the buffer
will save the allocate/free time.
Here is the diagram of Mpp buffer status transaction.
+----------+ +---------+
| create | | commit |
+-----+----+ +----+----+
| |
| |
| +----v----+
+----------+ unused <-----------+
| +---------+ |
| | | |
| | | |
+-----v----+ | | +-----+----+
| malloc | | | | free |
+----------+ | | +----------+
| inc_ref | +---------+ | dec_ref |
+-----+----+ +-----^----+
| |
| |
| +---------+ |
+----------> used +-----------+
+---------+
| |
| |
| |
| |
| |
+----^----+
|
|
+----+----+
| import |
+---------+

112
doc/mpp_design.txt Normal file
View File

@@ -0,0 +1,112 @@
MPP (Media Process Platform) design (2016.10.12)
================================================================================
The mpp is a middleware library for Rockchip SoC's cross platform media process.
The main purpose of mpp is to provide very high performance, high flexibility
and expansibility on multimedia (mainly video and image) process.
The design target of mpp is to connect different Rockchip hardware kernel driver
and different userspace application.
Rockchip has two sets of hardware kernel driver.
The first one is vcodec_service/vpu_service/mpp_service which is a high
performance stateless frame base hardware kernel driver. This driver supports
all available codecs that hardware can provide. This driver is used on Android/
Linux.
The second one is v4l2 driver which is developed for ChromeOS. It currently
supports H.264/H.265/vp8/vp9. This driver is used on ChomeOS/Linux.
Mpp plans to support serval userspace applications including OpenMax, FFmpeg,
gstreamer, libva.
Feature explaination
================================================================================
1. Cross Platform
The target OS platform including Android, Linux, ChromeOS and windows. Mpp uses
cmake to compile on different platform.
2. High Performance
Mpp supports sync / async interface to reduce the time blocked in interface. And
mpp internally make hardware and software run parallelly. When hardware is
running sofware will prepare next hardware task at the same time.
3. High Flexibility
mpi (Media Process Interface) is easy to extend by different control function.
The input/output element packet/frame/buffer is easy to extend different
components.
System Diagram
================================================================================
+---------------------------------------+
| |
| ffmpeg / OpenMax / gstreamer / libva |
| |
+---------------------------------------+
+-------------------- MPP ----------------------+
| |
| +-------------------------+ +--------+ |
| | | | | |
| | MPI / MPP | | | |
| | buffer queue manage | | | |
| | | | | |
| +-------------------------+ | | |
| | | |
| +-------------------------+ | | |
| | | | | |
| | codec | | OSAL | |
| | decoder / encoder | | | |
| | | | | |
| +-------------------------+ | | |
| | | |
| +-----------+ +-----------+ | | |
| | | | | | | |
| | parser | | HAL | | | |
| | control | | reg_gen | | | |
| | | | | | | |
| +-----------+ +-----------+ +--------| |
| |
+-------------------- MPP ----------------------+
+---------------------------------------+
| |
| kernel |
| RK vcodec_service / v4l2 |
| |
+---------------------------------------+
Mpp is composed of four main sub-modules:
OSAL (Operation System Abstraction Layer)
This module shutters the differences between different operation systems and
provide basic components including memory, time, thread, log and hardware memory
allocator.
MPI (Media Process Interface) / MPP
This module is on charge of interaction with external user. Mpi layer has two
ways for user. The simple way - User can use put/get packet/frame function set.
The advanced way - User has to config MppTask and use dequeue/enqueue funciton
set to communicate with mpp. MppTask can carry different meta data and complete
complex work.
Codec (encoder / decoder)
This module implements the high efficiency internal work flow. The codec module
provides a general call flow for different video format. Software process will
be separated from hardware specified process. The software will communicate with
hardware with a common task interface which combines the buffer information and
codec specified infomation.
Parser/Controller and hal (Hardware Abstraction Layer)
This layer provides the implement function call of different video format and
different hardware. For decoder parser provide the video stream parse function
and output format related syntax structure to hal. The hal will translate the
syntax structure to register set on different hardware. Current hal supports
vcodec_service kernel driver and plan to support v4l2 driver later.

149
readme.txt
View File

@@ -9,10 +9,12 @@ Rules:
1. header file arrange rule 1. header file arrange rule
a. inc directory in each module folder is for external module usage. a. inc directory in each module folder is for external module usage.
b. module internal header file should be put along with the implement file. b. module internal header file should be put along with the implement file.
c. header file should not contain any relative path or absolute path, all include path should be keep in Makefile. c. header file should not contain any relative path or absolute path, all
include path should be keep in Makefile.
2. compiling system rule 2. compiling system rule
a. for cross platform compiling use cmake as the compiling management system. a. for cross platform compiling use cmake as the compiling management system.
b. use cmake out-of-source build, final binary and library will be install to out/ directory. b. use cmake out-of-source build, final binary and library will be install to
out/ directory.
3. header file include order 3. header file include order
a. MODULE_TAG a. MODULE_TAG
b. system header b. system header
@@ -20,7 +22,10 @@ c. osal header
d. module header d. module header
NOTE: NOTE:
1. when run on window pthreadVC2.dll needed to be copied to the system directory. 1. when run on window pthreadVC2.dll is needed to be copied to system directory.
2. Current MPP only support new chipset like RK3228/RK3229/RK3399/RK1108.
Old chipset like RK29xx/RK30xx/RK31XX/RK3288/RK3368 is not supported due to
lack of corresponding hardware register generation module.
---- top ---- top
| |
@@ -36,24 +41,43 @@ NOTE:
| | | |
| |----- vc12-x86_64 visual studio 2013 on x86_64 build directory | |----- vc12-x86_64 visual studio 2013 on x86_64 build directory
| |
|----- inc header file for external usage, including platform header and mpi header |----- doc design documents of mpp
| |
|----- mpp Media Process Platform : mpi function private implement and mpp infrastructure (vpu_api private layer) |----- inc header file for external usage, including
| platform header and mpi header
|
|----- mpp Media Process Platform : mpi function private
| | implement and mpp infrastructure (vpu_api
| | private layer)
| | | |
| |----- common video codec protocol syntax interface for both codec parser and hal | |----- base base components including MppBuffer, MppFrame,
| | MppPacket, MppTask, MppMeta, etc.
| | | |
| |----- codec all video codec parser, convert stream to protocol structure | |----- common video codec protocol syntax interface for both
| | codec parser and hal
| |
| |----- codec all video codec parser, convert stream to
| | | protocol structure
| | | | | |
| | |----- inc header files provided by codec module for external usage | | |----- inc header files provided by codec module for
| | | external usage
| | | | | |
| | |----- dec | | |----- dec
| | | | | | | |
| | | |----- dummy decoder parser work flow sample | | | |----- dummy decoder parser work flow sample
| | | | | | | |
| | | |----- h263
| | | |
| | | |----- h264 | | | |----- h264
| | | | | | | |
| | | |----- h265 | | | |----- h265
| | | | | | | |
| | | |----- m2v mpeg2 parser
| | | |
| | | |----- mpg4 mpeg4 parser
| | | |
| | | |----- vp8
| | | |
| | | |----- vp9 | | | |----- vp9
| | | | | | | |
| | | |----- jpeg | | | |----- jpeg
@@ -80,25 +104,39 @@ NOTE:
| | | | | |
| | |----- deinter deinterlace function module including pp/iep/rga | | |----- deinter deinterlace function module including pp/iep/rga
| | | | | |
| | |----- rkdec rockchip hardware decoder register generation library | | |----- rkdec rockchip hardware decoder register generation
| | | | | | | |
| | | |----- h264d generate register file from H.264 structure created by codec parser | | | |----- h264d generate register file from H.264 syntax info
| | | | | | | |
| | | |----- h265d generate register file from H.265 structure created by codec parser | | | |----- h265d generate register file from H.265 syntax info
| | | | | | | |
| | | |----- vp9d generate register file from vp9 structure created by codec parser | | | |----- vp9d generate register file from vp9 syntax info
| | | | | |
| | |----- vpu vpu register generation library | | |----- vpu vpu register generation library
| | | | | |
| | |----- h264d generate register file from H.264 structure created by codec parser | | |----- h263d generate register file from H.263 syntax info
| | | | | |
| | |----- h265d generate register file from H.265 structure created by codec parser | | |----- h264d generate register file from H.264 syntax info
| | |
| | |----- h265d generate register file from H.265 syntax info
| | |
| | |----- jpegd generate register file from jpeg syntax info
| | |
| | |----- jpege generate register file from jpeg syntax info
| | |
| | |----- m2vd generate register file from mpeg2 syntax info
| | |
| | |----- mpg4d generate register file from mpeg4 syntax info
| | |
| | |----- vp8d generate register file from vp8 syntax info
| | | |
| |----- legacy generate new libvpu to include old vpuapi path and new mpp path | |----- legacy generate new libvpu to include old vpuapi path
| | and new mpp path
| | | |
| |----- test mpp internal video protocol unit test and demo | |----- test mpp internal video protocol unit test and demo
| |
|----- test mpp buffer/packet component unit test and mpi demo |----- test mpp buffer/packet component unit test and
| mpp/mpi/vpu_api demo
| |
|----- out final release binary output directory |----- out final release binary output directory
| | | |
@@ -108,12 +146,16 @@ NOTE:
| | | |
| |----- lib library file output directory | |----- lib library file output directory
| |
|----- osal Operation System Abstract Layer: abstract layer for different operation system |----- osal Operation System Abstract Layer: abstract layer
| | for different operation system
| | | |
| |----- mem mpi memory subsystem for hardware | |----- allocator supported allocator including Android ion and
| | Linux drm
| | | |
| |----- android google's android | |----- android google's android
| | | |
| |----- inc osal header file for mpp modules
| |
| |----- linux mainline linux kernel | |----- linux mainline linux kernel
| | | |
| |----- window microsoft's window | |----- window microsoft's window
@@ -127,31 +169,50 @@ NOTE:
Here is the mpp implement overall framework: Here is the mpp implement overall framework:
+-------------------------+ +--------+ +---------------------------------------+
| | | | | |
| MPI / MPP | | | | ffmpeg / OpenMax / gstreamer / libva |
| buffer queue manage | | | | |
| | | | +---------------------------------------+
+-------------------------+ | |
| | +-------------------- MPP ----------------------+
+-------------------------+ | | | |
| | | | | +-------------------------+ +--------+ |
| codec | | OSAL | | | | | | |
| decoder / encoder | | | | | MPI / MPP | | | |
| | | | | | buffer queue manage | | | |
+-------------------------+ | | | | | | | |
| | | +-------------------------+ | | |
+-----------+ +-----------+ | | | | | |
| | | | | | | +-------------------------+ | | |
| parser | | HAL | | | | | | | | |
| recoder | | reg_gen | | | | | codec | | OSAL | |
| | | | | | | | decoder / encoder | | | |
+-----------+ +-----------+ +--------+ | | | | | |
| +-------------------------+ | | |
| | | |
| +-----------+ +-----------+ | | |
| | | | | | | |
| | parser | | HAL | | | |
| | recoder | | reg_gen | | | |
| | | | | | | |
| +-----------+ +-----------+ +--------| |
| |
+-------------------- MPP ----------------------+
+---------------------------------------+
| |
| kernel |
| RK vcodec_service / v4l2 |
| |
+---------------------------------------+
Here is the Media Process Interface hierarchical structure Here is the Media Process Interface hierarchical structure
MpiPacket and MpiFrame is the stream I/O data structure. MpiPacket and MpiFrame is the stream I/O data structure.
And MpiBuffer encapsulates different buffer implement like Linux's dma-buf and Android's ion. And MpiBuffer encapsulates different buffer implement like Linux's dma-buf and
Android's ion.
This part is learned from ffmpeg. This part is learned from ffmpeg.
+-------------------+ +-------------------+
@@ -185,10 +246,12 @@ This part is learned from ffmpeg.
Take H.264 deocder for example. Video stream will first queued by MPI/MPP layer, MPP will send the stream to codec layer, Take H.264 deocder for example. Video stream will first queued by MPI/MPP layer,
codec layer parses the stream header and generates a protocol standard output. This output will be send to HAL to generate MPP will send the stream to codec layer, codec layer parses the stream header
register file set and communicate with hardware. Hardware will complete the task and resend information back. MPP notify and generates a protocol standard output. This output will be send to HAL to
codec by hardware result, codec output decoded frame by display order. generate register file set and communicate with hardware. Hardware will complete
the task and resend information back. MPP notify codec by hardware result, codec
output decoded frame by display order.
MPI MPP decoder parser HAL MPI MPP decoder parser HAL