mirror of
https://github.com/nyanmisaka/mpp.git
synced 2025-09-26 21:15:53 +08:00
[doc]: add documents for MPP design
Change-Id: Icaa01b6ceaa0cc828e62679ca7671f62f75f457f Signed-off-by: Herman Chen <herman.chen@rock-chips.com>
This commit is contained in:
Herman Chen
70
doc/kernel_driver.txt
Normal file
70
doc/kernel_driver.txt
Normal file
@@ -0,0 +1,70 @@
|
||||
Hardware kernel driver design (2016.10.17)
|
||||
================================================================================
|
||||
|
||||
Rockchip has two sets of hardware kernel driver.
|
||||
|
||||
The first one is vcodec_service/vpu_service/mpp_service which is a high
|
||||
performance stateless frame base hardware kernel driver. This driver supports
|
||||
all available codecs that hardware can provide. This driver is used on Android/
|
||||
Linux.
|
||||
|
||||
Here is the vcodec_service kernel driver framework diagram.
|
||||
|
||||
+-------------+ +-------------+ +-------------+
|
||||
| client A | | client B | | client C |
|
||||
+-------------+ +-------------+ +-------------+
|
||||
|
||||
userspace
|
||||
+------------------------------------------------------------------------------+
|
||||
kernel
|
||||
|
||||
+-------------+ +-------------+ +-------------+
|
||||
| session A | | session B | | session C |
|
||||
+-------------+ +-------------+ +-------------+
|
||||
| | | | | |
|
||||
waiting done waiting done waiting done
|
||||
| | | | | |
|
||||
+---+---+ +---+---+ +---+---+ +---+---+ +---+---+
|
||||
| task3 | | task0 | | task1 | | task0 | | task0 |
|
||||
+---+---+ +-------+ +-------+ +-------+ +---+---+
|
||||
| |
|
||||
+---+---+ +---+---+
|
||||
| task2 | | task1 |
|
||||
+-------+ +-------+
|
||||
+-----------+
|
||||
| service |
|
||||
+---------+-----------+---------+
|
||||
| | |
|
||||
waiting running done
|
||||
| | |
|
||||
+-----+-----+ +-----+-----+ +-----+-----+
|
||||
| task A2 | | task A1 | | task C0 |
|
||||
+-----+-----+ +-----------+ +-----+-----+
|
||||
| |
|
||||
+-----+-----+ +-----+-----+
|
||||
| task B1 | | task C1 |
|
||||
+-----+-----+ +-----+-----+
|
||||
| |
|
||||
+-----+-----+ +-----+-----+
|
||||
| task A3 | | task A0 |
|
||||
+-----------+ +-----+-----+
|
||||
|
|
||||
+-----+-----+
|
||||
| task B0 |
|
||||
+-----------+
|
||||
|
||||
The principle of this design is to separate user task handling and hardware
|
||||
resource management and minimize the kernel serial process time between two
|
||||
hardware process operation.
|
||||
|
||||
The driver uses session as communication channel. Each userspace client (client)
|
||||
will have a kernel session. Client will commit tasks to session. Then hardware
|
||||
is managed by service (vpu_service/vcodec_service). Service will provide the
|
||||
ability to process tasks in sessions.
|
||||
|
||||
When client commits a task to kernel the task will be set to waiting status and
|
||||
link to both session waiting list and service waiting list. Then service will
|
||||
get task from waiting list to running list and run. When hardware finishs a task
|
||||
the task will be moved to done list and put to both service done list and
|
||||
session done list. Finally client will get the finished task from session.
|
||||
|
||||
57
doc/mpp_buffer.txt
Normal file
57
doc/mpp_buffer.txt
Normal file
@@ -0,0 +1,57 @@
|
||||
Mpp buffer design (2016.10.12)
|
||||
================================================================================
|
||||
|
||||
Mpp buffer is the warpper of the buffer used by hardware. Hardware usually can
|
||||
not use the buffer malloc by cpu. Then we design MppBuffer for different memory
|
||||
allocator on different platform. Currently it is designed for ion buffer on
|
||||
Android and drm buffer on Linux. Later may support vb2_buffer in v4l2 devices.
|
||||
|
||||
In order to manage buffer usage in different user mpp buffer module introduces
|
||||
MppBufferGroup as bufffer manager. All MppBuffer will connect to its manager.
|
||||
MppBufferGroup provides allocator service and buffer reuse ability.
|
||||
|
||||
Different MppBufferGroup has different allocator. And besides normal malloc/free
|
||||
function the allocator can also accept buffer from external file descriptor.
|
||||
|
||||
MppBufferGroup has two lists, unused buffer list and used buffer list. When
|
||||
buffer is free buffer will not be released immediately. Buffer will be moved to
|
||||
unused list for later reuse. There is a good reason for doing so. When video
|
||||
resolution comes to 4K the buffer size will be above 12M. It will take a long
|
||||
time to allocate buffer and generate map table for it. So reusing the buffer
|
||||
will save the allocate/free time.
|
||||
|
||||
Here is the diagram of Mpp buffer status transaction.
|
||||
|
||||
+----------+ +---------+
|
||||
| create | | commit |
|
||||
+-----+----+ +----+----+
|
||||
| |
|
||||
| |
|
||||
| +----v----+
|
||||
+----------+ unused <-----------+
|
||||
| +---------+ |
|
||||
| | | |
|
||||
| | | |
|
||||
+-----v----+ | | +-----+----+
|
||||
| malloc | | | | free |
|
||||
+----------+ | | +----------+
|
||||
| inc_ref | +---------+ | dec_ref |
|
||||
+-----+----+ +-----^----+
|
||||
| |
|
||||
| |
|
||||
| +---------+ |
|
||||
+----------> used +-----------+
|
||||
+---------+
|
||||
| |
|
||||
| |
|
||||
| |
|
||||
| |
|
||||
| |
|
||||
+----^----+
|
||||
|
|
||||
|
|
||||
+----+----+
|
||||
| import |
|
||||
+---------+
|
||||
|
||||
|
||||
112
doc/mpp_design.txt
Normal file
112
doc/mpp_design.txt
Normal file
@@ -0,0 +1,112 @@
|
||||
MPP (Media Process Platform) design (2016.10.12)
|
||||
================================================================================
|
||||
|
||||
The mpp is a middleware library for Rockchip SoC's cross platform media process.
|
||||
The main purpose of mpp is to provide very high performance, high flexibility
|
||||
and expansibility on multimedia (mainly video and image) process.
|
||||
|
||||
The design target of mpp is to connect different Rockchip hardware kernel driver
|
||||
and different userspace application.
|
||||
|
||||
Rockchip has two sets of hardware kernel driver.
|
||||
|
||||
The first one is vcodec_service/vpu_service/mpp_service which is a high
|
||||
performance stateless frame base hardware kernel driver. This driver supports
|
||||
all available codecs that hardware can provide. This driver is used on Android/
|
||||
Linux.
|
||||
|
||||
The second one is v4l2 driver which is developed for ChromeOS. It currently
|
||||
supports H.264/H.265/vp8/vp9. This driver is used on ChomeOS/Linux.
|
||||
|
||||
Mpp plans to support serval userspace applications including OpenMax, FFmpeg,
|
||||
gstreamer, libva.
|
||||
|
||||
|
||||
Feature explaination
|
||||
================================================================================
|
||||
|
||||
1. Cross Platform
|
||||
The target OS platform including Android, Linux, ChromeOS and windows. Mpp uses
|
||||
cmake to compile on different platform.
|
||||
|
||||
2. High Performance
|
||||
Mpp supports sync / async interface to reduce the time blocked in interface. And
|
||||
mpp internally make hardware and software run parallelly. When hardware is
|
||||
running sofware will prepare next hardware task at the same time.
|
||||
|
||||
3. High Flexibility
|
||||
mpi (Media Process Interface) is easy to extend by different control function.
|
||||
The input/output element packet/frame/buffer is easy to extend different
|
||||
components.
|
||||
|
||||
|
||||
System Diagram
|
||||
================================================================================
|
||||
|
||||
+---------------------------------------+
|
||||
| |
|
||||
| ffmpeg / OpenMax / gstreamer / libva |
|
||||
| |
|
||||
+---------------------------------------+
|
||||
|
||||
+-------------------- MPP ----------------------+
|
||||
| |
|
||||
| +-------------------------+ +--------+ |
|
||||
| | | | | |
|
||||
| | MPI / MPP | | | |
|
||||
| | buffer queue manage | | | |
|
||||
| | | | | |
|
||||
| +-------------------------+ | | |
|
||||
| | | |
|
||||
| +-------------------------+ | | |
|
||||
| | | | | |
|
||||
| | codec | | OSAL | |
|
||||
| | decoder / encoder | | | |
|
||||
| | | | | |
|
||||
| +-------------------------+ | | |
|
||||
| | | |
|
||||
| +-----------+ +-----------+ | | |
|
||||
| | | | | | | |
|
||||
| | parser | | HAL | | | |
|
||||
| | control | | reg_gen | | | |
|
||||
| | | | | | | |
|
||||
| +-----------+ +-----------+ +--------| |
|
||||
| |
|
||||
+-------------------- MPP ----------------------+
|
||||
|
||||
+---------------------------------------+
|
||||
| |
|
||||
| kernel |
|
||||
| RK vcodec_service / v4l2 |
|
||||
| |
|
||||
+---------------------------------------+
|
||||
|
||||
Mpp is composed of four main sub-modules:
|
||||
|
||||
OSAL (Operation System Abstraction Layer)
|
||||
This module shutters the differences between different operation systems and
|
||||
provide basic components including memory, time, thread, log and hardware memory
|
||||
allocator.
|
||||
|
||||
MPI (Media Process Interface) / MPP
|
||||
This module is on charge of interaction with external user. Mpi layer has two
|
||||
ways for user. The simple way - User can use put/get packet/frame function set.
|
||||
The advanced way - User has to config MppTask and use dequeue/enqueue funciton
|
||||
set to communicate with mpp. MppTask can carry different meta data and complete
|
||||
complex work.
|
||||
|
||||
Codec (encoder / decoder)
|
||||
This module implements the high efficiency internal work flow. The codec module
|
||||
provides a general call flow for different video format. Software process will
|
||||
be separated from hardware specified process. The software will communicate with
|
||||
hardware with a common task interface which combines the buffer information and
|
||||
codec specified infomation.
|
||||
|
||||
Parser/Controller and hal (Hardware Abstraction Layer)
|
||||
This layer provides the implement function call of different video format and
|
||||
different hardware. For decoder parser provide the video stream parse function
|
||||
and output format related syntax structure to hal. The hal will translate the
|
||||
syntax structure to register set on different hardware. Current hal supports
|
||||
vcodec_service kernel driver and plan to support v4l2 driver later.
|
||||
|
||||
|
||||
149
readme.txt
149
readme.txt
@@ -9,10 +9,12 @@ Rules:
|
||||
1. header file arrange rule
|
||||
a. inc directory in each module folder is for external module usage.
|
||||
b. module internal header file should be put along with the implement file.
|
||||
c. header file should not contain any relative path or absolute path, all include path should be keep in Makefile.
|
||||
c. header file should not contain any relative path or absolute path, all
|
||||
include path should be keep in Makefile.
|
||||
2. compiling system rule
|
||||
a. for cross platform compiling use cmake as the compiling management system.
|
||||
b. use cmake out-of-source build, final binary and library will be install to out/ directory.
|
||||
b. use cmake out-of-source build, final binary and library will be install to
|
||||
out/ directory.
|
||||
3. header file include order
|
||||
a. MODULE_TAG
|
||||
b. system header
|
||||
@@ -20,7 +22,10 @@ c. osal header
|
||||
d. module header
|
||||
|
||||
NOTE:
|
||||
1. when run on window pthreadVC2.dll needed to be copied to the system directory.
|
||||
1. when run on window pthreadVC2.dll is needed to be copied to system directory.
|
||||
2. Current MPP only support new chipset like RK3228/RK3229/RK3399/RK1108.
|
||||
Old chipset like RK29xx/RK30xx/RK31XX/RK3288/RK3368 is not supported due to
|
||||
lack of corresponding hardware register generation module.
|
||||
|
||||
---- top
|
||||
|
|
||||
@@ -36,24 +41,43 @@ NOTE:
|
||||
| |
|
||||
| |----- vc12-x86_64 visual studio 2013 on x86_64 build directory
|
||||
|
|
||||
|----- inc header file for external usage, including platform header and mpi header
|
||||
|----- doc design documents of mpp
|
||||
|
|
||||
|----- mpp Media Process Platform : mpi function private implement and mpp infrastructure (vpu_api private layer)
|
||||
|----- inc header file for external usage, including
|
||||
| platform header and mpi header
|
||||
|
|
||||
|----- mpp Media Process Platform : mpi function private
|
||||
| | implement and mpp infrastructure (vpu_api
|
||||
| | private layer)
|
||||
| |
|
||||
| |----- common video codec protocol syntax interface for both codec parser and hal
|
||||
| |----- base base components including MppBuffer, MppFrame,
|
||||
| | MppPacket, MppTask, MppMeta, etc.
|
||||
| |
|
||||
| |----- codec all video codec parser, convert stream to protocol structure
|
||||
| |----- common video codec protocol syntax interface for both
|
||||
| | codec parser and hal
|
||||
| |
|
||||
| |----- codec all video codec parser, convert stream to
|
||||
| | | protocol structure
|
||||
| | |
|
||||
| | |----- inc header files provided by codec module for external usage
|
||||
| | |----- inc header files provided by codec module for
|
||||
| | | external usage
|
||||
| | |
|
||||
| | |----- dec
|
||||
| | | |
|
||||
| | | |----- dummy decoder parser work flow sample
|
||||
| | | |
|
||||
| | | |----- h263
|
||||
| | | |
|
||||
| | | |----- h264
|
||||
| | | |
|
||||
| | | |----- h265
|
||||
| | | |
|
||||
| | | |----- m2v mpeg2 parser
|
||||
| | | |
|
||||
| | | |----- mpg4 mpeg4 parser
|
||||
| | | |
|
||||
| | | |----- vp8
|
||||
| | | |
|
||||
| | | |----- vp9
|
||||
| | | |
|
||||
| | | |----- jpeg
|
||||
@@ -80,25 +104,39 @@ NOTE:
|
||||
| | |
|
||||
| | |----- deinter deinterlace function module including pp/iep/rga
|
||||
| | |
|
||||
| | |----- rkdec rockchip hardware decoder register generation library
|
||||
| | |----- rkdec rockchip hardware decoder register generation
|
||||
| | | |
|
||||
| | | |----- h264d generate register file from H.264 structure created by codec parser
|
||||
| | | |----- h264d generate register file from H.264 syntax info
|
||||
| | | |
|
||||
| | | |----- h265d generate register file from H.265 structure created by codec parser
|
||||
| | | |----- h265d generate register file from H.265 syntax info
|
||||
| | | |
|
||||
| | | |----- vp9d generate register file from vp9 structure created by codec parser
|
||||
| | | |----- vp9d generate register file from vp9 syntax info
|
||||
| | |
|
||||
| | |----- vpu vpu register generation library
|
||||
| | |
|
||||
| | |----- h264d generate register file from H.264 structure created by codec parser
|
||||
| | |----- h263d generate register file from H.263 syntax info
|
||||
| | |
|
||||
| | |----- h265d generate register file from H.265 structure created by codec parser
|
||||
| | |----- h264d generate register file from H.264 syntax info
|
||||
| | |
|
||||
| | |----- h265d generate register file from H.265 syntax info
|
||||
| | |
|
||||
| | |----- jpegd generate register file from jpeg syntax info
|
||||
| | |
|
||||
| | |----- jpege generate register file from jpeg syntax info
|
||||
| | |
|
||||
| | |----- m2vd generate register file from mpeg2 syntax info
|
||||
| | |
|
||||
| | |----- mpg4d generate register file from mpeg4 syntax info
|
||||
| | |
|
||||
| | |----- vp8d generate register file from vp8 syntax info
|
||||
| |
|
||||
| |----- legacy generate new libvpu to include old vpuapi path and new mpp path
|
||||
| |----- legacy generate new libvpu to include old vpuapi path
|
||||
| | and new mpp path
|
||||
| |
|
||||
| |----- test mpp internal video protocol unit test and demo
|
||||
|
|
||||
|----- test mpp buffer/packet component unit test and mpi demo
|
||||
|----- test mpp buffer/packet component unit test and
|
||||
| mpp/mpi/vpu_api demo
|
||||
|
|
||||
|----- out final release binary output directory
|
||||
| |
|
||||
@@ -108,12 +146,16 @@ NOTE:
|
||||
| |
|
||||
| |----- lib library file output directory
|
||||
|
|
||||
|----- osal Operation System Abstract Layer: abstract layer for different operation system
|
||||
|----- osal Operation System Abstract Layer: abstract layer
|
||||
| | for different operation system
|
||||
| |
|
||||
| |----- mem mpi memory subsystem for hardware
|
||||
| |----- allocator supported allocator including Android ion and
|
||||
| | Linux drm
|
||||
| |
|
||||
| |----- android google's android
|
||||
| |
|
||||
| |----- inc osal header file for mpp modules
|
||||
| |
|
||||
| |----- linux mainline linux kernel
|
||||
| |
|
||||
| |----- window microsoft's window
|
||||
@@ -127,31 +169,50 @@ NOTE:
|
||||
|
||||
Here is the mpp implement overall framework:
|
||||
|
||||
+-------------------------+ +--------+
|
||||
| | | |
|
||||
| MPI / MPP | | |
|
||||
| buffer queue manage | | |
|
||||
| | | |
|
||||
+-------------------------+ | |
|
||||
| |
|
||||
+-------------------------+ | |
|
||||
| | | |
|
||||
| codec | | OSAL |
|
||||
| decoder / encoder | | |
|
||||
| | | |
|
||||
+-------------------------+ | |
|
||||
| |
|
||||
+-----------+ +-----------+ | |
|
||||
| | | | | |
|
||||
| parser | | HAL | | |
|
||||
| recoder | | reg_gen | | |
|
||||
| | | | | |
|
||||
+-----------+ +-----------+ +--------+
|
||||
+---------------------------------------+
|
||||
| |
|
||||
| ffmpeg / OpenMax / gstreamer / libva |
|
||||
| |
|
||||
+---------------------------------------+
|
||||
|
||||
+-------------------- MPP ----------------------+
|
||||
| |
|
||||
| +-------------------------+ +--------+ |
|
||||
| | | | | |
|
||||
| | MPI / MPP | | | |
|
||||
| | buffer queue manage | | | |
|
||||
| | | | | |
|
||||
| +-------------------------+ | | |
|
||||
| | | |
|
||||
| +-------------------------+ | | |
|
||||
| | | | | |
|
||||
| | codec | | OSAL | |
|
||||
| | decoder / encoder | | | |
|
||||
| | | | | |
|
||||
| +-------------------------+ | | |
|
||||
| | | |
|
||||
| +-----------+ +-----------+ | | |
|
||||
| | | | | | | |
|
||||
| | parser | | HAL | | | |
|
||||
| | recoder | | reg_gen | | | |
|
||||
| | | | | | | |
|
||||
| +-----------+ +-----------+ +--------| |
|
||||
| |
|
||||
+-------------------- MPP ----------------------+
|
||||
|
||||
+---------------------------------------+
|
||||
| |
|
||||
| kernel |
|
||||
| RK vcodec_service / v4l2 |
|
||||
| |
|
||||
+---------------------------------------+
|
||||
|
||||
|
||||
|
||||
Here is the Media Process Interface hierarchical structure
|
||||
MpiPacket and MpiFrame is the stream I/O data structure.
|
||||
And MpiBuffer encapsulates different buffer implement like Linux's dma-buf and Android's ion.
|
||||
And MpiBuffer encapsulates different buffer implement like Linux's dma-buf and
|
||||
Android's ion.
|
||||
This part is learned from ffmpeg.
|
||||
|
||||
+-------------------+
|
||||
@@ -185,10 +246,12 @@ This part is learned from ffmpeg.
|
||||
|
||||
|
||||
|
||||
Take H.264 deocder for example. Video stream will first queued by MPI/MPP layer, MPP will send the stream to codec layer,
|
||||
codec layer parses the stream header and generates a protocol standard output. This output will be send to HAL to generate
|
||||
register file set and communicate with hardware. Hardware will complete the task and resend information back. MPP notify
|
||||
codec by hardware result, codec output decoded frame by display order.
|
||||
Take H.264 deocder for example. Video stream will first queued by MPI/MPP layer,
|
||||
MPP will send the stream to codec layer, codec layer parses the stream header
|
||||
and generates a protocol standard output. This output will be send to HAL to
|
||||
generate register file set and communicate with hardware. Hardware will complete
|
||||
the task and resend information back. MPP notify codec by hardware result, codec
|
||||
output decoded frame by display order.
|
||||
|
||||
MPI MPP decoder parser HAL
|
||||
|
||||
|
||||
Reference in New Issue
Block a user