diff --git a/doc/kernel_driver.txt b/doc/kernel_driver.txt new file mode 100644 index 00000000..a627a097 --- /dev/null +++ b/doc/kernel_driver.txt @@ -0,0 +1,70 @@ +Hardware kernel driver design (2016.10.17) +================================================================================ + +Rockchip has two sets of hardware kernel driver. + +The first one is vcodec_service/vpu_service/mpp_service which is a high +performance stateless frame base hardware kernel driver. This driver supports +all available codecs that hardware can provide. This driver is used on Android/ +Linux. + +Here is the vcodec_service kernel driver framework diagram. + + +-------------+ +-------------+ +-------------+ + | client A | | client B | | client C | + +-------------+ +-------------+ +-------------+ + +userspace ++------------------------------------------------------------------------------+ + kernel + + +-------------+ +-------------+ +-------------+ + | session A | | session B | | session C | + +-------------+ +-------------+ +-------------+ + | | | | | | + waiting done waiting done waiting done + | | | | | | ++---+---+ +---+---+ +---+---+ +---+---+ +---+---+ +| task3 | | task0 | | task1 | | task0 | | task0 | ++---+---+ +-------+ +-------+ +-------+ +---+---+ + | | ++---+---+ +---+---+ +| task2 | | task1 | ++-------+ +-------+ + +-----------+ + | service | + +---------+-----------+---------+ + | | | + waiting running done + | | | + +-----+-----+ +-----+-----+ +-----+-----+ + | task A2 | | task A1 | | task C0 | + +-----+-----+ +-----------+ +-----+-----+ + | | + +-----+-----+ +-----+-----+ + | task B1 | | task C1 | + +-----+-----+ +-----+-----+ + | | + +-----+-----+ +-----+-----+ + | task A3 | | task A0 | + +-----------+ +-----+-----+ + | + +-----+-----+ + | task B0 | + +-----------+ + +The principle of this design is to separate user task handling and hardware +resource management and minimize the kernel serial process time between two +hardware process operation. + +The driver uses session as communication channel. Each userspace client (client) +will have a kernel session. Client will commit tasks to session. Then hardware +is managed by service (vpu_service/vcodec_service). Service will provide the +ability to process tasks in sessions. + +When client commits a task to kernel the task will be set to waiting status and +link to both session waiting list and service waiting list. Then service will +get task from waiting list to running list and run. When hardware finishs a task +the task will be moved to done list and put to both service done list and +session done list. Finally client will get the finished task from session. + diff --git a/doc/mpp_buffer.txt b/doc/mpp_buffer.txt new file mode 100644 index 00000000..5daec4f5 --- /dev/null +++ b/doc/mpp_buffer.txt @@ -0,0 +1,57 @@ +Mpp buffer design (2016.10.12) +================================================================================ + +Mpp buffer is the warpper of the buffer used by hardware. Hardware usually can +not use the buffer malloc by cpu. Then we design MppBuffer for different memory +allocator on different platform. Currently it is designed for ion buffer on +Android and drm buffer on Linux. Later may support vb2_buffer in v4l2 devices. + +In order to manage buffer usage in different user mpp buffer module introduces +MppBufferGroup as bufffer manager. All MppBuffer will connect to its manager. +MppBufferGroup provides allocator service and buffer reuse ability. + +Different MppBufferGroup has different allocator. And besides normal malloc/free +function the allocator can also accept buffer from external file descriptor. + +MppBufferGroup has two lists, unused buffer list and used buffer list. When +buffer is free buffer will not be released immediately. Buffer will be moved to +unused list for later reuse. There is a good reason for doing so. When video +resolution comes to 4K the buffer size will be above 12M. It will take a long +time to allocate buffer and generate map table for it. So reusing the buffer +will save the allocate/free time. + +Here is the diagram of Mpp buffer status transaction. + + +----------+ +---------+ + | create | | commit | + +-----+----+ +----+----+ + | | + | | + | +----v----+ + +----------+ unused <-----------+ + | +---------+ | + | | | | + | | | | + +-----v----+ | | +-----+----+ + | malloc | | | | free | + +----------+ | | +----------+ + | inc_ref | +---------+ | dec_ref | + +-----+----+ +-----^----+ + | | + | | + | +---------+ | + +----------> used +-----------+ + +---------+ + | | + | | + | | + | | + | | + +----^----+ + | + | + +----+----+ + | import | + +---------+ + + diff --git a/doc/mpp_design.txt b/doc/mpp_design.txt new file mode 100644 index 00000000..5a088de9 --- /dev/null +++ b/doc/mpp_design.txt @@ -0,0 +1,112 @@ +MPP (Media Process Platform) design (2016.10.12) +================================================================================ + +The mpp is a middleware library for Rockchip SoC's cross platform media process. +The main purpose of mpp is to provide very high performance, high flexibility +and expansibility on multimedia (mainly video and image) process. + +The design target of mpp is to connect different Rockchip hardware kernel driver +and different userspace application. + +Rockchip has two sets of hardware kernel driver. + +The first one is vcodec_service/vpu_service/mpp_service which is a high +performance stateless frame base hardware kernel driver. This driver supports +all available codecs that hardware can provide. This driver is used on Android/ +Linux. + +The second one is v4l2 driver which is developed for ChromeOS. It currently +supports H.264/H.265/vp8/vp9. This driver is used on ChomeOS/Linux. + +Mpp plans to support serval userspace applications including OpenMax, FFmpeg, +gstreamer, libva. + + +Feature explaination +================================================================================ + +1. Cross Platform +The target OS platform including Android, Linux, ChromeOS and windows. Mpp uses +cmake to compile on different platform. + +2. High Performance +Mpp supports sync / async interface to reduce the time blocked in interface. And +mpp internally make hardware and software run parallelly. When hardware is +running sofware will prepare next hardware task at the same time. + +3. High Flexibility +mpi (Media Process Interface) is easy to extend by different control function. +The input/output element packet/frame/buffer is easy to extend different +components. + + +System Diagram +================================================================================ + + +---------------------------------------+ + | | + | ffmpeg / OpenMax / gstreamer / libva | + | | + +---------------------------------------+ + + +-------------------- MPP ----------------------+ + | | + | +-------------------------+ +--------+ | + | | | | | | + | | MPI / MPP | | | | + | | buffer queue manage | | | | + | | | | | | + | +-------------------------+ | | | + | | | | + | +-------------------------+ | | | + | | | | | | + | | codec | | OSAL | | + | | decoder / encoder | | | | + | | | | | | + | +-------------------------+ | | | + | | | | + | +-----------+ +-----------+ | | | + | | | | | | | | + | | parser | | HAL | | | | + | | control | | reg_gen | | | | + | | | | | | | | + | +-----------+ +-----------+ +--------| | + | | + +-------------------- MPP ----------------------+ + + +---------------------------------------+ + | | + | kernel | + | RK vcodec_service / v4l2 | + | | + +---------------------------------------+ + +Mpp is composed of four main sub-modules: + +OSAL (Operation System Abstraction Layer) +This module shutters the differences between different operation systems and +provide basic components including memory, time, thread, log and hardware memory +allocator. + +MPI (Media Process Interface) / MPP +This module is on charge of interaction with external user. Mpi layer has two +ways for user. The simple way - User can use put/get packet/frame function set. +The advanced way - User has to config MppTask and use dequeue/enqueue funciton +set to communicate with mpp. MppTask can carry different meta data and complete +complex work. + +Codec (encoder / decoder) +This module implements the high efficiency internal work flow. The codec module +provides a general call flow for different video format. Software process will +be separated from hardware specified process. The software will communicate with +hardware with a common task interface which combines the buffer information and +codec specified infomation. + +Parser/Controller and hal (Hardware Abstraction Layer) +This layer provides the implement function call of different video format and +different hardware. For decoder parser provide the video stream parse function +and output format related syntax structure to hal. The hal will translate the +syntax structure to register set on different hardware. Current hal supports +vcodec_service kernel driver and plan to support v4l2 driver later. + + diff --git a/readme.txt b/readme.txt index f778ba95..0b92a214 100644 --- a/readme.txt +++ b/readme.txt @@ -9,10 +9,12 @@ Rules: 1. header file arrange rule a. inc directory in each module folder is for external module usage. b. module internal header file should be put along with the implement file. -c. header file should not contain any relative path or absolute path, all include path should be keep in Makefile. +c. header file should not contain any relative path or absolute path, all + include path should be keep in Makefile. 2. compiling system rule a. for cross platform compiling use cmake as the compiling management system. -b. use cmake out-of-source build, final binary and library will be install to out/ directory. +b. use cmake out-of-source build, final binary and library will be install to + out/ directory. 3. header file include order a. MODULE_TAG b. system header @@ -20,7 +22,10 @@ c. osal header d. module header NOTE: -1. when run on window pthreadVC2.dll needed to be copied to the system directory. +1. when run on window pthreadVC2.dll is needed to be copied to system directory. +2. Current MPP only support new chipset like RK3228/RK3229/RK3399/RK1108. + Old chipset like RK29xx/RK30xx/RK31XX/RK3288/RK3368 is not supported due to + lack of corresponding hardware register generation module. ---- top | @@ -36,24 +41,43 @@ NOTE: | | | |----- vc12-x86_64 visual studio 2013 on x86_64 build directory | - |----- inc header file for external usage, including platform header and mpi header + |----- doc design documents of mpp | - |----- mpp Media Process Platform : mpi function private implement and mpp infrastructure (vpu_api private layer) + |----- inc header file for external usage, including + | platform header and mpi header + | + |----- mpp Media Process Platform : mpi function private + | | implement and mpp infrastructure (vpu_api + | | private layer) | | - | |----- common video codec protocol syntax interface for both codec parser and hal + | |----- base base components including MppBuffer, MppFrame, + | | MppPacket, MppTask, MppMeta, etc. | | - | |----- codec all video codec parser, convert stream to protocol structure + | |----- common video codec protocol syntax interface for both + | | codec parser and hal + | | + | |----- codec all video codec parser, convert stream to + | | | protocol structure | | | - | | |----- inc header files provided by codec module for external usage + | | |----- inc header files provided by codec module for + | | | external usage | | | | | |----- dec | | | | | | | |----- dummy decoder parser work flow sample | | | | + | | | |----- h263 + | | | | | | | |----- h264 | | | | | | | |----- h265 | | | | + | | | |----- m2v mpeg2 parser + | | | | + | | | |----- mpg4 mpeg4 parser + | | | | + | | | |----- vp8 + | | | | | | | |----- vp9 | | | | | | | |----- jpeg @@ -80,25 +104,39 @@ NOTE: | | | | | |----- deinter deinterlace function module including pp/iep/rga | | | - | | |----- rkdec rockchip hardware decoder register generation library + | | |----- rkdec rockchip hardware decoder register generation | | | | - | | | |----- h264d generate register file from H.264 structure created by codec parser + | | | |----- h264d generate register file from H.264 syntax info | | | | - | | | |----- h265d generate register file from H.265 structure created by codec parser + | | | |----- h265d generate register file from H.265 syntax info | | | | - | | | |----- vp9d generate register file from vp9 structure created by codec parser + | | | |----- vp9d generate register file from vp9 syntax info | | | | | |----- vpu vpu register generation library | | | - | | |----- h264d generate register file from H.264 structure created by codec parser + | | |----- h263d generate register file from H.263 syntax info | | | - | | |----- h265d generate register file from H.265 structure created by codec parser + | | |----- h264d generate register file from H.264 syntax info + | | | + | | |----- h265d generate register file from H.265 syntax info + | | | + | | |----- jpegd generate register file from jpeg syntax info + | | | + | | |----- jpege generate register file from jpeg syntax info + | | | + | | |----- m2vd generate register file from mpeg2 syntax info + | | | + | | |----- mpg4d generate register file from mpeg4 syntax info + | | | + | | |----- vp8d generate register file from vp8 syntax info | | - | |----- legacy generate new libvpu to include old vpuapi path and new mpp path + | |----- legacy generate new libvpu to include old vpuapi path + | | and new mpp path | | | |----- test mpp internal video protocol unit test and demo | - |----- test mpp buffer/packet component unit test and mpi demo + |----- test mpp buffer/packet component unit test and + | mpp/mpi/vpu_api demo | |----- out final release binary output directory | | @@ -108,12 +146,16 @@ NOTE: | | | |----- lib library file output directory | - |----- osal Operation System Abstract Layer: abstract layer for different operation system + |----- osal Operation System Abstract Layer: abstract layer + | | for different operation system | | - | |----- mem mpi memory subsystem for hardware + | |----- allocator supported allocator including Android ion and + | | Linux drm | | | |----- android google's android | | + | |----- inc osal header file for mpp modules + | | | |----- linux mainline linux kernel | | | |----- window microsoft's window @@ -127,31 +169,50 @@ NOTE: Here is the mpp implement overall framework: - +-------------------------+ +--------+ - | | | | - | MPI / MPP | | | - | buffer queue manage | | | - | | | | - +-------------------------+ | | - | | - +-------------------------+ | | - | | | | - | codec | | OSAL | - | decoder / encoder | | | - | | | | - +-------------------------+ | | - | | - +-----------+ +-----------+ | | - | | | | | | - | parser | | HAL | | | - | recoder | | reg_gen | | | - | | | | | | - +-----------+ +-----------+ +--------+ + +---------------------------------------+ + | | + | ffmpeg / OpenMax / gstreamer / libva | + | | + +---------------------------------------+ + + +-------------------- MPP ----------------------+ + | | + | +-------------------------+ +--------+ | + | | | | | | + | | MPI / MPP | | | | + | | buffer queue manage | | | | + | | | | | | + | +-------------------------+ | | | + | | | | + | +-------------------------+ | | | + | | | | | | + | | codec | | OSAL | | + | | decoder / encoder | | | | + | | | | | | + | +-------------------------+ | | | + | | | | + | +-----------+ +-----------+ | | | + | | | | | | | | + | | parser | | HAL | | | | + | | recoder | | reg_gen | | | | + | | | | | | | | + | +-----------+ +-----------+ +--------| | + | | + +-------------------- MPP ----------------------+ + + +---------------------------------------+ + | | + | kernel | + | RK vcodec_service / v4l2 | + | | + +---------------------------------------+ + Here is the Media Process Interface hierarchical structure MpiPacket and MpiFrame is the stream I/O data structure. -And MpiBuffer encapsulates different buffer implement like Linux's dma-buf and Android's ion. +And MpiBuffer encapsulates different buffer implement like Linux's dma-buf and +Android's ion. This part is learned from ffmpeg. +-------------------+ @@ -185,10 +246,12 @@ This part is learned from ffmpeg. -Take H.264 deocder for example. Video stream will first queued by MPI/MPP layer, MPP will send the stream to codec layer, -codec layer parses the stream header and generates a protocol standard output. This output will be send to HAL to generate -register file set and communicate with hardware. Hardware will complete the task and resend information back. MPP notify -codec by hardware result, codec output decoded frame by display order. +Take H.264 deocder for example. Video stream will first queued by MPI/MPP layer, +MPP will send the stream to codec layer, codec layer parses the stream header +and generates a protocol standard output. This output will be send to HAL to +generate register file set and communicate with hardware. Hardware will complete +the task and resend information back. MPP notify codec by hardware result, codec +output decoded frame by display order. MPI MPP decoder parser HAL