Accelerating Face Detection on GMSL-based Camera with Vitis AI Part 3

Please Read Accelerating Face Detection on GMSL-based Camera with Vitis AI Part 2

As of now, Vitis AI runtime libraries are provided in a docker container.  While it is possible to copy the sources for facedetect into the runtime docker and compile it there, this tutorial demonstrates an alternative approach which allows building in petalinux without using the docker for the build (though the runtime docker is still needed on the host, at least the first time the repository is created). The approach is to copy the dependencies out of the docker into a local repository that is fetched by a Yocto recipe during the petalinux build. The makefile target is shown below:

The primary tasks of this target are: 1) prepare the directory that’s shared between docker and the host 2) execute the scripts/copy_libs.sh scripts inside the docker container and 3) cleanup. The scripts/copy_libs.sh script simply copies a number of libraries and header files out of the cross compile environment in the docker container into repos/Vitis-AI/Rebuild-Model-Zoo/libs directory. Once complete, the makefile organizes it into repos/vitis-ai-pfm. In a later step, Petalinux recipes will pick up these files and insert them into the target’s rootfs.

Now that the .xsa file is available, a petalinux project can be created to generate the linux image, boot files, and sysroot.

source <petalinux_install>/2019.2/settings.sh

make petalinux

This target does several things:

• Create an initial petalinux project using --template zynqMP
• Copies pre-defined petalinux config into the project
• Copies pre-defined rootfs config into the project
• Imports the .xsa file
• Copies pre-defined device tree into the project
• Copies the following recipes into the project
  • bringupcam - Initialization software for the image sensor, serializer, deserializer, along with their associated V4L2 and Media Control configuration
  • autostart – Scripts that run automatically at boot/login to prepare environment and execute other initialization tasks (including the bringupcam application)
  • facedetect – Vitis AI application which uses OpenCV to capture frames, process them with the DPU, draws bounding boxes around faces, and outputs the result for display
  • recipes-apps/vitis-ai – Imports runtime dependencies/headers that are necessary for Vitis AI and the DPU
  • recipes-kernel – Import dummy_src patch
• Configures the petalinux project to enable the imported recipes
• Builds the petalinux project
• Packages the boot files

The petalinux configuration is in petalinux/configs/config_2019.2 and contains the following interesting modifications

•  ZCU102 machine is targeted

•  Rootfs is set to EXT4 partition at /dev/mmcblk0p2 (second partition of the SD card)

• Kernel boot arguments are modified to mount rootfs from /dev/mmcblk0p2

The rootfs configuration is in petalinux/configs/rootfs_config_2019.2 and contains the following interesting modifications

• i2c-tools is enabled which is useful for debug

• xrt, xrt-dev, and zocl to support acceleration

• gdb for debugging and printing dot graphs from media-ctl (also useful for debugging)

• json-c for DPU metadata

• v4l and media-ctl for configuration of image capture pipeline

• matchbox-desktop and window manager for X11

• Gstreamer package group for processing pipeline creation

• OpenCV package group for handling for DPU processing and bounding box drawing

• V4l package group

• X11 package group for display

• Custom apps and recipes are also enabled

A number of device tree modifications are necessary to set up the image capture pipeline and also the Zynq OpenCL framework. The device tree customizations are in petalinux/devicetree/system-user_2019.2.dtsi.

The MIPI RX Subsystem and the Demosaic are connected together normally with appropriate settings for the design

The output of the Demosaic feeds a video capture node for the Framebuffer Write IP

Normally, the input of the MIPI CSI2 RX Subsystem will be connected to an upstream V4L2 node for the image sensor driver which would complete the pipeline. In this case, the camera sensor driver is not so simple because of the serializer/deserializer in the I2C path that need to be configured and managed. The bringupcam userspace application takes the place of this driver and handles all the I2C accesses to the serializer/deserializer and image sensor. However, the MIPI CSI2 RX subsystem input cannot be left unconnected in the devicetree or else a /dev/media node will not be correctly created. One of 3 options were considered to solve this problem:

1. Use an AXI Stream Switch IP in static routing mode (which can otherwise be used to model changes to the V4L2 pipeline configuration without any actual hardware accesses) – This would not work because the xlnx,num-si-slots property must be >1 or else the driver will fail
2. Add a ‘dummy’ node in the device tree that uses the Xilinx TPG driver – This option also will not work directly because the TPG driver actually needs to access registers in the TPG to configure it. If no TPG exists in the PL design, the driver will hang the system
   • This option could be extended to actually include a ‘dummy’ TPG/VTC in the FPGA design but the AXI Stream interface left unconnected. This would satisfy the driver, but is not ideal because it is wasteful of valuable FPGA resources
3. Custom driver implementation to act as a dummy source to satisfy the pipeline

Ultimately, option 3 above was chosen for the cleanest implementation. This driver is provided as a kernel patch in repos/dummy_src_driver.

This dummy_src driver needs a device tree node as well. Since it’s implementation is based off the Xilinx TPG driver, the properties are similar.

Finally, the ZOCL driver needs to be informed about the interrupts that are available to the AXI Interrupt controller

The bringupcam application is modular software service whose purpose is to perform I2C configuration of an image sensor, serializer, and deserializer on the FMC card. It looks for hardware.conf file in /etc in the target filesystem. It is a relatively large codebase and the implementation/operation is beyond the scope of this article. The source is provided for those who are interested.

petalinux/recipes/autostart is a Yocto recipe that inserts some initialization scripts into the root filesystem that run automatically. petalinux/recipes/autostart/autostart.bb is the recipe itself shown here

This recipe is picked up by Petalinux and processed by Yocto back end during the build. The SRC_URI variable indicates the two files that are installed into the rootfs (autostart.sh and loginenv.sh). INITSCRIPT_NAME and INITSCRIPT_PARAMS are instructions for how update-rc.d is to handle inserting autostart.sh as an init script. The do_install() callback handles the install/copying of the scripts into /etc/init.d and /etc/profile.d in the target rootfs.

The autostart.sh script itself is shown below

Note that the dpu.xclbin is copied from the first partition on the SD card into /usr/lib. When the facedetect application runs, XRT locates and loads it.

The facedetect application is the primary application for this tutorial. The main() function is shown below

The OpenCV VideoCapture class is used to grab frames from a Gstreamer capture pipeline. By default, v4l2src uses /dev/video0. The cur_frame Mat object is passed to the DPU through the ml­_task (which is itself a xilinx::ai::FaceDetect object). The output of the face detection algorithm is passed to process­_result() function which draws bounding boxes using OpenCV rect() function. The final output frames are written to OpenCV VideoWriter class to a Gstreamer display pipeline that renders the display using the Mali GPU onto the Displayport Interface (glimagesink plugin). XR24 is the source code that corresponds to the RGB pixel format that is used. See https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/videodev.html#videodev

petalinux/recipes/vitis-ai simply contains a bitbake recipe which is shown below

Note that the FILESEXTRAPATHS_prepend and SRC_URI point to repos/vitis-ai-pfm created previously. There are a number of dependencies and configurations to package the libraries. The runtime dependencies are then installed into the target rootfs.

Continue to Accelerating Face Detection on GMSL-based Camera with Vitis AI Part 4