Heterogeneous Inter-processor Communication (HIPC)

Introduction

The RD-N2-Automotive FVP contains Armv9.0-A (Primary Compute) and Armv8-R64 (Safety Island) heterogeneous processing elements which share data via the Message Handling Unit (MHUv2) and shared Dynamic random-access memory (DRAM). The MHUv2 is a mailbox controller used for signal transmission and the shared memory is used for data exchange.

As noted in Use-Cases the Primary Compute runs Linux and the Safety Island runs Zephyr.

Virtual Network Communication

RPMsg Protocol

RPMsg (Remote Processor Messaging) is a messaging protocol enabling heterogeneous communication, which can be used by Linux as well as real-time OSes.

In the Linux RPMsg framework, it is implemented on top of the virtio-rpmsg bus and remoteproc framework. The virtio-rpmsg implementation is generic and based on virtio vrings to transmit/receive messages to/from the remote CPU over shared memory.

The remoteproc framework supports SoCs (System-on-a-Chip) which have heterogeneous remote processor devices in asymmetric multiprocessing configurations. In the RD-N2-Automotive FVP, Linux regards the Armv8-R64 processor as its remote processor. A platform-specific driver is added to this framework.

This remote processor cannot be booted by the Armv9-A processor because it needs to monitor the other hardware including the Primary Compute. Therefore, the initial status of the remote processor in the driver is “RPROC_DETACHED”, which means the remote processor has been booted independently from the Armv9-A processor. This driver also implements the notification handler using an MHUv2-based mailbox, which notifies other processors when new messages are sent to the virtual queue. The memory regions of the resource table, vrings and message buffers are configured in the device tree.

On the Armv8-R64 side, Zephyr has imported OpenAMP as an external module. The OpenAMP library implements the RPMsg backend based on virtio, which is compatible with the upstream Linux remoteproc and RPMsg components. This library can be used with the Zephyr kernel or Zephyr applications to behave as an RPMsg backend service for communication with the Armv9-A processors.

Virtual Network Device over RPMsg

RPMsg provides a set of user APIs for RPMsg endpoints to transmit/receive messages to/from the endpoints. These APIs can be used for some basic inter-processor communication. However, many existing user applications are not implemented based on RPMsg APIs. More often, they use BSD sockets for IPC. This is because BSD sockets can shield the difference between inter-processor communication and intra-processor communication, which makes applications generic and portable. In order to meet the needs of such applications, an RPMsg based virtual network device is added to the reference stack.

On the Armv8-R64 side, it creates a network device over an RPMsg endpoint with a specific service name. The RPMsg endpoint announces its existence by sending a name service message to the Armv9-A processors. This message is then handled by the RPMsg bus to create an RPMsg endpoint and a corresponding network device. After that, the network communication is established based on this pair of virtual network devices.

On the Primary Compute side, the RPMsg frame needs to be copied to the skb buffer used by the network stack. When the traffic exceeds the performance limitation, the skb buffer will be exhausted. At this time, the RPMsg frame will be silently dropped.

There are some issues and limitations of the virtual network device over RPMsg. Please refer to the changelog Known Issues and Limitations section.

Xen Guest Device Passthrough in the DomU

In the Virtualization Architecture of the Reference Stack, a virtual network device over RPMsg is supported in the DomU guest of the Primary Compute by means of MHUv2 and shared memory passthrough. Xen is responsible for stage 2 translation of register and memory addresses. This is configured in the Xen tool parameters and passthrough device tree.

Note that Dom0 does not have a communication channel with Safety Island.

As the non-IOMMU device passthrough is not officially supported in the Xen DomU, several downstream patches are added in Linux and Xen.

There is a limitation of the virtual network in the DomU. Please refer to the changelog Limitations section.

Device Tree

In Linux, a remoteproc binding is needed for Armv8-R64 remote processors. It includes MHUv2 transmit/receive channels for signaling and several memory regions for data exchange.

The Linux device tree on Baremetal and Dom0 on Virtualization Architecture, and passthrough device tree for DomU are located at components/primary_compute/devicetree.

In Zephyr, there is an overlay device tree for the network over RPMsg application, which also defines the MHUv2 channels and device memory regions.

The Zephyr overlay device tree for fvp_rdn2_automotive_cortex_r82 board is located at components/safety_island/zephyr/src/boards/arm64/fvp_rdn2_automotive_cortex_r82.