Heterogeneity of Computing Environments Using Cross-Compilation

With the advent of open-source software and the acceptance of these solutions in creating complex systems, the ability to develop applications that can run seamlessly across multiple hardware platforms becomes inherently important. There is a constant need to develop the software on one architecture but have the capability to execute these on other target architectures. One common technique to achieve this is cross-compilation of the application for the target architecture. 

Cross-compilation is significant in embedded systems where the intent is to run applications on specialized hardware like ARM and PowerPC boards. These systems are resource-constrained and hence a direct compilation is not an option. Thus, developers will leverage the common x86 architecture as a host and use toolchains specific to the target hardware, generating binaries compatible with the target hardware. 

This article covers one such case study where cross-compilation of an open-source package was done for PowerPC. The article will cover the details of the tools and toolchains leveraged and a step-by-step tutorial on how cross-compilation was achieved for this architecture. 

The Problem Statement  

Given a target board with PowerPC architecture, the intent was to add L3 routing capabilities to this board. For this purpose, a popular open-source routing protocol suite, FRRouting (FRR), was considered. 

FRR is a protocol suite that enables any Linux machine to behave like a full-fledged router. It is packaged for amd64, arm64, armhf, and i386 but not for PowerPC. This necessitated FRR’s cross-compilation.

Table 1. Difference in Build and Target platform

The Cross-Compilation Journey

There are two major stages in cross-compilation: 

Configuring the Build Environment and Pre-Compiled Toolchain

1. Install the required build tools in the environment. Common build tools include autoconf, make, cmake, build-essentials, pkg-config, libtool, etc. 

2. Set up the pre-compiled toolchain specific to the target host environment. CPU/Board vendors provide their own architecture-specific toolchains. The target board-specific toolchain was obtained from the vendor’s product website. 

3. The toolchain comes with an environment file, which is used to set the environment variables like CC, GCC, PKG_CONFIG_PATH, etc, that are required for cross-compilation. Edit the environment file /opt/fsl-qoriq/2.0/environment-setup-ppce5500-fsl-linux and update the path of variables with respect to the path of the toolchain directory. 

export SDKTARGETSYSROOT=/opt/fsl-qoriq/2.0/sysroots/ppce5500-fsl-linux

export PATH=/opt/fsl-qoriq/2.0/sysroots/x86_64-fslsdk-linux/usr/bin:/opt/fsl-qoriq/2.0/sysroots/x86_64-fslsdk-linux/usr/bin/../x86_64-fslsdk-linux/bin:/opt/fsl-qoriq/2.0/sysroots/x86_64-fslsdk-linux/usr/bin/powerpc-fsl-linux:/opt/fsl-qoriq/2.0/sysroots/x86_64-fslsdk-linux/usr/bin/powerpc-fsl-linux-uclibc:/opt/fsl-qoriq/2.0/sysroots/x86_64-fslsdk-linux/usr/bin/powerpc-fsl-linux-musl:$PATH

export CCACHE_PATH=/opt/fsl-qoriq/2.0/sysroots/x86_64-fslsdk-linux/usr/bin:/opt/fsl-qoriq/2.0/sysroots/x86_64-fslsdk-linux/usr/bin/../x86_64-fslsdk-linux/bin:/opt/fsl-qoriq/2.0/sysroots/x86_64-fslsdk-linux/usr/bin/powerpc-fsl-linux:/opt/fsl-qoriq/2.0/sysroots/x86_64-fslsdk-linux/usr/bin/powerpc-fsl-linux-uclibc:/opt/fsl-qoriq/2.0/sysroots/x86_64-fslsdk-linux/usr/bin/powerpc-fsl-linux-musl:$CCACHE_PATH

export PKG_CONFIG_SYSROOT_DIR=$SDKTARGETSYSROOT

export PKG_CONFIG_PATH=$SDKTARGETSYSROOT/usr/lib/pkgconfig

export CONFIG_SITE=/opt/fsl-qoriq/2.0/site-config-ppce5500-fsl-linux

export PYTHONHOME=/opt/fsl-qoriq/2.0/sysroots/x86_64-fslsdk-linux/usr unset command_not_found_handle

export CC="powerpc-fsl-linux-gcc -m32 -mhard-float -mcpu=e5500 --sysroot=$SDKTARGETSYSROOT"

export CXX="powerpc-fsl-linux-g++ -m32 -mhard-float -mcpu=e5500 --sysroot=$SDKTARGETSYSROOT"

export CPP="powerpc-fsl-linux-gcc -E -m32 -mhard-float -mcpu=e5500 --sysroot=$SDKTARGETSYSROOT"

export AS="powerpc-fsl-linux-as "

export LD="powerpc-fsl-linux-ld --sysroot=$SDKTARGETSYSROOT"

export GDB=powerpc-fsl-linux-gdb

export STRIP=powerpc-fsl-linux-strip

export RANLIB=powerpc-fsl-linux-ranlib

export OBJCOPY=powerpc-fsl-linux-objcopy

export OBJDUMP=powerpc-fsl-linux-objdump

export AR=powerpc-fsl-linux-ar

export NM=powerpc-fsl-linux-nm

export M4=m4

export TARGET_PREFIX=powerpc-fsl-linux-

export CONFIGURE_FLAGS="--target=powerpc-fsl-linux --host=powerpc-fsl-linux --build=x86_64-linux --with-libtool-sysroot=$SDKTARGETSYSROOT"

export CFLAGS=" -O2 -pipe -g -feliminate-unused-debug-types"

export CXXFLAGS=" -O2 -pipe -g -feliminate-unused-debug-types"

export LDFLAGS="-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed"

export CPPFLAGS=""

export KCFLAGS="--sysroot=$SDKTARGETSYSROOT"

export OECORE_DISTRO_VERSION="2.0"

export OECORE_SDK_VERSION="2.0"

export ARCH=powerpc

export CROSS_COMPILE=powerpc-fsl-linux-

Resolving Dependencies in Pre-Compiled Toolchain

Each software has its own set of dependencies(tools/libraries), which needs to be resolved before the cross-compilation.

Depending on the package availability of these tools/libraries in the target architecture, there are two options to resolve them. Either install them directly from available packages or cross-compile them as well from the source.

Specific to the FRR compilation, libpcre2, libyang, clippy, libelf, and json-c libraries need to be built from the source. Aside from these, protobuf and libcap library packages were available for PPC(PowerPC) arch, which can be installed directly into the toolchain.

1. Installing Libraries from Packages

Library packages available for the target architecture can be installed into the toolchain sysroot using two ways:

1. The first way uses Ubuntu/Debian-based dpkg-debtool for directly installing Debian packages as mentioned below: 
   Shell

    

   $ dpkg-deb -x <pkg_name>.deb <toolchain_directory_path>

   ​

   #Example of libcap: 

   $ wget http://launchpadlibrarian.net/222364908/libcap-dev_2.24-12_powerpc.deb

   $ dpkg-deb -x libcap-dev_2.24-12_powerpc.deb /opt/fsl-qoriq/2.0/sysroots/ppce5500-fsl-linux/

   Note:

   • Download all dependency packages and install them in order.
   • Library packages may get installed to different directory structure. Copy those library files to the correct directories as per the toolchain.
2. In the second way debian/rpm packages are extracted and manually placed to the toolchain directory path as mentioned below:
   1. For extracting debian package, use ar and tartools as mentioned below:
      Plain Text

       

      $ ar -x <package>.deb

      $ tar -xJf data.tar.xz

      Note: This method is useful for systems without dpkg-deb support.

   2. For rpm package extraction, use rpm2cpiotool as the command below:
      Plain Text

       

      $ rpm2cpio <package>.rpm | cpio -idmv

   3. Package extraction and file placement with libcap example:

      Shell

       

      # Extract .deb package

      $ ar -x libcap-dev_2.24-12_powerpc.deb

      ​

      # Three files will be extracted (control.tar.gz, data.tar.xz, debian-binary)

      $ ls

      control.tar.gz  data.tar.xz  debian-binary  libcap-dev_2.24-12_powerpc.deb

      ​

      # data.tar.xz has the package’s program files

      # control.tar.gz has metadata of package

      # debian-binary has version of deb file format

      ​

      # Untar data.tar.xz to extract the package’s program files.

      $ tar -xJf data.tar.xz

      ​

      # NOTE: rpm2cpio <package>.rpm will directly extract program files.

      ​

      # Do the same for all dependent debian or rpm packages at the same path, this will extract all program files and symlinks required in same directory structure. 

      # Extracted files under usr/lib, usr/include and usr/bin directories should be copied to toolchain_directory_path’s /usr directory, alongside the existing files already present.

      ​

      $ cp usr/include/sys/capability.h /opt/fsl-qoriq/2.0/sysroots/ppce5500-fsl-linux/usr/include/sys/

      ​

   4. To verify if the packages/libraries got installed successfully, run the command below:

      Shell

       

      # Make sure to export PKG_CONFIG_PATH variable from toolchain’s env file

      $ pkg-config --list-all | grep <package_library_name>

      Note: Install pkg-config if not already installed.

2. Cross-Compiling Libraries

Library packages are not available for the target arch and are compiled from the source. Before starting compilation, load the environment file packaged with the toolchain to set all the necessary parameters required for cross-compilation.

$ source <env_file_path>

Follow the compilation steps given in the library’s README file. Additionally, set the below parameters in the build procedure steps:

1. Set --host parameter when running ./configurescript
   Shell

    

   $ ./configure --host=<target_host_parameter>

   ​

   # Example:

   $ ./configure --host=powerpc-fsl-linux

   ​

   Note:

   • <target_host_parameter> is the system for which the library/tool is being built. It can be found in the environment file of the toolchain. It is a common prefix found in $CC, $LD, etc.
   • There will be two types of dependent libraries. One is only required for the compilation process, and the other is a dependency requirement for execution at the target host. Set the --host parameter accordingly.
2. When using “make install” for building dependent libraries, set DESTDIR to the toolchain sysroot directory.
   Shell

    

   $ make DESTDIR=<toolchain_directory_path> install

   ​

   Examlple:

   $ make DESTDIR=/opt/fsl-qoriq/2.0/sysroots/ppce5500-fsl-linux/ install

   ​

Conclusion

The differences in system architectures, libraries, dependencies, and toolchains make cross-compilation a complex technique to execute. To ease out complexities, this article uncovers the phases of cross-compilation. FRR and PowerPC were taken as examples for the desired software and the target hardware, respectively. However, the steps covered in this article provide a reasonable strategy for the cross-compilation of any software on a specific hardware. The environment settings, the toolchain, and the dependencies will vary based on the requirements.