How to build a cross compliatio enviroment for arm based SBC

It always painful to compile programs for arm-based single board computer such as odroid. This tutorial is aimed to build up a cross compiling system for arm-based programs.

My computer has x64 OpenSuse Tumbleweed installed and the target is to install an armhf(armv7l) ubuntu 14.04 chroot system in it. There will be two sections:

I. use docker to install a ubuntu x64 14.04 container.
II. build up the chroot system inside the ubuntu docker container.

Section I: install ubuntu 14.04 x64 in a docker container.

OpenSuse's document about chroot is quite limited. HDL:chroot describes how to chroot into an armv7 opensuse build from x64 opensuse host. You barely can find any useful documents about how to chroot into an armv7 ubuntu build from x64 opensuse host. I have tried a lot and finally given up the idea to chroot armv7 ubuntu from x64 opensuse host. In terms of documentation and stability, ubuntu is the best linux distro for non advanced users.

if you have a x64 ubuntu host, can you skip this section.

Instead I installed a x64 ubuntu container through docker in the x64 opensuse host and use the x64 ubuntu as host to chroot into an armv7 ubuntu.

The container concept in docker is a bit like the virtual machine though some evaluations from IBM said it has almost the same performances as physical machine in most area. also I have no intent to install a ubuntu os just for compiling purpose. So docker is a good choice compared with virtual machine.

the first step is to install docker:

sudo zypper in docker

there are all kinds of system containers hosted on docker-hub. It is very simple to create a container.
You can "docker search keyword" to search the images whose names contain keyword.

In my case, it is as below:

docker search ubuntu

ubuntu                            Ubuntu is a Debian-based Linux operating s...   3672      [OK]       
ubuntu-upstart                    Upstart is an event-based replacement for ...   61        [OK]       
torusware/speedus-ubuntu          Always updated official Ubuntu docker imag...   25                   [OK]
ubuntu-debootstrap                debootstrap --variant=minbase --components...   24        [OK]       
rastasheep/ubuntu-sshd            Dockerized SSH service, built on top of of...   23                   [OK]
nickistre/ubuntu-lamp             LAMP server on Ubuntu                           6                    [OK]
nickistre/ubuntu-lamp-wordpress   LAMP on Ubuntu with wp-cli installed            5                    [OK]
nimmis/ubuntu                     This is a docker images different LTS vers...   4                    [OK]
nuagebec/ubuntu                   Simple always updated Ubuntu docker images...   4                    [OK]
maxexcloo/ubuntu                  Docker base image built on Ubuntu with Sup...   2                    [OK]
sylvainlasnier/ubuntu             Ubuntu 15.10 root docker images with commo...   2                    [OK]
darksheer/ubuntu                  Base Ubuntu Image -- Updated hourly             1                    [OK]
admiringworm/ubuntu               Base ubuntu images based on the official u...   1                    [OK]
jordi/ubuntu                      Ubuntu Base Image                               1                    [OK]
rallias/ubuntu                    Ubuntu with the needful                         0                    [OK]
lynxtp/ubuntu                     https://github.com/lynxtp/docker-ubuntu         0                    [OK]
life360/ubuntu                    Ubuntu is a Debian-based Linux operating s...   0                    [OK]
esycat/ubuntu                     Ubuntu LTS                                      0                    [OK]
widerplan/ubuntu                  Our basic Ubuntu images.                        0                    [OK]
teamrock/ubuntu                   TeamRock's Ubuntu image configured with AW...   0                    [OK]
webhippie/ubuntu                  Docker images for ubuntu                        0                    [OK]
konstruktoid/ubuntu               Ubuntu base image                               0                    [OK]
ustclug/ubuntu                    ubuntu image for docker with USTC mirror        0                    [OK]
suzlab/ubuntu                     ubuntu                                          0                    [OK]
uvatbc/ubuntu                     Ubuntu images with unprivileged user            0                    [OK]

By using "docker pull ubuntu:version", you can download the corresponding image to you local drive.

docker pull ubuntu:latest

#it displays all images pulled from docker-hub.

docker images

REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
ubuntu               latest              b72889fa879c        3 days ago          187.9 MB
armv7/armhf-ubuntu   14.04.3             73915db97566        8 months ago        184.9 MB

#delete the cached image by specifying IMAGE_ID.

docker rmi IMAGE_ID

Now since you have the image, it is possible to create a container though "docker run"

docker run -it --privileged --name ubuntu ubuntu /bin/bash

#-it means tty interactive
#docker disables many system level operations by default. In my case, I need "--privileged" option to bring full control to the container in order to mount and install qemu-user-static.
#--name option is used to specify a name to the container for easy reference.
#/bin/bash tells docker to use /bin/bash as interpreter.

after running that command, you should see the container is created and it enters the container like ssh.
You can use "uname -a" to confirm it.

type "exit" to exit from the container.

docker ps -a

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
748f592e387b        ubuntu              "/bin/bash"         2 hours ago         Up 2 hours                              ubuntu

You can see the container is still on

#stop the container from running

docker stop ubuntu

Now you need to re-enter into the container.

docker start ubuntu
docker exec -it ubuntu /bin/bash

It is easy to share files between host and container.

docker cp host_file ubuntu:path
docker cp ubuntu:file host path

#here ubuntu is the container's name, ":path" is the target path inside the container.

Up to now, the ubuntu container is set. The command grammars of docker is also very elegant and easy to use. We will move to section 2. You can also use the container as the test environment of x64 ubuntu.

Section II set up a chroot env for armv7 ubuntu

make sure you enters into ubunut container. commands below are executed in the container.

cd /opt
mkdir rootfs
sudo apt-get install debootstrap schroot qemu qemu-user-static

If there is warning message says:
update-binfmts: warning: Couldn't load the binfmt_misc module.
That is because in latest ubuntu, binfmt_misc is not mounted properly. Using following command to mount binfmt_misc first:

mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc/

#install the necessary packages to build a chroot env.

debootstrap --verbose --variant=buildd --foreign --include=iproute,iputils-ping --arch armhf trusty ./rootfs http://ports.ubuntu.com/ubuntu-ports/

#ports.ubuntu.com/ubuntu-ports hold all arm based dists.
#--foreigh options is needed for different arch chroot
#--arch armhf armv7 arch, choose different arch if it is arm64.
#trusty means ubuntu 14.04 release.
#--include specify addtional packages out of standard release.

Cited from https://wiki.ubuntu.com/ARM/RootfsFromScratch/QemuDebootstrap#Creating_a_filesystem_from_a_directory
QEMU sports two types of emulation:

syscall emulation: this translates code from one architecture to another, remapping system calls (calls to the kernel) between the two architectures

machine emulation: this emulates a complete computer, including a virtual CPU, a virtual video card, a virtual clock, a virtual SCSI controller etc.

QEMU's syscall emulation is much faster than machine emulation, but both are relatively slow when compared to native programs for your computer. One major drawback of syscall emulation is that it that some syscalls are not emulated, so some programs might now work, also when building programs, these might configure themselve for the features of the kernel they run on instead of what the target architecture actually supports.

For rootfs creation, using syscall emulation is probably fine for core packages, but you might run into issues with higher level ones.

Using syscall emulation

While qemu-arm can be used to run a single armel binary from the command-line, it's impractical to use it to run a program which will fork and execute other programs. Also, because qemu-arm itself uses shared libraries and the ld.so dynamic linker/loader from the build environment, it's impractical to copy it in the rootfs to use it with chroot-ed programs. So the following instructions expect that you're using a static version of qemu-arm ("qemu-arm-static"), and that your build environment supports the binfmt-misc module, this modules allows running any executable matching some configured pattern with an interpreter of your choice.

cat /proc/sys/fs/binfmt_misc/qemu-arm

#verify qemu-arm interpreter is register in the container system.

Because we're going to chroot into the rootfs directory, the kernel will look for the interepreter in the chroot; copy the interpreter in the chroot:

cp /usr/bin/qemu-arm-static rootfs/usr/bin

before the chroot, several commands needs to be run to bring up network to chroot env.

mount -o bind /dev     rootfs/dev
mount -o bind /dev/pts rootfs/dev/pts
mount -o bind /proc    rootfs/proc
mount -o bind /sys     rootfs/sys
cp /etc/resolv.conf    rootfs/etc/resolv.conf

now you can chroot into the armv7 ubuntu 14.04

chroot rootfs /bin/bash
/debootstrap/debootstrap --second-stage

uname -a

#check if the system is armv7 arch

#configure system(optional)

locale-gen en_US.UTF-8
locale-gen zh_CN.UTF-8
dpkg-reconfigure locales
export LC_ALL="en_US.UTF-8"

It looks any modification to change the locale in the chroot env won't take effect. Since I only want to use this env to compile programs. So my lazy solution is just to manually export LC_ALL each time.

#add ubuntu repo

echo "deb http://ports.ubuntu.com/ trusty main restricted universe multiverse" > /etc/apt/sources.list
echo "deb http://ports.ubuntu.com/ trusty-security main restricted universe multiverse" >> /etc/apt/sources.list
echo "deb http://ports.ubuntu.com/ trusty-updates main restricted universe multiverse" >> /etc/apt/sources.list
echo "deb http://ports.ubuntu.com/ trusty-backports main restricted universe multiverse" >> /etc/apt/sources.list

#update system

apt-get update
apt-get upgrade

every time when the docker container is stopped, it looks like docker will clean up the content inside /proc/sys/fs/binfmt_misc. You need to run

update-binfmts --import 

to bring the qemu-arm registration back.

In the end, you can compile a program to make sure the cross compiling system is OK.

phantomjs might be a good example because its dependencies are quite complex though the compiling time is very long.

following the instructions in http://phantomjs.org/build.html

git crashes in the chroot env. So my solution is to git the src in the container and move that into rootfs.

it takes around 7-8 hours on my 16GB RAM and quad core HT CPU computer, which i think is pretty good considering three layer stacking structure.

Enjoy!

If you want to exit the chroot env. You can type "exit" twice to come back all the way to x64 opensuse host.