Saturday, April 16, 2016

How to build a cross compliatio enviroment for arm based SBC

It always painful to compile programs for arm-based single board computer such as odroid. This tutorial is aimed to build up a cross compiling system for arm-based programs.

My computer has x64 OpenSuse Tumbleweed installed and the target is to install an armhf(armv7l) ubuntu 14.04 chroot system in it. There will be two sections:

I. use docker to install a ubuntu x64 14.04 container.
II. build up the chroot system inside the ubuntu docker container.

Section I: install ubuntu 14.04 x64 in a docker container.

OpenSuse's document about chroot is quite limited. HDL:chroot describes how to chroot into an armv7 opensuse build from x64 opensuse host. You barely can find any useful documents about how to chroot into an armv7 ubuntu build from x64 opensuse host. I have tried a lot and finally given up the idea to chroot armv7 ubuntu from x64 opensuse host. In terms of documentation and stability, ubuntu is the best linux distro for non advanced users.

if you have a x64 ubuntu host, can you skip this section.

Instead I installed a x64 ubuntu container through docker in the x64 opensuse host and use the x64 ubuntu as host to chroot into an armv7 ubuntu.

The container concept in docker is a bit like the virtual machine though some evaluations from IBM said it has almost the same performances as physical machine in most area. also I have no intent to install a ubuntu os just for compiling purpose. So docker is a good choice compared with virtual machine.

the first step is to install docker:
sudo zypper in docker

there are all kinds of system containers hosted on docker-hub. It is very simple to create a container.
You can "docker search keyword" to search the images whose names contain keyword.

In my case, it is as below:
docker search ubuntu
ubuntu Ubuntu is a Debian-based Linux operating s... 3672 [OK] ubuntu-upstart Upstart is an event-based replacement for ... 61 [OK] torusware/speedus-ubuntu Always updated official Ubuntu docker imag... 25 [OK] ubuntu-debootstrap debootstrap --variant=minbase --components... 24 [OK] rastasheep/ubuntu-sshd Dockerized SSH service, built on top of of... 23 [OK] nickistre/ubuntu-lamp LAMP server on Ubuntu 6 [OK] nickistre/ubuntu-lamp-wordpress LAMP on Ubuntu with wp-cli installed 5 [OK] nimmis/ubuntu This is a docker images different LTS vers... 4 [OK] nuagebec/ubuntu Simple always updated Ubuntu docker images... 4 [OK] maxexcloo/ubuntu Docker base image built on Ubuntu with Sup... 2 [OK] sylvainlasnier/ubuntu Ubuntu 15.10 root docker images with commo... 2 [OK] darksheer/ubuntu Base Ubuntu Image -- Updated hourly 1 [OK] admiringworm/ubuntu Base ubuntu images based on the official u... 1 [OK] jordi/ubuntu Ubuntu Base Image 1 [OK] rallias/ubuntu Ubuntu with the needful 0 [OK] lynxtp/ubuntu 0 [OK] life360/ubuntu Ubuntu is a Debian-based Linux operating s... 0 [OK] esycat/ubuntu Ubuntu LTS 0 [OK] widerplan/ubuntu Our basic Ubuntu images. 0 [OK] teamrock/ubuntu TeamRock's Ubuntu image configured with AW... 0 [OK] webhippie/ubuntu Docker images for ubuntu 0 [OK] konstruktoid/ubuntu Ubuntu base image 0 [OK] ustclug/ubuntu ubuntu image for docker with USTC mirror 0 [OK] suzlab/ubuntu ubuntu 0 [OK] uvatbc/ubuntu Ubuntu images with unprivileged user 0 [OK]
By using "docker pull ubuntu:version", you can download the corresponding image to you local drive.
docker pull ubuntu:latest

#it displays all images pulled from docker-hub.
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE ubuntu latest b72889fa879c 3 days ago 187.9 MB armv7/armhf-ubuntu 14.04.3 73915db97566 8 months ago 184.9 MB
#delete the cached image by specifying IMAGE_ID.
docker rmi IMAGE_ID

Now since you have the image, it is possible to create a container though "docker run"
docker run -it --privileged --name ubuntu ubuntu /bin/bash
#-it means tty interactive
#docker disables many system level operations by default. In my case, I need "--privileged" option to bring full control to the container in order to mount and install qemu-user-static.
#--name option is used to specify a name to the container for easy reference.
#/bin/bash tells docker to use /bin/bash as interpreter.

after running that command, you should see the container is created and it enters the container like ssh.
You can use "uname -a" to confirm it.

type "exit" to exit from the container.

docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 748f592e387b ubuntu "/bin/bash" 2 hours ago Up 2 hours ubuntu
You can see the container is still on

#stop the container from running
docker stop ubuntu

Now you need to re-enter into the container.
docker start ubuntu
docker exec -it ubuntu /bin/bash

It is easy to share files between host and container.
docker cp host_file ubuntu:path
docker cp ubuntu:file host path
#here ubuntu is the container's name, ":path" is the target path inside the container.

Up to now, the ubuntu container is set. The command grammars of docker is also very elegant and easy to use. We will move to section 2. You can also use the container as the test environment of x64 ubuntu.

Section II set up a chroot env for armv7 ubuntu

make sure you enters into ubunut container. commands below are executed in the container.
cd /opt
mkdir rootfs
sudo apt-get install debootstrap schroot qemu qemu-user-static

If there is warning message says:
update-binfmts: warning: Couldn't load the binfmt_misc module.
That is because in latest ubuntu, binfmt_misc is not mounted properly. Using following command to mount binfmt_misc first:
mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc/

#install the necessary packages to build a chroot env.
debootstrap --verbose --variant=buildd --foreign --include=iproute,iputils-ping --arch armhf trusty ./rootfs hold all arm based dists.
#--foreigh options is needed for different arch chroot
#--arch armhf armv7 arch, choose different arch if it is arm64.
#trusty means ubuntu 14.04 release.
#--include specify addtional packages out of standard release.

Cited from
QEMU sports two types of emulation:

syscall emulation: this translates code from one architecture to another, remapping system calls (calls to the kernel) between the two architectures

machine emulation: this emulates a complete computer, including a virtual CPU, a virtual video card, a virtual clock, a virtual SCSI controller etc.

QEMU's syscall emulation is much faster than machine emulation, but both are relatively slow when compared to native programs for your computer. One major drawback of syscall emulation is that it that some syscalls are not emulated, so some programs might now work, also when building programs, these might configure themselve for the features of the kernel they run on instead of what the target architecture actually supports.

For rootfs creation, using syscall emulation is probably fine for core packages, but you might run into issues with higher level ones.

Using syscall emulation

While qemu-arm can be used to run a single armel binary from the command-line, it's impractical to use it to run a program which will fork and execute other programs. Also, because qemu-arm itself uses shared libraries and the dynamic linker/loader from the build environment, it's impractical to copy it in the rootfs to use it with chroot-ed programs. So the following instructions expect that you're using a static version of qemu-arm ("qemu-arm-static"), and that your build environment supports the binfmt-misc module, this modules allows running any executable matching some configured pattern with an interpreter of your choice.

cat /proc/sys/fs/binfmt_misc/qemu-arm
#verify qemu-arm interpreter is register in the container system.

Because we're going to chroot into the rootfs directory, the kernel will look for the interepreter in the chroot; copy the interpreter in the chroot:
cp /usr/bin/qemu-arm-static rootfs/usr/bin

before the chroot, several commands needs to be run to bring up network to chroot env.
mount -o bind /dev     rootfs/dev
mount -o bind /dev/pts rootfs/dev/pts
mount -o bind /proc    rootfs/proc
mount -o bind /sys     rootfs/sys
cp /etc/resolv.conf    rootfs/etc/resolv.conf

now you can chroot into the armv7 ubuntu 14.04
chroot rootfs /bin/bash
/debootstrap/debootstrap --second-stage

uname -a
#check if the system is armv7 arch

#configure system(optional)
locale-gen en_US.UTF-8
locale-gen zh_CN.UTF-8
dpkg-reconfigure locales
export LC_ALL="en_US.UTF-8"
It looks any modification to change the locale in the chroot env won't take effect. Since I only want to use this env to compile programs. So my lazy solution is just to manually export LC_ALL each time.

#add ubuntu repo
echo "deb trusty main restricted universe multiverse" > /etc/apt/sources.list
echo "deb trusty-security main restricted universe multiverse" >> /etc/apt/sources.list
echo "deb trusty-updates main restricted universe multiverse" >> /etc/apt/sources.list
echo "deb trusty-backports main restricted universe multiverse" >> /etc/apt/sources.list

#update system
apt-get update
apt-get upgrade

every time when the docker container is stopped, it looks like docker will clean up the content inside /proc/sys/fs/binfmt_misc. You need to run
update-binfmts --import 
to bring the qemu-arm registration back.

In the end, you can compile a program to make sure the cross compiling system is OK.

phantomjs might be a good example because its dependencies are quite complex though the compiling time is very long.

following the instructions in

git crashes in the chroot env. So my solution is to git the src in the container and move that into rootfs.

it takes around 7-8 hours on my 16GB RAM and quad core HT CPU computer, which i think is pretty good considering three layer stacking structure.


If you want to exit the chroot env. You can type "exit" twice to come back all the way to x64 opensuse host.

No comments: