Mesos on Raspberry Pi

At this Friday, I saw a email mentioned compile Mesos for Raspberry Pi.
It is a bit interesting and attracting me, it reminds me the happy time about playing Raspberry Pi when I still was a student two years ago.
Now Raspberry Pi is more cheaper over that time (Raspberry Pi Zero ID: 2885 - $5.00).

To make Mesos compiled for Raspberry Pi successfully, we have three possible approaches:

  1. Compile Mesos on Raspberry Pi directly.
  2. Use cross compiling tool chains to compile Mesos for Raspberry Pi in ARM.
  3. Compile Mesos on Raspberry Pi virtual machine.

For first approach, there are a lot of header files in Mesos stout depenency. This requires huge memory when compiling Mesos and make it impossible to compile Mesos on current Raspberry Pi because of lack of memory.

For second approach, it should work theoretically. But Mesos would check the dependent libraries by compiling and running it during configure stage. This mean we need remove those checks in configure file manually. After I removed those checks, I found I fall into another trap: dependencies loops. The minimize dependent libraries for Mesos are zlib, apr, apr-util, subversion. I need to cross compiling and perpare them for Mesos firstly. But I blocked by cross compiling subversion eventually. I found need to solve its dependencies, its dependencies dependencies, its dependencies dependencies dependencies and so on. Seems the only remain possible way is to compile Mesos on Raspberry Pi virtual machine.

qemu-arm could simulate the Raspberry Pi architecture in x86 machine. I found a related vagrant file in github which make it easiler to set up the Raspberry Pi develop environment. However, it has already been out of maintain over 3 years. The debian version it used (Wheezy) is not new enough to compile Mesos as well. So I create a docker image haosdent/raspberry based on the newest Raspberry Pi operate system(Jessie) according to its puppet files.

By this docker image, we could start to compile Mesos for Raspberry Pi.

1. Launch the Raspberry Pi Development Environment.

1
docker run -i -t --net=host --volume=`pwd`/mesos:/root/mesos haosdent/raspberry /bin/bash

At above command, I mount my local Mesos code into docker container. Keep in mind here, we have to use the master branch of Mesos because of the bundle zookeeper package has updated in the master branch recently. Otherwise we would fail when compiling zookeeper on Raspberry Pi if use the old versions of Mesos.

Then we use sb2 -eR to enter the virtual machine of Raspberry Pi. The shell prompt looks like

1
[SB2 emulate raspberry] root@raspberry ~ #

if you enter the virtual machine of Raspberry Pi successfully.

2. Patch pivot_root in Mesos Code

Currenlty Mesos still would fail when compiling on Raspberry Pi because of the undefined of __NR_pivot_root. I take a look at @lyda’s patch for this before. However, it looks incorrect for me. So I modify it to

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
diff --git a/src/linux/fs.cpp b/src/linux/fs.cpp
index 2087b4a..f29ce8a 100644
--- a/src/linux/fs.cpp
+++ b/src/linux/fs.cpp
@@ -444,6 +444,16 @@ Try<Nothing> pivot_root(
// number for 'pivot_root' on the powerpc architecture, see
// https://w3challs.com/syscalls/?arch=powerpc_64
int ret = ::syscall(203, newRoot.c_str(), putOld.c_str());
+#elif __thumb__
+ // A workaround for arm thumb mode. The magic number '218' is the syscall
+ // number for 'pivot_root' on the arm thumb mode, see
+ // https://w3challs.com/syscalls/?arch=arm_thumb
+ int ret = ::syscall(218, newRoot.c_str(), putOld.c_str());
+#elif __arm__
+ // A workaround for arm. The magic number '9437402' is the syscall
+ // number for 'pivot_root' on the arm architecture, see
+ // https://w3challs.com/syscalls/?arch=arm_strong
+ int ret = ::syscall(9437402, newRoot.c_str(), putOld.c_str());
#else
#error "pivot_root is not available"
#endif

3. Following Mesos Getting Started Guide

After finish above prepare works, we could use following commands to start to compile Mesos.

1
2
3
4
5
6
7
8
9
apt-get install -y tar wget git
apt-get install -y autoconf libtool
apt-get -y install build-essential python-dev python-boto libcurl4-nss-dev libsasl2-dev libsasl2-modules maven libapr1-dev libsvn-dev
cd ~/mesos
./bootstrap
mkdir build
cd build
../configure --disable-python --disable-java
make

Noted that we disable java and python when compiling. I encounter some tricky problems when compiled with java in Raspberry Pi and not yet to
investigate.

The stage would take quite a long time. It taked more than 8 hours in my slow machine. You could use make -j <number of cores> if your machine have more cpu cores.

4. Launch Your First Mesos Task in the Raspberry Pi

You should build it successfully if your finished above stages. And we could copy the whole packages to our real Raspberry Pi machine. However, some Mesos feautres don’t work in the Raspberry Pi correctly. For example, replicated_log and cgroups. So we need use following commands to start Mesos Master and Mesos Agent.

4.1 Start Mesos Master

1
./bin/mesos-master.sh --work_dir=/tmp/mesos --ip=127.0.0.1 --hostname=127.0.0.1 --registry=in_memory

4.2 Start Mesos Agent

1
./bin/mesos-slave.sh --work_dir=/tmp/mesos --ip=127.0.0.1 --hostname=127.0.0.1 --master=127.0.0.1:5050 --launcher=posix

4.3 Submit a Mesos Task

After starting these Mesos components successfully, we can use mesos-execute to launch our Mesos task.

1
2
3
export LIBPROCESS_IP=127.0.0.1
export LIBPROCESS_HOSTNAME=127.0.0.1
./src/mesos-execute --master=127.0.0.1:5050 --name=test --command="ls /"
1
./bin/mesos-slave.sh --work_dir=/tmp/mesos --ip=127.0.0.1 --hostname=127.0.0.1 --master=127.0.0.1:5050 --launcher=posix

Now you should found something like

1
2
3
4
5
6
7
8
I0424 07:16:12.291718  2722 scheduler.cpp:177] Version: 0.29.0
Subscribed with ID 'b383e094-b7f2-4841-a737-c4899ef5c81b-0000'
Submitted task 'test' to agent 'b383e094-b7f2-4841-a737-c4899ef5c81b-S0'
Received status update TASK_RUNNING for task 'test'
source: SOURCE_EXECUTOR
Received status update TASK_FINISHED for task 'test'
message: 'Command exited with status 0'
source: SOURCE_EXECUTOR

and the stout logs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
[SB2 emulate raspberry] root@precise64 build # cat /tmp/mesos/slaves/b383e094-b7f2-4841-a737-c4899ef5c81b-S0/frameworks/b383e094-b7f2-4841-a737-c4899ef5c81b-0000/executors/test/runs/latest/stdout
Registered executor on 127.0.0.1
Starting task test
sh -c 'ls /'
Forked command at 2794
bin
boot
dev
etc
home
lib
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var
Command exited with status 0 (pid: 2794)
Shutting down
Sending SIGTERM to process tree at pid 2794
Sent SIGTERM to the following process trees:
[

]

Yes, they indicate that our Mesos task succeed.

In additional, if you encounter below error when you want to lauch Mesos in the Raspberry Pi virtual machine. Please make sure you run the docker container with the option --cpuset-cpus=0 because qemu had a bug in multi cores environment.

1
/qemu-1.3.0/tcg/tcg.c:1440: tcg fatal error

In above content, we show how to compile Mesos for the Raspberry Pi and launch Mesos in the Raspberry Pi. But we still have a lot of works to make Mesos could work on the Raspberry Pi perfectly. Hope this could help you if you are seeking to run Mesos on ARM.