Aarch64 baremetal Hello World program on QEMU - arm

I have compiled and ran a simple hello world program on ARM Foundation Platform. The code is given below.
#include <stdio.h>
int main(int argc, char *argv[])
{
printf("Hello world!\n");
return 0;
}
This program is compiled and linked using ARM GNU tool chain as given below.
$aarch64-none-elf-gcc -c -march=armv8-a -g hello.c hello.o
aarch64-none-elf-gcc: warning: hello.o: linker input file unused because linking not done
$aarch64-none-elf-gcc -specs=aem-ve.specs -Wl,-Map=linkmap.txt hello.o -o hello.axf
I could successfully execute this program on ARM Foundation Platform (I think, foundation platform is similar to ARM Fixed Virtual Platform) and it prints "Hello world!"
The contents of 'aem-ve.specs' file is given below:
cat ./aarch64-none-elf/lib/aem-ve.specs
# aem-ve.specs
#
# Spec file for AArch64 baremetal newlib, libgloss on VE platform with version 2
# of AngelAPI semi-hosting.
#
# This Spec file is also appropriate for the foundation model.
%rename link old_link
*link:
-Ttext-segment 0x80000000 %(old_link)
%rename lib libc
*libgloss:
-lrdimon
*lib:
cpu-init/rdimon-aem-el3.o%s --start-group %(libc) %(libgloss) --end-group
*startfile:
crti%O%s crtbegin%O%s %{!pg:rdimon-crt0%O%s} %{pg:rdimon-crt0%O%s}
Is it possible to execute the same binary on QEMU? If so could you please share the sample command. If not, could you please share the right and best way to execute it on QEMU?
I have traced instruction execution of Foundation platform using the "trace" option and analysed using 'tarmac-profile' tool and 'Tarmac-calltree' tool. The outputs are given below:
$ ./tarmac-profile hello.trace --image hello.axf
Address Count Time Function name
0x80000000 1 120001
0x80001030 1 30001 register_fini
0x80001050 1 60001 deregister_tm_clones
0x800010c0 1 210001 __do_global_dtors_aux
0x80001148 1 23500001 frame_dummy
0x800012c0 1 10480001 main
0x80002000 1 40001 main
0x80003784 1 360001 main
0x80003818 1 460001 _cpu_init_hook
0x80003870 1 590001 _write_r
0x800038d0 1 1090001 __call_exitprocs
...
...
./tarmac-calltree hello.trace --image hello.axf
o t:10000 l:2288 pc:0x80001148 - t:23510000 l:7819 pc:0x80008060 : frame_dummy
- t:240000 l:2338 pc:0x800011a8 - t:720000 l:2443 pc:0x800011ac
o t:250000 l:2340 pc:0x80003818 - t:710000 l:2442 pc:0x80003828 : _cpu_init_hook
- t:260000 l:2343 pc:0x8000381c - t:320000 l:2354 pc:0x80003820
o t:270000 l:2345 pc:0x80002000 - t:310000 l:2353 pc:0x80002010 : main
- t:320000 l:2354 pc:0x80003820 - t:700000 l:2436 pc:0x80003824
o t:330000 l:2356 pc:0x80003784 - t:690000 l:2435 pc:0x80003814 : main
- t:760000 l:2453 pc:0x800011bc - t:2010000 l:2970 pc:0x800011c0
o t:770000 l:2455 pc:0x80004200 - t:2000000 l:2969 pc:0x800042c0 : memset
- t:2010000 l:2970 pc:0x800011c0 - t:4870000 l:3587 pc:0x800011c4
o t:2020000 l:2972 pc:0x80007970 - t:4860000 l:3586 pc:0x80007b04 : initialise_monitor_handles
- t:2960000 l:3165 pc:0x80007b24 - t:4340000 l:3465 pc:0x80007b28
I have tried the following method to execute on QEMU, without any success.
I have started the QEMU with gdb based debugging through TCP Port
$ qemu-system-aarch64 -semihosting -m 128M -nographic -monitor none -serial stdio -machine virt,gic-version=2,secure=on,virtualization=on -cpu cortex-a53 -kernel hello.axf -S -gdb tcp::9000
The result of debug session is given below:
gdb) target remote localhost:9000
Remote debugging using localhost:9000
_start () at /data/jenkins/workspace/GNU-toolchain/arm-11/src/newlib-cygwin/libgloss/aarch64/crt0.S:90
90 /data/jenkins/workspace/GNU-toolchain/arm-11/src/newlib-cygwin/libgloss/aarch64/crt0.S: No such file or directory.
(gdb) si
<The system hangs here>
I have tried to disassemble the code using gdb and its output is given below. It looks like the code is not loaded correctly
(gdb) disas frame_dummy
Dump of assembler code for function frame_dummy:
0x0000000080001110 <+0>: udf #0
0x0000000080001114 <+4>: udf #0
0x0000000080001118 <+8>: udf #0
0x000000008000111c <+12>: udf #0
0x0000000080001120 <+16>: udf #0
0x0000000080001124 <+20>: udf #0
0x0000000080001128 <+24>: udf #0
0x000000008000112c <+28>: udf #0
0x0000000080001130 <+32>: udf #0
0x0000000080001134 <+36>: udf #0
0x0000000080001138 <+40>: udf #0
0x000000008000113c <+44>: udf #0
0x0000000080001140 <+48>: udf #0
0x0000000080001144 <+52>: udf #0
End of assembler dump.
Could you please shed some light on this. Any hint given is appreciated very much.

The Foundation Model and the QEMU 'virt' board are not the same machine type. They have different devices at different physical addresses, and in particular they do not have RAM at the same address. To run bare metal code on the 'virt' machine type you will need to adjust your code. This is normal for bare metal -- the whole point is that you are running directly on some piece of (emulated) hardware and need to match the details of it.
Specifically, the minimum change you need to make here is that the RAM on the 'virt' board starts at 0x4000_0000, not the 0x8000_0000 that the Foundation Model uses. There may be others as well, but that is the immediate cause of what you're seeing.

Related

Macbook M1 assembly lldb displays only 3 source lines then switches to object code display

First attempt at ARM64 (apple M1) assembly coding. Have basic 'hello world' code which assembles and runs correctly but when I run it in lldb, only the first three lines are displayed in full source code format like this:
Abenaki:hello jiml$ ~/llvm/clang+llvm-15.0.2-arm64-apple-darwin21.0/bin/lldb hello
(lldb) target create "hello"
Current executable set to '/Users/jiml/Projects/GitRepos/ARM/hello/hello/hello/hello' (arm64).
(lldb) b main
Breakpoint 1: where = hello`main + 4, address = 0x0000000100003f7c
(lldb) r
Process 5017 launched: '/Users/jiml/Projects/GitRepos/ARM/hello/hello/hello/hello' (arm64)
Process 5017 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
frame #0: 0x0000000100003f7c hello`main at hello.s:19
16
17 _main:
18 mov x0, #0x0 // stdout
-> 19 adrp x1, msg#PAGE // pointer to string
20 add x1, x1, msg#PAGEOFF
21 ldr x2, =msg_len // bytes to output
22 mov x16, #0x04 // sys_write
warning: This version of LLDB has no plugin for the language "assembler". Inspection of frame variables will be limited.
(lldb)
After three steps, the display reverts to bare object code like this:
(lldb) s
Process 5017 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
frame #0: 0x0000000100003f88 hello`main + 16
hello`main:
-> 0x100003f88 <+16>: mov x16, #0x4
0x100003f8c <+20>: svc #0x80
0x100003f90 <+24>: adrp x1, 1
0x100003f94 <+28>: mov x2, #0x0
dwarfdump -a shows that all source lines are present in the .o; same behavior for .dSYM assembly. Using the 'list' command in lldb however displays all source lines correctly.
Is this a known issue for LLVM (clang, lldb) development? Any help appreciated...
I have tried LLVM version 14 and 15, same behavior, searched for similar issues but no help.
I did find this https://stackoverflow.com/questions/73778648/why-is-it-that-assembling-linking-in-one-step-loses-debug-info-for-my-assembly-s but it did not solve my issue.
So I think I have this resolved but not sure if it is actual compiler bug.
I wrote hello world in C, compiled and confirmed complete source display in lldb. I then reran clang with -S to generate the assembler source.
I then assembled that source...
clang -g -c -o hello.o hello.s
clang -o hello hello.o -lSystem -arch arm64
and confirmed it also runs in lldb with complete source display. Then I moved my hand written code line-by-line in order to figure out where the problem occurs. Seems my string data and length calculation is problematic. In the data section I originally had:
msg: ascii "Hello ARM64\n"
msg_len = . - msg
Coming from Intel world this seems perfectly natural ;-) Adding that length calculation caused some sort of corruption of the debug data. However, the executable has a proper OSO statement pointing at hello.o (nm -ap hello) and further the object file has references for all source statements in the source file (dwarfdump --debug-line hello.o) but still doesn't display source code after the third step. Curious that 'source info -f hello.s' within lldb only listed four lines.
I found three work-arounds. First adding a label between the two statements seems to allow correct behavior:
msg: ascii "Hello ARM64\n"
nothing:
msg_len = . - msg
Second, using equate:
msg: ascii "Hello ARM64\n"
.equ msg_len, . - msg
Third, using two labels:
msg: ascii "Hello ARM64\n"
msg_end:
msg_len = msg_end - msg
Will file report with llvm and see what they say.

Why is LLDB generating EXC_BAD_INSTRUCTION with user compiled library on MacOS?

I want to debug OpenSSL on MacOS to see how it creates an elliptic curve point. So, I compiled OpenSSL with debug symbols and no optimizations. However, when I run with lldb, it doesn't work
$ cat ec.c
#include <crypto/ec.h>
#include <stdio.h>
int main() {
EC_GROUP *group = EC_GROUP_new_by_curve_name(NID_secp384r1);
EC_POINT *point = EC_POINT_new(group);
printf("done!\n");
return 0;
}
Here is how I compiled and ran the program:
$ gcc ec.c -o ec -I../openssl/include ../openssl/libcrypto.dylib -ggdb3 -O0
$ DYLD_INSERT_LIBRARIES=../openssl/libcrypto.dylib ./ec
done!
Here is what happens when I run lldb and try to break at main:
$ lldb ./ec
(lldb) process launch --environment DYLD_INSERT_LIBRARIES=../openssl/libcrypto.dylib ./ec
Process 3948 launched: '/Users/seanthomas/repos/ec/ec' (arm64)
Process 3948 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=1, subcode=0x4a03000)
frame #0: 0x00000001009651a8 libcrypto.3.dylib`_armv8_sve_probe
libcrypto.3.dylib`:
-> 0x1009651a8 <+0>: eor z0.d, z0.d, z0.d
0x1009651ac <+4>: ret
libcrypto.3.dylib`:
0x1009651b0 <+0>: xar z0.d, z0.d, z0.d, #0x20
0x1009651b4 <+4>: ret
Target 0: (ec) stopped.
(lldb)
Does anyone know how to fix this?
I'm not sure whether this can help you or not. But perhaps, there're bugs in libcrypto.3.dylib on Arm arch.
I met this problem also. It works when I run the program by cmd line in shell. But it meets this problem when I want to debug this program in VSCode using lldb.
How ever, when I delete the libcrypto.3.dylib and libssl.3.dylib, build the openssl using tag OpenSSL_1_1_1m, and rebuild the program. It works!
Yes, confirm the issue on Mac M1/M2. Reverted back to the old version of openssl and the problem got fixed. I use openssl lib in MacOs app of mine and on app launch in Debug mode it gets crashing right away.

Emulate ARM Cortex-M7 with qemu-system-arm.exe

I'm using Eclipse based CubeIDE and QEMU debugging plugin.
I'm working in assembler and can debug simple project (adding two numbers in registers) on STM32 Cortex M7 board (STM32H750DK). Now I'd like to do the same using QEMU and have problems, because I can't find suitable generic Cortex M7 machine. I've tried mps2-an500, but it doesn't work.
I've done similar already under qemu-system-gnuarmeclipse.exe on STM32F407 board and it's supported even visually (board picture), and I can see LED diodes blinking.
I'd be enough just plain emulation (not visual) for cortex M7 board.
Has anyone done something similar or any other advice how to do this properly ?
Thanks.
I think you are supposed to show some code for the attempts you made when asking a question on Stackoverflow.
In your case, this could have been a minimal, reproducible example of your attempts to use the mps2-an500 QEMU virtual machine.
This being said, a procedure for building and debugging a program using QEMU and targeting the mps2-an500 virtual machine could be:
Download QEMU 7.10 for Windows, and install it into the \opt\qemu-7.1.0 directory - you will have to create it,
Download the arm-gnu-toolchain-11.3.rel1-mingw-w64-i686-arm-none-eabi toolchain,and install into the \opt\arm\11 directory - you will have to create it,
In a directory of your choice, create the following files:
build.cmd:
#set CROSS_COMPILE=\opt\arm\11\arm-gnu-toolchain-11.3.rel1-mingw-w64-i686-arm-none-eabi\bin\arm-none-eabi-
#set CC=%CROSS_COMPILE%gcc
#set OBJDUMP=%CROSS_COMPILE%objdump
#set GDB=%CROSS_COMPILE%gdb
#set QEMU_SYSTEM_ARM=\opt\qemu-7.1.0\qemu-system-arm
#%CC% -g -mthumb -mtune=cortex-m7 -nostdlib -nostartfiles -ffreestanding -Wl,-Ttext,0x00000000 -o mps2-an500.elf startup.s
#%OBJDUMP% -d mps2-an500.elf > mps2-an500.objdump
#echo QEMU/GDB commands:
#echo %QEMU_SYSTEM_ARM% -m 16M -nographic -machine mps2-an500 -S -cpu cortex-m7 -gdb tcp::2345,ipv4 -kernel mps2-an500.elf
#echo %GDB%
startup.s:
.file "startup.s"
.arch armv7-a
.fpu vfpv3-d16
.thumb
.syntax unified
.equ __StackTop, 0x21000000
.global _start
.align 2
.long __StackTop
.long _start
_start:
mov r0,#3
mov r1,#5
add r2, r0, r1
wait: b wait
.size _start, .-_start
.end
Execute the build.cmd batch procedure, it will create mps2-an500.elf and mps2-an500.lst, and display the QEMU and GDB commands you will use for debugging the (very) basic example:
build.cmd
QEMU/GDB commands:
\opt\qemu-7.1.0\qemu-system-arm -m 16M -nographic -machine mps2-an500 -S -cpu cortex-m7 -gdb tcp::2345,ipv4 -kernel mps2-an500.elf
\opt\arm\11\arm-gnu-toolchain-11.3.rel1-mingw-w64-i686-arm-none-eabi\bin\arm-none-eabi-gdb
In one console mode session, execute:
\opt\qemu-7.1.0\qemu-system-arm -m 16M -nographic -machine mps2-an500 -S -cpu cortex-m7 -gdb tcp::2345,ipv4 -kernel mps2-an500.elf
in another, execute the GDB command:
\opt\arm\11\arm-gnu-toolchain-11.3.rel1-mingw-w64-i686-arm-none-eabi\bin\arm-none-eabi-gdb --command=mps2-an500.gdb
In the GDB session, execute the following commands in sequence:
target remote localhost:2345
file mps2-an500.elf
break _start
break wait
set $sp = 0x21000000
set $pc = _start
stepi
stepi
stepi
info registers
The transcript for the GDB session should look like:
\opt\arm\11\arm-gnu-toolchain-11.3.rel1-mingw-w64-i686-arm-none-eabi\bin\arm-none-eabi-gdb
GNU gdb (Arm GNU Toolchain 11.3.Rel1) 12.1.90.20220802-git
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=i686-w64-mingw32 --target=arm-none-eabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.linaro.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) target remote localhost:2345
Remote debugging using localhost:2345
warning: No executable has been specified and target does not support
determining executable automatically. Try using the "file" command.
0x00000014 in ?? ()
(gdb) file mps2-an500.elf
A program is being debugged already.
Are you sure you want to change the file? (y or n) y
Reading symbols from mps2-an500.elf...
(gdb) break _start
Breakpoint 1 at 0x8: file startup.s, line 12.
(gdb) break wait
Breakpoint 2 at 0x14: file startup.s, line 15.
(gdb) set $sp = 0x21000000
(gdb) set $pc = _start
(gdb) stepi
13 mov r1,#5
(gdb) stepi
14 add r2, r0, r1
(gdb) stepi
Breakpoint 2, _start () at startup.s:15
15 wait: b wait
(gdb) info registers
r0 0x3 3
r1 0x5 5
r2 0x8 8
r3 0x0 0
r4 0x0 0
r5 0x0 0
r6 0x0 0
r7 0x0 0
r8 0x0 0
r9 0x0 0
r10 0x0 0
r11 0x0 0
r12 0x0 0
sp 0x21000000 0x21000000
lr 0xfffffff9 -7
pc 0x14 0x14 <_start+12>
xpsr 0x41000003 1090519043
fpscr 0x0 0
(gdb)

Linux Kernel function memblock_alloc_range_nid is not presented in the address space

I'm trying to debug physical memory allocation to understand what part of the Linux Kernel use memblock_alloc_range_nid on x86-64 and how.
I'm running the latest Linux Kernel 5.19-rc2 built from upstream with Ubuntu 20.04 inside QEMU. The problem is it's not possible to access memory address the function memblock_alloc_range_nid is located at. While other Kernel functions can be easily disassembled.
Here is what I have in my gdb connected to the QEMU VM:
(gdb) disas memblock_alloc_range_nid
Cannot access memory at address 0xffffffff831a05d1
(gdb) disas native_safe_halt
Dump of assembler code for function native_safe_halt:
#...
End of assembler dump.
What's wrong with the function memblock_alloc_range_nid? Why is it not possible to access its address? It seems all the function from memblock.c cannot be accessed.
As Margaret and Roi have noted in the above comments, memblock_alloc_range_nid() is annotated with __init. Functions annotated with __init, __head and similar macros are only needed during kernel initialization right after boot. After the kernel has finished initializing things, the special memory section containing all those functions is unmapped from memory since they are not needed anymore.
If you want to debug any such function, you will have to break very early, for example at start_kernel(), then you can inspect the function or set a breakpoint and continue execution to hit it.
Important: add -S (uppercase) to your QEMU command line options to make it stop and wait for the debugger before starting the kernel, and start the kernel with KASLR disabled using -append "nokaslr" (or adding nokaslr if you are already specifying -append).
The following GDB script should be what you need:
$ cat script.gdb
directory /path/to/kernel-source-dir
file /path/to/kernel-source-dir/vmlinux
target remote localhost:1234
break start_kernel
continue
Launch gdb -x script.gdb (after starting QEMU), and when you hit the start_kernel breakpoint, you can add another one for memblock_alloc_range_nid, then continue:
0x000000000000fff0 in exception_stacks ()
Breakpoint 1 at 0xffffffff82fbab4c
Breakpoint 1, 0xffffffff82fbab4c in start_kernel ()
(gdb) b memblock_alloc_range_nid
Breakpoint 2 at 0xffffffff82fe2879
(gdb) c
Continuing.
Breakpoint 2, 0xffffffff82fe2879 in memblock_alloc_range_nid ()
(gdb) disassemble
Dump of assembler code for function memblock_alloc_range_nid:
=> 0xffffffff82fe2879 <+0>: push r15
0xffffffff82fe287b <+2>: mov r15,rcx
0xffffffff82fe287e <+5>: push r14
0xffffffff82fe2880 <+7>: mov r14,rdx
0xffffffff82fe2883 <+10>: push r13
0xffffffff82fe2885 <+12>: mov r13d,r9
...

How do i run assembly (.s) files on VSCode in Linux subsystem

I have just got into ARM programming. Learned a few basics but am having issues with running code. I use VSCode on Linux Subsystem as my IDE.
I have nothing installed on my computer and i would like to run ARM code. I have read online something about "qemu" and "kernel" and stuff, but am not sure what they mean. It would be great if someone provides a detailed walkthrough for such a problem. I do not have a raspberry pi.
For example, how do i run the following division.s file on VSCode?
.global _start
_start:
MOV R1, #X
MOV R2, #Y
MOV R3, #Z
CMP R1, R2 # is x>y ?
BGT _tryx
CMP R2, R3 # is y>z ?
BGT _isy
MOV R4, R3
B _exit
_isy:
MOV R4, R2
B _exit
_tryx:
CMP R1, R3 # is x>z ?
BGT _isx
MOV R4, R3
B _exit
_isx:
MOV R4, R1
_exit:
MOV R0, R4
MOV R7, #1
SWI 0
.data
.equ X, 3
.equ Y, 5
.equ Z, 4
Are there any extensions i need to install? Is there anything i need to download? I have used gcc to compile C code. Can it be used here too?
Thx in advance! :D
Your question is rather a broad one. This being said, a slightly modified version of your program can be executed in WSL using the following procedure:
sudo apt-get install qemu-user
sudo mkdir -p /opt/arm/10
wget 'https://developer.arm.com/-/media/Files/downloads/gnu-a/10.2-2020.11/binrel/gcc-arm-10.2-2020.11-x86_64-arm-none-linux-gnueabihf.tar.xz?revision=d0b90559-3960-4e4b-9297-7ddbc3e52783&la=en&hash=985078B758BC782BC338DB947347107FBCF8EF6B' -O gcc-arm-10.2-2020.11-x86_64-arm-none-linux-gnueabihf.tar.xz
sudo tar Jxf gcc-arm-10.2-2020.11-x86_64-arm-none-linux-gnueabihf.tar.xz -C /opt/arm/10
/tmp/division.s:
# count how often we can take Y from X
.global main
main:
MOV R1, #X
MOV R2, #Y
MOV R3, #0 # Q
_loop:
CMP R1, R2
BLT _exit
SUB R1, R2
ADD R3, #1
B _loop
_exit:
MOV R0, R3
MOV R7, #1
SWI 0
.data
.equ X, 23
.equ Y, 4
Compiling:
/opt/arm/10/gcc-arm-10.2-2020.11-x86_64-arm-none-linux-gnueabihf/bin/arm-none-linux-gnueabihf-gcc -static -o /tmp/division /tmp/division.s
Executing - WSL:
qemu-arm /tmp/division
echo $?
5
Which is the expected result, since 23 div 4 is 5.
Executing - Windows 10:
C:\>c:\Windows\System32\bash -c "qemu-arm /tmp/division; echo $?"
5
C:\>
Or:
C:\>c:\Windows\System32\bash -c "qemu-arm /tmp/division"
C:\>echo %ERRORLEVEL%
5
Note that division.s may have been compiled in Windows 10 as well by downloading/installing gcc-arm-10.2-2020.11-mingw-w64-i686-arm-none-linux-gnueabihf.tar.xz instead of gcc-arm-10.2-2020.11-x86_64-arm-none-linux-gnueabihf.tar.xz - your choice.
I let it to you than to go into the details of using the information above for running your program from vscode, you question being a bit too broad IMHO.
Update: division.s was compiled statically on purpose for avoiding having to specify the locations for any non-needed dynamic libraries.
Compiling it without using the -static option, and executing it would result in the following error message to be displayed:
qemu-arm division
/lib/ld-linux-armhf.so.3: No such file or directory
It can be avoided by telling qemu-arm where to look for required dynamic libraries, /lib/ld-linux-armhf.so.3 in this case:
qemu-arm -L /opt/arm/10/gcc-arm-10.2-2020.11-x86_64-arm-none-linux-gnueabihf/arm-none-linux-gnueabihf/libc division
echo $?
5
That looks like a Linux program. And I guess you have an 64-bit x86 computer.
You want QEMU, specifically the executable qemu-arm which emulates an ARM processor running Linux on another Linux compatible system. In Debian derived Linux distributions it is in the package qemu-arm.
And on ebian derived Linux distributions for a compiler you can install the gcc-arm-linux-gnueabihf package for compiling for ARMv7 and ARMv8 with hardware floating point (Debian port name armhf), or gcc-arm-linux-gnueabi for some older ARMs with no hardware floating point Debian port name armel).
If you wish to install libraries, you need add the architecture to dpkg's list, for example:
dpkg --add-architecture armhf

Resources