Related
I am writing a program for an Arm Cortex M7 based microcontroller, where I am facing an issue with (I guess) compiler optimization.
I compile using arm-none-eabi-gcc v10.2.1 with -mcpu=cortex-m7 -std=gnu11 -Ofast -mthumb.
I extracted the relevant parts, they are as below. I copy some data from a global variable into a local one, and from there the data is written to a hardware
CRC engine to compute the CRC over the data. Before writing it to the engine, the byte order is to be reversed.
#include <stddef.h>
#include <stdint.h>
#define MEMORY_BARRIER asm volatile("" : : : "memory")
#define __REV(x) __builtin_bswap32(x) // Reverse byte order
// Dummy struct, similiar as it is used in our program
typedef struct {
float n1, n2, n3, dn1, dn2, dn3;
} test_struct_t;
// CRC Engine Registers
typedef struct {
volatile uint32_t DR; // CRC Data register Address offset: 0x00
} CRC_TypeDef;
#define CRC ((CRC_TypeDef*) (0x40000000UL))
// Write data to CRC engine
static inline void CRC_write(uint32_t* data, size_t size)
{
for(int index = 0; index < (size / 4); index++) {
CRC->DR = __REV(*(data + index));
}
}
// Global variable holding some data, similiar as it is used in our program
volatile test_struct_t global_var = { 0 };
int main()
{
test_struct_t local_tmp;
MEMORY_BARRIER;
// Copy the current data into a local variable
local_tmp = global_var;
// Now, write the data to the CRC engine to compute CRC over the data
CRC_write((uint32_t*) &local_tmp, sizeof(local_tmp));
MEMORY_BARRIER;
return 0;
}
This compiles to:
main:
push {r4, r5, lr}
sub sp, sp, #28
ldr r4, .L4
mov ip, sp
ldr r3, [sp] ; load first float into r3
mov lr, #1073741824 ; load address of CRC engine DR register into lr
rev r5, r3 ; reverse byte order of first float and store in r5
ldmia r4!, {r0, r1, r2, r3} ; load first 4 floats into r0-r3
stmia ip!, {r0, r1, r2, r3} ; copy first 4 floats into local variable
ldm r4, {r0, r1} ; load float 5 and 6 into r0 and r1
stm ip, {r0, r1} ; copy float 5 and 6 into local variable
str r5, [lr] ; write reversed bytes from r5 to CRC engine
ldr r3, [sp, #4]
rev r3, r3
str r3, [lr]
ldr r3, [sp, #8]
rev r3, r3
str r3, [lr]
ldr r3, [sp, #12]
rev r3, r3
str r3, [lr]
ldr r3, [sp, #16]
rev r3, r3
str r3, [lr]
ldr r3, [sp, #20]
rev r3, r3
str r3, [lr]
movs r0, #0
add sp, sp, #28
pop {r4, r5, pc}
.L4:
.word .LANCHOR0
global_var:
The problem is now that the first float is loaded and its byte order is reversed (rev r5, r3) before the data has been copied from the global variable (happens with the ldmia/sdmia).
This is clearly not intended, leads to a wrong CRC, and I wonder why the compiler does not recognize this data dependency. Am I doing anything wrong or is there a bug in GCC?
Code in compiler explorer: https://godbolt.org/z/ErWThc5dx
I'm working on STM32CubeIDE.
To mix C and arm assembly, we initially used EXPORT.
Below is main.c:
#include <stdio.h>
extern int calc(int num, int* cnt);
int main()
{
int cnt = 5;
calc(4, &cnt);
return 0;
}
And then calc.s:
AREA calculator, CODE
EXPORT calc
ALIGN
calc:PROC
PUSH {r4,r5,lr}
MOV r5, #13
UDIV r4, r0, r5
MUL r5, r4, r5
SUB r4, r0, r5
CMP r4,#0
BEQ ace
CMP r4,#8
BGT jqk
then: ADD r4, r4, #1
B exit
ace:ADD [r1],#1
MOV r4, #11
B exit
jqk:MOV r4, #10
exit:MOV r0, r4
POP {r4, r5, pc}
ENDP
I put the two files in the same place and built main.c, but I get an error that says it's an unknown external reference.
So after giving up, I tried to put the ASM sentence in the c file using inline assembly.
calc PROC
PUSH {r4, lr}
AND r4, r0, #12
CMP r4, #0
BEQ ace
CMP r4, #8
BGT jqk
then ADD r4, r4, #1
B exit
ace ADD r1, r1, #1
MOV r4, #11
B exit
jqk MOV r4, #10
exit MOV r0, r4
ENDP
I've written these assembly codes, and I've adapted them to inline grammar.
reference : Labels in GCC inline assembly
int calc(int num, int *cnt)
{
int rst=0;
int c = *cnt;
__asm volatile("MOV R0, %0": :"r"(num));
__asm volatile("AND R0, R0, %0": :"r"(0x12));
__asm volatile("MOV R1, %0": :"r"(c));
__asm volatile("CMP R0, #0");
__asm volatile("BEQ ace%=");
__asm volatile("CMP R0,#8");
__asm volatile("BGT jqk%=");
__asm volatile("ADD R2, R0, #1");
__asm volatile("B exit%=");
__asm volatile("ace%=: ADD R1, R1, #1");
__asm volatile("MOV R2, #11");
__asm volatile("B exit%=");
__asm volatile("jqk%=: MOV R2, #10");
__asm volatile("exit%=: LDR %0, R2":"=r"(rst):);
__asm volatile("LDR %0, R1":"=r"(c):);
__asm volatile("POP {R0, R1, R2}");
*cnt = c;
return rst;
}
But even in this case, an error appears.
What should I change in my code?
'''
int calc(int num, int *cnt)
{
int tmp = num % 13;
if (tmp == 0)
{
tmp = *cnt;
*cnt = tmp+1;
return 11;
}
else if (tmp > 8)
return 10;
else
return tmp + 1;
}
'''
You actually did the right thing initially, putting the asm code in a separate .s file and not using inline asm at all. The only thing is that you need to explicitly compile and link the calc.s file along with main.c
cc -o program main.c calc.s
should compile and assemble both files and link them. If you're using an IDE, you need to specify both main.c and calc.s as source files of the project.
I'm writing embedded C/assembler code for the NXP LPC810 microcontroller (just a hobby project).
I have a function fn. I also have an exact copy of that function's machine code in an array of uint8_t. (I have checked the hex file.)
I create a function pointer fnptr, with the same type as fn and point it at the array, using a cast.
It all cross-compiles without warnings.
When the MCU executes fn it works correctly.
When the MCU executes fnptr it crashes (I can't see any debug, as there are only 8 pins, all in use).
The code is position independent.
The array has the correct 4 byte alignment.
fn is in the .text section of the elf file.
The array is forced into the .text section of the elf file (still in flash, not RAM).
I have assumed that there is no NX-like functionality on such a basic Coretex M0+ MCU. (Cortex M3 and M4 do have some form of read-only memory protection for code.)
Are there other reasons why the machine code in the array does not work?
Update:
Here is the code:
#include "stdio.h"
#include "serial.h"
extern "C" void SysTick_Handler() {
// generate an interrupt for delay
}
void delay(int millis) {
while (--millis >= 0) {
__WFI(); // wait for SysTick interrupt
}
}
extern "C" int fn(int a, int b) {
return a + b;
}
/* arm-none-eabi-objdump -d firmware.elf
00000162 <fn>:
162: 1840 adds r0, r0, r1
164: 4770 bx lr
166: 46c0 nop ; (mov r8, r8)
*/
extern "C" const uint8_t machine_code[6] __attribute__((aligned (4))) __attribute__((section (".text"))) = {
0x40,0x18,
0x70,0x47,
0xc0,0x46
};
int main() {
LPC_SWM->PINASSIGN0 = 0xFFFFFF04UL;
serial.init(LPC_USART0, 115200);
SysTick_Config(12000000/1000); // 1ms ticks
int(*fnptr)(int a, int b) = (int(*)(int, int))machine_code;
for (int a = 0; ; a++) {
int c = fnptr(a, 1000000);
printf("Hello world2 %d.\n", c);
delay(1000);
}
}
And here is the disassembled output from arm-none-eabi-objdump -D -Mforce-thumb firmware.elf:
00000162 <fn>:
162: 1840 adds r0, r0, r1
164: 4770 bx lr
166: 46c0 nop ; (mov r8, r8)
00000168 <machine_code>:
168: 1840 adds r0, r0, r1
16a: 4770 bx lr
16c: 46c0 nop ; (mov r8, r8)
16e: 46c0 nop ; (mov r8, r8)
00000170 <main>:
...
I amended the code to call the original fn though a function pointer too, in order to be able to generate working and non-working assembly code that was hopefully near-identical.
machine_code has become much longer, as I am now using no optimisation (-O0).
#include "stdio.h"
#include "serial.h"
extern "C" void SysTick_Handler() {
// generate an interrupt for delay
}
void delay(int millis) {
while (--millis >= 0) {
__WFI(); // wait for SysTick interrupt
}
}
extern "C" int fn(int a, int b) {
return a + b;
}
/*
000002bc <fn>:
2bc: b580 push {r7, lr}
2be: b082 sub sp, #8
2c0: af00 add r7, sp, #0
2c2: 6078 str r0, [r7, #4]
2c4: 6039 str r1, [r7, #0]
2c6: 687a ldr r2, [r7, #4]
2c8: 683b ldr r3, [r7, #0]
2ca: 18d3 adds r3, r2, r3
2cc: 1c18 adds r0, r3, #0
2ce: 46bd mov sp, r7
2d0: b002 add sp, #8
2d2: bd80 pop {r7, pc}
*/
extern "C" const uint8_t machine_code[24] __attribute__((aligned (4))) __attribute__((section (".text"))) = {
0x80,0xb5,
0x82,0xb0,
0x00,0xaf,
0x78,0x60,
0x39,0x60,
0x7a,0x68,
0x3b,0x68,
0xd3,0x18,
0x18,0x1c,
0xbd,0x46,
0x02,0xb0,
0x80,0xbd
};
int main() {
LPC_SWM->PINASSIGN0 = 0xFFFFFF04UL;
serial.init(LPC_USART0, 115200);
SysTick_Config(12000000/1000); // 1ms ticks
int(*fnptr)(int a, int b) = (int(*)(int, int))fn;
//int(*fnptr)(int a, int b) = (int(*)(int, int))machine_code;
for (int a = 0; ; a++) {
int c = fnptr(a, 1000000);
printf("Hello world2 %d.\n", c);
delay(1000);
}
}
I compiled the code above, generating firmware.fn.elf and firmware.machinecode.elf by uncommenting //int(*fnptr)(int a, int b) = (int(*)(int, int))machine_code; (and commenting-out the line above).
The first code (fn) worked, the second code (machine_code) crashed.
fn's text and the code at machine_code are identical:
000002bc <fn>:
2bc: b580 push {r7, lr}
2be: b082 sub sp, #8
2c0: af00 add r7, sp, #0
2c2: 6078 str r0, [r7, #4]
2c4: 6039 str r1, [r7, #0]
2c6: 687a ldr r2, [r7, #4]
2c8: 683b ldr r3, [r7, #0]
2ca: 18d3 adds r3, r2, r3
2cc: 1c18 adds r0, r3, #0
2ce: 46bd mov sp, r7
2d0: b002 add sp, #8
2d2: bd80 pop {r7, pc}
000002d4 <machine_code>:
2d4: b580 push {r7, lr}
2d6: b082 sub sp, #8
2d8: af00 add r7, sp, #0
2da: 6078 str r0, [r7, #4]
2dc: 6039 str r1, [r7, #0]
2de: 687a ldr r2, [r7, #4]
2e0: 683b ldr r3, [r7, #0]
2e2: 18d3 adds r3, r2, r3
2e4: 1c18 adds r0, r3, #0
2e6: 46bd mov sp, r7
2e8: b002 add sp, #8
2ea: bd80 pop {r7, pc}
000002ec <main>:
...
The only difference in the calling code is the location of the code called:
$ diff firmware.fn.bin.xxd firmware.machine_code.bin.xxd
54c54
< 0000350: 0040 0640 e02e 0000 bd02 0000 4042 0f00 .#.#........#B..
---
> 0000350: 0040 0640 e02e 0000 d402 0000 4042 0f00 .#.#........#B..
The second address d402 is the address of the machine_code array.
Curiously, the first address bd02 is a little-endian odd number (d is odd in hex).
The address of fn is 02bc (bc02 in big endian), so the pointer to fn is not the address of fn, but the address of fn plus one (or with the low bit set).
Changing the code to:
...
int main() {
LPC_SWM->PINASSIGN0 = 0xFFFFFF04UL;
serial.init(LPC_USART0, 115200);
SysTick_Config(12000000/1000); // 1ms ticks
//int(*fnptr)(int a, int b) = (int(*)(int, int))fn;
int machine_code_addr_low_bit_set = (int)machine_code | 1;
int(*fnptr)(int a, int b) = (int(*)(int, int))machine_code_addr_low_bit_set;
for (int a = 0; ; a++) {
int c = fnptr(a, 1000000);
printf("Hello world2 %d.\n", c);
delay(1000);
}
}
Makes it work.
Googling, I found:
The mechanism for switching makes use of the fact that all instructions must be (at least) halfword-aligned, which means that bit[0] of the branch target address is redundant. Therefore this bit can be re-used to indicate the target instruction set at that address. Bit[0] cleared to 0 means ARM and bit[0] set to 1 means Thumb.
on http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.faqs/ka12545.html
tl;dr
You need to set the low bit on function pointers when executing data as code on ARM Thumb.
I'm working with 2 memories on my device, DDR and SRAM. The device is running a pre-OS code written in C and ARM.
I would like to conduct a DDR calibration, for that I need to copy a few functions to the SRAM, jump to it, run the calibration code, and go back to DDR when done.
In order to do so, I've modify my scatter file (.lds) so the relevant functions will be mapped to SRAM (instructions, data etc.).
After compiling the image, he is copied into the DDR and start running from there.
My problem is as follows:
How can I locate the starting address and size of these functions on DDR, so I'll be able to copy them to the SRAM and jump there?
Thanks you all in advance!
I am assuming you are talking about ARM architecture:
compile the code with __attribute__((always_inline)); on all related functions and compile with -fpic -fPIC read here for more info.
disassembling it and put it as-is on SRAM, e.g at adress 0xd1001000
reserve {r4-r15} on SRAM.
set pc to 0xd1001000 and sp properly to point the stack.
restore {r4-r15}
jump back to DDR.
You can look at here for a good resource of how to use the right gcc flags.
here is a refernce from uboot - it doesn't jump back to the initial place:
/*
* void relocate_code (addr_sp, gd, addr_moni)
*
* This "function" does not return, instead it continues in RAM
* after relocating the monitor code.
*
*/
.globl relocate_code
relocate_code:
mov r4, r0 /* save addr_sp */
mov r5, r1 /* save addr of gd */
mov r6, r2 /* save addr of destination */
/* Set up the stack */
stack_setup:
mov sp, r4
adr r0, _start
cmp r0, r6
moveq r9, #0 /* no relocation. relocation offset(r9) = 0 */
beq clear_bss /* skip relocation */
mov r1, r6 /* r1 <- scratch for copy_loop */
ldr r3, _image_copy_end_ofs
add r2, r0, r3 /* r2 <- source end address */
copy_loop:
ldmia r0!, {r9-r10} /* copy from source address [r0] */
stmia r1!, {r9-r10} /* copy to target address [r1] */
cmp r0, r2 /* until source end address [r2] */
blo copy_loop
#ifndef CONFIG_SPL_BUILD
/*
* fix .rel.dyn relocations
*/
ldr r0, _TEXT_BASE /* r0 <- Text base */
sub r9, r6, r0 /* r9 <- relocation offset */
ldr r10, _dynsym_start_ofs /* r10 <- sym table ofs */
add r10, r10, r0 /* r10 <- sym table in FLASH */
ldr r2, _rel_dyn_start_ofs /* r2 <- rel dyn start ofs */
add r2, r2, r0 /* r2 <- rel dyn start in FLASH */
ldr r3, _rel_dyn_end_ofs /* r3 <- rel dyn end ofs */
add r3, r3, r0 /* r3 <- rel dyn end in FLASH */
fixloop:
ldr r0, [r2] /* r0 <- location to fix up, IN FLASH! */
add r0, r0, r9 /* r0 <- location to fix up in RAM */
ldr r1, [r2, #4]
and r7, r1, #0xff
cmp r7, #23 /* relative fixup? */
beq fixrel
cmp r7, #2 /* absolute fixup? */
beq fixabs
/* ignore unknown type of fixup */
b fixnext
fixabs:
/* absolute fix: set location to (offset) symbol value */
mov r1, r1, LSR #4 /* r1 <- symbol index in .dynsym */
add r1, r10, r1 /* r1 <- address of symbol in table */
ldr r1, [r1, #4] /* r1 <- symbol value */
add r1, r1, r9 /* r1 <- relocated sym addr */
b fixnext
fixrel:
/* relative fix: increase location by offset */
ldr r1, [r0]
add r1, r1, r9
fixnext:
str r1, [r0]
add r2, r2, #8 /* each rel.dyn entry is 8 bytes */
cmp r2, r3
blo fixloop
b clear_bss
_rel_dyn_start_ofs:
.word __rel_dyn_start - _start
_rel_dyn_end_ofs:
.word __rel_dyn_end - _start
_dynsym_start_ofs:
.word __dynsym_start - _start
#endif /* #ifndef CONFIG_SPL_BUILD */
clear_bss:
#ifdef CONFIG_SPL_BUILD
/* No relocation for SPL */
ldr r0, =__bss_start
ldr r1, =__bss_end__
#else
ldr r0, _bss_start_ofs
ldr r1, _bss_end_ofs
mov r4, r6 /* reloc addr */
add r0, r0, r4
add r1, r1, r4
#endif
mov r2, #0x00000000 /* clear */
clbss_l:str r2, [r0] /* clear loop... */
add r0, r0, #4
cmp r0, r1
bne clbss_l
/*
* We are done. Do not return, instead branch to second part of board
* initialization, now running from RAM.
*/
jump_2_ram:
/*
* If I-cache is enabled invalidate it
*/
#ifndef CONFIG_SYS_ICACHE_OFF
mcr p15, 0, r0, c7, c5, 0 # invalidate icache
mcr p15, 0, r0, c7, c10, 4 # DSB
mcr p15, 0, r0, c7, c5, 4 # ISB
#endif
ldr r0, _board_init_r_ofs
adr r1, _start
add lr, r0, r1
add lr, lr, r9
/* setup parameters for board_init_r */
mov r0, r5 /* gd_t */
mov r1, r6 /* dest_addr */
/* jump to it ... */
mov pc, lr
_board_init_r_ofs:
.word board_init_r - _start
I am confused with the calculation of CGMA. I know that CGMA = # of operations / # of memory fetches. Firstly, when x = g_A[idx], should I count the write operation to x or ignore it because it is stored in register? Likewise, in z = (x*y) + (y/x) + (y-x);, should I count read of x and y as memory reads in the calculation of CGMA? Finally, should I count all the operations in the Kernel function (those five lines)?
__global__ void PerformSomeOperations(int* g_A,int* g_B,int* g_C, int Size)
{
const int idx = threadIdx.x + (blockIdx.x*blockDim.x);
if(idx < Size)
{
int x = g_A[idx];
int y = g_B[idx];
int z = 0;
z = (x*y) + (y/x) + (y-x);
g_C[idx] = z;
}
}
The disassembled code corresponding to your kernel (compiled for compute_20,sm_20) is the following
/*0000*/ MOV R1, c[0x1][0x100];
/*0008*/ S2R R0, SR_CTAID.X;
/*0010*/ S2R R2, SR_TID.X;
/*0018*/ IMAD R0, R0, c[0x0][0x8], R2;
/*0020*/ ISETP.GE.AND P0, PT, R0, c[0x0][0x2c], PT;
/*0028*/ #P0 EXIT ;
/*0030*/ SHL R0, R0, 0x2;
/*0038*/ IADD R2, R0, c[0x0][0x20];
/*0040*/ IADD R3, R0, c[0x0][0x24];
/*0048*/ IADD R0, R0, c[0x0][0x28];
/*0050*/ LD R2, [R2]; R2 = x = g_A[idx]
/*0058*/ LD R3, [R3]; R3 = y = g_B[idx]
/*0060*/ I2I.S32.S32 R5, |R2|;
/*0068*/ I2F.F32.U32.RP R4, R5; R4 = (float)x
/*0070*/ MUFU.RCP R4, R4; R4 = 1/R4
/*0078*/ IADD32I R4, R4, 0xffffffe;
/*0080*/ F2I.FTZ.U32.F32.TRUNC R4, R4;
/*0088*/ IMUL.U32.U32 R6, R5, R4; R6 = x * (1/y)
/*0090*/ I2I.S32.S32 R7, -R6;
/*0098*/ I2I.S32.S32 R6, |R3|;
/*00a0*/ IMAD.U32.U32.HI R7, R4, R7, R4;
/*00a8*/ IMUL.U32.U32.HI R4, R7, R6;
/*00b0*/ LOP.XOR R7, R3, R2;
/*00b8*/ IMAD.U32.U32 R6, -R5, R4, R6;
/*00c0*/ ISETP.GE.AND P1, PT, R7, RZ, PT;
/*00c8*/ ISETP.LE.U32.AND P0, PT, R5, R6, PT;
/*00d0*/ #P0 ISUB R6, R6, R5;
/*00d8*/ #P0 IADD R4, R4, 0x1;
/*00e0*/ ISETP.GE.U32.AND P0, PT, R6, R5, PT;
/*00e8*/ LOP.PASS_B R6, RZ, ~R2;
/*00f0*/ ISUB R5, R3, R2;
/*00f8*/ #P0 IADD R4, R4, 0x1;
/*0100*/ #!P1 I2I.S32.S32 R4, -R4;
/*0108*/ ICMP.EQ R4, R6, R4, R2;
/*0110*/ IADD R4, R5, R4;
/*0118*/ IMAD R2, R3, R2, R4;
/*0120*/ ST [R0], R2;
/*0128*/ EXIT ;
From the above code, there are the following floating point operations
I2F.F32.U32.RP R4, R5; Integer to Float conversion
MUFU.RCP R4, R4; Multifunction Floating Point Operation (Reciprocal)
F2I.FTZ.U32.F32.TRUNC R4, R4; Float to Integer conversion
Those operations appear to be related to (x/y) which is a division between two integers but require conversion to floating point. I'm not really aware of whether the conversions are counted as floating point operations or not. I do not see any other floating point operation within the code.
The global memory operations are the following 3
LD R2, [R2];
LD R3, [R3];
ST [R0], R2;
I would say that CGMA = 3/3 = 1 for your case (counting the int2float and float2int conversions as floating point operations).
Looks like CGMA stands for Compute to Global Memory Access and is defined as the number
of floating-point calculations performed for each access to the global memory within a
region of a CUDA program.
The best way to calculate the ratio will be to run your program in a CUDA profiler and use the performance counters for memory accesses and floating point operations. According to the definition I found, your kernel has a CGMA of zero because it performs integer arithmetic, not floating point. If you change the definition, then x = g_A[idx] is one read operation and no write operations. That is because the register file is not stored in global memory (the "G" in CGMA). There are no global memory reads in z = (x*y) + (y/x) + (y-x);, so count that as 5 operations. If all threads run with idx < Size, then you have 3 global memory accesses and 8 operations. Note, though, that in CUDA, performance of global memory accesses depends on if they are coalesced. Many coalesced memory accesses can run much faster than a few uncoalesced ones. So the CGMA is not necessarily going to give an accurate picture of the performance potential of your kernel.
References:
http://www.greatlakesconsortium.org/events/GPUMulticore/Chapter4-CudaMemoryModel.pdf
http://cs.nyu.edu/courses/spring12/CSCI-GA.3033-012/lecture6.pdf