const arrays get loaded into flash, or else they are in RAM.
How can I load a large const array, apart from typing in thousands of numbers by hand?
I am using the IAR compiler with an STM32F303 (Cortex M4)
You always can write an application which will generate the array from what you need and then just include it in your source file.
What are the numbers? Typically you can use off-line tools to generate C code that holds the numbers in a const array of the suitable type. This is often done for i.e. look-up tables in embedded software, and so on.
You cannot do this at run-time, since it's the linker's job to arrange the segments of the program into the various available memory-blocks.
Also, flash memory is not generally "easy" to write, i.e. you can't typically expect to be able to have a regular C pointer into flash, and just write to it and have it "stick". Programming flash generally requires dancing with the flash memory controller, and keeping in mind things like block erasure, erasing time, minimum programming page size, programming time per page, and so on. Flash memory is not so much RAM, as it is ROM that happens to be reprogrammable in software if you know how.
Related
I've used few types microcontrollers.
When I write code like this:
const int var = 5;
usually var is kept in flash. I understand that const variables are not always kept only in flash. Sometimes (depending compiler, processor, options like pic etc.) they are loaded from flash to RAM before main. Is there a case, when it is better to load var into RAM?
Many microcontrollers have a unified address space, but some (such as AVRs) do not. When there are multiple address spaces, then you may need to use different instructions to access data in those spaces. (For AVR, the LPM (Load Program Memory) instruction must be used to access data in Flash, instead of one of the ordinary LD (Load) instructions.)
If you imagine a pointer to your const variable, a function receiving the pointer would not be able to access the data unless it knows which address space the pointer points into. In a case like that, you either have to copy the data to RAM, even though it's const, or the user of the pointer has to know which instruction to use to access the data.
Assuming a Microcontroller architecture like ARM Cortex or Microchip MIPS (and many others), RAM and Flash are mapped to different parts of the internal address space, like a huge array. So the Assembly commands reading from RAM are the same like reading from Flash. No difference here.
Access times of RAM and Flash shouldn't be too different, so no waiting needed on any of the controllers I've worked with.
The only case I can imagine where storing const vars in flash could cause problems is in some sort of bootloader app, when the flash is written. Of course, writing to a flash range where you are executing from is a bad idea and will cause much heavier problems than overwritten const values.
Is there any way to check or prevent stack area from crossing the RAM data (.data or .bss) area in the limited memory (RAM/ROM) embedded systems comprising microcontrollers? There are tools to do that, but they come with very costly license fees like C-STAT and C-RUN in IAR.
You need no external tools to view and re-map your memory layout. The compiler/linker you are using should provide means of doing so. How to do this is of course very system-specific.
What you do is to open up the system-specific linker file in which all memory segments have been pre-defined to a default for the given microcontroller. You should have the various RAM segments listed there, de facto standard names are: .stack .data .bss and .heap.
Each such segment will have an address range specified. Change the addresses and you will move the segments. However, these linker files usually have some obscure syntax that you need to study before you touch anything. If you are (un)lucky it uses GNU linker scripts, which is a well-documented, though rather complex standard.
There could also be some manufacturer-supplied start-up code that sets the stack pointer. You might have to modify that code manually, in addition to tweaking the linker file.
Regarding the stack: you need to check the CPU core manual and see if the stack pointer moves upwards or downwards on your given system. Most common is downwards, but the alternative exists. You should ensure that in the direction that the stack grows, there is no other read/write data segment which it can overwrite upon stack overflow. Ideally the stack should overflow into non-mapped memory where access would cause a CPU hardware interrupt/exception.
Here is an article describing how to do this.
In small micros that do not have the necessary hardware support for this, a very simple method is to have a periodic task (either under a multitasker or via a regular timed interrupt) check the 'threshold' RAM address which you must have initialized to some 'magic' pattern, like 0xAA55
Once the periodic task sees this memory address change contents, you have a problem!
In microcontrollers with limited resources, it is always a good idea to prevent stack overflow via simple memory usage optimizations:
Reduce overall RAM usage by storing read-only variables in non-volatile (e.g. flash) memory. A good target for this are constant strings in your code, like the ones used on printf() format strings, for example. This can free a lot of memory for your stack to grow. Check you compiler documentation about how to allocate these variables in flash.
Avoid recursive calls - they are not a good idea in resource-constrained or safety-critical systems, as you have little control over how the stack grows.
Avoid passing large parameters by value in function calls - pass them as const references whenever possible (e.g. for structs or classes).
Minimize unnecessary usage of local variables. Look particularly for the large ones, like local buffers for example. Often you can find ways to just remove them, or to use a shared resource instead without compromising your code.
I know I can force an array into FLASH in ARM by declaring it "const". But this is not truly an array of consts: I want to be able to write to it regularly. I have three large arrays that take up ~50k of the 128kB of SRAM I have available, but I have an order of magnitude more FLASH than I need. How can I force these three arrays into FLASH without declaring them const? Using IAR, BTW.
Tried using the __no_init keyword; according to the linker map files this had no effect.
To answer the original question, you can write a linker script to force any variable to reside in a predetermined area of memory (declaring them const does not force the compiler to put it in FLASH, it is merely a strong suggestion).
On the other hand, overabundance of FLASH is not in itself a good reason to keep a non-const array in flash. Among the reasons are: 1) depending on the chip, the access to FLASH memory can be much slower than RAM access (especially for writing) 2) FLASH can only be rewritten a limited number of times: it is not a problem for an occasional rewrite but if your code constantly rewrites FLASH memory, you can ruin it rather quicky. 3) There are special procedures to write to FLASH (ARM makes it easy but it is still not as simple as writing to RAM).
The C language, compilers, etc are not able to generate chip/board specific flash routines. If you wish to use flash pages to store read/write data you are going to have to have at least a page worth of ram and some read/write routines. You would need to have very many of these variables to overcome the cost of ram and execution time needed to keep the master copy in flash. In general every time you write the value to flash, you will need to read out the whole page, erase the page, then write back the whole page with the one item changed. Now if you know how your flash works (Generally it is erase to ones and write zeros) you could read the prior version, compare differences and if an erase is not needed then do a write of that one item.
if you dont have dozens of variables you wish to do this with then, dont bother. You would want to declare a const something offset for each variable in this flash and have a read/write routine
const unsigned int off_items=0x000;
const unsigned int off_dollars=0x004;
...
unsigned int flash_read_nv ( unsigned int offset );
void flash_write_nv ( unsigned int offset, unsigned int data);
so this code that uses .data:
items++;
dollars=items*5;
Using your desire to keep the variables in flash becomes:
unsigned int ra,rb;
ra= flash_read_nv(off_items);
rb= flash_read_nv(off_dollars);
ra++;
rb=ra*5;
flash_write_nv(off_items,ra);
flash_write_nv(off_dollars,ra);
And of course the flash writes take hundreds to thousands of clock cycles or more to execute. Plus require 64, 128, or 256 bytes of ram (or more) depending on the flash page size.
Believe it or not, this page is the best thing that pops up when looking how to store data in flash with the AVR ARM compiler.
On page 359 of the manual (the one that comes with v7.50 of the compiler), does show this as a way to put data in flash:
#define FLASH _Pragma("location=\"FLASH\"")
On page 332 of the manual, however, it says this:
"Static and global objects declared const are allocated in ROM."
I tested it and it seems the pragma is unnecessary with the IAR compiler, as specifying const puts it in flash already (as stated in the original question).
From the other answers, it seems like the OP didn't really want to use flash. If anyone like me comes to this page to figure out how to store data in flash with the AVR ARM compiler, however, I hope my answer saves them some time.
In IAR, you can declare your array as follows:
__root __no_init const uint8_t myVect[50000] #0x12345678
Where, of course, 0x12345678 is the address in FLASH.
I'm trying to get a ram-resident image to checksum itself, which is proving easier said than done.
The code is first compiled on a cross development platform, generating an .elf output. A utility is used to strip out the binary image, and that image is burned to flash on the target platform, along with the image size. When the target is started, it copies the binary to the correct region of ram, and jumps to it. The utility also computes a checksum of all the words in the elf that are destined for ram, and that too is burned into the flash. So my image theoretically could checksum its own ram resident image using the a-priori start address and the size saved in flash, and compare to the sum saved in flash.
That's the theory anyway. The problem is that once the image begins executing, there is change in the .data section as variables are modified. By the time the sum is done, the image that has been summed is no longer the image for which the utility calculated the sum.
I've eliminated change due to variables defined by my application, by moving the checksum routine ahead of all other initializations in the app (which makes sense b/c why run any of it if an integrity check fails, right?), but the killer is the C run time itself. It appears that there are some items relating to malloc and pointer casting and other things that are altered before main() is even entered.
Is the entire idea of self-checksumming C code lame? If there was a way to force app and CRT .data into different sections, I could avoid the CRT thrash, but one might argue that if the goal is to integrity check the image before executing (most of) it, that initialized CRT data should be part of that. Is there a way to make code checksum itself in RAM like this at all?
FWIW, I seem stuck with a requirement for this. Personally I'd have thought that the way to go is to checksum the binary in the flash, before transfer to ram, and trust the loader and the ram. Paranoia has to end somewhere right?
Misc details: tool chain is GNU, image contains .text, .rodata and .data as one contiguously loaded chunk. There is no OS, this is bare metal embedded. Primary loader essentially memcpy's my binary into ram, at a predetermined address. No relocations occur. VM is not used. Checksum only needs testing once at init only.
updated
Found that by doing this..
__attribute__((constructor)) void sumItUp(void) {
// sum it up
// leave result where it can be found
}
.. that I can get the sum done before almost everything except the initialization of the malloc/sbrk vars by the CRT init, and some vars owned by "impure.o" and "locale.o". Now, the malloc/sbrk value is something I know from the project linker script. If impure.o and locale.o could be mitigated, might be in business.
update
Since I can control the entry point (by what's stated in flash for the primary loader), it seems the best angle of attack now is to use a piece of custom assembler code to set up stack and sdata pointers, call the checksum routine, and then branch into the "normal" _start code.
If the checksum is done EARLY enough, you could use ONLY stack variables, and not write to any data-section variables - that is, make EVERYTHING you need to perform the checksumming [and all preceding steps to get to that point] ONLY use local variables for storing things in [you can read global data of course].
I'm fairly convinced that the right way is to trust the flash & loader to load what is in the flash. If you want to checksum the code, sure, go and do that [assuming it's not being modified by the loader of course - for example runtime loading of shared libraries or relocation of the executable itself, such as random virtual address spaces and such]. But the data loaded from flash can't be relied upon once execution starts properly.
If there is a requirement from someone else that you should do this, then please explain to them that this is not feasible to implement, and that "the requirement, as it stands" is "broken".
I'd suggest approaching this like an executable packer, like upx.
There are several things in the other answers and in your question that, for lack of a better term, make me nervous.
I would not trust the loader or anything in flash that wasn't forced on me.
There is source code floating around on the net that was used to secure one of, I think, HTCs recent phones. Look around on forum.xda-developers.com and see if you can find it and use it for an example.
I would push back on this requirement. Cellphone manufacturers spend a lot of time on keeping their images locked down and, eventually, all of them are beaten. This seems like a vicious circle.
Can you use the linker script to place impure.o and locale.o before or after everything else, allowing you to checksum everything but those and the malloc/sbrk stuff? I'm guessing malloc and sbrk are called in the bootloader that loads your application, so the thrash caused by those cannot be eliminated?
It's not an answer to just tell you to fight this requirement, but I agree that this seems to be over-thought. I'm sure you can't go into any amount of detail, but I'm assuming the spec authors are concerned about malicious users/hackers, rather than regular memory corruption due to cosmic rays, etc. In this case, if a malicious user/hacker can change what's loaded into RAM, they can just change your checksumming routine (which is itself running from RAM, correct?) to always return a happy status, no matter how well the checksum routine they aren't running anymore is designed.
Even if they are concerned about regular memory corruption, this checksum routine would only catch that if the error occurred during the original copy to memory, which is actually the least likely time such an error would occur, simply because the system hasn't been running long enough to have a high probability of a corruption event.
In general, what you want to do is impossible, since on many (most?) platforms the program loader may "relocate" some program address constants.
Can you update the loader to perform the checksum test on the flash resident binary image, before it is copied to ram?
I am working on an embedded system, so memory is precious for me.
One issue that has been recurring is that I've been running out of memory space when attempting to compile a program for it. This is usually fixed by limiting the number of typedefs, etc that can take up a lot of space.
There is a macro generator that I use to create a file with a lot of #define's in it.
Some of these are simple values, others are boundary checks
ie
#define SIGNAL1 (float)0.03f
#define SIGNAL1_ISVALID(value) ((value >= 0.0f) && (value <= 10.0f))
Now, I don't use all of these defines. I use some, but not actually the majority.
I have been told that they don't actually take up any memory if they are not used, but I was unsure on this point. I'm hoping that by cutting out the unused ones that I can free up some extra memory (but again, I was told this is pointless).
Do unused #define's take up any memory space?
No, #defines take up no space unless they are used - #defines work like find/replace; whenever the compiler sees the left half, it'll replace it with the right half before it actually compiles.
So, if you have:
float f = SIGNAL1;
The compiler will literally interpret the statement:
float f = (float)0.03f;
It will never see the SIGNAL1, it won't show up in a debugger, etc.
This is usually fixed by limiting the number of typedefs, etc that can take up a lot of space.
You seem somewhat confused because typedef's do not take up space at runtime. They are merely aliases for data types. Now you may have instances of large structures (typedef'd or otherwise), but it is the instance that takes space, not the type definition. I wonder what 'etc' might cover in this statement.
Macro instances are replaced in the source code with their definition, and code generated accordingly, an unused macro does not result in any generated code.
Things that take up space are:
Executable code (functions/member functions)
Data instantiation (including C++ object instances)
The amount of space allocated to the stack (or stacks in a multi-threaded system).
What is left is typically available for dynamic memory allocation (RAM), or is unused or made for non-volatile storage (Flash/EPROM).
Reducing memory usage is primarily a case of selecting/designing efficient data structures, using appropriate data types, and efficient code and algorithm design. It is best to target the area that will get the greatest benefit. To see the size of objects and code in your application, get the linker to generate a map file. That will tell you which are the largest functions, as well as the sizes of global and static objects.
Source file text length is not a good guide to code size. Large amounts of C code is declarative (typically header files are all declarative), and do not generate memory occupying code or data.
An embedded system does not necessarily imply small memory, so you should specify. I have worked on systems with 64Mb RAM and 2Mb Flash, and even that is modest compared with many systems. A typical micro-controller with on-chip resources however, will generally have much less (especially SRAM which takes up a lot of chip area). Also whether your system is Harvard or Von Neumann architecture is relevant here, since in a Harvard architecture data and code spaces are separate, so we need to know what it is you are short of. If Von Neumann, the code/data usage is still relevant if the code is running from ROM, or is it is copied from ROM to RAM at run-time (i.e. different types of memory, even if they are in the same address space).
Clifford
Well, yes and no.
No, unused #defines won't increase the size of the resulting binary.
Yes, all #defines (whether used or unused) must be known by the compiler when building the binary.
By your question it's a bit ambiguous how you use the compiler, but it almost seems that you try to build directly on an embedded device; have you tried a cross-compiler? :)
unused #defines doesn't take up space in the resulting executable. They do take up memory in the compiler itself whist compiling.