Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 11 days ago.
Improve this question
There's a lot of file formats that require writing a set of integers to the file for various reasons. And I've recently found myself doing this
uint32_t u32;
u32 = ...
fwrite(&u32, 4, 1, file);
u32 = ...
fwrite(&u32, 4, 1, file);
...
which seems unnecessary and tedious. Is this a good practice? Or is there a better solution?
"Or is there a better solution?"
Yes. At a minimum use if (fwrite(&u32, sizeof u32, 1, file) == 1) ....
(Avoid magic numbers, test I/O result, ...)
Better beyond that, best solution depends on context and coding goals not stated.
When reading and writing binary-format files, my preference is to define sets of "get" and "put" functions, like this:
/* read a 4-byte little-endian integer */
int32_t get32(FILE *fp)
{
int32_t ret = 0;
ret |= getc(fp);
ret |= getc(fp) << 8;
ret |= (uint32_t)getc(fp) << 16;
ret |= (uint32_t)getc(fp) << 24;
return ret;
}
/* write a 4-byte little-endian integer */
void put32(int32_t val, FILE *fp)
{
putc( val & 0xff, fp);
putc((val >> 8) & 0xff, fp);
putc((val >> 16) & 0xff, fp);
putc((val >> 24) & 0xff, fp);
}
Then instead of your fwrite(&u32, 4, 1, file), I'd call put32(u32, file).
This is, admittedly, not that much more convenient than fwrite, but it gives precise control over byte order, which is the other big issue when it comes to reading/writing binary data files.
Also, getting these "get" and "put" routines absolutely right can be a little tricky. The simplified versions I've shown aren't perfect, as they play a little fast-and-loose with signed/unsigned distinctions, and the get32 I've shown doesn't have any EOF handling.
Typing in all the get's and put's can indeed be a nuisance, but in my experience it's actually a lot less of a nuisance, in the end, than some of the allegedly "more convenient" techniques people try to use. For example, rather than reading and writing one variable at a time, it's quite popular to put all your variables into a structure and then read and write whole instances of that structure — but then you get to spend many, many hours on the joyless task of trying to manage your structure padding and alignment.
If you truly have so many of these to read and write that manually maintaining the get's and put's is unworkable, I'd say it's time to look into some kind of interface generator to write the serialization and deserialization code for you, based on a nice, high-level description of your data file format and your mapping of data file fields to variable or member names in your code.
Related
I'm trying to implement a lossless audio codec that will be able to process data coming in at roughly 190 kHz to then be stored to an SD card using SPI DMA. I've found that the algorithm basically works, but has certain bottlenecks that I can't seem to overcome. I was hoping to get some advice on how to best optimize a certain portion of the code that I found to be the "slowest". I'm writing in C on a TI DSP and am using -O3 optimization.
for (j = packet_to_write.bfp_bits; j>0; j--)
{
encoded_data[(filled/16)] |= ((buf_filt[i] >> (j- 1)) & 1) << (filled++ % 16);
}
In this section of code, I am taking X number of bits from the original data and fitting it into a buffer of encoded data. I've found that the loop is fairly costly and when I am working with a set of data represented by 8+ bits, then this code is too slow for my application. Loop unrolling doesn't really work here since each block of data can be represented by a different number of bits. The "filled" variable represents a bit counter filling up Uint16 indices in the encoded_data buffer.
I'd like some help understanding where bottlenecks may come from in this snippet of code (and hopefully I can take those findings and apply that to other areas of the algo). The authors of the paper that I'm reading (whose algorithm I'm trying to replicate) noted that they used a mixture of C and assembly code, but I'm not sure how assembly would be useful in this case.
Finally, the code itself is functional and I have done some extensive testing on actual audio samples. It's just not fast enough for real-time!
Thanks!
You really need to change the representation that you use for the output data. Instead of just a target buffer and the number of bits written, expand this to:
//complete words that have been written
uint16_t *encoded_data;
//number of complete words that have been written
unsigned filled_words;
//bits waiting to be written to encoded_data, LSB first
uint32_t encoded_bits;
//number of bits in encoded_bits
unsinged filled_bits;
This uses a single 32-bit word to buffer bits until we have enough to write out a complete uint16_t. This greatly simplifies the shifting and masking, because you always have at least 16 free bits to write into.
Then you can write out n bits of any source word like this:
void write_bits(uint16_t bits, unsigned n) {
uint32_t mask = ((uint32_t)0x0FFFF) >> (16-n);
encoded_bits |= (bits&mask) << filled_bits;
filled_bits += n;
if (filled_bits >= 16) {
encoded_data[filled_words++] = (uint16_t)encoded_bits;
encoded_bits >>= 16;
filled_bits -= 16;
}
}
and instead of your loop, you just write
write_bits(buf_filt[i], packet_to_write.bfp_bits);
No one-bit-at-a-time operations are required.
Just a quick question concerning the rust programming language.
Assume you had the following in C:
uint8_t *someblockofdata; /* has certain length of 4 */
uint32_t *anotherway = (uint32_t*) someblockofdata;
Regardless of the code not being all that useful and rather ugly, how would I go about doing that in rust? Say you have a &[u8]with a length divisible by 4, how would you "convert" it to a &[u32] and back (preferrably avoiding unsafe code as much as possible and retaining as much speed as possible).
Just to be complete, the case where I would want to do that is an application which reads u8s from a file and then manipulates those.
Reinterpret casting a pointer is defined between pointers to objects of alignment-compatible types, and it may be valid in some implementations, but it's non-portable. For one thing, the result depends on the endianness (byte order) of your data, so you may lose performance anyway through byte-swapping.
First rewrite your C as follows, verify that it does what you expect, and then translate it to Rust.
// If the bytes in the file are little endian (10 32 means 0x3210), do this:
uint32_t value = someblockofdata[0] | (someblockofdata[1] << 8)
| (someblockofdata[2] << 16) | (someblockofdata[3] << 24);
// If the bytes in the file are big endian (32 10 means 0x3210), do this:
uint32_t value = someblockofdata[3] | (someblockofdata[2] << 8)
| (someblockofdata[1] << 16) | (someblockofdata[0] << 24);
// Middle endian is left as an exercise for the reader.
I wrote my implementation of a program, in C, in which I can encrypt a text file and vice versa.
The BLOWFISH algorithm is the standard one provided.
But then my thought is this: if I create a set of 4 chars in a long file, let's say 0x12345678, I can decode it because I know the proper order in which I read the file.
On the other hand, using a pre-made function like memcpy(), the content read is ordered like as 0x87654321, not as my previous function do. But the algorithm used is the same.
Is there a "standard" way to read and acquire data from a file, or both of the previous examples are fine? In an online site (blowfish online) the version used with memcpy() does not comply with that, when using the ECB mode. The version that acquires the data like 0x1234567 is working fine with the site. (Working means making an encrypted file with my program and decrypting it online).
For example, if I code and decode stuff with my program, that stuff should be (knowing the key) coded/decoded by other people who don't know my program (as general rule, at least)?
EDIT: the memcpy() function translate the lowest index of the array to the right end of the INT number.
This is the code which manipulate data for 64bit block:
memcpy(&cl, &file_cache[i], sizeof(unsigned long));
memcpy(&cr, &file_cache[i + 4], sizeof(unsigned long));
And this is the core part (is working fine, by correctly rearranging the read from the buffer, i.e. looping 8 times for each block) of the same portion which uses bitwise magic instead of memcpy() and comply with the endianess problem:
if (i==0){
cl <<= 24;
L |= 0xff000000 & cl;
}
else if (i==1){
cl <<= 16;
L |= 0x00ff0000 & cl;
}
else if (i==2){
cl <<= 8;
L |= 0x0000ff00 & cl;
}
else if (i==3){
//cl <<= 24;
L |= 0x000000ff & cl;
}
else if (i==4){
cl <<= 24;
R |= 0xff000000 & cl;
}
else if (i==5){
cl <<= 16;
R |= 0x00ff0000 & cl;
}
else if (i==6){
cl <<= 8;
R |= 0x0000ff00 & cl;
}
else if (i==7){
//cl <<= 8;
R |= 0x000000ff & cl;
}
Then L and R are sent to be encrypted. This last implementation works if I use other blowfish versions on line, so in principle should be better.
Which implementation is faster/better/lighter/stronger?
If the memcpy() is the one adviced, there's a convenient and faster way to reverse/mirroring the content of cl and cr?
Note that the leftmost byte is usually the "first byte send/received" for cryptography; i.e. if you have an array then the lowest index is to the left. If nothing has been specified, then this is the ad-hoc standard.
However, the Blowfish test vectors - as indicated by GregS - explicitly specify this default order, so there is no need to guess:
...
All data is shown as a hex string with 012345 loading as
data[0]=0x01;
data[1]=0x23;
data[2]=0x45;
...
As long as your code produces the same test vectors then you're OK, keeping in mind that your input / output should comply with the order of the test vectors.
It is highly recommended to make any cryptographic API operate on bytes (or rather, octets), not on other data types even if those bytes are internally handled as 32 or 64 bit words. The time required for conversion to/from bytes should be minimal compared to the actual encryption/decryption.
If you read the file as a sequence of 4-byte words then you would need to account for the endianness of those words in the memory layout, swapping the bytes as required to ensure the individual bytes are handled in a consistent order.
However, if you read/write your file as a sequence of bytes, and stored directly in sequence in memory (in an unsigned char array for example) then the data in file should have the same layout as in memory. That way you can obtain a consistent encoding/decoding whether you encode directly from/to memory or from/to file.
I am programming a PIC18F94K20 to work in conjunction with a MCP7941X I2C RTCC ship and a 24AA128 I2C CMOS Serial EEPROM device. Currently I have code which successfully intialises the seconds/days/etc values of the RTCC and starts the timer, toggling a LED upon the turnover of every second.
I am attempting to augment the code to read back the correct data for these values, however I am running into trouble when I try to account for the various 'extra' bits in the values. The memory map may help elucidate my problem somewhat:
Taking, for example, the hours column, or the 02h address. Bit 6 is set as 1 to toggle 12 hour time, adding 01000000 to the hours bit. I can read back the entire contents of the byte at this address, but I want to employ an if statement to detect whether 12 or 24 hour time is in place, and adjust accordingly. I'm not worried about the 10-hour bits, as I can calculate that easily enough with a BCD conversion loop (I think).
I earlier used the bitwise OR operator in C to augment the original hours data to 24. I initialised the hours in this particular case to 0x11, and set the 12 hour control bit which is 0x64. When setting the time:
WriteI2C(0x11|0x64);
which as you can see uses the bitwise OR.
When reading back the hours, how can I incorporate operators into my code to separate the superfluous bits from the actual time bits? I tried doing something like this:
current_seconds = ReadI2C();
current_seconds = ST & current_seconds;
but that completely ruins everything. It compiles, but the device gets 'stuck' on this sequence.
How do I separate the ST / AMPM / VBATEN bits from the actual data I need, and what would a good method be of implementing for loops for the various circumstances they present (e.g. reading back 12 hour time if bit 6 = 0 and 24 hour time if bit6 = 1, and so on).
I'm a bit of a C novice and this is my first foray into electronics so I really appreciate any help. Thanks.
To remove (zero) a bit, you can AND the value with a mask having all other bits set, i.e., the complement of the bits that you wish to zero, e.g.:
value_without_bit_6 = value & ~(1<<6);
To isolate a bit within an integer, you can AND the value with a mask having only those bits set. For checking flags this is all you need to do, e.g.,
if (value & (1<<6)) {
// bit 6 is set
} else {
// bit 6 is not set
}
To read the value of a small integer offset within a larger one, first isolate the bits, and then shift them right by the index of the lowest bit (to get the least significant bit into correct position), e.g.:
value_in_bits_4_and_5 = (value & ((1<<4)|(1<<5))) >> 4;
For more readable code, you should use constants or #defined macros to represent the various bit masks you need, e.g.:
#define BIT_VBAT_EN (1<<3)
if (value & BIT_VBAT_EN) {
// VBAT is enabled
}
Another way to do this is to use bitfields to define the organisation of bits, e.g.:
typedef union {
struct {
unsigned ones:4;
unsigned tens:3;
unsigned st:1;
} seconds;
uint8_t byte;
} seconds_register_t;
seconds_register_t sr;
sr.byte = READ_ADDRESS(0x00);
unsigned int seconds = sr.seconds.ones + sr.seconds.tens * 10;
A potential problem with bitfields is that the code generated by the compiler may be unpredictably large or inefficient, which is sometimes a concern with microcontrollers, but obviously it's nicer to read and write. (Another problem often cited is that the organisation of bit fields, e.g., endianness, is largely unspecified by the C standard and thus not guaranteed portable across compilers and platforms. However, it is my opinion that low-level development for microcontrollers tends to be inherently non-portable, so if you find the right bit layout I wouldn't consider using bitfields “wrong”, especially for hobbyist projects.)
Yet you can accomplish similarly readable syntax with macros; it's just the macro itself that is less readable:
#define GET_SECONDS(r) ( ((r) & 0x0F) + (((r) & 0x70) >> 4) * 10 )
uint8_t sr = READ_ADDRESS(0x00);
unsigned int seconds = GET_SECONDS(sr);
Regarding the bit masking itself, you are going to want to make a model of that memory map in your microcontroller. The simplest, cudest way to do that is to #define a number of bit masks, like this:
#define REG1_ST 0x80u
#define REG1_10_SECONDS 0x70u
#define REG1_SECONDS 0x0Fu
#define REG2_10_MINUTES 0x70u
...
And then when reading each byte, mask out the data you are interested in. For example:
bool st = (data & REG1_ST) != 0;
uint8_t ten_seconds = (data & REG1_10_SECONDS) >> 4;
uint8_t seconds = (data & REG1_SECONDS);
The important part is to minimize the amount of "magic numbers" in the source code.
Writing data:
reg1 = 0;
reg1 |= st ? REG1_ST : 0;
reg1 |= (ten_seconds << 4) & REG1_10_SECONDS;
reg1 |= seconds & REG1_SECONDS;
Please note that I left out the I2C communication of this.
How to construct a request message with a given message specification, and then send to server thought c socket? Binary protocol is employed for Client and Server communication. Are the following approaches correct?
Given message specification:
Field Fomat Length values
------------ ------ ------ --------
requesID Uint16 2 20
requestNum Uint16 2 100
requestTitle String 10 data sring
/************** approach 1 ****************/
typedef unsigned short uint16;
typedef struct {
uint16 requesID [2];
uint16 requestNum [2];
unsigned char requestTitle [10];
}requesMsg;
…
requesMsg rqMsg;
memcpy(rqMsg.requesID, "\x0\x14", 2); //20
memcpy(rqMsg.requesNum, "\x0\x64", 2); //100
memcpy(rqMsg.requesTitle, "title01 ", 10);
…
send(sockfd, &rqMsg, sizeof(rqMsg), 0);
/************** approach 2 ****************/
unsigned char rqMsg[14];
memset(rqMsg, 0, 14);
memcpy(rqMsg, "\x0\x14", 2);
memcpy(rqMsg+2, "\x0\x64", 2);
memcpy(rqMsg+4, "title01 ", 10);
…
send(sock, &rqMsg, sizeof(rqMsg), 0);
I'm afraid you are misunderstanding something: The length column appears to tell you the length in bytes, so if you receive a uint16 you receive 2 bytes.
Your first approach could lead to serious problem through data structure alignment. If I were in your shoes I'd prefer the second approach and fill in the bytes on my own into a byte array.
A general note about filling fields here: I'ts useless to use memcpy for "native" fields like uint16, etc. It might work but is simply a waste of runtime. You can fill in fields of a struct simply assigning them a value like rqMsg.requesID = 20;
Another issue is the question of byte order or endianness of your binary protocol.
As a whole package, I'd implement a "serializeRequest" function taking fields of your struct and convert it into a byte array according to the protocol.
Both of them are at least partially correct but I much prefer the first one because it allows for quick and natural data manipulations and access and leaves less space for errors compared to manual indexing. As a bonus you can even copy and assign structure values as a whole and in C it works as expected.
But for any outgoing data you should make sure to use a "packed" struct. Not only it will reduce the amount of data transmitted down to the array-based implementation figure but it will also make sure that the fields alignments are the same in all the programs involved. For most C compilers I tried (GCC included) it can be done with __attribute__((__packed__)) attribute, but there are different compilers that require different attributes or even a different keyword.
Also endianness control may be required if your application is going to run on different architectures (ARM clients vs x86_64 server is a major example). I just use some simple macros like these to preprocess each field individually before doing any calculations or data output:
#define BYTE_SWAP16(num) ( ((num & 0xFF) << 8) | ((num >> 8) & 0xFF) )
#define BYTE_SWAP32(num) ( ((num>>24)&0xff) | ((num<<8)&0xff0000) | ((num>>8)&0xff00) | ((num<<24)&0xff000000) )
But you can use different approaches like BCD encoding, separate decoding functions or something else.
Also notice that uint16_t is already a 2-byte value. You probably don't need two of them to store your single values.