I'm using Ryyst's code from here - How do I base64 encode (decode) in C? - to base64 encode an image file and insert it into a HTML document.
It works! - except on the second line of base64-encoded output there is a single stray "X" at the end of the line.
It's always the second line, and only the second line, no matter how large the binary file (I've tried many).
If I remove the stray "X" manually, the encoded data exactly matches the output of the base64 utility, and the image is correctly decoded by the browser.
I've tried adding "\0" to the ends of each char array to make sure they are properly terminated (made no difference). I've checked that "buffer" is always 60 bytes, and that output_length is always 80 bytes (they are). I've read and re-read Ryyst's code to see if anything there could cause it (didn't see anything, but I am a C n00b). I did a rain dance. I searched for a virgin to toss down a volcano (can't find either one around here). The bug is still there.
Here are the important bits of the code -
while (cgiFormFileRead(CoverImageFile, buffer, BUFFERLEN, &got) ==cgiFormSuccess)
{
if(got>0)
{
fputs(base64_encode(buffer, got, &output_length), targetfile);
fputs("\n", targetfile);
}
}
And the base64_encode function is -
char *base64_encode(const unsigned char *data, size_t input_length,
size_t *output_length)
{
*output_length = 4 * ((input_length + 2) / 3);
char *encoded_data = malloc(*output_length);
if (encoded_data == NULL)
return NULL;
int i = 0, j = 0;
for (i = 0, j = 0; i < input_length;)
{
uint32_t octet_a = i < input_length ? data[i++] : 0;
uint32_t octet_b = i < input_length ? data[i++] : 0;
uint32_t octet_c = i < input_length ? data[i++] : 0;
uint32_t triple = (octet_a << 0x10) + (octet_b << 0x08) + octet_c;
encoded_data[j++] = encoding_table[(triple >> 3 * 6) & 0x3F];
encoded_data[j++] = encoding_table[(triple >> 2 * 6) & 0x3F];
encoded_data[j++] = encoding_table[(triple >> 1 * 6) & 0x3F];
encoded_data[j++] = encoding_table[(triple >> 0 * 6) & 0x3F];
}
for (i = 0; i < mod_table[input_length % 3]; i++)
encoded_data[*output_length - 1 - i] = '=';
return encoded_data;
}
(as you can see, I'm also using the cgic library v 205, but I don't think the problem is from there because its giving the right number of bytes)
(And BUFFERLEN is a constant, equals 60.)
What am I doing wrong, guys?
(Even more frustratingly, I /did/ get Ryyst's algorithm to work flawlessly once before, so his code /does/ work.)
I'm compiling using gcc on an ARM-based Debian Linux system, if that makes any difference.
Comparing your function with the original you've deleted:
encoded_data[j++] = encoding_table[(triple >> 0 * 6) & 0x3F];
Apart from that, the function is the same, I'm guessing that's just a copy error.
The problem is you are using BUFFERLEN rather than looking at got, which returns the amount of data read, the second line doesn't read the full 60 characters so you are encoding whatever junk is at the end of the buffer.
Related
This development is being done on Windows in usermode.
I have two (potentially quite large) buffers, and I would like to know the number of bytes different between the two of them.
I wrote this myself just checking byte by byte, but this resulted in a quite slow implementation. As I'm comparing on the order of hundreds of megabytes, this is undesirable. I'm aware that I could optimize this though many different means, but this seems like a common problem that's probably got optimized solutions already out there, and there's no way I'm going to optimize this as effectively as if it was written by optimization experts.
Perhaps my Googling is inadequate, but I'm unable to find any other C or C++ functions that can count the number of different bytes between two buffers. Is there such a built in function to the C standard library, WinAPI, or C++ standard library that I just don't know of? Or do I need to manually optimize this?
I ended up writing this (perhaps somewhat poorly) optimized code to do the job for me. I was hoping it would vectorize this under the hood, but that doesn't appear to be happening unfortunately, and I didn't feel like digging around the SIMD intrinsics to do it manually. As a result, my bit fiddling tricks may end up making it slower, but it's still fast enough that it's no more than about 4% of my code's runtime (and almost all of that was memcmp). Whether or not it could be better, it's good enough for me.
I'll note that this is designed to be fast for my use case, where I'm expecting only rare differences.
inline size_t ComputeDifferenceSmall(
_In_reads_bytes_(size) char* buf1,
_In_reads_bytes_(size) char* buf2,
size_t size) {
/* size should be <= 0x1000 bytes */
/* In my case, I expect frequent differences if any at all are present. */
size_t res = 0;
for (size_t i = 0; i < (size & ~0xF); i += 0x10) {
uint64_t diff1 = *reinterpret_cast<uint64_t*>(buf1) ^
*reinterpret_cast<uint64_t*>(buf2);
if (!diff1) continue;
/* Bit fiddle to make each byte 1 if they're different and 0 if the same */
diff1 = ((diff1 & 0xF0F0F0F0F0F0F0F0ULL) >> 4) | (diff1 & 0x0F0F0F0F0F0F0F0FULL);
diff1 = ((diff1 & 0x0C0C0C0C0C0C0C0CULL) >> 2) | (diff1 & 0x0303030303030303ULL);
diff1 = ((diff1 & 0x0202020202020202ULL) >> 1) | (diff1 & 0x0101010101010101ULL);
/* Sum the bytes */
diff1 = (diff1 >> 32) + (diff1 & 0xFFFFFFFFULL);
diff1 = (diff1 >> 16) + (diff1 & 0xFFFFULL);
diff1 = (diff1 >> 8) + (diff1 & 0xFFULL);
diff1 = (diff1 >> 4) + (diff1 & 0xFULL);
res += diff1;
}
for (size_t i = (size & ~0xF); i < size; i++) {
res += (buf1[i] != buf2[i]);
}
return res;
}
size_t ComputeDifference(
_In_reads_bytes_(size) char* buf1,
_In_reads_bytes_(size) char* buf2,
size_t size) {
size_t res = 0;
/* I expect most pages to be identical, and both buffers should be page aligned if
* larger than a page. memcmp has more optimizations than I'll ever come up with,
* so I can just use that to determine if I need to check for differences
* in the page. */
for (size_t pn = 0; pn < (size & ~0xFFF); pn += 0x1000) {
if (memcmp(&buf1[pn], &buf2[pn], 0x1000)) {
res += ComputeDifferenceSmall(&buf1[pn], &buf2[pn], 0x1000);
}
}
return res + ComputeDifferenceSmall(
&buf1[size & ~0xFFF], &buf2[size & ~0xFFF], size & 0xFFF);
}
I'm trying to write the binary number of 16-bit signed integer to a file. I searched a lot and ofcourse I found many examples which converts integer variables to binary. But in my case these functions will not be efficient, because I need to convert 50e6 samples/s. Calling a function to convert each sample will need a lot of computing time.
So what I want to do is:
int array[] = {233, 431, 1024, ...}
for (i = 0; i < sizeof(array); i++){
fprintf(outfile, "%any_binary_format \n", array[i]);
}
result in the file should be:
0000000011101001
0000000110101111
0000010000000000
fprintf is intended for formatted output - the formatting being "human readable" text, it is therefore not the appropriate function to use if you want binary output. For that you should use fwrite():
for (i = 0; i < sizeof(array) / sizeof(*array); i++ )
{
fwrite (&array[i], sizeof(*array), 1, outfile ) ;
}
Note I have also fixed your loop termination to correctly iterate the number of elements in the array. But in fact the loop is unnecessary - the output is binary, the array is binary - you can just output the entire array thus:
fwrite( array, sizeof(array), 1, outfile ) ;
Your performance requirement of 50Msps will require write performance of around 95Mb/s sustained - that is a lot to ask, and unlikely to be achieved by writing one sample at a time. You may be better off using a memory mapped file, but unless you are using a real-time OS, there are no guarantees that you will sustain that output rate indefinitely - it only takes some other process to access the drive, and it may introduce an unacceptable delay.
Also note that the file must have been opened for binary output - especially on Windows to prevent translation of CR to CR+LF which will be disastrous for your sample data.
If you want to use printf you can use something like this:
#define BYTE_TO_BINARY_PATTERN "%c%c%c%c%c%c%c%c\n"
#define BYTE_TO_BINARY(byte) \
(byte & 0x80 ? '1' : '0'), \
(byte & 0x40 ? '1' : '0'), \
(byte & 0x20 ? '1' : '0'), \
(byte & 0x10 ? '1' : '0'), \
(byte & 0x08 ? '1' : '0'), \
(byte & 0x04 ? '1' : '0'), \
(byte & 0x02 ? '1' : '0'), \
(byte & 0x01 ? '1' : '0')
int main()
{
uint8_t value = 5;
printf(BYTE_TO_BINARY_PATTERN, BYTE_TO_BINARY(value));
return 0;
}
Should print 00000101. I use this sometimes in embedded code when debugging to check register values. Just replace printf with fprintf if you want to write the ascii binary strings to file.
If your compiler supports inline you don't need to worry about the overhead of a small function, take a look at this.
Anyway you can simply implement the function as a macro.
If you want a faster approach you can use a larger buffer (the size for the faster runtime is machine-dependent) for example char str[1 << 16], writing the results to the buffer and using fwrite/write to the out stream.
Another approach is to map the process via mmap/msync.
Anyway you don't need to look at a faster function, but rather a deeper knowledge of the system you're working on.
#define SHORT_WIDTH 16
#define TEST 1
#define PADDING 1 /* set to 0 if you don't need the leading 0s */
char *ShortToBin(unsigned short x, char *buffer) {
#if PADDING
int i;
for(i = 0; i < SHORT_WIDTH; ++i)
buffer[SHORT_WIDTH - i - 1] = '0' + ((x >> i) & 1);
return buffer;
#else
char *ptr = buffer + SHORT_WIDTH;
do {
*(--ptr) = '0' + (x & 1);
x >>= 1;
} while(x);
return ptr;
#endif
}
#if TEST
#include <stdio.h>
int main() {
short n;
char str[SHORT_WIDTH+1]; str[SHORT_WIDTH]='\0';
while(scanf("%hd", &n) == 1)
puts(ShortToBin(n, str));
return 0;
}
#endif
I am trying to thoroughly understand the code I found on Github. I am running this code on Eclipse (Version: 3.6.1
Build id: M20100909-0800).
I want to efficiently debug these lines of code:
for (index_X = 0; index_X < nb_MCU_X; index_X++) {
for (index_Y = 0; index_Y < nb_MCU_Y; index_Y++) {
for (index = 0; index < SOS_section.n; index++) {
uint32_t component_index = component_order[index];
int nb_MCU = ((SOF_component[component_index].HV >> 4) & 0xf) * (SOF_component[component_index].HV & 0x0f);
for (chroma_ss = 0; chroma_ss < nb_MCU; chroma_ss++) {
unpack_block(movie, & scan_desc, index, MCU);
iqzz_block(MCU, unZZ_MCU, DQT_table[SOF_component[component_index].q_table]);
IDCT(unZZ_MCU, YCbCr_MCU_ds[component_index] + (64 * chroma_ss));
}
upsampler(YCbCr_MCU_ds[component_index], YCbCr_MCU[component_index],
max_ss_h / ((SOF_component[component_index].HV >> 4) & 0xf), max_ss_v / ((SOF_component[component_index].HV) & 0xf), max_ss_h, max_ss_v);
}
if (color && (SOF_section.n > 1)) {
YCbCr_to_ARGB(YCbCr_MCU, RGB_MCU, max_ss_h, max_ss_v);
} else {
to_NB(YCbCr_MCU, RGB_MCU, max_ss_h, max_ss_v);
}
screen_cpyrect(index_Y * MCU_sy * max_ss_h, index_X * MCU_sx * max_ss_v, MCU_sy * max_ss_h, MCU_sx * max_ss_v, RGB_MCU);
}
}
The code above contains a number of loops and stepping over every line of code many times is laborious (nb_MCU_X is 18 and nb_MCU_Y is 32).
I tried to change the values of index_X and index_Y in Debug mode. I thought doing so would take me to a point in the program where more of the code will have been processed. However, Only index_X and index_Y changed to the values I gave them but all other dependent values did not change with them. Consequently, the behavior of the program was distorted and it began behaving erratically.
I tried setting breakpoints immediately after this section of code. However, it does not allow me to see the next step that occurs after the section of the code above is processed. I want to know instantly what the condition of the code will be when index_X or index_Y is at any value of my choosing.
Is there a way for me in Eclipse to go forward in time and have more iterations processed instead of stepping over each line of the code?
What should I do if, for example, index_Y is currently 0 but I want to go instantly to a point in the program where index_Y is 7 and the rest of the code has also changed accordingly?
In GIMP, you're able to save an image as a C header file. I did so with an XPM file, which looks like the image below:
If I were to save the XPM image as a C header file, GIMP will output this C header file.
In order to process each pixel of the given image data, the header pixel is called repeatedly. What I don't understand is what the header pixel does to process the data in the first place.
#define HEADER_PIXEL(data,pixel) {\
pixel[0] = (((data[0] - 33) << 2) | ((data[1] - 33) >> 4)); \
pixel[1] = ((((data[1] - 33) & 0xF) << 4) | ((data[2] - 33) >> 2)); \
pixel[2] = ((((data[2] - 33) & 0x3) << 6) | ((data[3] - 33))); \
data += 4; \
}
When I saw it in use in another person's code, they stated the byte order was in the wrong order and rearranged it themselves. They used it like this:
char *pixel, *data = header_data;
int i = width * height;
*processed_data = pixel = malloc(i * 4 + 1);
while(i-- > 0) {
pixel[0] = ((((data[2] - 33) & 0x3) << 6) | ((data[3] - 33)));
pixel[1] = ((((data[1] - 33) & 0xF) << 4) | ((data[2] - 33) >> 2));
pixel[2] = (((data[0] - 33) << 2) | ((data[1] - 33) >> 4));
pixel[3] = 0;
data += 4;
pixel += 4;
}
But that didn't really help me understand what is going on with all the bit shifting and bitwise or's and "why minus 33?" and so forth. If anyone can give an explanation on what is going on to process to the image data in the header, that would be much appreciated.
Thanks in advance!
Each pixel is represented by 3 bytes. These pixels are defined as a character array, named header_data.
The problem is that not every byte is a printable character that could exist in that header file.
This is solved by only using the printable characters 33 through 97. That gives 6 bits of information, so every four characters will give 24 bits, which can represent all permutations of 3 bytes.
The current project in Systems Programming is to come up with an ASCII compressor that removes the top zero bit and writes the contents to the file.
In order to facilitate decompression, the original file size is written to file, then the compressed char bytes. There are two files to run tests on- one that is 63 bytes long, and the other is 5344213 bytes. My code below works as expected for the first test file, as it writes 56 bytes of compressed text plus 4 bytes of file header.
However, when I try it on the long test file, the compressed version is 3 bytes shorter than the original, when it should be roughly 749KiB smaller, or 14% of original size. I've worked out the binary bit shift values for the first two write loops of the long test file, and they match up what is being recorded on my test printout.
while ( (characters= read(openReadFile, unpacked, BUFFER)) >0 ){
unsigned char packed[7]; //compression storage
int i, j, k, writeCount, endLength, endLoop;
//loop through the buffer array
for (i=0; i< characters-1; i++){
j= i%7;
//fill up the compressed array
packed[j]= packer(unpacked[i], unpacked[i+1], j);
if (j == 6){
writeCalls++; //track how many calls made
writeCount= write(openWriteFile, packed, sizeof (packed));
int packedSize= writeCount;
for (k=0; k<7 && writeCalls < 10; k++)
printf("%X ", (int)packed[k]);
totalWrittenBytes+= packedSize;
printf(" %d\n", packedSize);
memset(&packed[0], 0, sizeof(packed)); //clear array
if (writeCount < 0)
printOpenErrors(writeCount);
}
//end of buffer array loop
endLength= characters-i;
if (endLength < 7){
for (endLoop=0; endLoop < endLength-1; endLoop++){
packed[endLoop]= packer(unpacked[endLoop], unpacked[endLoop+1], endLoop);
}
packed[endLength]= calcEndBits(endLength, unpacked[endLength]);
}
} //end buffer array loop
} //end file read loop
The packer function:
//calculates the compressed byte value for the array
char packer(char i, char j, int k){
char packStyle;
switch(k){
//shift bits based on mod value with 8
case 0:
packStyle= ((i & 0x7F) << 1) | ((j & 0x40) >> 6);
break;
case 1:
packStyle= ((i & 0x3F) << 2) | ((j & 0x60) >> 5);
break;
case 2:
packStyle= ((i & 0x1F) << 3) | ((j & 0x70) >> 4);
break;
case 3:
packStyle= ((i & 0x0F) << 4) | ((j & 0x78) >> 3);
break;
case 4:
packStyle= ((i & 0x07) << 5) | ((j & 0x7C) >> 2);
break;
case 5:
packStyle= ((i & 0x03) << 6) | ((j & 0x7E) >> 1);
break;
case 6:
packStyle= ( (i & 0x01 << 7) | (j & 0x7F));
break;
}
return packStyle;
}
I've verified that there are 7 bytes written out every time the packed buffer is flushed, and there are 763458 write calls made for the long file, which match up to 5344206 bytes written.
I'm getting the same hex codes from the printout that I worked out in binary beforehand, and I can see the top bit of every byte removed. So why aren't the bit shifts being reflected in the results?
Ok, since this is homework I'll just give you a few hints without giving out a solution.
First are you sure that the 56 bytes you get on the first file are the right bytes? Sure the count looks good, but you got lucky on count (proof is the second test file). I can immediately see at least two key mistakes in the code.
To make sure you have the right output, the byte count is not enough. You need to dig deeper. How about checking the bytes themselves one by one. 63 characters is not that much to go heh? There are many ways you can do this. You could use od (a pretty good Linux/Unix tool to look at the binary contents of files, if you're on Windows use some Hex editor). Or you could print out debug information from within your program.
Good luck.
Why do you expect the output to be 14% shorter than the input? How could it, when you store a byte into packed as many times as there are input bytes, except for the last group? The size of the output will always be within 7 of the size of the input.