Why is zlib deflate() hanging? - c

My issue is that my program hangs on use of zlib's deflate() function.
I first initialize my z_stream, as follows:
int setupGzipOutputStream(z_stream zStream) {
int zError;
zStream.zalloc = Z_NULL;
zStream.zfree = Z_NULL;
zStream.opaque = Z_NULL;
zError = deflateInit(&zStream, Z_COMPRESSION_LEVEL);
/* error handling code to test if zError != Z_OK... */
return EXIT_SUCCESS;
}
I attempt to write data to my z-stream with the following function:
int compressDataToGzipOutputStream(unsigned char *myData, z_stream zStream, Boolean flushZStreamFlag) {
int zError;
int zOutHave;
FILE *outFp = stdout;
unsigned char zBuffer[Z_BUFFER_MAX_LENGTH] = {0};
zStream.next_in = myData;
zStream.avail_in = strlen(myData); /* myData is a null-terminated string */
do {
zStream.avail_out = Z_BUFFER_MAX_LENGTH;
zStream.next_out = zBuffer;
zError = deflate(&zStream, (flushZStreamFlag == kFalse) ? Z_NO_FLUSH : Z_FINISH);
/* error handling code to test if zError != Z_OK... */
zOutHave = Z_BUFFER_MAX_LENGTH - zStream.avail_out;
fwrite(zBuffer, sizeof(unsigned char), zOutHave, outFp);
fflush(outFp);
} while (zStream.avail_out == 0);
return EXIT_SUCCESS;
}
I call these two functions (with simplifications for the purpose of asking this question) as follows:
z_stream zOutStream;
setupGzipOutputStream(zOutStream);
compressDataToGzipOutputStream(data, zOutStream, kFalse);
compressDataToGzipOutputStream(data, zOutStream, kFalse);
...
compressDataToGzipOutputStream(data, zOutStream, kTrue);
I then break down the zOutStream struct with deflateEnd().
The kTrue value on the last compression step sends the Z_FINISH flag to deflate(), instead of Z_NO_FLUSH.
It hangs on the following line:
zError = deflate(&zStream, (flushZStreamFlag == kFalse) ? Z_NO_FLUSH : Z_FINISH);
I then tried using gdb. I set a break at this line, the line where the program hangs.
At this breakpoint, I can see the values of the variables zStream, flushZStreamFlag and others. The zStream variable is not NULL, which I can verify with print zStream, print zStream.next_in, etc. which are populated with my data of interest.
If I type next in gdb, then this line of code is processed and the entire process hangs, which I verify with log statements before and after this line of code. The "before" log statement shows up, but the "after" statement does not.
My question is: Why is deflate() hanging here? Am I not initializing the output stream correctly? Not using deflate() correctly? I've been banging my head on the wall trying to solve this, but no luck. Thanks for any advice you might have.

Your functions should take a pointer to a z_stream, rather than passing the struct in. Your init function is initialising what is effectively a local copy, which will be discarded. Then your compression function will have a garbage z_stream passed to it.
e.g:
int setupGzipOutputStream(z_stream *zStream) {
int zError;
zStream->zalloc = Z_NULL;
...
}
... etc.
It also looks like your compression function is not taking into account the null on the end of the string, so that might cause you problems when you try to re-inflate your data.
zStream.avail_in = strlen(myData);
Might want to be:
zStream.avail_in = strlen(myData) + 1;

Related

FFMPEG remux sample without writing to file

Let's consider this very nice and easy to use remux sample by horgh.
I'd like to achieve the same task: convert an RTSP H264 encoded stream to a fragmented MP4 stream.
This code does exactly this task.
However I don't want to write the mp4 onto disk at all, but I need to get a byte buffer or array in C with the contents that would normally written to disk.
How is that achievable?
This sample uses vs_open_output to define the output format and this function needs an output url.
If I would get rid of outputting the contents to disk, how shall I modify this code?
Or there might be better alternatives as well, those are also welcomed.
Update:
As szatmary recommended, I have checked his example link.
However as I stated in the question I need the output as buffer instead of a file.
This example demonstrates nicely how can I read my custom source and give it to ffmpeg.
What I need is how can open the input as standard (with avformat_open_input) then do my custom modification with the packets and then instead writing to file, write to a buffer.
What have I tried?
Based on szatmary's example I created some buffers and initialization:
uint8_t *buffer;
buffer = (uint8_t *)av_malloc(4096);
format_ctx = avformat_alloc_context();
format_ctx->pb = avio_alloc_context(
buffer, 4096, // internal buffer and its size
1, // write flag (1=true, 0=false)
opaque, // user data, will be passed to our callback functions
0, // no read
&IOWriteFunc,
&IOSeekFunc
);
format_ctx->flags |= AVFMT_FLAG_CUSTOM_IO;
AVOutputFormat * const output_format = av_guess_format("mp4", NULL, NULL);
format_ctx->oformat = output_format;
avformat_alloc_output_context2(&format_ctx, output_format,
NULL, NULL)
Then of course I have created 'IOWriteFunc' and 'IOSeekFunc':
static int IOWriteFunc(void *opaque, uint8_t *buf, int buf_size) {
printf("Bytes read: %d\n", buf_size);
int len = buf_size;
return (int)len;
}
static int64_t IOSeekFunc (void *opaque, int64_t offset, int whence) {
switch(whence){
case SEEK_SET:
return 1;
break;
case SEEK_CUR:
return 1;
break;
case SEEK_END:
return 1;
break;
case AVSEEK_SIZE:
return 4096;
break;
default:
return -1;
}
return 1;
}
Then I need to write the header to the output buffer, and the expected behaviour here is to print "Bytes read: x":
AVDictionary * opts = NULL;
av_dict_set(&opts, "movflags", "frag_keyframe+empty_moov", 0);
av_dict_set_int(&opts, "flush_packets", 1, 0);
avformat_write_header(output->format_ctx, &opts)
In the last line during execution, it always runs into segfault, here is the backtrace:
#0 0x00007ffff7a6ee30 in () at /usr/lib/x86_64-linux-gnu/libavformat.so.57
#1 0x00007ffff7a98189 in avformat_init_output () at /usr/lib/x86_64-linux-gnu/libavformat.so.57
#2 0x00007ffff7a98ca5 in avformat_write_header () at /usr/lib/x86_64-linux-gnu/libavformat.so.57
...
The hard thing for me with the example is that it uses avformat_open_input.
However there is no such thing for the output (no avformat_open_ouput).
Update2:
I have found another example for reading: doc/examples/avio_reading.c.
There are mentions of a similar example for writing (avio_writing.c), but ffmpeg does not have this available (at least in my google search).
Is this task really this hard to solve? standard rtsp input to custom avio?
Fortunately ffmpeg.org is down. Great.
It was a silly mistake:
In the initialization part I called this:
avformat_alloc_output_context2(&format_ctx, output_format,
NULL, NULL)
However before this I already put the avio buffers into format_ctx:
format_ctx->pb = ...
Also, this line is unnecessary:
format_ctx = avformat_alloc_context();
Correct order:
AVOutputFormat * const output_format = av_guess_format("mp4", NULL, NULL);
avformat_alloc_output_context2(&format_ctx, output_format,
NULL, NULL)
format_ctx->pb = avio_alloc_context(
buffer, 4096, // internal buffer and its size
1, // write flag (1=true, 0=false)
opaque, // user data, will be passed to our callback functions
0, // no read
&IOWriteFunc,
&IOSeekFunc
);
format_ctx->flags |= AVFMT_FLAG_CUSTOM_IO;
format_ctx->oformat = output_format; //might be unncessary too
Segfault is gone now.
You need to write a AVIOContext implementation.

Strategy for cycling trough preexisting set of variables in c

I’m trying to program a HMI console to read a file from an USB pen drive and display its data on the screen. This is a csv file and the objective is to store the interpreted data to HMI console memory, which the HMI console later interprets. The macros on these consoles run in C (not C++).
I have no issue with both reading and interpreting the file, the issue that the existing function (not accessible to me, shown below) to write in the console memory only interprets char.
int WriteLocal( const char *type, int addr, int nRegs, void *buf , int flag );
Parameter: type is the string of "LW","LB" etc;
address is the Operation address ;
nRegs is the length of read or write ;
buf is the buffer which store the reading or writing data
flag is 0,then codetype is BIN,is 1 then codetype is BCD;
return value : 1 , Operation success
0 , Operation fail.
As my luck would have it I need to write integer values. What are available to me are the variables for each memory position. These are preexisting and are named individually such as:
int WR_LW200;
int WR_LW202;
int WR_LW204;
...
int WR_LW20n;
Ideally we could have a vector with all the names of the variables but unfortunately this is not possible. I could manually write every single variable but I need to do 300 of these…
must be a better way, right?
Just to give you a look on how it ended up looking:
int* arr[50][5] = { {&WR_LW200, &WR_LW400, &WR_LW600, &WR_LW800, &WR_LW1000},
{&WR_LW202, &WR_LW402, &WR_LW602, &WR_LW802, &WR_LW1002},
{&WR_LW204, &WR_LW404, &WR_LW604, &WR_LW804, &WR_LW1004},
{&WR_LW206, &WR_LW406, &WR_LW606, &WR_LW806, &WR_LW1006},
{&WR_LW208, &WR_LW408, &WR_LW608, &WR_LW808, &WR_LW1008},
{&WR_LW210, &WR_LW410, &WR_LW610, &WR_LW810, &WR_LW1010},
{&WR_LW212, &WR_LW412, &WR_LW612, &WR_LW812, &WR_LW1012},
{&WR_LW214, &WR_LW414, &WR_LW614, &WR_LW814, &WR_LW1014},
{&WR_LW216, &WR_LW416, &WR_LW616, &WR_LW816, &WR_LW1016},
{&WR_LW218, &WR_LW418, &WR_LW618, &WR_LW818, &WR_LW1018},
{&WR_LW220, &WR_LW420, &WR_LW620, &WR_LW820, &WR_LW1020},
{&WR_LW222, &WR_LW422, &WR_LW622, &WR_LW822, &WR_LW1022},
{&WR_LW224, &WR_LW424, &WR_LW624, &WR_LW824, &WR_LW1024},
{&WR_LW226, &WR_LW426, &WR_LW626, &WR_LW826, &WR_LW1026},
{&WR_LW228, &WR_LW428, &WR_LW628, &WR_LW828, &WR_LW1028},
{&WR_LW230, &WR_LW430, &WR_LW630, &WR_LW830, &WR_LW1030},
{&WR_LW232, &WR_LW432, &WR_LW632, &WR_LW832, &WR_LW1032},
{&WR_LW234, &WR_LW434, &WR_LW634, &WR_LW834, &WR_LW1034},
{&WR_LW236, &WR_LW436, &WR_LW636, &WR_LW836, &WR_LW1036},
{&WR_LW238, &WR_LW438, &WR_LW638, &WR_LW838, &WR_LW1038},
{&WR_LW240, &WR_LW440, &WR_LW640, &WR_LW840, &WR_LW1040},
{&WR_LW242, &WR_LW442, &WR_LW642, &WR_LW842, &WR_LW1042},
{&WR_LW244, &WR_LW444, &WR_LW644, &WR_LW844, &WR_LW1044},
{&WR_LW246, &WR_LW446, &WR_LW646, &WR_LW846, &WR_LW1046},
{&WR_LW248, &WR_LW448, &WR_LW648, &WR_LW848, &WR_LW1048},
{&WR_LW250, &WR_LW450, &WR_LW650, &WR_LW850, &WR_LW1050},
{&WR_LW252, &WR_LW452, &WR_LW652, &WR_LW852, &WR_LW1052},
{&WR_LW254, &WR_LW454, &WR_LW654, &WR_LW854, &WR_LW1054},
{&WR_LW256, &WR_LW456, &WR_LW656, &WR_LW856, &WR_LW1056},
{&WR_LW258, &WR_LW458, &WR_LW658, &WR_LW858, &WR_LW1058},
{&WR_LW260, &WR_LW460, &WR_LW660, &WR_LW860, &WR_LW1060},
{&WR_LW262, &WR_LW462, &WR_LW662, &WR_LW862, &WR_LW1062},
{&WR_LW264, &WR_LW464, &WR_LW664, &WR_LW864, &WR_LW1064},
{&WR_LW266, &WR_LW466, &WR_LW666, &WR_LW866, &WR_LW1066},
{&WR_LW268, &WR_LW468, &WR_LW668, &WR_LW868, &WR_LW1068},
{&WR_LW270, &WR_LW470, &WR_LW670, &WR_LW870, &WR_LW1070},
{&WR_LW272, &WR_LW472, &WR_LW672, &WR_LW872, &WR_LW1072},
{&WR_LW274, &WR_LW474, &WR_LW674, &WR_LW874, &WR_LW1074},
{&WR_LW276, &WR_LW476, &WR_LW676, &WR_LW876, &WR_LW1076},
{&WR_LW278, &WR_LW478, &WR_LW678, &WR_LW878, &WR_LW1078},
{&WR_LW280, &WR_LW480, &WR_LW680, &WR_LW880, &WR_LW1080},
{&WR_LW282, &WR_LW482, &WR_LW682, &WR_LW882, &WR_LW1082},
{&WR_LW284, &WR_LW484, &WR_LW684, &WR_LW884, &WR_LW1084},
{&WR_LW286, &WR_LW486, &WR_LW686, &WR_LW886, &WR_LW1086},
{&WR_LW288, &WR_LW488, &WR_LW688, &WR_LW888, &WR_LW1088},
{&WR_LW290, &WR_LW490, &WR_LW690, &WR_LW890, &WR_LW1090},
{&WR_LW292, &WR_LW492, &WR_LW692, &WR_LW892, &WR_LW1092},
{&WR_LW294, &WR_LW494, &WR_LW694, &WR_LW894, &WR_LW1094},
{&WR_LW296, &WR_LW496, &WR_LW696, &WR_LW896, &WR_LW1096},
{&WR_LW298, &WR_LW498, &WR_LW698, &WR_LW898, &WR_LW1098} };
Big right? I had consurns that this HMI would have issues with such an approach but it did the job. The code below runs trough a string that comes from the csv file. This code runs inside another while cycle to cycle trough the multi dimensional array.
it's a little crude but works.
while (i<=5)
{
memset(lineTemp, 0, sizeof lineTemp); // clear lineTemp array
while (lineFromFile[index] != delimiter)
{
if (lineFromFile[index] != delimiter && lineFromFile[index] != '\0') { lineTemp[j] = lineFromFile[index]; index++; j++; }
if (lineFromFile[index] == '\0') { i = 5; break; }
}
index++;
lineTemp[j] = '\0'; // NULL TERMINATION
j = 0;
if (i == -1) { WriteLocal("LW",temp,3,lineTemp,0); }
if (i >= 0 && i<=5) { *(arr[x][i]) = atoi(lineTemp); }
i++;
}
Thanks again for the tip.
Cheers

Why does malloc provoke memory corruption here?

I keep getting the following error:
*** Error in `./vice': malloc(): memory corruption: 0x08e77530 ***
Aborted (core dumped)
The relevant code is:
open_result *
open_file_1_svc(open_args *argp, struct svc_req *rqstp)
{
static open_result result;
int obtained_fd;
int just_read;
int total_read = 0;
int max_bytes_read = 1024;
char *ptr_file;
char *pathName = "MyFiles/"; // strlen = 8
int toReserve;
xdr_free((xdrproc_t)xdr_open_result, (char *)&result);
// Construct full name of the file (in "MyFiles")
toReserve = strlen(argp->fname) + strlen(pathName) + 1; // "\0"
char *fullName = malloc(toReserve*sizeof(char));
fullName = strdup(pathName);
fullName = strcat(fullName, argp->fname);
// Call to open in POSIX
obtained_fd = open(fullName, argp->flags);
result.fd = obtained_fd;
/* If there was an error while reading, the error code will be sent, but not
the file (it might not even exist) */
if (obtained_fd < 0) {
result.characters = "";
result.number_characters = 0;
}
/* If the file opening was successful,
both the fd and the file will be sent */
else {
char *file_just_read = malloc(max_bytes_read * sizeof(char)); // This is the problem
ptr_file = file_just_read;
/* Reading the file byte by byte */
while((just_read = read(obtained_fd, ptr_file, max_bytes_read)) > 0) {
total_read += just_read;
file_just_read = realloc(file_just_read, (total_read+max_bytes_read) * sizeof(char));
ptr_file = file_just_read + total_read;
}
result.characters = file_just_read;
result.number_characters = total_read;
}
return &result;
}
Let me explain what the code does. This is a server named "vice" which communicates with its clients via RPC. This function is supposed to receive "open_args" and return "open_result". These are defined in the "vice.x" file. The relevant part of this file is:
struct open_args {
string fname<>;
int flags;
};
struct open_result {
string characters<>;
int number_characters;
int fd;
};
open_file_1_svc is supposed to try to open a file with the name given in argp->fname in the directory MyFiles. If open is successful, open_file_1_svc will attempt to copy the contents of the file in result.characters, sending a copy of the contents of the file to the client this way. The number_characters will allow me to know if there are any null bytes in between.
The error I'm getting appears when I attempt to allocate some memory for the part of the file I'm about to read.
I've been reading about this type of error, but I don't understand what's wrong with this particular case.
malloc does not "provoke" the corruption; malloc detects it.
This error is telling you that something had scribbled over the heap meta data before malloc was called (this time); you probably have a buffer overrun.
Both malloc calls in this code are before anything writes to memory, so the overrun is most likely elsewhere. (I've not done a detailed check that this code is right, but it's after-the-fact here.)
Edit: I missed the implicit malloc call inside the strdup. This will cause an overrun because the duplicated string has a smaller allocation. I think you mean strcpy, not strdup.

Splitting a comma-delimited string of integers

My background is not in C (it's in Real Studio - similar to VB) and I'm really struggling to split a comma-delimited string since I'm not used to low-level string handling.
I'm sending strings to an Arduino over serial. These strings are commands in a certain format. For instance:
#20,2000,5!
#10,423,0!
'#' is the header indicating a new command and '!' is the terminating footer marking the end of a command. The first integer after '#' is the command id and the remaining integers are data (the number of integers passed as data may be anywhere from 0 - 10 integers).
I've written a sketch that gets the command (stripped of the '#' and '!') and calls a function called handleCommand() when there is a command to handle. The problem is, I really don't know how to split this command up to handle it!
Here's the sketch code:
String command; // a string to hold the incoming command
boolean commandReceived = false; // whether the command has been received in full
void setup() {
// put your setup code here, to run once:
Serial.begin(9600);
}
void loop() {
// main loop
handleCommand();
}
void serialEvent(){
while (Serial.available()) {
// all we do is construct the incoming command to be handled in the main loop
// get the incoming byte from the serial stream
char incomingByte = (char)Serial.read();
if (incomingByte == '!')
{
// marks the end of a command
commandReceived = true;
return;
}
else if (incomingByte == '#')
{
// marks the start of a new command
command = "";
commandReceived = false;
return;
}
else
{
command += incomingByte;
return;
}
}
}
void handleCommand() {
if (!commandReceived) return; // no command to handle
// variables to hold the command id and the command data
int id;
int data[9];
// NOT SURE WHAT TO DO HERE!!
// flag that we've handled the command
commandReceived = false;
}
Say my PC sends the Arduino the string "#20,2000,5!". My sketch ends up with a String variable (called command) that contains "20,2000,5" and the commandRecieved boolean variable is set to True so the handleCommand() function is called.
What I would like to do in the (currently useless) handleCommand() function is assign 20 to a variable called id and 2000 and 5 to an array of integers called data, i.e: data[0] = 2000, data[1] = 5, etc.
I've read about strtok() and atoi() but frankly I just can't get my head around them and the concept of pointers. I'm sure my Arduino sketch could be optimised too.
Since you're using the Arduino core String type, strtok and other string.h functions aren't appropriate. Note that you can change your code to use standard C null-terminated strings instead, but using Arduino String will let you do this without using pointers.
The String type gives you indexOf and substring.
Assuming a String with the # and ! stripped off, finding your command and arguments would look something like this:
// given: String command
int data[MAX_ARGS];
int numArgs = 0;
int beginIdx = 0;
int idx = command.indexOf(",");
String arg;
char charBuffer[16];
while (idx != -1)
{
arg = command.substring(beginIdx, idx);
arg.toCharArray(charBuffer, 16);
// add error handling for atoi:
data[numArgs++] = atoi(charBuffer);
beginIdx = idx + 1;
idx = command.indexOf(",", beginIdx);
}
data[numArgs++] = command.substring(beginIdx);
This will give you your entire command in the data array, including the command number at data[0], while you've specified that only the args should be in data. But the necessary changes are minor.
seems to work, could be buggy:
#include<stdio.h>
#include <string.h>
int main(){
char string[]="20,2000,5";
int a,b,c;
sscanf(string,"%i,%i,%i",&a,&b,&c);
printf("%i %i %i\n",a,b,c);
a=b=c=0;
a=atoi(strtok(string,","));
b=atoi(strtok(0,","));
c=atoi(strtok(0,","));
printf("%i %i %i\n",a,b,c);
return 0;
}

fwrite() and file corruption

I'm trying to write a wchar array to a file in C, however there is some sort of corruption and unrelevant data like variables and paths like this
c.:.\.p.r.o.g.r.a.m. .f.i.l.e.s.\.m.i.c.r.o.s.o.f.t. .v.i.s.u.a.l. .s.t.u.d.i.o. 1.0...0.\.v.c.\.i.n.c.l.u.d.e.\.x.s.t.r.i.n.g..l.i.s.t...i.n.s.e.r.t
are written on to the file along with the correct data (example) I have confirmed that the buffer is null-terminated and contains proper data.
Heres my code:
myfile = fopen("logs.txt","ab+");
fseek(myfile,0,SEEK_END);
long int size = ftell(myfile);
fseek(myfile,0,SEEK_SET);
if (size == 0)
{
wchar_t bom_mark = 0xFFFE;
size_t written = fwrite(&bom_mark,sizeof(wchar_t),1,myfile);
}
// in another func
while (true)
{
[..]
unsigned char Temp[512];
iBytesRcvd = recv(sclient_socket,(char*)&Temp,iSize,NULL);
if(iBytesRcvd > 0 )
{
WCHAR* unicode_recv = (WCHAR*)&Temp;
fwrite(unicode_recv,sizeof(WCHAR),wcslen(unicode_recv),myfile);
fflush(myfile);
}
[..]
}
What could be causing this?
recv() will not null-terminate &Temp, so wcslen() runs over the bytes actually written by recv(). You will get correct results if you just use iBytesReceived as byte count for fwrite() instead of using wcslen() and hoping the data received is correctly null-terminated (wide-NULL-terminated, that is):
fwrite(unicode_recv, 1, iBytesReceived, myfile);

Resources