Related
I am working on a project in C to implement CBC mode on top of a skeleton code for DES with OpenSSL. We are not allowed to use a function that does the CBC mode automatically, in the sense that we must implement it ourselves. I am getting output but I have result files and my output is not matching up completely with the intended results. I also am stuck on figuring out how to pad the file to ensure all the blocks are of equal size, which is probably one of the reasons why I'm not receiving the correct output. Any help would be appreciated. Here's my modification of the skeleton code so far:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <openssl/des.h>
#include <sys/time.h>
#include <unistd.h>
#define ENC 1
#define DEC 0
DES_key_schedule key;
int append(char*s, size_t size, char c) {
if(strlen(s) + 1 >= size) {
return 1;
}
int len = strlen(s);
s[len] = c;
s[len+1] = '\0';
return 0;
}
int getSize (char * s) {
char * t;
for (t = s; *t != '\0'; t++)
;
return t - s;
}
void strToHex(const_DES_cblock input, unsigned char *output) {
int arSize = 8;
unsigned int byte;
for(int i=0; i<arSize; i++) {
if(sscanf(input, "%2x", &byte) != 1) {
break;
}
output[i] = byte;
input += 2;
}
}
void doBitwiseXor(DES_LONG *xorValue, DES_LONG* data, const_DES_cblock roundOutput) {
DES_LONG temp[2];
memcpy(temp, roundOutput, 8*sizeof(unsigned char));
for(int i=0; i<2; i++) {
xorValue[i] = temp[i] ^ data[i];
}
}
void doCBCenc(DES_LONG *data, const_DES_cblock roundOutput, FILE *outFile) {
DES_LONG in[2];
doBitwiseXor(in, data, roundOutput);
DES_encrypt1(in,&key,ENC);
printf("ENCRYPTED\n");
printvalueOfDES_LONG(in);
printf("%s","\n");
fwrite(in, 8, 1, outFile);
memcpy(roundOutput, in, 2*sizeof(DES_LONG));
}
int main(int argc, char** argv)
{
const_DES_cblock cbc_key = {0x01,0x23,0x45,0x67,0x89,0xab,0xcd,0xef};
const_DES_cblock IV = {0x01,0x23,0x45,0x67,0x89,0xab,0xcd,0xef};
// Initialize the timing function
struct timeval start, end;
gettimeofday(&start, NULL);
int l;
if ((l = DES_set_key_checked(&cbc_key,&key)) != 0)
printf("\nkey error\n");
FILE *inpFile;
FILE *outFile;
inpFile = fopen("test.txt", "r");
outFile = fopen("test_results.txt", "wb");
if(inpFile && outFile) {
unsigned char ch;
// A char array that will hold all 8 ch values.
// each ch value is appended to this.
unsigned char eight_bits[8];
// counter for the loop that ensures that only 8 chars are done at a time.
int count = 0;
while(!feof(inpFile)) {
// read in a character
ch = fgetc(inpFile);
// print the character
printf("%c",ch);
// append the character to eight_bits
append(eight_bits,1,ch);
// increment the count so that we only go to 8.
count++;
const_DES_cblock roundOutput;
// When count gets to 8
if(count == 8) {
// for formatting
printf("%s","\n");
// Encrypt the eight characters and store them back in the char array.
//DES_encrypt1(eight_bits,&key,ENC);
doCBCenc(eight_bits, roundOutput, outFile);
// prints out the encrypted string
int k;
for(k = 0; k < getSize(eight_bits); k++){
printf("%c", eight_bits[k]);
}
// Sets count back to 0 so that we can do another 8 characters.
count = 0;
// so we just do the first 8. When everything works REMOVE THE BREAK.
//break;
}
}
} else {
printf("Error in opening file\n");
}
fclose(inpFile);
fclose(outFile);
// End the timing
gettimeofday(&end, NULL);
// Initialize seconds and micros to hold values for the time output
long seconds = (end.tv_sec - start.tv_sec);
long micros = ((seconds * 1000000) + end.tv_usec) - (start.tv_usec);
// Output the time
printf("The elapsed time is %d seconds and %d microseconds\n", seconds, micros);
}
Your crypto is at least half correct, but you have a lot of actual or potential other errors.
As you identified, raw CBC mode can only encrypt data which is a multiple of the block size, for DES 64 bits or 8 bytes (on most modern computers and all where you could use OpenSSL). In some applications this is okay; for example if the data is (always) an MD5 or SHA-256 or SHA-512 hash, or a GUID, or an IPv6 (binary) address, then it is a block multiple. But most applications want to handle at least any length in bytes, so they need to use some scheme to pad on encrypt and unpad on decrypt the last block (all blocks before the last already have the correct size). Many different schemes have been developed for this, so you need to know which to use. I assume this is a school assignment (since no real customer would set such a stupid and wasteful combination of requirements) and this should either have been specified or clearly left as a choice. One padding scheme very common today (although not for single-DES, because that is broken, unsafe, obsolete, and not common) is the one defined by PKCS5 and generalized by PKCS7 and variously called PKCS5, PKCS7, or PKCS5/7 padding, so I used that as an example.
Other than that:
you try to test feof(inpFile) before doing fgetc(inpFile). This doesn't work in C. It results in your code treating the low 8 bits of EOF (255 aka 0xFF on practically all implementations) as a valid data character added to the characters that were actually in the file. The common idiom is to store the return of getchar/getc/fgetc in a signed int and compare to EOF, but that would have required more changes so I used an alternate.
you don't initialize eight_bits which is a local-scope automatic duration variable, so its contents are undefined and depending on the implementation are often garbage, which means trying to 'append' to it by using strlen() to look for the end won't work right and might even crash. Although on some implementations at least some times it might happen to contain zero bytes, and 'work'. In addition it is possible in C for a byte read from a file (and stored here) to be \0 which will also make this work wrong, although if this file contains text, as its name suggests, it probably doesn't contain any \0 bytes.
once you fill eight_bits you write 'off-the-end' into element [8] which doesn't exist. Technically this is Undefined Behavior and anything at all can happen, traditionally expressed on Usenet as nasal demons. Plus after main finishes the first block it doesn't change anything in eight_bits so all further calls to append find it full and discard the new character.
while you could fix the above points separately, a much simple solution is available: you are already using count to count the number of bytes in the current block, so just use it as the subscript.
roundOutput is also an uninitialized local/auto variable within the loop, which is then used as the previous block for the CBC step, possibly with garbage or wrong value(s). And you don't use the IV at all, as is needed. You should allocate this before the loop (so it retains its value through all iterations) and initialize it to the IV, and then for each block in the loop your doCBCenc can properly XOR it to the new block and then leave the encrypted new block to be used next time.
your code labelled 'prints out the encrypted string' prints plaintext not ciphertext -- which is binary and shouldn't be printed directly anyway -- and is not needed because your file-read loop already echoes each character read. But if you do want to print a (validly null-terminated) string it's easier to just use fputs(s) or [f]printf([f,]"%s",s) or even fwrite(s,1,strlen(s),f).
your doCBCenc has a reference to printvalueofDES_LONG which isn't defined anywhere, and which along with two surrounding printf is clearly not needed.
you should use a cast to convert the first argument to doCBCenc -- this isn't strictly required but is good style and a good compiler (like mine) complains if you don't
finally, when an error occurs you usually print a message but then continue running, which will never work right and may produce symptoms that disguise the problem and make it hard to fix.
The below code fixes the above except that last (which would have been more work for less benefit) plus I removed routines that are now superfluous, and the timing code which is just silly: Unix already has builtin tools to measure and display process time more easily and reliably than writing code. Code I 'removed' is under #if 0 for reference, and code I added under #else or #if 1 except for the cast. The logic for PKCS5/7 padding is under #if MAYBE so it can be either selected or not. Some consider it better style to use sizeof(DES_block) or define a macro instead of the magic 8's, but I didn't bother -- especially since it would have required changes that aren't really necessary.
// SO70209636
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <openssl/des.h>
#include <sys/time.h>
#include <unistd.h>
#define ENC 1
#define DEC 0
DES_key_schedule key;
#if 0
int append(char*s, size_t size, char c) {
if(strlen(s) + 1 >= size) {
return 1;
}
int len = strlen(s);
s[len] = c;
s[len+1] = '\0';
return 0;
}
int getSize (char * s) {
char * t;
for (t = s; *t != '\0'; t++)
;
return t - s;
}
void strToHex(const_DES_cblock input, unsigned char *output) {
int arSize = 8;
unsigned int byte;
for(int i=0; i<arSize; i++) {
if(sscanf(input, "%2x", &byte) != 1) {
break;
}
output[i] = byte;
input += 2;
}
}
#endif
void doBitwiseXor(DES_LONG *xorValue, DES_LONG* data, const_DES_cblock roundOutput) {
DES_LONG temp[2];
memcpy(temp, roundOutput, 8*sizeof(unsigned char));
for(int i=0; i<2; i++) {
xorValue[i] = temp[i] ^ data[i];
}
}
void doCBCenc(DES_LONG *data, const_DES_cblock roundOutput, FILE *outFile) {
DES_LONG in[2];
doBitwiseXor(in, data, roundOutput);
DES_encrypt1(in,&key,ENC);
#if 0
printf("ENCRYPTED\n");
printvalueOfDES_LONG(in);
printf("%s","\n");
#endif
fwrite(in, 8, 1, outFile);
memcpy(roundOutput, in, 2*sizeof(DES_LONG));
}
int main(int argc, char** argv)
{
const_DES_cblock cbc_key = {0x01,0x23,0x45,0x67,0x89,0xab,0xcd,0xef};
const_DES_cblock IV = {0x01,0x23,0x45,0x67,0x89,0xab,0xcd,0xef};
#if 0
// Initialize the timing function
struct timeval start, end;
gettimeofday(&start, NULL);
#endif
int l;
if ((l = DES_set_key_checked(&cbc_key,&key)) != 0)
printf("\nkey error\n");
#if 1
DES_cblock roundOutput; // must be outside the loop
memcpy (roundOutput, IV, 8); // and initialized
#endif
FILE *inpFile;
FILE *outFile;
inpFile = fopen("test.txt", "r");
outFile = fopen("test.encrypt", "wb");
if(inpFile && outFile) {
unsigned char ch;
// A char array that will hold all 8 ch values.
// each ch value is appended to this.
unsigned char eight_bits[8];
// counter for the loop that ensures that only 8 chars are done at a time.
int count = 0;
#if 0
while(!feof(inpFile)) {
// read in a character
ch = fgetc(inpFile);
#else
while( ch = fgetc(inpFile), !feof(inpFile) ){
#endif
// print the character
printf("%c",ch);
#if 0
// append the character to eight_bits
append(eight_bits,1,ch);
// increment the count so that we only go to 8.
count++;
#else
eight_bits[count++] = ch;
#endif
#if 0
const_DES_cblock roundOutput;
#endif
// When count gets to 8
if(count == 8) {
// for formatting
printf("%s","\n");
// Encrypt the eight characters and store them back in the char array.
//DES_encrypt1(eight_bits,&key,ENC);
doCBCenc((DES_LONG*)eight_bits, roundOutput, outFile);
#if 0
// prints out the encrypted string
int k;
for(k = 0; k < getSize(eight_bits); k++){
printf("%c", eight_bits[k]);
}
#endif
// Sets count back to 0 so that we can do another 8 characters.
count = 0;
// so we just do the first 8. When everything works REMOVE THE BREAK.
//break;
}
}
#if MAYBE
memset (eight_bits+count, 8-count, 8-count); // PKCS5/7 padding
doCBCenc((DES_LONG*)eight_bits, roundOutput, outFile);
#endif
} else {
printf("Error in opening file\n");
}
fclose(inpFile);
fclose(outFile);
#if 0
// End the timing
gettimeofday(&end, NULL);
// Initialize seconds and micros to hold values for the time output
long seconds = (end.tv_sec - start.tv_sec);
long micros = ((seconds * 1000000) + end.tv_usec) - (start.tv_usec);
// Output the time
printf("The elapsed time is %d seconds and %d microseconds\n", seconds, micros);
#endif
}
PS: personally I wouldn't put the fwrite in doCBCenc; I would only do the encryption and let the caller do whatever I/O is appropriate which might in some cases not be fwrite. But what you have is not wrong for the requirements you apparently have.
I am using PortAudio to implement a real-time audio processing.
My primary task is to acquire data from mic continuously and provide 100 samples for processing (each FRAME = 100 samples at a time) to some other processing thread.
Here is my callback collecting 100 samples each time on a continuous basis -
static int paStreamCallback( const void* input, void* output,
unsigned long samplesPerFrame,
const PaStreamCallbackTimeInfo* timeInfo,
PaStreamCallbackFlags statusFlags,
void* userData ) {
paTestData *data = (paTestData*) userData;
const SAMPLE *readPtr = (const SAMPLE*)input; // Casting input read to valid input type SAMPLE (float)
unsigned long totalSamples = TOTAL_NUM_OF_SAMPLES ; // totalSamples = 100 here
(void) output;
(void) timeInfo;
(void) statusFlags;
static int count = 1;
if(data->sampleIndex < count * samplesPerFrame){
data->recordedSample[data->sampleIndex] = *readPtr;
data->sampleIndex++;
}
else if(data->sampleIndex == count * samplesPerFrame){
processSampleFrame(data->recordedSample);
count++;
finished = paContinue;
}
return finished;
}
My `main function -
int main(void){
// Some Code here
data.sampleIndex = 0;
data.frameIndex = 1;
numBytes = TOTAL_NUM_OF_SAMPLES * sizeof(SAMPLE);
data.recordedSample = (SAMPLE*)malloc(numBytes);
for(i=0; i < TOTAL_NUM_OF_SAMPLES; i++)
data.recordedSample[i] = 0;
// Open Stream Initialization
err = Pa_StartStream(stream);
/* Recording audio here..Speak into the MIC */
printf("\nRecording...\n");
fflush(stdout);
while((err = Pa_IsStreamActive(stream)) == 1)
Pa_Sleep(10);
if(err < 0)
goto done;
err = Pa_CloseStream(stream);
// Some more code here
}
Sending each Frame of 100 samples to processSampleFrame.
void processSampleFrame(SAMPLE *singleFrame){
// Free buffer of this frame
// Processing sample frame here
}
The challenge is that I need to implement a way in which the time processSampleFrame is processing the samples, my callBack should be active and keep acquiring the next Frame of 100 samples and (may be more depending upon the processing time of processSampleFrame).
Also the buffer should able to free itself of the frame so sooner it has passed it to processSampleFrame.
EDIT 2 :
I tried implementing with PaUtilRingBuffer that PortAudio provides. Here is my callback.
printf("Inside callback\n");
paTestData *data = (paTestData*) userData;
ring_buffer_size_t elementsWritable = PaUtil_GetRingBufferWriteAvailable(&data->ringBuffer);
ring_buffer_size_t elementsToWrite = rbs_min(elementsWritable, (ring_buffer_size_t)(samplesPerFrame * numChannels));
const SAMPLE *readPtr = (const SAMPLE*)input;
printf("Sample ReadPtr = %.8f\n", *readPtr);
(void) output; // Preventing unused variable warnings
(void) timeInfo;
(void) statusFlags;
data->frameIndex += PaUtil_WriteRingBuffer(&data->ringBuffer, readPtr, elementsToWrite);
return paContinue;
And main :
int main(void){
paTestData data; // Object of paTestData structure
fflush(stdout);
data.frameIndex = 1;
long numBytes = TOTAL_NUM_OF_SAMPLES * LIMIT;
data.ringBufferData = (SAMPLE*)PaUtil_AllocateMemory(numBytes);
if(PaUtil_InitializeRingBuffer(&data.ringBuffer, sizeof(SAMPLE), ELEMENTS_TO_WRITE, data.ringBufferData) < 0){
printf("Failed to initialise Ring Buffer.\n");
goto done;
err = Pa_StartStream(stream);
/* Recording audio here..Speak into the MIC */
printf("\nRecording samples\n");
fflush(stdout);
while((err = Pa_IsStreamActive(stream)) == 1)
Pa_Sleep(10);
if(err < 0)
goto done;
err = Pa_CloseStream(stream);
// Error Handling here
}
PaTestData Structure :
typedef struct{
SAMPLE *ringBufferData;
PaUtilRingBuffer ringBuffer;
unsigned int frameIndex;
}
paTestData;
I am facing the same issue of seg-fault after successful acquisition for the allocated space because of not being able to use any free in the callback as suggested by PortAudio documentation.
Where can I free the buffer of the allocated frame given to the processing thread. May be a method of obtaining a thread-synchronization can be really useful here. Your help is appreciated.
Example code of processing audio input:
#define FRAME_SIZE 1024
#define CIRCULAR_BUFFER_SIZE (FRAME_SIZE * 4)
float buffer[CIRCULAR_BUFFER_SIZE];
typedef struct {
int read, write;
float vol;
} paData;
static int paStreamCallback(const void* input, void* output,
unsigned long samplesPerFrame,
const PaStreamCallbackTimeInfo* timeInfo,
PaStreamCallbackFlags statusFlags,
void* userData) {
paData *data = (paData *)userData;
// Write input buffer to our buffer: (memcpy function is fine, just a for loop of writing data
memcpy(&buffer[write], input, samplesPerFrame); // Assuming samplesPerFrame = FRAME_SIZE
// Increment write/wraparound
write = (write + FRAME_SIZE) % CIRCULAR_BUFFER_SIZE;
// dummy algorithm for processing blocks:
processAlgorithm(&buffer[read]);
// Write output
float *dummy_buffer = &buffer[read]; // just for easy increment
for (int i = 0; i < samplesPerFrame; i++) {
// Mix audio input and processed input; 50% mix
output[i] = input[i] * 0.5 + dummy_buffer[i] * 0.5;
}
// Increment read/wraparound
read = (read + FRAME_SIZE) % CIRCULAR_BUFFER_SIZE;
return paContinue;
}
int main(void) {
// Initialize code here; any memory allocation needs to be done here! default buffers to 0 (silence)
// initialize paData too; write = 0, read = 3072
// read is 3072 because I'm thinking about this like a delay system.
}
Take this with a grain of salt; Obviously better ways of doing this but it can be used as a starting point.
I am trying to get the flite speech synthesis library to work on my Mac, but my sound architecture isn't supported within the flite library. To fix that problem, I am using PortAudio to playback the synthesized audio; so I had to do a little bit of hacking within the audio.c file to get flite to use that library. I managed to get everything compiling just fine after mucking around in the GNU AutoTools for a while, but then I run the program and get this output:
$ ./flite -t "test"
frameIndex: 0
maxFrameIndex: 0
numChannels: 1
numSamples: 7225
sampleRate: 8000
=== Now playing back. ===
Waiting for playback to finish.
frameIndex in callback: -2008986336
maxFrameIndex in callback: 32655
numChannels in callback: 152579008
numSamples in callback: 0
sampleRate in callback: 0
Segmentation fault: 11
$ ./flite -t "test"
frameIndex: 0
maxFrameIndex: 0
numChannels: 1
numSamples: 7225
sampleRate: 8000
=== Now playing back. ===
Waiting for playback to finish.
frameIndex in callback: -71217888
maxFrameIndex in callback: 32712
numChannels in callback: 232979392
numSamples in callback: 0
sampleRate in callback: 0
Segmentation fault: 11
Here is the relevant code from the audio.c file, which is called when I supply the command line argument -t. I marked the area of interest where the segmentation fault occurs in the playCallback() function after a bit of debugging.
static int playCallback( const void *inputBuffer, void *outputBuffer,
unsigned long framesPerBuffer,
const PaStreamCallbackTimeInfo* timeInfo,
PaStreamCallbackFlags statusFlags,
void *userData )
{
cst_wave *data = (cst_wave*)userData;
short *rptr = &data->samples[data->frameIndex * data->num_channels];
short *wptr = (short*)outputBuffer;
unsigned int i;
int finished;
unsigned int framesLeft = cst_wave_maxFrameIndex(data) - cst_wave_frameIndex(data);
(void) inputBuffer; /* Prevent unused variable warnings. */
(void) timeInfo;
(void) statusFlags;
(void) userData;
printf("frameIndex in callback: %d\n", cst_wave_frameIndex(data));
printf("maxFrameIndex in callback: %d\n", cst_wave_maxFrameIndex(data));
printf("numChannels in callback: %d\n", cst_wave_num_channels(data));
printf("numSamples in callback: %d\n", cst_wave_num_samples(data));
printf("sampleRate in callback: %d\n\n", cst_wave_sample_rate(data));
if( framesLeft < framesPerBuffer )
{
/* final buffer... */
for( i=0; i<framesLeft; i++ )
{
*wptr++ = *rptr++; /* left */
if( cst_wave_num_channels(data) == 2 ) *wptr++ = *rptr++; /* right */
}
for( ; i<framesPerBuffer; i++ )
{
*wptr++ = 0; /* left */
if( cst_wave_num_channels(data) == 2) *wptr++ = 0; /* right */
}
data->frameIndex += framesLeft;
finished = paComplete;
}
else
{
for( i=0; i<framesPerBuffer; i++ )
{
*wptr++ = *rptr++; /* left */
if( cst_wave_num_channels(data) == 2 ) *wptr++ = *rptr++; /* right */
}
cst_wave_set_frameIndex(data, framesPerBuffer);
finished = paContinue;
}
return finished;
}
int play_wave(cst_wave *w)
{
PaStream* stream;
PaStreamParameters outputParameters;
cst_wave_set_frameIndex(w, 0);
cst_wave_set_maxFrameIndex(w, (cst_wave_num_samples(w) / cst_wave_sample_rate(w)) * cst_wave_num_channels(w) * sizeof(short));
int err = 0;
err = Pa_Initialize();
outputParameters.device = Pa_GetDefaultOutputDevice();
if (outputParameters.device == paNoDevice)
{
fprintf(stderr,"Error: No default output device.\n");
return -5;
}
printf("frameIndex: %d\n", cst_wave_frameIndex(w));
printf("maxFrameIndex: %d\n", cst_wave_maxFrameIndex(w));
printf("numChannels: %d\n", cst_wave_num_channels(w));
printf("numSamples: %d\n", cst_wave_num_samples(w));
printf("sampleRate: %d\n", cst_wave_sample_rate(w));
outputParameters.channelCount = cst_wave_num_channels(w);
outputParameters.sampleFormat = paInt16;
outputParameters.suggestedLatency = Pa_GetDeviceInfo( outputParameters.device )->defaultLowOutputLatency;
outputParameters.hostApiSpecificStreamInfo = NULL;
puts("=== Now playing back. ===");
err = Pa_OpenStream(&stream,
NULL, /* no input */
&outputParameters,
cst_wave_sample_rate(w),
512,
paClipOff,
playCallback,
&w);
if( stream )
{
err = Pa_StartStream( stream );
if( err != paNoError ) goto done;
puts("Waiting for playback to finish.");
while((err = Pa_IsStreamActive(stream)) == 1) Pa_Sleep(100);
if( err < 0 ) goto done;
err = Pa_CloseStream( stream );
if( err != paNoError ) goto done;
puts("Done.");
}
done:
Pa_Terminate();
free(cst_wave_samples(w));
}
Because it is relevant, I also slightly modified the cst_wave structure in cst_wave.h so that it contains my data I have to store, as well as adding a few #defines to the ones that were already present:
typedef struct cst_wave_struct {
const char *type;
int frameIndex;
int maxFrameIndex;
int sample_rate;
int num_samples;
int num_channels;
short *samples;
} cst_wave;
#define cst_wave_num_samples(w) (w?w->num_samples:0)
#define cst_wave_num_channels(w) (w?w->num_channels:0)
#define cst_wave_sample_rate(w) (w?w->sample_rate:0)
#define cst_wave_samples(w) (w->samples)
#define cst_wave_frameIndex(w) (w->frameIndex)
#define cst_wave_maxFrameIndex(w) (w->maxFrameIndex)
#define cst_wave_set_num_samples(w,s) w->num_samples=s
#define cst_wave_set_num_channels(w,s) w->num_channels=s
#define cst_wave_set_sample_rate(w,s) w->sample_rate=s
#define cst_wave_set_frameIndex(w,s) w->frameIndex=s
#define cst_wave_set_maxFrameIndex(w,s) w->maxFrameIndex=s
Update 1:
Following the advice of #Rohan now gives me this output:
$ ./bin/flite -t "test"
frameIndex: 0
maxFrameIndex: 0
numChannels: 1
numSamples: 7225
sampleRate: 8000
=== Now playing back. ===
Waiting for playback to finish.
frameIndex in callback: 0
maxFrameIndex in callback: 0
numChannels in callback: 1
numSamples in callback: 7225
sampleRate in callback: 8000
Done.
flite(68929,0x7fff71c0d310) malloc: *** error for object 0x7fd6e2809800: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6
To fix that, I removed the free(cst_wave_samples(w));. Now the program executes normally with no visible errors, but there is still no audio output on my Mac. Any suggestions?
It looks to me like the problem is probably elsewhere.
The routine where you've added the comment is really pretty trivial when all is said and done. It's basically just copying a buffer full of data from one place to another, and if the data doesn't fill the input buffer, zero-filling the remainder. If I were writing the code, I'd probably do something more along these general lines:
const unsigned frame_size = sizeof(short) * data->num_channels;
char *source = &data->samples[data->frameIndex * data->num_channels];
char *dest = outputBuffer;
unsigned framesLeft = data->maxFrameIndex - data->frameIndex;
unsigned framesEmpty = framesPerBuffer - framesLeft;
memcpy(source, dest, framesLeft * frame_size);
memset(dest+framesLeft * frame_size, 0, framesEmpty * frame_size);
data->frameIndex += framesPerBuffer;
Although rather clumsily written, the if/else in the question just skips doing the memset part at all if the size that needs to be filled is zero.
So, this copies a buffer full of data from one place to another, and zero-fills any remainder. If you're getting a segfault, whatever's allocating the destination buffer apparently hasn't allocated enough space there. Without doing some looking, it's impossible to guess whether the allocation happens in Pa_Initialize, Pa_OpenStream, Pa_StartStream, or possibly somewhere else--and most likely you care less about the code that actually does the allocation than the code that computes how much space to allocate (which might be in one of the above, or somewhere else completely).
In your play_wave function you are calling:
err = Pa_OpenStream(&stream,
NULL, /* no input */
&outputParameters,
cst_wave_sample_rate(w),
512,
paClipOff,
playCallback,
&w);
Here as last parameter you are passing &w, so you are passing cst_wave ** as w is defined as cst_wave *w.
But in playCallback() you are using it as
cst_wave *data = (cst_wave*)userData;
So in this function you are incorrectly accessing cst_wave ** as cst_wave *. So at some point you will access the invalid memory while using some member of w.
Also, this is the reason you are getting the incorrect output for other parameters e.g. frameIndex, maxFrameIndex etc. as your output shows.
Solution is just pass w to Pa_OpenStream() function rather than &w.
Your next problem is that you aren't setting your maxFrameIndex correctly. As you stated in the comments, this shouldn't be 0. In order to set it properly, you should have something like this:
cst_wave_set_maxFrameIndex(w, cst_wave_num_samples(w) * cst_wave_num_channels(w));
Lastly, it looks like your callback could be messing things up a bit. Here is a better and more efficient way to write it:
static int playCallback( const void *inputBuffer, void *outputBuffer,
unsigned long framesPerBuffer,
const PaStreamCallbackTimeInfo* timeInfo,
PaStreamCallbackFlags statusFlags,
void *userData )
{
cst_wave *data = (cst_wave*)userData;
short *rptr = &data->samples[data->frameIndex * data->num_channels];
short *wptr = (short*)outputBuffer;
int finished;
unsigned int framesLeft = data->maxFrameIndex - data->frameIndex;
(void) inputBuffer; /* Prevent unused variable warnings. */
(void) timeInfo;
(void) statusFlags;
(void) userData;
if( framesLeft < framesPerBuffer )
{
/* final buffer... */
memcpy(wptr, rptr, sizeof(*wptr) * data->num_channels * framesLeft);
memset(wptr, sizeof(*wptr) * data->num_channels * framesPerBuffer, 0);
data->frameIndex += framesLeft;
finished = paComplete;
}
else
{
memcpy(wptr, rptr, sizeof(*wptr) * data->num_channels * framesPerBuffer);
data->frameIndex += framesPerBuffer;
finished = paContinue;
}
return finished;
}
You're in luck. I was able to compile both PortAudio and flite on my own Mac, and solve your problem.
You have several issues other than those mentioned before, all of which I have addressed in the code dump below.
Minor: You don't use consistently your own API for cst_wave.
Minor: I prefer to enclose my while and if blocks with {} always. This has a habit of preventing mysterious bugs.
Max Frames was being set to zero. That's because in (cst_wave_num_samples(w) / cst_wave_sample_rate(w)) * cst_wave_num_channels(w) * sizeof(short), you were dividing by the sample rate, which was greater than your number of samples. Given that integer division is left-associative and truncating, yadda yadda yadda zero.
Max Frames is still wrong, as a frame includes all channel samples. The number of frames is thus agnostic to both number of channels and the size of the samples themselves. Allowing myself to guess that flite misuses sample to mean frame, your max frame index is just cst_wave_num_samples(w). Else it will be cst_wave_num_samples(w) / cst_wave_num_channels(w).
PortAudio's documentation states you should call Pa_StopStream(stream) after the stream becomes inactive, whether or not you were waiting until it became so.
I simplified the callback, and corrected it for
Minor: Consistent use of your API
MAJOR: Ehm... cst_wave_set_frameIndex(data, framesPerBuffer); is definitely wrong. You're pinning yourself at frame index 512 instead of incrementing! That's because you asked for 512 frames per buffer when opening the stream and you're not incrementing the frame index by framesPerBuffer, you're setting the frame index to framesPerBuffer. You hadn't made it that far because your maxFrameIndex was 0 anyways so you were exiting. I fixed it so that the frame index increments - with your API of course.
Here is the code, which I took the freedom of documenting and cleaning until it approached my standards of elegance. Enjoy!
#include <stdio.h>
#include <string.h>
/**
* Audio play callback.
*
* Follows the PaStreamCallback signature, wherein:
*
* #param input and
* #param output are either arrays of interleaved samples or; if
* non-interleaved samples were requested using the
* paNonInterleaved sample format flag, an array of buffer
* pointers, one non-interleaved buffer for each channel.
* #param frameCount The number of sample frames to be processed by the
* stream callback.
* #param timeInfo Timestamps indicating the ADC capture time of the first
* sample in the input buffer, the DAC output time of the
* first sample in the output buffer and the time the
* callback was invoked. See PaStreamCallbackTimeInfo and
* Pa_GetStreamTime()
* #param statusFlags Flags indicating whether input and/or output buffers
* have been inserted or will be dropped to overcome
* underflow or overflow conditions.
* #param userData The value of a user supplied pointer passed to
* Pa_OpenStream() intended for storing synthesis data
* etc.
*/
static int playCallback(const void* inputBuffer,
void* outputBuffer,
unsigned long framesPerBuffer,
const PaStreamCallbackTimeInfo* timeInfo,
PaStreamCallbackFlags statusFlags,
void* userData){
(void) inputBuffer; /* Prevent unused variable warnings. */
(void) timeInfo;
(void) statusFlags;
(void) userData;
/**
* Compute current processing state.
*/
cst_wave* data;
short* rptr;
short* wptr;
unsigned int framesLeft, /* Number of frames of data remaining within the stream ***as a whole*** */
frames, /* Number of frames of data to be written for this buffer. */
framesPad, /* Number of frames of padding required within the final buffer. */
samples, /* Number of samples of data to be written for this buffer. */
samplesPad, /* Number of samples of padding required within the final buffer. */
numBytes, /* Number of bytes of data to be written for this buffer. */
numBytesPad;/* Number of bytes of padding required within the final buffer. */
int finalBuffer;/* Stores whether or not this is the final buffer. */
data = (cst_wave*)userData;
rptr = &data->samples[cst_wave_frameIndex (data) *
cst_wave_num_channels(data)];
wptr = (short*)outputBuffer;
framesLeft = cst_wave_maxFrameIndex(data) - cst_wave_frameIndex(data);
finalBuffer = framesLeft <= framesPerBuffer;
frames = finalBuffer ? framesLeft : framesPerBuffer;
framesPad = framesPerBuffer - frames;
samples = frames * cst_wave_num_channels(data);
samplesPad = framesPad * cst_wave_num_channels(data);
numBytes = samples * sizeof(short);
numBytesPad = samplesPad * sizeof(short);
/**
* Debug code. Comment out in production.
*/
printf("framesLeft in callback: %u\n", framesLeft);
printf("framesPerBuffer in callback: %lu\n", framesPerBuffer);
printf("frames in callback: %u\n", frames);
printf("frameIndex in callback: %d\n", cst_wave_frameIndex(data));
printf("maxFrameIndex in callback: %d\n", cst_wave_maxFrameIndex(data));
printf("numChannels in callback: %d\n", cst_wave_num_channels(data));
printf("numSamples in callback: %d\n", cst_wave_num_samples(data));
printf("sampleRate in callback: %d\n\n", cst_wave_sample_rate(data));
/**
* Output data. We handle the final buffer specially, padding it with zeros.
*/
memcpy(wptr, rptr, numBytes);
wptr += samples;
rptr += samples;
cst_wave_set_frameIndex(data, cst_wave_frameIndex(data) + frames);
memset(wptr, 0, numBytesPad);
wptr += samplesPad;
rptr += samplesPad;
/**
* Return a completion or continue code depending on whether this was the
* final buffer or not respectively.
*/
return finalBuffer ? paComplete : paContinue;
}
/**
* Play wave function.
*
* Plays the given cst_wave data as audio, blocking until this is done.
*/
int play_wave(cst_wave *w){
PaStream* stream;
PaStreamParameters outputParameters;
int err;
/**
* Initialize custom fields in cst_wave struct.
*/
cst_wave_set_frameIndex(w, 0);
cst_wave_set_maxFrameIndex(w, (cst_wave_num_samples(w)));
// / cst_wave_sample_rate(w) * cst_wave_num_channels(w) * sizeof(short)
/**
* Initialize Port Audio device and stream parameters.
*/
err = Pa_Initialize();
outputParameters.device = Pa_GetDefaultOutputDevice();
if (outputParameters.device == paNoDevice){
fprintf(stderr,"Error: No default output device.\n");
return -5;
}
printf("frameIndex: %d\n", cst_wave_frameIndex(w));
printf("maxFrameIndex: %d\n", cst_wave_maxFrameIndex(w));
printf("numChannels: %d\n", cst_wave_num_channels(w));
printf("numSamples: %d\n", cst_wave_num_samples(w));
printf("sampleRate: %d\n", cst_wave_sample_rate(w));
outputParameters.channelCount = cst_wave_num_channels(w);
outputParameters.sampleFormat = paInt16;
outputParameters.suggestedLatency = Pa_GetDeviceInfo( outputParameters.device )->defaultLowOutputLatency;
outputParameters.hostApiSpecificStreamInfo = NULL;
/**
* Open the stream for playback.
*/
puts("=== Now playing back. ===");
err = Pa_OpenStream(&stream,
NULL, /* no input */
&outputParameters,
cst_wave_sample_rate(w),
512,
paClipOff,
playCallback,
w);
if(stream){
/**
* Start the stream.
*/
err = Pa_StartStream(stream);
if(err != paNoError){
goto done;
}
/**
* Block while it plays.
*/
puts("Waiting for playback to finish.");
while((err = Pa_IsStreamActive(stream)) == 1){
Pa_Sleep(100);
}
if(err < 0){
goto done;
}
/**
* Stop and close the stream. Both are necessary.
*/
Pa_StopStream(stream);
err = Pa_CloseStream(stream);
if(err != paNoError){
goto done;
}
puts("Done.");
}
/**
* Terminate and leave.
*/
done:
Pa_Terminate();
return 0;
}
I'm trying to read in multiple files using MPI-IO in C. I'm following this example : http://users.abo.fi/Mats.Aspnas/PP2010/examples/MPI/readfile1.c
However I'm reading in a matrix a doubles instead of a string of chars. Here is that implementation:
/*
Simple MPI-IO program that demonstrate parallel reading from a file.
Compile the program with 'mpicc -O2 readfile1.c -o readfile1'
*/
#include <stdlib.h>
#include <stdio.h>
#include "mpi.h"
#define FILENAME "filename.dat"
double** ArrayAllocation() {
int i;
double** array2D;
array2D= (double**) malloc(num_procs*sizeof(double*));
for(i = 0; i < num_procs; i++) {
twoDarray[i] = (double*) malloc(column_size*sizeof(double));
}
return array2D;
}
int main(int argc, char* argv[]) {
int i, np, myid;
int bufsize, nrchar;
double *buf; /* Buffer for reading */
double **matrix = ArrayAllocation();
MPI_Offset filesize;
MPI_File myfile; /* Shared file */
MPI_Status status; /* Status returned from read */
/* Initialize MPI */
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &myid);
MPI_Comm_size(MPI_COMM_WORLD, &np);
/* Open the files */
MPI_File_open (MPI_COMM_WORLD, FILENAME, MPI_MODE_RDONLY,
MPI_INFO_NULL, &myfile);
/* Get the size of the file */
MPI_File_get_size(myfile, &filesize);
/* Calculate how many elements that is */
filesize = filesize/sizeof(double);
/* Calculate how many elements each processor gets */
bufsize = filesize/np;
/* Allocate the buffer to read to, one extra for terminating null char */
buf = (double *) malloc((bufsize)*sizeof(double));
/* Set the file view */
MPI_File_set_view(myfile, myid*bufsize*sizeof(double), MPI_DOUBLE,
MPI_DOUBLE,"native", MPI_INFO_NULL);
/* Read from the file */
MPI_File_read(myfile, buf, bufsize, MPI_DOUBLE, &status);
/* Find out how many elemyidnts were read */
MPI_Get_count(&status, MPI_DOUBLE, &nrchar);
/* Set terminating null char in the string */
//buf[nrchar] = (double)0;
printf("Process %2d read %d characters: ", myid, nrchar);
int j;
for (j = 0; j <bufsize;j++){
matrix[myid][j] = buf[j];
}
/* Close the file */
MPI_File_close(&myfile);
if (myid==0) {
printf("Done\n");
}
MPI_Finalize();
exit(0);
}
However when I try to call MPI_File_open after I close the first file, I get an error. Do I need multiple communicators to perform this? Any tips will be appreciated.
The code in ArrayAllocation above does not quite match the logic of the main program. The matrix is allocated as an array of pointers to vectors of doubles before MPI is initialized, therefore it is impossible to set the number of rows to the number of MPI processes.
The column_size is also not known until the file size is determined.
It is a general convention in the C language to store matrices by rows. Violating this convention might confuse you or the reader of your code.
All in all in order to get this program working you need to declare
int num_procs, column_size;
as global variables before the definition of ArrayAllocation and move the call to this function down below the line where bufsize is calculated:
...
/* Calculate how many elements each processor gets */
bufsize = filesize/np;
num_procs = np;
column_size = bufsize;
double **matrix = ArrayAllocation();
...
With the above modifications this example should work on any MPI implementation that supports MPI-IO. I have tested it with OpenMPI 1.2.8.
In order to generate a test file you could use for instance the following code:
FILE* f = fopen(FILENAME,"w");
double x = 0;
for(i=0;i<100;i++){
fwrite(&x, 1,sizeof(double), f);
x +=0.1;
}
fclose(f);
I'm trying to implement a program that uses a thread to read data from a file and write to a buffer of an arbitrary size, while two other threads read info from this buffer. Everything works fine, except for when I specify a buffer size of 1. When I do that, everything locks up. I'm more or less adapting the classic "consumer/producer" example from here Here's my code:
Struct I'm using:
struct prodcons {
char** buffer; pthread_mutex_t lock;
pthread_cond_t notempty; pthread_cond_t notfull;
int readpos, writepos, finished;
};
My "add to buffer" thread
static void buf_add(struct prodcons *b, char* data) {
/* Add the data after locking the buffer */
pthread_mutex_lock(&b-> lock);
printf("Reader adding %s\n", data);
int err;
/*Wait until buffer is not full*/
while ((b->writepos + 1) % numlines == b->readpos) {
err = pthread_cond_wait(&b->notfull, &b->lock);
if (err != 0) { fprintf(stderr, "cond wait");}
}
/* Needed to stop writes */
if (data != NULL) {
b->buffer[b->writepos] = strdup(data);
} else {
//fprintf(stderr, "End of file reached, adding NULL\n");
b->buffer[b->writepos] = NULL;
}
/* Increments the writing position */
(*b).writepos++;
if ((*b).writepos >= numlines) {
printf("Resetting write position\n");
(*b).writepos = 0;
}
/* Unlock */
pthread_cond_signal(&b->notempty);
pthread_mutex_unlock(&b->lock);
}
Here's what output looks like:
Reader adding 64.233.173.85
And then it just hangs. It's clear that it's never getting beyond that first while loop. It works with any other size, but not with 1. what would be the best way of implement a fix for this? If this helps, here is my "get from buffer" method.
static void *read_from_buffer(struct prodcons *b) {
pthread_mutex_lock(&b -> lock);
/* We have to wait for the buffer to have something in it */
while ((*b).writepos == (*b).readpos) {
pthread_cond_wait(&b->notempty, &b->lock);
}
/* Set the thread delay */
thread_delay.tv_sec = threaddelay / 100000;
thread_delay.tv_nsec = 1000*threaddelay%100000;
char *t = NULL;
/* Read the data and advance the reader pointer */
if ((*b).buffer[(*b).readpos] != NULL) {
t = (char*)malloc(strlen ((*b).buffer[(*b).readpos] ) + 1);
strcpy(t, (*b).buffer[(*b).readpos]);
printf("Consumer %u reading from buffer: got %s\n", (unsigned int)pthread_self(), t);
/*At this point, we should probably check is FQDN*/
if (strcmp(t, "-1") == 0) {
(*b).finished = 1;
} else {
nanosleep(&thread_delay, &thread_delay_rem);
check_cache(t, &cache);
}
}
/* We have to adjust the reading position */
(*b).readpos++;
if ( (*b).readpos >= numlines) {
(*b).readpos = 0;
}
/*Need to signal and unlock */
pthread_cond_signal (&b->notfull);
pthread_mutex_unlock(&b->lock);
return t;
}
I'm sure there is a fairly simple fix to handle this edge case, but I can't seem to figure it out. Any suggestions would be much appreciated!
EDIT: I also initialize my buffer like so:
static void init(struct prodcons *temp) {
(*temp).buffer = (char**)malloc(numlines * sizeof(char*));
Not having stepped through your code, but you're writing a terminating NUL '\0' byte, and that would take up an entire 1-byte buffer. The writer waits forever for space in the buffer.
while ((b->writepos + 1) % numlines == b->readpos) { /* always satisfied */