Writing a VTK ASCII Legacy File to draw contours in VisIt - c

I'm trying to write a legacy .vtk file to be read into VisIt using C. Unfortunately my installed VisIt program refuses to render the VTK file that I have writing, reading: 'local host failed'
Below is the code used to read data from one file and convert it to a legacy VTK file. I use the macros XPIX, YPIX, and ZPIX to describe the dimensions of a pixel grid. Each pixel contains a scalar density value. I have listed the pixels in a 'grid-file' using row-major ordering: i.e.
int list_index(x,y,z) = YPIX * ZPIX * x + ZPIX * y + z;
Every entry in this pixel list is read into an array called grid[] of type double, and written to outfile beneath the legacy VTK header data:
/*Write vtk header */
fprintf(outfile,"# vtk DataFile Version 3.0\n");
fprintf(outfile,"Galaxy density grid\nASCII\nDATASET STRUCTURED_POINTS\n");
fprintf(outfile,"DIMENSIONS %d %d %d \n", (XPIX+1), (YPIX+1), (ZPIX+1));
fprintf(outfile,"ORIGIN 0 0 0\n");
fprintf(outfile,"SPACING 1 1 1\n");//or ASPECT_RATIO
fprintf(outfile,"CELL_DATA %d\n", totalpix);
fprintf(outfile,"SCALARS cell_density float 1\n");
fprintf(outfile, "LOOKUP_TABLE default\n");
/*Create Memory Space to store Pixel Grid*/
double *grid;
grid = malloc(XPIX * YPIX * ZPIX * sizeof(double));
if (grid == NULL ){
fprintf(stderr, "Pixel grid of type double failed to initialize\n");
exit(EXIT_FAILURE);
}
fprintf(stderr,"Pixel grid has been initialized.\n Now reading infile\n");
/*Read infile contents into double grid[], using Row-Major Indexing*/
double rho;
char newline;
int i, j, k;
for(i = 0; i < XPIX; i++){
for(j = 0; j < YPIX; j++){
for(k = 0; k < ZPIX; k++){
fscanf(infile, "%lf", &rho);
grid[getindex(i,j,k)] = rho;
}
}
fprintf(stderr,"%d\n", i);
}
fprintf(stderr,"Finished reading\n");
#if !DEBUG
/*Write out grid contents in Row major order*/
fprintf(stderr,"Now writing vtk file");
for(i = 0; i < XPIX; i++){
for(j = 0; j < YPIX; j++){
for(k = 0; k < ZPIX; k++){
fprintf(outfile, "%lf ", grid[getindex(i,j,k)]);
}
fprintf(outfile,"\n");
}
}
fprintf(stderr,"Finished Writing to outfile\n");
#endif
After running the grid data list through this routine, I have XPIX*YPIX lines in the lookup_table, each with ZPIX entries. Is this an incorrect format? VisIt continues to fail reading the input file. I'm aware of the fact that structured_points may use column major indexing, but my first goal of course is to get some sort of result from VisIt. I would like to draw a contour using the scalar cell_density eventually. Is my data set simply too large?

Have you seen the accepted answer to the question vtk data format error? The question is debugging a VTK writer in C++ but it is very similar to your code (and of course should yield the same results).
The key point from the accepted answer is that data is written in column major order, not row major order (you seem to hint at this in your question: "I'm aware of the fact that structured_points may use column major indexing").
Also, it is always helpful (if you can) to compare your code with something which you know works. For example, VisIt provides a small C library for writing legacy VTK file formats called VisItWriterLib. Compare the output from your code and the VisItWriterLib to see where your data files differ. I would recommend using VisItWriterLib for your VTK IO rather than writing your own routines - no need to reinvent the wheel.
Edit: To answer a couple of your other questions:
After running the grid data list through this routine, I have
XPIX*YPIX lines in the lookup_table, each with ZPIX entries. Is this
an incorrect format?
This is not the correct format. LOOKUP_TABLE should be a list of XPIX*YPIX*ZPIX lines, with one element per line (or alternatively, VisIt will accept one line with XPIX*YPIX*ZPIX elements). See the section Dataset Attribute Format in the VTK File Formats document (www.vtk.org/VTK/img/file-formats.pdf).
Is my data set simply too large?
I doubt it. VisIt is designed to handle huge datasets and, AFAIK, can render petabyte data sets. I would be very surprised if your data is that large.
However, if you are concerned about having large files, you can split your data into multiple files and tell VisIt to read these files is parallel. To do this, write a bit a your data into separate files, e.g. domain1.vtk, domain2.vtk, ... domainN.vtk etc. Then write a .visit master file, which has the structure
!NBLOCKS N
domain1.vtk
domain2.vtk
...
domainN.vtk
Save this as, for example, mydata.visit and then open this .visit file, rather than the .vtk files, in VisIt.

Related

Errors with gnuplot after execution

I've been wanting to plot some data with gnuplot. What I've written compiles fine, and also executes. The plot itself even seems to work in gnuplot.
However, I'm getting an warning that I want to understand:
Warning: slow font initializationqt_processTermEvent received a GE_fontprops event. This should not have happened
What does this error mean, and how can I avoid it? Does this even have any effect on the plot?
I've included my code below, which is a crude population simulation to see chaotic behavior.
#include <stdio.h>
int main(int argc, char const *argv[]) {
float currentPop;
float nextPop = 0;
float growthRate;
FILE *fp = NULL;
FILE *gnupipe = NULL;
char *GnuCommands [] = {"set title \"Popsim\"", "plot 'data.tmp'"};
fp = fopen("data.tmp", "w");
gnupipe = _popen("gnuplot -persistent", "w");
printf("Enter current population and growth rate:\n");
scanf("%f %f", &currentPop, &growthRate);
for (int counter = 0; counter < 30; counter++) {
nextPop = growthRate * currentPop * (1 - currentPop);
fprintf(fp, "%d %f\n", counter, nextPop);
currentPop = nextPop;
}
for (int i = 0; i < 2; i++) {
fprintf(gnupipe, "%s\n", GnuCommands[i]);
}
return 0;
}
Also, a bonus question: How do I make gnuplot draw a line between the points it plots, so that it's easier for me to visualise?
Gnuplot does not do its own font handling. Depending on what output mode (gnuplot calls it "terminal") is being used it queries various system libraries or subsystems to estimate how much space will be occupied by a particular string of characters that will be placed on the graph.
If the response to the query is an error, or is too slow, it prints that warning and continues with a best-guess estimate of the space required.
As I understand it, the most common underlying cause for this is that the font being requested is not currently in the system font cache so the response to the first query is very slow. After that the font is in the cache so the response is quick and you don't see the message again, at least not from the same gnuplot session.
There is a command-line option that tells the program to wait longer for the system to respond:
gnuplot --slow
That will probably avoid the warning message, and possibly improve text layout on the first plot, but it may result in a noticeable delay in producing that first plot while the system font cache is updated.

How does fread() in C work inside a for loop?

I am new to C programming, but I need it to read some binary file which I describe below.
The India Meteorological Department (IMD) has provided historical weather data in .GRD files in their website. They have also provided sample C code to read those files. From their sample C code, I have written the following code that extracts the daily minimum temperatures on 15 April 1980 recorded on a 31x31 grid over India.
/* This program reads binary data for 365/366 days and writes in ascii file. */
#include <stdio.h>
int main() {
float t[31][31];
int i,j ,k;
FILE *fin,*fout;
fin = fopen("C:\\New folder\\Mintemp_MinT_1980.GRD","rb"); // Input file
fout = fopen("C:\\New folder\\MINT15APR1980.TXT","w"); // Output file
fprintf(fout,"Daily Minimum Tempereture for 15 April 1980\n");
if(fin == NULL) {
printf("Can't open file");
return 0;
}
if(fout == NULL) {
printf("Can't open file");
return 0;
}
for(k=0 ; k<366 ; k++) {
fread(&t,sizeof(t),1,fin);
if(k == 105) {
for(i=0 ; i < 31 ; i++) {
fprintf(fout,"\n") ;
for(j=0 ; j < 31 ; j++)
fprintf(fout,"%6.2f",t[i][j]);
}
}
}
fclose(fin);
fclose(fout);
return 0;
}
/* end of main */
The file Mintemp_MinT_1980.GRD can be downloaded from the IMD website by selecting the year as 1980 against Minimum Temperature.
What I don't understand is that how the fread() function actually works in the line fread(&t,sizeof(t),1,fin) within the loop for(k=0 ; k<366 ; k++). In plain sight, the arguments of fread() here do not depend on the looping variable k, and so it should read the same data to the matrix t[31][31] for every k. However, I have checked that, surprisingly, the data extracted by this program are different for different values of k in the line if(k == 105), i.e., the data extracted for k == 105 and k == 32 are different, for example.
I would very much appreciate if one can please explain the above.
Files contain sequential data. All the file operators are based on the premise that whatever you do to a file, you'll generally be doing it in a sequential way.
So when you read data, and then read more data, you will be getting sequential chunks of the file. The both the FILE datatype and the operating system itself do a number of things for you, including keeping track of your current position in the file and doing block buffering in memory to improve performance.
If you wanted to reread the same data over, or skip around in the file, you would need to use fseek() to change positions in the file before doing your next read.

gnu FORTRAN unformatted file record markers stored as 64-bit width?

I have a legacy code and some unformatted data files that it reads, and it worked with gnu-4.1.2. I don't have access to the method that originally generated these data files. When I compile this code with a newer gnu compiler (gnu-4.7.2) and attempt to load the old data files on a different computer, it is having difficulty reading them. I start by opening the file and reading in the first record which consists of three 32-bit integers:
open(unit, file='data.bin', form='unformatted', status='old')
read(unit) x,y,z
I am expecting these three integers here to describe x,y,z spans so that next it can load a 3D matrix of float values with those same dimensions. However, instead it's loading a 0 for the first value, then the next two are offset.
Expecting:
x=26, y=127, z=97 (1A, 7F, 61 in hex)
Loaded:
x=0, y=26, z=127 (0, 1A, 7F in hex)
When I checked the data file in a hex editor, I think I figured out what was happening.
The first record marker in this case has a value of 12 (0C in hex) since it's reading three integers at 4 bytes each. This marker is stored both before and after the record. However, I notice that the 32bits immediately after each record marker is 00000000. So either the record markers are treated as 64bit integers (little-Endian) or there is a 32-bit zero padding after each record marker. Either way, the code generated with the new compiler is reading the record markers as 32-bit integers and not expecting any padding. This effectively intrudes/corrupts the data being read in.
Is there an easy way to fix this non-portable issue? The old and new hardware are 64 bit architecture and so is the executable I compiled. If I try to use the older compiler version again will it solve the problem, or is it hardware dependent? I'd prefer to use the newer compilers because they are more efficient, and I really don't want to edit the source code to open all the files as access='stream' and manually read in a trailing 0 integer after each record marker, both before and after each record.
P.S. I could probably write a C++ code to alter the data files and remove these zero paddings if there is no easier alternative.
See the -frecord-marker= option in the gfortran manual. With -frecord-marker=8 you can read the old style unformatted sequential files produced by older versions of gfortran.
Seeing as how Fortran doesn't have a standardization on this, I opted to convert the data files to a new format that uses 32-bit wide record lengths instead of 64-bit wide. In case anyone needs to do this in the future I've included some Visual C++ code here that worked for me and should be easily modifiable to C or another language. I have also uploaded a Windows executable (fortrec.zip) here.
CFile OldFortFile, OutFile;
const int BUFLEN = 1024*20;
char pbuf[BUFLEN];
int i, iIn, iRecLen, iRecLen2, iLen, iRead, iError = 0;
CString strInDir = "C:\folder\";
CString strIn = "file.dat";
CString strOutDir = "C:\folder\fortnew\"
system("mkdir \"" + strOutDir + "\""); //create a subdir to hold the output files
strIn = strInDir + strIn;
strOut = strOutDir + strIn;
if(OldFortFile.Open(strIn,CFile::modeRead|CFile::typeBinary)) {
if(OutFile.Open(strOut,CFile::modeCreate|CFile::modeWrite|CFile::typeBinary)) {
while(true) {
iRead = OldFortFile.Read(&iRecLen, sizeof(iRecLen)); //Read the record's raw data
if (iRead < sizeof(iRecLen)) //end of file reached
break;
OutFile.Write(&iRecLen, sizeof(iRecLen));//Write the record's raw data
OldFortFile.Read(&iIn, sizeof(iIn));
if (iIn != 0) {//this is the padding we need to ignore, ensure it's always zero
//Padding not found
iError++;
break;
}
i = iRecLen;
while (i > 0) {
iLen = (i > BUFLEN) ? BUFLEN : i;
OldFortFile.Read(&pbuf[0], iLen);
OutFile.Write(&pbuf[0], iLen);
i -= iLen;
}
if (i != 0) { //Buffer length mismatch
iError++;
break;
}
OldFortFile.Read(&iRecLen2, sizeof(iRecLen2));
if (iRecLen != iRecLen2) {//ensure we have reached the end of the record proeprly
//Record length mismatch
iError++;
break;
}
OutFile.Write(&iRecLen2, sizeof(iRecLen));
OldFortFile.Read(&iIn, sizeof(iIn));
if (iIn != 0) {//this is the padding we need to ignore, ensure it's always zero
//Padding not found
break;
}
}
OutFile.Close();
OldFortFile.Close();
}
else { //Could not create the ouput file.
OldFortFile.Close();
return;
}
}
else { //Could not open the input file
}
if (iError == 0)
//File successfully converted
else
//Encountered error

How to use vlfeat sift matching function in C code?

I just found one similar question here. But I just want to do a matching based on the description result from vlfeat. The goal to detect if an image contains the object in another image, based on sift feature description extracting and matching. And I need to do it in C, not Matlab.
So how can I call vl_ubcmatch function in C code?
So how can I call vl_ubcmatch function in C code?
This is a MEX function which is only intented to be called from MATLAB. You cannot re-use it as-is from a general purpose C program.
The goal to detect if an image contains the object in another image [...] How to do SIFT matching algorithm if I use vlfeat?
VLFeat C API does not provide SIFT matching functions out-of-the box. So basically you need to adapt the so-called ratio test [1] code section from this MATLAB C code section which is fairly easy (see below).
The main drawback if you want to perform robust matching is that this function does not take into account the geometry, i.e the keypoints coordinates.
What you need in addition is a geometrical consistency check which is typically performed by figuring out if there is an homography between the two images (using as input the descriptor correspondences obtained with the ratio test). This is done with an algorithm like RANSAC since the correspondences may include outliers.
But also you can speed up correspondences computation with a kd-tree.
So an alternative if you need a plain C implementation is relying on Open SIFT by Rob Hess which includes everything you need, as well as a ready-to-use command-line tool (and thus example) of matching:
See match.c.
typedef struct {
int k1;
int k2;
double score;
} Pair;
Pair *
compare(
Pair *pairs,
const float *descr1,
const float *descr2,
int K1,
int K2,
int ND,
float thresh
)
{
int k1, k2;
/* Loop over 1st image descr. */
for (k1 = 0; k1 < K1; ++k1, descr1 += ND ) {
float best = FLT_MAX;
float second_best = FLT_MAX;
int bestk = -1;
/* Loop over 2nd image descr. and find the 1st and 2nd closest descr. */
for (k2 = 0; k2 < K2; ++k2, descr2 += ND ) {
int bin;
float acc = 0;
/* Compute the square L2 distance between descriptors */
for (bin = 0 ; bin < ND ; ++bin) {
float delta = descr1[bin] - descr2[bin];
acc += delta*delta;
if (acc >= second_best)
break;
}
if (acc < best) {
second_best = best;
best = acc;
bestk = k2;
}
else if (acc < second_best) {
second_best = acc;
}
}
/* Rewind */
descr2 -= ND*K2;
/* Record the correspondence if the best descr. passes the ratio test */
if (thresh * best < second_best && bestk != -1) {
pairs->k1 = k1;
pairs->k2 = bestk;
pairs->score = best;
pairs++;
}
}
return pairs;
}
K1: number of descriptors in image 1,
K2: number of descriptors in image 2,
ND: descriptor dimension (= 128 for SIFT),
descr1 and descr2: descriptors of image 1 and 2 resp. in row major order, e.g K1 lines x ND columns),
thresh: ratio test threshold value, e.g 1.5 in MATLAB code.
[1] see 7.1 Keypoint Matching from D. Lowe's paper.
You will use the vlfeat library the same way you use any other library with C. First make sure you have the library installed on your computer and know where it is installed. You will need to include the required header for each part of vlfeat you are using. Generally a generic library header for vlfeat and then a specific header for sift (e.g. #include "sift.h")(sometimes there is no general header). You will need to insure gcc or g++ command includes the proper INCLUDE_PATH and LIBRARY_PATH for your environment that will allow gcc to find your vlfeat files. (e.g. -I/path/to/dir/holding_sift.h and -L/path/to/vlfeatlib) So you will end up with something like this for C:
gcc -o exename exename.c -I/path/to/dir/holding_sift.h -L/path/to/vlfeatlib -lvl
There is documentation on line that will help. See: how to setup a basic C++ project which uses the VLFeat library If you have further questions, just drop a line in the comments.

Create an array of values from different text files in C

I'm working in C on 64-bit Ubuntu 14.04.
I have a number of .txt files, each containing lines of floating point values (1 value per line). The lines represent parts of a complex sample, and they're stored as real(a1) \n imag(a1) \n real(a2) \n imag(a2), if that makes sense.
In a specific scenario there are 4 text files each containing 32768 samples (thus 65536 values), but I need to make the final version dynamic to accommodate up to 32 files (the maximum samples per file would not exceed 32768 though). I'll only be reading the first 19800 samples (depending on other things) though, since the entire signal is contained in those 39600 points (19800 samples).
A common abstraction is to represent the files / samples as a matrix, where columns represent return signals and rows represent the value of each signal at a sampling instant, up until the maximum duration.
What I'm trying to do is take the first sample from each return signal and move it into an array of double-precision floating point values to do some work on, move on to the second sample for each signal (which will overwrite the previous array) and do some work on them, and so forth, until the last row of samples have been processed.
Is there a way in which I can dynamically open files for each signal (depending on the number of pulses I'm using in that particular instance), read the first sample from each file into a buffer and ship that off to be processed. On the next iteration, the file pointers will all be aligned to the second sample, it would then move those into an array and ship it off again, until the desired amount of samples (19800 in our hypothetical case) has been reached.
I can read samples just fine from the files using fscanf:
rx_length = 19800;
int x;
float buf;
double *range_samples = calloc(num_pulses, 2 * sizeof(range_samples));
for (i=0; i < 2 * rx_length; i++){
x = fscanf(pulse_file, "%f", &buf);
*(range_samples) = buf;
}
All that needs to happen (in my mind) is that I need to cycle both sample# and pulse# (in that order), so when finished with one pulse it would move on to the next set of samples for the next pulse, and so forth. What I don't know how to do is to somehow declare file pointers for all return signal files, when the number of them can vary inbetween calls (e.g. do the whole thing for 4 pulses, and on the next call it can be 16 or 64).
If there are any ideas / comments / suggestions I would love to hear them.
Thanks.
I would make the code you posted a function that takes an array of file names as an argument:
void doPulse( const char **file_names, const int size )
{
FILE *file = 0;
// declare your other variables
for ( int i = 0; i < size; ++i )
{
file = fopen( file_names[i] );
// make sure file is open
// do the work on that file
fclose( file );
file = 0;
}
}
What you need is a generator. It would be reasonably easy in C++, but as you tagged C, I can imagine a function, taking a custom struct (the state of the object) as parameter. It could be something like (pseudo code) :
struct GtorState {
char *files[];
int filesIndex;
FILE *currentFile;
};
void gtorInit(GtorState *state, char **files) {
// loads the array of file into state, set index to 0, and open first file
}
int nextValue(GtorState *state, double *real, double *imag) {
// read 2 values from currentFile and affect them to real and imag
// if eof, close currentFile and open files[++currentIndex]
// if real and imag were found returns 0, else 1 if eof on last file, 2 if error
}
Then you main program could contain :
GtorState state;
// initialize the list of files to process
gtorInit(&state, files);
double real, imag);
int cr;
while (0 == (cr = nextValue(&state, &real, &imag)) {
// process (real, imag)
}
if (cr == 2) {
// process (at least display) error
}
Alternatively, your main program could iterate the values of the different files and call a function with state analog of the above generator that processes the values, and at the end uses the state of the processing function to get the results.
Tried a slightly different approach and it's working really well.
In stead of reading from the different files each time I want to do something, I read the entire contents of each file into a 2D array range_phase_data[sample_number][pulse_number], and then access different parts of the array depending on which range bin I'm currently working on.
Here's an excerpt:
#define REAL(z,i) ((z)[2*(i)])
#define IMAG(z,i) ((z)[2*(i)+1])
for (i=0; i<rx_length; i++){
printf("\t[%s] Range bin %i. Samples %i to %i.\n", __FUNCTION__, i, 2*i, 2*i+1);
for (j=0; j<num_pulses; j++){
REAL(fft_buf, j) = range_phase_data[2*i][j];
IMAG(fft_buf, j) = range_phase_data[2*i+1][j];
}
printf("\t[%s] Range bin %i done, ready to FFT.\n", __FUNCTION__, i);
// do stuff with the data
}
This alleviates the need to dynamically allocate file pointers and in stead just opens the files one at a time and writes the data to the corresponding column in the matrix.
Cheers.

Resources