Extracting data from an unknown encoding file - arrays

We use testing equipment (1995 year manufacturing) powered by MS DOS. Analog-digital converter records information in the file.
In [picture1] is shown the structure of that file.
In [picture2] is shown the oscillogram that constructed according to the data from the file (program for opening the file on MS DOS).
Below I placed link to this file (google drive).
This file contains the data that need for me - the massive of point of oscillogram. I want have opportunities to keep, analyze and print this chart on Windows or Linux (not MS DOS). So I need to extract data from the file.
But I can't make it. And no program (known to me) can't open this file. I analyzed a few first byte and they point to program 'TRAS v4.99'. This program is on MS DOS.
But I really hope, that it is really to get data without this program.
P.S. If anyone will say it is impossible - it is will well too because I haven't found point of view yet:)
Thank you for your time! Best regards!
LINK TO FILE ON GOOGLE DISK - 00014380.K00
STRUCTURE OF FILE
OPENING FILE VIA PROGRAM IN MS DOS

Here is an idea on how you can tackle this problem. Since the format is relatively well specified in the handbook you can use the Java programming language for example with something like java.io.RandomAccessFile to read arrays of bytes. These arrays of bytes can then be converted to Java primitive types OR to string according to the data type. After this conversion you can the print out the data in a human readable format.
Below you can find some sample code to give you an idea of what you could do with this approach (I have not tested the code, it is not complete, it is just to give you an idea of what you can do):
public static void readBinaryfile() throws IOException {
java.io.RandomAccessFile randomAccessFile = new RandomAccessFile("test.bin", "r");
byte[] addKenStrBytes = new byte[12];
randomAccessFile.read(addKenStrBytes);
String addKenStr = new String(addKenStrBytes, "UTF-8");
// TODO: Do something with addKenStr.
System.out.println(addKenStr);
byte[] kopfSizeBytes = new byte[2];
randomAccessFile.read(kopfSizeBytes);
// TODO: Do something with kopfSizeBytes
System.out.println(convertToInt(kopfSizeBytes));
byte[] addRufNrCounterBytes = new byte[6];
randomAccessFile.read(addRufNrCounterBytes);
long addRufNrCounter = convertToLong(addRufNrCounterBytes);
// TODO: Do something with addRufNrCounter
System.out.println(addRufNrCounter);
byte[] endAdrBytes = new byte[4];
randomAccessFile.read(endAdrBytes);
// TODO: Do something with endAdrBytes
System.out.println(convertToLong(endAdrBytes));
// Continue here and after you reached the end of the record repeat until you reached the end off the file
}
private static int convertToInt(byte[] bytes) {
if(bytes.length > 4) {
throw new IllegalArgumentException();
}
int buffer = 0;
for(byte b : bytes) {
buffer |= b;
buffer = buffer << 8;
}
return buffer;
}
private static long convertToLong(byte[] bytes) {
if(bytes.length > 8) {
throw new IllegalArgumentException();
}
long buffer = 0L;
for(byte b : bytes) {
buffer |= b;
buffer = buffer << 8;
}
return buffer;
}
Note that fields with more than 8 bytes need to be most probably converted to strings. This is not complete code, just an example to give you an idea on how you can tackle this problem.

Related

How do I read OpenVINO IR models from memory with the OpenVINO C API

I am having trouble reading OpenVINO IR networks (XML and bin) from memory using ie_core_read_network_from_memory() in the OpenVINO 2021.4 C API ie_c_api.h.
I suspect that I am creating the network weight blob wrong, but I cannot find any information on how to create weight blobs correctly for networks.
I have read the OpenVINO C API docs but cannot deduce from docs what I am doing wrong. The OpenVINO code repo contains some C code samples, but none of the samples seem to use ie_core_read_network_from_memory().
Below is a cut out of the code I am having trouble with.
// void* dmem->data - network memory buffer (float32)
// size_t dmem->size - size of network memory buffer (bytes)
ie_core_t* ov_core = NULL;
IEStatusCode status = ie_core_create("", &ov_core);
if (status != OK)
{
// error handling
}
const dimensions_t weights_tensor_dims =
{ 4, { 1, 1, 1, dmem->size/sizeof(float) } };
tensor_desc_t weights_tensor_desc = { OIHW, weights_tensor_dims, FP32 };
ie_blob_t* ov_model_weight_blob = NULL;
status = ie_blob_make_memory_from_preallocated(
&weights_tensor_desc, dmem->data, dmem->size, &ov_model_weight_blob);
if (status != OK)
{
// error handling
}
// char* model_xml_desc - the model's XML string
uint8_t* ov_model_xml_content = (uint8_t*)model_xml_desc;
ie_network_t* ov_network = NULL;
size_t xml_sz = strlen(ov_model_xml_content);
status = ie_core_read_network_from_memory(
ov_core, ov_model_xml_content, xml_sz, ov_model_weight_blob, &ov_network);
if (status != OK)
{
// Always get "GENERAL_ERROR (-1)"
}
The code works fine down to the ie_core_read_network_from_memory() call which results in "GENERAL_ERROR".
I have tried two models that were converted from Tensorflow. One is a simple [X] -> [Y] regression model (single input value, single output value). The other is also a regression model [X_1, X_2, ..., X_9] -> [Y] (nine input values, single output value). They work fine when reading them from file with ie_core_read_network(), but for my use case I must provide the network as a binary memory buffer and XML string.
I would appreciate any help, either by pointing out what I am getting wrong or directing me to some code samples that use ie_core_read_network_from_memory().
System information:
Windows 10
OpenVINO v2021.4.689
Microsoft Visual Studio 2019
UPDATE: An Intel employee reached out to me in another forum and pointed out that there is a unit test for ie_core_read_network_from_memory(). The unit test successfully reads a network from memory and made clear that I was in fact using a faulty tensor description to produce the weight blob, just as I suspected. Apparently the weight blob descriptor should be one dimensional, have memory layout ANY and datatype U8 even though the model weights are fp32.
From the unit test:
std::string bin_std = TestDataHelpers::generate_model_path("test_model", "test_model_fp32.bin");
const char* bin = bin_std.c_str();
//...
std::vector<uint8_t> weights_content(content_from_file(bin, true));
tensor_desc_t weights_desc { ANY, { 1, { weights_content.size() } }, U8 };
However, simply changing the tensor descriptor was not enough to get my code to work so it remains for me to properly translate the C++ code from the unit test to my C environment before the issue to can be considered solved.
Thanks
Refer to tensor_desc struct and standard layout format.
Apart from that, it is recommended to use the Benchmark_app tool to test the inference performance.

Split ogg vorbis stream without BOS

Input: a stream of ogg/vorbis coming from an encoder chip of an embedded system.
Problem: create output chunks of one second without transcoding.
Issue: the stream is being read "in the middle", so the first page with BOS (Beginning of Stream) is not available. Since the encoder chip has always the same parameters, I'd like to recreate the BOS page using the BOS page of a stream that was read from the start (reference stream).
I am trying to use vcut. I modified it so that it creates infinite chunks of one second. It was easy, and it works with files and streams with BOS.
I also hacked it so that I wrote to a file the first pages of the reference stream and then read them before reading the production stream with no BOS. In this way, vs->headers are populated. When I detect a page serial number change, I change it so that vcut and libogg do not freak:
int process_page(vcut_state *s, ogg_page *page) {
...
else if(vs->serial != ogg_page_serialno(page))
{
// fprintf(stderr, _("Multiplexed bitstreams are not supported.\n"));
vs->stream_in.serialno = ogg_page_serialno(page);
vs->serial = ogg_page_serialno(page);
vs->granulepos = -1;
vs->initial_granpos = 0;
// ogg_stream_init(&vs->stream_in, vs->serial);
// vorbis_info_init(&vs->vi);
// vorbis_comment_init(&vs->vc);
s->vorbis_init = 1;
}
However, this gigantic hack does not work. How to solve this issue?
It actually works: see VS1053 split ogg.
What I needed to do was to consider that starting reading in the middle of the stream, granulepos was naturally high. So it was mine logical mistake.
In process_audio_packet, I added:
int process_audio_packet(vcut_state *s,
vcut_vorbis_stream *vs, ogg_packet *packet)
{
...
if(packet->granulepos >= 0)
{
if (!firstNonZeroGranule) { // my addition
firstNonZeroGranule = 1;
vs->initial_granpos = packet->granulepos - bs;
if(vs->initial_granpos < 0)
vs->initial_granpos = 0;
} else if(vs->granulepos == 0 && packet->granulepos != bs) {
...

Wrapping a java.nio.ByteBuffer in a new instance, with offset, loses bytes

I'm trying to write the contents of a byte buffer to a file, using the offset (position). It works when I convert to an input stream, but not when I wrap in a new ByteBuffer
This works:
new ByteArrayInputStream(byteBuffer.array(), byteBuffer.position(), byteBuffer.array().length - byteBuffer.position())
This doesn't
ByteBuffer.wrap(byteBuffer.array(), byteBuffer.position(), byteBuffer.array().length - byteBuffer.position())
More specifically, when I say it doesn't work, writing the contents of the buffer to a file:
Files.write(path, ByteBuffer.wrap(byteBuffer.array(), byteBuffer.position(), byteBuffer.array().length - byteBuffer.position()).array())
results in bytes written to the file but it is not complete, so the jpeg cannot be viewed, but if I write the same buffer, wrapping in a ByteArrayInputStream, it does work:
val in = new ByteArrayInputStream(byteBuffer.array(), byteBuffer.position(), byteBuffer.array().length - byteBuffer.position())
Iterator.continually (in.read).takeWhile (-1 != _).foreach (fileOutputStream.write)
So I must be doing something silly and perhaps I don't understand how ByteBuffer works
ByteBuffer.wrap(byteBuffer.array(), <ANYTHING>, <ANYTHING>).array() means just byteBuffer.array(), and <ANYTHING> isn't taken into account.
Also, the whole
ByteBuffer.wrap(byteBuffer.array(), byteBuffer.position(), byteBuffer.array().length - byteBuffer.position())
is just a cumbersome way to create a shallow copy of byteBuffer, why do you do it instead of just using the byteBuffer itself?
Looks like what you want is something like
try (FileChannel outCh = new FileOutputStream(filename).getChannel()) {
outCh.write(byteBuffer);
}

fread returns no data in my buffer in spite of saying it read 4096 bytes

I'm porting some C code that loads sprites from files containing multiple bitmaps. Basically the code fopens the file, fgetcs some header info, then freads the bitmap data. I can see that the fgetcs are returning proper data, but the outcome of the fread is null. Here's the code - fname does exist, the path is correct, fil is non-zero, num is the number of sprites in the file (encoded into the header, little-endian), pak is an array of sprites, sprite is a typedef of width, height and bits, and new_sprite inits one for you.
FILE *fil;
uint8 *buffu;
uint8 read;
int32 x,num;
int32 w,h,c;
fil = fopen(fname, "rb");
if (!fil) return NULL;
num = fgetc(fil);
num += fgetc(fil)*256;
if (num > max) max = num;
for (x=0;x<max;x++) {
// header
w=fgetc(fil);
w+=fgetc(fil)*256;
h=fgetc(fil);
h+=fgetc(fil)*256;
fgetc(fil); // stuff we don't use
fgetc(fil);
fgetc(fil);
fgetc(fil);
// body
buffu = (uint8*)malloc(w * h);
read=fread(buffu,1,w*h,fil);
pak->spr[x]=new_sprite(w,h);
memcpy(pak->spr[x]->data, buffu, w*h);
// done
free(buffu);
}
I've stepped through this code line by line, and I can see that w and h are getting set up properly, and read=4096, which is the right number of bits. However, buffer is "" after the fread, so of course memcpy does nothing useful and my pak is filled with empty sprites.
My apologies for what is surely a totally noob question, but I normally use Cocoa so this pure-C file handling is new to me. I looked all over for examples of fread, and they all look like the one here - which apparently works fine on Win32.
Since fgetc seems to work, you could try this as a test
int each;
int byte;
//body
buffu = malloc(w * h);
for (each = 0; each < w*h; each++) {
byte = fgetc(fil);
if ( byte == EOF) {
printf("End of file\n");
break;
}
buffu[each] = (uint8)byte;
printf ("byte: %d each: %d\n", byte, each);
}
pak->spr[x]=new_sprite(w,h);
memcpy(pak->spr[x]->data, buffu, w*h);
// done
You say:
However, buffer is "" after the fread, so of course memcpy does nothing useful
But that is not true at all. memcpy() is not a string function, it will copy the requested number of bytes. Every time. If that isn't "useful", then something else is wrong.
Your buffer, when treated as a string (which it is not, it's a bunch of binary data) will look like an empty string if the first byte happens to be 0. The remaining 4095 bytes can be whatever, to C's string printing functions it will look "empty".

Reading unsingned chars with Qt

Hi fellow stack overflowers,
I'm currently parsing a file which both contains text and binary data. Currently, I'm reading the file in following manner:
QTextStream in(&file);
int index = 0;
while(!in.atEnd()) {
if (index==0) {
QString line = in.readLine(); // parse file here
} else {
QByteArray raw_data(in.readAll().toAscii());
data = new QByteArray(raw_data);
}
index++;
}
where data refers to the binary data I'm looking for. I'm not sure if this is what I want, since the QString is encoded into ascii and I have no idea if some bytes are lost.
I checked the documentation, and it recommends using a QDataStream. How can I combine both approaches, i.e. read lines with an encoding and also read the binary dump, after one line break?
Help is greatly appreciated!
This will do what you want.
QTextStream t(&in);
QString line;
QByteArray raw_data;
if(!in.atEnd()) {line = t.readLine();}
in.reset();
int lineSize = line.toLocal8Bit().size() + 1;
in.seek(lineSize);
if(!in.atEnd())
{
int len = in.size() - lineSize;
QDataStream d(&in);
char *raw = new char[len]();
d.readRawData(raw, len);
raw_data = QByteArray(raw, len);
delete raw;
}
PS: if file format is yours, it will be better to create file with QDataStream and write data with <<, read with >>. This way you can store QByteArray and QString in file without such problems.

Resources