I have a file where I'm trying to read each line into a struct in C to further work with it.
The file looks like this:
Bread,212,2.7,36,6,9.8,0.01,0.01,10,500
Pasta,347,2.5,64,13,7,0.01,0.01,6,500
Honey,340,0.01,83,0.01,0.01,0.01,0.01,22.7,425
Olive-oil,824,92,0.01,0.01,0.01,0.01,13.8,35,500
White-beans,320,2.7,44,21,18,0.01,0.01,11,400
Flaxseed-oil,828,92,0.01,0.01,0.01,52,14,100,100
Cereal,363,6.5,58,13,9.9,0.01,0.01,11,1000
Hazelnuts,644,61.6,10.5,12,0.01,0.09,7.83,16.74,252
So I wrote a for-loop to iterate over the lines in the file, trying to store each value into fields of a struct. I try to print the fields of the struct, but its already going wrong with the first argument, the string.
It is printing:
scanresult: 1, name: ■B, kcal: 0.00, omega 3: 0.00, omega 6: 0.00, carb: 0.00, protein: 0.00, fib: 0.00, price: 0.00, weight: 0.00g
Scanres should be 10, not 1, and the values should match the ones of the first line of the file.
I have tried with or without whitespace in front of the argument in the formatted string. Also I tried compiler warnings -Wall and -pedantic. No issues found.
What else could cause this problem?
The code looks like this:
#include <stdio.h>
#define MAX_CHAR 100
#define SIZE_OF_SHELF 8
typedef struct {
char name[MAX_CHAR];
double kcal, fat, omega_3, omega_6, carb, protein, fib, price, weight;
} Food;
int main(void) {
int i = 0, scanresult;
Food Shelf[SIZE_OF_SHELF];
FILE *fp;
fp = fopen("foods.txt", "r");
if (! fp) {
printf("error loading file. bye.\n");
return 0;
}
for (i = 0; !feof(fp); i++) {
scanres = fscanf(fp, " %[^,],%lf,%lf,%lf,%lf,%lf,%lf,%lf,%lf,%lf ",
Shelf[i].name,
&Shelf[i].kcal, &Shelf[i].fat,
&Shelf[i].carb, &Shelf[i].protein,
&Shelf[i].fib, &Shelf[i].omega_3,
&Shelf[i].omega_6, &Shelf[i].price,
&Shelf[i].weight);
printf("scanres: %d, name: %s, kcal: %.2f, omega 3: %.2f, omega 6: %.2f, carb: %.2f, protein: %.2f, fib: %.2f, price: %.2f, weight: %.2fg\n",
scanres, Shelf[i].name, Shelf[i].kcal,
Shelf[i].omega_3, Shelf[i].omega_6, Shelf[i].carb,
Shelf[i].protein, Shelf[i].fib, Shelf[i].price,
Shelf[i].weight);
}
return 0;
}
If anybody can spot what I'm doing wrong, please let me know.
Check if the file has a Byte Order Mark (BOM) in the first three characters. You can use hexdump (or any binary editor) to inspect it.
File with BOM:
00000000 ef bb bf 42 72 65 61 64 2c 32 31 32 2c 32 2e 37 |...Bread,212,2.7|
00000010 2c 33 36 2c 36 2c 39 2e 38 2c 30 2e 30 31 2c 30 |,36,6,9.8,0.01,0|
00000020 2e 30 31 2c 31 30 2c 35 30 30 20 0a 50 61 73 74 |.01,10,500 .Past|
00000030 61 2c 33 34 37 2c 32 2e 35 2c 36 34 2c 31 33 2c |a,347,2.5,64,13,|
...
File without BOM :
00000000 42 72 65 61 64 2c 32 31 32 2c 32 2e 37 2c 33 36 |Bread,212,2.7,36|
00000010 2c 36 2c 39 2e 38 2c 30 2e 30 31 2c 30 2e 30 31 |,6,9.8,0.01,0.01|
00000020 2c 31 30 2c 35 30 30 20 0a 50 61 73 74 61 2c 33 |,10,500 .Pasta,3|
00000030 34 37 2c 32 2e 35 2c 36 34 2c 31 33 2c 37 2c 30 |47,2.5,64,13,7,0|
...
It's likely that, besides having a Byte Order Mark (BOM), the original copy of the foods.txt file was encoded using UTF-16, instead of ASCII or the more popular and compatible UTF-8. Taking a cue from wildplasser's answer, here is a hex dump of the first portion of the file in the little-endian variant of that encoding:
00000000 ff fe 42 00 72 00 65 00 61 00 64 00 2c 00 32 00 |..B.r.e.a.d.,.2.|
00000010 31 00 32 00 2c 00 32 00 2e 00 37 00 2c 00 33 00 |1.2.,.2...7.,.3.|
00000020 36 00 2c 00 36 00 2c 00 39 00 2e 00 38 00 2c 00 |6.,.6.,.9...8.,.|
00000030 30 00 2e 00 30 00 31 00 2c 00 30 00 2e 00 30 00 |0...0.1.,.0...0.|
00000040 31 00 2c 00 31 00 30 00 2c 00 35 00 30 00 30 00 |1.,.1.0.,.5.0.0.|
00000050 20 00 0a 00 50 00 61 00 73 00 74 00 61 00 2c 00 | ...P.a.s.t.a.,.|
00000060 33 00 34 00 37 00 2c 00 32 00 2e 00 35 00 2c 00 |3.4.7.,.2...5.,.|
The leading ff fe represents the byte order mark, and would account for the mysterious ■ that showed up in the output name: ■B. Thereafter, every other byte is 0, which is why "Bread" was truncated to "B". And then fscanf's first %lf sees "r\0e\0a\0d", and can't parse that as a double, which is why fscanf returns 1 instead of 10.
copying the contents of the .txt file into a new .txt file solved the problem. It was originated in an .xls file, my guess is, there the weird BOM stuff, mentioned by some of you, comes from.
I was trying to interpret the bytes in a DBus Message as specified in https://dbus.freedesktop.org/doc/dbus-specification.html. This is taken from a pcap while using the Frida tool.
The bytes are
0000 6c 01 00 01 08 00 00 00 01 00 00 00 70 00 00 00
0010 01 01 6f 00 15 00 00 00 2f 72 65 2f 66 72 69 64
0020 61 2f 48 6f 73 74 53 65 73 73 69 6f 6e 00 00 00
0030 02 01 73 00 16 00 00 00 72 65 2e 66 72 69 64 61
0040 2e 48 6f 73 74 53 65 73 73 69 6f 6e 31 35 00 00
0050 08 01 67 00 05 61 7b 73 76 7d 00 00 00 00 00 00
0060 03 01 73 00 17 00 00 00 47 65 74 46 72 6f 6e 74
0070 6d 6f 73 74 41 70 70 6c 69 63 61 74 69 6f 6e 00
0080 00 00 00 00 00 00 00 00
There are some fields which I am uncertain what they mean. Appreciate if anyone can provide some guidance on this.
0x6C: Refers to little endian
0x01: Message Type (Method Call)
0x00: Bitwise of OR flags
0x01: Major Protocol Version
0x08000000: Length of Message Body (Little Endian), starting from end of Header. This should be referring to the eight null bytes at the end?
0x01000000: Serial of this Message (Little Endian)
0x70000000: (Little Endian) Not sure what this represents? This value does correspond to the length of the array of struct, excluding trailing null bytes, that starts from 0x0010 and ends at 0x007F.
0x01: Decimal Code for Object Path
0x01: Not sure what this represents?
0x6F: DBus Type 'o' for Object Path
0x15: Length of the Object Path string
You want to look at the part of the specification that tells you what the message format is.
But to answer your questions:
0x08000000: Length of Message Body (Little Endian), starting from end of Header. This should be referring to the eight null bytes at the end?
Correct.
0x70000000: (Little Endian) Not sure what this represents? This value does correspond to the length of the array of struct, excluding trailing null bytes, that starts from 0x0010 and ends at 0x007F.
That's the length of the array in the header. The DBus header is of a variable size - after the first few bytes, it is an array of struct(byte,variant). As per the documentation, that looks like a(yv) if you were to express this as a DBus type signature.
0x01: Decimal Code for Object Path
0x01: Not sure what this represents?
This is where the parsing gets interesting: in our struct, the signature is yv, so the first 0x01 is telling us that this struct entry is the header field for Object Path, as you have seen. However, we now need to parse what the variant contains inside of it. To marshal a variant, you first marshal a signature, which in this case is 1 byte long: 01 6f 00. Note that signatures can be a max of 255 bytes long, so unlike other strings they only have a 1-byte length at the front. As a string, that is o, which tells us that this variant contains an object path inside of it. Since object paths are strings, we then decode the next bytes as a string(keeping note that the leading 4 bytes are the string length): 15 00 00 00 2f 72 65 2f 66 72 69 64 61 2f 48 6f 73 74 53 65 73 73 69 6f 6e 00
If I've done the conversion correctly, that says /re/frida/HostSession
This is taken from a pcap
If it's a standard pcap (or pcapng) D-Bus capture file, using the LINKTYPE_DBUS link-layer type, then Wireshark should be able to read it and, at least to some degree, interpret the messages (i.e., it has code that understands the message format, as defined by the specification to which #rm5248 referred (and to which the LINKTYPE_DBUS entry in the list of link-layer header types refers), so you might not have to interpret all the bytes by yourself.
I'm having a problem trying to decode XML data returned by an instance of MS SQL Server 2014 to an app written in D7. (the version of Indy is the one which came with it, 9.00.10).
Update When I originally wrote this q, I was under the impression that the contents of the blob field needed to be Base64-decoded, but it seems that that was wrong. Having followed Remy Lebeau's suggestion, the blob stream contains recognisable text in the field names and field values before decoding but not afterwards.
In the code below, the SQL in the AdoQuery is simply
Select * from Authors where au_lname = 'White' For XML Auto
the Authors table being the one in the demo 'pubs' database. I've added the "Where" clause to restrict the size of the result set so I can show a hex dump of the returned blob.
According to the Sql Server OLH, the default type of the returned data when 'For XML Auto' is specified is 'binary base64-encoded format'. The data type of the single field of the AdoQuery is ftBlob, if I let the IDE create this field.
Executing the code below generates an exception "Uneven size in DecodeToStream". At the call to IdDecoderMIME.DecodeToString(S), the length of the string S is 3514, and 3514 mod 4 is 2, not 0 as it apparently should be, hence the exception. I've confirmed that the number of bytes in the field's value is 3514, so there's no difference between the size of the variant and the length of the string, i.e. nothing has gone awol in between.
procedure TForm1.FormCreate(Sender: TObject);
var
SS : TStringStream;
Output : String;
S : String;
IdDecoderMIME : TIdDecoderMIME;
begin
SS := TStringStream.Create('');
IdDecoderMIME := TIdDecoderMIME.Create(Nil);
try
AdoQuery1.Open;
TBlobField(AdoQuery1.Fields[0]).SaveToStream(SS);
S := SS.DataString;
IdDecoderMIME.FillChar := #0;
Output := IdDecoderMIME.DecodeToString(S);
Memo1.Lines.Text := S;
finally
SS.Free;
IdDecoderMIME.Free;
end;
end;
I'm using this code:
procedure TForm1.FormCreate(Sender: TObject);
var
SS : TStringStream;
MS : TMemoryStream;
Output : String;
begin
SS := TStringStream.Create('');
MS := TMemoryStream.Create;
try
AdoQuery1.Open;
TBlobField(AdoQuery1.Fields[0]).SaveToStream(SS);
SS.WriteString(#13#10);
Output := SS.DataString;
SS.Position := 0;
MS.CopyFrom(SS, SS.Size);
MS.SaveToFile(ExtractFilePath(Application.ExeName) + 'Blob.txt');
finally
SS.Free;
MS.Free;
end;
end;
A hex dump of the Blob.Txt file looks like this
00000000 44 05 61 00 75 00 5F 00 69 00 64 00 44 08 61 00 D.a.u._.i.d.D.a.
00000010 75 00 5F 00 6C 00 6E 00 61 00 6D 00 65 00 44 08 u._.l.n.a.m.e.D.
00000020 61 00 75 00 5F 00 66 00 6E 00 61 00 6D 00 65 00 a.u._.f.n.a.m.e.
00000030 44 05 70 00 68 00 6F 00 6E 00 65 00 44 07 61 00 D.p.h.o.n.e.D.a.
00000040 64 00 64 00 72 00 65 00 73 00 73 00 44 04 63 00 d.d.r.e.s.s.D.c.
00000050 69 00 74 00 79 00 44 05 73 00 74 00 61 00 74 00 i.t.y.D.s.t.a.t.
00000060 65 00 44 03 7A 00 69 00 70 00 44 08 63 00 6F 00 e.D.z.i.p.D.c.o.
00000070 6E 00 74 00 72 00 61 00 63 00 74 00 44 07 61 00 n.t.r.a.c.t.D.a.
00000080 75 00 74 00 68 00 6F 00 72 00 73 00 01 0A 02 01 u.t.h.o.r.s.....
00000090 10 E4 04 00 00 0B 00 31 37 32 2D 33 32 2D 31 31 .......172-32-11
000000A0 37 36 02 02 10 E4 04 00 00 05 00 57 68 69 74 65 76.........White
000000B0 02 03 10 E4 04 00 00 07 00 4A 6F 68 6E 73 6F 6E .........Johnson
000000C0 02 04 0D E4 04 00 00 0C 00 34 30 38 20 34 39 36 .........408 496
000000D0 2D 37 32 32 33 02 05 10 E4 04 00 00 0F 00 31 30 -7223.........10
000000E0 39 33 32 20 42 69 67 67 65 20 52 64 2E 02 06 10 932 Bigge Rd....
000000F0 E4 04 00 00 0A 00 4D 65 6E 6C 6F 20 50 61 72 6B ......Menlo Park
00000100 02 07 0D E4 04 00 00 02 00 43 41 02 08 0D E4 04 .........CA.....
As you can see, some of it is legible (field names and contents), some of it not. Does anyone recognise this format and know how to clean it up into the plain text I get from executing the same query in SS Management Studio, i.e. how do I successfully extract the XML from the result set?
Btw, I get the same result (including the contents of the Blob.Txt file) using both the MS OLE DB Provider for Sql Server and the Sql Server Native Client 11 provider, and using Delphi Seattle in place of D7.
Given that the code accesses an external database, this code is the closest I can get to an MCVE.
Update #2 The decoding problem vanishes if I change the Sql query to
select Convert(Text,
(select * from authors where au_lname = 'White' for xml AUTO
))
which gives the result (in SS) of
<authors au_id="172-32-1176" au_lname="White" au_fname="Johnson" phone="408 496-7223" address="10932 Bigge Rd." city="Menlo Park" state="CA" zip="94025" contract="1"/>
but I'm still interested to know how to get this to work without needing the Convert(). I've noticed that if I remove the Where clause from the Sql, what is returned is not well-formed XML - it contains a series of nodes, one per data row, but there is no enclosing root node.
Also btw, I realise that I can avoid this problem by not using "For XML Auto", I'm just interested in how to do it correctly. Also, I don't need any help parsing the XML once I've managed to extract it.
Add the TYPE Directive to specify that you want XML returned.
select *
from Authors
where au_lname = 'White'
for xml auto, type
You can't simply decode the binary blob into XML.
You can use TADOCommand and direct its output stream to an XML document object e.g.:
const
adExecuteStream = 1024;
var
xmlDoc, RecordsAffected: OleVariant;
cmd: TADOCommand;
xmlDoc := CreateOleObject('MSXML2.DOMDocument.3.0'); // or CoDomDocument30.Create;
xmlDoc.async := False;
cmd := TADOCommand.Create(nil);
// specify your connection string
cmd.ConnectionString := 'Provider=SQLOLEDB;Data Source=(local);...';
cmd.CommandType := cmdText;
cmd.CommandText := 'select top 1 * from items for xml auto';
cmd.Properties['Output Stream'].Value := xmlDoc;
cmd.Properties['XML Root'].Value := 'RootNode';
cmd.CommandObject.Execute(RecordsAffected, EmptyParam, adExecuteStream);
xmlDoc.save('d:\test.xml');
cmd.Free;
This results a well-formed XML with enclosing root node RootNode.