When I am sending a DNS query to the DNS it returns the header with the format bit set.
Indicating there is a problem with the format, but I am failing to see what it is. Its possible I have misinterpreted the RFC, or misread it but right now I cant seem to work it out.
The DNS structure I am sending looks like this in hex.
Header
00 01 - ID = 1
01 00 - RD = 1
00 01 - QD = 1
00 00 - AN
00 00 - NS
00 00 - NR
Question for www.google.com
03 77 - 3 w
77 77 - w w
06 67 - 6 g
6f 6f - o o
67 6c - g l
65 03 - e 3
63 6f - c o
6d 00 - m 0
00 01 - QTYPE
00 01 - QCLASS
I then flip the bytes for any field that is two bytes, to convert to big endian for the network format. So each row of the header, and then QTYPE and QCLASS ...
Here's what a byte-by-byte hexdump of that query packet should look like (tested and working!):
00000000 00 01 01 00 00 01 00 00 00 00 00 00 03 77 77 77 |.............www|
00000010 06 67 6f 6f 67 6c 65 03 63 6f 6d 00 00 01 00 01 |.google.com.....|
I think your problem is that the third and fourth bytes of the packet (flags and rcode) are two single-byte fields, not one 2-byte field - it looks like you might be treating it as a 16 bit integer and swapping the bytes?
To get these you can use netcat and dig.
# nc –uip 53 > dnsreqdump
# dig www.example.com #localhost
# nc –u 8.8.8.8 53 <dnsreqdump >dnsrespdump
Now you can inspect them in hexedit or your favorite hex editor.
I tend to think that your problem depends on how are you actually "flipping the bits to convert to network format".
Typical C library implementations provide the htonl() function family to do the conversion from host into network order and viceversa.
Of course, without seeing the code, I cannot be sure that this is the problem.
Related
I was trying to interpret the bytes in a DBus Message as specified in https://dbus.freedesktop.org/doc/dbus-specification.html. This is taken from a pcap while using the Frida tool.
The bytes are
0000 6c 01 00 01 08 00 00 00 01 00 00 00 70 00 00 00
0010 01 01 6f 00 15 00 00 00 2f 72 65 2f 66 72 69 64
0020 61 2f 48 6f 73 74 53 65 73 73 69 6f 6e 00 00 00
0030 02 01 73 00 16 00 00 00 72 65 2e 66 72 69 64 61
0040 2e 48 6f 73 74 53 65 73 73 69 6f 6e 31 35 00 00
0050 08 01 67 00 05 61 7b 73 76 7d 00 00 00 00 00 00
0060 03 01 73 00 17 00 00 00 47 65 74 46 72 6f 6e 74
0070 6d 6f 73 74 41 70 70 6c 69 63 61 74 69 6f 6e 00
0080 00 00 00 00 00 00 00 00
There are some fields which I am uncertain what they mean. Appreciate if anyone can provide some guidance on this.
0x6C: Refers to little endian
0x01: Message Type (Method Call)
0x00: Bitwise of OR flags
0x01: Major Protocol Version
0x08000000: Length of Message Body (Little Endian), starting from end of Header. This should be referring to the eight null bytes at the end?
0x01000000: Serial of this Message (Little Endian)
0x70000000: (Little Endian) Not sure what this represents? This value does correspond to the length of the array of struct, excluding trailing null bytes, that starts from 0x0010 and ends at 0x007F.
0x01: Decimal Code for Object Path
0x01: Not sure what this represents?
0x6F: DBus Type 'o' for Object Path
0x15: Length of the Object Path string
You want to look at the part of the specification that tells you what the message format is.
But to answer your questions:
0x08000000: Length of Message Body (Little Endian), starting from end of Header. This should be referring to the eight null bytes at the end?
Correct.
0x70000000: (Little Endian) Not sure what this represents? This value does correspond to the length of the array of struct, excluding trailing null bytes, that starts from 0x0010 and ends at 0x007F.
That's the length of the array in the header. The DBus header is of a variable size - after the first few bytes, it is an array of struct(byte,variant). As per the documentation, that looks like a(yv) if you were to express this as a DBus type signature.
0x01: Decimal Code for Object Path
0x01: Not sure what this represents?
This is where the parsing gets interesting: in our struct, the signature is yv, so the first 0x01 is telling us that this struct entry is the header field for Object Path, as you have seen. However, we now need to parse what the variant contains inside of it. To marshal a variant, you first marshal a signature, which in this case is 1 byte long: 01 6f 00. Note that signatures can be a max of 255 bytes long, so unlike other strings they only have a 1-byte length at the front. As a string, that is o, which tells us that this variant contains an object path inside of it. Since object paths are strings, we then decode the next bytes as a string(keeping note that the leading 4 bytes are the string length): 15 00 00 00 2f 72 65 2f 66 72 69 64 61 2f 48 6f 73 74 53 65 73 73 69 6f 6e 00
If I've done the conversion correctly, that says /re/frida/HostSession
This is taken from a pcap
If it's a standard pcap (or pcapng) D-Bus capture file, using the LINKTYPE_DBUS link-layer type, then Wireshark should be able to read it and, at least to some degree, interpret the messages (i.e., it has code that understands the message format, as defined by the specification to which #rm5248 referred (and to which the LINKTYPE_DBUS entry in the list of link-layer header types refers), so you might not have to interpret all the bytes by yourself.
TL;DR: I want to read raw h264 streams from AVI/MP4 files, even broken/incomplete.
Almost every document about h264 tells me that it consists of NAL packets. Okay. Almost everywhere told to me that the packet should start with a signature like 00 00 01 or 00 00 00 01. For example, https://stackoverflow.com/a/18638298/8167678, https://stackoverflow.com/a/17625537/8167678
The format of H.264 is that it’s made up of NAL Units, each starting
with a start prefix of three bytes with the values 0x00, 0x00, 0x01
and each unit has a different type depending on the value of the 4th
byte right after these 3 starting bytes. One NAL Unit IS NOT one frame
in the video, each frame is made up of a number of NAL Units.
Okay.
I downloaded random_youtube_video.mp4 and strip out one frame from it:
ffmpeg -ss 10 -i random_youtube_video.mp4 -frames 1 -c copy pic.avi
And got:
Red part - this is part of AVI container, other - actual data.
As you can see, here I have 00 00 24 A9 instead of 00 00 00 01
This AVI file plays perfectly
I do same for mp4 container:
As you can see, here exact same bytes.
This MP4 file plays perfectly
I try to strip out raw data:
ffmpeg -i pic.avi -c copy pic.h264
This file can't play in VLC or even ffmpeg, which produced this file, can't parse it:
I downloaded mp4 stream analyzer and got:
MP4Box tells me:
Cannot find H264 start code
Error importing pic.h264: BitStream Not Compliant
It very hard to learn internals of h264, when nothing works.
So, I have questions:
What actual data inside mp4?
What I must read to decode that data (I mean different annex-es)
How to read stream and get decoded image (even with ffmpeg) from this "broken" raw stream?
UPDATE:
It seems bug in ffmpeg:
When I do double conversion:
ffmpeg -ss 10 -i random_youtube_video.mp4 -frames 1 -c copy pic.mp4
ffmpeg pic.mp4 -c copy pic.h264
But when I convert file directly:
ffmpeg -ss 10 -i random_youtube_video.mp4 -frames 1 -c copy pic.h264
I have NALs signatures and one extra NAL unit. Other bytes are same (selected).
This is bug?
UPDATE
Not, this is not bug, U must use option -bsf h264_mp4toannexb to save stream as "Annex B" format (with prefixes)
"I want to read raw h264 streams from AVI files, even broken/incomplete."
"Almost everywhere told to me that the packet should start with a signature like : 00 00 01 or 00 00 00 01"
"...As you can see, here I have 00 00 24 A9 instead of 00 00 00 01"
Your H264 is in AVCC format which means it uses data sizes (instead of data start codes). It is only Annex-B that will have your mentioned signature as start code.
You seek frames, not by looking for start codes, but instead you just do skipping by frame sizes to reach the final correct offset of a (requested) frame...
AVI processing :
Read size (four) bytes (32-bit integer, Little Endian).
Extract the next following bytes up to size amount.
This is your H.264 frame (in AVCC format), decode the bytes to view image.
To convert into Annex-B, try replacing first 4 bytes of H.264 frame bytes with 00 00 00 01.
Consider your shown AVI bytes (see first picture) :
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00 00 00 00 4C 49 53 54 BA 24 00 00 6D 6F 76 69 ....LISTº$..movi
30 30 64 63 AD 24 00 00 00 00 24 A9 65 88 84 27 00dc.$....$©eˆ„'
C7 11 FE B3 C7 83 08 00 08 2A 7B 6E 59 B5 71 E1 Ç.þ³Çƒ...*{nYµqá
E3 9C 0E 73 E7 10 50 00 18 E9 25 F7 AA 7D 9C 30 ãœ.sç.P..é%÷ª}œ0
E6 2F 0F 20 00 3A 64 AA CA 5E 4F CA FF AE 20 04 æ/. .:dªÊ^OÊÿ® .
07 81 40 00 48 00 0A 28 71 21 84 48 06 18 90 0C ..#.H..(q!„H....
31 14 57 9E 7A CD 63 A0 E0 9B 96 69 C5 18 AE F2 1.WžzÍc à›–iÅ.®ò
E6 07 02 29 01 20 10 70 A1 0F 8C BC 73 F0 78 FA æ..). .p¡.Œ¼sðxú
9E 1D E1 C2 BF 8C 62 CE CE AC 14 5A A4 E1 45 44 ž.á¿ŒbÎά.Z¤áED
38 38 85 DB 12 57 3E F6 E0 FB AE 03 04 21 62 8D 88…Û.W>öàû®..!b.
F6 F1 1E 37 1C A2 FF 75 1C F1 02 66 0C 92 07 06 öñ.7.¢ÿu.ñ.f.’..
15 7C 90 15 6F 7D FC BD 13 1E 2B 0C 14 3C 0C 00 .|..o}ü½..+..<..
B0 EA 6F 53 B4 98 D7 80 7A 68 3E 34 69 20 D2 FA °êoS´˜×€zh>4i Òú
F0 91 FC 75 C6 00 01 18 C0 00 3B 9A C5 E2 7D BF ð‘üuÆ...À.;šÅâ}¿
Some explanation :
Ignore leading multiple 00 bytes.
4C 49 53 54 D6 3C 00 00 6D 6F 76 69 including 30 30 64 63 = AVI "List" header.
AD 24 00 00 == decimal 9389 is AVI's own size of H264 item (must read in Little Endian).
Notice that the AVI bytes include...
- a note of item's total size (AD 24 00 00... or reverse for Little Endian : 00 00 24 AD)
- followed by item data (00 00 24 A9 65 88 84 27 ... etc ... C5 E2 7D BF).
This size includes both the 4 bytes of the AVI's"size" entry + expected bytes length of the item's own bytes. Can be written simply as:
AVI_Item_Size = ( 4 + item_H264_Frame.length );
H.264 video frame bytes in AVI :
Next follows the item data, which is the H.264 video frame. By sheer coincidence of formats/bytes layout, it too holds a 4-byte entry for data's size (since your H264 is in AVCC format, if it was Annex-B then you would be seeing start code bytes here instead of size bytes).
Unlike AVI bytes, these H264 size bytes are written in Big Endian format.
00 00 24 A9 = size of bytes for this video frame (instead of start code : 00 00 00 01).
65 88 84 27 C7 11 FE B3 C7 = H.264 keyframe (always begins X5, where the X value is based on other settings).
Remember after four size bytes (or even start codes) if followed by...
byte X5 = keyframe (IDR), example byte 65.
byte X1 = P or B frame, example byte 41.
byte X6 = SEI (Supplemental Enhancement Information).
byte X7 = SPS (Sequence Parameter Set).
byte X8 = PPS (Picture Parameter Set).
bytes 00 00 00 X9 = Access unit delimiter.
You can find the H.264 if you search for exact same bytes within AVI file. See third picture, these are your H.264 bytes (they are cut & pasted into the AVI container).
Sometimes a frame is sliced into different NAL units. So if you extract a key frame and it only shows 1/2 or 1/3 instead of full image, just grab next one or two NAL and re-try the decode.
I´m playing around with two Xbees, one defined as coordinator, another as router. I want to read information about the network interoperably so i decided to use the ZDO messages.
I send a message like this ((profile ID 0x00 00, cluster ID 0x 00 31) and receive for example the following response from the router:
7E 00 2D 91 00 13 A2 00 40 E5 F0 B4 FB CE 00 00 80 31 00 00 01 2C 00 01 00 01 58 CE C1 8D 7A 3F 2D 40 AB F0 E5 40 00 A2 13 00 00 00 04 02 00 FF 33
Correct answer cluster ID: 0x 80 31
Focussing on the RF Data i have the following:
2C 00 01 00 01 58 CE C1 8D 7A 3F 2D 40 AB F0 E5 40 00 A2 13 00 00 00 04 02 00 FF
I now try to decode this hex string and face some problems.
From my point of view, this string should be encoded like defined within the ZigBee Spec from 2012, at Table 2.126 and 2.127
Unfortunately this don´t work for me. If i ignore, that the first byte should be the status and take the first two of them, i can read out NeighborTableEntries, StartIndex, NeighborTabelListCount. But when it comes to the NeighTableList i only can read out the Extended PAN id, the extended address and the network address, the rest of the string does not fit to the standard. Am i doing something wrong here or does the xbee´s don´t stick to the standard?
2C = Sequence Number
00 = Status (Success)
01 = 1 entry (total)
00 = starting at index 0
01 = 1 entry (in packet)
58 CE C1 8D 7A 3F 2D 40 = Extended Pan ID
AB F0 E5 40 00 A2 13 00 = IEEE address
00 00 = NodeId
04 = (Coordinator, RxOnWhenIdle)
02 = (Unknown Permit Join)
00 = (Coordinator)
FF = (LQI)
The values after the NodeId are bitmasks, not bytes.
The 2006 version of the free offline Chinese sentence dictionary Jukuu contains a collection of 100,000 publicly sourced example sentences in Chinese and English in a .dll file.
The application size is about 80mb, but once installed a 500mb DLL dictionary file is created with the source text. For whatever reason the application doesn't run on my computer, and I'd like to extract all the example sentences so I can do some POS analysis on them.
Opening the 500mb .DLL file is mostly gibberish, except for some fragments of text here and there and references to other resources.
I'm wondering if there is any way I can extract the information in plain text?
The application can be downloaded here: http://www.jukuu.com/down/download.html
Thanks!
Edit: Never-mind, It looks like when viewed in HEX, the file is ordered in a way that is not conducive to sentence mining at all:
00 06 00 02 00 07 03 00 01 00 00 00 FF FE 6E 65 76 65 72 7E 73 74 61
6E 64 20 75 70 0B 00 00 80 00 00 00 00 00 00 00 00 FF FE 33 30 32
32 31 38 32 36 38 2D 00 16 00 06 00 02 00 07 03 00 01 00 00 00 FF
FE 65 78 74 72 65 6D 65 6C 79 7E 63 6C 6F 73 65 0B 00 00 80 00 00
00 00 00 00 00
Something like ÿþnever~stand upÿþ302218268-ÿþextremely~close
Any other ideas on how to mine sentences from the application? Maybe a batch script?
I'm trying to read the content of a Mifare Ultralight card using the NFC Reader Library.
I'm totally new with NFC and I'm using this github repository to start.
The code in this repo allows to detect which type of card is detected (Mifare, Mifare ultralight ...) and read the UID of the card. I added this code in order to read the content of a Mifare ultralight card:
uint8_t bBufferReader[96];
memset(bBufferReader, '\0', 0x60);
PH_CHECK_SUCCESS_FCT(status, phalMful_Read(&alMful, 4, bBufferReader));
int i;
for(i = 0; i < 96; i++){
printf("%02X", bBufferReader[i]);
}
I have a card that contains the text "Hello world" and when I read it, the piece of code above print the following bytes:
0103A010440312D1010E5402667248650000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
So I'm reading some stuff out of my card, however there is no traces of a "Hello world"
text.
Probably missing something (might be a big something or, hopefully, a little something). Any help would be great !
Edit
So I made some good progress. Mifare ultralight contains 16 pages of 4 bytes, 0 - 3 are for internal usage (serial number, lock etc ...) and 4 - 15 are for user data. I can now read the content of my cards, however, just a few question remains:
I'm reading a card that contains an URL, www.google.com, here is what I got:
03 0F D1 01 -> Page 1, 4 bytes of non text data, not sure what it is
0B 55 01 67 -> Page 2, 3 bytes of non text data, then 1 bytes for the "g"
6F 6F 67 6C -> Page 3, 4 bytes for "oogl"
65 2E 63 6F -> Page 4, 4 bytes for "e.co"
6D FE 00 00 -> Page 6, 1 byte for "m", 1 byte for I don't know
00 00 00 00 -> Other pages are just empty
00 00 00 00
00 00 00 00
00 00 00 00
00 00 00 00
00 00 00 00
00 00 00 00
So I have got 7 bytes of data + my url, "google.com" + 1 byte FE
I can't find what are these 7 prefix bytes + this 1 trailing byte...
Edit again
Ok got it, it's the NDEF message format.
Yes it is NDEF format!
03 NDEF Message
0F length
Record 1
D1 - MB, ME, SR, TNF=”NFC Forum well-known type”
01 Type length
0B Payload length
55 Type - “U”(Abbrivation for URL)
67 6F 6F 67 6C 65 2E 63 6F 6D (google.com)
Record 2
FE Terminator NDEF