The Microsoft documentation for NTFS describes the structure of an attribute. It shows it as follows:
typedef struct _ATTRIBUTE_RECORD_HEADER {
ATTRIBUTE_TYPE_CODE TypeCode;
ULONG RecordLength;
UCHAR FormCode;
UCHAR NameLength;
USHORT NameOffset;
USHORT Flags;
USHORT Instance;
union {
struct {
ULONG ValueLength;
USHORT ValueOffset;
UCHAR Reserved[2];
} Resident;
struct {
VCN LowestVcn;
VCN HighestVcn;
USHORT MappingPairsOffset;
UCHAR Reserved[6];
LONGLONG AllocatedLength;
LONGLONG FileSize;
LONGLONG ValidDataLength;
LONGLONG TotalAllocated;
} Nonresident;
} Form;
} ATTRIBUTE_RECORD_HEADER, *PATTRIBUTE_RECORD_HEADER;
The final member of a nonresident attribute, TotalAllocated, does not seem to exist. 3rd party documentation does not mention it, and actual NTFS filesystem do not contain such a member (the ValidDataLength is immediately followed by the the data runs, as specified in MappingPairsOffset.
From the documentation itself, it is supposed to record the total number of clusters (as opposed to the total number of bytes).
TotalAllocated
The total allocated for the file (the sum of the allocated clusters).
Does anyone recognize this?
The field actually does exist, but only on compressed files (i.e. bit 1 on the Flags field is set); The MappingPairsOffset is then 0x48 instead of the usual 0x40 to make room for the extra field.
This is mentioned in a footnote to 3rd party NTFS documentation here.
Related
When we use #pragma pack (push, 1) to pack the C structure and convert to go structure by cgo, some of the fields are missing in the converted Go structure.
I am using Go version: 1.16.6 Windows/386
For example, my C struct is like this:
#pragma pack (push, 1)
typedef struct _sample_struct
{
USHORT usNo;
DWORD ft;
DWORD fit;
CHAR cID[5];
CHAR cCID[3];
ULONG ulVal;
ULONG ulIn;
ULONG ulCnt;
ULONG ulMax;
USHORT usStat;
BOOL bAlk;
LPSTRUCT2 lpNNL;
USHORT usNPCU;
LPSTRUCT4 * lppP;
LPSTR lpBuff;
LPUSHORT lpusIDS;
WORD usType;
LPSTR lpszCUName;
ULONG ulICnt;
ULONG ulDCnt;
ULONG ulPCnt;
ULONG ulRCnt;
ULONG ulRJCnt;
ULONG ulMin;
} SAMPLESTRUCT, *LPSAMPLESTRUCT;
#pragma pack (pop)
Converted to Go, the structure looks like below:
type _Ctype_struct__sample_struct struct {
usNo _Ctype_USHORT
_ [8]byte
cID [5]_Ctype_CHAR
cCID [3]_Ctype_CHAR
_ [16]byte
usStat _Ctype_USHORT
bAlk _Ctype_BOOL
lpNNL _Ctype_LPSTRUCT2
usNPCU _Ctype_USHORT
_ [12]byte
usType _Ctype_WORD
lpszCUName _Ctype_LPSTR
ulICnt _Ctype_ULONG
ulDCnt _Ctype_ULONG
ulPCnt _Ctype_ULONG
ulRCnt _Ctype_ULONG
ulRJCnt _Ctype_ULONG
ulMin _Ctype_ULONG
}
As we can see, some of the fields are not converted properly; instead, it has "_".**
Conversion is correct without the #pragma pack lines; however, I need packing in place because the library that I use imposes packing for the structures that are sent to our application.
Is there any solution to this problem?
I actually wrote a whole blog post on this: https://medium.com/#liamkelly17/working-with-packed-c-structs-in-cgo-224a0a3b708b
The quick notes are:
Look at the Go wiki page on GitHub, in the cgo section they explicitly mention cgo will not pack structs
You can very easily pack the structs yourself in a similar to encoding a struct to JSON before sending it. By using the binary package you can iterate through struct members and place them into a []byte. The []byte can then be passed as a packed struct to C functions. (do the opposite to receive packed structs from c functions)
There is a function to set the "valid data length" value: SetFileValidData, but I didn't find a way to get the "valid data length" value.
I want to know about given file if the EOF is different from the VDL, because writing after the VDL in case of VDL<EOF will cause a performance penalty as described here.
I found this page, claims that:
there is no mechanism to query the value of the VDL
So the answer is "you can't".
If you care about performance you can set the VDL to the EOF, but then note that you may allow access old garbage on your disk - the part between those two pointers, that supposed to be zeros if you would access that file without setting the VDL to point the EOF.
Looked into this. No way to get this information via any API, even the e.g. NtQueryInformationFile API (FileEndOfFileInformation only worked with NtSetInformationFile). So finally I read this by manually reading NTFS records. If anyone has a better way, please tell! This also obviously only works with full system access (and NTFS) and might be out of sync with the in-memory information Windows uses.
#pragma pack(push)
#pragma pack(1)
struct NTFSFileRecord
{
char magic[4];
unsigned short sequence_offset;
unsigned short sequence_size;
uint64 lsn;
unsigned short squence_number;
unsigned short hardlink_count;
unsigned short attribute_offset;
unsigned short flags;
unsigned int real_size;
unsigned int allocated_size;
uint64 base_record;
unsigned short next_id;
//char padding[470];
};
struct MFTAttribute
{
unsigned int type;
unsigned int length;
unsigned char nonresident;
unsigned char name_lenght;
unsigned short name_offset;
unsigned short flags;
unsigned short attribute_id;
unsigned int attribute_length;
unsigned short attribute_offset;
unsigned char indexed_flag;
unsigned char padding1;
//char padding2[488];
};
struct MFTAttributeNonResident
{
unsigned int type;
unsigned int lenght;
unsigned char nonresident;
unsigned char name_length;
unsigned short name_offset;
unsigned short flags;
unsigned short attribute_id;
uint64 starting_vnc;
uint64 last_vnc;
unsigned short run_offset;
unsigned short compression_size;
unsigned int padding;
uint64 allocated_size;
uint64 real_size;
uint64 initial_size;
};
#pragma pack(pop)
HANDLE GetVolumeData(const std::wstring& volfn, NTFS_VOLUME_DATA_BUFFER& vol_data)
{
HANDLE vol = CreateFileW(volfn.c_str(), GENERIC_WRITE | GENERIC_READ,
FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
if (vol == INVALID_HANDLE_VALUE)
return vol;
DWORD ret_bytes;
BOOL b = DeviceIoControl(vol, FSCTL_GET_NTFS_VOLUME_DATA,
NULL, 0, &vol_data, sizeof(vol_data), &ret_bytes, NULL);
if (!b)
{
CloseHandle(vol);
return INVALID_HANDLE_VALUE;
}
return vol;
}
int64 GetFileValidData(HANDLE file, HANDLE vol, const NTFS_VOLUME_DATA_BUFFER& vol_data)
{
BY_HANDLE_FILE_INFORMATION hfi;
BOOL b = GetFileInformationByHandle(file, &hfi);
if (!b)
return -1;
NTFS_FILE_RECORD_INPUT_BUFFER record_in;
record_in.FileReferenceNumber.HighPart = hfi.nFileIndexHigh;
record_in.FileReferenceNumber.LowPart = hfi.nFileIndexLow;
std::vector<BYTE> buf;
buf.resize(sizeof(NTFS_FILE_RECORD_OUTPUT_BUFFER) + vol_data.BytesPerFileRecordSegment - 1);
NTFS_FILE_RECORD_OUTPUT_BUFFER* record_out = reinterpret_cast<NTFS_FILE_RECORD_OUTPUT_BUFFER*>(buf.data());
DWORD bout;
b = DeviceIoControl(vol, FSCTL_GET_NTFS_FILE_RECORD, &record_in,
sizeof(record_in), record_out, 4096, &bout, NULL);
if (!b)
return -1;
NTFSFileRecord* record = reinterpret_cast<NTFSFileRecord*>(record_out->FileRecordBuffer);
unsigned int currpos = record->attribute_offset;
MFTAttribute* attr = nullptr;
while ( (attr==nullptr ||
attr->type != 0xFFFFFFFF )
&& record_out->FileRecordBuffer + currpos +sizeof(MFTAttribute)<buf.data() + bout)
{
attr = reinterpret_cast<MFTAttribute*>(record_out->FileRecordBuffer + currpos);
if (attr->type == 0x80
&& record_out->FileRecordBuffer + currpos + attr->attribute_offset+sizeof(MFTAttributeNonResident)
< buf.data()+ bout)
{
if (attr->nonresident == 0)
return -1;
MFTAttributeNonResident* dataattr = reinterpret_cast<MFTAttributeNonResident*>(record_out->FileRecordBuffer
+ currpos + attr->attribute_offset);
return dataattr->initial_size;
}
currpos += attr->length;
}
return -1;
}
[...]
NTFS_VOLUME_DATA_BUFFER vol_data;
HANDLE vol = GetVolumeData(L"\\??\\D:", vol_data);
if (vol != INVALID_HANDLE_VALUE)
{
int64 vdl = GetFileValidData(alloc_test->getOsHandle(), vol, vol_data);
if(vdl>=0) { [...] }
[...]
}
[...]
The SetValidData (according to MSDN) can be used to create for example a large file without having to write to the file. For a database this will allocate a (contiguous) storage area.
As a result, it seems the file size on disk will have changed without any data having been written to the file.
By implication, any GetValidData (which does not exist) just returns the size of the file, so you can use GetFileSize which returns the "valid" file size.
I think you are confused as to what "valid data length" actually means. Check this answer.
Basically, while SetEndOfFile lets you increase the length of a file quickly, and allocates the disk space, if you skip to the (new) end-of-file to write there, all the additionally allocated disk space would need to be overwritten with zeroes, which is kind of slow.
SetFileValidData lets you skip that zeroing-out. You're telling the system, "I am OK with whatever is in those disk blocks, get on with it". (This is why you need the SE_MANAGE_VOLUME_NAME priviledge, as it could reveal priviledged data to unpriviledged users if you don't overwrite the data. Users with this priviledge can access the raw drive data anyway.)
In either case, you have set the new effective size of the file. (Which you can read back.) What, exactly, should a seperate "read file valid data" report back? SetFileValidData told the system that whatever is in those disk blocks is "valid"...
Different approach of explanation:
The documentation mentions that the "valid data length" is being tracked; the purpose for this is for the system to know which range (from end-of-valid-data to end-of-file) it still needs to zero out, in the context of SetEndOfFile, when necessary (e.g. you closing the file). You don't need to read back this value, because the only way it could be different from the actual file size is because you, yourself, did change it via the aforementioned functions...
I am trying to design a data structure (I have made it much shorter to save space here but I think you get the idea) to be used for byte level communication:
/* PACKET.H */
#define CM_HEADER_SIZE 3
#define CM_DATA_SIZE 16
#define CM_FOOTER_SIZE 3
#define CM_PACKET_SIZE (CM_HEADER_SIZE + CM_DATA_SIZE + CM_FOOTER_SIZE)
// + some other definitions
typedef struct cm_header{
uint8_t PacketStart; //Start Indicator 0x5B [
uint8_t DeviceId; //ID Of the device which is sending
uint8_t PacketType;
} CM_Header;
typedef struct cm_footer {
uint16_t DataCrc; //CRC of the 'Data' part of CM_Packet
uint8_t PacketEnd; //should be 0X5D or ]
} CM_Footer;
//Here I am trying to conver a few u8[4] tp u32 (4*u32 = 16 byte, hence data size)
typedef struct cm_data {
union {
struct{
uint8_t Value_0_0:2;
uint8_t Value_0_1:2;
uint8_t Value_0_2:2;
uint8_t Value_0_3:2;
};
uint32_t Value_0;
};
//same thing for Value_1, 2 and 3
} CM_Data;
typedef struct cm_packet {
CM_Header Header;
CM_Data Data;
CM_Footer Footer;
} CM_Packet;
typedef struct cm_inittypedef{
uint8_t DeviceId;
CM_Packet Packet;
} CM_InitTypeDef;
typedef struct cm_appendresult{
uint8_t Result;
uint8_t Reason;
} CM_AppendResult;
extern CM_InitTypeDef cmHandler;
The goal here is to make reliable structure for transmitting data over USB interface. At the end the CM_Packet should be converted to an uint8_t array and be given to data transmit register of an mcu (stm32).
In the main.c file I try to init the structure as well as some other stuff related to this packet:
/* MAIN.C */
uint8_t packet[CM_PACKET_SIZE];
int main(void) {
//use the extern defined in packet.h to init the struct
cmHandler.DeviceId = 0x01; //assign device id
CM_Init(&cmHandler); //construct the handler
//rest of stuff
while(1) {
CM_GetPacket(&cmHandler, (uint8_t*)packet);
CDC_Transmit_FS(&packet, CM_PACKET_SIZE);
}
}
And here is the implementation of packet.h which screws up everything so bad. I added the packet[CM_PACKET_SIZE] to watch but it is like it is just being generated randomly. Sometimes by pure luck I can see in this array some of the values that I am interested in! but it is like 1% of the time!
/* PACKET.C */
CM_InitTypeDef cmHandler;
void CM_Init(CM_InitTypeDef *cm_initer) {
cmHandler.DeviceId = cm_initer->DeviceId;
static CM_Packet cmPacket;
cmPacket.Header.DeviceId = cm_initer->DeviceId;
cmPacket.Header.PacketStart = CM_START;
cmPacket.Footer.PacketEnd = CM_END;
cm_initer->Packet = cmPacket;
}
CM_AppendResult CM_AppendData(CM_InitTypeDef *handler, uint8_t identifier,
uint8_t *data){
CM_AppendResult result;
switch(identifier){
case CM_VALUE_0:
handler->Packet.Data.Value_0_0 = data[0];
handler->Packet.Data.Value_0_1 = data[1];
handler->Packet.Data.Value_0_2 = data[2];
handler->Packet.Data.Value_0_3 = data[3];
break;
//Also cases for CM_VALUE_0, 1 , 2
//to build up the CM_Data sturct of CM_Packet
default:
result.Result = CM_APPEND_FAILURE;
result.Reason = CM_APPEND_CASE_ERROR;
return result;
break;
}
result.Result = CM_APPEND_SUCCESS;
result.Reason = 0x00;
return result;
}
void CM_GetPacket(CM_InitTypeDef *handler, uint8_t *packet){
//copy the whole struct in the given buffer and later send it to USB host
memcpy(packet, &handler->Packet, sizeof(CM_PACKET_SIZE));
}
So, the problem is this code gives me 99% of the time random stuff. It never has the CM_START which is the start indicator of packet to the value I want to. But most of the time it has the CM_END byte correctly! I got really confused and cant find out the reason. Being working on an embedded platform which is hard to debugg I am kind of lost here...
If you transfer data to another (different) architecture, do not just pass a structure as a blob. That is the way to hell: endianess, alignment, padding bytes, etc. all can (and likely will) cause trouble.
Better serialize the struct in a conforming way, possily using some interpreted control stream so you do not have to write every field out manually. (But still use standard functions to generate that stream).
Some areas of potential or likely trouble:
CM_Footer: The second field might very well start at a 32 or 64 bit boundary, so the preceeding field will be followed by padding. Also, the end of that struct is very likely to be padded by at least 1 bytes on a 32 bit architecture to allow for proper alignment if used in an array (the compiler does not care you if you actually need this). It might even be 8 byte aligned.
CM_Header: Here you likely (not guaranteed) get one uint8_t with 4*2 bits with the ordering not standardized. The field my be followed by 3 unused bytes which are required for the uint32_t interprettion of the union.
How do you guarantee the same endianess (for >uint8_t: high byte first or low byte first?) for host and target?
In general, the structs/unions need not have the same layout for host and target. Even if the same compiler is used, their ABIs may differ, etc. Even if it is the same CPU, there might be other system constraints. Also, for some CPUs, different ABIs (application binary interface) exist.
I use Microsoft Visual Studio 2010, write in C++ and compile binary for x64 platform. In my project I have useful structure for memory blocks with data pointer and size:
typedef struct _MEMORY_BLOCK
{
DWORD64 Length;
LPBYTE lpData;
}
MEMORY_BLOCK, *LPMEMORY_BLOCK;
In other file I have key definition:
BYTE PublicKey[] = { 0x01, 0x02, 0x03, ... };
DWORD64 PublicKeyLength = (DWORD64)sizeof(PublicKey);
MEMORY_BLOCK ServerKey = { PublicKeyLength, PublicKey };
On x86 platform structure initialization of ServerKey works fine, but on x64 platform it results in MEMORY_BLOCK struct filled with zeroes. If I change members order in structure (e.g. lpData at first, and second is Length), lpData initializes properly, but Length is still equal to zero.
Now I have workaround with function that sets ServerKey values in runtime, but I need to know why I could not initialize DWORD64 struct member in ServerKey definition.
I'm currently working on a C-project where I should hide text in an .bmp image.
Therefore I open an image and write the file header and the info header in two structures:
typedef struct _BitmapFileHeader_
{
uint16_t bfType_;
uint32_t bfSize_;
uint32_t bfReserved_;
uint32_t bfOffBits_;
}
__attribute__((packed))
BitmapFileHeader;
typedef struct _BitmapInfoHeader_
{
uint32_t biSize_;
int32_t biWidth_;
int32_t biHeight_;
uint16_t biPlanes_;
uint16_t biBitCount_;
uint32_t biCompression_;
uint32_t biSizeImage_;
int32_t biXPelsPerMeter_;
int32_t biYPelsPerMeter_;
uint32_t biClrUsed_;
uint32_t biClrImportant_;
}BitmapInfoHeader;
BitmapFileHeader* bitmap_headerdata = NULL;
BitmapInfoHeader* bitmap_infodata = NULL;
filename is defined previously
int readBitmapFile (char* filename, BitmapFileHeader** bitmap_headerdata,
BitmapInfoHeader** bitmap_infodata, unsigned char** bitmap_colordata)
{
FILE* bmp_file;
bmp_file = fopen(filename, "rb");
fseek(bmp_file, 0, SEEK_SET); // Set File Cursor to beginning
fread((*bitmap_headerdata), sizeof(**bitmap_headerdata), 1, bmp_file);
fseek(bmp_file, sizeof(**bitmap_headerdata), SEEK_SET);
fread((*bitmap_infodata), sizeof(**bitmap_infodata), 1, bmp_file);
int checkinfo = sizeof(**bitmap_infodata);
int checkheader = sizeof(**bitmap_headerdata);
printf("Size of Infodata: %d\nSize of Headerdata: %d\n", checkinfo, checkheader);
....
}
When I open a valid Bitmap (24bit, not compressed) and I compare the values bfType_, biBitCount and biCompression to 19778,24,0 on Linux it works just fine but when I try to run it on Windows the Program stops when it compares biBitCount to 24.
When I debugged the program I noticed that all the Values from "bitmap_infodata" are one line above from where they should be (when I look at it like a table).
Then I compared sizeof(**bitmap_headerdata) on Linux and on Windows and noticed that it's 14 on Linux and 16 on Windows?
Shouldn't that be the same? And why does the structure bitmap_headerdata have same values on both OS but bitmap_infodata is different?
Bernhard
The problem is that structs are padded differently in different environments.
There are 2 solutions to the problem.
1: Read the header field by field.
2: Remove struct padding. The syntax to do that varies. Some compilers use #PRAGMA PACK. You are using __attribute__((__packed__)) which apparantly doesn't work on both platforms.
The packed attribute is broken in mingw gcc:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=52991
You might want to consider using:
#pragma pack(1)
struct BitmapFileHeader {
...
};
#pragma pack()
And -mno-ms-bitfields flag.