When we use #pragma pack (push, 1) to pack the C structure and convert to go structure by cgo, some of the fields are missing in the converted Go structure.
I am using Go version: 1.16.6 Windows/386
For example, my C struct is like this:
#pragma pack (push, 1)
typedef struct _sample_struct
{
USHORT usNo;
DWORD ft;
DWORD fit;
CHAR cID[5];
CHAR cCID[3];
ULONG ulVal;
ULONG ulIn;
ULONG ulCnt;
ULONG ulMax;
USHORT usStat;
BOOL bAlk;
LPSTRUCT2 lpNNL;
USHORT usNPCU;
LPSTRUCT4 * lppP;
LPSTR lpBuff;
LPUSHORT lpusIDS;
WORD usType;
LPSTR lpszCUName;
ULONG ulICnt;
ULONG ulDCnt;
ULONG ulPCnt;
ULONG ulRCnt;
ULONG ulRJCnt;
ULONG ulMin;
} SAMPLESTRUCT, *LPSAMPLESTRUCT;
#pragma pack (pop)
Converted to Go, the structure looks like below:
type _Ctype_struct__sample_struct struct {
usNo _Ctype_USHORT
_ [8]byte
cID [5]_Ctype_CHAR
cCID [3]_Ctype_CHAR
_ [16]byte
usStat _Ctype_USHORT
bAlk _Ctype_BOOL
lpNNL _Ctype_LPSTRUCT2
usNPCU _Ctype_USHORT
_ [12]byte
usType _Ctype_WORD
lpszCUName _Ctype_LPSTR
ulICnt _Ctype_ULONG
ulDCnt _Ctype_ULONG
ulPCnt _Ctype_ULONG
ulRCnt _Ctype_ULONG
ulRJCnt _Ctype_ULONG
ulMin _Ctype_ULONG
}
As we can see, some of the fields are not converted properly; instead, it has "_".**
Conversion is correct without the #pragma pack lines; however, I need packing in place because the library that I use imposes packing for the structures that are sent to our application.
Is there any solution to this problem?
I actually wrote a whole blog post on this: https://medium.com/#liamkelly17/working-with-packed-c-structs-in-cgo-224a0a3b708b
The quick notes are:
Look at the Go wiki page on GitHub, in the cgo section they explicitly mention cgo will not pack structs
You can very easily pack the structs yourself in a similar to encoding a struct to JSON before sending it. By using the binary package you can iterate through struct members and place them into a []byte. The []byte can then be passed as a packed struct to C functions. (do the opposite to receive packed structs from c functions)
Related
The Microsoft documentation for NTFS describes the structure of an attribute. It shows it as follows:
typedef struct _ATTRIBUTE_RECORD_HEADER {
ATTRIBUTE_TYPE_CODE TypeCode;
ULONG RecordLength;
UCHAR FormCode;
UCHAR NameLength;
USHORT NameOffset;
USHORT Flags;
USHORT Instance;
union {
struct {
ULONG ValueLength;
USHORT ValueOffset;
UCHAR Reserved[2];
} Resident;
struct {
VCN LowestVcn;
VCN HighestVcn;
USHORT MappingPairsOffset;
UCHAR Reserved[6];
LONGLONG AllocatedLength;
LONGLONG FileSize;
LONGLONG ValidDataLength;
LONGLONG TotalAllocated;
} Nonresident;
} Form;
} ATTRIBUTE_RECORD_HEADER, *PATTRIBUTE_RECORD_HEADER;
The final member of a nonresident attribute, TotalAllocated, does not seem to exist. 3rd party documentation does not mention it, and actual NTFS filesystem do not contain such a member (the ValidDataLength is immediately followed by the the data runs, as specified in MappingPairsOffset.
From the documentation itself, it is supposed to record the total number of clusters (as opposed to the total number of bytes).
TotalAllocated
The total allocated for the file (the sum of the allocated clusters).
Does anyone recognize this?
The field actually does exist, but only on compressed files (i.e. bit 1 on the Flags field is set); The MappingPairsOffset is then 0x48 instead of the usual 0x40 to make room for the extra field.
This is mentioned in a footnote to 3rd party NTFS documentation here.
I am trying to design a data structure (I have made it much shorter to save space here but I think you get the idea) to be used for byte level communication:
/* PACKET.H */
#define CM_HEADER_SIZE 3
#define CM_DATA_SIZE 16
#define CM_FOOTER_SIZE 3
#define CM_PACKET_SIZE (CM_HEADER_SIZE + CM_DATA_SIZE + CM_FOOTER_SIZE)
// + some other definitions
typedef struct cm_header{
uint8_t PacketStart; //Start Indicator 0x5B [
uint8_t DeviceId; //ID Of the device which is sending
uint8_t PacketType;
} CM_Header;
typedef struct cm_footer {
uint16_t DataCrc; //CRC of the 'Data' part of CM_Packet
uint8_t PacketEnd; //should be 0X5D or ]
} CM_Footer;
//Here I am trying to conver a few u8[4] tp u32 (4*u32 = 16 byte, hence data size)
typedef struct cm_data {
union {
struct{
uint8_t Value_0_0:2;
uint8_t Value_0_1:2;
uint8_t Value_0_2:2;
uint8_t Value_0_3:2;
};
uint32_t Value_0;
};
//same thing for Value_1, 2 and 3
} CM_Data;
typedef struct cm_packet {
CM_Header Header;
CM_Data Data;
CM_Footer Footer;
} CM_Packet;
typedef struct cm_inittypedef{
uint8_t DeviceId;
CM_Packet Packet;
} CM_InitTypeDef;
typedef struct cm_appendresult{
uint8_t Result;
uint8_t Reason;
} CM_AppendResult;
extern CM_InitTypeDef cmHandler;
The goal here is to make reliable structure for transmitting data over USB interface. At the end the CM_Packet should be converted to an uint8_t array and be given to data transmit register of an mcu (stm32).
In the main.c file I try to init the structure as well as some other stuff related to this packet:
/* MAIN.C */
uint8_t packet[CM_PACKET_SIZE];
int main(void) {
//use the extern defined in packet.h to init the struct
cmHandler.DeviceId = 0x01; //assign device id
CM_Init(&cmHandler); //construct the handler
//rest of stuff
while(1) {
CM_GetPacket(&cmHandler, (uint8_t*)packet);
CDC_Transmit_FS(&packet, CM_PACKET_SIZE);
}
}
And here is the implementation of packet.h which screws up everything so bad. I added the packet[CM_PACKET_SIZE] to watch but it is like it is just being generated randomly. Sometimes by pure luck I can see in this array some of the values that I am interested in! but it is like 1% of the time!
/* PACKET.C */
CM_InitTypeDef cmHandler;
void CM_Init(CM_InitTypeDef *cm_initer) {
cmHandler.DeviceId = cm_initer->DeviceId;
static CM_Packet cmPacket;
cmPacket.Header.DeviceId = cm_initer->DeviceId;
cmPacket.Header.PacketStart = CM_START;
cmPacket.Footer.PacketEnd = CM_END;
cm_initer->Packet = cmPacket;
}
CM_AppendResult CM_AppendData(CM_InitTypeDef *handler, uint8_t identifier,
uint8_t *data){
CM_AppendResult result;
switch(identifier){
case CM_VALUE_0:
handler->Packet.Data.Value_0_0 = data[0];
handler->Packet.Data.Value_0_1 = data[1];
handler->Packet.Data.Value_0_2 = data[2];
handler->Packet.Data.Value_0_3 = data[3];
break;
//Also cases for CM_VALUE_0, 1 , 2
//to build up the CM_Data sturct of CM_Packet
default:
result.Result = CM_APPEND_FAILURE;
result.Reason = CM_APPEND_CASE_ERROR;
return result;
break;
}
result.Result = CM_APPEND_SUCCESS;
result.Reason = 0x00;
return result;
}
void CM_GetPacket(CM_InitTypeDef *handler, uint8_t *packet){
//copy the whole struct in the given buffer and later send it to USB host
memcpy(packet, &handler->Packet, sizeof(CM_PACKET_SIZE));
}
So, the problem is this code gives me 99% of the time random stuff. It never has the CM_START which is the start indicator of packet to the value I want to. But most of the time it has the CM_END byte correctly! I got really confused and cant find out the reason. Being working on an embedded platform which is hard to debugg I am kind of lost here...
If you transfer data to another (different) architecture, do not just pass a structure as a blob. That is the way to hell: endianess, alignment, padding bytes, etc. all can (and likely will) cause trouble.
Better serialize the struct in a conforming way, possily using some interpreted control stream so you do not have to write every field out manually. (But still use standard functions to generate that stream).
Some areas of potential or likely trouble:
CM_Footer: The second field might very well start at a 32 or 64 bit boundary, so the preceeding field will be followed by padding. Also, the end of that struct is very likely to be padded by at least 1 bytes on a 32 bit architecture to allow for proper alignment if used in an array (the compiler does not care you if you actually need this). It might even be 8 byte aligned.
CM_Header: Here you likely (not guaranteed) get one uint8_t with 4*2 bits with the ordering not standardized. The field my be followed by 3 unused bytes which are required for the uint32_t interprettion of the union.
How do you guarantee the same endianess (for >uint8_t: high byte first or low byte first?) for host and target?
In general, the structs/unions need not have the same layout for host and target. Even if the same compiler is used, their ABIs may differ, etc. Even if it is the same CPU, there might be other system constraints. Also, for some CPUs, different ABIs (application binary interface) exist.
I use Microsoft Visual Studio 2010, write in C++ and compile binary for x64 platform. In my project I have useful structure for memory blocks with data pointer and size:
typedef struct _MEMORY_BLOCK
{
DWORD64 Length;
LPBYTE lpData;
}
MEMORY_BLOCK, *LPMEMORY_BLOCK;
In other file I have key definition:
BYTE PublicKey[] = { 0x01, 0x02, 0x03, ... };
DWORD64 PublicKeyLength = (DWORD64)sizeof(PublicKey);
MEMORY_BLOCK ServerKey = { PublicKeyLength, PublicKey };
On x86 platform structure initialization of ServerKey works fine, but on x64 platform it results in MEMORY_BLOCK struct filled with zeroes. If I change members order in structure (e.g. lpData at first, and second is Length), lpData initializes properly, but Length is still equal to zero.
Now I have workaround with function that sets ServerKey values in runtime, but I need to know why I could not initialize DWORD64 struct member in ServerKey definition.
I'm trying to write some code dealing with a register interface. The register interface is generic and used all over the chip I'm working with. There are several different types of instantiations which use a subset of the register interface.
I want to know, is there some way in C to create a struct with the master register interface, then have the various subsets be structs that only can access their used registers?
So, for example, let's call the different possible subsets A, B, and C.
The register interface may look something like this:
OFFSET NAME NEEDED BY
0x00 -- CFG A B C
0x04 -- VERSION A B C
0x08 -- SIZE B C
0x0C -- TYPE A C
0x10 -- USER A
So, using the same format that ARM uses for their SCS register interface, I can create the following struct for master:
typedef volatile struct {
uint32 cfg;
uint32 version;
uint32 size;
uint32 type;
uint32 user;
} master_t;
That part's relatively straightforward. But I'm having trouble thinking of a way to implement the a_t, b_t, and c_t types to have their contents still point to the right offsets, short of just creating separate structs in the exact same method as above (which would make it extremely painful to propagate changes/bugfixes; the actual use case has more like 50 total registers for master_t).
Essentially, my end goal is that if you pointed an a_t to the right memory address, then tried to write code that would (e.g.) try to access a_t.size, the compiler would throw an error. Is there any way this is possible in C?
Perhaps like this:
typedef struct
{
uint32 cfg;
uint32 version;
uint32 filler;
uint32 type;
uint32 user;
} SUB_A_T;
typedef struct
{
uint32 cfg;
uint32 version;
uint32 size;
} SUB_B_T;
typedef struct
{
uint32 cfg;
uint32 version;
uint32 size;
uint32 type;
} SUB_C_T;
typedef UNION
{
SUB_A_T a;
SUB_B_T b;
SUB_C_T c;
} MASTER_T;
To access:
MASTER_T master;
master.a.cfg = 1;
master.a.version = 2;
master.a.type = 3;
master.a.user = 4;
// master.a.size = 11 This line would cause a compiler error.
Or...
master.b.cfg = 10;
master.b.version = 20;
master.b.size = 30;
// master.b.type = 22 This line would cause a compiler error.
// master.b.user = 33 This line would cause a compiler error.
Or...
master.c.cfg = 100;
master.c.version = 200;
master.c.size = 300;
master.c.type = 400;
// master.c.user = 44 This line would cause a compiler error.
I'm currently working on a C-project where I should hide text in an .bmp image.
Therefore I open an image and write the file header and the info header in two structures:
typedef struct _BitmapFileHeader_
{
uint16_t bfType_;
uint32_t bfSize_;
uint32_t bfReserved_;
uint32_t bfOffBits_;
}
__attribute__((packed))
BitmapFileHeader;
typedef struct _BitmapInfoHeader_
{
uint32_t biSize_;
int32_t biWidth_;
int32_t biHeight_;
uint16_t biPlanes_;
uint16_t biBitCount_;
uint32_t biCompression_;
uint32_t biSizeImage_;
int32_t biXPelsPerMeter_;
int32_t biYPelsPerMeter_;
uint32_t biClrUsed_;
uint32_t biClrImportant_;
}BitmapInfoHeader;
BitmapFileHeader* bitmap_headerdata = NULL;
BitmapInfoHeader* bitmap_infodata = NULL;
filename is defined previously
int readBitmapFile (char* filename, BitmapFileHeader** bitmap_headerdata,
BitmapInfoHeader** bitmap_infodata, unsigned char** bitmap_colordata)
{
FILE* bmp_file;
bmp_file = fopen(filename, "rb");
fseek(bmp_file, 0, SEEK_SET); // Set File Cursor to beginning
fread((*bitmap_headerdata), sizeof(**bitmap_headerdata), 1, bmp_file);
fseek(bmp_file, sizeof(**bitmap_headerdata), SEEK_SET);
fread((*bitmap_infodata), sizeof(**bitmap_infodata), 1, bmp_file);
int checkinfo = sizeof(**bitmap_infodata);
int checkheader = sizeof(**bitmap_headerdata);
printf("Size of Infodata: %d\nSize of Headerdata: %d\n", checkinfo, checkheader);
....
}
When I open a valid Bitmap (24bit, not compressed) and I compare the values bfType_, biBitCount and biCompression to 19778,24,0 on Linux it works just fine but when I try to run it on Windows the Program stops when it compares biBitCount to 24.
When I debugged the program I noticed that all the Values from "bitmap_infodata" are one line above from where they should be (when I look at it like a table).
Then I compared sizeof(**bitmap_headerdata) on Linux and on Windows and noticed that it's 14 on Linux and 16 on Windows?
Shouldn't that be the same? And why does the structure bitmap_headerdata have same values on both OS but bitmap_infodata is different?
Bernhard
The problem is that structs are padded differently in different environments.
There are 2 solutions to the problem.
1: Read the header field by field.
2: Remove struct padding. The syntax to do that varies. Some compilers use #PRAGMA PACK. You are using __attribute__((__packed__)) which apparantly doesn't work on both platforms.
The packed attribute is broken in mingw gcc:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=52991
You might want to consider using:
#pragma pack(1)
struct BitmapFileHeader {
...
};
#pragma pack()
And -mno-ms-bitfields flag.