C Program Written in VS2012 Works w/ Win7/8/2008R2/2012, but not 2003/XP/32bit? - c

I have to start by saying that I am very much a programming noob. I do not understand all the compiler options or nuances of the IDE, not by a longshot. But I am trying to teach myself more about native programming languages. (I'm decent with C#, but that is much easier than C as I am discovering.)
Today, I wrote this small program in C. It is a console/command line program. I used Visual Studio 2012 and my development machine alternates between Windows 7 and 8, 64 bit. To start, what I did was create a new VC++ project, and I chose a Blank Project. Then I created a new app.c file. I also created a *.rc file to give the executable some extra properties like "File Version" and "Company Name" when you browse the file properties in Windows Explorer. Then I went to the properties of the project, chose Configuration Properties -> C/C++ -> Code Generation and I changed Runtime Library to "Multi-threaded (/MT) so that I wouldn't have to distribute the msvcr100.dll file along with my executable.
In the app.c file, I placed the following code:
#include <stdio.h>
#include <string.h>
#include <Windows.h>
#include <WtsApi32.h>
#pragma comment(lib, "WtsApi32.lib")
void main(int argc, char *argv[])
{
char *helpMsg = "blah";
char *hostName, *connState = "";
char *addrFamily = "";
HANDLE hHost = NULL;
...stuff and so forth and so on...
}
Then I built/compiled the program, and the executable works just fine on Windows 7, 8, Server 2008R2, Server 2012, all 64 bit. But when I try to run the program on Server 2003 (and I am guessing WinXP, etc., as well,) I am greeted with the Windows dialog box:
"Foo.exe is not a valid Win32 application."
So my question is, is there something obvious/simple that I am missing that will allow this executable to also work on earlier XP/2003/32bit platforms that I am missing? I do not believe that I am using any 64-bit exclusive features in my program. But I figured that since I did choose "Blank Project" instead of "Win32 Console Application" that I may be missing some setting.
Edit: Here is the dumpbin.exe /headers output when run against my exe:
Microsoft (R) COFF/PE Dumper Version 11.00.50727.1
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file C:\users\me\Release\foo.exe
PE signature found
File Type: EXECUTABLE IMAGE
FILE HEADER VALUES
14C machine (x86)
5 number of sections
50F604BC time date stamp Tue Jan 15 19:39:08 2013
0 file pointer to symbol table
0 number of symbols
E0 size of optional header
102 characteristics
Executable
32 bit word machine
OPTIONAL HEADER VALUES
10B magic # (PE32)
11.00 linker version
7800 size of code
A200 size of initialized data
0 size of uninitialized data
16A7 entry point (004016A7) _mainCRTStartup
1000 base of code
9000 base of data
400000 image base (00400000 to 00414FFF)
1000 section alignment
200 file alignment
6.00 operating system version
0.00 image version
6.00 subsystem version
0 Win32 version
15000 size of image
400 size of headers
0 checksum
3 subsystem (Windows CUI)
8140 DLL characteristics
Dynamic base
NX compatible
Terminal Server Aware
100000 size of stack reserve
1000 size of stack commit
100000 size of heap reserve
1000 size of heap commit
0 loader flags
10 number of directories
0 [ 0] RVA [size] of Export Directory
D374 [ 3C] RVA [size] of Import Directory
11000 [ 538] RVA [size] of Resource Directory
0 [ 0] RVA [size] of Exception Directory
0 [ 0] RVA [size] of Certificates Directory
12000 [ C04] RVA [size] of Base Relocation Directory
9160 [ 38] RVA [size] of Debug Directory
0 [ 0] RVA [size] of Architecture Directory
0 [ 0] RVA [size] of Global Pointer Directory
0 [ 0] RVA [size] of Thread Storage Directory
CF98 [ 40] RVA [size] of Load Configuration Directory
0 [ 0] RVA [size] of Bound Import Directory
9000 [ 118] RVA [size] of Import Address Table Directory
0 [ 0] RVA [size] of Delay Import Directory
0 [ 0] RVA [size] of COM Descriptor Directory
0 [ 0] RVA [size] of Reserved Directory
SECTION HEADER #1
.text name
7670 virtual size
1000 virtual address (00401000 to 0040866F)
7800 size of raw data
400 file pointer to raw data (00000400 to 00007BFF)
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
60000020 flags
Code
Execute Read
SECTION HEADER #2
.rdata name
49E2 virtual size
9000 virtual address (00409000 to 0040D9E1)
4A00 size of raw data
7C00 file pointer to raw data (00007C00 to 0000C5FF)
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
40000040 flags
Initialized Data
Read Only
Debug Directories
Time Type Size RVA Pointer
-------- ------ -------- -------- --------
50F604BC cv 61 0000CFE0 BBE0 Format: RSDS, {582D0FF2-59C1-4633-AF2A-E4A4AD6BFA2C}, 1, C:\Users\me\Release\users.pdb
50F604BC feat 10 0000D044 BC44 Counts: Pre-VC++ 11.00=0, C/C++=116, /GS=116, /sdl=0
SECTION HEADER #3
.data name
2C04 virtual size
E000 virtual address (0040E000 to 00410C03)
E00 size of raw data
C600 file pointer to raw data (0000C600 to 0000D3FF)
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
C0000040 flags
Initialized Data
Read Write
SECTION HEADER #4
.rsrc name
538 virtual size
11000 virtual address (00411000 to 00411537)
600 size of raw data
D400 file pointer to raw data (0000D400 to 0000D9FF)
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
40000040 flags
Initialized Data
Read Only
SECTION HEADER #5
.reloc name
235C virtual size
12000 virtual address (00412000 to 0041435B)
2400 size of raw data
DA00 file pointer to raw data (0000DA00 to 0000FDFF)
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
42000040 flags
Initialized Data
Discardable
Read Only
Summary
3000 .data
5000 .rdata
3000 .reloc
1000 .rsrc
8000 .text
I have also tried going to Project Properties -> Linker -> System: Minimum Required Version and changing that to 5.00 and 1.00 or whatever, but it has no effect. dumpbin.exe still reports the OS version as 6.00. I have even used editbin.exe /version 5.00 on the exe and no errors were reported... and yet dumpbin.exe still reports 6.00 for the OS version.

VS2012 originally shipped without supporting XP/2003. The updated CRT and runtime support libraries are using too many Windows api functions that are not available on those operating systems. This created quite a stir among its customers, to put it mildly, and they re-engineered the libraries to dynamically bind to these functions and limp along it they are missing. This was made available in Update 1, you'll need to use Project + Properties, General, Platform Toolset = v110_xp to build programs that use those libraries.
Note how it changes a linker setting, the important one, Linker > System > Minimum Required Version = "5.01". Which ensures that the executable file is marked to be compatible with the XP sub-system version. You'll also build against SDK version 7.1, the last one that is still compatible with XP.
When you use the default toolset (v110) then you target sub-system 6.00 and SDK version 8. Version 6.00 was the last major kernel revision, started with Vista.
A brief overview of the new api functions being used to give you a (very rough) idea what is missing in the XP version:
FlsAlloc, FlsFree, FlsGetValue, FlsSetValue : safe thread-local storage
InitializeCriticalSectionEx, CreateSemaphoreEx : safety
SetThreadStackGuarantee : stability
CreateThreadPoolTimer, SetThreadPoolTimer, WaitForThreadPoolTimerCallbacks, CloseThreadPoolTimer : cheaper timers
CreateThreadPoolWait, SetThreadPoolWait, CloseThreadPoolWait : cheaper waits?
FlushProcessWriteBuffers, GetCurrentProcessorNumber, GetLogicalProcessorInformation : threading
FreeLibraryWhenCallbackReturns : stability?
CreateSymbolicLink : functionality
InitOnceExecuteOnce : unknown
SetDefaultDllDirectories : unknown
EnumLocalesEx, CompareStringEx, GetDateFormatEx, GetLocalInfoEx, GetTimeFormatEx, GetUserDefaultLocaleName, IsValidLocaleName, LCMapStringEx : better locale support

I figured it out myself. (But thank you Hans for steering me in the right direction.) For some reason, even with Update 1 and even after setting my toolset to v110_xp, and setting the minimum required version to 5.01 in the Linker options, the resulting dumpbin app.exe /headers still reports a minimum operating system version of 6.0.
So I simply ran
editbin.exe app.exe /SUBSYSTEM:CONSOLE,5.01 /OSVERSION:5.1
And the executable now runs just fine on older operating systems. I'm thinking there still might be a little bit of a bug somewhere in Visual Studio.

The MSVC Team Blog says that when using MSBuild or DEVENV from the command-line with the v110_xp platform toolset, no other changes are necessary. This information is incorrect/incomplete. The /SUBSYSTEM linker argument and associated "Minimum Required Version" must also be set appropriately.
The MSDN documentation for /ENTRY states that, if the /SUBSYSTEM argument is not specified that the SUBSYSTEM and ENTRY POINT are determined automatically. My hunch is that when this happens, the SUBSYSTEM's "Minimum Required Version" argument is also automatically overridden.
The v110_xp toolset automatically specifies the SUBSYSTEM's MRV ("5.1" (WindowsXP)) but not the SUBSYSTEM. As such, the MRV will be overridden, for example, by the linker to "6.0". Running the application will then cause WindowsXP to show the error message stating that the application "is not a valid Win32 application."

Related

where is uboot environment variables stored in emmc?

I'm trying to understand where uboot stores the environment variables in emmc. I have the following set in the uboot config file -
CONFIG_ENV_IS_NOWHERE=y
CONFIG_ENV_IS_IN_MMC=y
CONFIG_SYS_MMC_ENV_DEV=2
CONFIG_SYS_MMC_ENV_PART=0
CONFIG_ENV_SIZE=0x4000
CONFIG_ENV_OFFSET=0x400000
CONFIG_ENV_SECT_SIZE=0x10000
uboot is able to save the env variables in emmc and I can read them from Linux. Variables set from Linux are also readable in uboot. I fail to understand which partition has the environment variables?
Output lsblk from Linux -
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mtdblock0 31:0 0 16M 0 disk
mmcblk2 179:0 0 7.3G 0 disk
|-mmcblk2p1 179:1 0 83.2M 0 part /run/media/mmcblk2p1
|-mmcblk2p2 179:2 0 1.7G 0 part /
|-mmcblk2p3 179:3 0 83.2M 0 part /run/media/mmcblk2p3
|-mmcblk2p4 179:4 0 1K 0 part
|-mmcblk2p5 179:5 0 1.7G 0 part /run/media/mmcblk2p5
`-mmcblk2p6 179:6 0 1G 0 part /run/media/mmcblk2p6
mmcblk2boot0 179:32 0 4M 1 disk
mmcblk2boot1 179:64 0 4M 1 disk
mmcblk2boot0 is where uboot is located which is 4MB in size, considering the environment is saved at an offset of 0x400000 in mmcblk2 device, it should be located just after mmcblk2boot0 which points to the parition mmcblk2boot1, but if I dump the strings "strings /dev/mmcblk2boot1", I get nothing.
I have provided a file /etc/fw_config which is used by fw_printenv and fw_saveenv, this file contains "/dev/mmcblk2 0x400000 0x10000". All the settings point to the fact that uboot environment is located at an offset of 0x400000 in mmcblk2 device. Any pointers on which partition listed in output of lsblk holds the uboot environment variables..?
Thanks,
Vinay
The confusing thing here is that CONFIG_SYS_MMC_ENV_PART does not refer to sofware partitions, such as /dev/mmcblk2p1 that you see under Linux, but rather hardware partitions such as /dev/mmcblk2 (partition 0) or /dev/mmcblk2boot0 (partition 1).

Reverse enginering a Linux based USB camera

I bought an IP camera on which is installed proprietary software (no HTTP server).
This prevents me to integrate it into my home network.
I want to replace the software (ELF closed source) by the motion package I already use and add some features.
I have no particular system competence and it's been over a week since I travel the net to learn but I can not get out.
I have access to the U-boot console (USB-TTL adapter) and telnet (root).
The webcam has a SD card reader that I could use if I need space.
I started by making a backup of the three partitions (with dd).
I unzipped the file mtdblock2 (binwalk -e). Which generates a classical Linux tree with links to Busybox, some binary system and proprietary software.
I tried to unzip mtdblock1 which generates zImage.
The decompression zImage generates two directories and one file (console).
Yet I need the kernel modules that are in it. What to do?
I also want to get the kernel compilation settings, is this possible?
I unpacked the firmware available on the manufacturer's website.
It contains only updating the ELF, one .so file and some Bash scripts.
At first I thought the three partitions directly migrate to Qemu.
But if I understand this is not possible because the memory addresses are hard-coded into the kernel.
I understand good?
So I think I have one solution: build a new kernel and rebuild a rootfs from scratch.
Is this only solution?
I started playing with Buildroot but I can not find the configuration file for board based on Hisilicon Hi3518.
I looked bad or is it useless?
For my first test I used board/qemu/arm-versatile. This is the right choice?
This will not prevent me from migrating to the physical machine?
For testing, if I managed to rebuild a kernel and rootfs I would install these partitions on the SD not to break anything.
For this, it is "sufficient" to modify kernel parameters (in bootargs variable) is that right?
So I don't need to rebuild a U-boat partition for my device?
In short, you guessed I ask myself a lot of questions (yet others but "one" thing at a time).
I need advice about whether I take the right road.
Please, if I am talking nonsense feel free to correct me.
If you have ideas or subjects of reflection I'm interested.
# cat /proc/cpuinfo
Processor : ARM926EJ-S rev 5 (v5l)
BogoMIPS : 218.72
Features : swp half thumb fastmult edsp java
CPU implementer : 0x41
CPU architecture: 5TEJ
CPU variant : 0x0
CPU part : 0x926
CPU revision : 5
Hardware : hi3518
Revision : 0000
Serial : 0000000000000000
# cat /proc/mtd
dev: size erasesize name
mtd0: 00100000 00010000 "boot"
mtd1: 00300000 00010000 "kernel"
mtd2: 00c00000 00010000 "rootfs"
# binwalk mtdblock0
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
122044 0x1DCBC CRC32 polynomial table, little endian
# binwalk mtdblock1
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
0 0x0 uImage header, header size: 64 bytes, header CRC: 0x853F419E, created: 2014-07-22 02:45:04, image size: 2890840 bytes, Data Address: 0x80008000, Entry Point: 0x80008000, data CRC: 0xB24E77CA, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: none, image name: "Linux-3.0.8"
22608 0x5850 gzip compressed data, maximum compression, from Unix, NULL date:
# binwalk zImage
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
113732 0x1BC44 ASCII cpio archive (SVR4 with no CRC), file name: "dev", file name length: "0x00000004", file size: "0x00000000"
113848 0x1BCB8 ASCII cpio archive (SVR4 with no CRC), file name: "dev/console", file name length: "0x0000000C", file size: "0x00000000"
113972 0x1BD34 ASCII cpio archive (SVR4 with no CRC), file name: "root", file name length: "0x00000005", file size: "0x00000000"
114088 0x1BDA8 ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"
1903753 0x1D0C89 Certificate in DER format (x509 v3), header length: 4, sequence length: 1284
4188800 0x3FEA80 Linux kernel version "3.0.8 (cwen#ubuntu) (gcc version 4.4.1 (Hisilicon_v100(gcc4.4-290+uclibc_0.9.32.1+eabi+linuxpthread)) ) #1 Tue Jul 22 10:45:00 H"
4403540 0x433154 CRC32 polynomial table, little endian
5053435 0x4D1BFB Unix path: /mtd/devices/hisfc350/hisfc350_spi_gd25qxxx.c
5054731 0x4D210B Unix path: /mtd/devices/hisfc350/hisfc350.c
5058939 0x4D317B Unix path: /net/wireless/rt2x00/rt2x00dev.c
5059323 0x4D32FB Unix path: /net/wireless/rt2x00/rt2x00config.c
5060683 0x4D384B Unix path: /net/wireless/rt2x00/rt2x00usb.c
5060851 0x4D38F3 Unix path: /net/wireless/rt2x00/rt2x00.h
5061171 0x4D3A33 Unix path: /net/wireless/rt2x00/rt73usb.c
5081107 0x4D8813 Unix path: /S70/S75/505V/F505/F707/F717/P8
5102399 0x4DDB3F Unix path: /mmc/host/himciv100/himci.c
5141264 0x4E7310 Neighborly text, "NeighborSolicits/ipv6/inet6_hashtables.c"
5141284 0x4E7324 Neighborly text, "NeighborAdvertisementses.c"
# binwalk mtdblock2
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
0 0x0 JFFS2 filesystem, little endian
722980 0xB0824 JFFS2 filesystem, little endian
732282 0xB2C7A Zlib compressed data, compressed
737031 0xB3F07 Zlib compressed data, compressed
738287 0xB43EF Zlib compressed data, compressed
.... most other lines in the same genre
IP Camera QQZM N5063
http://www.zmvideo.com/product/detail.php?id=60
Firmware
http://bbs.zmmcu.com/forum.php?mod=attachment&aid=MzU2fDBiY2M4NDdjfDE0MTkxMTEzODl8MzQ4fDIwMzc%3D
First of all, you do not want to replace U-Boot as this may render your device unbootable. On the U-Boot console, check if you can boot from the SD card mmc rescan 0; fatload mmc 0 ${loadaddr} uImage or from the network dhcp ${loadaddr} ${serverip}:uImage. You'll need to look for documentation for these commands to get more help.
But perhaps you don't even need to replace the kernel. You already know it's a 3.0.8 kernel, so you can build a userspace for this kernel version. And any proprietary modules that are used by it can be lifted from the jffs2 filesystem. On your telnet session, do lsmod to find out which modules are loaded. You can mount an SD card and copy them to it. The modules are located in /lib/modules/3.0.8.
So you probably don't even need to build a kernel in buildroot, only the rootfs. First, check in the telnet session which filesystems are supported: cat /proc/filesystems. Then choose the appropriate filesystem in the buildroot configuration. For the target architecture, choose arm926t. And select the 3.0 kernel headers in the toolchain configuration, or choose the Arago ARMv5 2011.09 external toolchain (it has old kernel headers).
As remarked by artless noise, you don't need to test it in qemu, since the SD card is safe.

Lua unable to load VC++ dll

I have compiled my litertools project as a .dll using the Visual Studio C/C++ toolchain and it correctly exports the lua_openlitertools function:
PS> dumpbin /exports .\bin\lib\litertools.dll
Microsoft (R) COFF/PE Dumper Version 14.00.22013.1
Dump of file .\bin\lib\litertools.dll
Section contains the following exports for litertools.dll
00000000 characteristics
5439D35E time date stamp Sun Oct 12 12:03:26 2014
0.00 version
1 ordinal base
1 number of functions
1 number of names
ordinal hint RVA name
1 0 000227AD luaopen_litertools = #ILT+1960(_luaopen_litertools)
Summary
1000 .data
2000 .idata
6000 .rdata
2000 .reloc
1000 .rsrc
44000 .text
21000 .textbss
When I set configure package.cpath and then require the litertools dll by doing:
CMD> lua -e "package.cpath=\".\\bin\\lib\\?.dll\"; require(\"litertools\")"
I receive the following error
lua: error loading module 'litertools' from file '.\bin\lib\litertools.dll':
The specified module could not be found.
stack traceback:
[C]: in ?
[C]: in function 'require'
(command line):1: in main chunk
[C]: in ?
Looking into it, I debugged the call through require (loadlib.c), and got to here:
static void *ll_load (lua_State *L, const char *path, int seeglb) {
HMODULE lib = LoadLibraryExA(path, NULL, LUA_LLE_FLAGS);
(void)(seeglb); /* not used: symbols are 'global' by default */
if (lib == NULL) pusherror(L);
return lib;
}
The call to LoadLibraryExA returns NULL. I am unsure as to why this happens, though I have compiled the same library using mingw-gcc and was able to require the dll.
I figured out that dependency was not being found through using:
http://www.dependencywalker.com/
Move the unit tests into a separate project and corrected solution settings.

In a compressed PE must the virtual size of the data section match the raw size?

In working with a compressed PE (Windows console EXE) that has a file alignment and section alignment of 4 bytes, I notice that if virtual size and raw size of the sections match, then the program loads, but if virtual size of the data section, the last section, does not match then Windows refuses to load it, even though by the specification you should be able to have a virtual size larger than a raw size.
Is this some kind of hidden constraint on compressed PEs?
I have pasted a dumpbin /headers of the exe below:
Microsoft (R) COFF/PE Dumper Version 10.00.30319.01
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file ba42x.exe
PE signature found
File Type: EXECUTABLE IMAGE
FILE HEADER VALUES
14C machine (x86)
2 number of sections
50AABC14 time date stamp Mon Nov 19 18:09:08 2012
0 file pointer to symbol table
0 number of symbols
60 size of optional header
10F characteristics
Relocations stripped
Executable
Line numbers stripped
Symbols stripped
32 bit word machine
OPTIONAL HEADER VALUES
10B magic # (PE32)
2.03 linker version
BD0 size of code
5000 size of initialized data
0 size of uninitialized data
CC entry point (004000CC)
CC base of code
C9C base of data
400000 image base (00400000 to 00403FFF)
4 section alignment
4 file alignment
4.00 operating system version
0.00 image version
4.00 subsystem version
0 Win32 version
4000 size of image
CC size of headers
0 checksum
3 subsystem (Windows CUI)
0 DLL characteristics
10000 size of stack reserve
1000 size of stack commit
0 size of heap reserve
0 size of heap commit
0 loader flags
0 number of directories
SECTION HEADER #1
.text name
BD0 virtual size
CC virtual address (004000CC to 00400C9B)
BD0 size of raw data
CC file pointer to raw data (000000CC to 00000C9B)
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
E0000020 flags
Code
Execute Read Write
SECTION HEADER #2
.data name
3102 virtual size
C9C virtual address (00400C9C to 00403D9D)
3102 size of raw data
C9C file pointer to raw data (00000C9C to 00003D9D)
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
C0000040 flags
Initialized Data
Read Write
Summary
3104 .data
BD0 .text
For example if you change the virtual size of the above .data section to 3106 the program will not load, even though the size of initialized data (0x5000) is more than enough to accomodate the additional memory.
No, there are not special constraints related to compressed images, since as long as your image is PE compliant, the loader does not care about the compression. Compression is handled by the stub, not the loader.
Can you provide your image for further analysis?
Just by looking at the output of dumpbin, the image looks unusual..There are no directory at all, pretty strange. It looks like the issue with the loader is not directly related to the alignement, but malformation of the image file. Did you try to have a look at your image file using other PE tools (e.g. PeStudio, CFF Explorer..?)

How do I create a sparse file programmatically, in C, on Mac OS X?

I'd like to create a sparse file such that all-zero blocks don't take up actual disk space until I write data to them. Is it possible?
There seems to be some confusion as to whether the default Mac OS X filesystem (HFS+) supports holes in files. The following program demonstrates that this is not the case.
#include <stdio.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
void create_file_with_hole(void)
{
int fd = open("file.hole", O_WRONLY|O_TRUNC|O_CREAT, 0600);
write(fd, "Hello", 5);
lseek(fd, 99988, SEEK_CUR); // Make a hole
write(fd, "Goodbye", 7);
close(fd);
}
void create_file_without_hole(void)
{
int fd = open("file.nohole", O_WRONLY|O_TRUNC|O_CREAT, 0600);
write(fd, "Hello", 5);
char buf[99988];
memset(buf, 'a', 99988);
write(fd, buf, 99988); // Write lots of bytes
write(fd, "Goodbye", 7);
close(fd);
}
int main()
{
create_file_with_hole();
create_file_without_hole();
return 0;
}
The program creates two files, each 100,000 bytes in length, one of which has a hole of 99,988 bytes.
On Mac OS X 10.5 on an HFS+ partition, both files take up the same number of disk blocks (200):
$ ls -ls
total 400
200 -rw------- 1 user staff 100000 Oct 10 13:48 file.hole
200 -rw------- 1 user staff 100000 Oct 10 13:48 file.nohole
Whereas on CentOS 5, the file without holes consumes 88 more disk blocks than the other:
$ ls -ls
total 136
24 -rw------- 1 user nobody 100000 Oct 10 13:46 file.hole
112 -rw------- 1 user nobody 100000 Oct 10 13:46 file.nohole
As in other Unixes, it's a feature of the filesystem. Either the filesystem supports it for ALL files or it doesn't. Unlike Win32, you don't have to do anything special to make it happen. Also unlike Win32, there is no performance penalty for using a sparse file.
On MacOS, the default filesystem is HFS+ which does not support sparse files.
Update: MacOS used to support UFS volumes with sparse file support, but that has been removed. None of the currently supported filesystems feature sparse file support.
This thread becomes a comprehensive source of info about the sparse files. Here is the missing part for Win32:
Decent article with examples
Tool that estimates if it makes sense to make file as sparse
Regards
hdiutil can handle sparse images and files but unfortunately the framework it links against is private.
You could try defining external symbols as defined by the DiskImages framework below but this is most likely not acceptable for production code, plus since the framework is private you'd have to reverse engineer its use cases.
cristi:~ diciu$ otool -L /usr/bin/hdiutil
/usr/bin/hdiutil:
/System/Library/PrivateFrameworks/DiskImages.framework/Versions/A/DiskImages (compatibility version 1.0.8, current version 194.0.0)
[..]
cristi:~ diciu$ nm /System/Library/PrivateFrameworks/DiskImages.framework/Versions/A/DiskImages | awk -F' ' '{print $3}' | c++filt | grep -i sparse
[..]
CSparseFile::sector2Band(long long)
CSparseFile::addIndexNode()
CSparseFile::readIndexNode(long long, SparseFileIndexNode*)
CSparseFile::readHeaderNode(CBackingStore*, SparseFileHeaderNode*, unsigned long)
[... cut for brevity]
Later Edit
You could use hdiutil as an external process and have it create an sparse disk image for you. From the C process you would then create a file in the (mounted) sparse disk image.
If you seek (fseek, ftruncate, ...) to past the end, the file size will be increased without allocating blocks until you write to the holes. But there's no way to create a magic file that automatically converts blocks of zeroes to holes. You have to do it yourself.
This may be helpful to look at (the OpenBSD cp command inserts holes instead of writing zeroes).
patch
If you want portability, the last resort is to write your own access function so that you manage an index and a set of blocks.
In essence you manage a single file as the OS manages the disk keeping the chain of the blocks that are part of the file, the bitmap of allocated/free blocks etc.
Of course this will lead to a non optimized and slower access, I would reccomend this apprach only if the requirement to save space is absolutely critical and you have enough time to write a robust set of access functions.
And even in that case, I would first investigate if your problem is in need of a different solution. Probably you should store your data differently?
It looks like OS X supports sparse files on UDF volumes. I tried titaniumdecoy's test program on OS X 10.9 and it did generate a sparse file on a UDF disk image. Also, not that UFS is no longer supported in OS X, so if you need sparse files, UDF is the only natively supported file system that supports them.
I also tried the program on SMB shares. When the server is Ubuntu (ext4 filesystem) the program creates a sparse file, but 'ls -ls' through SMB doesn't show that. If you do 'ls -ls' on the Ubuntu host itself it does show the file is sparse. When the server is Windows XP (NTFS filesystem) the program does not generate a sparse file.

Resources