File IO in the apache portable runtime library - c

While working through Zed Shaw's learn C the Hard Way, I encountered the function apr_dir_make_recursive() which according to the documentation here has the type signature
apr_status_t apr_dir_make_recursive(const char *path, apr_fileperms_t perm, apr_pool_t *pool)
Which makes the directory, identical to the Unix command mkdir -p.
Why would the IO function need a memory pool in order to operate?
My first thought was that it was perhaps an optional argument to populate the newly made directory, however the code below uses an initialized but presumptively empty memory pool. Does this mean that the IO function itself needs a memory pool, that we are passing in for it to use? But that doesn't seem likely either; couldn't the function simply create a local memory pool for it to use which is then destroyed upon return or error?
So, what use is the memory pool? The documentation linked is unhelpful on this point.
Code shortened and shown below, for the curious.
int DB_init()
{
apr_pool_t *p = NULL;
apr_pool_initialize();
apr_pool_create(&p, NULL);
if(access(DB_DIR, W_OK | X_OK) == -1) {
apr_status_t rc = apr_dir_make_recursive(DB_DIR,
APR_UREAD | APR_UWRITE | APR_UEXECUTE |
APR_GREAD | APR_GWRITE | APR_GEXECUTE, p);
}
if(access(DB_FILE, W_OK) == -1) {
FILE *db = DB_open(DB_FILE, "w");
check(db, "Cannot open database: %s", DB_FILE);
DB_close(db);
}
apr_pool_destroy(p);
return 0;
}

If you pull up the source, you'll see: apr_dir_make_recursive() calls path_remove_last_component():
static char *path_remove_last_component (const char *path, apr_pool_t *pool)
{
const char *newpath = path_canonicalize (path, pool);
int i;
for (i = (strlen(newpath) - 1); i >= 0; i--) {
if (path[i] == PATH_SEPARATOR)
break;
}
return apr_pstrndup (pool, path, (i < 0) ? 0 : i);
}
This function is creating copies of the path in apr_pstrndup(), each representing a smaller component of it.
To answer your question - because of how it was implemented. Would it be possible to do the same without allocating memory, yes. I think in this case everything came out cleaner and more readable by copying the necessary path components.

The implementation of the function (found here) shows that the pool is used to allocate strings representing the individual components of the path.
The reason the function does not create its own local pool is because the pool may be reused across multiple calls to the apr_*() functions. It just so happens that DB_init() does not have a need to reuse an apr_pool_t.

Related

getpwuid_r and getgrgid_r are slow, how can I cache them?

I am writing a clone of find while learning C. When implementing the -ls option I've stumbled upon a problem that the getpwuid_r and getgrgid_r calls are really slow, the same applies to getpwuid and getgrgid. I need them to display the user/group names from ids provided by stat.h.
For example, listing the whole filesystem gets 3x slower:
# measurements were made 3 times and the fastest run was recorded
# with getgrgid_r
time ./myfind / -ls > list.txt
real 0m4.618s
user 0m1.848s
sys 0m2.744s
# getgrgid_r replaced with 'return "user";'
time ./myfind / -ls > list.txt
real 0m1.437s
user 0m0.572s
sys 0m0.832s
I wonder how GNU find maintains such a good speed. I've seen the sources, but they are not exactly easy to understand and to apply without special types, macros etc:
time find / -ls > list.txt
real 0m1.544s
user 0m0.884s
sys 0m0.648s
I thought about caching the uid - username and gid - groupname pairs in a data structure. Is it a good idea? How would you implement it?
You can find my complete code here.
UPDATE:
The solution was exactly what I was looking for:
time ./myfind / -ls > list.txt
real 0m1.480s
user 0m0.696s
sys 0m0.736s
Here is a version based on getgrgid (if you don't require thread safety):
char *do_get_group(struct stat attr) {
struct group *grp;
static unsigned int cache_gid = UINT_MAX;
static char *cache_gr_name = NULL;
/* skip getgrgid if we have the record in cache */
if (cache_gid == attr.st_gid) {
return cache_gr_name;
}
/* clear the cache */
cache_gid = UINT_MAX;
grp = getgrgid(attr.st_gid);
if (!grp) {
/*
* the group is not found or getgrgid failed,
* return the gid as a string then;
* an unsigned int needs 10 chars
*/
char group[11];
if (snprintf(group, 11, "%u", attr.st_gid) < 0) {
fprintf(stderr, "%s: snprintf(): %s\n", program, strerror(errno));
return "";
}
return group;
}
cache_gid = grp->gr_gid;
cache_gr_name = grp->gr_name;
return grp->gr_name;
}
getpwuid:
char *do_get_user(struct stat attr) {
struct passwd *pwd;
static unsigned int cache_uid = UINT_MAX;
static char *cache_pw_name = NULL;
/* skip getpwuid if we have the record in cache */
if (cache_uid == attr.st_uid) {
return cache_pw_name;
}
/* clear the cache */
cache_uid = UINT_MAX;
pwd = getpwuid(attr.st_uid);
if (!pwd) {
/*
* the user is not found or getpwuid failed,
* return the uid as a string then;
* an unsigned int needs 10 chars
*/
char user[11];
if (snprintf(user, 11, "%u", attr.st_uid) < 0) {
fprintf(stderr, "%s: snprintf(): %s\n", program, strerror(errno));
return "";
}
return user;
}
cache_uid = pwd->pw_uid;
cache_pw_name = pwd->pw_name;
return pwd->pw_name;
}
UPDATE 2:
Changed long to unsigned int.
UPDATE 3:
Added the cache clearing. It is absolutely necessary, because pwd->pw_name may point to a static area. getpwuid can overwrite its contents if it fails or simply when executed somewhere else in the program.
Also removed strdup. Since the output of getgrgid and getpwuid should not be freed, there is no need to require free for our wrapper functions.
The timings indeed indicate a strong suspicion on these functions.
Looking at your function do_get_group, there are some issues:
You use sysconf(_SC_GETPW_R_SIZE_MAX); for every call to do_get_group and do_get_user, definitely cache that, it will not change during the lifetime of your program, but you will not gain much.
You use attr.st_uid instead of attr.st_gid, which probably causes the lookup to fail for many files, possibly defeating the cacheing mechanism, if any. Fix this first, this is a bug!
You return values that should not be passed to free() by the caller, such as grp->gr_name and "". You should always allocate the string you return. The same issue is probably present in do_get_user().
Here is a replacement for do_get_group with a one shot cache. See if this improves the performance:
/*
* #brief returns the groupname or gid, if group not present on the system
*
* #param attr the entry attributes from lstat
*
* #returns the groupname if getgrgid() worked, otherwise gid, as a string
*/
char *do_get_group(struct stat attr) {
char *group;
struct group grp;
struct group *result;
static size_t length = 0;
static char *buffer = NULL;
static gid_t cache_gid = -1;
static char *cache_gr_name = NULL;
if (!length) {
/* only allocate the buffer once */
long sysconf_length = sysconf(_SC_GETPW_R_SIZE_MAX);
if (sysconf_length == -1) {
sysconf_length = 16384;
}
length = (size_t)sysconf_length;
buffer = calloc(length, 1);
}
if (!buffer) {
fprintf(stderr, "%s: malloc(): %s\n", program, strerror(errno));
return strdup("");
}
/* check the cache */
if (cache_gid == attr.st_gid) {
return strdup(cache_gr_name);
}
/* empty the cache */
cache_gid = -1;
free(cache_gr_name);
cache_gr_name = NULL;
if (getgrgid_r(attr.st_gid, &grp, buffer, length, &result) != 0) {
fprintf(stderr, "%s: getpwuid_r(): %s\n", program, strerror(errno));
return strdup("");
}
if (result) {
group = grp.gr_name;
} else {
group = buffer;
if (snprintf(group, length, "%ld", (long)attr.st_gid) < 0) {
fprintf(stderr, "%s: snprintf(): %s\n", program, strerror(errno));
return strdup("");
}
}
/* load the cache */
cache_gid = attr.st_gid;
cache_gr_name = strdup(group);
return strdup(group);
}
Whether the getpwuid and getgrgid calls are cached depends on how they are implemented and on how your system is configured. I recently wrote an implementation of ls and ran into a similar problem.
I found that on all modern systems I tested, the two functions are uncached unless you run the name service caching dæmon (nscd) in which case nscd makes sure that the cache stays up to date. It's easy to understand why this happens: Without an nscd, caching the information could lead to outdated output which is a violation of the specification.
I don't think you should rely on these functions caching the group and passwd databases because they often don't. I implemented custom caching code for this purpose. If you don't require to have up-to-date information in case the database contents change during program execution, this is perfectly fine to do.
You can find my implementation of such a cache here. I'm not going to publish it on Stack Overflow as I do not desire to publish the code under the MIT license.

TCL_LINK_STRING causing segmentation fault (core dumped)

I'm trying to share a variable with c and tcl, the problem is when i try to read the variable in the c thread from tcl, it causes segmentation error, i'm not sure this is the right way to do it, but it seems to work for ints. The part that is causing the segmentation fault is this line is when i try to print "Var" but i want to read the variable to do the corresponding action when the variable changes.
Here is the C code that i'm using
void mode_service(ClientData clientData) {
while(1) {
char* Var = (char *) clientData;
printf("%s\n", Var);
usleep(100000); //100ms
}
}
static int mode_thread(ClientData cdata, Tcl_Interp *interp, int objc, Tcl_Obj *const objv[]) {
Tcl_ThreadId id;
ClientData limitData;
limitData = cdata;
id = 0;
Tcl_CreateThread(&id, mode_service, limitData, TCL_THREAD_STACK_DEFAULT, TCL_THREAD_NOFLAGS);
printf("Tcl_CreateThread id = %d\n", (int) id);
// Wait thread process, before returning to TCL prog
int i, aa;
for (i=0 ; i<100000; i++) {aa = i;}
// Return thread ID to tcl prog to allow mutex use
Tcl_SetObjResult(interp, Tcl_NewIntObj((int)id));
printf("returning\n");
return TCL_OK;
}
int DLLEXPORT Modemanager_Init(Tcl_Interp *interp){
if (Tcl_InitStubs(interp, TCL_VERSION, 0) == NULL) {
return TCL_ERROR;
}
if (Tcl_PkgProvide(interp, "PCIe", "1.0") == TCL_ERROR) {
return TCL_ERROR;
}
// Create global Var
int *sharedPtr=NULL;
//sharedPtr = sharedPtr = (char *) Tcl_Alloc(sizeof(char));
Tcl_LinkVar(interp, "mode", (char *) &sharedPtr, TCL_LINK_STRING);
Tcl_CreateObjCommand(interp, "mode_thread", mode_thread, sharedPtr, NULL);
return TCL_OK;
}
In the tcl code, i'm changing the variable mode whenever the user presses a button for example:
set mode "Idle"
button .startSamp -text "Sample Start" -width 9 -height 3 -background $btnColor -relief flat -state normal -command {set mode "Sampling"}
set threadId [mode_thread]
puts "Created thread $threadId, waiting"
Your code is a complete mess! You need to decide what you are doing and then do just that. In particular, you are using Tcl_LinkVar so you need to decide what sort of variable you are linking to. If you get a mismatch between the storage, the C access pattern and the declared semantic type, you'll get crashes.
Because your code is in too complicated a mess for me to figure out exactly what you want to do, I'll illustrate with less closely related examples. You'll need to figure out from them how to change things in your code to get the result you need.
Linking Integer Variables
Let's do the simple case: a global int variable (declared outside any function).
int sharedVal;
You want your C code to read that variable and get the value. Easy! Just read it as it is in scope. You also want Tcl code to be able to write to that variable. Easy! In the package initialization function, put this:
Tcl_LinkVar(interp /* == the Tcl interpreter context */,
"sharedVal" /* == the Tcl name */,
(char *) &sharedVal /* == pointer to C variable */,
TCL_LINK_INT /* == what is it! An integer */);
Note that after that (until you Tcl_UnlinkVar) whenever Tcl code reads from the Tcl variable, the current value will be fetched from the C variable and converted.
If you want that variable to be on the heap, you then do:
int *sharedValPtr = malloc(sizeof(int));
C code accesses using *sharedValPtr, and you bind to Tcl with:
Tcl_LinkVar(interp /* == the Tcl interpreter context */,
"sharedVal" /* == the Tcl name */,
(char *) sharedValPtr /* == pointer to C variable */,
TCL_LINK_INT /* == what is it! An integer */);
Linking String Variables
There's a bunch of other semantic types as well as TCL_LINK_INT (see the documentation for a list) but they all follow that pattern except for TCL_LINK_STRING. With that, you do:
char *sharedStr = NULL;
Tcl_LinkVar(interp, "sharedStr", (char *) &sharedStr, TCL_LINK_STRING);
You also need to be aware that the string will always be allocated with Tcl_Alloc (which is substantially faster than most system memory allocators for typical Tcl memory usage patterns) and not with any other memory allocator, and so will also always be deallocated with Tcl_Free. Practically, that means if you set the string from the C side, you must use Tcl_Alloc to allocate the memory.
Posting Update Notifications
The final piece to note is when you set the variable from the C side but want Tcl to notice that the change has set (e.g., because a trace has been set or because you've surfaced the value in a Tk GUI), you should do Tcl_UpdateLinkedVar to let Tcl know that a change has happened that it should pay attention to. If you never use traces (or Tk GUIs, or the vwait command) to watch the variable for updates, you can ignore this API call.
Donal's answer is correct, but I try to show you what you did with your ClientData.
To clarify: All (or almost all, Idk) Tcl functions that take a function pointer also take a parameter of type ClientData that is passed to your function when Tcl calls it.
Let's take a look at this line:
Tcl_CreateObjCommand(interp, "mode_thread", mode_thread, NULL, NULL);
// ------------------------------------------------------^^^^
You always pass NULL as ClientData to the mode_thread function.
In the mode_thread function you use the passed ClientData (NULL) to pass it as ClientData to the new Thread:
limitData = cdata;
// ...
Tcl_CreateThread(&id, mode_service, limitData, TCL_THREAD_STACK_DEFAULT, TCL_THREAD_NOFLAGS);
In the mode_service function you use the ClientData (which is still NULL) as pointer to a char array:
char* Var = (char *) clientData;
Which is a pointer to the address 0x00.
And then you tell printf to dereference this NULL pointer:
printf("%s\n", Var);
Which obviously crashes your program.

C memory management in server application

I am student and I am writing HTTP proxy application in C. I have trouble with memory management. In all my previous applications I simply wrote a wrapper around malloc which aborted when malloc failed.
void *xmalloc(size_t size)
{
void *ptr;
assert(size);
ptr = malloc(size);
if (!ptr)
abort();
return ptr;
}
This I now find insufficient as I just want to refuse client and continue serving other clients when memory allocation fails due to temporary shortage of memory. If I don't want to clutter my code with checks after each malloc call (I have quite lot of them per function in parsing code), what are other options to handle memory management and which one is the best for my purposes and how what is a common way for server applications to handle memory management and shortage of memory?
Consider this function from my current code which parses one line from header portion of HTTP message (xstrndup calls xmalloc):
int http_header_parse(http_hdr_table *t, const char *s)
{
const char *p;
const char *b;
char *tmp_name;
char *tmp_value;
int ret = -1;
assert(t);
assert(s);
p = b = s;
/* field name */
for (; ; p++) {
if (*p == ':') {
if (p-b <= 0) goto out;
tmp_name = xstrndup(b, p-b);
b = ++p;
break;
}
if (is_ctl_char(*p) || is_sep_char(*p)) goto out;
}
while (*p == ' ' || *p == '\t') {
p++; b++;
}
/* field value */
for (; ; p++) {
if (is_crlf(p)) {
if (p-b <= 0) goto err_value;
tmp_value = xstrndup(b, p-b);
p += 2;
break;
}
if (!*p) goto err_value;
}
http_hdr_table_set(t, tmp_name, tmp_value);
ret = 0;
xfree(tmp_value);
err_value:
xfree(tmp_name);
out:
return ret;
}
I would like to keep things simple and handle memory allocation errors at one place and to not clutter code with malloc error handling code. What should I do? Thank you.
P.S: I am writing the application to run on POSIX/Unix-like systems. Also feel free to criticize my current coding style and practices.
If you want to use a relatively low level language like C, then you shouldn't be too worried about adding something like if(tmp_value == NULL) goto out; in 2 places.
If you can't stand the idea of 2 trivial lines of extra code, then maybe try a language that supports exceptions properly (e.g. C++) and add throw/try/catch instead. Note: I really don't like C++, but using C++ would have to make more sense than implementing your own "exception like" features and an entire layer of automated resource de-allocation in C.
Modern languages give you garbage collection and exceptions. C doesn't, so you have to work hard. There's no magical solution here.
Some tips:
Create a session structure, and keep all your allocated memory pointed from it. When the session is aborted, always call a cleanup function. This way, even if you have to check for failures in many places, at least all failures are handled the same way.
You can even create a session_allocate() function, which allocates memory and keeps it on a linked list pointed from the session structure. Everything you allocate using this function would be freed when the session is destroy.
Try to concentrate all allocations in the beginning of the session. After you've allocated all you need, the rest of your code won't need to worry about failures.
If you're on a system that supports fork(), which linux does, you can run each client connection in it's own process. When a client connection is first established, you fork your main process into a child process to handle the rest of the request. Then you can abort() like you always have and only the specific client connection is affected. This is a classic unix server model.
If you don't want to or can't use fork(), you need to abort the request by throwing an exception. In C, that would be done by using setjump() when the connection is first established and then calling longjump() when out of memory is detected. This will reset execution and the stack back to where setjump() was called.
The problem is, this will leak all the resources allocated up to that point (for example, other memory allocations that had succeeded up to the point of getting out of memory). So additionally, your memory allocator will have to track all the memory allocations for each request. When longjump() is called, the setjump() return location will then have to free all the memory that was associated with the aborted request.
This is what apache does using pools. Apache uses pools to track resource allocations so it can auto free them in the case of an abort or because the code just didn't free it: http://www.apachetutor.org/dev/pools.
You should also consider the pool model and not just simply wrap malloc() so one client can't use up all the memory in the system.
Another possibility would be to use Boehm's GC by using its GC_malloc instead of malloc (you won't need to call free or GC_free); its
GC_oom_fn function pointer (called internally from GC_malloc when no memory is available any more) can be set to your particular out of memory handler (which would deny the incoming HTTP request, perhaps with a longjmp)
The major advantage of using Boehm GC is that you don't care any more about free-ing your dynamically allocated data (provided it was allocated using GC_malloc or friends, e.g. GC_malloc_atomic for data without any pointers inside).
Notice that memory management is not a modular property. The liveness of some given data is a whole program property, see garbage collection wikipage, and RAII programming idiom.
You could of course use alloca, but that has issues that mean it must be used with care. Alternatively, you can write your code so that you minimise and localise the use of malloc. For example your function above could be rewritten to localise the allocations:
static size_t field_name_length(const char *s)
{
const char *p = s;
for ( ; *p != ':'; ++p) {
if (is_ctl_char(*p) || is_sep_char(*p))
return 0;
}
return (size_t) (p - s);
}
static size_t value_length(const char *s)
{
const char *p = s;
for (; *p && !is_crlf(p); p+=2) {
/* nothing */
}
return *p ? (size_t) (p - s) : 0;
}
int http_header_parse(http_hdr_table *t, const char *s)
{
const char *v;
int ret = -1;
size_t v_len = 0;
size_t f_len = field_name_length(s);
if (f_len) {
v = s + f_len + 1;
v = s + strspn(s, " \t");
v_len = value_length(s);
}
if (v_len > 0 && f_len > 0) {
/* Allocation is localised to this block */
const char *name = xstrndup(s, f_len);
const char *value = xstrndup(v, v_len);
if (name && value) {
http_hdr_table_set(t, name, value);
ret = 0;
}
xfree(value);
xfree(name);
}
return ret;
}
Or, even better, you could modify http_hdr_table_set to accept the pointers and lengths and avoid allocation completely.

tmpfile() on windows 7 x64

Running the following code on Windows 7 x64
#include <stdio.h>
#include <errno.h>
int main() {
int i;
FILE *tmp;
for (i = 0; i < 10000; i++) {
errno = 0;
if(!(tmp = tmpfile())) printf("Fail %d, err %d\n", i, errno);
fclose(tmp);
}
return 0;
}
Gives errno 13 (Permission denied), on the 637th and 1004th call, it works fine on XP (haven't tried 7 x86). Am I missing something or is this a bug?
I've got similar problem on Windows 8 - tmpfile() was causing win32 ERROR_ACCESS_DENIED error code - and yes, if you run application with administrator privileges - then it works fine.
I guess problem is mentioned over here:
https://lists.gnu.org/archive/html/bug-gnulib/2007-02/msg00162.html
Under Windows, the tmpfile function is defined to always create
its temporary file in the root directory. Most users don't have
permission to do that, so it will often fail.
I would suspect that this is kinda incomplete windows port issue - so this should be an error reported to Microsoft. (Why to code tmpfile function if it's useless ?)
But who have time to fight with Microsoft wind mills ?! :-)
I've coded similar implementation using GetTempPathW / GetModuleFileNameW / _wfopen. Code where I've encountered this problem came from libjpeg - I'm attaching whole source code here, but you can pick up code from jpeg_open_backing_store.
jmemwin.cpp:
//
// Windows port for jpeg lib functions.
//
#define JPEG_INTERNALS
#include <Windows.h> // GetTempFileName
#undef FAR // Will be redefined - disable warning
#include "jinclude.h"
#include "jpeglib.h"
extern "C" {
#include "jmemsys.h" // jpeg_ api interface.
//
// Memory allocation and freeing are controlled by the regular library routines malloc() and free().
//
GLOBAL(void *) jpeg_get_small (j_common_ptr cinfo, size_t sizeofobject)
{
return (void *) malloc(sizeofobject);
}
GLOBAL(void) jpeg_free_small (j_common_ptr cinfo, void * object, size_t sizeofobject)
{
free(object);
}
/*
* "Large" objects are treated the same as "small" ones.
* NB: although we include FAR keywords in the routine declarations,
* this file won't actually work in 80x86 small/medium model; at least,
* you probably won't be able to process useful-size images in only 64KB.
*/
GLOBAL(void FAR *) jpeg_get_large (j_common_ptr cinfo, size_t sizeofobject)
{
return (void FAR *) malloc(sizeofobject);
}
GLOBAL(void) jpeg_free_large (j_common_ptr cinfo, void FAR * object, size_t sizeofobject)
{
free(object);
}
//
// Used only by command line applications, not by static library compilation
//
#ifndef DEFAULT_MAX_MEM /* so can override from makefile */
#define DEFAULT_MAX_MEM 1000000L /* default: one megabyte */
#endif
GLOBAL(long) jpeg_mem_available (j_common_ptr cinfo, long min_bytes_needed, long max_bytes_needed, long already_allocated)
{
// jmemansi.c's jpeg_mem_available implementation was insufficient for some of .jpg loads.
MEMORYSTATUSEX status = { 0 };
status.dwLength = sizeof(status);
GlobalMemoryStatusEx(&status);
if( status.ullAvailPhys > LONG_MAX )
// Normally goes here since new PC's have more than 4 Gb of ram.
return LONG_MAX;
return (long) status.ullAvailPhys;
}
/*
Backing store (temporary file) management.
Backing store objects are only used when the value returned by
jpeg_mem_available is less than the total space needed. You can dispense
with these routines if you have plenty of virtual memory; see jmemnobs.c.
*/
METHODDEF(void) read_backing_store (j_common_ptr cinfo, backing_store_ptr info, void FAR * buffer_address, long file_offset, long byte_count)
{
if (fseek(info->temp_file, file_offset, SEEK_SET))
ERREXIT(cinfo, JERR_TFILE_SEEK);
size_t readed = fread( buffer_address, 1, byte_count, info->temp_file);
if (readed != (size_t) byte_count)
ERREXIT(cinfo, JERR_TFILE_READ);
}
METHODDEF(void)
write_backing_store (j_common_ptr cinfo, backing_store_ptr info, void FAR * buffer_address, long file_offset, long byte_count)
{
if (fseek(info->temp_file, file_offset, SEEK_SET))
ERREXIT(cinfo, JERR_TFILE_SEEK);
if (JFWRITE(info->temp_file, buffer_address, byte_count) != (size_t) byte_count)
ERREXIT(cinfo, JERR_TFILE_WRITE);
// E.g. if you need to debug writes.
//if( fflush(info->temp_file) != 0 )
// ERREXIT(cinfo, JERR_TFILE_WRITE);
}
METHODDEF(void)
close_backing_store (j_common_ptr cinfo, backing_store_ptr info)
{
fclose(info->temp_file);
// File is deleted using 'D' flag on open.
}
static HMODULE DllHandle()
{
MEMORY_BASIC_INFORMATION info;
VirtualQuery(DllHandle, &info, sizeof(MEMORY_BASIC_INFORMATION));
return (HMODULE)info.AllocationBase;
}
GLOBAL(void) jpeg_open_backing_store(j_common_ptr cinfo, backing_store_ptr info, long total_bytes_needed)
{
// Generate unique filename.
wchar_t path[ MAX_PATH ] = { 0 };
wchar_t dllPath[ MAX_PATH ] = { 0 };
GetTempPathW( MAX_PATH, path );
// Based on .exe or .dll filename
GetModuleFileNameW( DllHandle(), dllPath, MAX_PATH );
wchar_t* p = wcsrchr( dllPath, L'\\');
wchar_t* ext = wcsrchr( p + 1, L'.');
if( ext ) *ext = 0;
wchar_t* outFile = path + wcslen(path);
static int iTempFileId = 1;
// Based on process id (so processes would not fight with each other)
// Based on some process global id.
wsprintfW(outFile, L"%s_%d_%d.tmp",p + 1, GetCurrentProcessId(), iTempFileId++ );
// 'D' - temporary file.
if ((info->temp_file = _wfopen(path, L"w+bD") ) == NULL)
ERREXITS(cinfo, JERR_TFILE_CREATE, "");
info->read_backing_store = read_backing_store;
info->write_backing_store = write_backing_store;
info->close_backing_store = close_backing_store;
} //jpeg_open_backing_store
/*
* These routines take care of any system-dependent initialization and
* cleanup required.
*/
GLOBAL(long)
jpeg_mem_init (j_common_ptr cinfo)
{
return DEFAULT_MAX_MEM; /* default for max_memory_to_use */
}
GLOBAL(void)
jpeg_mem_term (j_common_ptr cinfo)
{
/* no work */
}
}
I'm intentionally ignoring errors from some of functions - have you ever seen GetTempPathW or GetModuleFileNameW failing ?
A bit of a refresher from the manpage of on tmpfile(), which returns a FILE*:
The file will be automatically deleted when it is closed or the program terminates.
My verdict for this issue: Deleting a file on Windows is weird.
When you delete a file on Windows, for as long as something holds a handle, you can't call CreateFile on something with the same absolute path, otherwise it will fail with the NT error code STATUS_DELETE_PENDING, which gets mapped to the Win32 code ERROR_ACCESS_DENIED. This is probably where EPERM in errno is coming from. You can confirm this with a tool like Sysinternals Process Monitor.
My guess is that CRT somehow wound up creating a file that has the same name as something it's used before. I've sometimes witnessed that deleting files on Windows can appear asynchronous because some other process (sometimes even an antivirus product, in reaction to the fact that you've just closed a delete-on-close handle...) will leave a handle open to the file, so for some timing window you will see a visible file that you can't get a handle to without hitting delete pending/access denied. Or, it could be that tmpfile has simply chosen a filename that some other process is working on.
To avoid this sort of thing you might want to consider another mechanism for temp files... For example a function like Win32 GetTempFileName allows you to create your own prefix which might make a collision less likely. That function appears to resolve race conditions by retrying if a create fails with "already exists", so be careful about deleting the temp filenames that thing generates - deleting the file cancels your rights to use it concurrently with other processes/threads.

Recursive CreateDirectory

I found many examples of CreatingDirectory recursively, but not the one I was looking for.
here is the spec
Given input
\\server\share\aa\bb\cc
c:\aa\bb\cc
USING helper API
CreateDirectory (char * path)
returns true, if successful
else
FALSE
Condition: There should not be any parsing to distinguish if the path is Local or Server share.
Write a routine in C, or C++
I think it's quite easier... here a version that works in every Windows version:
unsigned int pos = 0;
do
{
pos = path.find_first_of("\\/", pos + 1);
CreateDirectory(path.substr(0, pos).c_str(), NULL);
} while (pos != std::string::npos);
Unicode:
pos = path.find_first_of(L"\\/", pos + 1);
Regards,
This might be exactly what you want.
It doesn't try to do any parsing to distinguish if the path is Local or Server share.
bool TryCreateDirectory(char *path){
char *p;
bool b;
if(
!(b=CreateDirectory(path))
&&
!(b=NULL==(p=strrchr(path, '\\')))
){
size_t i;
(p=strncpy((char *)malloc(1+i), path, i=p-path))[i]='\0';
b=TryCreateDirectory(p);
free(p);
b=b?CreateDirectory(path):false;
}
return b;
}
The algorithm is quite simple, just pass the string of higher level directory recursively while creation of current level of directory fails until one success or there is no more higher level. When the inner call returns with succeed, create the current. This method do not parse to determ the local or server it self, it's according to the CreateDirectory.
In WINAPI, CreateDirectory will never allows you to create "c:" or "\" when the path reaches that level, the method soon falls in to calling it self with path="" and this fails, too. It's the reason why Microsoft defines file sharing naming rule like this, for compatibility of DOS path rule and simplify the coding effort.
Totally hackish and insecure and nothing you'd ever actually want to do in production code, but...
Warning: here be code that was typed in a browser:
int createDirectory(const char * path) {
char * buffer = malloc((strlen(path) + 10) * sizeof(char));
sprintf(buffer, "mkdir -p %s", path);
int result = system(buffer);
free(buffer);
return result;
}
How about using MakeSureDirectoryPathExists() ?
Just walk through each directory level in the path starting from the root, attempting to create the next level.
If any of the CreateDirectory calls fail then you can exit early, you're successful if you get to the end of the path without a failure.
This is assuming that calling CreateDirectory on a path that already exists has no ill effects.
The requirement of not parsing the pathname for server names is interesting, as it seems to concede that parsing for / is required.
Perhaps the idea is to avoid building in hackish expressions for potentially complex syntax for hosts and mount points, which can have on some systems elaborate credentials encoded.
If it's homework, I may be giving away the algorithm you are supposed to think up, but it occurs to me that one way to meet those requirements is to start trying by attempting to mkdir the full pathname. If it fails, trim off the last directory and try again, if that fails, trim off another and try again... Eventually you should reach a root directory without needing to understand the server syntax, and then you will need to start adding pathname components back and making the subdirs one by one.
std::pair<bool, unsigned long> CreateDirectory(std::basic_string<_TCHAR> path)
{
_ASSERT(!path.empty());
typedef std::basic_string<_TCHAR> tstring;
tstring::size_type pos = 0;
while ((pos = path.find_first_of(_T("\\/"), pos + 1)) != tstring::npos)
{
::CreateDirectory(path.substr(0, pos + 1).c_str(), nullptr);
}
if ((pos = path.find_first_of(_T("\\/"), path.length() - 1)) == tstring::npos)
{
path.append(_T("\\"));
}
::CreateDirectory(path.c_str(), nullptr);
return std::make_pair(
::GetFileAttributes(path.c_str()) != INVALID_FILE_ATTRIBUTES,
::GetLastError()
);
}
void createFolders(const std::string &s, char delim) {
std::stringstream ss(s);
std::string item;
char combinedName[50]={'\0'};
while (std::getline(ss, item, delim)) {
sprintf(combinedName,"%s%s%c",combinedName,item.c_str(),delim);
cout<<combinedName<<endl;
struct stat st = {0};
if (stat(combinedName,&st)==-1)
{
#if REDHAT
mkdir(combinedName,0777);
#else
CreateDirectory(combinedName,NULL);
#endif
}
}
}

Resources