I need to recode malloc in C for a school project.
I'd like to send a GET request to a server in the malloc function.
All the solutions I've tried, produce a nasty infinite loop when loaded through LD_PRELOAD because they use malloc.
So my question is really simple: is there a way to make a send request using C, without using any of the *alloc functions? I don't care about what the server returns, I just want to send data.
Doing an HTTP transaction--which can take several seconds, and possibly fail, and calls lots of other library functions like malloc--inside your own malloc is never going to work. If you want a documented malloc, the better way to do it is have a piece of code inside your malloc that puts the data to be documented in a queue--fast, no mess. Then, periodically, perhaps in a separate thread or some other slack time in the application, remove the data from the queue and post it.
A nice way to do this is Linux would be to write the data to an open pipe, and have a program on the other end of the pipe do the posting.
Related
I'm in the midst of wrapping a C library with cgo to be usable by normal Go code.
My problem is that I'd like to propagate error strings up to the Go API, but the C library in question makes error strings available via thread-local storage; there's a global get_error() call that returns a pointer to thread local character data.
My original plan was to call into C via cgo, check if the call returned an error, and if so, wrap the error string using C.GoString to convert it from a raw character pointer into a Go string. It'd look something like C.GoString(C.get_error()).
The problem that I foresee here is that TLS in C works on the level of native OS threads, but in my understanding, the calling Go code will be coming from one of potentially N goroutines that are multiplexed across some number of underlying native threads in a thread pool managed by the Go scheduler.
What I'm afraid of is running into a situation where I call into the C routine, then after the C routine returns, but before I copy the error string, the Go scheduler decides to swap the current goroutine out for another one. When the original goroutine gets swapped back in, it could potentially be on a different native thread for all I know, but even if it gets swapped back onto the same thread, any goroutines that ran there in the intervening time could've changed the state of the TLS, causing me to load an error string for an unrelated call.
My questions are these:
Is this a reasonable concern? Am I misunderstanding something about the go scheduler, or the way it interacts with cgo, that would cause this to not be an issue?
If this is a reasonable concern, how can I work around it?
cgo somehow manages to propagate errno values back to the calling Go code, which are also stored in TLS, which makes me think there must be a safe way to do this.
I can't think of a way that the C code itself could get preempted by the go scheduler, so should I introduce a wrapper C function and have IT make the necessary call and then conditionally copy the error string before returning back up to goland?
I'm interested in any solution that would allow me to propagate the error strings out to the rest of Go, but I'm hoping to avoid any solution that would require me to serialize accesses around the TLS, as adding a lock just to grab an error string seems greatly unfortunate to me.
Thanks in advance!
What I'm afraid of is running into a situation where I call into the C routine, then after the C routine returns, but before I copy the error string, the Go scheduler decides to swap the current goroutine out for another one. ...
Is this a reasonable concern?
Yes. The cgo "call C code" wrappers lock on to one POSIX / OS thread for the duration of each call, but the thread they lock is not fixed for all time; it does in fact bop around, as it were, to multiple different threads over time, as long as your goroutines are operating normally. (Since Go is cooperatively scheduled in the current implementations, you can, in some circumstances, be careful not to do anything that might let you switch underlying OS threads, but this is probably not a good plan.)
You can use runtime.LockOSThread here, but I think the best plan is otherwise:
how can I work around it?
Grab the error before Go resumes its normal scheduling algorithm (i.e., before unlocking the goroutine from the C / POSIX thread).
cgo somehow manages to propagate errno values ...
It grabs the errno value before unlocking the goroutine from the POSIX thread.
My original plan was to call into C via cgo, check if the call returned an error, and if so, wrap the error string using C.GoString to convert it from a raw character pointer into a Go string. It'd look something like C.GoString(C.get_error()).
If there is a variant of this that takes the error number (rather than fishing it out of a TLS variable), that plan should still work: just make sure that your C routines provide both the return value and the error number.
If not, write your own C wrapper, just as you suggested:
ftype wrapper_for_realfunc(char **errp, arg1type arg1, arg2type arg2) {
ftype ret = realfunc(arg1, arg2);
if IS_ERROR(ret) {
*errp = get_error();
} else {
*errp = NULL;
}
return ret;
}
Now your Go wrapper simply calls the wrapper, which fills in a pointer to C memory with an extra *C.char argument, setting it to nil if there is no error, and setting it to something on which you can use C.GoString if there is an error.
If that's not feasible for some reason, consider using runtime.LockOSThread and its counterpart, runtime.UnlockOSThread.
I was reading this: https://stackoverflow.com/a/26648931/6872717 and decided to fix that code and improve it to be able to use it as a library.
It is one of the examples in the libcurl web page: https://curl.haxx.se/libcurl/c/fopen.html
I found out that although the libcurl documentation states this referring to the function curl_multi_init():
This init call MUST have a corresponding call to curl_multi_cleanup
when the operation is complete.
the example code doesn't call it, ever.
In a program, it can be easy to add that call at the end of the main, but for a library, it is more difficult (or maybe impossible) to know if the multi handle can be cleaned up. Is it valid to omit the call?
I guess that constitutes a memory leak, but not a very big one, and it's only once, and I don't know how to avoid it.
Would it be OK to write a __attribute__((destructor)) url_deinit() function so that if the user forgets to call it, it would be called anyway, or would the resources already be destroyed at that moment and produce UB?
If you never clean it up, you will never get the memory and resources "back" that is allocated in relation to that handle.
In the fopen.c example, the multi handle is global and is reused and is indeed never freed. That's fine if you're fine with never getting the memory back.
When your program exits, all memory and resources will be freed forcibly anyway.
I am trying to understand/make sure about the nature of "callback function" utilized by libcurl.
As usual, after setting all the options using curl_easy_setopt, I would call curl_easy_perform.
But when there is a call back function will libcurl actually return absolutely all the data before exiting curl_easy_perform.
I understand that the multi-interface is there to provide non-blocking capabilities. But "call back functions" are meant to be called "later in time" while other stuff is taking place, right? So for easy-interface is it really blocking till all data is received?
How can I test this?
I have been researching and I put below two quotes from the libcurl docs. But I am stuck at trying to comprehend the concepts of call back functions and blocking manner
http://curl.haxx.se/libcurl/c/curl_easy_perform.html
curl_easy_perform - man page:
curl_easy_perform performs the entire request in a blocking manner and returns when done, or if it failed. For non-blocking behavior, see curl_multi_perform.”
http://curl.haxx.se/libcurl/c/curl_multi_perform.html
curl_multi_perform - man page:
This function handles transfers on all the added handles that need attention in an non-blocking fashion”
Please note that the aim is to make sure that after the end of my function call, the application must have ALL the data. We are doing things strictly sequentially and can not afford chunks of data flying in at different times.
Yes, the easy interface is blocking until the entire request is complete. You can test this by doing lots of requests and verifying that it works this way - or just trust the docs and the thousands of users who depend on this behavior.
"Callbacks" means that they are functions you write and provide that get "called back" from the function you invoke. So, you call curl_easy_perform() and then libcurl itself calls back to your callback function(s) according to the documentation all the way until either something failed or the transfer is complete and then curl_easy_perform() returns back to your program again.
When malloc() fails, which would be the best way to handle the error? If it fails, I want to immediately exit the program, which I would normally do with using exit(). But in this special case, I'm not quite sure if exit() would be the way to go here.
In library code, it's absolutely unacceptable to call exit or abort under any circumstances except when the caller broke the contact of your library's documented interface. If you're writing library code, you should gracefully handle any allocation failures, freeing any memory or other resources acquired in the attempted operation and returning an error condition to the caller. The calling program may then decide to exit, abort, reject whatever command the user gave which required excessive memory, free some unneeded data and try again, or whatever makes sense for the application.
In all cases, if your application is holding data which has not been synchronized to disk and which has some potential value to the user, you should make every effort to ensure that you don't throw away this data on allocation failures. The user will almost surely be very angry. It's best to design your applications so that the "save" function does not require any allocations, but if you can't do that in general, you might instead want to perform frequent auto-save-to-temp-file operations or provide a way of dumping the memory contents to disk in a form that's not the standard file format (which might for example require ugly XML and ZIP libraries, each with their own allocation needs, to write) but instead a more "raw dump" which you application can read and recover from on the next startup.
If malloc() returns NULL it means that the allocation was unsuccessful. It's up to you to deal with this error case. I personally find it excessive to exit your entire process because of a failed allocation. Deal with it some other way.
Use Both?
It depends on whether the core file will be useful. If no one is going to analyze it, then you may as well simply _exit(2) or exit(3).
If the program will sometimes be used locally and you intend to analyze any core files produced, then that's an argument for using abort(3).
You could always choose conditionally, so, with --debug use abort(3) and without it use exit.
I have a client using select() to check if there is anything to be received, else it times out and the user is able to send(). Which works well enough. However, the program locks up waiting for user input, so it cant recv() again until the user has sent something.
I am not having much luck with using threads either, as I cannot seem to find a good resource, which shows me how to use them.
I have tried a creating two threads (using CreateThread) for send and recv functions which are basically two functions using while loops to keep sending and receiving. And then the two CreateThreads() are wrapped up in a while loop, because otherwise it just seemed to drop out.
I have basically no previous experience with threads, so my description of what i've been doing will probably sound ridiculous. But I would appreciate any help, in using them properly for this kind of use, or an alternative method would also be great.
Can't find a good resource on socket programming? Bah. Read the bible:
Unix Network Programming
this has a strong feel of assignment work...
I am not too much into Windows programming but when I did do some earlier, there used to be a kbhit (probably) function that allowed you to check if the user had sent any inputs. You could look for something similar and try a non-blocking user input check before starting your select again.
You could then get this working without multi-threading
(unless, er, you have to use that).