I am trying to understand/make sure about the nature of "callback function" utilized by libcurl.
As usual, after setting all the options using curl_easy_setopt, I would call curl_easy_perform.
But when there is a call back function will libcurl actually return absolutely all the data before exiting curl_easy_perform.
I understand that the multi-interface is there to provide non-blocking capabilities. But "call back functions" are meant to be called "later in time" while other stuff is taking place, right? So for easy-interface is it really blocking till all data is received?
How can I test this?
I have been researching and I put below two quotes from the libcurl docs. But I am stuck at trying to comprehend the concepts of call back functions and blocking manner
http://curl.haxx.se/libcurl/c/curl_easy_perform.html
curl_easy_perform - man page:
curl_easy_perform performs the entire request in a blocking manner and returns when done, or if it failed. For non-blocking behavior, see curl_multi_perform.”
http://curl.haxx.se/libcurl/c/curl_multi_perform.html
curl_multi_perform - man page:
This function handles transfers on all the added handles that need attention in an non-blocking fashion”
Please note that the aim is to make sure that after the end of my function call, the application must have ALL the data. We are doing things strictly sequentially and can not afford chunks of data flying in at different times.
Yes, the easy interface is blocking until the entire request is complete. You can test this by doing lots of requests and verifying that it works this way - or just trust the docs and the thousands of users who depend on this behavior.
"Callbacks" means that they are functions you write and provide that get "called back" from the function you invoke. So, you call curl_easy_perform() and then libcurl itself calls back to your callback function(s) according to the documentation all the way until either something failed or the transfer is complete and then curl_easy_perform() returns back to your program again.
Related
I'm in the midst of wrapping a C library with cgo to be usable by normal Go code.
My problem is that I'd like to propagate error strings up to the Go API, but the C library in question makes error strings available via thread-local storage; there's a global get_error() call that returns a pointer to thread local character data.
My original plan was to call into C via cgo, check if the call returned an error, and if so, wrap the error string using C.GoString to convert it from a raw character pointer into a Go string. It'd look something like C.GoString(C.get_error()).
The problem that I foresee here is that TLS in C works on the level of native OS threads, but in my understanding, the calling Go code will be coming from one of potentially N goroutines that are multiplexed across some number of underlying native threads in a thread pool managed by the Go scheduler.
What I'm afraid of is running into a situation where I call into the C routine, then after the C routine returns, but before I copy the error string, the Go scheduler decides to swap the current goroutine out for another one. When the original goroutine gets swapped back in, it could potentially be on a different native thread for all I know, but even if it gets swapped back onto the same thread, any goroutines that ran there in the intervening time could've changed the state of the TLS, causing me to load an error string for an unrelated call.
My questions are these:
Is this a reasonable concern? Am I misunderstanding something about the go scheduler, or the way it interacts with cgo, that would cause this to not be an issue?
If this is a reasonable concern, how can I work around it?
cgo somehow manages to propagate errno values back to the calling Go code, which are also stored in TLS, which makes me think there must be a safe way to do this.
I can't think of a way that the C code itself could get preempted by the go scheduler, so should I introduce a wrapper C function and have IT make the necessary call and then conditionally copy the error string before returning back up to goland?
I'm interested in any solution that would allow me to propagate the error strings out to the rest of Go, but I'm hoping to avoid any solution that would require me to serialize accesses around the TLS, as adding a lock just to grab an error string seems greatly unfortunate to me.
Thanks in advance!
What I'm afraid of is running into a situation where I call into the C routine, then after the C routine returns, but before I copy the error string, the Go scheduler decides to swap the current goroutine out for another one. ...
Is this a reasonable concern?
Yes. The cgo "call C code" wrappers lock on to one POSIX / OS thread for the duration of each call, but the thread they lock is not fixed for all time; it does in fact bop around, as it were, to multiple different threads over time, as long as your goroutines are operating normally. (Since Go is cooperatively scheduled in the current implementations, you can, in some circumstances, be careful not to do anything that might let you switch underlying OS threads, but this is probably not a good plan.)
You can use runtime.LockOSThread here, but I think the best plan is otherwise:
how can I work around it?
Grab the error before Go resumes its normal scheduling algorithm (i.e., before unlocking the goroutine from the C / POSIX thread).
cgo somehow manages to propagate errno values ...
It grabs the errno value before unlocking the goroutine from the POSIX thread.
My original plan was to call into C via cgo, check if the call returned an error, and if so, wrap the error string using C.GoString to convert it from a raw character pointer into a Go string. It'd look something like C.GoString(C.get_error()).
If there is a variant of this that takes the error number (rather than fishing it out of a TLS variable), that plan should still work: just make sure that your C routines provide both the return value and the error number.
If not, write your own C wrapper, just as you suggested:
ftype wrapper_for_realfunc(char **errp, arg1type arg1, arg2type arg2) {
ftype ret = realfunc(arg1, arg2);
if IS_ERROR(ret) {
*errp = get_error();
} else {
*errp = NULL;
}
return ret;
}
Now your Go wrapper simply calls the wrapper, which fills in a pointer to C memory with an extra *C.char argument, setting it to nil if there is no error, and setting it to something on which you can use C.GoString if there is an error.
If that's not feasible for some reason, consider using runtime.LockOSThread and its counterpart, runtime.UnlockOSThread.
What is the right (GIO/Glib/GTK/Gnome) way to process GInputStream in non-blocking manner and chunk-by-chunk?
I have an application which is downloading (through libsoup) and processing a data stream in chunks and doing other actions in parallel. I am calling g_input_stream_read_async on GInputStream (received from soup_session_send_finish and giving it a reasonable size of chunk to read (in my case 2048 bytes).
After I receive a g_input_stream_read_async callback, I want to continue reading bytes. So, the first idea is to recursively call g_input_stream_read_async from the callback handler, passing itself as next callback. But that seems clumsy and not quite right to me (and I'm not sure if it's safe to pass to GIO the callback which is currently still executing).
Alternative could be to spin off a thread and do usual blocking reads in a loop calling g_input_stream_read.
But how it is usually done in GTK / Gnome world? What is the right way? Any simple working example (preferably from developers related to GTK / Gnome) will be appreciated.
After I receive a g_input_stream_read_async callback, I want to continue reading bytes. So, the first idea is to recursively call g_input_stream_read_async from the callback handler, passing itself as next callback.
This is valid and works well.
I need to recode malloc in C for a school project.
I'd like to send a GET request to a server in the malloc function.
All the solutions I've tried, produce a nasty infinite loop when loaded through LD_PRELOAD because they use malloc.
So my question is really simple: is there a way to make a send request using C, without using any of the *alloc functions? I don't care about what the server returns, I just want to send data.
Doing an HTTP transaction--which can take several seconds, and possibly fail, and calls lots of other library functions like malloc--inside your own malloc is never going to work. If you want a documented malloc, the better way to do it is have a piece of code inside your malloc that puts the data to be documented in a queue--fast, no mess. Then, periodically, perhaps in a separate thread or some other slack time in the application, remove the data from the queue and post it.
A nice way to do this is Linux would be to write the data to an open pipe, and have a program on the other end of the pipe do the posting.
Whenever i attempt to write anything on my pendrive, a write system call is generated. What i want to do is, this write call should be trapped and and the user should be requested to input predecided password( which i can define during coding itself).
Please tell me whether this is possible or not? and if yes than how should i do it?
The windows DDK has an example of hooking the file reads/writes/copies in filesys\minifilter, with both pre and post op callbacks, that should have you set for the kernel side of things. For the gui part you'll need something to do a non-blocking spin till the drives signals an event, you'll probably also want a pipe or mapped memory view to pass data around
EasyHook is supposed to give you the ability to hook kernel functions. I have not tried it, so your mileage may vary. Be sure to hook functions cautiously - you may degrade the performance of your machine to a point where it's unusable. What you want is to interact with the user, meaning that you must put the hooked function on hold, and issue a callback into user space. This is probably not an exercise for mere mortals.
At any rate, good luck!
I have a client using select() to check if there is anything to be received, else it times out and the user is able to send(). Which works well enough. However, the program locks up waiting for user input, so it cant recv() again until the user has sent something.
I am not having much luck with using threads either, as I cannot seem to find a good resource, which shows me how to use them.
I have tried a creating two threads (using CreateThread) for send and recv functions which are basically two functions using while loops to keep sending and receiving. And then the two CreateThreads() are wrapped up in a while loop, because otherwise it just seemed to drop out.
I have basically no previous experience with threads, so my description of what i've been doing will probably sound ridiculous. But I would appreciate any help, in using them properly for this kind of use, or an alternative method would also be great.
Can't find a good resource on socket programming? Bah. Read the bible:
Unix Network Programming
this has a strong feel of assignment work...
I am not too much into Windows programming but when I did do some earlier, there used to be a kbhit (probably) function that allowed you to check if the user had sent any inputs. You could look for something similar and try a non-blocking user input check before starting your select again.
You could then get this working without multi-threading
(unless, er, you have to use that).