Ive seen few other posts in stack, but none really answered my question.
Is it possible to emulate or somehow map or have a driver that emulates specific memory addresses that otherwise dont exist on host machine?
Im looking at piece of embedded code that i need to troubleshoot, but its reading values from memory space that doesn't exist on my pc. I could change this one instance of address, but those nonexistent addressees are all over the place on the code, so changing them all just isn't practical.
It could be possible to make the code run for testing purposes, but how easy it would be depends on two things: How exactly is the code referencing the addresses and what are the memory accesses supposed to do?
Generally speaking, you could write a library to mock the other hardware. Writing such a library could be anything between easy and difficult, depending on what those accesses are supposed to do. In an easy case, the accesses wouldn't affect each other in a way that would need to be emulated. In a more difficult case, you might need some kind of a state machine to track what is going on. In a medium case, there might be a need to remember some values from earlier accesses but no need for a more complex state machine.
Writing a library may not be enough, though. You would also need to make the code that you are troubleshooting use that library. In practice, the code needs to call functions offered by the library. If the memory accesses are already done via some function calls, you could simply have your own versions of those functions in your library. Otherwise, more effort is needed.
If accesses are not via function calls but by directly referencing the addresses, one solution might be to try writing regular expressions to match the references. If you had regular expressions matching all or at least most of them, you could use them and some tool like sed to replace the references with function calls more or less automatically.
Related
Suppose that I have a C library that requires initialization and cleanup functions that aren’t thread-safe. Specifically, these functions may invoke other thread-unsafe functions in other libraries. I don’t know (in a default build) which libraries these will be.
Now consider the case of writing Java bindings to this library. Java spawns multiple threads before running any Java code. Worse, in the case of (say) an Eclipse plugin, there could be multiple threads running Java code by the time my code receives control. Some of the other threads could be using the aforementioned unsafe functions.
My current plan is to statically link the C library (in my case, libcurl) and all transitive dependencies – in my case, a TLS library (probably mbedTLS) and (on Windows platforms) the CRT. Fortunately, libcurl cleans up everything it has allocated, so problems related to allocating from one heap and freeing it on another should not arise. Because everything is statically linked, and won’t try to load any other shared libraries, I can then initialize libcurl from a static initializer.
Will this even work? Is there a better way?
Edit: The reason that serializing library calls won’t work, and that I believe that my solution might work, is that the global state is stored not only in libcurl itself, but also in libraries libcurl depends on. Some of these libraries (ex. OpenSSL) might be in use by other code when my code is loaded. So I would need to lock against the entire process.
The reason I believe that isolating the global state would work is that libcurl (and every library it depends on) is thread safe after initialization. I need to make sure that the initialization of libcurl doesn’t create race conditions. Afterwards I am fine.
[Updated and revised]
Your concern seems to be that you will have both direct and indirect bindings to some native library -- say mbedTLS --, that that native library requires one-time initialization that is not thread-safe, and that, beyond your ability to detect or control, different threads of the process may concurrently attempt to initialize that library, or perhaps may (unsafely) attempt to initialize it more than once. That certainly seems to be a worst-case scenario.
On the other hand, you postulate that you can successfully build a monolithic, dynamically-loadable library containing the native library you want along with the transitive closure of all its dependencies (outside the kernel), so as to ensure that this library does not share state with any other library loaded by the process. You assert that after a non-thread-safe initialization, the combined stack will be thread safe, at least as you intend to use it. You want to know about how to initialize the library.
Java promises that each class will be initialized by exactly one thread, and that afterward its initialized state will be visible to all threads. Although that does not explicitly address the question, it certainly implies that if the initialization of your native libraries is performed entirely as part of the initialization of a class -- e.g. via a static initializer, as you propose -- then the correct initialized state will be visible to all Java threads. That adequately addresses the problem as I understand it.
I remain dubious that building the monolithic library is necessary, but if you truly have to deal with the worst-case scenario you seem to anticipate then perhaps it is. Inasmuch as you cannot isolate the library from conflicting demands on the kernel, however, it is conceivable that the strategy will not be sufficient. That would be one of the few conceivable good reasons for a library to rely on the kind of shared state you postulate, and your strategy would thwart that particular purpose. I cannot judge how probable such an eventuality might be, but I doubt it's very likely.
I'm implementing a pthread replacement library which is extremely lightweight. There are several reasons why I want to disable __thread completely.
It's a waste of memory. If I'm creating a thousand threads which has nothing to do with the context that declares a variable with __thread they will still allocate the program will still have allocated 1000*the size of that data bytes and never use it. It's simply not memory compatible with the mass-concurrency model. If we need extremely lightweight fibers with only 8K of stack, a TLS block of just 4K would be an overhead of 50% of the memory used by each thread. In some scenarios the TLS overhead would be enormous.
TLS is a complex standard and I simply don't have the time/resources to support it. It's to expensive. Personally I think the standard is poorly designed. It should have defined standard functions that had to be provided by the linker so thread libraries can take control over where TLS allocation takes place and insert relevant offsets and addresses it requires. Also the standard ELF implementation has been infected with pthread, expecting pthread sized structs to calculate offsets making it really hard to adapt to something else.
It's just a bad pattern. It encourages using globals and creating functions with static functions/side effects. This is not a territory we want to be in if we're creating correct programs that are easy to analyze.
If we really need "thread context" for some magic that tracks thread state behind the scenes (like for allocation or cancellation tracking) why not just expose the magic that TLS uses to understand that context in the first place? Personally I'm just going to use the %fs register directly. This would not be possible in libraries for obvious reasons but why should they be thread aware to begin with? Why not just design them correctly so they get the context related data they need in the first place right in the argument list?
My question is simply: What is the simplest way to disable __thread support and make clang emit errors if you accidentally used it? How can I get errors if I load a dynamic library which happens to require TLS?
I believe the simplest way is adding something like this unconditionally to your CFLAGS (perhaps from clang's equivalent of the gcc specfile if you want it to be system-global):
-D__thread='^-^'
where the righthand side can be anything that's syntactically invalid (a constraint violation) at any point in a C program.
As for preventing loading of libraries with TLS, you'd have to patch the linker and/or dynamic linker to reject them. If you're just talking about dlopen, your program could first read the file and parse the ELF headers for TLS relocations, then reject the library (without passing it to dlopen) if it has any. This might even be possible with an LD_PRELOAD wrapper.
I agree with you that, especially in its current implementation, TLS is something whose use should generally be avoided, but may I ask if you've measured the costs? I think stamping it out completely will be fairly difficult on a system that's designed to use it, and there's much lower-hanging fruit for cutting bloat. Which libc are you using? If it's glibc, I'm pretty sure glibc has lots of TLS it uses internally itself these days... Of course if you're writing your own threads implementation, that's going to entail a lot of interaction with the rest of the standard library, so perhaps you're already patching it out...?
By the way (shameless plug follows), we have an extremely light-weight threads implementation in musl libc which presently does not have TLS. I don't think it would be easy to integrate with another libc (and I'm sure if you're writing your own you will find it difficult to integrate with glibc, especially glibc's dynamic linker, which expects TLS to be supported) but if you can use the whole library as-is, it may meet your needs for specific projects, or have useful code you can borrow (license is MIT).
I remembering reading this concept somewhere. I do not remember where though.
I have a file say file.c, which along with other files I compile along with some other files as a library for use by applications.
Now suppose i compile the same file and build it with a Kernel module. Hence now the same file object is in both user space and kernel space and it allows me to access kernel data structures without invoking a system call. I mean i can have api's in the library by which applications can access kernel data structures without system calls. I am not sure if I can write anything into the kernel (which i think is impossile in this manner), but reading some data structures from kernel this way would be fine?
Can anyone give me more details about this approach. I could not find anything in google regarding this.
I believe this is a conceptually flawed approach, unless I misunderstand what you're talking about.
If I understand you correctly, you want to take the same file and compile it twice: once as a module and once as a userspace program. Then you want to run both of them, so that they can share memory.
So, the obvious problem with that is that even though the programs come from the same source code, they would still exist as separate executables. The module won't be its own process: it only would get invoked when the kernel get's going (i.e. system calls). So by itself, it doesn't let you escape the system call nonsense.
A better solution depends on what your goal is: do you simply want to access kernel data structures because you need something that you can't normally get at? Or, are you concerned about performance and want to access these structures faster than a system call?
For (1), you can create a character device or a procfs file. Both of these allow your userspace programs to reach their dirty little fingers into the kernel.
For (2), you are in a tough spot, and the problem gets a lot nastier (and more insteresting). To solve the speed issue, it depends a lot on what exact data you're trying to extract.
Does this help?
There are two ways to do this, the most common being what's called a Character Device, and the other being a Block Device (i.e. something "disk-like").
Here's a guide on how to create drivers that register chardevs.
I'm looking for a quick guide to basic dll hooking in windows with C, but all the guides I can find are either not C, or not windows.
(The DLL is not part of windows, but a third party program)
I understand the principle, but I don't know how to go about it.
I have pre-existing source code in C++ that shows what I need to hook into, but I don't have any libraries for C, or know how to hook from scratch.
The detours license terms are quite restrictive.
If you merely want to hook certain functions of a DLL it is often cheaper to use a DLL-placement attack on the application whose DLL you want to hook. In order to do this, provide a DLL with the same set of exports and forward those that you don't care about and intercept the rest. Whether that's C or C++ doesn't really matter. This is often technically feasible even with a large number of exports but has its limitations with exported data and if you don't know or can't discern the calling convention used.
If you must use hooking there are numerous ways including to write a launcher and rewrite the prepopulated (by the loader) IAT to point to your code while the main thread of the launched application is still suspended (see the respective CreateProcess flag). Otherwise you are likely going to need at least a little assembly knowledge to get the jumps correct. There are plenty of liberally licensed disassembler engines out there that will allow you to calculate the proper offsets for patching (because you don't want to patch the middle of a multi-byte opcode, for example).
You may want to edit your question again to include what you wrote in the comments (keyword: "DLL hooking").
loading DLLs by LoadLibrary()
This is well known bad practice.
You might want to look up "witch" or "hctiw", the infamous malware dev. there's a reason he's so infamous - he loaded DLLs with LoadLibrary(). try to refrain from bad practice like that.
If I want to clone a library and change just one function, say memcpy or memmove, and have an already built executable link to it for debugging/exploration purposes, what is the proper way to do this?
I am guessing I need to recompile the entire library with my modifications, but is there another way to do this?
I understand that there are things like malloc hooks but this seems to be a special case for malloc.
I am curious about the specifics for how valgrind and gdb do this from within another program, if someone has a resource on that.
I am interested in mac and linux solutions. On linux I've used LD_LIBRARY_PATH before - is this all that I need to do besides have the library names the same? How would I do this on mac?
For those curious as to why I want to do this, the purpose is for experimental music. I am doing this to sonify memory operations, so memcpy/memmove will work as normal but the data accessed will also be sent to the sound card. I know there are other ways of doing this (I have done a few other methods already,) but currently I am interested in focusing on memcpy/memmove, so I will appreciate it if you can restrict your answers to this focus.
You can use LD_LIBRARY_PATH to cause a program to load a shared object library different from the usual one. But if you want to replace just one function (or a few) rather than a whole library, you can use LD_PRELOAD to cause the linker (ld.so) to load a particular shared object early on, and your program will use the symbols (functions) in there rather than looking for them in the usual places.