I'd love to know where some of the hidden functions that aren't officially in the open kernel are documented.
The current WinAPI and NTAPI documentation lacks these functions.
#include <ntapi.h>
Nothing is even documented in the files.
There are plenty of functions in the NTAPI that are not documented. Primarily because they could be subject to change at any point during Microsoft's implementation. An example like NtCreateProcess I believe is undocumented. NtQueryProcessInformation. NtdelayExecution. Many many more. One area you can look for undocumented functions is here.
NtInternals Also reversing kernel modules or even ntdll can help you discover some undocumented functions.
Related
I was looking at the NtDll export table on my Windows 10 computer, and I found that it exports standard C runtime functions, like memcpy, sprintf, strlen, etc.
Does that mean that I can call them dynamically at runtime through LoadLibrary and GetProcAddress? Is this guaranteed to be the case for every Windows version?
If so, it is possible to drop the C runtime library altogether (by just using the CRT functions from NtDll), therefore making my program smaller?
There is absolutely no reason to call these undocumented functions exported by NtDll. Windows exports all of the essential C runtime functions as documented wrappers from the standard system libraries, namely Kernel32. If you absolutely cannot link to the C Runtime Library*, then you should be calling these functions. For memory, you have the basic HeapAlloc and HeapFree (or perhaps VirtualAlloc and VirtualFree), ZeroMemory, FillMemory, MoveMemory, CopyMemory, etc. For string manipulation, the important CRT functions are all there, prefixed with an l: lstrlen, lstrcat, lstrcpy, lstrcmp, etc. The odd man out is wsprintf (and its brother wvsprintf), which not only has a different prefix but also doesn't support floating-point values (Windows itself had no floating-point code in the early days when these functions were first exported and documented.) There are a variety of other helper functions, too, that replicate functionality in the CRT, like IsCharLower, CharLower, CharLowerBuff, etc.
Here is an old knowledge base article that documents some of the Win32 Equivalents for C Run-Time Functions. There are likely other relevant Win32 functions that you would probably need if you were re-implementing the functionality of the CRT, but these are the direct, drop-in replacements.
Some of these are absolutely required by the infrastructure of the operating system, and would be called internally by any CRT implementation. This category includes things like HeapAlloc and HeapFree, which are the responsibility of the operating system. A runtime library only wraps those, providing a nice standard-C interface and some other niceties on top of the nitty-gritty OS-level details. Others, like the string manipulation functions, are just exported wrappers around an internal Windows version of the CRT (except that it's a really old version of the CRT, fixed back at some time in history, save for possibly major security holes that have gotten patched over the years). Still others are almost completely superfluous, or seem so, like ZeroMemory and MoveMemory, but are actually exported so that they can be used from environments where there is no C Runtime Library, like classic Visual Basic (VB 6).
It is also interesting to point out that many of the "simple" C Runtime Library functions are implemented by Microsoft's (and other vendors') compiler as intrinsic functions, with special handling. This means that they can be highly optimized. Basically, the relevant object code is emitted inline, directly in your application's binary, avoiding the need for a potentially expensive function call. Allowing the compiler to generate inlined code for something like strlen, that gets called all the time, will almost undoubtedly lead to better performance than having to pay the cost of a function call to one of the exported Windows APIs. There is no way for the compiler to "inline" lstrlen; it gets called just like any other function. This gets you back to the classic tradeoff between speed and size. Sometimes a smaller binary is faster, but sometimes it's not. Not having to link the CRT will produce a smaller binary, since it uses function calls rather than inline implementations, but probably won't produce faster code in the general case.
* However, you really should be linking to the C Runtime Library bundled with your compiler, for a variety of reasons, not the least of which is security updates that can be distributed to all versions of the operating system via updated versions of the runtime libraries. You have to have a really good reason not to use the CRT, such as if you are trying to build the world's smallest executable. And not having these functions available will only be the first of your hurdles. The CRT handles a lot of stuff for you that you don't normally even have to think about, like getting the process up and running, setting up a standard C or C++ environment, parsing the command line arguments, running static initializers, implementing constructors and destructors (if you're writing C++), supporting structured exception handling (SEH, which is used for C++ exceptions, too) and so on. I have gotten a simple C app to compile without a dependency on the CRT, but it took quite a bit of fiddling, and I certainly wouldn't recommend it for anything remotely serious. Matthew Wilson wrote an article a long time ago about Avoiding the Visual C++ Runtime Library. It is largely out of date, because it focuses on the Visual C++ 6 development environment, but a lot of the big picture stuff is still relevant. Matt Pietrek wrote an article about this in the Microsoft Journal a long while ago, too. The title was "Under the Hood: Reduce EXE and DLL Size with LIBCTINY.LIB". A copy can still be found on MSDN and, in case that becomes inaccessible during one of Microsoft's reorganizations, on the Wayback Machine. (Hat tip to IInspectable and Gertjan Brouwer for digging up the links!)
If your concern is just the need to distribute the C Runtime Library DLL(s) alongside your application, you can consider statically linking to the CRT. This embeds the code into your executable, and eliminates the requirement for the separate DLLs. Again, this bloats your executable, but does make it simpler to deploy without the need for an installer or even a ZIP file. The big caveat of this, naturally, is that you cannot benefit to incremental security updates to the CRT DLLs; you have to recompile and redistribute the application to get those fixes. For toy apps with no other dependencies, I often choose to statically link; otherwise, dynamically linking is still the recommended scenario.
There are some C runtime functions in NtDll. According to Windows Internals these are limited to string manipulation functions. There are other equivalents such as using HeapAlloc instead of malloc, so you may get away with it depending on your requirements.
Although these functions are acknowledged by Microsoft publications and have been used for many years by the kernel programmers, they are not part of the official Windows API and you should not use of them for anything other than toy or demo programs as their presence and function may change.
You may want to read a discussion of the option for doing this for the Rust language here.
Does that mean that I can call them dynamically at runtime through
LoadLibrary and GetProcAddress?
yes. even more - why not use ntdll.lib (or ntdllp.lib) for static binding to ntdll ? and after this you can direct call this functions without any GetProcAddress
Is this guaranteed to be the case for every Windows version?
from nt4 to win10 exist many C runtime functions in ntdll, but it set is different. usual it grow from version to version. but some of then less functional compare msvcrt.dll . say for example printf from ntdll not support floating point format, but in general functional is same
it is possible to drop the C runtime library altogether (by just using
the CRT functions from NtDll), therefore making my program smaller?
yes, this is 100% possible.
I understand it's necessary to "load" functions for opengl by having special functions locate pointers to the function based on the name of the opengl function. I've never seen something similar to this done before and I'm wondering how this works. Where are the functions actually located? How are they retrieved? Why is it done this way?
Where are the functions actually located?
This is implementation dependent. I'd wager that they're stored in a hash table of function names to function pointers. They're still in the shared library, but usually don't have their symbols exposed.
How are they retrieved?
glXGetProcAddress or wglGetProcAddress, depending on the platform. Most libraries that create OpenGL windows (GLFW, SDL) have their own, cross-platform function that uses the above.
Why is it done this way?
I can think of a few reasons: implementations can change what extensions are available without breaking ABI compatibility, and what functions you have access to may depend on the type or version of context you request (ex. for Mesa, post OpenGL 3.1 extensions are only available in a 3.1+ context, not in anything lower).
This is a curiosity question for anybody who has worked for, known somebody who's worked for, or otherwise had any sort of affiliation with the Microsoft team responsible for defining these macros.
I understand what __stdcall is and I know why it's used, I just don't understand why Microsoft would make three separate macros for the same thing. The only benefit I could see would be to provide some semantic meaning in a source file, but other than that it provides no other benefit as far as I can tell. Obviously there was a point to doing it, I just want to know what it is! :)
It seems you're right about the different macros being used to provide semantic information. MSDN says this about CALLBACK:
CALLBACK, WINAPI, and APIENTRY are all used to define functions with
the __stdcall calling convention. Most functions in the Windows API
are declared using WINAPI. You may wish to use CALLBACK for the
callback functions that you implement to help identify the function as
a callback function.
Both WINAPI and APIENTRY are said to be:
The calling convention for system functions.
I don't know why there's two macros for system functions.
The link to ooga's answer is https://learn.microsoft.com/en-us/windows/win32/winprog/windows-data-types
More details are provided by Microsoft at 'Understanding SAL':
"The Microsoft source-code annotation language (SAL) provides a set of annotations that you can use to describe how a function uses its parameters, the assumptions that it makes about them, and the guarantees that it makes when it finishes. The annotations are defined in the header file <sal.h>. Visual Studio code analysis for C++ uses SAL annotations to modify its analysis of functions. For more information about SAL 2.0 for Windows driver development, see SAL 2.0 Annotations for Windows Drivers.
Natively, C and C++ provide only limited ways for developers to consistently express intent and invariance. By using SAL annotations, you can describe your functions in greater detail so that developers who are consuming them can better understand how to use them."
My program is written in C. I want to use library winnt.h, but I don't use Windows anymore.
Seems like a strange question; you should probably clarify which function(s) you actually need from winnt.h so that you can learn the Linux equivalent. winnt.h isn't really a general purpose "library", it's just an interface to built in Windows-specific functions.
With that as a major caveat, you may get some degree of what you want by attempting to run your app with the help of Wine. See http://www.winehq.org/ If you're just trying to run an existing app, that's may be a reasonable solution. If you're trying to make a Linux version of your app, though, that won't help you very much.
No, well you could but it's not going to do any good - the.h file just declares functions that are defined in libs that are only on windows
No. You can't.
winnt.h contains lots of macros that depend on a Windows environment and a lot of function declarations that only exist in Windows-specific libraries. So, it's not really useful (or possible) to use winnt.h on Linux.
That said, you can use Winelib, which includes most of the functionality exposed by those Windows-specific headers, and you can get those features by linking your program with Winelib. In general, this is probably not a good idea, because Winelib is relatively unstable (the functionality of a given API function may be absent, incomplete, buggy, or incompatible compared to the native Windows version). It is a much better idea to look for a Linux-native alternative to what you need.
What parts of winnt.h do you want to use? Of course, if you need some nice macroses or type definitions from it, you can freely copy it to your own header file (of course, with dependencies). But if you include all winnt.h file to your program in linux environment, you will get tons of error messages. One of the reasons for it is pronounced by Martin Beckett in his reply.
I want to write a C program which makes use of the linux crypto-api for digital signatures. Unfortunately I cannot find good documentation about the linux api and the functions defined in linux/crypto.h (googling doesn't help, man pages for those functions don't exist). So now I wonder if anyone here can help me with a good link, a book would also be appreciated.
2nd short question: All the time within this crypto stuff the term "tfm" comes up, as in a struct crypto_tfm or in functions, but I can't find out what tfm actually means.
The Linux crypto-API is an internal kernel API used for things such as IPsec and dm-crypt. It's not directly usable by user-mode applications. If you want to use it in a kernel driver, read the headers, and look at these files: http://lxr.linux.no/linux+v2.6.37/Documentation/crypto
In general, anything under the linux/ directory should not be used by ordinary applications directly. Anything in there you're allowed to use is re-exported under a sys/ include somewhere, and if it's a function it'll come with a manual page as well. Kernel functions are designed with the assumption that you will read the kernel source code (since you're working on the kernel, right?) and so the API details are documented primarily in the header and source files themselves.
Finally, "tfm" stands for "transformation".
This might be enlightening http://thesweeheng.files.wordpress.com/2007/11/6451.pdf. It explains some of the details regarding the design of the kernel crypto api.