I need to modify a network adapter driver to increase its performance for my use, and I need a huge physical memory chunk to be contiguous.
I will need several of these chunks based on the number of ports. Each chunk should be around 64MB.
Currently I am looking at my option to be CMA and bootmem.
Is there any other option for same and I haven't used any of it till date so can someone give me a direction on how to use it? as in are there inbuilt functions to manage this allocated memory or will I have to manage it all in my driver?
There is program running which has globally initialized large arrays. I want to verify which index of that large array is getting accessed from another program running in the same machine simultaneously.
I saw how to get a set of pages in the physical memory for a given process proc_id and I also saw how the physical memory of system can be accessed directly. But how to verify which block is getting accessed currently.
How to access the global data segment access pattern of another process is the question. If anyone can direct me to any code example or any api that has this functionality it would be helpful.
I'm developing a kernel PnP driver to map my FPGA. I need four 32Mb buffer of contiguous memory as I use a non scatter gather DMA. Right now I have a problem allocating them with WdfCommonBufferCreate as sometime it works, sometimes don't. I don't understand why the allocation fails sporadically as the device memory and devices does not changes.
Is there a way to ensure my buffer will be created? What can cause the sporadic fail?
I also thought removing 128Mb from Windows with Bcdedit and use the space left for my buffer. I have no problem doing that since the driver is for a specific platform in a controlled environnement but I did not found a way to know memory size with Windows Driver API.
Is there a way to know the size of actual memory? Can I actually use the memory left and if yes, how?
Thanks for your help
That's a lot of contiguous memory. The Windows Driver Framework can break up large DMA transactions into a size your driver can handle, if you tell it the maximum amount of scatter/gather descriptors you have through WdfDmaEnablerSetMaximumScatterGatherElements. Just use a smaller fixed number of scatter/gather elements.
I would like to know how memory can be protected without MMU support. I have tried to google it, but have not seen any worthwile papers or research on it. And those which deal with it only deals it for bugs, such as uninitialized pointers and not memory corruption due to a soft error, that is, due to a hardware transient fault corrupting an instruction that writes to a memory location.
The reason I want to know this is because I am working on a proprietary manycore platform without any Memory Protection. Now my question is, can software be used to protect memory, especially for wild writes due to soft erros (as opposed to mistakes by a programmer). Any help on this would be really appreciated.
If you're looking for Runtime memory protection the sane only option is hardware support. Hardware is the only way to intervene in a bad memory access before it can cause damage. Any software solution would be vulnerable to the very memory errors it is trying to protect against.
With software you could possibly implement a verification/detection scheme. You could periodically check portions of memory that the currently running program should not have access and see if they have changed (probably by CRCing these areas). But of course if the rogue program damages the area where the checksums are held, or where the checking program's code is held, then all bets are off.
Even this software checking solution would be more of a debugging utility than a permanent runtime protection. It is likely that a device with no MMU is a small embedded device which won't have the spare cycles to be constantly checking the device's memory.
Usually devices without MMUs are designed to run a single program with no kernel or anything else, and thus there is nothing to protect. If you need to run multiple programs and feel you need protection, you probably need a more advanced piece of hardware that supports the kind of features you're looking for.
If you want software implemented memory protection, then you will need support from your compiler and its associated libraries. I expect that there is one compiler only on this platform and so you should contact the vendor. I wouldn't hold out much hope for a positive response. Even if they had such tools, I would expect the performance of software memory protection to be unacceptable.
MMU less systems are present in several embedded solutions.
The memory is managed by the kernel code. The entire memory (heap) is divided into heap lists of various sizes (heap lists can be of sizes 4 bytes, 8 bytes, 16 bytes ..... upto 1024 bytes)and there's a header attached to each heap block that tells whether the particular heap block is taken or not. So, that when u need to assign a new heap block, you can browse through the heap lists and see which heap blocks are free and can assign them to the requesting application. And the same is the case when you free a particular sized heap block, the headers of that block are updated to reflect that it has been freed.
Now, this implementation has to take care of the scenario when the application requested a particular size of heap block and that size of heap list is full. In that case you break up a block from the next size of heap list or join together smaller sized heap blocks and add to the requested sized heap list.
The implementation is much simpler than it seems.
Depends on what application platform will run. There is technology called Type-Safe Language (ATS, for instance) which can protect from software errors. And such languages may have good performance (again ATS, for instance).
Does any know a good rule of thumb for the appropriate pagefile size for a Windows 2003 server running SQL Server?
With all due respect to Remus (whom I respect greatly), I strongly disagree. If your page file is large enough to support a full dump, it will perform a full dump every time. If you have a very large amount of RAM, this can cause a tiny blip to became a major outage.
You do NOT want your server to have to write out 1 TB of RAM to disk if there is a one-time transient issue. If there is a recurring issue, you can increase the page file to capture a full dump. I would wait to do this until you have been isntructed by PSS (or someone else qualified to analyze a full dump) request you to capture a full dump. An extremely small percentage of DBAs know how to analyze a full dump. A mini-dump is sufficent for troubleshooting most issues that pop up anyway.
Plus, if your server is configured to allow a 1 TB full dump and a recurring issue occurs, how much free disk space would you recommend having on hand? You could fill up an entire SAN in a single weekend.
A page file 1.5*RAM was the norm back in the days when you were lucky to have a SQL Server with 3 or 4 GB of RAM. This is not the case any more. I leave the page file at Windows default size and settings on all production servers (except for an SSAS server that is experiencing memory pressure).
And just for clarification, I've worked with servers ranging from 2 GB of RAM to 2 TB of RAM. After more than 11 years, I have only had to increae the paging file to capture a full dump one time.
Irrelevant of the size of the RAM, you still need a pagefile at least 1.5 times the amount of physical RAM. This is true even if you have a 1 TB RAM machine, you'll need 1.5 TB pagefile on disk (sounds crazy, but is true).
When a process asks MEM_COMMIT memory via VirtualAlloc/VirtualAllocEx, the requested size needs to be reserved in the pagefile. This was true in the first Win NT system, and is still true today see Managing Virtual Memory in Win32:
When memory is committed, physical
pages of memory are allocated and
space is reserved in a pagefile.
Bare some extreme odd cases, SQL Server will always ask for MEM_COMMIT pages. And given the fact that SQL uses a Dynamic Memory Management policy that reserves upfront as much buffer pool as possible (reserves and commits in terms of VAS), SQL Server will request at start up a huge reservation of space in the pagefile. If the pagefile is not properly sized errors 801/802 will start showing up in SQL's ERRORLOG file and operations.
This always causes some confusion, as administrators erroneously assume that a large RAM eliminates the need for a pagefile. In truth the contrary happens, a large RAM increases the need for pagefile, just because of the inner workings of the Windows NT memory manager. The reserved pagefile is, hopefully, never used.
According to Microsoft, "as the amount of RAM in a computer increases, the need for a page file decreases." The article then goes on to describe how to use Performance Logs to determine how much of the page file is actually being used. Try setting your page file to 1.5X system memory for a start, then do the recommended monitoring and make adjustments from there.
How to determine the appropriate page file size for 64-bit versions of Windows
The bigger the better up to the size of the working set of the application where you will start to get into diminishing returns. You can try to find this by slowly increasing or decreasing the size until you see a significant change in cache hit rates. However, if the cache hit rate is over 90% or so you're probably OK. Generally you should keep an eye on this on a production system to make sure it hasn't outgrown its RAM allocation.
We were recently having some performance issues with one of our SQL Server that we weren't able to completely narrow down, and actually used one of our Microsoft support tickets to have them help troubleshoot. The optimal pagefile size to use with SQL Server came up, and Microsoft's recommendation is that it be 1 1/2 times the amount of RAM.
In this case, the normal recommendation of 1.5 times total physical RAM is not the best. This very general recommendation is provided under the assumption that all memory is being used by "normal" processes, which can generally have their least-used pages moved to disk without generating massive performance issues for the application process the memory belongs to.
For servers running SQL Server (generally with very large amounts of RAM), the majority of the physical RAM is committed to the SQL Server process and should be (if configured correctly) locked in physical memory, preventing it from being paged out to the pagefile. SQL Server manages its own memory very carefully with performance in mind, using a large part of the RAM allocated to its process as a data cache to reduce disk I/O. It does not make sense to page out those data cache pages to the pagefile, as the sole purpose of having that data in RAM in the first place is to reduce disk I/O. (Note that the Windows OS also uses available RAM similarly as disk cache to speed up system operation.) Since SQL Server already manages its own memory space, this memory space should not be considered "pageable", and not included in a calculation for pagefile size.
In regard to MEM_COMMIT mentioned by Remus, the terminology is confusing because in the virtual memory parlance, "reserved" never refers to actual allocation, but to preventing use of an address space (not physical space) by another process. Memory available to be "committed" is basically equal to the sum of physical RAM and pagefile size, and doing a MEM_COMMIT just decrements the amount available in the committed pool. It does not allocate a matching page in the pagefile at that time. When a committed memory page is actually written to, that is when the virtual memory system will allocate a physical memory page and possibly bump another memory page from physical RAM to the pagefile. See MSDN's VirtualAlloc function reference.
The Windows OS keeps track of memory pressures between application processes and its own disk cache mechanism and decides when it should bump non-locked memory pages from physical to the pagefile. My understanding is that having a pagefile that is way too large compared to the actual non-locked memory space can result in Windows overzealously paging out application memory to the pagefile, resulting in those applications suffering the consequences of page misses (slow performance).
As long as the server is not running other memory-hungry processes, a pagefile size of 4GB should be plenty. If you have set SQL Server to allow locking pages in memory, you should also consider setting SQL Server's max memory setting so that it leaves some physical RAM available to the OS for itself and other processes.
802 errors in SQL Server indicate that the system cannot commit any more pages for the data cache. Increasing the pagefile size will only help in this situation insofar as Windows is able to page out memory from non-SQL Server processes. Allowing SQL Server memory to grow into the pagefile in this situation might get rid of the error messages, but it is counterproductive, due to the point earlier about the reason for the data cache in the first place.
If you're looking for high performance, you are going to want to avoid paging completely, so the page file size becomes less significant. Invest in as much RAM as feasible for the DB server.
After much research our dedicated SQL Servers running Enterprise x64 on Windows 2003 Enterprise x64 have no page file.
Simply, the page file is a cache for files that gets managed by the OS, and SQL has it's own internal memory management system.
The MS article referenced does not qualify that the advice is for the OS running out-of-the-box services such as file sharing.
Having a page file simply burdens the disk I/O because Windows is trying to help, when only the SQL OS can do the job.