Arch=x86_64
I am working through a DMA solution following the process outlined in this question,
Direct Memory Access in Linux
My call to ioremap successfully returns with an address, pt.
In my call to remap_pfn_range I use, virt_to_phys(pt) >> PAGE_SHIFT, to specify the pfn of the area generated by the ioremap call.
When the userspace application using mmap executes and the call to remap_pfn_range is made, the machine crashes. I assume the mapping is off and I am forcing the system to use memory that is already allocated (screen glitches before exit), however I'm not clear on where the mismatch is occurring. The system has 4 Gigs of Ram and I reserved 2Gigs by using the kernel boot option mem=2048M.
I use BUFFER_SIZE=1024u*1024u*1024u and BUFFER_OFFSET=2u*1024u*1024u*1024u.
putting these into pt=ioremap(BUFFER_SIZE,BUFFER_OFFSET) I believe pt should equal a virtual address to the physical memory located at the 2GB boundary up to the 3GB boundary. Is this assumption accurate?
When I execute my kernel module, but I change my remap_pfn_range to use vma->vm_pgoff>>PAGE_SHIFT as the target pfn the code executes with no error and I can read and write to the memory. However this is not using the reserved physical memory that I intended.
Since everything works when using vma->vm_pgoff>>PAGE_SHIFT I believe my culprit is between my ioremap and the remap_pfn_range
Thanks for any suggestions!
The motivation behind the use of this kernel module is the need for large contiguous buffers for DMA from a PCI device. In this application, recompiling the kernel isn't an option so I'm trying to accomplish it with a module + hardware.
My call to ioremap successfully returns with an address, pt.
In my call to remap_pfn_range I use, virt_to_phys(pt) >> PAGE_SHIFT,
to specify the pfn of the area generated by the ioremap call.
This is illegal, because ioremap reserves virtual region in vmalloc area. The virt_to_phys() is OK only for linearly mapped part of memory.
putting these into pt=ioremap(BUFFER_SIZE,BUFFER_OFFSET) I believe pt
should equal a virtual address to the physical memory located at the
2GB boundary up to the 3GB boundary. Is this assumption accurate?
That is not exactly true, for example on my machine
cat /proc/iomem
...
00001000-0009ebff : System RAM
...
00100000-1fffffff : System RAM
...
There may be several memory banks, and the memory not obligatory will start at address 0x0 of physical address space.
This might be usefull for you Dynamic DMA mapping Guide
Related
I am trying to make use of a contiguous memory i reserved while passing the "mem" parameter to Linux when booting.
Now, i have the physical address of this space i reserved earlier, and the length of it, and i wish to make use of this reserved space for DMA purposes in my driver.
Normally i would use dma_alloc_coherent() , and if i were using CMA i would use that too, but in this case, its different.
Now, i have read that an acceptable way of mapping a physical space to kernel virtual space is to use ioremap
And, an acceptable way of "taking over" a contiguous space for DMA purposes is to use dma_map_single (mapping it for bus address)
I'm having trouble combining the two. ioremap works and returns a virtual address. Now, i have read that this is no ordinary virtual address and i should only be using access methods to read/write from this memory.
Thing is, when i try to pass this virtual address to dma_map_single , it doesn't report an error, but i suspect that this is wrong.
Am i doing it right? What can i do to make it work like it should?
10x
You are doing right
You don't need to allocate the memory because you already set it on boot time but you need to use dam_map_single to prevent cache problems for example if you want to do a DMA from memory to the device but the RAM is not synchronised with the L2 cache (the cache has a newer version) you will get the wrong data so you need to map and unmap before and after the DMA operation
I'm on 32bit machine. From what I understand, User space's address ranges from 0x00000000 to 0xbfffffff, and kernel's ranges from 0xc0000000 to 0xffffffff.
But when I used pmap to see a process's memory allocation, I see that the library are loaded in around 0xf7777777. Please see the attached screenshot. Does it mean those libraries are loaded in kernel space? And when I used mmap(), I got the address from 0xe0000000. So, mmap() got memory from kernel space?
I'm on 32bit machine. From what I understand, User space's address
ranges from 0x00000000 to 0xbfffffff, and kernel's ranges from
0xc0000000 to 0xffffffff.
Not exactly. Kernel memory space starts at 0xC0000000, but it doesn't have to fill the entire GB. In fact, it fills up to virtual address 0xF7FFFFFF. This covers 896MB of physical memory. Virtual addresses 0xF8000000 and up are used as a 128MB window for the kernel to map any region of physical memory beyond the 896MB limit.
All user processes share the same memory map for virtual addresses 0xC0000000 and beyond, so if the kernel does not use its entire GB of virtual space, it may reuse part of it to map commonly used shared libraries, so every process can see them.
I understand if I try to print the address of an element of an array it would be an address from virtual memory not from real memory (physical memory) i.e DRAM.
printf ("Address of A[5] and A[6] are %u and %u", &A[5], &A[6]);
I found addresses were consecutive (assuming elements are chars). In reality they may not be consecutive at least not in the DRAM. I want to know the real addresses. How do I get that?
I need to know this for either Windows or Linux.
You can't get the physical address for a virtual address from user code; only the lowest levels of the kernel deal with physical addresses, and you'd have to intercept things there.
Note that the physical address for a virtual address may not be constant while the program runs — the page might be paged out from one physical address and paged back in to a different physical address. And if you make a system call, this remapping could happen between the time when the kernel identifies the physical address and when the function call completes because the program requesting the information was unscheduled and partially paged out and then paged in again.
The simple answer is that, in general, for user processes or threads in a multiprocessing OS such as Windows or Linux, it is not possible to find the address even of of a static variable in the processor's memory address space, let alone the DRAM address.
There are a number of reasons for this:
The memory allocated to a process is virtual memory. The OS can remap this process memory from time-to-time from one physical address range to another, and there is no way to detect this remaping in the user process. That is, the physical address of a variable can change during the lifetime of a process.
There is no interface from userspace to kernel space that would allow a userspace process to walk through the kernel's process table and page cache in order to find the physical address of the process. In Linux you can write a kernel module or driver that can do this.
The DRAM is often mapped to the procesor address space through a memory management unit (MMU) and memory cache. Although the MMU maping of DRAM to the processor address space is usually done only once, during system boot, the processor's use of the cache can mean that values written to a variable might not be written through to the DRAM in all cases.
There are OS-specific ways to "pin" a block of allocated memory to a static physical location. This is often done by device drivers that use DMA. However, this requires a level of privilege not available to userspace processes, and, even if you have the physical address of such a block, there is no pragma or directive in the commonly used linkers that you could use to allocate the BSS for a process at such a physical address.
Even inside the Linux kernel, virtual to physical address translation is not possible in the general case, and requires knowledge about the means that were used to allocate the memory to which a particular virtual address refers.
Here is a link to an article called Translating Virtual to Physical Address on Windows: Physical Addresses that gives you a hint as to the extreme ends to which you must go to get physical addresses on Windows.
I have a PCI device which exposes a BAR and few offsets in the bar for accessing the device.
On one of the Bar offset, i need to program a 64KB allocated memory. In my Linux driver, i allocate a 64KB of memory using kmalloc() which as i know returns virtual address. If this is programmed into the offset, HW won't be able to see the same. How do i convert this virtual address to physical ?
When i Google, i see few links pointing to virt_to_phys() but few responses says this doesn't work well with kmalloc(). Any idea how to go about this?
You normally use pci_resource_start() / pci_resource_end() from within a kernel driver. I assume you are writing a device driver ?
I won't map the memory yourself : That's where the kernel functions are for. That way, you're sure it works on all platforms. That 64k block I assume is some memory map that is provided by the PCI device ? if yes, then the above is correct. If no, please give more information.
Rather than using kmalloc() , use alloc_pages() function.
struct page* alloc_pages(gfp_t gfp_mask, 4 );
one page is of 4K, so it will allocate 2^4=16 pages which is equal to 16 * 4K = 64K memory and returns physical address.
I reserve the memory chunk using a memmap=8G$4G linux kernel boot parameter.
Is it needed to ioremap this memory ?
ioremap man pages say :
ioremap performs a platform specific sequence of operations to make
bus memory CPU accessible via the readb/readw/readl/writeb/
writew/writel functions and the other mmio helpers. The returned
address is not guaranteed to be usable directly as a virtual address.
So if i can't use the returned address of ioremap as a virtual address for directly addressing the memory, then a broader question is when should we ioremap the memory ?
Yes, you have to ioremap this region to access it. The kernel does not set up the Page Directory Entries for this memory region as you instructed the kernel to ignore this region.
The addresses returned by ioremap may not be used directly if you remapped the addresses of io-port address space. When you remap the addresses from the memory address space then it is OK to use them directly.
However, please take look at https://unix.stackexchange.com/questions/37729/how-can-i-reserve-a-block-of-memory-from-the-linux-kernel
As per my experience with reserving ( or blocking) the memory is as followed.
if you are trying to reserve a particular volume of memory you may have to remap the already existing memory map provided by BIOS.
If your system doesnt enable you to do so then you will have to identify which area is free in the BIOS provided memory map and only that can be reserved.