I'm using Windriver Jungo for testing PCI express device connected to my PC. I can able to read/write memory in 8/16/32/64 bit modes and the board responding very well. But my doubt is that, according to PCI express specification we can send a TLP packet from 128 bytes to 4096 bytes. How can i achieve this transfer using driver. Now its transfer only 8 byte(64 bit mode) data. If i give more data to sent then it split the data to 8byte packets and then sent. Please clear my doubt and suggest any other driver options for PCI express in windows or Linux ?
You are trying PIO read write, not DMA. PIO read/write TLPs will have only a doubleword length data in a packet. For burst data transfers(which have data up to 4096 bytes but it will be probably limited to 128 byte by your motherboard chipset), you have to initate read/write from your PCIE device to PC, not from PC to PCIE.
I used a windriver to communicate with a PCIE FPGA kit before, it works fine for me at any payload the program specify (128-4K). It might be the way you write to the BAR location. check if you are sending the data to the same BAR address location or different BAR address location.
Related
I am working on stm32f107VCx microprocessor.
my compiler is keil and I am using spl (standard peripheral library).
I can send and recieve data with UDP protocol but I dont know how to save this strings that I have given from UDP.
Actually I want to save this strings into a file in my PC.
My suggestion: connect the STM.. and the PC via a RS-232 bus.
Then have the STM.. send the strings over the bus to the PC.
Have the PC read the RS-232 bus and write the resulting data to some file.
if it were me, I would implement some protocol so the STM.. can tell the PC how many bytes are in each string to be saved and have the STM.. append some checksum so the PC can validate the string.
Suggest the PC reply with a ACK or NAK so the STM.. knows if the transfer of the string was successful (or not)
Do note that the RS-232 bus will be MUCH slower than the UDP communications, so some strings may be lost as the PC cannot keep up with the rate the data is coming into the STM..
Given the UDP is not a guaranteed communication protocol, I would expect the losses would be acceptable.
I'm developing a new Kernel-Mode driver, that should run on Windows 10 (64-bit).
This driver should allocate 48GB of continuous physical memory, and map it (its base address) to a virtual address in the user space of the Windows application that will use it. The system actually has 64GB of RAM installed on it, so it might be needed to make a segment of the memory dedicated for this use, perhaps by changing a boot entry information.
In addition, the driver should reveal its base address somehow to an FPGA-based device, located on the PCIe slot. The purpose of this, is to use this 48GB of continuous physical allocated memory, as a DMA (Direct Memory Access). Namely, the FPGA-based device will generate data, and write it at the appropriate location in the DMA. The host software will try to read from that location, in a cyclic fashion. That is, the FPGA will override the data in the buffer, and the host will try to keep the pace, and read that data before it is overridden.
Please note that this question only deals with the host side (the driver), and not with the FPGA side.
So, basically my questions are:
How do I make such an allocation (as described above)?
How do I map the base address from the virtual address in the Kernel-Space, to the appropriate virtual address in the User-Space?
How do I reveal that base address to the FPGA (located on the PCIe slot), so it will know where to perform its write operations?
What other Callback-Functions should this driver implement, in terms of events handling?
Thanks a lot!
I've gathered some level of knowledge on several components (including software and hardware) which are involved in general DMA transactions in ARM based boards, but I don't understand how is it all perfectly integrated, I didn't find a full coherent description about this.
I'll write down the high level of the knowledge I already have and I hope that someone could fix me where I'm wrong and complete the missing parts so the whole picture would be clear. My description starts with the userspace software and drills down to the hardware components. The misunderstood parts are in italic-bold format.
The user-mode application requests to read/write from some device, i.e. makes I/O operation.
The operating system receives the request and hand it to the appropriate driver (every OS has its own mechanism to do this, I don't need a further drill down here but if you want to share insights here you are welcome)
The driver which is on charge to handle the I/O request, has to know the address to which the device is mapped to (since I'm interested in ARM based boards, afaik there is only memory-mapped I/O and no port I/O). In most of the cases (if we consider smartphone-like boards) there is a linux kernel that parses the devices addresses from the device-tree which is given from the bootloader at the boot time (the modern approach), or the linux is precompiled for the specific model family and board with the device addresses within it (hardcoded in its source code) (in older and obsolete? approach). In some cases (happens a lot in smartphones) part of the drivers are precompiled and are just packaged into the kernel, i.e. their source is closed, thus, the addresses correspond to the devices are unknown. Is it correct?
Given that the driver knows the address of the relevant registers of the device it want to communicate with, it allocate a buffer (usually in the kernel space) to which the device would write its data (with the help of the DMA). The driver needs to inform the device about the location of that buffer, but the addresses that the devices work with (to manipulate memory) are different from the addresses that the drivers (cpu) work with, hence, the driver needs to inform the device about the 'bus address' of the buffer it has just allocated. How does the driver inform the device about that address? How popular is to use an IOMMU? when using IOMMU is there one hardware component that manages addressing or one per device?
Then the driver commands the device to do its job (by manipulating its registers) and the device transfers output data directly to the allocated buffer in the memory. Here I'm confused a bit with the relation of device-driver:bus:bus-controller:actual-device. Take for example some imaginary device which knows to communicate in the I2C protocol; the SoC specify an I2C bus interface - what is this actually? does the I2C bus has some kind of bus controller? Does the cpu communicate with the I2C bus interface or directly with the device? (i.e. the I2C bus interface is seamless). I guess that someone with some experience with device drivers could answer this easily..
The device populates a DMA channel. Since the device is not connected directly to the memory but rather is connected through some bus to the DMA controller (which masters the bus), it interacts with the DMA to transfer the required data to the allocated buffer in the memory. When the board vendor uses ARM IP cores and bus specifications then this step involves transactions over a bus from the AMBA spec (i.e. AHB/multi-AHB/AXI), and some protocol between the device and a DMAC on top of it. I would like to know more about this step, what actually happens? There are many specifications for DMA controller by ARM, which one is the popular? which is obsolete?
When the device is done, it sends an interrupt, which travel to the OS through the interrupt controller, and the OS's interrupt handler direct it to the appropriate driver which now knows that the DMA transfer is completed.
You've slightly conflated two things here - there are some devices (e.g. UARTs, MMC controllers, audio controllers, typically lower-bandwidth devices) which rely on an external DMA controller ("DMA engine" in Linux terminology), but many devices are simply bus masters in their own right and perform their own DMA directly (e.g. GPUs, USB host controllers, and of course the DMA controllers themselves). The former involves a bunch of extra complexity with the CPU programming the DMA controller, so I'm going to ignore it and just consider straightforward bus-master DMA.
In a typical ARM SoC, the CPU clusters and other master peripherals, and the memory controller and other slave peripherals, are all connected together with various AMBA interconnects, forming a single "bus" (generally all mapped to the "platform bus" in Linux), over which masters address slaves according to the address maps of the interconnect. You can safely assume that the device drivers know (whether by device tree or hardcoded) where devices appear in the CPU's physical address map, because otherwise they'd be useless.
On simpler systems, there is a single address map, so the physical addresses used by the CPU to address RAM and peripherals can be freely shared with other masters as DMA addresses. Other systems are more complex - one of the more well-known is the Raspberry Pi's BCM2835, in which the CPU and GPU have different address maps; e.g. the interconnect is hard-wired such that where the GPU sees peripherals at "bus address" 0x7e000000, the CPU sees them at "physical address" 0x20000000. Furthermore, in LPAE systems with 40-bit physical addresses, the interconnect might need to provide different views to different masters - e.g. in the TI Keystone 2 SoCs, all the DRAM is above the 32-bit boundary from the CPUs' point of view, so the 32-bit DMA masters would be useless if the interconnect didn't show them a different addresses map. For Linux, check out the dma-ranges device tree property for how such CPU→bus translations are described. The CPU must take these translations into account when telling a master to access a particular RAM or peripheral address; Linux drivers should be using the DMA mapping API which provides appropriately-translated DMA addresses.
IOMMUs provide more flexibility than fixed interconnect offsets - typically, addresses can be remapped dynamically, and for system integrity masters can be prevented from accessing any addresses other than those mapped for DMA at any given time. Furthermore, in an LPAE or AArch64 system with more than 4GB of RAM, an IOMMU becomes necessary if a 32-bit peripheral needs to be able to access buffers anywhere in RAM. You'll see IOMMUs on a lot of the current 64-bit systems for the purpose of integrating legacy 32-bit devices, but they are also increasingly popular for the purpose of device virtualisation.
IOMMU topology depends on the system and the IOMMUs in use - the system I'm currently working with has 7 separate ARM MMU-401/400 devices in front of individual bus-master peripherals; the ARM MMU-500 on the other hand can be implemented as a single system-wide device with a separate TLB for each master; other vendors have their own designs. Either way, from a Linux perspective, most device drivers should be using the aforementioned DMA mapping API to allocate and prepare physical buffers for DMA, which will also automatically set up the appropriate IOMMU mappings if the device is attached to one. That way, individual device drivers need not care about the presence of an IOMMU or not. Other drivers (typically GPU drivers) however, depend on an IOMMU and want complete control, so manage the mappings directly via the IOMMU API. Essentially, the IOMMU's page tables are set up to map certain ranges of physical addresses* to ranges of I/O virtual addresses, those IOVAs are given to the device as DMA (i.e. bus) addresses, and the IOMMU translates the IOVAs back to physical addresses as the device accesses them. Once the DMA operation is finished, the driver typically removes the IOMMU mapping, both to free up IOVA space and so that the device no longer has access to RAM.
Note that in some cases the DMA transfer is cyclic and never "finishes". With something like a display controller, the CPU might just map a buffer for DMA, pass that address to the controller and trigger it to start, and it will then continuously perform DMA reads to scan out whatever the CPU writes to that buffer until it is told to stop.
Other peripheral buses beyond the SoC interconnect, like I2C/SPI/USB/etc. work as you suspect - there is a bus controller (which is itself a device on the AMBA bus, so any of the above might apply to it) with its own device driver. In a crude generalisation, the CPU doesn't communicate directly with devices on the external bus - where a driver for an AMBA device says "write X to register Y", that just happens by the CPU performing a store to a memory-mapped address; where an I2C device driver says "write X to register Y", the OS usually has some bus abstraction layer which the bus controller driver implements, whereby the CPU programs the controller with a command saying "write X to register Y on device Z", the bus controller hardware will go off and do that, then notify the OS of the peripheral device's response via an interrupt or some other means.
* technically, the IOMMU itself, being more or less "just another device", could have a different address map in the interconnect as previously described, but I would doubt the sanity of anyone actually building a system like that.
My question is going to be rather vague but I will try to explain as detailed as I can what I am trying to resolve.
Trying to learn Linux kernel USB stack I have started to think of making a simple USB driver for my Atmel evaluation board based on ARM M0+ MCU to run away from Windows tools (Visual Studio plugin).
I have spent few days learning kernel's USB API and come to conclusion of how to make this. My driver aims to make my board connected to PC through USB cable act like a simple USB flash drive. Making that I then can easily program it with a new version of firmware written by me.
I have found that I need to find out specific interface (I am talking about interface in terms of USB specification, not interface we used to use as a code abstraction) that holds an endpoint (pipe) responsible for interaction with flash memory. And then I can map it to character device and interact with it using standard I/O operations that are described in struct file_operations structure.
Simply using cat on /proc/* file descriptor that was created by USB Core subsystem I have investigated that interface responsible for interaction with flash memory holds bulk endpoint (likewise, this terms come from USB specification, CMIIAW) that act as a "descriptor". Linux kernel USB Core subsystem gives neat interfaces to talk to different kind of endpoints whether it control, interrupt, bulk or asynchronous endpoint.
Now I have come closer to my very question.
Also the main transfer unit in communication between two USB devices is abstraction called urb - you allocate it, you fill it, you send it to USB Core subsystem, you read it if it was IN type of urb and, finally, you free it. What is confusing for me and tightly related to my question is the next API include/linux/usb.h:
static inline void usb_fill_bulk_urb(struct urb *urb,
struct usb_device *dev,
unsigned int pipe,
void *transfer_buffer,
int buffer_length,
usb_complete_t complete_fn,
void *context)
Assume I have obtained an information from board's datasheet about where to write a program code. Let's say, we have 0x00100 - 0x10000 memory region. I will compile my code, obtain a binary and then using standard Linux tools or writing a simple user-space wrapper application I will use lseek to set file's offset to 0x00100 and write system call provided with a buffer (binary compiled previously) and it's length.
In kernel space, I will have to allocate urb in write system call handler, fill it with a buffer sent from user space and submit this urb to USB Core.
BUT I can not find a way how to specify an OFFSET set earlier by lseek. Do I miss something? Maybe I missed some concepts or, perhaps, I am watching in a wrong way?
When your embedded Linux device acts as a USB mass storage device, the flash as a peripheral on Linux device is unmounted, and the gadget driver is loaded. Linux then loses control to the flash, and now the PC connected to your Linux device fully controls the flash. This is because a flash as a USB device can only has one USb host.
The gadget driver works purely in kernel space. It does not receive or transmit data from/to user space. It calls vfs_read() and vfs_write() to access the files on the flash, with an field offset. The offset is got from the USB commands sent from your host - Windows PC.
There is no way to specify offset using USB subsystem's API. I misunderstood whole conception of USB as communication protocol, unwise me. You must first learn underlying protocol your device uses to communicate with others.
If your device acts as a USB HID device then learning specification of how to exchange data with USB HID device is the way to go. If there is something proprietary then you can do nothing but reverse engineer it (listening USB packets with a sniffer on system where a driver for your device exists).
As for my board it has embedded debugger that serves as a communication module besides being debugger itself. Specifically, my device is equipped with EDBG and here is a link on description of protocol it uses for communication.
I am a bit confused regarding DMA transfers with a PCIe device.
Say, for example, I have a slave PCIe device, and I want to transfer a block of data from the device to the RAM, using a DMA transaction. Note that the device is slave, and does not have a DMA "machine" on it.
I know I need to obtain a DMA-able buffer in RAM (either by allocating a coherent one, or by mapping a page) first.
But what's next? what's the API to start a DMA transfer of N bytes from address S to address D?
Can modern systems issue a DMA transfer to/from a slave pci device? if so, what is the Linux API for that?
As explained here:
[ISA]
In the original IBM PC, there was only one Intel 8237 DMA controller [...]
A PCI architecture has no central DMA controller, unlike ISA. Instead, any PCI component can request control of the bus ("become the bus master") and request to read from and write to system memory
The PCI bus does not have a "central" DMA controller - instead, each device can be a DMA "controller".
First of all, there are no slaves and slave holders inside modern PC. There is south bridge (in PCI) or Root Complex (root of PCI-express device tree) and there are some other PCI/PCIe actors, like bridges, soldered chips, plugged cards, hardware debuggers etc. I'll assume that you are asking about plugged card or some other peripheral device, like soldered Sound Card or Ethernet chip.
According to this detailed description of "Transaction Layer Packet" (TLP, "PCIe’s uppermost layer"), there is "Bus Mastership (DMA)":
On PCIe, it’s significantly less exotic. ... anyone on the bus can send read and write TLPs on the bus, exactly like the Root Complex. This allows the peripheral to access the CPU’s memory directly (DMA) or exchange TLPs with peer peripherals (to the extent that the switching entities support that).
Also, there is some benefits of DMA capability from plugged devices: DMA attack. And PCIe is listed as capable of initiating DMA transfer:
Systems may be vulnerable to a DMA attack by an external device if they have a FireWire, ExpressCard, Thunderbolt, or other expansion port that, like PCI and PCI-Express in general, hooks up attached devices directly to the physical address space.
I think, there is no universal API for programming DMA transfers that are initiated from the peripheral device itself. This depends on the what the device is, when the DMA should be started and what will be sent.