Is it possible to map physical address as data fragment in sk_buff?
I am working on Zynq Ultrascale+ platform (FPGA + ARM SOC). I have memory buffer mapped to physical address. The goal is to efficiently send that data over UDP. By efficiently I mean ZEROCOPY. What I am trying to do is to develop linux driver that would map that physical address into kernel memory and append it to sk_buff as fragment.
I started with:
#define PACKET_LEN 1024
struct page *pag;
struct net_device *dev;
struct sk_buff *skb = NULL;
skb = alloc_skb(LL_RESERVED_SPACE(dev) + PACKET_LEN + ip_header_l +
udp_header_l, GFP_ATOMIC);
udp = skb_push(skb, udp_header_l);
//Fill up udp header
...
ip = skb_push(skb, ip_header_l);
//fill up ip header
...
dev_hard_header(skb, dev, ETH_P_IP, addr, myaddr, dev->addr_len);
skb->dev = dev;
//map page with data as fragment
skb_fill_page_desc(skb, 0, pag, 0, PACKET_LEN);
//send data
dev_queue_xmit(skb);
And as long as page is created by:
pagebuff = vmalloc(PACKET_LEN);
pag = vmalloc_to_page(pagebuff);
It all works fine. Packet gets send. Packet is send by two DMA transactions (Scatter Gather).
Going towards my goal I replaced vmalloced page with:
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
membase = devm_ioremap_resource(&pdev->dev, res);
pag = virt_to_page(membase);
Physical address is 0xb0000000 and is mapped to virtual address 0xffffff800ad30000 page is at 0xffffffbf0025e280.
After dev_queue_xmit packet goes to network queue and ends up being mapped for DMA.
Problem arises when swiotlb_map_page uses 0x00ad30000 as phys_addr, which is different than original 0xb0000000.
virt_to_phys is used in swiotlb_map_page to calculate physical address and it basically takes lower 32 bits as phys address. Is there a different way to map memory region so it can be used as sk_buff fragment?
As a temporary fix I created fake page like this:
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
pag = alloc_page(0); //create fake page
memset(pag, 0, sizeof(struct page));
pag->private = res->start;
And patched ethernet driver to use page private data as mapping address:
mapping = skb_frag_page(frag)->private;
if (mapping) {
// printk("macb mapping override to %p\n",mapping);
}
else {
mapping = skb_frag_dma_map(&bp->pdev->dev, frag, offset, size, DMA_TO_DEVICE);
if (dma_mapping_error(&bp->pdev->dev, mapping))
goto dma_error;
}
With such a hack it all works. Data is filled with contents of 0xb0000000. Although it works fine I really doubt it is the right way to do it. Nevertheless it shows there is no hardware limitation to do it. Does anyone know how to map that memory correctly?
P.S. I also tried to map physical address to fixed virtual address in such manner that swiotlb_map_page would calculate correct address (and virt_to_phys did), but it ended with "Unable to handle kernel paging request at virtual address" error.
membase = phys_to_virt(res->start);
i = ioremap_page_range(membase, membase + resource_size(res),
res->start, PAGE_KERNEL);
//I tried both
pag = phys_to_page(res->start);
pag = virt_to_page(membase);
Maybe I am looking for page at wrong address or maybe it is nonexistent.
Can anyone point me in the right direction. Is there a way to accomplish the goal without such a nasty hack?
Related
This question has been asked at least a dozen times but I cannot figure out where is my issue.
I am writing a kernel module that must read data from a reserved memory range. These data are written by an external device.
To control the the device, we have a second register within which I want to write some data.
And this is where I start to get lost...
This is the part of the code that, from my understanding, should create a virtual mapping from the reg input in my device tree:
// Read the control memory and map to virtual address
res_ctrl = platform_get_resource(pdev, IORESOURCE_MEM, 0);
if (!res_ctrl) {
dev_err(&pdev->dev, "can't get device resources\n");
return -ENOENT;
}
p3chvideo_device->pchv_ctrl.paddr = res_ctrl->start;
p3chvideo_device->pchv_ctrl.size = resource_size(res_ctrl);
struct resource* res_d = request_mem_region(p3chvideo_device->pchv_ctrl.paddr, p3chvideo_device->pchv_ctrl.size, "p3chv");
p3chvideo_device->pchv_ctrl.vaddr = ioremap_nocache(p3chvideo_device->pchv_ctrl.paddr, p3chvideo_device->pchv_ctrl.size);
if (!p3chvideo_device->pchv_ctrl.vaddr) {
pr_info("Control buffer allocated vaddr: 0x%0llX paddr: 0x%0llX (size: 0x%0llX)\n", p3chvideo_device->pchv_ctrl.vaddr, p3chvideo_device->pchv_ctrl.paddr, p3chvideo_device->pchv_ctrl.size);
return -EADDRNOTAVAIL;
}
//p3chvideo_device->pchv_ctrl.vaddr = devm_ioremap(&pdev->dev, p3chvideo_device->pchv_ctrl.paddr, p3chvideo_device->pchv_ctrl.size);//ioremap(p3chvideo_device->pchv_ctrl.paddr, p3chvideo_device->pchv_ctrl.size);
pr_info("Control buffer allocated vaddr: 0x%0llX paddr: 0x%0llX (size: 0x%0llX)\n", p3chvideo_device->pchv_ctrl.vaddr, p3chvideo_device->pchv_ctrl.paddr, p3chvideo_device->pchv_ctrl.size);
From the messages in the kernel, the register is correctly detected (offset, size). In addition, I do see in /proc/iomem the reserved memory.
However, when I try to write then read the results, it doesn't work, the value I read is different from the value I wrote... It is as if the register value wasn't altered by the write operation.
static void buffer_loaded_enable_interrupt(void) {
pr_info("buffer loaded enable interrupt 0x%0llX 0x%0X\n", p3chvideo_device->pchv_ctrl.vaddr + IRQ_ENABLE_BUFFER_LOADED, (u32)(1 << 0));
// Clear buffer Loaded Interrupt
//wmb();
pr_info("Stored value before: 0x%0X", readl(p3chvideo_device->pchv_ctrl.vaddr + IRQ_ENABLE_BUFFER_LOADED));
//*(p3chvideo_device->pchv_ctrl.vaddr + IRQ_ENABLE_BUFFER_LOADED) = (u32)(1 << 0);
iowrite32((u32)(1 << 0), p3chvideo_device->pchv_ctrl.vaddr + IRQ_ENABLE_BUFFER_LOADED);
udelay(100);
pr_info("Stored value: 0x%0X", readl(p3chvideo_device->pchv_ctrl.vaddr + IRQ_ENABLE_BUFFER_LOADED));
}
If I use a devmem approach, I can write ahd check the read and it works...
What am I missing?
I finally found the issue and I am posting it so if anyone has the same kind of problem, maybe this could help!
The problem was not in the code I put above but in the type of vaddr. It was set as a ssize_t.
ssize_t vaddr;
As soon as I set it to void __iomem *, everything worked as expected!
void __iomem *vaddr;
Let's hope this post will help someone like me :)
Thanks
I've got the TCP Echo example working well on my hardware, and yesterday figured out how to get a UDP Broadcast working. After further thought, I've realized is that what I really need is to be able to set up a TCP Connection to a Static IP, the idea being that my hardware can connect to a server of some sort and then use that connection for all its transactions. The difference is that whereas the echo example sets up a passive connection, that binds with the incoming source (as I understand it), I want to initiate the connection deliberately to a known IP.
Based on what I found on Wikia Here Here
I've attempted as a base case to implement a function that can send a packet to a Defined IP. I'm simply trying to send a packet to my PC, and I'm looking for it on Wireshark.
void echo_tx_tcp()
{
err_t wr_err = ERR_OK;
struct tcp_pcb *l_tcp_pcb;
l_tcp_pcb = tcp_new();
ip_addr_t dest_ip =
{ ((u32_t)0x0C0C0C2BUL) };
wr_err = tcp_bind(l_tcp_pcb, &dest_ip, 12);
wr_err = tcp_connect(l_tcp_pcb, &dest_ip, 12, echo_accept);
tcp_sent(l_tcp_pcb, echo_sent);
struct pbuf *p = pbuf_alloc(PBUF_TRANSPORT, 1024, PBUF_RAM);
unsigned char buffer_send[1024] = "My Name Is TCP";
p->payload = buffer_send;
p->len = 1024;
p->tot_len = 1024;
wr_err = tcp_write(l_tcp_pcb, p->payload, p->len, 1);
wr_err = tcp_output(l_tcp_pcb);
if(wr_err == ERR_OK)
{
p->len++;
}
return;
}
The last if statement just exists so that I can inspect the wr_err value with a debugger. The err is coming back OK but the packet is not seen on wireshark. My setup is my hardare as well as my PC connected to a router in an isolated manner. The IP Address of the PC locally is 12.12.12.43
Am I missing a step here?
The tcp_write() function will fail and return ERR_MEM if:
The length of the data exceeds the current send buffer size.
The length of the queue of the outgoing segment is larger than the upper limit defined in lwipopts.h.
The number of bytes available in the output queue can be retrieved with the tcp_sndbuf() function.
Potential solution(s):
Try again but send less data.
Monitor the amount of space available in the send buffer and only send (more) data when there is space available in the send buffer.
Suggestions:
tcp_snd_buf() can be used to find out how much send buffer space is available.
tcp_sent() can be implemented with callback function, that will be called when send butter space is available.
When I tried to build a bio and use generic_make_request, I got flooded with this kind of messages.
The pseudocode is following
struct page *page = kmalloc(sizeof(struct page), GFP_KERNEL);
struct bio *bio = bio_alloc(GFP_KERNEL);
set_bio(bio);
add_bio_page(bio, page);
submit_bio(bio);
Then the log will be flooded with messages like:
nommu_map_sg overflow xxxxxxxxxxx+4096 of device mask ffffffff
When I change allocation of page to
struct page *page = alloc_page(GFP_KERNEL);
Kernel just hung up and I can see large CPU consumption of the VM I use.
I can't point to the exact error, but your code lacks a few elements:
the bi_bdev field of struct bio should point to a block device
the bi_end_io field should point to an I/O completion routine
Do you have idea how to initialize array of structs starting from specific address in memory (not virtual, physical DDR memory). I am working on implementation of TxRx on SoC (ARM-FPGA). Basically ARM (PS) and FPGA (PL) communicate to each other by using shared RAM memory. Currently I am working on transmitter side, so I need to constantly load packets that I get from MAC layer to memory, then my Tx reads data and sends it in air. To achieve this I want to implement circular FIFO buffer on (ARM) side, in way that I can store up to 6 packets into buffer and send them one by one, in same time loading other packets on places of already sent packages. Because I need to use specific memory addresses I am interested is it possible to initialize array of structure that will be stored on specific addresses in memory. For example I want that my array starts at adress 0x400000 and ends at address 0x400000 + MaximumNumberOfPackets x SizeOfPackets I know how to do it for one instantiate of structure for example like this:
buffer_t *tmp = (struct buffer_t *)234881024;
But how to do it for array of structures?
A pointer to a single struct (or int, float, or anything else) is inherently a pointer to an array of them. The pointer type provides the sizeof() value for an array entry, and thus allows pointer arithmetic to work.
Thus, given a struct buffer you can simply do
static struct buffer * const myFIFO = (struct buffer *) 0x40000
and then simply access myFIFO as an array
for (size_t i = 0; i < maxPackets; ++i)
{
buffer[i].someField = initialValue1;
buffer[i].someOtherField = 42;
}
This works just the way you expect.
What you can't do (using pure standard C) is declare an array at a particular address like this:
struct buffer myFIFO[23] # 0x400000;
However, your compiler may have extensions to allow it. Many embedded compilers do (after all, that's often how they declare memory-mapped device registers), but it will be different for every compiler vendor, and possibly for every chip because it is a vendor extension.
GCC does allow it for AVR processors via an attribute, for example
volatile int porta __attribute__((address (0x600)));
But it doesn't seem to support it for an ARM.
Generally #kdopen is right but for arm you should create an entry in MEMORY section linker script that shows to linker where is your memory:
MEMORY
{
...
ExternalDDR (w) : ORIGIN = 0x400000, LENGTH = 4M
}
And than, when you are declaring variable just use the
__attribute__((section("ExternalDDR")))
I found the way how to do it. So could I do it like this. I set this into linker script:
MEMORY {
ps7_ddr_0_S_AXI_BASEADDR : ORIGIN = 0x00100000, LENGTH = 0x1FF00000
ps7_ram_0_S_AXI_BASEADDR : ORIGIN = 0x00000000, LENGTH = 0x00030000
ps7_ram_1_S_AXI_BASEADDR : ORIGIN = 0xFFFF0000, LENGTH = 0x0000FE00
DAC_DMA (w) : ORIGIN = 0xE000000, LENGTH = 64K
}
.dacdma : {
__dacdma_start = .;
*(.data)
__dacdma_end = .;
} > DAC_DMA
And then I set this into code
static buffer_t __attribute__((section("DAC_DMA"))) buf_pool[6];
I am writing a kernel module which forms its own packet at ip level and then sends the data . I just want to know how to find the ip address of a local network interface.
There are defined ioctls SIOCGIFADDR (to get if address) and SIOCSIFADDR (to set if address).
All device configuration for IPv4 is done in net/ipv4/devinet.c. You can refer to this source file to get more information.
Wanting to read interface addresses reeks of a design problem. However, if you are looking for determining the preferred source address to use when contacting a remote peer is given in struct rt6_info.rt6i_prefsrc after obtaining it with ip6_route_output. Something along the lines of (with no implied guarantees):
int pick_addr(struct in6_addr *saddr, struct net *net, const struct in6_addr *daddr)
{
struct rt6_info *rt;
struct flowi6 fl6;
int ret;
memset(&fl6, 0, sizeof(fl6));
memcpy(&fl6.daddr, daddr, sizeof(*daddr));
dst = (struct rt6_info *)ip6_route_output(net, NULL /* or sk if you have it */, &fl6);
ret = rt->dst.error;
if (ret == 0)
memcpy(saddr, &rt->rt6i_prefsrc.addr, sizeof(*saddr));
dst_release(&rt->dst);
return ret;
}
We have if_getconfig function available in linux-x.y.z/Documentation/networking/ifenslave.c file which is a very good example of how to use ioctls and fetch address from kernel space