Im using https://github.com/gaby64/sf
When i connect localy over 127.0.0.1 everything is fine
This is a 32 byte fragment of the data i should get and do get locally:
00000000
00000000
00000000
00000100
00000000
00001010
01000111
01000101
01010100
01000001
01010000
01010000
01010000
01001011
01000111
00000000
11111111
11101000
01111011
00100000
00100010
01101001
01101101
01100111
00100010
00111010
00100000
01011011
00100000
01011101
00101100
00100000
This is the data i get when connecting via my own external internet ip:
00100000
00100000
00100000
00100000
00100000
00100000
00100000
00100000
01011100
00101111
01011100
00101111
00100000
01100100
01110010
01101001
01101100
01101100
00100000
01100100
01101111
01110111
01101110
00100000
01110100
01101111
00100000
01110100
01101000
01100101
00100000
01101001
it appears to be completely mixed and is mostly NULL, for the 84kb multipart message that is.
small messages dont seem to be affected.
is there something i am failing to consider?
A few months ago it was working, since then i had to reinstall linux so i cannot test with exactly the same conditions i had when it was functional.
And there is another bit of weirdness, a web page served using libwebsockets only loads partially up to 20kb when i try to access it via my external ip, also works fine through localhost.
I should mention its all repeatable, always getting transformed the same way.
ran wire-shark to analyze traffic, im not much of an expert on tcp/ip protocol but looks like everything is ok when looking at the packets. kernel must be responsible, could my application have a leak that effects that kernel? despite no warnings from valgrind.
I finally figured it out,
i was not ware that TCP cannot traverse NAT routers.
connecting directly to my modem solves the problem.
You have to use some method to check if you receive the same that you have sent. There are many ways to do this. One well known and easy to find on the Internet is the checksum method. Search for it.
Related
I learned from this link Why is address 0x400000 chosen as a start of text segment in x86_64 ABI? that 64-bit Linux process start address by default should be 0x400000, but on my Ubuntu, I only found my bash process starts from a very high base address (0x55971cea6000).
Any one knows why? and how does dynamic linker choose the start address for a 64-bit process?
$ uname -r
5.15.0-25-generic
$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS"
...
$ file /usr/bin/bash
/usr/bin/bash: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=33a5554034feb2af38e8c75872058883b2988bc5, for GNU/Linux 3.2.0, stripped
$ ld -verbose | grep -i text-segment
PROVIDE (__executable_start = SEGMENT_START("text-segment", 0x400000)); . = SEGMENT_START("text-segment", 0x400000) + SIZEOF_HEADERS;
$ cat maps
55971ce77000-55971cea6000 r--p 00000000 08:02 153 /usr/bin/bash
55971cea6000-55971cf85000 r-xp 0002f000 08:02 153 /usr/bin/bash
55971cf85000-55971cfbf000 r--p 0010e000 08:02 153 /usr/bin/bash
55971cfc0000-55971cfc4000 r--p 00148000 08:02 153 /usr/bin/bash
55971cfc4000-55971cfcd000 rw-p 0014c000 08:02 153 /usr/bin/bash
55971cfcd000-55971cfd8000 rw-p 00000000 00:00 0
...
$ readelf -h /usr/bin/bash
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: DYN (Position-Independent Executable file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x32eb0
Start of program headers: 64 (bytes into file)
Start of section headers: 1394600 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 13
Size of section headers: 64 (bytes)
Number of section headers: 30
Section header string table index: 29
I learned from this link Why is address 0x400000 chosen as a start of text segment in x86_64
That address is used for executables (ELF type ET_EXEC).
I only found my bash process starts from a very high base address (0x55971cea6000). Any one knows why?
Because your bash is (newer) position-independent executable (ELF type ET_DYN). It behaves much like a shared library, and is relocated to random address at runtime.
The 0x55971cea6000 address you found will vary from one execution to another. In contrast, ET_EXEC executables can only run correctly when loaded at their "linked at" address (typically 0x400000).
how does dynamic linker choose the start address for a 64-bit process?
The dynamic linker doesn't choose the start address of the executable -- the kernel does (by the time the dynamic linker starts running, the executable has already been mmaped into memory).
The kernel looks at the .e_type in the ELF header and .p_vaddr field of the first program header and goes from there. IFF .e_type == ET_EXEC, then the kernel maps executable segments at their .p_vaddr addresses. For ET_DYN, if ASLR is in effect, the kernel performs mmaps at a random address.
After many years away from programming, I've decided to take it up again for fun, and finding myself enjoying it quite a lot. In the process of finding things to code, I've found this data which is openly available from Network Rail in the United Kingdom.
Among other things, you can obtain schedule data, which is a list of all train, bus and ferry journeys.
A schedule record for a train journey might look like this:
BSNY819581902281902280001000 PEE5A99 122112002 EMU390 125
BX VTY
LOMNCRPIC 2131 00008 FL TB
LIARDWCKJ 2133 00000000
LISLDLJN 2134H00000000 SL H
LIHTNOJN 2138 00000000 FL 1H
LISTKP 2140H000000002 SL
LISTKPE1 2141H00000000
LIADSWDRD 2142H00000000
LICHDH 2143 000000002
LIWLMSL 2146 000000004 1
LIALDEDGE 2148 00000000 1 3
LISBCH 2200 000000001 FL
LICREWSBG 2203 00000000
LICREWUML 2204 00000000
LICREWE 2206 000000001 FL H
LICREWBHJ 2207H00000000 1
LIMADELEY 2212 00000000 FL FL 5H
LINTNB 2222H00000000 FL FL
LISTAFFDJ 2226H00000000 SL
LISTAFFRD 2228H2231H 000000004 SL SL A C
LISTAFTVJ 2233 00000000
LIPNKRDG 2236H00000000 1 1
LIBSBYJN 2244 00000000
LIPBLJWM 2248 00000000
LIDRLSTNJ 2251H00000000
LIBSCTSTA 2252H00000000 1
LIPRYBRNJ 2257 00000000 7
LIASTON 2306H000000002
LISTECHFD 2311H00000000
LIBHAMINT 2315 000000004
LIBKSWELL 2318H00000000 1H
LICOVNTRY 2324 000000001 2 3
LIRUGBTVJ 2336 00000000 UNL 3
LIRUGBY 2340 000000005 UNLUNL 1H
LIHMTNJ 2343 00000000 1H
LIDVNTYNJ 2346H00000000
LILNGBKBY 2351 00000000 1 1
LINMPTN 0001H000000001 6
LIHANSLPJ 0016 00000000 SL
LIMKNSCEN 0021 000000001 SL SL
LIBLTCHLY 0023 000000004 SL SL 5 1H
LILEDBRNJ 0036 00000000 SL SL 2
LITRING 0042 000000002 SL SL 2H
LIBONENDJ 0048H00000000 SL SL 1H
LIWATFDJ 0056 000000009 SL SL 1H
LIHROW 0101H000000006 SL SL 3
LIWMBY 0107H000000006 CL
LTWMBYICD 0117H0000 TF
tl;dr the first two lines describe what type of train is running, when it runs, how fast, etc. The other lines describe the points the train will pass, and what time they are expected to do so. The main takeaway is that each record has a different length depending on the journey.
When I saw this, I thought "This would be a great thing to try and mess around with in COBOL." I went to polytech and learned PASCAL and COBOL, but only had to deal with files with a consistent length and consistent data, not something like this.
I spent a couple of hours trying to find some sort of answer to this on Google, but nothing really showed, hence my asking.
Just for reference, I have managed to do this in GW-BASIC, and could do it in elementary Python if needed, but COBOL, being what it is, is a whole different kettle of fish.
Is it possible to read something like this in to COBOL without having to resort to witchcraft, or is it just in the "too hard" basket? I'm only doing this for fun, so it's really no big deal.
Any responses or feedback would be most welcome.
Many thanks,
Joseph.
Yes it is possible. For the file use Line Sequential
The file definition
select lineseq assign to "lineseq.dat"
organization is line sequential.
To split the lines up use UNSTRING. i.e.
UNSTRING in-line
DELIMITED BY SPACES
into item-1, item-2, item-3
END-UNSTRING
It is probably easier to do in languages like python
Actually (after reformatting the question) I think COBOL is
perfect for this job as the data is fixed-length (may came out of COBOL, too...)
define the file (line-sequential may not even be needed if it contains, like your post even trailing spaces; but as this may change, line-sequential would be fine)
OPEN INPUT file, READ until end of file
put the complete record read into a local record with defined sub-fields and just access the data from the sub-fields, after validating (the file may be broken for many reasons)
Depending on the amount of lines in that data you may either process the records direct, move them to a table (have a look at OCCURS), or WRITE them to another file (likely and INDEXED with multiple KEY definitions)
To expand on #Simon Sobisch's answer.
Looking at the data, and trying to work it out, I can see these things.
Top two lines as you say are the type of train and that.
Then you have a line starting LO which must be the start of the journey. The next 7 characters would be the station, with MNCRPIC being presumably "Manchester Piccadily". Then there's a space, then four digits which would be a time.
Then you have a load of lines starting LI which are intermediate points. Some of those have a "H" after the time, others don't. This will be a problem if you're going to do UNSTRING DELIMITED BY SPACE. I'm going to presume that H means Halt.
LISTAFFRD 2228H2231H 000000004 SL SL A C
is an odd looking line.
At the end we have LT which is the end of the journey, arriving at WMBYICD at 0117.
01 TRAIN-SCHEDULE.
03 RECORD-TYPE PIC XX.
88 JOURNEY-START VALUE 'LO'.
88 JOURNEY-INTERMEDIATE VALUE 'LI'.
88 JOURNEY-TERMINATE VALUE 'LT'.
03 TRAIN-STATION PIC X(7).
03 FILLER PIC X(11).
03 TRAIN-TIME.
05 TRAIN-TIME-HH PIC 99.
05 TRAIN-TIME-MM PIC 99.
03 TRAIN-HALT-FLAG PIC X.
88 TRAIN-STOPS-HERE VALUE 'H'.
And so on.
COBOL input files can be VARIABLE or FIXED. If it is variable, the first positions will be the number of cols of the row.
Here you have the information at IBM official webpage.
Processing Files with Variable Length Records
I am exploring the possibility to back the text and data segment with hugepages following the guide in https://github.com/libhugetlbfs/libhugetlbfs/blob/master/HOWTO.
I have relinked the application as suggested by adding "-B/usr/share/libhugetlbfs -Wl,--hugetlbfs-align" and started the application with "hugectl --text --data --heap --bss /path/to/my/application".
But I am not very sure how to verify whether the data and text segments are indeed copied to the files on the hugetlbfs filesystem.
Checking the /proc/{pid}/maps, it could be seen that the hugepages are used for heap but not for text and data segments as the first two address ranges are mapped to the application but not the hugepage file system.
Is my understanding correct? Actually I suspect my conclusion that hugepages are used for heap from /proc/{pid}/maps is also incorrect.
How should I verify whether the data and text segments are backed in the hugepages? I know that data and text segments will be copied to hugetlbfs filesystem if successful but how to verify it?
Thanks!
output of /proc/{pid}/maps
00400000-00d2c000 r-xp 00000000 fd:02 46153351 /path/to/my/application
00f2b000-00fa3000 rw-p 0092b000 fd:02 46153351 /path/to/my/application
00fa3000-00fbb000 rw-p 00000000 00:00 0
02a0c000-02a2d000 rw-p 00000000 00:00 0 [heap]
40000000-80000000 rw-p 00000000 00:15 2476090 /dev/hugepages-1G/libhugetlbfs.tmp.nS7exn (deleted)
check
/proc/$pid/numa_maps
contains information about each memory area used by a given process allowing--among other information--the determination of which nodes were used for the pages.
for formar see http://linux.die.net/man/5/numa_maps
If you set HUGETLB_DEBUG=1 variable, it will tell you whole lot of useful information. One of them is this:
INFO: Segment 2's aligned memsz is too small: 0x864 < 0xffffffffffffffff
If it is successful it looks like:
libhugetlbfs [zupa:154297]: INFO: Segment 0 (phdr 2): 0x400000-0x400864 (filesz=0x864) (prot = 0x5)
libhugetlbfs [zupa:154297]: INFO: Segment 1 (phdr 3): 0x600868-0x600af8 (filesz=0x27c) (prot = 0x3)
libhugetlbfs [zupa:154297]: DEBUG: Total memsz = 0xaf4, memsz of largest segment = 0x864
libhugetlbfs [zupa:154297]: INFO: libhugetlbfs version: 2.16 (modified)
libhugetlbfs [zupa:154951]: INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0x864 bytes and 0 extra bytes from 0x400000...done
libhugetlbfs [zupa:154297]: INFO: Prepare succeeded
libhugetlbfs [zupa:154952]: INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0x27c bytes and 0 extra bytes from 0x600868...done
libhugetlbfs [zupa:154297]: INFO: Prepare succeeded
I'm just writing a Linux kernel module and getting a warning message that my kernel is tainted:
[ 1194.552013] ------------[ cut here ]------------
[ 1194.552639] WARNING: CPU: 0 PID: 176 at /build/linux-mcJ30I/linux-3.16.7-ckt20/block/blk.h:231 generic_make_request_checks+0x46a/0x480()
[ 1194.552639] Modules linked in: <my kernel muldule>(O) <various other Standard kernel modules>
[ 1194.552639] CPU: 0 PID: 176 Comm: systemd-udevd Tainted: G O 3.16.0-4-586 #1 Debian 3.16.7-ckt20-1+deb8u1
[ 1194.552639] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[ 1194.552639] ebeb3b70 ebeb3b70 c103bf71 c14f2d68 00000000 000000b0 c1520644 000000e7
[ 1194.552639] c11ddd1a 000000e7 c11ddd1a ec0d4800 edcb5c00 ffffffff ebeb3b80 c103c032
[ 1194.552639] 00000009 00000000 ebeb3bf8 c11ddd1a 06010006 000001a7 00000000 00000000
[ 1194.552639] Call Trace:
[ 1194.552639] [<c103bf71>] ? warn_slowpath_common+0x81/0xa0
[ 1194.552639] [<c11ddd1a>] ? generic_make_request_checks+0x46a/0x480
[ 1194.552639] [<c11ddd1a>] ? generic_make_request_checks+0x46a/0x480
...
The message just appears once, my module is working as it should and I'm not getting any Oops message or kernel Panic or similar.
I read several questions, answers and tutorials (here, here or here) but basically all I found out is that the currently running kernel is not supported by the community (anymore) and can't be debugged by the community. For me, this is obvious since my kernel module is installed and they don't have the source of it.
So for me it seems like there is nothing dangerous with that warning message, is that correct? I already tried to find an error for several hours but I didn't find one. So is the message just showing up because I'm using my kernel module?
Edit: I don't know if it's important but I'm running Linux kernel 3.16.0-4-586
I'm wring a toy OS for my raspberry pi and trying to setup the MMU. I want to split the virtual memory between 3G:1G, so I think my code should be linked at 0xC0008000, while loaded to 0x8000 on execution. (The 0x8000 is the address the current bootloaders expect to find the kernel - since they are built for linux).
I think everything is setup fine by poking around with objdump, but it doesn't work. After some debugging with qemu, I think the bootloader doesn't find my code at all.
I believe the problem is with my linkscript, since the kernel starts fine if I move the starting code into its own section that's both linked and loaded at 0x8000.
I've extracted out the script and the minimal code. As following,
$ cat kernel.ld
ENTRY(_start)
SECTIONS
{
/* must == KERNLINK */
. = 0xC0008000;
.text : AT(0x8000) {
*(.text)
}
.bss : {
*(.bss)
}
.data : {
*(.data)
}
.rodata : {
*(.rodata)
}
}
-
$ cat source/entry.S
#include "mem.h"
.globl _start
_start = V2P(entry)
.globl entry
entry:
loop$:
b loop$
(The "b loop$" won't work since it's generated as "b·c0008000" instead of using a relative branch. But never mind that part, the problem is it never reaches entry).
$ cat source/mem.h
#define KERNLOAD 0x8000
#define KERNBASE 0xC0000000
#define KERNLINK (KERNBASE+KERNLOAD)
#define V2P(va) ((va) - KERNBASE)
Those are the only three source files. There should be nothing interesting in the Makefile, but the output from make is,
$ make
arm-none-eabi-gcc -g -Wall -c -o build/entry.o source/entry.S
arm-none-eabi-ld --no-undefined -T kernel.ld build/entry.o -Map kernel.map -o build/output.elf
arm-none-eabi-objcopy build/output.elf -O binary kernel.img
And objdump,
$ arm-none-eabi-objdump -h build/output.elf
build/output.elf: file format elf32-littlearm
Sections:
Idx Name Size VMA LMA File off Algn
0 .text 00000004 c0008000 00008000 00008000 2**0
CONTENTS, ALLOC, LOAD, READONLY, CODE
1 .ARM.attributes 00000014 00000000 00000000 00008004 2**0
CONTENTS, READONLY
2 .debug_line 0000003c 00000000 00000000 00008018 2**0
CONTENTS, READONLY, DEBUGGING
3 .debug_info 00000054 00000000 00000000 00008054 2**0
CONTENTS, READONLY, DEBUGGING
4 .debug_abbrev 00000014 00000000 00000000 000080a8 2**0
CONTENTS, READONLY, DEBUGGING
5 .debug_aranges 00000020 00000000 00000000 000080c0 2**3
CONTENTS, READONLY, DEBUGGING
I start to believe I overlooked some obvious yet precious details.
==== update ====
As noted in my own answer below, the confusion is caused by debugging in qemu. The breakpoints are set by virtual addresses. "b entry" doesn't work, because gdb is thinking about virtual address while MMU hasn't been enabled and we're running by physical address.
So before MMU is enabled, we have to use "b *0x8000". This sets a breakpoint that's correctly hit. GDB seems still confused though, since it doesn't show any debugging info (no source code, like 0x00008004 in ?? ()). That's not a big issue since I have the listing produced by "objdump -D".
After MMU is enabled and we branch to main, gdb works perfectly. The crux is to jump to a virtual address, using an absolute branch. b/bl would issue relative jumps. So I use ldr pc =main. bx works too.
Finally solved it...
The code and the linkscript don't have problems. It's just that debugging doesn't seem to work on the qemu I used. It does work when VMA==LMA, but in this case the kernel already runs and qemu doesn't know about it. Or, the breakpoint is lost and never caught by gdb.
Anyway, by starting afresh and adding bits one by one, I have it working on my rpi (real hardware), and then it works on qemu too (gdb still won't catch the breakpoint).
So my question should really be how to debug such a scenario using qemu/gdb.
(My original problem is solved by changing the "domain" permission to "manager", instead of using "client". Yeah I know this is probably temporary and I should set a variable elsewhere).