How pass a file pointer from c to a call in asm - c

Question

I'm messing around with nasm, and after doing a hello world with no problem, I though I'd try to do some c integration.
I'm using c to open a file, and I then want to use the pointer returned for the open file to process the text. However, when I call fgetc with the pointer in rdi, I get a "no such file or directory", followed by a segfault.
What am I doing wrong?
int64_t asmFunc(FILE* a, char* b);
int main()
{
int num;
FILE *fptr;
size_t line_buf_size = 0;
char *ret = malloc(100);
fptr = fopen("./test.txt","r");
if(fptr == NULL)
{
printf("Error!");
exit(1);
}
printf("%ld", asmFunc(fptr, ret));
return 0;
}
global asmFunc
section .text
extern fgetc
asmFunc:
call fgetc ; segfault occurs here.
(...)
ret

Answer 1

The first instruction of asmFunc isn't a call either, but I removed some setup stuff for later operations to make it easier to read.
Well that just defeats the entire purpose of an MCVE. You need to re-run your test after simplifying to make sure it still shows the same problem as your full version. But for this answer, I'll assume your setup didn't clobber the fptr arg in RDI or modify RSP.
asmFunc:
call fgetc ; segfault occurs here.
fptr will still be in RDI, where your caller passed it, so that's correct for int fgetc(FILE *fp).
So presumably fgetc is segfaulting because you called it with a misaligned stack. (It was 16-byte aligned before the call that jumped to asmFunc, but you don't have an odd number of pushes or any sub rsp, 8*n). Modern builds of glibc actually do depend on 16-byte alignment for scanf (glibc scanf Segmentation faults when called from a function that doesn't align RSP) so it's easy to imagine that fgetc includes code that also compiles to include a movaps of something on the stack.
Once you fix this bug, you will have the problem that call fgetc destroys your char *ret arg, because your caller passes it in RSI. Arg-passing registers are call-clobbered. What registers are preserved through a linux x86-64 function call
asmFunc: ; (FILE *fptr, char *ret)
push rsi ; save ret
call fgetc
pop rsi
mov [rsi], al
ret
A C compiler would normally save/restore RBX and use mov to save ret there.
asmFunc: ; (FILE *fptr, char *ret)
push rbx
mov rbx, rsi ; save ret
call fgetc
mov [rbx], al
pop rbx ; restore rbx
ret
However, when I call fgetc with the pointer in rdi, I get a "no such file or directory", followed by a segfault.
No idea how you're getting "no such file or directory". Is that from your debugger looking source for glibc functions? If it's part of what your program itself prints, that makes near zero sense, because you do exit(1) correctly when fptr == NULL. And you don't use perror() or anything else that looks up errno codes to generate standard error strings.

Answer 2

You need to learn and follow the calling conventions documented in Linux x86-64 ABI specification, in particular its §3.2.3 Parameter passing section. So the pointer value fptr is in %rdi, and the pointer value ret is in %rsi and you probably should push a call frame for your asmFunc
Read also the x86 calling conventions wikipage.
If you are able to code the equivalent (even a simplified one) of asmFunc in C in some example.c file, I recommend compiling it with gcc -O -fverbose-asm -Wall -S example.c and looking into the emitted example.s assembler file for inspiration. Most of the time, the first machine instruction of such a function is not a call (but something, called the function prologue, changing the stack pointer %esp and allocating some call frame on the call stack)
For example, on my Linux/Debian/x86-64 with gcc-8
void asmfunc(FILE* fil, char*s) {
fputc ('\t', fil);
fputs (s, fil);
fputc ('\n', fil);
fflush (fil);
}
is compiled into:
.text
.globl asmfunc
.type asmfunc, #function
asmfunc:
.LFB11:
.cfi_startproc
pushq %rbp #
.cfi_def_cfa_offset 16
.cfi_offset 6, -16
pushq %rbx #
.cfi_def_cfa_offset 24
.cfi_offset 3, -24
subq $8, %rsp #,
.cfi_def_cfa_offset 32
movq %rdi, %rbx # fil, fil
movq %rsi, %rbp # s, s
# /tmp/example.c:4: fputc ('\t', fil);
movq %rdi, %rsi # fil,
movl $9, %edi #,
call fputc#PLT #
# /tmp/example.c:5: fputs (s, fil);
movq %rbx, %rsi # fil,
movq %rbp, %rdi # s,
call fputs#PLT #
# /tmp/example.c:6: fputc ('\n', fil);
movq %rbx, %rsi # fil,
movl $10, %edi #,
call fputc#PLT #
# /tmp/example.c:7: fflush (fil);
movq %rbx, %rdi # fil,
call fflush#PLT #
# /tmp/example.c:8: }
addq $8, %rsp #,
.cfi_def_cfa_offset 24
popq %rbx #
.cfi_def_cfa_offset 16
popq %rbp #
.cfi_def_cfa_offset 8
ret
.cfi_endproc
.LFE11:
.size asmfunc, .-asmfunc
.ident "GCC: (Debian 8.3.0-6) 8.3.0"
Notice however that in some cases, GCC is capable (e.g. with -O2) of tail-call optimizations and might call some leaf-functions specially.