Running pipe of subprocess in ESP32 using ANSI C - c

I am getting into the ESP32 using esp-idf and I would like to make a program able to run subprocess while using C.
Banally, I have tried to call a function like the following:
...
#include <stdio.h>
char *cmd = "ls -l";
char buf[BUFSIZE];
FILE *fp;
if ((fp = popen(cmd, "r")) == NULL) {
printf("############## Error opening pipe!\n");
return -1;
}
while (fgets(buf, BUFSIZE, fp) != NULL) {
printf("############## OUTPUT: %s", buf);
}
if(pclose(fp)) {
printf("############## Command not found or exited with error status\n");
return -1;
}
return 0;
...
but the application returns
c:\esp\../main/app_main.c:139: undefined reference to `popen`
Has somebody experience with this one?

popen() is a system call available on POSIX-compliant and similar systems (UNIX, Linux, FreeBSD). The software that the ESP32 runs is not remotely POSIX-compliant.
Pipes and subprocesses are meaningless on the ESP32 - they're not supported. There is no popen() call.
The ESP32 runs a lightweight operating system called FreeRTOS. FreeRTOS provides lightweight tasks and very simple memory management and not much else.
FreeRTOS tasks share an address space; they are generally non-preemptive, voluntarily yield the processor to allow other tasks to run, and execute a subroutine (C function). POSIX processes each have their own address space, are preemptive and run POSIX binary applications. They're totally different, incompatible things. The ESP32 does not have the resources or memory management hardware to properly support POSIX processes.
ESP-IDF ("IoT Development Framework") is a layer built above FreeRTOS that provides a thin API providing access to the ESP32's hardware (I2C, SPI, wifi, Bluetooth, timers and the like).
Other programming environments like the Arduino Core, Python and LUA are all built on top of ESP-IDF.
ESP-IDF runs one program at a time, unlike POSIX-style systems. That program is the one you compiled, linked, and stored in its flash memory. POSIX-style systems run hundreds or thousands of programs at once. The ESP32 and ESP-IDF do not have the ability to do that.
As you start building applications for the ESP32, not only can you not use POSIX-compatible system calls, you'll also need to learn to program in a very resource-limited environment. On Linux and similar systems programmers commonly write code as if there were no constraints on memory, filesystem size and durability. None of these things are true on the ESP32 - it's trivial to write a program that accidentally needs more memory than its 520KB available to it; it's trivial to write a program that expects more persistent storage than is available in its flash memory; it's trivial to write a program that writes to the flash memory so frequently that it destroys it. You'll need to learn how to program within the constraints of the ESP32, not just how to write code that calls the ESP-IDF instead of a POSIX system.
Take a read through the ESP-IDF documentation that you linked to. That's the environment you're working in. You may find starting out in the Arduino Core to be easier - it's a C/C++ environment, there are a ton of tutorials out there for it, and it's an easier build environment, and you can still easily call ESP-IDF functions from it if you need to.

Related

How to execute system commands using raspberry pi pico

I wanted to know if it is possible to run system commands on computer using raspberry pi pico when plugging it into the USB?
I've tried to do it like a normal C program :
#include "pico/stdlib.h"
#include <stdlib.h>
#include <stdio.h>
int main() {
printf("Start");
stdio_init_all();
char *cmd = "ls";
system(cmd);
return 0;
}
Although printf works very well, system functions (eg: readfile) doesn't work.
Is there any way that i could run some kind of actions on PC using pico usb output ?
Also, there is a project called picoducky which allows the pico to run commands on system but unfortunately it has written in python and i'm looking for c code.
Pico communicates with Pc using usb-to-serial and vice versa. Unless there is a server on Pc that interprets data send through serial than executes a cmd command, its not possible with the convetional way. However, you can make Pico emulate a USB HID device and send keystrokes. Check Tinyusb and this pico-superkey-board project
To be able to run system() function call in C in an embedded system, you need to know several things first:
system() calls the user shell with the string passed as argument as a parameter. Are you running an operating system at all? A shell is normally an operating system component, and it is not normally present in many embedded systems.
system() requires /bin/sh normally to work. To run ls, you need in adition to have it (the ls command) installed.
You don't say what operating system you have in your Raspberry pi pico, but I'm afraid it is not linux (or any unix flavour) so probably all of this is forbidden to you.
Normally, the requirements to have a unix like environment in a small system impede to use a high technology operating system in such systems. Linux requires large amoutns of memory (as there are in the normal raspberry pi, but not in the small versions), a large capacity storage system (bein a usb disk, flash memory card, etc. but normally several Gb for a minimum installation)
In your case, 264kb of ram are very small to have a non-mmu handled microprocessor, capable of addressing virtual memory spaces. Also 2Mb of flash gives you to write large programs, but not to install an operating system like linux.
Had you an emulator of system() you should be able to run other programs, but how? A raspberry pi pico has space to run just one program (the one you write to the flash and nothing else) Even if you write a kernel of a multitasking operating system, you would lack space to run filesystem stored programs, as you normally have limited access to the flash where programs are installed.

passing variables to a process from linux kernel

I want to make a program that will gather information about the keystrokes of a user (keycode, press and release times) and will use them as a biometric for authenticating the user continuously. My approach is to gather the keystrokes using a kernel module (because you can't just kill a kernel module), than the kernel module will send the information to another process that will analyze the data gathered by the kernel module, it will save it to a database and will return an answer to the kernel (the user is authenticated or not) and the kernel will lock the computer if the user is not authenticated. the whole module will not be distributed.
my questions are:
1. How can I call a process from the kernel and also send him the data?
2. how can I return a message to the kernel from the process?
#basile-starynkevitch 's answer and his arguments notwithstanding there is an approach you can take that is perfectly correct and technically allowed by the linux kernel.
Register a keyboard notifier call back function using the kernel call register_keyboard_notifier() in your kernel module. As a matter of fact it's designed for exactly this!
Your notifier call back function will look something like:
int keysniffer_callback(struct notifier_block *notifier_block,
unsigned long scancode,
void *param)
{
// do something with the scancode
return NOTIFY_OK;
}
See https://www.kernel.org/doc/Documentation/input/notifier.txt for starters.
I want to make a program that will gather information about the keystrokes of a user
That should go in practice into your display server, which you did not mention (Xorg, Wayland, MIR, ...?). Details matter a big lot!
My approach is to gather the keystrokes using a kernel module
I strongly believe this is a wrong approach, you don't need any kernel module.
I want to make a program that gathers data about the user keystrokes
Then use ordinary Unix machinery. The keyboard is some character device (and you could have several keyboards, or none, or some virtual one...) and you could read(2) from it. If you want to code a keylogger, please tell that explicitly.
(be aware that a keylogger or any other cyberspying activity can be illegal when used without consent and without permission; in most countries, that could send you to jail: in France, Article 323-1 du Code Pénal punishes that by at least 2 years of jail; and most other countries have similar laws)
the kernel module will send the information to another process [....] it will save it to a database
This is practically very difficult to get (and you look confused). Databases are in user-land (e.g. some RDBMS like PostGresSQL, or some library accessing files like sqlite). Notice that a kernel driver cannot (easily and reliably) even access to files.
All application programs (and most daemons & servers) on Linux are started with execve(2) (e.g. by some unix shell process, or by some daemon, etc...), and I see no reason for you to make an exception. However, some programs (mostly init, but also a few others, e.g. /sbin/hotplug) are started by the kernel, but this is exceptional (and should be avoided, and you don't need that).
How can I call a process from the kernel
You should not do that. I see no reason for your program to avoid being started by execve from some other process (perhaps your init, e.g. systemd).
and also send him the data?
Your process, as all other processes, is interacting with the kernel thru system calls (listed in syscalls(2)). So your application program could use read(2), write(2), poll(2) etc.. Be aware of netlink(7).
how can I return a message to the kernel from the process?
You don't. Use system calls, initiated by application code.
the kernel will lock the computer if the user is not authenticated.
This does not have any sense. Screen locking is a GUI artefact (so is not done by kernel code, but by ad-hoc daemon processes). Of course some processes do continue to run when locking is enabled. And many processes are daemons or servers which don't belong to "the" user (and continue to run when "the computer is locked"). At heart, Linux & POSIX is a multi-user and multi-tasking operating system. Even on a desktop Linux system used by a single physical person, you have dozens of users (i.e. uid-s many of them specialized to a particular feature, look into your /etc/passwd file, see passwd(5)) and more than a hundred processes (each having its pid), use top(1) or ps(1) as ps auxw to list them.
I believe you have the wrong approach. Take first several days or weeks to understand more about Linux from the application point of view. So read some book about Linux programming, e.g. ALP or something newer. Read also something like: Operating Systems: Three Easy Pieces
Be aware that in practice, most Linux systems having a desktop environment are using some display server. So the (physical) keyboard is handled by the X11 or Wayland server. You need to read more about your display server (with X11, things like EWMH).
Hence, you need to be much more specific. You are likely to need to interact with the display server, not the kernel directly.
At last, a rule of thumb is to avoid bloating your kernel with extra and useless driver code. You very probably can do your thing in userland entirely.
So, spend a week or more reading about OSes & Linux before coding a single line of code. Avoid kernel modules, they will bite you, and you probably don't need them (but you might need to hack your display server or simply your window manager; of course details are different with X11 and with Wayland). Read also about multiseat Linux systems.
At last, most Linux distributions are made of free software, whose source code you can study. So take time to look into the source code of relevant software for your (ill-defined) goals. Use also strace(1) to understand the system calls dynamically done by commands and processes.

cannot find pthread.c in linux folder

I have downloaded kernel and the kernel resides in folder called Linux-2.6.32.28 in which I can find /Kernel/Kthread.c
I found Kthread.c but I cannot find pthread.c in Linux-2.6.32.28
I found Kthread.c in Linux-3.13/Kernel and Linux-4.7.2/Kernel
locate pthread.c finds file pthread.c in Computer/usr folder that comes when I install Ubuntu but pthread.c is not available in downloaded folders Linux-2.6.32.28, Linux-3.13, Linux-4.7.2
MORE: There are two sets of function calls. 1. System Calls 2. Library Calls.
For a computer to do any task it has to use hardware resources. So, how library calls differ from system calls?
System calls always use kernel which means hardware.
Library calls means no usage of kernel or hardware?
I know that library calls sometimes may resolve to system call.
What I want to know is that, if every set of function calls uses hardware then to what degree system calls will use hardware resources when compared with library calls and vice-versa.
Whether a function call is System or Library, at least hardware resource like RAM has to be utilized. Right?
Read first pthreads(7). It explains you that pthreads are implemented in the C standard library as nptl(7).
The C standard library is the cornerstone of Linux systems, and you might have several variants of it; however, most Linux distributions have only one libc, often the GNU glibc, which contains the NPTL. You might use another C standard library (such as musl-libc or dietlibc). With care, you can have several C standard libraries co-existing on your system.
The C standard library (and every user-space program) is using system calls to interact with the kernel. They are listed in syscalls(2). BTW most C standard library implementations on Linux are free software, so you can study (or even improve) their source code if you want to. You often use a system call thru the small wrapper C function (e.g. write(2)) for it in your C standard library, and even more often thru some higher-level function (e.g. fprintf(3)) provided by your C standard library.
Pthreads are implemented (in the NPTL layer of glibc) using low-level stuff like clone(2) and futex(7) and a bit of assembler code. But you generally won't use these directly unless you are implementing a thread library (like NPTL).
Most programs are using the libc and linking with it (and also with crt0) as a shared library, which is /lib/x86_64-linux-gnu/libc.so.6 on my Debian/Sid/x86-64. However, you might (but you usually don't) invoke system calls directly thru some assembler code (e.g. using a SYSCALL or SYSENTER machine instruction). See also this.
The question was edited to also ask
What I want to know is that, if every set of function calls uses hardware then to what degree system calls will use hardware resources
Please read a lot more about operating systems. So read carefully Operating Systems: Three Easy pieces (a freely downloadable textbook) and read about Instruction Set Architecture and Computer Architecture. Study several of them, e.g. x86-64 and RISC-V (or ARM, PowerPC, etc...). Read about CPU modes and virtual memory.
You'll find out that the OS manages physical resources (including RAM and cores of processors). Each process has its own virtual address space. So from a user-space point of view, a process don't use directly hardware (e.g. it runs in virtual address space, not in RAM), it runs in some virtual machine (provided by the OS kernel) defined by the system calls and the ISA (the unpriviledged machine instructions).
Whether a function call is System or Library, at least hardware resource like RAM has to be utilized. Right?
Wrong, from a user-space point of view. All hardware resources are (by definition) managed by the operating system (which provides abstractions thru system calls). An ordinary application executable program uses the abstractions and software resources (files, processes, file descriptors, sockets, memory mappings, virtual address space, etc etc...) provided by the OS.
(so several books are required to really answer your questions; I gave some references, please follow them and read a lot more; we can't explain everything here)
Regarding your second cluster of questions: Everything a computer does is ultimately done by hardware. But we can make some distinctions between different kinds of hardware within the computer, and different kinds of programs interacting with them in different ways.
A modern computer can be divided into three major components: central processing unit (CPU), main memory (random access memory, RAM), and "peripherals" such as disks, network transceivers, displays, graphics cards, keyboards, mice, etc. Things that the CPU and RAM can do by themselves without involving peripherals in any way are called computation. Any operation involving at least one peripheral is called input/output (I/O). All programs do some of both, but some programs mostly do computation, and other programs mostly do I/O.
A modern operating system is also divided into two major components: the kernel, which is a self-contained program that is given special abilities that no other program has (such as the ability to communicate with peripherals and the ability to control the allocation of RAM), and is responsible for supervising execution of all the other programs; and the user space, which is an unbounded set of programs that have no such special abilities.
User space programs can do computation by themselves, but to do I/O they must, essentially, ask the kernel to do it for them. A system call is a request from a user-space program to the kernel. Many system calls are requests to perform I/O, but the kernel can also be requested to do other things, such as provide general information, set up communication channels among programs, allocate RAM, etc. A library is a component of a user space program. It is, essentially, a collection of "functions" that someone else has written for you, that you can use as if you wrote them yourself. There are many libraries, and most of them don't do anything out of the ordinary. For instance, zlib is a library that (provides functions that) compress and uncompress data.
However, "the C library" is special because (on all modern operating systems, except Windows) it is responsible for interacting directly with the kernel. Nearly all programs, and nearly all libraries, will not make system calls themselves; they will instead ask the C library to do it for them. Because of this, it can often be confusing trying to tell whether a particular C library function does all of its work itself or whether it "wraps" one or more system calls. The pthreads functions in particular tend to be a complicated tangle of both. If you want to learn how operating systems are put together at their lower levels, I recommend you start with something simpler, such as how the stdio.h "buffered I/O" routines (fopen, fclose, fread, fwrite, puts, etc) ultimately call the unistd.h/fcntl.h "raw I/O" routines (open, close, read, write, etc) and how the latter set of functions are just wrappers around system calls.
(Assigning the task of wrapping system calls to the runtime library for the C programming language is a historical accident and we probably wouldn't do it that way again if we were going to start over from scratch.)
pthread is POSIX thread. It is a library that helps application to create threads in OS. Kthread in kernel source code is for kernel threads in linux.
POSIX is a standard defined by IEEE to maintain the compatibility between different OS.
So pthread source code cannot be seen in linux kernel source code.
you may refer this link for pthread source code http://www.gnu.org/software/hurd/libpthread.html

Dependency of Run time library on operating system

I was going through this tutorial about how to write a minimalist kernel. I read this in between :
The Run-Time Library
A major part of writing code for your OS is rewriting the run-time library, also known as libc. This is because
the RTL is the most OS-dependent part of the compiler package: the C
RTL provides enough functionality to allow you to write portable
programs, but the inner workings of the RTL are dependent on the OS in
use. In fact, compiler vendors often use different RTLs for the same
OS: Microsoft Visual C++ provides different libraries for the various
combinations of debug/multi-threaded/DLL, and the older MS-DOS
compilers offered run-time libraries for up to 6 different memory
models.
I am kind of confused with this part. Suppose I write my kernel in C code and against the advice use the inbuilt printf() function to print something. Finally my code will be translated to the machine code. When it will be executed, processer will directly run it. Why does the author says :
inner workings of the RTL are dependent on the OS in use ?
There are two separate issues:
What will printf() do when run inside of your kernel? Most likely it will crash or do nothing, since the RTL of the C compiler you use to develop your kernel is probably assuming some runtime environment with console, operating system, etc. Even if you're using a freestanding implementation of C/C++, the runtime will likely take over serial ports or whatnot to perform the output. You don't want that, probably, since your kernel's drivers will control the I/O. So you need to reimplement the underlying file I/O from the RTL.
What will printf() do when run in a user process that runs on top of your kernel? If the kernel protects access to hardware resources, it can't do anything. The underlying file I/O code from the RTL has to be aware of how to communicate with the kernel to open whatever passes for standard input/output "files" and to perform data exchange.
You need to be aware of whether you're using a free-standing or hosted implementation of the C/C++ compiler + RTL, and all of the implications. For kernel development, you'll be using a free-standing implementation. For userspace development, you'll want a hosted implementation, perhaps a cross-compiler, but the runtime library must be written as for a hosted implementation. Note that in both cases you can use the same compiler, you just need to point it to appropriate header files and libraries. On Linux, for example, kernel and userspace development can be done using the very same gcc compiler, with different headers and libraries.
The processor has no clue what a console is, or what a kernel is. Some code has to actually access the hardware. When you take printf() from a hosted C/C++ implementation, that implementation, somewhere deep in its guts, will invoke a system call for the particular platform it was meant to run on. That system call is meant to write to some abstraction that wraps the "console". On the other side of this system call is kernel code that will push this data to some hardware. It may not even be hardware directly, it may well be userspace of another process.
For example, whenever you run things in a GUI-based terminal on a Unix machine (KDE's Konsole, X11 xterm, OS X Terminal, etc.), the userland process invoking printf() has very, very far to go before anything hits hardware. Namely (even this is simplified!):
printf() writes data to a buffer
The buffer is flushed to (written to) a file handle. The write() library function is called.
The write() library function invokes a system call that transfers the control over to the kernel.
The kernel code copies the data from the userspace pages, since those can vanish at any time, to a kernel-side non-paged buffer.
The kernel code invokes the write handler for a given file handle - a file handle, in many kernels, is implemented as class with virtual methods.
The file handle happens to be a pseudo-terminal (pty) slave. The write method passes the data to the pty master.
The pty master fills up the read buffer of given pseudo-terminal, and wakes up the process waiting on the related file handle.
The process implementing the GUI terminal wakes up and read()s the file handle. This goes through the library to a syscall.
The kernel invokes the read handler for the pty master file handle.
The read handler copies its buffered data to the userspace.
The syscall returns.
The terminal process takes the data, parses it for control codes, and updates its internal data structure representing the emulated screen. It also queues an update event in the event queue. Control returns to the event loop of the GUI library/framework. This is done through an event since those events are usually coalesced. If there's a lot of data available, it will be all processed to update the screen data structure before anything gets repainted.
The event dispatcher dispatches the update/repaint event to the "screen" widget/window.
The event handler code in the widget/window uses the internal data structure to "paint" somewhere. Usually it'd be on a bitmap backing store.
The GUI library/framework code signals the operating system's graphics driver that new data is available on the backing store.
Again, through a syscall, the control is passed over to the kernel. The graphics driver running in the kernel will do the necessary magic on the graphics hardware to pass the backing bitmap to the screen. It may be an explicit memory copy, or a simple queuing of a texture copy with the graphics hardware.
Printf() is a high-level function that can be independent of the OS. It is however just part of the puzzle, it has dependencies itself. It needs to be able to write to stdout. Which will result in low-level OS dependent system calls, like create() to open the stdout stream and write() to send printf output there. Different OSes have different system calls so there's always an adaption layer, there will be in yours.
So sure, you can make printf() work in your kernel. Actually seeing the output of calls to printf() is going to be the real problem to solve. Nothing like a terminal window in kernel mode.

Debugging tools for Multithreaded Apps

What free tools can I use to debug multithreaded programs created using the pthread library in Linux? (Besides pen & paper, of course...)
The usual way of printing debug messages isn't working out very well.
Debugging concurrent programs is inherently difficult, because the debugging tool tends to change the scheduling (often making it tamer so that the bug disappears).
One technique I've had some success with is to log to a data structure that is not protected by a lock. Then, once the system is idle, print the data structure or look at it in a debugger. The important thing is to avoid making a system call or invoking a synchronization primitive when logging, so that logging has minimal influence on the scheduler.
static char *log_buffer[LOG_BUFFER_LENGTH];
static size_t log_index;
#define LOG(message) (log_buffer[log_index++] = (message))
If your thread is interrupted in the middle of logging, the log buffer will become inconsistent. This is improbable enough to be useful for debugging, though it must be kept in mind. I've never tried this on a multiprocessor machine; I would expect the lack of synchronization to make the log buffer inconsistent very quickly in practice¹.
¹
Which is one more reason not to do multithreaded programming on multiprocessor machines. Use message passing instead.
Both the GNU gdb debugger and its ddd graphical front-end support threads.

Resources