I used to handle my log with Syslog. I am currently trying to switch to syslog-ng and also to update the format of my log to follow JSON standard.
First I define a syslog-ng configuration file :
#version: 3.2X
filter f_middleware {
facility("local1");
};
template t_json_filetemplate {
template("{\"timestamp\":\"${ISODATE}\",${MESSAGE}\n");
};
source s_kernel {
file("/proc/kmsg" program_override("kernel"));
};
source s_middleware {
unix-stream("/dev/log");
};
destination d_kernel {
file("/data/logs/kern.log");
};
destination d_middleware {
file("/data/logs/middleware.log", template(t_json_filetemplate));
};
log {
source(s_kernel);
destination(d_kernel);
};
log {
source(s_middleware);
filter(f_middleware);
destination(d_middleware);
};
In my middleware c code, I
openlog("middleware", 0, LOG_LOCAL1);
I format my log with macro in my code to handle the expected json format from my custom server:
#ifdef DEBUG
#define _PRINT_DEBUG(M, ...) \
do { \
syslog(LOG_DEBUG, "\"category\":\"%s\",\"level\":\"DEBUG\",\"message\":\"" M "%s\"}\n", __DIR__, __VA_ARGS__); \
} while (0)
#else
The problem is if I tried to display a file content with Syslog, the end of the message
is truncated. it is working correctly for other types of message of course.
Here for example I try to print the content of the file /proc/cmdline
{"timestamp":"2020-04-20T16:55:50+02:00","category":"XXX_MANAGER","level":"DEBUG","message":"root ptr : root=/dev/xxx
The '}' is missing at the end. It is because there is a '\n' at the end of the file ?
I also tried to add some stuff about escaping in my config file :
destination d_middleware {
file("/data/logs/middleware.log", template("{\"timestamp\":\"${ISODATE}\",${MESSAGE}\n") template-escape(no));
No better result...
A guess that I am using syslog-ng in a bad way...
More complete explanation of the problem after some investigation
https://lists.balabit.hu/pipermail/syslog-ng/2020-April/025836.html
It is because there is a '\n' at the end of the file?
Yes, unix-stream() uses \n to separate messages. You should escape your message in your middleware.
Alternatively, you can use, for example, unix-dgram() with flags(no-parse). You may need to send your data to a SOCK_DGRAM socket in that case, but on Linux, syslog() is actually dgram-based.
From the glibc manual:
openlog may or may not open the /dev/log socket, depending on option. If it does, it tries to open it and connect it as a stream socket. If that doesn’t work, it tries to open it and connect it as a datagram socket.
Related
I have an application project running on Linux environment, which includes libuv and another third-party library, the third-party library provides APIs for starting a TCP connection to remote server (say xxx_connect()) and getting file descriptor of the active connection (say xxx_get_socket()) . So far I managed to get valid file descriptor from xxx_get_socket() after xxx_connect() completed successfully, and initialize uv_poll_t handle with that file descriptor in my program.
Currently I am working on reconnecting function, after reconnecting the same server (by running xxx_connect() again), xxx_get_socket() returns different file descriptor, that means it is necessary to update io_watcher.fd member of a uv_poll_t handle to receive data in the new active connection.
AFAIK uv_poll_init() internally invokes uv__io_check_fd() , uv__nonblock() and uv__io_init() , it seems possible to modify io_watcher.fd of a uv_poll_t handle without closing the handle and then initializing it again (see sample code below), which has extra latency. However I'm not sure if it is safe to do so, I don't know whether io_watcher.fd member of a uv_poll_t handle is referenced elsewhere in libuv (e.g. uv_run()) which makes thing more complex. Is my approach feasible or should I re-initialize the uv_poll_t handle in such case ? Appreciate any feedback.
Possible approach , simplified sample code :
int uv_poll_change_fd( uv_poll_t *handle, int new_fd ) {
if (uv__fd_exists(handle->loop, new_fd))
// ..... some code ....
err = uv__io_check_fd(handle->loop, new_fd);
if(err)
// ..... some code ....
err = uv__nonblock(new_fd, 1);
// ..... some code ....
handle->io_watcher.fd = new_fd;
}
I just realized, that the PerlIO layer seems to do something more than just (more or less) easily wrap the stdio.h-functions.
If I try to use a file-descriptor resolved via PerlIO_stdout() and PerlIO_fileno() with functions from stdio.h, this fails.
For example:
PerlIO* perlStdErr = PerlIO_stderr();
fdStdErrOriginal = PerlIO_fileno(perlStdErr);
relocatedStdErr = dup(fdStdOutOriginal);
_write(relocatedStdErr, "something", 8); //<-- this fails
I've tried this with VC10. The embedded perl program is executed from a different context - so it's not possible to use PerlIO from the context where the write to the relocatedStdErr is performed.
For the curious: I need to execute a perl script and forward the output of the script's stdout/stderr to a log whilst keeping the ability to write on stdout for myself. Moreover this should work platform independent (linux, windows console application, win32 desktop application). Just to forward the stdout/stderr doesn't work in Win32 desktop applications since there is none ;) - you need to use the perl's stdout/stderr.
Needed solution: Be able to write on a filehandle (or descriptor) derived from perlio NOT using the PerlIO stack.
EDIT - my solution:
As Story Teller was pointing to PerlIO_findFILE, this did the trick.
So here an excerpt of the code - see the comments inside for descriptions:
FILE* stdErrFILE = PerlIO_findFILE(PerlIO_stderr()); //convert to Perl's stderr to stdio FILE handle
fdStdErrOriginal = _fileno(stdErrFILE); //get descriptor using MSVC
if (fdStdErrOriginal >= 0)
{
relocatedStdErr = _dup(fdStdErrOriginal); //relocate stdErr for external writing using MSVC
if (relocatedStdErr >= 0)
{
if (pipe(fdPipeStdErr) == 0) //create pipe for forwarding stdErr - USE PERL's IO since win32subsystem(non-console) "_pipe" doesn't work
{
if (dup2(fdPipeStdErr[1], fdStdErrOriginal) >= 0) //hang pipe on stdErr - USE PERL's IO (since it's created by perl)
{
close(fdPipeStdErr[1]); //close the now duplicated writer on stdErr for further usage - USE PERL's IO (since it's created by perl)
//"StreamForwarder" creates a thread that catches/reads the pipe's input and forwards it to the processStdErrOutput function (using the PerlIO)
stdErrForwarder = new StreamForwarder(fdPipeStdErr[0], &processStdErrOutput, PerlIO_stderr());
return relocatedStdErr; //return the relocated stdErr to be able to '_write' onto it
}
}
}
}
...
...
_write(relocatedStdErr, "Hello Stackoverflow!", 20); //that works :)
One interesting thing that I actually don't understand is, that the perl documentation says that is't necessary to #define PERLIO_NOT_STDIO 0 to be able to use PerlIO_findFILE(). But for me, that works fine without it and further I like to use PerlIO and the stdio together anyway. That's a point I didn't figured out what is going on.
I need to save UDP packets to a file and would like to use the pcap format to reuse the various tools available (wireshark, tcpdump, ...).
There are some information in this thread but I can't find how to write the global file header 'struct pcap_file_header'.
pcap_t* pd = pcap_open_dead(DLT_RAW, 65535);
pcap_dumper_t* pdumper = pcap_dump_open(pd, filename);
struct pcap_file_header file_hdr;
file_hdr.magic_number = 0xa1b2c3d4;
file_hdr.version_major = 2;
file_hdr.version_minor = 4;
file_hdr.thiszone = 0;
file_hdr.sigfigs = 0;
file_hdr.snaplen = 65535;
file_hdr.linktype = 1;
// How do I write file_hdr to m_pdumper?
while( (len = recvmsg(sd, &msg_hdr, 0)) > 0 )
pcap_dump((u_char*)m_pdumper, &m_pcap_pkthdr, (const u_char*)&data);
How should I write the global file header?
If there is no specific pcap function available, how can I retrieve the file descriptor to insert the header using write()?
You shouldn't need to write that header, pcap_open_dead should do it for you. You only need to fill out and write that header yourself if you want to write the file directly instead of using pcap_dump and friends. There's an example here of a trivial program write out a pcap file with those functions.
original answer, concerning writing the file directly:
I can't remember exactly how this works, but I wrote a patch to redir a while ago that would write out pcap files, you may be able to use it as an example.
You can find it attached to this debian bug. (bug link fixed.)
Some of it is for faking the ethernet and IP headers, and may not be applicable as you're using pcap_dump_open and pcap_dump where as the patch linked above writes out the pcap file without using any libraries, but I'll leave this here anyway in case it helps.
If you are interested in UDP and TCP only, you should use DLT_EN10MB instead of DLT_RAW ( cf pcap_open_dead to simulate full UDP packets capture ).
It is much better when editing in WireShak.
By default, libavformat writes error messages to stderr, Like:
Estimating duration from bitrate, this may be inaccurate
How can I turn it off? or better yet, pipe it to my own neat logging function?
Edit: Redirecting stderr to somewhere else is not acceptable since I need it for other logging purposes, I just want libavformat to not write to it.
Looking through the code, it appears you can change the behavior by writing your own callback function for the av_log function.
From the description of this function in libavutil/log.h:
Send the specified message to the log if the level is less than or equal
to the current av_log_level. By default, all logging messages are sent to
stderr. This behavior can be altered by setting a different av_vlog callback
function.
The API provides a function that will allow you to define your own callback:
void av_log_set_callback(void (*)(void*, int, const char*, va_list));
In your case, you could write a simple callback function that discards the messages altogether (or redirects them to a dedicated log, etc.) without tainting your stderr stream.
Give av_log_set_level(level) a try!
include this header file
#include <libavutil/log.h>
add this code will disable the log
av_log_set_level(AV_LOG_QUIET);
You can redirect them to a custom file, it will redirect all cerr entry:
#include <iostream>
#include <fstream>
using namespace std;
int main()
{
ofstream file("file.txt");
streambuf *old_cerr = cerr.rdbuf();
cerr.rdbuf (file.rdbuf());
cerr << "test test test" << endl; // writes to file.txt
// ...
cerr.rdbuf (old_cerr); // restore orginal cerr
return 0;
}
Edit: After editing the question, i warn about above code that it will redirect all cerr entry stream to file.txt
I'm not familiar with libavformat, but if its code is unchangeable, you can temporary redirect cerr to a file before calling library's api and redirect it to original cerr again.(However this is ugly way)
Here is a telnet site:
telnet://202.85.101.136:8604/
It is from Hong Kong public library, can I write some programme to get the string / result from the telnet service, and send the request from C / Objective C? thz u.
Sure its possible. Telnet is a pretty simple protocol, you simply need to open a TCP socket and connect it to that IP and Port. When you first connect, the telnet server will send some negotiation requests using the binary protocol defined in RFC854, to which your client is expected to respond. Once negotiation is completed you communicate by simply sending and receiving ASCII data, normally a line at a time.
For a simple "get some data from a host" telnet sessions where you aren't trying to have a real interactive session, it sometimes works to simply accept all the servers negotiation settings to avoid implementing the whole negotiation protocol. To do this, just look for the server to send you several 3-byte commands in the format of: 0xFF 0xFD xx, which is basically the server telling you "I want you to use option X", just respond to this with 0xFF 0xFB xx, which basically is just you agreeing to whatever the server is asking for. Then when you get passed negotiations, you just have to receive lines with a socket read and send commands with a socket write.
If you have a telnet program already on your system, you can use it to do all the connection work for you. Here's a program for gnu/Linux that you can use as a starting point.
It uses popen to execute the system's telnet command. Then it just reads all data from the pipe (stdout if you just executed the telnet command by itself from the shell) and prints it. When there's no more data to read, it exits.
You can send data to the server by opening the pipe in rw mode instead of r and then writing like you would any other file. You could conditionally do stuff like scan your input for Username: and then send a username string too, for instance.
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char *argv[])
{
const char *cmd = "telnet 202.85.101.136 8604";
char buffer[256];
FILE *pipe = popen(cmd, "r");
if( !pipe ) { perror("popen"); exit(-1); }
while( fgets(buffer, sizeof(buffer), pipe) != NULL &&
!feof(pipe) )
{
if( ferror(pipe) ) { perror("fgets"); break; }
/* Here you do whatever you want with the data. */
printf("%s", buffer);
}
pclose(pipe);
return 0;
}
If you're using Windows, this link explains the alternative to popen.
There's also a program called Expect that can help you automate stuff like this.