As described in this question: Openssl C++ get expiry date, there is the possibility to write an ASN1 time into a BIO buffer and then read it back into a custom buffer buf:
BIO *bio;
int write = 0;
bio = BIO_new(BIO_s_mem());
if (bio) {
if (ASN1_TIME_print(bio, tm))
write = BIO_read(bio, buf, len-1);
BIO_free(bio);
}
buf[write]='\0';
return write;
How could this be achieved without using BIO at all? The ASN1_TIME_print function is only present when OPENSSL_NO_BIO is not defined. Is there a way to write the time directly into a given buffer?
You can try the sample code below. It doesn't use BIO, but should give you the same output as the OP's example. If you don't trust the ASN1_TIME string, you'll want to add some error checking for:
notBefore->data is > 10 chars
each char value is between '0' and '9'
values for year, month, day, hour, minute, second
type
You should test for the type (i.e. UTC), if you expect multiple types.
You should also test whether or not the date/time is GMT and add that to the string if you want the output to match exactly as if using BIOs. see:
openssl/crypto/asn1/t_x509.c - ASN1_UTCTIME_print or ASN1_GENERALIZEDTIME_print
ASN1_TIME* notBefore = NULL;
int len = 32;
char buf[len];
struct tm tm_time;
notBefore = X509_get_notBefore(x509_cert);
// Format ASN1_TIME with type UTC into a tm struct
if(notBefore->type == V_ASN1_UTCTIME){
strptime((const char*)notBefore->data, "%y%m%d%H%M%SZ" , &tm_time);
strftime(buf, sizeof(char) * len, "%h %d %H:%M:%S %Y", &tm_time);
}
// Format ASN1_TIME with type "Generalized" into a tm struct
if(notBefore->type == V_ASN1_GENERALIZEDTIME){
// I didn't look this format up, but it shouldn't be too difficult
}
I think this should be possible, at least in terms of writing the time directly into a given buffer -- but you'll still need to use BIOs.
Ideally, BIO_new_mem_buf would suit, given that it creates an in-memory BIO using a given buffer as the source. Unfortunately, that function treats the given buffer as read-only, which is not what we want. However, we can create our own function (let's call it BIO_new_mem_buf2), based on the BIO_new_mem_buf source code:
BIO *BIO_new_mem_buf2(void *buf, int len)
{
BIO *ret;
BUF_MEM *b;
size_t sz;
if (!buf) {
BIOerr(BIO_F_BIO_NEW_MEM_BUF, BIO_R_NULL_PARAMETER);
return NULL;
}
sz = (size_t)len;
if (!(ret = BIO_new(BIO_s_mem())))
return NULL;
b = (BUF_MEM *)ret->ptr;
b->data = buf;
b->length = sz;
b->max = sz;
return ret;
}
This is just like BIO_new_mem_buf, except that a) the len argument must indicate the size of the given buffer, and b) the BIO is not marked "readonly".
With the above, you should now be able to call:
ASN1_TIME_print(bio, tm)
and have the time appear in your given buffer.
Note that I have not tested the above code, so YMMV. Hope this helps!
Related
I want to trim this string below (which is a log.txt file) into the substring below.
Since there are no delimeters, I can't use strtok().
So how could I do it?
Log file's contents:
[INFO][2019-10-2323:21:45.638]{"cmd":"set","objects":[{"type":"switch","data":["zwave-dc53:4-1"],"execution":{"command":"OnOff","params":{"on":true}}}],"raw":"DC530401010001","reqid": "0001"}
[INFO][2019-10-2323:22:11.936]{"cmd":"status","objects":[{"bridge_key":"zwave","data":[{"hash":"zwave-dc53:8-0","states":{"OnOff":{"on":false}}}],"type":"switch"}],"raw":"DC53010401000000","reqid": "0001"}
[INFO][2019-10-2323:22:29.232]{"cmd":"set","objects":[{"type":"switch","data":["zwave-dc53:4-1"],"execution":{"command":"OnOff","params":{"on":true}}}],"raw":"DC530401010001","reqid": "0002"}
[INFO][2019-10-2323:22:29.256]{"cmd":"status","objects":[{"bridge_key":"zwave","data":[{"hash":"zwave-dc53:8-0","states":{"OnOff":{"on":false}}}],"type":"switch"}],"raw":"DC53010401000000","reqid": "0002"}
[INFO][2019-10-2323:22:33.192]{"cmd":"set","objects":[{"type":"switch","data":["zwave-dc53:4-1"],"execution":{"command":"OnOff","params":{"on":true}}}],"raw":"DC530401010001","reqid": "0003}
[INFO][2019-10-2323:22:48.075]{"cmd":"status","objects":[{"bridge_key":"zwave","data":[{"hash":"zwave-dc53:8-0","states":{"OnOff":{"on":false}}}],"type":"switch"}],"raw":"DC53010401000000","reqid": "0003"}
[INFO][2019-10-2323:22:48.098]{"cmd":"set","objects":[{"type":"switch","data":["zwave-dc53:4-1"],"execution":{"command":"OnOff","params":{"on":true}}}],"raw":"DC530401010001","reqid": "0004"}
[INFO][2019-10-2323:22:52.034]{"cmd":"status","objects":[{"bridge_key":"zwave","data":[{"hash":"zwave-dc53:8-0","states":{"OnOff":{"on":false}}}],"type":"switch"}],"raw":"DC53010401000000","reqid": "0004"}
[INFO][2019-10-2323:25:58.509]{"cmd":"set","objects":[{"type":"switch","data":["zwave-dc53:4-1"],"execution":{"command":"OnOff","params":{"on":true}}}],"raw":"DC530401010001","reqid": "0005"}
[INFO][2019-10-2323:26:42.425]{"cmd":"status","objects":[{"bridge_key":"zwave","data":[{"hash":"zwave-dc53:8-0","states":{"OnOff":{"on":false}}}],"type":"switch"}],"raw":"DC53010401000000","reqid": "0005"}
[INFO][2019-10-2323:27:15.467]{"cmd":"set","objects":[{"type":"switch","data":["zwave-dc53:4-1"],"execution":{"command":"OnOff","params":{"on":true}}}],"raw":"DC530401010001","reqid": "0006"}
[INFO][2019-10-2323:27:42.030]{"cmd":"status","objects":[{"bridge_key":"zwave","data":[{"hash":"zwave-dc53:8-0","states":{"OnOff":{"on":false}}}],"type":"switch"}],"raw":"DC53010401000000","reqid": "0006"}
[INFO][2019-10-2323:32:45.088]{"cmd":"set","objects":[{"type":"switch","data":["zwave-ffa2:4-1"],"execution":{"command":"OnOff","params":{"on":true}}}],"raw":"FFA20401010001","reqid": "0033"}
[INFO][2019-10-2323:33:11.934]{"cmd":"status","objects":[{"bridge_key":"zwave","data":[{"hash":"zwave-ffa2:8-0","states":{"OnOff":{"on":false}}}],"type":"switch"}],"raw":"FFA2010401000000","reqid": "0007"}
[INFO][2019-10-2323:36:39.262]{"cmd":"set","objects":[{"type":"switch","data":["zwave-ffa2:4-1"],"execution":{"command":"OnOff","params":{"on":true}}}],"raw":"FFA20401010001","reqid": "0008"}
[INFO][2019-10-2323:36:39.267]{"cmd":"status","objects":[{"bridge_key":"zwave","data":[{"hash":"zwave-ffa2:8-0","states":{"OnOff":{"on":false}}}],"type":"switch"}],"raw":"FFA2010401000000","reqid": "0008"}
[INFO][2019-10-2323:36:39.267]{"cmd":"set","objects":[{"type":"switch","data":["zwave-ffa2:4-1"],"execution":{"command":"OnOff","params":{"on":true}}}],"raw":"FFA20401010001","reqid": "0022"}
[INFO][2019-10-2323:36:39.332]{"cmd":"status","objects":[{"bridge_key":"zwave","data":[{"hash":"zwave-ffa2:8-0","states":{"OnOff":{"on":false}}}],"type":"switch"}],"raw":"FFA2010401000000","reqid": "0009"}
The substring I want to find is the raw data's value, for example: FFA2010401000000
This should work to extract the raw data, assuming it's hex-encoded data (error checking, memory freeing, and proper headers are omitted for clarity):
#define RAW_STR ",\"raw\":\""
FILE *logfile = fopen( filename, "rb" );
char *line = NULL;
size_t len = 0;
for ( ;; )
{
ssize_t bytesRead = getline( &line, &len, logfile );
if ( bytesRead == -1 )
{
break;
}
char *rawData = strstr( line, RAW_STR );
if ( !raw )
{
continue;
}
// jump over the "raw":" string to the actual value
rawData += strlen( RAW_STR );
// assume the data is hex
unsigned long long value = strtoull( rawData, NULL, 16 );
...
}
This simple method depends on the log file being consistently formatted. If the log file doesn't always have the RAW_STR in that exact format, it won't work.
I've also assumed you're running on a POSIX system and have access to getline().
You might find regular expression useful in this scenario. Specifically regexec seems appropriate here.
int regexec(const regex_t *preg, const char *string, size_t nmatch,
regmatch_t pmatch[], int eflags);
Or a proper json parser. Checkout the c library section on the json format description website.
Let's consider this very nice and easy to use remux sample by horgh.
I'd like to achieve the same task: convert an RTSP H264 encoded stream to a fragmented MP4 stream.
This code does exactly this task.
However I don't want to write the mp4 onto disk at all, but I need to get a byte buffer or array in C with the contents that would normally written to disk.
How is that achievable?
This sample uses vs_open_output to define the output format and this function needs an output url.
If I would get rid of outputting the contents to disk, how shall I modify this code?
Or there might be better alternatives as well, those are also welcomed.
Update:
As szatmary recommended, I have checked his example link.
However as I stated in the question I need the output as buffer instead of a file.
This example demonstrates nicely how can I read my custom source and give it to ffmpeg.
What I need is how can open the input as standard (with avformat_open_input) then do my custom modification with the packets and then instead writing to file, write to a buffer.
What have I tried?
Based on szatmary's example I created some buffers and initialization:
uint8_t *buffer;
buffer = (uint8_t *)av_malloc(4096);
format_ctx = avformat_alloc_context();
format_ctx->pb = avio_alloc_context(
buffer, 4096, // internal buffer and its size
1, // write flag (1=true, 0=false)
opaque, // user data, will be passed to our callback functions
0, // no read
&IOWriteFunc,
&IOSeekFunc
);
format_ctx->flags |= AVFMT_FLAG_CUSTOM_IO;
AVOutputFormat * const output_format = av_guess_format("mp4", NULL, NULL);
format_ctx->oformat = output_format;
avformat_alloc_output_context2(&format_ctx, output_format,
NULL, NULL)
Then of course I have created 'IOWriteFunc' and 'IOSeekFunc':
static int IOWriteFunc(void *opaque, uint8_t *buf, int buf_size) {
printf("Bytes read: %d\n", buf_size);
int len = buf_size;
return (int)len;
}
static int64_t IOSeekFunc (void *opaque, int64_t offset, int whence) {
switch(whence){
case SEEK_SET:
return 1;
break;
case SEEK_CUR:
return 1;
break;
case SEEK_END:
return 1;
break;
case AVSEEK_SIZE:
return 4096;
break;
default:
return -1;
}
return 1;
}
Then I need to write the header to the output buffer, and the expected behaviour here is to print "Bytes read: x":
AVDictionary * opts = NULL;
av_dict_set(&opts, "movflags", "frag_keyframe+empty_moov", 0);
av_dict_set_int(&opts, "flush_packets", 1, 0);
avformat_write_header(output->format_ctx, &opts)
In the last line during execution, it always runs into segfault, here is the backtrace:
#0 0x00007ffff7a6ee30 in () at /usr/lib/x86_64-linux-gnu/libavformat.so.57
#1 0x00007ffff7a98189 in avformat_init_output () at /usr/lib/x86_64-linux-gnu/libavformat.so.57
#2 0x00007ffff7a98ca5 in avformat_write_header () at /usr/lib/x86_64-linux-gnu/libavformat.so.57
...
The hard thing for me with the example is that it uses avformat_open_input.
However there is no such thing for the output (no avformat_open_ouput).
Update2:
I have found another example for reading: doc/examples/avio_reading.c.
There are mentions of a similar example for writing (avio_writing.c), but ffmpeg does not have this available (at least in my google search).
Is this task really this hard to solve? standard rtsp input to custom avio?
Fortunately ffmpeg.org is down. Great.
It was a silly mistake:
In the initialization part I called this:
avformat_alloc_output_context2(&format_ctx, output_format,
NULL, NULL)
However before this I already put the avio buffers into format_ctx:
format_ctx->pb = ...
Also, this line is unnecessary:
format_ctx = avformat_alloc_context();
Correct order:
AVOutputFormat * const output_format = av_guess_format("mp4", NULL, NULL);
avformat_alloc_output_context2(&format_ctx, output_format,
NULL, NULL)
format_ctx->pb = avio_alloc_context(
buffer, 4096, // internal buffer and its size
1, // write flag (1=true, 0=false)
opaque, // user data, will be passed to our callback functions
0, // no read
&IOWriteFunc,
&IOSeekFunc
);
format_ctx->flags |= AVFMT_FLAG_CUSTOM_IO;
format_ctx->oformat = output_format; //might be unncessary too
Segfault is gone now.
You need to write a AVIOContext implementation.
I am writing a clone of find while learning C. When implementing the -ls option I've stumbled upon a problem that the getpwuid_r and getgrgid_r calls are really slow, the same applies to getpwuid and getgrgid. I need them to display the user/group names from ids provided by stat.h.
For example, listing the whole filesystem gets 3x slower:
# measurements were made 3 times and the fastest run was recorded
# with getgrgid_r
time ./myfind / -ls > list.txt
real 0m4.618s
user 0m1.848s
sys 0m2.744s
# getgrgid_r replaced with 'return "user";'
time ./myfind / -ls > list.txt
real 0m1.437s
user 0m0.572s
sys 0m0.832s
I wonder how GNU find maintains such a good speed. I've seen the sources, but they are not exactly easy to understand and to apply without special types, macros etc:
time find / -ls > list.txt
real 0m1.544s
user 0m0.884s
sys 0m0.648s
I thought about caching the uid - username and gid - groupname pairs in a data structure. Is it a good idea? How would you implement it?
You can find my complete code here.
UPDATE:
The solution was exactly what I was looking for:
time ./myfind / -ls > list.txt
real 0m1.480s
user 0m0.696s
sys 0m0.736s
Here is a version based on getgrgid (if you don't require thread safety):
char *do_get_group(struct stat attr) {
struct group *grp;
static unsigned int cache_gid = UINT_MAX;
static char *cache_gr_name = NULL;
/* skip getgrgid if we have the record in cache */
if (cache_gid == attr.st_gid) {
return cache_gr_name;
}
/* clear the cache */
cache_gid = UINT_MAX;
grp = getgrgid(attr.st_gid);
if (!grp) {
/*
* the group is not found or getgrgid failed,
* return the gid as a string then;
* an unsigned int needs 10 chars
*/
char group[11];
if (snprintf(group, 11, "%u", attr.st_gid) < 0) {
fprintf(stderr, "%s: snprintf(): %s\n", program, strerror(errno));
return "";
}
return group;
}
cache_gid = grp->gr_gid;
cache_gr_name = grp->gr_name;
return grp->gr_name;
}
getpwuid:
char *do_get_user(struct stat attr) {
struct passwd *pwd;
static unsigned int cache_uid = UINT_MAX;
static char *cache_pw_name = NULL;
/* skip getpwuid if we have the record in cache */
if (cache_uid == attr.st_uid) {
return cache_pw_name;
}
/* clear the cache */
cache_uid = UINT_MAX;
pwd = getpwuid(attr.st_uid);
if (!pwd) {
/*
* the user is not found or getpwuid failed,
* return the uid as a string then;
* an unsigned int needs 10 chars
*/
char user[11];
if (snprintf(user, 11, "%u", attr.st_uid) < 0) {
fprintf(stderr, "%s: snprintf(): %s\n", program, strerror(errno));
return "";
}
return user;
}
cache_uid = pwd->pw_uid;
cache_pw_name = pwd->pw_name;
return pwd->pw_name;
}
UPDATE 2:
Changed long to unsigned int.
UPDATE 3:
Added the cache clearing. It is absolutely necessary, because pwd->pw_name may point to a static area. getpwuid can overwrite its contents if it fails or simply when executed somewhere else in the program.
Also removed strdup. Since the output of getgrgid and getpwuid should not be freed, there is no need to require free for our wrapper functions.
The timings indeed indicate a strong suspicion on these functions.
Looking at your function do_get_group, there are some issues:
You use sysconf(_SC_GETPW_R_SIZE_MAX); for every call to do_get_group and do_get_user, definitely cache that, it will not change during the lifetime of your program, but you will not gain much.
You use attr.st_uid instead of attr.st_gid, which probably causes the lookup to fail for many files, possibly defeating the cacheing mechanism, if any. Fix this first, this is a bug!
You return values that should not be passed to free() by the caller, such as grp->gr_name and "". You should always allocate the string you return. The same issue is probably present in do_get_user().
Here is a replacement for do_get_group with a one shot cache. See if this improves the performance:
/*
* #brief returns the groupname or gid, if group not present on the system
*
* #param attr the entry attributes from lstat
*
* #returns the groupname if getgrgid() worked, otherwise gid, as a string
*/
char *do_get_group(struct stat attr) {
char *group;
struct group grp;
struct group *result;
static size_t length = 0;
static char *buffer = NULL;
static gid_t cache_gid = -1;
static char *cache_gr_name = NULL;
if (!length) {
/* only allocate the buffer once */
long sysconf_length = sysconf(_SC_GETPW_R_SIZE_MAX);
if (sysconf_length == -1) {
sysconf_length = 16384;
}
length = (size_t)sysconf_length;
buffer = calloc(length, 1);
}
if (!buffer) {
fprintf(stderr, "%s: malloc(): %s\n", program, strerror(errno));
return strdup("");
}
/* check the cache */
if (cache_gid == attr.st_gid) {
return strdup(cache_gr_name);
}
/* empty the cache */
cache_gid = -1;
free(cache_gr_name);
cache_gr_name = NULL;
if (getgrgid_r(attr.st_gid, &grp, buffer, length, &result) != 0) {
fprintf(stderr, "%s: getpwuid_r(): %s\n", program, strerror(errno));
return strdup("");
}
if (result) {
group = grp.gr_name;
} else {
group = buffer;
if (snprintf(group, length, "%ld", (long)attr.st_gid) < 0) {
fprintf(stderr, "%s: snprintf(): %s\n", program, strerror(errno));
return strdup("");
}
}
/* load the cache */
cache_gid = attr.st_gid;
cache_gr_name = strdup(group);
return strdup(group);
}
Whether the getpwuid and getgrgid calls are cached depends on how they are implemented and on how your system is configured. I recently wrote an implementation of ls and ran into a similar problem.
I found that on all modern systems I tested, the two functions are uncached unless you run the name service caching dæmon (nscd) in which case nscd makes sure that the cache stays up to date. It's easy to understand why this happens: Without an nscd, caching the information could lead to outdated output which is a violation of the specification.
I don't think you should rely on these functions caching the group and passwd databases because they often don't. I implemented custom caching code for this purpose. If you don't require to have up-to-date information in case the database contents change during program execution, this is perfectly fine to do.
You can find my implementation of such a cache here. I'm not going to publish it on Stack Overflow as I do not desire to publish the code under the MIT license.
I'm using Nanopb to try and send protobuf messages from a VxWorks based National Instruments Compact RIO (9025). My cross compilation works great, and I can even send a complete message with data types that don't require extra encoding. What's getting me is the callbacks. My code is cross compiled and called from LabVIEW and the callback based structure of Nanopb seems to break (error out, crash, target reboots, whatever) on the target machine. If I run it without any callbacks it works great.
Here is the code in question:
bool encode_string(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
{
char *str = "Woo hoo!";
if (!pb_encode_tag_for_field(stream, field))
return false;
return pb_encode_string(stream, (uint8_t*)str, strlen(str));
}
extern "C" uint16_t getPacket(uint8_t* packet)
{
uint8_t buffer[256];
uint16_t packetSize;
ExampleMsg msg = {};
pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
msg.name.funcs.encode = &encode_string;
msg.value = 17;
msg.number = 18;
pb_encode(&stream, ExampleMsg_fields, &msg);
packetSize = stream.bytes_written;
memcpy(packet, buffer, 256);
return packetSize;
}
And here's the proto file:
syntax = "proto2"
message ExampleMsg {
required int32 value = 1;
required int32 number = 2;
required string name = 3;
}
I have tried making the callback an extern "C" as well and it didn't change anything. I've also tried adding a nanopb options file with a max length and either didn't understand it correctly or it didn't work either.
If I remove the string from the proto message and remove the callback, it works great. It seems like the callback structure is not going to work in this LabVIEW -> C library environment. Is there another way I can encode the message without the callback structure? Or somehow embed the callback into the getPacket() function?
Updated code:
extern "C" uint16_t getPacket(uint8_t* packet)
{
uint8_t buffer[256];
for (unsigned int i = 0; i < 256; ++i)
buffer[i] = 0;
uint16_t packetSize;
ExampleMsg msg = {};
pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
msg.name.funcs.encode = &encode_string;
msg.value = 17;
msg.number = 18;
char name[] = "Woo hoo!";
strncpy(msg.name, name, strlen(name));
pb_encode(&stream, ExampleMsg_fields, &msg);
packetSize = stream.bytes_written;
memcpy(packet, buffer, sizeof(buffer));
return packetSize;
}
Updated proto file:
syntax = "proto2"
import "nanopb.proto";
message ExampleMsg {
required int32 value = 1;
required int32 number = 2;
required string name = 3 [(nanopb).max_size = 40];
}
You can avoid callbacks by giving a maximum size for the string field using the option (nanopb).max_size = 123 in the .proto file. Then nanopb can generate a simple char array in the structure (relevant part of documentation).
Regarding why callbacks don't work: just a guess, but try adding extern "C" also to the callback function. I assume you are using C++ there, so perhaps on that platform the C and C++ calling conventions differ and that causes the crash.
Does the VxWorks serial console give any more information about the crash? I don't remember if it does that for functions called from LabView, so running some test code directly from the VxWorks shell may be worth a try also.
Perhaps the first hurdle is how the code handles strings.
LabVIEW's native string representation is not null-terminated like C, but you can configure LabVIEW to use a different representation or update your code to handle LabVIEW's native format.
LabVIEW stores a string in a special format in which the first four bytes of the array of characters form a 32-bit signed integer that stores how many characters appear in the string. Thus, a string with n characters requires n + 4 bytes to store in memory.
LabVIEW Help: Using Arrays and Strings in the Call Library Function Node
http://zone.ni.com/reference/en-XX/help/371361L-01/lvexcodeconcepts/array_and_string_options/
I'm trying to write a wchar array to a file in C, however there is some sort of corruption and unrelevant data like variables and paths like this
c.:.\.p.r.o.g.r.a.m. .f.i.l.e.s.\.m.i.c.r.o.s.o.f.t. .v.i.s.u.a.l. .s.t.u.d.i.o. 1.0...0.\.v.c.\.i.n.c.l.u.d.e.\.x.s.t.r.i.n.g..l.i.s.t...i.n.s.e.r.t
are written on to the file along with the correct data (example) I have confirmed that the buffer is null-terminated and contains proper data.
Heres my code:
myfile = fopen("logs.txt","ab+");
fseek(myfile,0,SEEK_END);
long int size = ftell(myfile);
fseek(myfile,0,SEEK_SET);
if (size == 0)
{
wchar_t bom_mark = 0xFFFE;
size_t written = fwrite(&bom_mark,sizeof(wchar_t),1,myfile);
}
// in another func
while (true)
{
[..]
unsigned char Temp[512];
iBytesRcvd = recv(sclient_socket,(char*)&Temp,iSize,NULL);
if(iBytesRcvd > 0 )
{
WCHAR* unicode_recv = (WCHAR*)&Temp;
fwrite(unicode_recv,sizeof(WCHAR),wcslen(unicode_recv),myfile);
fflush(myfile);
}
[..]
}
What could be causing this?
recv() will not null-terminate &Temp, so wcslen() runs over the bytes actually written by recv(). You will get correct results if you just use iBytesReceived as byte count for fwrite() instead of using wcslen() and hoping the data received is correctly null-terminated (wide-NULL-terminated, that is):
fwrite(unicode_recv, 1, iBytesReceived, myfile);