Documentation for pb_ostream_from_buffer says
After writing, you can check stream.bytes_written to find out how much
valid data there is in the buffer. This should be passed as the
message length on decoding side.
So ideally, when I send the serialized data I need to also send the bytes_written as a parameter separate from the buffer.
The problem is that my interface only allows me to send one variable: the buffer.
QUESTION
How do I specify always serialize the struct with no optimizations so that bufsize in
pb_istream_from_buffer(const pb_byte_t *buf, size_t bufsize)
can be a constant (i.e. the macro that specifies the maximum size) instead of needing to pass stream.bytes_written?
According to the Protocol Buffers encoding specification there are variable size types (like int32, int64, string, etc) and fixed size types (like fixed32, fixed64, double, etc). Now, this variable size encoding is more than just an optimization, it's a part of the design and specification. So disabling this "optimization" by the means of Protocol Buffers is only possible if your data consists exclusively of fixed length types and has no repeated fields as long as the number of repetitions is not fixed. I presume that this is not the case, since you're asking this question. So the short answer is no, it's not possible by means of the library because it would violate the encoding specification.
But in my opinion the desired effect could be easily achieved by encoding the size into the buffer with little CPU and RAM overhead. I presume you know the maximum size of the message generated by nanopb, we denote it by MAX_MSG_SIZE. We call this message the payload message. Suppose that this MAX_MSG_SIZE can be represented by some integer type, which we denote by wrapped_size_t (e.g. uint16_t).
The idea is simple:
allocate the buffer slightly larger than MAX_MSG_SIZE;
write the payload message generated by nanopb at some offset into the allocated buffer;
use this offset to encode the size of the payload message at the beginning of the buffer;
transmit the whole buffer having the fixed size equal to MAX_MSG_SIZE + sizeof(wrapped_size_t) to the receiver;
upon reception decode the size of the payload message and pass both the decoded size and the payload message to pb_istream_from_buffer.
I attach the code to illustrate the idea. I used an example from nanopb repository:
#include <stdio.h>
#include <inttypes.h>
#include <string.h>
#include <pb_encode.h>
#include <pb_decode.h>
#include "simple.pb.h"
//#define COMMON_ENDIANNES
#ifdef COMMON_ENDIANNES
#define encode_size encode_size_ce
#define decode_size decode_size_ce
#else
#define encode_size encode_size_le
#define decode_size decode_size_le
#endif
typedef uint16_t wrapped_size_t;
/* Maximum size of the message returned by bytes_written */
const size_t MAX_MSG_SIZE = 11;
/* Size of the field storing the actual size of the message
* (as returned by bytes_written) */
const size_t SIZE_FIELD = sizeof(wrapped_size_t);
/* Fixed wrapped message size */
const size_t FIXED_MSG_SIZE = MAX_MSG_SIZE + sizeof(wrapped_size_t);
void print_usage(char *prog);
/* Get the address of the payload buffer from the transmitted buffer */
uint8_t* payload_buffer(uint8_t *buffer);
/* Encode the payload size into the transmitted buffer (common endiannes) */
void encode_size_ce(uint8_t *buffer, size_t size);
/* Decode the payload size into the transmitted buffer (common endiannes) */
wrapped_size_t decode_size_ce(uint8_t *buffer);
/* Encode the payload size into the transmitted buffer (little endian) */
void encode_size_le(uint8_t *buffer, size_t size);
/* Decode the payload size into the transmitted buffer (little endian) */
size_t decode_size_le(uint8_t *buffer);
int main(int argc, char* argv[])
{
/* This is the buffer where we will store our message. */
uint8_t buffer[MAX_MSG_SIZE + sizeof(wrapped_size_t)];
bool status;
if(argc > 2 || (argc == 2 && (!strcmp(argv[1], "-h") || !strcmp(argv[1], "--help"))))
{
print_usage(argv[0]);
return 1;
}
/* Encode our message */
{
/* Allocate space on the stack to store the message data.
*
* Nanopb generates simple struct definitions for all the messages.
* - check out the contents of simple.pb.h!
* It is a good idea to always initialize your structures
* so that you do not have garbage data from RAM in there.
*/
SimpleMessage message = SimpleMessage_init_zero;
/* Create a stream that will write to our buffer. */
pb_ostream_t stream = pb_ostream_from_buffer(payload_buffer(buffer),
MAX_MSG_SIZE);
if(argc > 1)
sscanf(argv[1], "%" SCNd32, &message.lucky_number);
else
{
printf("Input lucky number: ");
scanf("%" SCNd32, &message.lucky_number);
}
/* Encode the payload message */
status = pb_encode(&stream, SimpleMessage_fields, &message);
/* Wrap the payload, i.e. add the size to the buffer */
encode_size(buffer, stream.bytes_written);
/* Then just check for any errors.. */
if (!status)
{
printf("Encoding failed: %s\n", PB_GET_ERROR(&stream));
return 1;
}
}
/* Now we could transmit the message over network, store it in a file, etc.
* Note, the transmitted message has a fixed length equal to FIXED_MSG_SIZE
* and is stored in buffer
*/
/* But for the sake of simplicity we will just decode it immediately. */
{
/* Allocate space for the decoded message. */
SimpleMessage message = SimpleMessage_init_zero;
/* Create a stream that reads from the buffer. */
pb_istream_t stream = pb_istream_from_buffer(payload_buffer(buffer),
decode_size(buffer));
/* Now we are ready to decode the message. */
status = pb_decode(&stream, SimpleMessage_fields, &message);
/* Check for errors... */
if (!status)
{
printf("Decoding failed: %s\n", PB_GET_ERROR(&stream));
return 1;
}
/* Print the data contained in the message. */
printf("Your lucky number was %d; payload length was %d.\n",
(int)message.lucky_number, (int)decode_size(buffer));
}
return 0;
}
void print_usage(char *prog)
{
printf("usage: %s [<lucky_number>]\n", prog);
}
uint8_t* payload_buffer(uint8_t *buffer)
{
return buffer + SIZE_FIELD;
}
void encode_size_ce(uint8_t *buffer, size_t size)
{
*(wrapped_size_t*)buffer = size;
}
wrapped_size_t decode_size_ce(uint8_t *buffer)
{
return *(wrapped_size_t*)buffer;
}
void encode_size_le(uint8_t *buffer, size_t size)
{
int i;
for(i = 0; i < sizeof(wrapped_size_t); ++i)
{
buffer[i] = size;
size >>= 8;
}
}
size_t decode_size_le(uint8_t *buffer)
{
int i;
size_t ret = 0;
for(i = sizeof(wrapped_size_t) - 1; i >= 0; --i)
ret = buffer[i] + (ret << 8);
return ret;
}
UPD Ok, if, for some reason, you still wish to stick to the original GPB encoding there's another option available: fill the unused part of the buffer (i.e. the part after the last byte written by nanopb) with some valid data which will be ignored. For instance, you can reserve a field number which doesn't mark any field in your *.proto file but is used to mark the data which will be discarded by the GPB decoder. Let's denote this reserved field number as RESERVED_FIELD_NUMBER. This is used for backward compatibility but you can use it for your purpose as well. Let's call this filling-in the buffer with the dummy data sealing (perhaps there's a better term). This method also requires that you have at least 2 free bytes available to you after pb_encode.
So the idea of sealing is even simpler:
calculate how many buffer bytes is left unfilled after pb_encode;
mark the rest of the buffer as array of bytes with RESERVED_FIELD_NUMBER.
I attach the updated code, the main function is bool seal_buffer(uint8_t *buffer, size_t size), call it after pb_encode to seal the buffer and you're done. Currently, it has a limitation of sealing no more than 2 ** 28 + 4 bytes, but it could be easily updated to overcome this limitation.
#include <stdio.h>
#include <assert.h>
#include <inttypes.h>
#include <pb_encode.h>
#include <pb_decode.h>
#include "simple.pb.h"
/* Reserved field_number shouldn't be used for field numbering. We use it
* to mark the data which will be ignored upon reception by GPB parser.
* This number should be 1 to 15 to fit into a single byte. */
const uint8_t RESERVED_FIELD_NUMBER = 15;
/* Maximum size of the message returned by bytes_written (payload size) */
const size_t MAX_MSG_SIZE = 200;
/* Size of the transmitted message (reserve 2 bytes for minimal sealing) */
const size_t FIXED_MSG_SIZE = MAX_MSG_SIZE + 2;
void print_usage(char *prog);
/* Sealing the buffer means filling it in with data which is valid
* in the sense that a GPB parser accepts it as valid but ignores it */
bool seal_buffer(uint8_t *buffer, size_t size);
int main(int argc, char* argv[])
{
/* This is the buffer where we will store our message. */
uint8_t buffer[FIXED_MSG_SIZE];
bool status;
if(argc > 2 || (argc == 2 && (!strcmp(argv[1], "-h") || !strcmp(argv[1], "--help"))))
{
print_usage(argv[0]);
return 1;
}
/* Encode our message */
{
/* Allocate space on the stack to store the message data.
*
* Nanopb generates simple struct definitions for all the messages.
* - check out the contents of simple.pb.h!
* It is a good idea to always initialize your structures
* so that you do not have garbage data from RAM in there.
*/
SimpleMessage message = SimpleMessage_init_zero;
/* Create a stream that will write to our buffer. */
pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
if(argc > 1)
sscanf(argv[1], "%" SCNd32, &message.lucky_number);
else
{
printf("Input lucky number: ");
scanf("%" SCNd32, &message.lucky_number);
}
/* Now we are ready to encode the message! */
status = pb_encode(&stream, SimpleMessage_fields, &message);
/* Then just check for any errors.. */
if (!status)
{
fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&stream));
return 1;
}
/* Now the main part - making the buffer fixed-size */
assert(stream.bytes_written + 2 <= FIXED_MSG_SIZE);
if(!seal_buffer(buffer + stream.bytes_written,
FIXED_MSG_SIZE - stream.bytes_written))
{
fprintf(stderr, "Failed sealing the buffer "
"(filling in with valid but ignored data)\n");
return 1;
}
}
/* Now we could transmit the message over network, store it in a file or
* wrap it to a pigeon's leg.
*/
/* But because we are lazy, we will just decode it immediately. */
{
/* Allocate space for the decoded message. */
SimpleMessage message = SimpleMessage_init_zero;
/* Create a stream that reads from the buffer. */
pb_istream_t stream = pb_istream_from_buffer(buffer, FIXED_MSG_SIZE);
/* Now we are ready to decode the message. */
status = pb_decode(&stream, SimpleMessage_fields, &message);
/* Check for errors... */
if (!status)
{
fprintf(stderr, "Decoding failed: %s\n", PB_GET_ERROR(&stream));
return 1;
}
/* Print the data contained in the message. */
printf("Your lucky number was %d.\n", (int)message.lucky_number);
}
return 0;
}
void print_usage(char *prog)
{
printf("usage: %s [<lucky_number>]\n", prog);
}
bool seal_buffer(uint8_t *buffer, size_t size)
{
size_t i;
if(size == 1)
{
fprintf( stderr, "Cannot seal the buffer, at least 2 bytes are needed\n");
return false;
}
assert(size - 5 < 1<<28);
if(size - 5 >= 1<<28)
{
fprintf( stderr, "Representing the size exceeding 2 ** 28 + 4, "
"although it's not difficult, is not yet implemented\n");
return false;
}
buffer[0] = (15 << 3) + 2;
/* encode the size */
if(size - 2 < 1<<7)
buffer[1] = size - 2;
else
{
/* Size is large enough to fit into 7 bits (1 byte).
* For simplicity we represent the remaining size by 4 bytes (28 bits).
* Note that 1 byte is used for encoding field_number and wire_type,
* plus 4 bytes for the size encoding, therefore the "remaining size"
* is equal to (size - 5)
*/
size -= 5;
for(i = 0; i < 4; ++i)
{
buffer[i + 1] = i < 3? (size & 0x7f) | 0x80: size & 0x7f;
size >>= 7;
}
}
return true;
}
I have a string creating function in C which accepts an array of structs as it's argument and outputs a string based on a predefined format (like a list of list in python).
Here's the function
typedef struct
{
PacketInfo_t PacketInfo;
char Gnss60[1900];
//and other stuff...
} Track_json_t;
typedef struct
{
double latitude;
double longitude;
} GPSPoint_t;
typedef struct
{
UInt16 GPS_StatusCode;
UInt32 fixtime;
GPSPoint_t point;
double altitude;
unsigned char GPS_Satilite_Num;
} GPS_periodic_t;
unsigned short SendTrack()
{
Track_json_t i_sTrack_S;
memset(&i_sTrack_S, 0x00, sizeof(Track_json_t));
getEvent_Track(&i_sTrack_S);
//Many other stuff added to the i_sTrack_S struct...
//Make a JSON format out of it
BuildTrackPacket_json(&i_sTrack_S, XPORT_MODE_GPRS);
}
Track_json_t *getEvent_Track(Track_json_t *trk)
{
GPS_periodic_t l_gps_60Sec[60];
memset(&l_gps_60Sec, 0x00,
sizeof(GPS_periodic_t) * GPS_PERIODIC_ARRAY_SIZE);
getLastMinGPSdata(l_gps_60Sec, o_gps_base);
get_gps60secString(l_gps_60Sec, trk->Gnss60);
return trk;
}
void get_gps60secString(GPS_periodic_t input[60], char *output)
{
int i = 0;
memcpy(output, "[", 1); ///< Copy the first char as [
char temp[31];
for (i = 0; i < 59; i++) { //Run for n-1 elements
memset(temp, 0, sizeof(temp));
snprintf(temp, sizeof(temp), "[%0.8f,%0.8f],",
input[i].point.latitude, input[i].point.longitude);
strncat(output, temp, sizeof(temp));
}
memset(temp, 0, sizeof(temp)); //assign last element
snprintf(temp, sizeof(temp), "[%0.8f,%0.8f]]",
input[i].point.latitude, input[i].point.longitude);
strncat(output, temp, sizeof(temp));
}
So the output of the function must be a string of format
[[12.12345678,12.12345678],[12.12345678,12.12345678],...]
But at times I get a string which looks like
[[12.12345678,12.12345678],[55.01[12.12345678,12.12345678],...]
[[21.28211567,84.13454083],[21.28211533,21.22[21.28211517,84.13454000],..]
Previously, I had a buffer overflow at the function get_gps60secString, I fixed that by using snprintf and strncat.
Note: This is an embedded application and this error occur once or twice a day (out of 1440 packets)
Question
1. Could this be caused by an interrupt during the snprintf/strncat process?
2. Could this be caused by a memory leak, overwriting the stack or some other segmentation issue caused else where?
Basically I would like to understand what might be causing a corrupt string.
Having a hard time finding the cause and fixing this bug.
EDIT:
I used chux's function. Below is the Minimal, Complete, and Verifiable Example
/*
* Test code for SO question https://stackoverflow.com/questions/5216413
* A Minimal, Complete, and Verifiable Example
*/
#include <stdio.h>
#include <stdlib.h>
#include <memory.h>
#include <stdbool.h>
#include <signal.h>
#include <unistd.h>
typedef unsigned short UInt16;
typedef unsigned long UInt32;
#define GPS_PERIODIC_ARRAY_SIZE 60
#define GPS_STRING_SIZE 1900
/* ---------------------- Data Structs --------------------------*/
typedef struct
{
char Gnss60[GPS_STRING_SIZE];
} Track_json_t;
typedef struct
{
double latitude;
double longitude;
} GPSPoint_t;
typedef struct
{
UInt16 GPS_StatusCode;
UInt32 fixtime;
GPSPoint_t point;
double altitude;
unsigned char GPS_Satilite_Num;
} GPS_periodic_t;
/* ----------------------- Global --------------------------------*/
FILE *fptr; //Global file pointer
int res = 0;
int g_last = 0;
GPS_periodic_t l_gps_60Sec[GPS_PERIODIC_ARRAY_SIZE];
/* ----------------------- Function defs --------------------------*/
/* At signal interrupt this function is called.
* Flush and close the file. And safly exit the program */
void userSignalInterrupt()
{
fflush(fptr);
fclose(fptr);
res = 1;
exit(0);
}
/* #brief From the array of GPS structs we create a string of the format
* [[lat,long],[lat,long],..]
* #param input The input array of GPS structs
* #param output The output string which will contain lat, long
* #param sz Size left in the output buffer
* #return 0 Successfully completed operation
* 1 Failed / Error
*/
int get_gps60secString(GPS_periodic_t input[GPS_PERIODIC_ARRAY_SIZE],
char *output, size_t sz)
{
int cnt = snprintf(output, sz, "[");
if (cnt < 0 || cnt >= sz)
return 1;
output += cnt;
sz -= cnt;
int i = 0;
for (i = 0; i < GPS_PERIODIC_ARRAY_SIZE; i++) {
cnt = snprintf(output, sz, "[%0.8f,%0.8f]%s",
input[i].point.latitude, input[i].point.longitude,
i + 1 == GPS_PERIODIC_ARRAY_SIZE ? "" : ",");
if (cnt < 0 || cnt >= sz)
return 1;
output += cnt;
sz -= cnt;
}
cnt = snprintf(output, sz, "]");
if (cnt < 0 || cnt >= sz)
return 1;
return 0; // no error
}
/* #brief Create a GPS struct with data for testing. It will populate the
* point field of GPS_periodic_t. Lat starts from 0.0 and increases by 1*10^(-8)
* and Long will dstart at 99.99999999 and dec by 1*10^(-8)
*
* #param o_gps_60sec Output array of GPS structs
*/
void getLastMinGPSdata(GPS_periodic_t *o_gps_60sec)
{
//Fill in GPS related data here
int i = 0;
double latitude = o_gps_60sec[0].point.latitude;
double longitude = o_gps_60sec[0].point.longitude;
for (i = 0; i < 60; i++)
{
o_gps_60sec[i].point.latitude = latitude + (0.00000001 * (float)g_last +
0.00000001 * (float)i);
o_gps_60sec[i].point.longitude = longitude - (0.00000001 * (float)g_last +
0.00000001 * (float)i);
}
g_last = 60;
}
/* #brief Get the GPS data and convert it into a string
* #param trk Track structure with GPS string
*/
int getEvent_Track(Track_json_t *trk)
{
getLastMinGPSdata(l_gps_60Sec);
get_gps60secString(l_gps_60Sec, trk->Gnss60, GPS_STRING_SIZE);
return 0;
}
int main()
{
fptr = fopen("gpsAno.txt", "a");
if (fptr == NULL) {
printf("Error!!\n");
exit(1);
}
//Quit at signal interrupt
signal(SIGINT, userSignalInterrupt);
Track_json_t trk;
memset(&l_gps_60Sec, 0x00, sizeof(GPS_periodic_t) * GPS_PERIODIC_ARRAY_SIZE);
//Init Points to be zero and 99.99999999
int i = 0;
for (i = 0; i < 60; i++) {
l_gps_60Sec[i].point.latitude = 00.00000000;
l_gps_60Sec[i].point.longitude = 99.99999999;
}
do {
memset(&trk, 0, sizeof(Track_json_t));
getEvent_Track(&trk);
//Write to file
fprintf(fptr, "%s", trk.Gnss60);
fflush(fptr);
sleep(1);
} while (res == 0);
//close and exit
fclose(fptr);
return 0;
}
Note: Error was not recreated in the above code.
Because this doesn't have the strcat pitfalls.
I tested this function in the embedded application.
Through this I was able to find that the snprintf returns an error and the string created ended up to be:
[17.42401750,78.46098717],[17.42402083,53.62
It ended there (because of the return 1).
Does this mean that the data which was passed to snprints corrupted? It's a float value. How can it get corrupted?
Solution
The error have not been seen since I changed the sprintf function with one that doesn't directly deal with 64 bits of data.
Here's the function modp_dtoa2
/** \brief convert a floating point number to char buffer with a
* variable-precision format, and no trailing zeros
*
* This is similar to "%.[0-9]f" in the printf style, except it will
* NOT include trailing zeros after the decimal point. This type
* of format oddly does not exists with printf.
*
* If the input value is greater than 1<<31, then the output format
* will be switched exponential format.
*
* \param[in] value
* \param[out] buf The allocated output buffer. Should be 32 chars or more.
* \param[in] precision Number of digits to the right of the decimal point.
* Can only be 0-9.
*/
void modp_dtoa2(double value, char* str, int prec)
{
/* if input is larger than thres_max, revert to exponential */
const double thres_max = (double)(0x7FFFFFFF);
int count;
double diff = 0.0;
char* wstr = str;
int neg= 0;
int whole;
double tmp;
uint32_t frac;
/* Hacky test for NaN
* under -fast-math this won't work, but then you also won't
* have correct nan values anyways. The alternative is
* to link with libmath (bad) or hack IEEE double bits (bad)
*/
if (! (value == value)) {
str[0] = 'n'; str[1] = 'a'; str[2] = 'n'; str[3] = '\0';
return;
}
if (prec < 0) {
prec = 0;
} else if (prec > 9) {
/* precision of >= 10 can lead to overflow errors */
prec = 9;
}
/* we'll work in positive values and deal with the
negative sign issue later */
if (value < 0) {
neg = 1;
value = -value;
}
whole = (int) value;
tmp = (value - whole) * pow10[prec];
frac = (uint32_t)(tmp);
diff = tmp - frac;
if (diff > 0.5) {
++frac;
/* handle rollover, e.g. case 0.99 with prec 1 is 1.0 */
if (frac >= pow10[prec]) {
frac = 0;
++whole;
}
} else if (diff == 0.5 && ((frac == 0) || (frac & 1))) {
/* if halfway, round up if odd, OR
if last digit is 0. That last part is strange */
++frac;
}
/* for very large numbers switch back to native sprintf for exponentials.
anyone want to write code to replace this? */
/*
normal printf behavior is to print EVERY whole number digit
which can be 100s of characters overflowing your buffers == bad
*/
if (value > thres_max) {
sprintf(str, "%e", neg ? -value : value);
return;
}
if (prec == 0) {
diff = value - whole;
if (diff > 0.5) {
/* greater than 0.5, round up, e.g. 1.6 -> 2 */
++whole;
} else if (diff == 0.5 && (whole & 1)) {
/* exactly 0.5 and ODD, then round up */
/* 1.5 -> 2, but 2.5 -> 2 */
++whole;
}
//vvvvvvvvvvvvvvvvvvv Diff from modp_dto2
} else if (frac) {
count = prec;
// now do fractional part, as an unsigned number
// we know it is not 0 but we can have leading zeros, these
// should be removed
while (!(frac % 10)) {
--count;
frac /= 10;
}
//^^^^^^^^^^^^^^^^^^^ Diff from modp_dto2
// now do fractional part, as an unsigned number
do {
--count;
*wstr++ = (char)(48 + (frac % 10));
} while (frac /= 10);
// add extra 0s
while (count-- > 0) *wstr++ = '0';
// add decimal
*wstr++ = '.';
}
// do whole part
// Take care of sign
// Conversion. Number is reversed.
do *wstr++ = (char)(48 + (whole % 10)); while (whole /= 10);
if (neg) {
*wstr++ = '-';
}
*wstr='\0';
strreverse(str, wstr-1);
}
Here's (part of) my unabashedly opinionated guide on safe string handling in C. Normally, I would promote dynamic memory allocation instead of fixed-length strings, but in this case I'm assuming that in the embedded environment that might be problematic. (Although assumptions like that should always be checked.)
So, first things first:
Any function which creates a string in a buffer must be told explicitly how long the buffer is. This is non-negotiable.
As should be obvious, it's impossible for a function filling a buffer to check for buffer overflow unless it knows where the buffer ends. "Hope that the buffer is long enough" is not a viable strategy. "Document the needed buffer length" would be fine if everyone carefully read the documentation (they don't) and if the required length never changes (it will). The only thing that's left is an extra argument, which should be of type size_t (because that's the type of buffer lengths in the C library functions which require lengths).
Forget that strncpy and strncat exist. Also forget about strcat. They are not your friends.
strncpy is designed for a specific use case: ensuring that an entire fixed-length buffer is initialised. It is not designed for normal strings, and since it doesn't guarantee that the output is NUL-terminated, it doesn't produce a string.
If you're going to NUL-terminate yourself anyway, you might as well use memmove, or memcpy if you know that the source and destination don't overlap, which should almost always be the case. Since you'll want the memmove to stop at the end of the string for short strings (which strncpy does not do), measure the string length first with strnlen: strnlen takes a maximum length, which is precisely what you want in the case that you are going move a maximum number of characters.
Sample code:
/* Safely copy src to dst where dst has capacity dstlen. */
if (dstlen) {
/* Adjust to_move will have maximum value dstlen - 1 */
size_t to_move = strnlen(src, dstlen - 1);
/* copy the characters */
memmove(dst, src, to_move);
/* NUL-terminate the string */
dst[to_move] = 0;
}
strncat has a slightly more sensible semantic, but it's practically never useful because in order to use it, you already have to know how many bytes you could copy. In order to know that, in practice, you need to know how much space is left in your output buffer, and to know that you need to know where in the output buffer the copy will start. [Note 1]. But if you already know where the copy will start, what's the point of searching through the buffer from the beginning to find the copy point? And if you do let strncat do the search, how sure are you that your previously computed start point is correct?
In the above code snippet, we already computed the length of the copy. We can extend that to do an append without rescanning:
/* Safely copy src1 and then src2 to dst where dst has capacity dstlen. */
/* Assumes that src1 and src2 are not contained in dst. */
if (dstlen) {
/* Adjust to_move will have maximum value dstlen - 1 */
size_t to_move = strnlen(src1, dstlen - 1);
/* Copy the characters from src1 */
memcpy(dst, src1, to_move);
/* Adjust the output pointer and length */
dst += to_move;
dstlen -= to_move;
/* Now safely copy src2 to just after src1. */
to_move = strnlen(src2, dstlen - 1);
memcpy(dst, src2, to_move);
/* NUL-terminate the string */
dst[to_move] = 0;
}
It might be that we want the original values of dst and dstlen after creating the string, and it might also be that we want to know how many bytes we inserted into dst in all. In that case, we would probably want to make copies of those variables before doing the copies, and save the cumulative sum of moves.
The above assumes that we're starting with an empty output buffer, but perhaps that isn't the case. Since we still need to know where the copy will start in order to know how many characters we can put at the end, we can still use memcpy; we just need to scan the output buffer first to find the copy point. (Only do this if there is no alternative. Doing it in a loop instead of recording the next copy point is Shlemiel the Painter's algorithm.)
/* Safely append src to dst where dst has capacity dstlen and starts
* with a string of unknown length.
*/
if (dstlen) {
/* The following code will "work" even if the existing string
* is not correctly NUL-terminated; the code will not copy anything
* from src, but it will put a NUL terminator at the end of the
* output buffer.
*/
/* Figure out where the existing string ends. */
size_t prefixlen = strnlen(dst, dstlen - 1);
/* Update dst and dstlen */
dst += prefixlen;
dstlen -= prefixlen;
/* Proceed with the append, as above. */
size_t to_move = strnlen(src, dstlen - 1);
memmove(dst, src, to_move);
dst[to_move] = 0;
}
Embrace snprintf. It really is your friend. But always check its return value.
Using memmove, as above, is slightly awkward. It requires you to manually check that the buffer's length is not zero (otherwise subtracting one would be disastrous since the length is unsigned), and it requires you to manually NUL-terminate the output buffer, which is easy to forget and the source of many bugs. It is very efficient, but sometimes it's worth sacrificing a little efficiency so that your code is easier to write and easier to read and verify.
And that leads us directly to snprintf. For example, you can replace:
if (dstlen) {
size_t to_move = strnlen(src, dstlen - 1);
memcpy(dst, src, to_move);
dst[to_move] = 0;
}
with the much simpler
int copylen = snprintf(dst, dstlen, "%s", src);
That does everything: checks that dstlen is not 0; only copies the characters from src which can fit in dst, and correctly NUL-terminates dst (unless dstlen was 0). And the cost is minimal; it takes very little time to parse the format string "%s" and most implementations are pretty well optimised for this case. [Note 2]
But snprintf is not a panacea. There are still a couple of really important warnings.
First, the documentation for snprintf makes clear that it is not permitted for any input argument to overlap the output range. (So it replaces memcpy but not memmove.) Remember that overlap includes NUL-terminators, so the following code which attempts to double the string in str instead leads to Undefined Behaviour:
char str[BUFLEN];
/* Put something into str */
get_some_data(str, BUFLEN);
/* DO NOT DO THIS: input overlaps output */
int result = snprintf(str, BUFLEN, "%s%s", str, str);
/* DO NOT DO THIS EITHER; IT IS STILL UB */
size_t len = strnlen(str, cap - 1);
int result = snprintf(str + len, cap - len, "%s", str);
The problem with the second invocation of snprintf is that the NUL which terminates str is precisely at str + len, the first byte of the output buffer. That's an overlap, so it's illegal.
The second important note about snprintf is that it returns a value, which must not be ignored. The value returned is not the length of the string created by snprintf. It's the length the string would have been had it not been truncated to fit in the output buffer.
If no truncation occurred, then the result is the length of the result, which must be strictly less than the size of the output buffer (because there must be room for a NUL terminator, which is not considered part of the length of the result.) You can use this fact to check whether truncation occurred:
if (result >= dstlen) /* Output was truncated */
This can be used, for example, to redo the snprintf with a larger, dynamically-allocated buffer (of size result + 1; never forget the need to NUL-terminate).
But remember that the result is an int -- that is, a signed value. That means that snprintf cannot cope with very long strings. That's not likely to be an issue in embedded code, but on systems where it's conceivable that strings exceed 2GB, you may not be able to safely use %s formats in snprintf. It also means that snprintf is allowed to return a negative value to indicate an error. Very old implementations of snprintf returned -1 to indicate truncation, or in response to being called with buffer length 0. That's not standard behaviour according to C99 (nor recent versions of Posix), but you should be prepared for it.
Standard-compliant implementations of snprintf will return a negative value if the buffer length argument is too big to fit in a (signed) int; it's not clear to me what the expected return value is if the buffer length is OK but the untruncated length is too big for an int. A negative value will also be returned if you used a conversion which resulted in an encoding error; for example, a %lc conversion whose corresponding argument contains an integer which cannot be converted to a multibyte (typically UTF-8) sequence.
In short, you should always check the return value of snprintf (recent gcc/glibc versions will produce a warning if you do not), and you should be prepared for it to be negative.
So, with all that behind us, let's write a function which produces a string of co-ordinate pairs:
/* Arguments:
* buf the output buffer.
* buflen the capacity of buf (including room for trailing NUL).
* points a vector of struct Point pairs.
* npoints the number of objects in points.
* Description:
* buf is overwritten with a comma-separated list of points enclosed in
* square brackets. Each point is output as a comma-separated pair of
* decimal floating point numbers enclosed in square brackets. No more
* than buflen - 1 characters are written. Unless buflen is 0, a NUL is
* written following the (possibly-truncated) output.
* Return value:
* If the output buffer contains the full output, the number of characters
* written to the output buffer, not including the NUL terminator.
* If the output was truncated, (size_t)(-1) is returned.
*/
size_t sprint_points(char* buf, size_t buflen,
struct Point const* points, size_t npoints)
{
if (buflen == 0) return (size_t)(-1);
size_t avail = buflen;
char delim = '['
while (npoints) {
int res = snprintf(buf, avail, "%c[%f,%f]",
delim, points->lat, points->lon);
if (res < 0 || res >= avail) return (size_t)(-1);
buf += res; avail -= res;
++points; --npoints;
delim = ',';
}
if (avail <= 1) return (size_t)(-1);
strcpy(buf, "]");
return buflen - (avail - 1);
}
Notes
You will often see code like this:
strncat(dst, src, sizeof(src)); /* NEVER EVER DO THIS! */
Telling strncat not to append more characters from src than can fit in src is obviously pointless (unless src is not correctly NUL-terminated, in which case you have a bigger problem). More importantly, it does absolutely nothing to protect you from writing beyond the end of the output buffer, since you have not done anything to check that dst has room for all those characters. So about all it does is get rid of compiler warnings about the unsafety of strcat. Since this code is exactly as unsafe as strcat was, you probably would be better off with the warning.
You might even find a compiler which understands snprintf will enough to parse the format string at compile time, so the convenience comes at no cost at all. (And if your current compiler doesn't do this, no doubt a future version will.) As with any use of the *printf family, you should never try to economize keystrokes by
leaving out the format string (snprintf(dst, dstlen, src) instead of snprintf(dst, dstlen, "%s", src).) That's unsafe (it has undefined behaviour if src contains an unduplicated %). And it's much slower because the library function has to parse the entire string to be copied looking for percent signs, instead of just copying it to the output.
Code is using functions that expect pointers to string, yet not always passing pointers to strings as arguments.
Stray characters seen at output of snprintf
A string must have a terminating null character.
strncat(char *, .... expects the first parameter to be a pointer to a string. memcpy(output, "[",1); does not insure that. #Jeremy
memcpy(output, "[",1);
...
strncat(output, temp,sizeof(temp));
This is a candidate source of stray characters.
strncat(...., ..., size_t size). itself is a problem as the size is the amount of space available for concatenating (minus the null character). The size available to char * output is not passed in. #Jonathan Leffler. Might as well do strcat() here.
Instead, pass in the size available to output to prevent buffer overflow.
#define N 60
int get_gps60secString(GPS_periodic_t input[N], char *output, size_t sz) {
int cnt = snprintf(output, sz, "[");
if (cnt < 0 || cnt >= sz)
return 1;
output += cnt;
sz -= cnt;
int i = 0;
for (i = 0; i < N; i++) {
cnt = snprintf(output, size, "[%0.8f,%0.8f]%s", input[i].point.latitude,
input[i].point.longitude, i + 1 == N ? "" : ",");
if (cnt < 0 || cnt >= sz)
return 1;
output += cnt;
sz -= cnt;
}
cnt = snprintf(output, sz, "]");
if (cnt < 0 || cnt >= sz)
return 1;
return 0; // no error
}
OP has posted more code - will review.
Apparently the buffer char *output is pre-filled with 0 before the get_gps60secString() so the missing null character from memcpy(output, "[",1); should not cause the issue - hmmmmmm
unsigned short SendTrack() does not return a value. 1) Using its result value is UB. 2) Enable all compiler warnings.
I have run into a strange bug and I cannot for the life of me get it figured out. I have a function that decodes a byte array into a string based on another encoding function. The function that decodes looks roughly like this:
char *decode_string( uint8_t *encoded_string, uint32_t length,
uint8_t encoding_bits ) {
char *sequence_string;
uint32_t idx = 0;
uint32_t posn_in_buffer;
uint32_t posn_in_cell;
uint32_t encoded_nucleotide;
uint32_t bit_mask;
// Useful Constants
const uint8_t CELL_SIZE = 8;
const uint8_t NUCL_PER_CELL = CELL_SIZE / encoding_bits;
sequence_string = malloc( sizeof(char) * (length + 1) );
if ( !sequence_string ) {
ERR_PRINT("could not allocate enough space to decode the string\n");
return NULL;
}
// Iterate over the buffer, converting one nucleotide at a time.
while ( idx < length ) {
posn_in_buffer = idx / NUCL_PER_CELL;
posn_in_cell = idx % NUCL_PER_CELL;
encoded_nucleotide = encoded_string[posn_in_buffer];
encoded_nucleotide >>= (CELL_SIZE - encoding_bits*(posn_in_cell+1));
bit_mask = (1 << encoding_bits) - 1;
encoded_nucleotide &= bit_mask;
sequence_string[idx] = decode_nucleotide( encoded_nucleotide );
// decode_nucleotide returns a char on integer input.
idx++;
}
sequence_string[idx] = '\0';
printf("%s", sequence_string); // prints the correct string
return sequence_string;
}
The bug is that the return pointer, if I try to print it, causes a segmentation fault. But calling printf("%s\n", sequence_string) inside of the function will print everything just fine. If I call the function like this:
const char *seq = "AA";
uint8_t *encoded_seq;
encode_string( &encoded_seq, seq, 2, 2);
char *decoded_seq = decode_string( encoded_seq, 2, 2);
if ( decoded_seq ) {
printf("%s\n",decoded_seq); // this crashes
if ( !strcmp(decoded_seq, seq) ) {
printf("Success!");
}
then it will crash on the print.
A few notes, the other functions seem to all work, I've tested them fairly thoroughly (i.e. decode_nucleotide, encode_string). The string also prints correctly inside the function. It is only after the function returns that it stops working.
My question is, what might cause this memory to become invalid just by returning the pointer from a function? Thanks in advance!
First (and not that important, but) in the statement:
sequence_string = malloc( sizeof(char) * (length + 1) );
sizeof(char) by definition is always == 1. so the statement becomes:
sequence_string = malloc(length + 1);
In this section of your post:
char *decoded_seq = decode_string( encoded_seq, 2, 2);
...since I cannot see your implementation of decode_string, I can only make assumptions about how you are verifying its output before returning it. I do however understand that you are expecting the return value to contain values that would be legal contents for a C string. I can also assume that because you are working with coding and decoding, that the output type is likely unsigned char. If I am correct, then a legal range of characters for an output type of unsigned char is 0-255.
You are not checking the output before sending the value to the printf statement. If the value at the memory address of decoded_seq happens to be 0, (in the range of unsigned char) your program would crash. String functions do not work well with null pointers.
You should verify the return of _decode_string_ sending it to printf
char *decoded_seq = decode_string( encoded_seq, 2, 2);
if(decoded_seq != NULL)
{
...
Here, I explain my problem, I am a beginner on the ptrace function and I would like to succeed in recovering the hard information of a structure.
For example with this command, I will have strace -e trace = fstat ls
a line: fstat (3, {st_mode = ..., st_size = ...}
and I would like to successfully retrieve the contents of the structure (st_mode) and (st_size).
I try this but to no avail:
int buffer(unsigned long long addr, pid_t child, size_t size, void *buffer)
{
size_t byte = 0;
size_t data;
unsigned long tmp;
while (byte < size) {
tmp = ptrace(PTRACE_PEEKDATA, child, addr + byte);
if ((size - byte) / sizeof(tmp))
data = sizeof(tmp);
else
data = size % sizeof(tmp);
memcpy((void *)(buffer + byte), &tmp, data);
byte += data;
}
}
and in params :
struct stat stat_i;
buffer(addr, pid, sizeof(stat_i), &stat_i);
printf("%lu", stat_i.st_size); -> fake value :/
Thank'ks !
From the man page,
PTRACE_PEEKTEXT, PTRACE_PEEKDATA
Read a word at the address addr in the tracee's memory,
returning the word as the result of the ptrace() call. Linux
does not have separate text and data address spaces, so these
two requests are currently equivalent. (data is ignored; but
see NOTES.)
Thus you must understand that tmp would hold the actually value that was read.
Your checks are wrong - you should set errno = 0 before the call and then check if it has changed. If it has - you've got an error. If it hasn't - you can be assured that tmp has the word from the remote process.
Try something like this:
int buffer(unsigned long long addr, pid_t child, size_t size, void *buffer)
{
size_t byte = 0;
size_t data;
unsigned long tmp;
// support for word aligned sizes only
if (size % sizeof(long) != 0)
return -1;
long * buffer_int = (long*) buffer;
while (byte < size) {
errno = 0;
tmp = ptrace(PTRACE_PEEKDATA, child, addr + byte);
if (errno)
return -1;
buffer_int[byte / sizeof(long)] = tmp;
byte += sizeof(long);
}
}