I am reading some HTTP POST data from an HTTP wrapped TCP socket. My apparatus works but there is a strange syndrome. Basically I know what the content length is (via HTTP header Content-length) but I more often than not seem to build up a buffer that is 2-3 bytes longer than expected. I know that I am not setting my buffer size on initialization but when I do I get a lot of compile errors. The following code almost works but often produces more data in the buffer than there should be.
long bytesRead;
unsigned long bytesRemaining;
sbyte *pBuffer;
sbyte *pTmpBuffer;
pBuffer = malloc(contentLength);
memset(pBuffer, 0, contentLength);
pTmpBuffer = pBuffer;
bytesRemaining = contentLength;
while(bytesRemaining > 0){
if(maxBuffSize < bytesRemaining){
chunkSize = maxBuffSize;
}
else {
chunkSize = bytesRemaining;
}
bytesRead = tcpBlockReader(pHttpData, pTempBuff, chunkSize);
bytesRemaining -= bytesRead;
pTempBuff += bytesRead;
}
printf("Data is %s\n", pBuffer);
printf("Length is %d\n", strlen(pBuffer));
Now sometimes it will be perfect, ie
Data is expected+data
Length is 13
And sometimes it will be
Data is expected+data+(weird characters)
Length is 15
So the problem here I think is I don't set a size for the buffer (ie pBuffer[contentLength]). When I do this though I get errors of incompatible types and what not. I am not a well versed C programmer (usually stick to chars and ints). What can I do to ensure that the buffer is not full of extra garbage at the end?
I was missing the elusive NULL terminator.
pBuffer = malloc(contentLength + 1)
...
pBuffer[contentLength] = '\0';
So, I have the following code which sends out my packet on TCP. Its working pretty well. I just have to test partial writes. So I write 1 byte at time either by setting sendbuf to 1 or do a hack as shown below. When i took a tcpdump, it was all incorrect except the first byte.. what am i doing wrong?
int tmi_transmit_packet(struct tmi_msg_pdu *tmi_pkt, int len, int *written_len)
{
int bytes;
// This works
bytes = write(g_tmi_mgr->tmi_conn_fd, (void*) tmi_pkt, len);
// This doesn't:
// bytes = write(g_tmi_mgr->tmi_conn_fd, (void*) tmi_pkt, 1);
if (bytes < 0) {
if (errno == EAGAIN) {
return (TMI_SOCK_FULL);
}
return (TMI_WRITE_FAILED);
} else if (bytes < len) {
*written_len += bytes;
tmi_pkt += bytes;
return (tmi_transmit_packet(tmi_pkt, len - bytes, written_len));
} else {
*written_len += len;
}
return TMI_SUCCESS;
}
This line
tmi_pkt += bytes;
most propably does not do what you expect.
It does increment tmi_pkt by sizeof(*tmp_pkt) * bytes and not only by bytes. For a nice explanation on pointer arithmetics you might like to click here and have a look at binky.
To get around this you might mod you code as follows:
...
else if (bytes < len) {
void * pv = ((char *) tmp_pkt) + bytes;
*written_len += bytes;
return (tmi_transmit_packet(pv, len - bytes, written_len));
}
...
Anyhow this somehow smells dirty as the data pointed to by the pointer passed into the write function does not necessarly need to correspond to it's type.
So a cleaner solution would be to not used struct tmi_msg_pdu *tmi_pkt but void * or char * as the function parameter declaration.
Although quiet extravagant the use of recursive calls here is not necessary nor recommended. For much data and/or a slow transmission it may run out of stack memory. A simple loop would do also. The latter has the advantage that you could use a temporary pointer to the buffer to be written and could stick to a typed interface.
[Update] I am offering a bonus for this. Frankly, I don't care which encryption method is used. Preferably something simple like XTEA, RC4, BlowFish ... but you chose.
I want minimum effort on my part, preferably just drop the files into my projects and build.
Idealy you should already have used the code to en/de-crypt a file in Delphi and C (I want to trade files between an Atmel UC3 micro-processor (coding in C) and a Windows PC (coding in Delphi) en-and-de-crypt in both directions).
I have a strong preference for a single .PAS unit and a single .C/.H file. I do not want to use a DLL or a library supporting dozens of encryption algorithms, just one (and I certainly don't want anything with an install program).
I hope that I don't sound too picky here, but I have been googling & trying code for over a week and still can't find two implementations which match. I suspect that only someone who has already done this can help me ...
Thanks in advance.
As a follow up to my previous post, I am still looking for some very simple code with why I can - with minimal effort - en-de crypt a file and exchange it between Delphi on a PC and C on an Atmel UC3 u-processor.
It sounds simple in theory, but in practice it's a nightmare. There are many possible candidates and I have spend days googling and trying them out - to no avail.
Some are humonous libraries, supporting many encryption algorithms, and I want something lightweight (especially on the C / u-processor end).
Some look good, but one set of source offers only block manipulation, the other strings (I would prefer whole file en/de-crypt).
Most seem to be very poorly documented, with meaningless parameter names and no example code to call the functions.
Over the past weekend (plus a few more days), I have burned my way through a slew of XTEA, XXTEA and BlowFish implementations, but while I can encrypt, I can't reverse the process.
Now I am looking at AES-256. Dos anyone know of an implementation in C which is a single AES.C file? (plus AES.H, of course)
Frankly, I will take anything that will do whole file en/de-crypt between Delphi and C, but unless anyone has actually done this themselves, I expect to hear only "any implementation that meets the standard should do" - which is a nice theoory but just not working out for me :-(
Any simple AES-256 in C out there? I have some reasonable looking Delphi code, but won't be sure until I try them together.
Thanks in advance ...
I would suggest using the .NET Micro Framework on a secondary microcontroller (e.g. Atmel SAM7X) as a crypto coprocessor. You can test this out on a Netduino, which you can pick up for around $35 / £30. The framework includes an AES implementation within it, under the System.Security.Cryptography namespace, alongside a variety of other cryptographic functions that might be useful for you. The benefit here is that you get a fully tested and working implementation, and increased security via type-safe code.
You could use SPI or I2C to communicate between the two microcontrollers, or bit-bang your own data transfer protocol over several I/O lines in parallel if higher throughput is needed.
I did exactly this with an Arduino and a Netduino (using the Netduino to hash blocks of data for a hardware BitTorrent device) and implemented a rudimentary asynchronous system using various commands sent between the devices via SPI and an interrupt mechanism.
Arduino is SPI master, Netduino is SPI slave.
A GPIO pin on the Netduino is set as an output, and tied to another interrupt-enabled GPIO pin on the Arduino that is set as an input. This is the interrupt pin.
Arduino sends 0xF1 as a "hello" initialization message.
Netduino sends back 0xF2 as an acknolwedgement.
When Arduino wants to hash a block, it sends 0x48 (ASCII 'H') followed by the data. When it is done sending data, it sets CS low. It must send whole bytes; setting CS low when the number of received bits is not divisible by 8 causes an error.
The Netduino receives the data, and sends back 0x68 (ASCII 'h') followed by the number of received bytes as a 2-byte unsigned integer. If an error occurred, it sends back 0x21 (ASCII '!') instead.
If it succeeded, the Netduino computes the hash, then sets the interrupt pin high. During the computation time, the Arduino is free to continue its job whilst waiting.
The Arduino sends 0x52 (ASCII 'R') to request the result.
The Netduino sets the interrupt pin low, then sends 0x72 (ASCII 'r') and the raw hash data back.
Since the Arduino can service interrupts via GPIO pins, it allowed me to make the processing entirely asynchronous. A variable on the Arduino side tracks whether we're currently waiting on the coprocessor to complete its task, so we don't try to send it a new block whilst it's still working on the old one.
You could easily adapt this scheme for computing AES blocks.
Small C library for AES-256 by Ilya Levin. Short implementation, asm-less, simple usage. Not sure how would it work on your current micro CPU, though.
[Edit]
You've mentioned having some delphi implementation, but in case something not working together, try this or this.
Also I've found arduino (avr-based) module using the Ilya's library - so it should also work on your micro CPU.
Can you compile C code from Delphi (you can compile Delphi code from C++ Builder, not sure about VV). Or maybe use the Free Borland Command line C++ compiler or even another C compiler.
The idea is to use the same C code in your Windows app as you use on your microprocessor.. That way you can be reasonably sure that the code will work in both directions.
[Update] See
http://www.drbob42.com/examines/examin92.htm
http://www.hflib.gov.cn/e_book/e_book_file/bcb/ch06.htm (Using C++ Code in Delphi)
http://edn.embarcadero.com/article/10156#H11
It looks like you need to use a DLL, but you can statically link it if you don't want to distribute it
Here is RC4 code. It is very lightweight.
The C has been used in a production system for five years.
I have added lightly tested Delphi code. The Pascal is a line-by-line port with with unsigned char going to Byte. I have only run the Pascal in Free Pascal with Delphi option turned on, not Delphi itself. Both C and Pascal have simple file processors.
Scrambling the ciphertext gives the original cleartext back.
No bugs reported to date. Hope this solves your problem.
rc4.h
#ifndef RC4_H
#define RC4_H
/*
* rc4.h -- Declarations for a simple rc4 encryption/decryption implementation.
* The code was inspired by libtomcrypt. See www.libtomcrypt.org.
*/
typedef struct TRC4State_s {
int x, y;
unsigned char buf[256];
} TRC4State;
/* rc4.c */
void init_rc4(TRC4State *state);
void setup_rc4(TRC4State *state, char *key, int keylen);
unsigned endecrypt_rc4(unsigned char *buf, unsigned len, TRC4State *state);
#endif
rc4.c
void init_rc4(TRC4State *state)
{
int x;
state->x = state->y = 0;
for (x = 0; x < 256; x++)
state->buf[x] = x;
}
void setup_rc4(TRC4State *state, char *key, int keylen)
{
unsigned tmp;
int x, y;
// use only first 256 characters of key
if (keylen > 256)
keylen = 256;
for (x = y = 0; x < 256; x++) {
y = (y + state->buf[x] + key[x % keylen]) & 255;
tmp = state->buf[x];
state->buf[x] = state->buf[y];
state->buf[y] = tmp;
}
state->x = 255;
state->y = y;
}
unsigned endecrypt_rc4(unsigned char *buf, unsigned len, TRC4State *state)
{
int x, y;
unsigned char *s, tmp;
unsigned n;
x = state->x;
y = state->y;
s = state->buf;
n = len;
while (n--) {
x = (x + 1) & 255;
y = (y + s[x]) & 255;
tmp = s[x]; s[x] = s[y]; s[y] = tmp;
tmp = (s[x] + s[y]) & 255;
*buf++ ^= s[tmp];
}
state->x = x;
state->y = y;
return len;
}
int endecrypt_file(FILE *f_in, FILE *f_out, char *key)
{
TRC4State state[1];
unsigned char buf[4096];
size_t n_read, n_written;
init_rc4(state);
setup_rc4(state, key, strlen(key));
do {
n_read = fread(buf, 1, sizeof buf, f_in);
endecrypt_rc4(buf, n_read, state);
n_written = fwrite(buf, 1, n_read, f_out);
} while (n_read == sizeof buf && n_written == n_read);
return (n_written == n_read) ? 0 : 1;
}
int endecrypt_file_at(char *f_in_name, char *f_out_name, char *key)
{
int rtn;
FILE *f_in = fopen(f_in_name, "rb");
if (!f_in) {
return 1;
}
FILE *f_out = fopen(f_out_name, "wb");
if (!f_out) {
close(f_in);
return 2;
}
rtn = endecrypt_file(f_in, f_out, key);
fclose(f_in);
fclose(f_out);
return rtn;
}
#ifdef TEST
// Simple test.
int main(void)
{
char *key = "This is the key!";
endecrypt_file_at("rc4.pas", "rc4-scrambled.c", key);
endecrypt_file_at("rc4-scrambled.c", "rc4-unscrambled.c", key);
return 0;
}
#endif
Here is lightly tested Pascal. I can scramble the source code in C and descramble it with the Pascal implementation just fine.
type
RC4State = record
x, y : Integer;
buf : array[0..255] of Byte;
end;
KeyString = String[255];
procedure initRC4(var state : RC4State);
var
x : Integer;
begin
state.x := 0;
state.y := 0;
for x := 0 to 255 do
state.buf[x] := Byte(x);
end;
procedure setupRC4(var state : RC4State; var key : KeyString);
var
tmp : Byte;
x, y : Integer;
begin
y := 0;
for x := 0 to 255 do begin
y := (y + state.buf[x] + Integer(key[1 + x mod Length(key)])) and 255;
tmp := state.buf[x];
state.buf[x] := state.buf[y];
state.buf[y] := tmp;
end;
state.x := 255;
state.y := y;
end;
procedure endecryptRC4(var buf : array of Byte; len : Integer; var state : RC4State);
var
x, y, i : Integer;
tmp : Byte;
begin
x := state.x;
y := state.y;
for i := 0 to len - 1 do begin
x := (x + 1) and 255;
y := (y + state.buf[x]) and 255;
tmp := state.buf[x];
state.buf[x] := state.buf[y];
state.buf[y] := tmp;
tmp := (state.buf[x] + state.buf[y]) and 255;
buf[i] := buf[i] xor state.buf[tmp]
end;
state.x := x;
state.y := y;
end;
procedure endecryptFile(var fIn, fOut : File; key : KeyString);
var
nRead, nWritten : Longword;
buf : array[0..4095] of Byte;
state : RC4State;
begin
initRC4(state);
setupRC4(state, key);
repeat
BlockRead(fIN, buf, sizeof(buf), nRead);
endecryptRC4(buf, nRead, state);
BlockWrite(fOut, buf, nRead, nWritten);
until (nRead <> sizeof(buf)) or (nRead <> nWritten);
end;
procedure endecryptFileAt(fInName, fOutName, key : String);
var
fIn, fOut : File;
begin
Assign(fIn, fInName);
Assign(fOut, fOutName);
Reset(fIn, 1);
Rewrite(fOut, 1);
endecryptFile(fIn, fOut, key);
Close(fIn);
Close(fOut);
end;
{$IFDEF TEST}
// Very small test.
const
key = 'This is the key!';
begin
endecryptFileAt('rc4.pas', 'rc4-scrambled.pas', key);
endecryptFileAt('rc4-scrambled.pas', 'rc4-unscrambled.pas', key);
end.
{$ENDIF}
It looks easier would be to get reference AES implementation (which works with blocks), and add some code to handle CBC (or CTR encryption).
This would need from you only adding ~30-50 lines of code, something like the following (for CBC):
aes_expand_key();
first_block = iv;
for (i = 0; i < filesize / 16; i++)
{
data_block = read(file, 16);
data_block = (data_block ^ iv);
iv = encrypt_block(data_block);
write(outputfile, iv);
}
// if filesize % 16 != 0, then you also need to add some padding and encrypt the last block
Assuming the encryption strength isn't an issue, as in satisfying an organization's Chinese Wall requirement, the very simple "Sawtooth" encryption scheme of adding (i++ % modulo 256) to fgetc(), for each byte, starting at the beginning of the file, might work just fine.
Declaring i as a UCHAR will eliminate the modulo requirement, as the single byte integer cannot help but cycle through its 0-255 range.
The code is so simple it's not worth posting. A little imagination, and you'll have some embellishments that can add a lot to the strength of this cypher. The primary vulnerability of this cypher is large blocks of identical characters. Rectifying this is a good place to start improving its strength.
This cypher works on every possible file type, and is especially effective if you've already 7Zipped the file.
Performance is phenomenal. You won't even know the code is there. Totally I/O bound.
Scatter-gather - readv()/writev()/preadv()/pwritev() - reads/writes a variable number of iovec structs in a single system call. Basically it reads/write each buffer sequentially from the 0th iovec to the Nth. However according to the documentation it can also return less on the readv/writev calls than was requested. I was wondering if there is a standard/best practice/elegant way to handle that situation.
If we are just handling a bunch of character buffers or similar this isn't a big deal. But one of the niceties is using scatter-gather for structs and/or discrete variables as the individual iovec items. How do you handle the situation where the readv/writev only reads/writes a portion of a struct or half of a long or something like that.
Below is some contrived code of what I am getting at:
int fd;
struct iovec iov[3];
long aLong = 74775767;
int aInt = 949;
char aBuff[100]; //filled from where ever
ssize_t bytesWritten = 0;
ssize_t bytesToWrite = 0;
iov[0].iov_base = &aLong;
iov[0].iov_len = sizeof(aLong);
bytesToWrite += iov[0].iov_len;
iov[1].iov_base = &aInt;
iov[1].iov_len = sizeof(aInt);
bytesToWrite += iov[1].iov_len;
iov[2].iov_base = &aBuff;
iov[2].iov_len = sizeof(aBuff);
bytesToWrite += iov[2].iov_len;
bytesWritten = writev(fd, iov, 3);
if (bytesWritten == -1)
{
//handle error
}
if (bytesWritten < bytesToWrite)
//how to gracefully continue?.........
Use a loop like the following to advance the partially-processed iov:
for (;;) {
written = writev(fd, iov+cur, count-cur);
if (written < 0) goto error;
while (cur < count && written >= iov[cur].iov_len)
written -= iov[cur++].iov_len;
if (cur == count) break;
iov[cur].iov_base = (char *)iov[cur].iov_base + written;
iov[cur].iov_len -= written;
}
Note that if you don't check for cur < count you will read past the end of iov which might contain zero.
AFAICS the vectored read/write functions work the same wrt short reads/writes as the normal ones. That is, you get back the number of bytes read/written, but this might well point into the middle of a struct, just like with read()/write(). There is no guarantee that the possible "interruption points" (for lack of a better term) coincide with the vector boundaries. So unfortunately the vectored IO functions offer no more help for dealing with short reads/writes than the normal IO functions. In fact, it's more complicated since you need to map the byte count into an IO vector element and offset within the element.
Also note that the idea of using vectored IO for individual structs or data items might not work that well; the max allowed value for the iovcnt argument (IOV_MAX) is usually quite small, something like 1024 or so. So if you data is contiguous in memory, just pass it as a single element rather than artificially splitting it up.
Vectored write will write all the data you have provided with one call to "writev" function. So byteswritten will be always be equal to total number of bytes provided as input. this is what my understanding is.
Please correct me if I am wrong
I am trying to send data between a client/Server, the data looks like
typedef Struct Message
{ int id;
int message_length;
char* message_str;
}message;
I am trying to Write and Read this message between a client and server constantly updating the elements in this struct. I have heard Writev may do the trick. i want to send a
message to the server and then the server pulls out the elements and uses those elements as conditionals to execute the proper method?
Assuming you want to do the serialization yourself and not use Google Protocol Buffers or some library to handle it for you, I'd suggest writing a pair of functions like this:
// Serializes (msg) into a flat array of bytes, and returns the number of bytes written
// Note that (outBuf) must be big enough to hold any Message you might have, or there will
// be a buffer overrun! Modifying this function to check for that problem and
// error out instead is left as an exercise for the reader.
int SerializeMessage(const struct Message & msg, char * outBuf)
{
char * outPtr = outBuf;
int32_t sendID = htonl(msg.id); // htonl will make sure it gets sent in big-endian form
memcpy(outPtr, &sendID, sizeof(sendID));
outPtr += sizeof(sendID);
int32_t sendLen = htonl(msg.message_length);
memcpy(outPtr, &sendLen, sizeof(sendLen));
outPtr += sizeof(sendLen);
memcpy(outPtr, msg.message_str, msg.message_length); // I'm assuming message_length=strlen(message_str)+1 here
outPtr += msg.message_length;
return (outPtr-outBuf);
}
// Deserializes a flat array of bytes back into a Message object. Returns 0 on success, or -1 on failure.
int DeserializeMessage(const char * inBuf, int numBytes, struct Message & msg)
{
const char * inPtr = inBuf;
if (numBytes < sizeof(int32_t)) return -1; // buffer was too short!
int32_t recvID = ntohl(*((int32_t *)inPtr));
inPtr += sizeof(int32_t);
numBytes -= sizeof(int32_t);
msg.id = recvID;
if (numBytes < sizeof(int32_t)) return -1; // buffer was too short!
int32_t recvLen = ntohl(*((int32_t *)inPtr));
inPtr += sizeof(int32_t);
numBytes -= sizeof(int32_t);
msg.message_length = recvLen; if (msg.message_length > 1024) return -1; /* Sanity check, just in case something got munged we don't want to allocate a giant array */
msg.message_str = new char[msg.message_length];
memcpy(msg.message_str, inPtr, numBytes);
return 0;
}
With these functions, you are now able to convert a Message into a simple char-array and back at will. So now all you have to do is send the char-array over the TCP connection, receive it at the far end, and then Deserialize the array back into a Message struct there.
One wrinkle with this is that your char arrays will be variable-length (due to the presence of a string which can be different lengths), so your receiver will need some easy way to know how many bytes to receive before calling DeserializeMessage() on the array.
An easy way to handle that is to always send a 4-byte integer first, before sending the char-array. The 4-byte integer should always be the size of the upcoming array, in bytes. (Be sure to convert the integer to big-endian first, via htonl(), before sending it, and convert it back to native-endian on the receiver via htonl() before using it).
Okay, I'll take a stab at this. I'm going to assume that you have a "message" object on the sending side and what you want to do is somehow send it across to another machine and reconstruct the data there so you can do some computation on it. The part that you may not be clear on is how to encode the data for communications and then decode it on the receiving side to recover the information. The simplistic approach of just writing the bytes contained in a "message" object (i.e. write(fd, msg, sizeof(*msg), where "msg" is a pointer to an object of type "message") won't work because you will end up sending the value of a virtual address in the memory of one machine to different machine and there's not much you can do with that on the receiving end. So the problem is to design a way to pass an two integers and a character string bundled up in a way that you can fish them back out on the other end. There are, of course, many ways to do this. Does this describe what you are trying to do?
You can send structs over socket, but you have to serialize them before sending the struct using boost serialization.
Here is a sample code :
#include<iostream>
#include<unistd.h>
#include<cstring>
#include <sstream>
#include <boost/archive/text_oarchive.hpp>
#include <boost/archive/text_iarchive.hpp>
using namespace std;
typedef struct {
public:
int id;
int message_length;
string message_str;
private:
friend class boost::serialization::access;
template <typename Archive>
void serialize(Archive &ar, const unsigned int vern)
{
ar & id;
ar & message_length;
ar & message_str;
}
} Message;
int main()
{
Message newMsg;
newMsg.id = 7;
newMsg.message_length = 14;
newMsg.message_str="Hi ya Whats up";
std::stringstream strData;
boost::archive::text_oarchive oa(strData);
oa << newMsg;
char *serObj = (char*) strData.str().c_str();
cout << "Serialized Data ::: " << serObj << "Len ::: " << strlen(serObj) << "\n";
/* Send serObj thru Sockets */
/* recv serObj from socket & deserialize it */
std::stringstream rcvdObj(serObj);
Message deserObj;
boost::archive::text_iarchive ia(rcvdObj);
ia >> deserObj;
cout<<"id ::: "<<deserObj.id<<"\n";
cout<<"len ::: "<<deserObj.message_length<<"\n";
cout<<"str ::: "<<deserObj.message_str<<"\n";
}
you can compile the program by
g++ -o serial boost.cpp /usr/local/lib/libboost_serialization.a
you must have libboost_serialization.a statically compiled in your machine.
Keeping the sockets 'blocking' will be good and you have to devise for reading these structs from recv buffer.