This works on Linux, but not (as I would like) on FreeBSD:
I wish to exercise my CD-ROM drive, to keep the dust off the lens. On Linux I run (as root) a C program I wrote which seeks back and forth, reading a single block each time as it goes. On FreeBSD this program doesn't get too far. I can open the device and seek to (say) block 1. But when I try to read the block, I get error 22 (EINVAL). It fails on the first read, at block 1, whether or not the device is mounted (-t cd9660). How do I proceed?
Full program is here. The relevant snippet:
lo_fd=Open(ar_argv[1],
O_RDONLY,
0
);
lo_high_bit=1;
while(lo_high_bit>0)
{
if(lseek(lo_fd,
lo_high_bit,
SEEK_SET
)
==
(off_t)-1
)
{
lo_high_bit>>=1;
break;
}
if(read(lo_fd,
lo_buffer,
1
)
!=
1
)
{
lo_high_bit>>=1;
break;
}
lo_high_bit<<=1;
}
It turns out that I was making two errors: trying to read a byte at a time, and lseek()ing to byte 1. fstat() on the device shows st_blksize of 4096.
Seeking to 4096 and reading 4096 bytes works.
Seeking to 2048 and reading 2048 bytes works.
Seeking to 2048 and reading 1024 bytes gives EINVAL on the read().
Seeking to 1024 and reading 2048 bytes gives EINVAL on the read().
Related
Turns out I misinterpreted wear leveling, I initially thought by accessing the drive as RAW I would lose this feature but as its a feature on the controller this explains why i am hitting millions of writes to the 'logical sector' I am testing
I am writing an application where I will be utilizing a RAW disk partition like a circular buffer, ie no filesystem.
I need somewhere to keep track of my read/write buffer heads that is persistent across boots, I was thinking i can create another partition to store these 2 pointers.
But I am worried about how many times I can write to a sector in the devices SOLID state drive before the sector dies so I wrote the code below to hammer a single sector and see how quickly it fails.
create random block (512 bytes)
write to sector 0 using pwrite
read block from sector 0 using pread
compare each byte
exit when difference found.
but it has been running for millions of sector writes now!!
I would have expected it to fail somewhere standard like between 10,000-100,000 times?
I'm using pread/pwrite as below with a random buffer each loop and a comparison of both buffers afterwards.
void WriteBlock(const char* device, unsigned char* dataBlock, size_t sector, int size)
{
int fd = open(device, O_WRONLY);
if(fd <= 0)
{
std::cout << "error opening " << device << " '" << strerror(errno) << "'\r\n";
exit(EXIT_FAILURE);
}
ssize_t r = pwrite(fd, dataBlock, size, sector*SECTOR_SIZE);
if (r<=0 || r<size)
{
std::cout << "failure writing '" << strerror(errno) << "'\r\n";
exit(EXIT_FAILURE);
}
close(fd);
}
void ReadBlock(const char* device, unsigned char* dataBlock, size_t sector, int size)
{
int fd = open(device, O_RDONLY);
if(fd <= 0)
{
std::cout << "error opening " << device << "\r\n";
exit(EXIT_FAILURE);
}
ssize_t r = pread(fd, dataBlock, size, sector*SECTOR_SIZE);
if (r<=0 || r<size)
{
std::cout << "failure writing '" << strerror(errno) << "'\r\n";
exit(EXIT_FAILURE);
}
close(fd);
}
the code just keeps on running with the write buffer equal to the read buffer every time.
FYI i'm not comparing the write buffer to itself! if i hard code a value into the read buffer it catches this and interprets as a failure.
I would have expected it to fail somewhere standard like between 10,000-100,000 times?
Most solid state drives have wear levelling. What this means is that when you write to logical block 0 the device says "Hey, the old data in logical block 0 is getting overwritten, so I can just pretend a completely different physical block is now logical block 0". By continually writing to the same logical block, you can be writing to many completely different physical blocks.
To defeat wear leveling (and actually write to the same physical block) you have to convince the device that all other blocks are in use. This isn't possible because there's spare capacity. For example, for a 1 TiB device you can fill all 1 TiB of logical blocks with data (without doing any "trim", etc), but there might be an extra 512 GiB of spare space and your writes to the same logical block will be spread across that 512 GiB of spare space; and by the time you actually see an error it may mean that every block in that 512 GiB of spare space has failed (and not just one block).
If you know how much spare space there actually is, then you might be able to do calculations based on that - e.g. if there's 1 thousand spare blocks and you do 1 billion writes before seeing an error, then you might be able to say "1 billion writes / 1 thousand blocks = 1 million writes to each physical block".
Now, imagine you're a manufacturer and you've got a 1000 GiB drive. You decide to sell it as a consumer drive (with the assumption that the drive will be mostly empty and wear leveling will be able to work well) and that you can say it's a 900 GiB drive (with 100 GiB of spare blocks) that will fail after 10000 writes. Then you decide to also sell the exact same drive as an enterprise drive (with the assumption that the drive will probably be full and wear leveling won't be so effective) and that you can say it's a 900 GiB drive (with 100 GiB of spare blocks) that will fail after 2000 writes. Then you decide you can increase the amount of spare space, and also sell the "mostly the same" drive as an 500 GiB enterprise drive (with 500 GiB of spare blocks) that will fail after 20000 writes.
Next; imagine that you test 1000 identical drives (with the software you're writing) and measure "20000 writes before failure on average, under the specific conditions used for testing (e.g. with the rest of the logical blocks all free/empty)". This information is mostly useless - if someone uses the same drive under different conditions (e.g. with less logical blocks free/empty) then it'll fail sooner (or later) than you said it would.
For a (potentially better) alternative; you might be able to use information obtained from "S.M.A.R.T." (see https://en.wikipedia.org/wiki/S.M.A.R.T. ). Unfortunately some of the most useful pieces of information you can get is manufacturer specific (e.g. Used Reserved Block Count Total, Unused Reserved Block Count Total), and some devices (USB flash) don't support it at all.
I have a client server program in which i read the file from the server to the client in chunks, over TCP. However, not all chunks being sent are of the size I have given. Some of them are less than the requested size or even 1.
This is my server side code for reading from the file and writing to socket
rd=read(filed, buffer,128);
write(socketd,buffer,128);
This true for only non text files. On using a txt file, it works perfectly, but fails for pdfs, ppts, images. Is there any way to read these non text files in chunks?
Sample of size in bytes being read in each iteration for a pdf file
128
128
128
128
116
128
128
36
You should only write the amount that was actually read from the file. If the size of the file isn't a multiple of 128, then the last read can't return 128 bytes, it just returns what's left in the file. Use rd as the amount to write to the socket.
while ((rd = read(filed, buffer, 128)) > 0) {
write(socketd, buffer, rd);
}
Issue resolved. I was using strlen(buffer) which gave those numbers. The integer returned by read() was 128, as desired.
I get confused with one line of code:
temp_uart_count = read(VCOM, temp_uart_data, 4096);
I found more about read function at http://linux.die.net/man/3/read, but if everything is okay it returns 0, so how we can get num of bytes received from that?
temp_uart_count is used to count how much bytes we received from virtual COM port and stored it to temp_uart_data which is 4096 bytes wide.
Am I really getting how much bytes i received with this line of code?
... but if everything is okay it returns 0, so how we can get num of bytes received from that?
A return code of zero simply means that read() was unable to provide any data.
Am I really getting how much bytes i received with this line of code?
Yes, a positive return code (i.e. >= 0) from read() is an accurate count of bytes that were returned in the buffer. Zero is a valid count.
If you're expecting more data, then simply repeat the read() syscall. (However you may have setup the termios arguments poorly, e.g. VMIN=0 and VTIME=0).
And - zero indicates end of file
If you get 0, it means that the end of file (or an equivalent condition) has been reached and there is nothing else to read.
The above (one from a comment, and the other in an answer) are incorrect.
Reading from a tty device (e.g. a serial port) is not like reading from a file on a block device, but is temporal. Data for reading is only available as it is received over the comm link.
A non-blocking read() will return with -1 and errno set to EAGAIN when there is no data available.
A blocking non-canonical read() will return zero when there is no data available. Correlate your termios configuration with this to confirm that a return of zero is valid (and does not indicate "end of file").
In either case, the read() can be repeated to get more data when/if it arrives.
Also when using non-canonical (aka raw) mode (or non-blocking reads), do not expect or rely on the the read() to perform message or packet management for you. You will need to add a layer to your program to read bytes, concatenate those bytes into a complete message datagram/packet, and validate it before that message can be processed.
ssize_t read(int fd, void *buf, size_t count); returns you the size of bytes he read and stores it into the value you passed in parameters. And when errors happen it returns -1 (with errno set to EINTR) or to return the number of bytes already read..
From the linux man :
On files that support seeking, the read operation commences at the current file offset, and the file offset is incremented by the number of bytes read. If the current file offset is at or past the end of file, no bytes are read, and read() returns zero.
Yes, temp_uart_count will contain the actual number of bytes read, and obviously that number will be smaller or equal to the number of elements of temp_uart_data. If you get 0, it means that the end of file (or an equivalent condition) has been reached and there is nothing else to read.
If it returns -1 this indicate that an error has occurred and you'll need to check the errno variable to understand what happened.
I'm currently working on a project that will read data from a micro-controller via serial communications.
As of right now, the program (on my computer), opens a /dev/tty* file and is able to read/write to it. The micro-controller will send a packet of n bytes at any time. I want to know if there is any way I can tell when all of the bytes have been written to the file?
I've been looking at the select() and poll() functions, but they seem to be only able to tell when a byte is ready, but not when every byte has been written.
Any help is appreciated. Thanks!
If your n is hardcoded you can just do (with pseudocode):
received_data = offset_from_last_round
while( received_data < n )
{
use select() for waiting data to arrive
read() all data you can, dont forget to check buffer oveflow here
received_data += how much data was red
}
full_message = buffer[ 0 ... N - 1 ]
offset_to_next_round = buffer[ N .. received_data ]
If n is not hardcoded you need to do something like what #Golgauth suggested, or add some "end of transmission" sequence/byte to your message (which is tricky if you have binary file/data to transmit). In short: You need some sort of protocol.
Well, the idea is that your binary file should start by some bytes which actually give the size that has to be read next.
Read N bytes => gives the DATASIZE, i.e. how many bytes remaining: (FILESIZE - N)
Read (DATASIZE) bytes => gives the data themselves (readable by blocks/packets of size n)
This is the kind of discussion we were having here actually (this about how to interpret a raw PCM wav sound file, but this is the same point: getting the number N of samples, to determine how many blocks are to be read next to get the file with concern to the integrity).
As the title says, is there a limit to the number of bytes that can be written at once on a connection-oriented socket?
If I want to send a buffer of, for example, 1024 bytes, can I use a
write(tcp_socket, buffer, 1024);
or should I use multiple write() calls with a lower amount of bytes for each one?
write() does not guarantee that all bytes will be written so multiple calls to write() are required. From man write:
The number of bytes written may be less than count if, for example, there is insufficient space on the underlying physical medium, or the RLIMIT_FSIZE resource limit is encountered (see setrlimit(2)), or the call was interrupted by a signal handler after having written less than count bytes. (See also pipe(7).)
write() returns the number of bytes written so a running total of the bytes written must be maintained and used as an index into buffer and to calculate the number of remaining bytes to be written:
ssize_t total_bytes_written = 0;
while (total_bytes_written != 1024)
{
assert(total_bytes_written < 1024);
ssize_t bytes_written = write(tcp_socket,
&buffer[total_bytes_written],
1024 - total_bytes_written);
if (bytes_written == -1)
{
/* Report failure and exit. */
break;
}
total_bytes_written += bytes_written;
}
From my experience, it is better to stay in the 1024 byte limit
There's no inherent limit. TCP/IP will fragment and reassemble packets as required. Your system may impose (a possibly tunable) upper limit, but it is likely to be in the multi-MB range. See your manpage for setsockopt() and always check write()'s return value.
The actual amount you can write will depend on the type of socket. In general, you need to check the return value to see how many bytes were actually written. The number of bytes written can vary depend on whether the socket is in blocking mode or not.
Also, if the socket is blocking, you may not want to wait for all the data to be written in one go. You may want to write some at a time in order to be able to something else in between write operations.
as you can see write socket the maximum buffer size is 1048576 bytes.