First off I need to say it's completely possible I'm missing something.
My assignment is to essentially implement 'fprintf'. Now while appending to the file isn't required, I like to go above and beyond.
My issue is, I can't find a definition for lseek in xv6, meaning I have to implement it on my own, but I genuinely don't know how to go about it.
Tried reading 512 bytes at a time on an infinite loop in attempt to move the cursor over to the end, as a way to hardcode it, but if the file isn't opened with O_RDWR or I try this with stdout it fails.
I've also tried writing an empty string on an infinite loop. Knew it wouldn't work, but tried anyways.
I can read xv6 fairly well (The user level programs), but I can't understand the source code of lseek for the life of me
It doesn't have to be a genuine lseek. I just need to be able to get to the end of an fd and continue writing, but this cannot be reliant on filemode.
Any help is greatly appreciated.
I found the solution.
The reason O_APPEND doesn't work is because the definition of open(), in sysfile.c, doesn't do anything with append.
In sys_open, they hardcode a value of 0 for f->off (offset), and this is what I need to change.
My planned solution is to figure out the filesize (in bytes) of the file, and set the offset to that number.
Probably gonna use stat().
Related
Is there any way to determine if a file can be deleted before trying DeleteFile on it. I just want to test if a DeleteFile will succeed without really deleting the file. What I want to do is to create a hardlink to a file if it is not in use and to copy it if it is in use without FILE_SHARE_DELETE(there is in the system a handle opened on it woithout FILE_SHARE_DELETE flag).
Thank you!
Disclaimer: I don't really know anything about Windows programming. But I read documentation.
According to the MSDN documentation for CreateFile, you can specify DELETE in the dwDesiredAccess parameter, which will cause the CreateFile call to fail if the access right is not available. (You can also specify OPEN_EXISTING for dwCreationDisposition if you don't actually want to create a new file.) That should cover your use case.
However, it's worth pointing out that there is an race condition; you might do this test and then just afterwards, before you have a chance to act on the result, someone else might open the file without the FILE_SHARE_DELETE. So for the test to be meaningful, you'd need to know more about the lifecycle of files than is mentioned in your question (which you indeed might know; it isn't really necessary to mention it, so if you do, just ignore this paragraph.)
Thank you everyone for your answers. In the end after 3 days of searching(after 2 days I posted my question here) I found I needed. At the beginning (3 days ago) I thought DELETE flag is enough for the test, but it does not work - I still am able to open a handle even there is another handle open in the system without FILE_SHARE_DELETE flag. Anyway, starting from here I manage to find a solution in the end - making a CreateFile with FILE_FLAG_DELETE_ON_CLOSE - if I receive a valid handle than the file is deletable. Before closing the handle using NtSetInformationFile with FILE_DISPOSITION_INFORMATION I remove FILE_FLAG_DELETE_ON_CLOSE so this way I avoid deleting the file when closing the handle. And indeed I do have control over the lifecycle of that file - meaning that after I make the test no one should open it without SHARE_DELETE - at least not the processes I control which should be enough for what I have to do.
I am currently working on a simple server implemented in C.
Processing jpg files works fine, btu png's give me a segmentation fault. I never get past this chunk of code. Why this might be?
fseek (file , 0 , SEEK_END);
lSize = ftell (file);
rewind (file);
Thanks.
It's far more likely that you were accessing those arrays in a problematic fashion. Check the logic in your buffering code. Make sure you have your buffer sizes #define'd in a central location, rather than hardcoding sizes and offsets. You made it quit crashing, but if you missed an underlying logic error, you may run into mysterious problems down the road when you change something else. It is probably worth your time to deliberately break the program again and figure out WHY it's broken. As others have suggested, a debugger would be an excellent idea at this point. Or post a more complete example of your code.
There are many similar questions, but nothing that answers this specifically after googling around quite a bit. Here goes:
Say we have a file (could be binary, and much bigger too):
abcdefghijklmnopqrztuvwxyz
what is the best way in C to "move" a right most portion of this file to the left, truncating the beginning of the file.. so, for example, "front truncating" 7 bytes would change the file on disk to be:
hijklmnopqrztuvwxyz
I must avoid temporary files, and would prefer not to use a large buffer to read the whole file into memory. One possible method I thought of is to use fopen with "rb+" flag, and constantly fseek back and forth reading and writing to copy bytes starting from offset to the beginning, then setEndOfFile to truncate at the end. That seems to be a lot of seeking (possibly inefficient).
Another way would be to fopen the same file twice, and use fgetc and fputc with the respective file pointers. Is this even possible?
If there are other ways, I'd love to read all of them.
You could mmap the file into memory and then memmove the contents. You would have to truncate the file separately.
You don't have to use an enormous buffer size, and the kernel is going to be doing the hard work for you, but yes, reading a buffer full from up the file and writing nearer the beginning is the way to do it if you can't afford to do the simpler job of create a new file, copy what you want into that file, and then copy the new (temporary) file over the old one. I wouldn't rule out the possibility that the approach of copying what you want to a new file and then either moving the new file in place of the old or copying the new over the old will be faster than the shuffling process you describe. If the number of bytes to be removed was a disk block size, rather than 7 bytes, the situation might be different, but probably not. The only disadvantage is that the copying approach requires more intermediate disk space.
Your outline approach will require the use of truncate() or ftruncate() to shorten the file to the proper length, assuming you are on a POSIX system. If you don't have truncate(), then you will need to do the copying.
Note that opening the file twice will work OK if you are careful not to clobber the file when opening for writing - using "r+b" mode with fopen(), or avoiding O_TRUNC with open().
If you are using Linux, since Kernel 3.15 you can use
#include <fcntl.h>
int fallocate(int fd, int mode, off_t offset, off_t len);
with the FALLOC_FL_COLLAPSE_RANGE flag.
http://manpages.ubuntu.com/manpages/disco/en/man2/fallocate.2.html
Note that not all file systems support it but most modern ones such as ext4 and xfs do.
Is there a Windows equivalent to Linux's readahead syscall?
EDIT:
I would like a full function signature if possible, showing the equivalent offset/count parameters (or lower/upper).
Eg:
The Linux function signature is:
ssize_t readahead(int fd, off64_t *offset, size_t count);
and an example of it's use is
readahead(file, 100, 500);
Where "file" is a file descriptor previously set by a function like mmap. This call is reading 500 bytes at index 100.
EDIT 2:
Please read this if you are unsure what readahead does: http://linux.die.net/man/2/readahead
Yes. It is FileSystemControl FSCTL_FILE_PREFETCH.
It is used in Windows Vista and above for prefetching both when an application starts and at boot time.
It is also used by the SuperFetch technology that uses heuristics to load applications at approximately the times of day you generally use them.
FSCTL_FILE_PREFETCH itself is not documented on MSDN, but it is easy to figure out the parameter format by examining the DeviceIoControl calls made on app startup: Just start an application in the debugger that already has a .pf file in the c:\Windows\Prefetch directory and break on DeviceIoControl (or if you're using a kernel debugger, break when the NTFS driver receives its first FSCTL_FILE_PREFETCH). Examine the buffer passed in and compare it with the .pf file and the range actually used later. I did this once out of curiosity but didn't record the details.
In case you are unfamiliar with DeviceIoControl and IRP_MJ_FILESYSTEM_CONTROL, here are some links to look at:
FileSystemControl at the IRP level IRP_MJ_FILESYSTEM_CONTROL
DeviceIoControl, which is used to invoke FileSystemControl IRPs
Structure of IO Control Codes
As of Windows 8, there exists a more or less direct equivalent to madvise(MADV_WILLNEED), which is effectively the same thing (Windows has an unified VM/cache system).
Assuming that you have memory-mapped the file, you can thus use PrefetchVirtualMemory to prefetch it.
This is still slightly more complicated than you'd wish, but not nearly as harsh as DeviceIoControl. Also note that you can easily prefetch several independent, discontinuous ranges.
I am not sure if I understand correctly, in what you said 'Where "file" is a file descriptor previously set by a function like mmap. This call is reading 500 bytes at index 100.' That sounds suspiciously like seeking to the offset and read 500 bytes... but you want it to be pre-fetched ahead...
In C Code, it would look something like this:
fseek(fp, 100, SEEK_CUR);
fread(&data, 500, 1, fp);
But prefetching it, I guess, you would want to hook up some kind of events using waithandles, and when the event gets raised, the data gets stored somewhere in a buffer...
To be honest, I have not come across a such thing that does pre-fetching data...but Ray's answer surprised me, but then again it is only for Vista upwards, if you want to maintain compatibility...that's something to keep in mind... but the links below may be of help...
Ok, there was a blog discussing this, a library written in Delphi, the source code is here, browsing the code here, ok, it may not be what you want but it may help you point in the direction... Sorry if its not what you are looking for...
Hope this helps,
Best regards,
Tom.
I'm writing a straightforward C program on Linux and wish to use an existing library's API which expects data from a file. I must feed it a file name as a const char*. But i have data, just like content of a file, already sitting in a buffer allocated on the heap. There is plenty of RAM and we want high performance. Wanting to avoid writing a temporary file to disk, what is a good way to feed the data to this API in a way that looks like a file?
Here's a cheap pretend version of my code:
marvelouslibrary.h:
int marvelousfunction(const char *filename);
normal-persons-usage.cpp, for which library was originally designed:
#include "marvelouslibrary.h"
int somefunction(char *somefilename)
{
return marvelousfunction(somefilename);
}
myprogram.cpp:
#include "marvelouslibrary.h"
int one_of_my_routines()
{
byte* stuff = new byte[1000000];
// fill stuff[] with...stuff!
// stuff[] holds same bytes as might be found in a file
/* magic goes here: make filename referring to stuff[] */
return marvelousfunction( ??? );
}
To be clear, the marvelouslibrary does not offer any API functions that accept data by pointer; it can only read a file.
I thought of pipes and mkfifo(), but seems meant for communicating between processes. I am no expert at these things. Does a named pipe work okay read and written in the same process? Is this a wise approach?
Or skip being clever, go with plan "B" which is to shuddup and just write a temp file. However, i'd like to learn something new and find out what's possible in this situation, beside getting high performance.
Given that you likely have a function like:
char *read_data(const char *fileName)
I think you will need to "skip being clever, go with plan "B" which is to shuddup and just write a temp file."
If you can dig around and find out if the call you are making is calling another function that takes a File * or an int for the file descriptor then you can do something better.
One thought that does come to mind, can you cahnge your code to write to a memory mapped file instead of to the heap? That way you would have a file on disk already and you would avoid the copying (though it'll still be on disk) and you can still give the function call the file name.
I'm not sure what kind of input the library function wants ... does it need a path/file name, or open file pointer, or open file descriptor?
If you don't want to hack the library and the function wants a string (path to a file), try making the temporary file in /dev/shm.
Otherwise, mmap might be the best option, please be sure to research posix_madvise() when using mmap() (or its counterpart posix_fadvise() if using a temporary file).
It looks like your talking about very little data to begin with, so I don't think you'll see a performance impact in whatever route you take.
Edit
Sorry, I just re-read your question .. perhaps I just read too fast. There is no way you are going to feed a function like:
char * foo(const char *filepath)
... with mmap().
If you can not modify the library to accept a file descriptor instead (or as an alternate to the path) .. just use /dev/shm and a temporary file, it will be quite cheap.
You're on Linux, can't you just grab the source of the library and hack in the function you need? If it's useful to others, you could even send a patch to the original author, so it will be in future versions for everyone.
Edit: Sorry. Just read the question. With my advise below, you fork a spare process, and the question of "does in work in a single process does not come up". I also see no reason you couldn't spawn a separate thread to do the push...
Not in the least elegant, but you could:
open a named pipe.
fork a streamer that does nothing but try to write to the pipe
pass the name of the pipe
which should be pretty robust...
mmap(), perhaps?