I've got a service which runs all the time and also keeps a log file. It basically adds new lines to the log file every few seconds. I'm written a small file which reads these lines and then parses them to various actions. The question I have is how can I delete the lines which I have already parsed from the log file without disrupting the writing of the log file by the service?
Usually when I need to delete a line in a file then I open the original one and a temporary one and then I just write all the lines to the temp file except the original which I want to delete. Obviously this method will not word here.
So how do I go about deleting them ?
In most commonly used file systems you can't delete a line from the beginning of a file without rewriting the entire file. I'd suggest instead of one large file, use lots of small files and rotate them for example once per day. The old files are deleted when you no longer need them.
Can't be done, unfortunately, without rewriting the file, either in-place or as a separate file.
One thing you may want to look at is to maintain a pointer in another file, specifying the position of the first unprocessed line.
Then your process simply opens the file and seeks to that location, processes some lines, then updates the pointer.
You'll still need to roll over the files at some point lest they continue to grow forever.
I'm not sure, but I'm thinking in this way:
New Line is a char, so you must delete chars for that line + New Line char
By the way, "moving" all characters back (to overwrite the old line), is like copying each character in a different position, and removing them from their old position
So no, I don't think you can just delete a line, you should rewrite all the file.
You can't, that just isn't how files work.
It sounds like you need some sort of message logging service / library that your program could connect to in order to log messages, which could then hide the underlying details of file opening / closing etc.
If each log line has a unique identifier (or even just line number), you could simply store in your log-parsing the identifier until which you got parsing. That way you don't have to change anything in the log file.
If the log file then starts to get too big, you could switch to a new one each day (for example).
Related
I'm using text files as a database for saving users' information for a game which i made using swi-prolog. The information is saved like this:user(Name,Password,Age,Points). What i want to do is to change a user's Points without having to rewrite the entire db. In other words, I am looking for something that will work like retractall(user(Name,_,_,_)), but with the text file. I know how to find the specific user using read/2, and how to assert a new fact using write/2, but i don't know how to delete one specific line in the text file.
Thank you for helping.
Take a look at SWI-Prolog's library(persistency). It removes a fact by adding a line that the fact is removed. If the file gets too big with add/remove lines, it provides db_sync/1 to write a clean file. OS file system operations do not allow to remove part of a file (except from truncating the end). The normal way to do this is to write a new file and, if successful, rename this to the existing one, so nothing is lost if you crash while writing the new file.
I'm running a loop that takes names from a file.
What I want to do is when the name is taken, it then deletes it from the file. How can I do this?
Try out fileinput.
It goes something like:
import fileinput
for line in fileinput.input(someFileName, inplace=True):
doSomething()
Any line iterated over is consumed, hence deleted from the file.
If you need to keep it just print it, it'll be rewritten to the file in the same location.
i.e. if you won't print the line back it'll disappear :)
Native files don't work that way. You'd have to rewrite the entire list of remaining files, each time through the loop. Performance will be bad (unless the list of files is always very short). And the failure modes are very bad if your process crashes while you're trying to update the file because you can't do the update in place. The best you could do is: write to a temp file, delete the original file, then rename (or move) the temp file.
Instead, you should consider using SQLite. You could either delete records from a File table, or have a status field that tracks which files have been processed.
I have opened a file using a and r+ but when I use fseek and ftell the file pointer is always 0.
My file looks like this:
1 -3
2 -8
And I want to add another line between the two but it is added in the end after the last line.
Someone in another forum said that when you open the file in append the pointer is always zero and you have to open it in r+ and if that doesn't work "you have to read the complete data and then insert the data in the variables and write it back." but I don't understand what they mean by that.
Can anyone help with inserting numbers in the middle of a file?
Thanks!
Would something like this work?
To transfer the data?
rewind(fp);
fscanf(fp,"%d",&ch);
fprintf(fp1,"%d",ch);
fseek(fp,1,0);
fscanf(fp,"%d",&ch);
fprintf(fp1,"%d",ch);
Like others already said, there's no easy way to insert data in the middle of a file. If you really want to do this, you can implement the following steps:
Create a second file
Copy all data before the place you want to insert to the second file
Insert the line you want to the second file
Copy the remaining data to the second file
Delete the original file
Rename the second file
Other approach is using binary files instead of text files. Although binary files are a bit harder to learn, once you understand how they work you'll see that working with them is much like working with arrays. To perform this task, for example, you'd not even need to use an auxiliary file.
There is no open mode that will allow you to "insert" data into a file at a random point. The only place you can add data without overwriting existing data is the end of the file (what you get opening with mode "a").
If you want to insert at a random position, you need to do it yourself.
One of the easier ways is to re-write the file completely (transfer the start of the old file to a new file, add your data to the new file, transfer the rest of the old file, and rename/overwrite at the end).
The hard way: you need to "shift" all the data from your insertion point to the end-of-file manually. That's not trivial to get right.
There isn't an easy way to insert data in the middle of the file. A file is basically an array of characters. To add a character in the middle, you need to copy everything following your insertion point down one location. With a file you need to read the data that follows and write it after your addition.
Generally, when you want to do something like this you create a new file. You copy the old file into it up to the point where you want to insert, then you write the data you want to insert, then you copy the rest of the old file. Finally, you rename the new file to the old file.
I am reading info (numbers) from a txt file and after that I am adding to those numbers, others I had in another file, with the same structure.
At the start of each line in the file is a number, that identifies a specific product. That code will allow me to search for the same product in the other file. In my program I have to add the other "variables" from one file to the other, and then replace it, in the same place in one of those files.
I didn't open any of those files with a or a+, I did it with r and r+ because i want to replace the information in the lines that may be in the middle of the file, and not in the end of it.
The program compiles, and runs, but when it comes to replace the info in the file, it just doesn't do anything.
How should I resolve the problem?
A program can replace (overwrite) text in the middle of the file. But the question is whether or not this should be performed.
In order to insert larger text or smaller text (and close up the gap), a new text file must be written. This is assuming the file is not fixed width. The fundamental rule is to copy all original text before the insertion to a new file. Write the new text. Finally write the remaining original text. This is a lot of work and will slow down even the simplest programs.
I suggest you design your data layout before you go any further. Also consider using a database, see my post: At what point is it worth using a database?
Your objective is to design the data to minimize duplication and data fetching.
my file pointer is pointing to end of a line. I want to remove all contents of that line, how do I do that?
I might need to move the file pointer to start of the line and then delete the contents.
You can only delete from the end of a file. To delete data from the middle of a file, you generally need to copy the subsequent data to cover up the gap (or, more easily as a rule, make a new copy of the file, skipping over the part you want to delete).
If you need to do things like this very often, you'll probably want to create some sort of indexed file so you can just delete from the index -- or, of course, use a database library to handle it for you.
You can't "delete" anything from a file. In C language files are accessed through streams, and streams don't support such operation as "delete a line" or "delete" anything at all. You can delete the entire file, but that's apparently not what you need.
Within the C language approach to working with files, all you can do is copy your original file to another file, skipping the line in question. The second file will look like the original one with the line deleted. After doing that you can destroy the original file and use the new one in its place.
There's a chance you might mean something else by your "delete" (what does your "delete" mean, BTW?). You might want to overwrite the contents of the line with space characters, for one example. If so, just move the current file pointer to the beginning of the line and write the appropriate number of space characters to the file.
You have to shift all of the content beyond the line back to the location where the line to be deleted begins.
If you're working in an environment that supports it, you could mmap(2) the file, work with the whole thing in memory and use memmove(3) to make the shifts.