I have a server log file which is constantly updated.
after some script execution, I want to collect the relevant part of my execution from the server log into an execution log.
Basically, I need to capture the last line number of my server log before the test started (which I already know how to do), say X, and after execution, copy the lines from X to the new end of server log, now, X+1000.
How do I copy only lines X to X+1000?
Thanks, Assaf
Try this
open("execution.log", "w").write("\n".join(open("server.log").readlines()[X:X+1000]))
Related
I have a log file (similar to a web server's access log) which I need to continuously read and use a regular expression to fetch a value in each line.
For example, if you could imagine reading from a web server's access log, I would be getting the IP address of each visitor as their visit is written to the log.
Of course this is easy to do using the linux command line (combination of tail and sed, or something like that), but I want to do it using C code.
I guess I could open the file, read X lines, save the number of the last line I read, and then on the next round open the file, move to the line X, and read from there, and so on, but this seems very clumsy.
Is there a known way or best practice for reading data from a continuously updating log file?
Thanks
I have an issue that I can't seem to resolve. I have a job (A) that calls another job (B). Iteration takes place in job B. There is a loop inside job B which reads one row at a time from a source file and writes to a text file.
Problem is the source file contains 37,000 rows but the execution stops at row 27,000. It crashes and gives me
"ERROR (version 6.0.0.0-353, build 1 from 2015-10-07 13.27.43 by
buildguy) : java.lang.StackOverflowError"
I have tried to gradually increase the stack in spoon.bat from 1g to 7g "-Xms7g" "-Xmx12g" "-XX:MaxPermSize=256m" but continues to crash. Any idea how can I solve this problem?
I finaly solved the the problem, i added one parameter ("-Xss512m") to my spoon.bat "%PENTAHO_DI_JAVA_OPTIONS%"=="" set PENTAHO_DI_JAVA_OPTIONS="-Xms2048m" "-Xmx4096m" "-Xss512m" "-XX:MaxPermSize=256m"
I want to create a log file in Scheme, but every time I add a new entry, I want it to be at the beginning of the file, so when I read X number of logs from the file again, it reads the X newest entries from new to old.
Example:
22/02/14 13:50 Newest log entry
22/02/14 13:45 Older log entry
22/02/14 13:40 Oldest log entry
Does anyone know how to do this using the 'open-input-file' and 'open-output-file' procedures?
The functionality you are requesting you need to write the whole logfile every time you need to write a new entry because you will overwrite the previous first entry with the next. Usually programs don't keep the commited parts of a logfile so this introduces more memory usage and your program must know when the log is being rotated to clear the buffer.
The standard way is to append a new entry, which leaves the previous log entries where the last log write put them.
As a compromise you might look for a program that displays a log file in the reverse order and prehaps tails it like that too. It's easy to implement so I guess it would exist already. Writing such an app if it doesn't exist would be trivial.
I am using unison to sync a bunch of folders together. Not just 2 roots, but I think my question can be made this simple...
Lets say I am syncing directory A and B using unison. If I remove file X from directory A, how does unison know what to do? Should it add X back to A from B or should it delete X from B?
Unison keeps a record of the contents of each path after each successful synchronization of that path (i.e., it remembers the contents at the last moment when they were the same in the two replicas).
We say that a path is updated (in some replica) if its current contents are different from its contents the last time it was successfully synchronized. Note that whether a path is updated has nothing to do with its last modification time—Unison considers only the contents when determining whether an update has occurred. This means that touching a file without changing its contents will not be recognized as an update. A file can even be changed several times and then changed back to its original contents; as long as Unison is only run at the end of this process, no update will be recognized.
In other words: Unison knows that you have deleted file X, because it's no longer on the disk in A, it knows it should delete it from B.
I've got a service which runs all the time and also keeps a log file. It basically adds new lines to the log file every few seconds. I'm written a small file which reads these lines and then parses them to various actions. The question I have is how can I delete the lines which I have already parsed from the log file without disrupting the writing of the log file by the service?
Usually when I need to delete a line in a file then I open the original one and a temporary one and then I just write all the lines to the temp file except the original which I want to delete. Obviously this method will not word here.
So how do I go about deleting them ?
In most commonly used file systems you can't delete a line from the beginning of a file without rewriting the entire file. I'd suggest instead of one large file, use lots of small files and rotate them for example once per day. The old files are deleted when you no longer need them.
Can't be done, unfortunately, without rewriting the file, either in-place or as a separate file.
One thing you may want to look at is to maintain a pointer in another file, specifying the position of the first unprocessed line.
Then your process simply opens the file and seeks to that location, processes some lines, then updates the pointer.
You'll still need to roll over the files at some point lest they continue to grow forever.
I'm not sure, but I'm thinking in this way:
New Line is a char, so you must delete chars for that line + New Line char
By the way, "moving" all characters back (to overwrite the old line), is like copying each character in a different position, and removing them from their old position
So no, I don't think you can just delete a line, you should rewrite all the file.
You can't, that just isn't how files work.
It sounds like you need some sort of message logging service / library that your program could connect to in order to log messages, which could then hide the underlying details of file opening / closing etc.
If each log line has a unique identifier (or even just line number), you could simply store in your log-parsing the identifier until which you got parsing. That way you don't have to change anything in the log file.
If the log file then starts to get too big, you could switch to a new one each day (for example).