I need to check in a script if a file was modified since I read it (another application can modify it in between). According to bash manual there is a "-N" test which should report if a file was modified since last read. I tried it in a small script but it seems like it doesn't work.
#!/bin/bash
file="test.txt"
echo "test" > $file
cat $file;
if [ -N $file ];
then echo "modified since read";
else
echo "not modified since read";
fi
I also tried an alternative way by touching another file and using
if [ "file1" -nt "file2 ];
but this works only on a seconds accuracy which may under rare conditions not be sufficient. Is there any other bash-inbuilt solution for this problem or I do really need to use diff or md5sum?
Comparing file hashes (md5/sha1) is probably the only robust way to know (and audit) file changes, lest someone attempt to circumvent your attempts to detect file modifications. Also, rather than (or in addition to) periodically polling, you could implement some kind of file change notification: see this s.u. question for suggestions (e.g., using inotifywait).
But if it seems your scripts are gradually converging on a revision control system, just put these files in a git repo and let git detect and manage the changes for you.
(Edit: I just noticed that these are scripts monitoring log files (I was thinking you were monitoring actual scripts for file changes), in which case putting them under version control really wouldn't be appropriate.)
Related
I have to maintain a batch script of about 3500 lines littered with GOTO. Seems that the original "developer" hasn't heard of this famous paper and modular programming.
What the script does?
The script deals with the (silent) installation/uninstallation/reinstallation of several programs using different options. It could be split in several files that deal with each program in part. The problem is that if you're trying to take a part in another file that part will still GOTO another section that needs to be in the original script.
Refactoring?
Normally you wouldn't do a refactoring without having automated tests (so you can be sure you didn't break anything), but I don't know how to do it. There's no testing framework for that.
Partial Solution
I have come up with a partial "solution" that is some kind of adaptation of characterization tests (from Working Effectively with Legacy Code by Michael Feathers) and approval tests:
- create another script: test.py that replaces all commands (like copy or msiexec) with echo,
- redirect the output to a text file (good.txt),
- change the original batch script,
- run the test.py script again and save the output to another text file (current.txt),
- diff good.txt and current.txt -> if there are no differences then I didn't break anything, but if they are different I need to check if I broke something.
Problem with partial solution
How can I capture and replace all the commands? I could make a list of commands to replace, but there are also a lot of string concatenations to get the name and path of the program to be installed.
CMD level capture/hook?
Is there any way I can hook into the command line interpreter (CMD.exe) so I can replace on the fly all the calls to installers with echo?
Other suggestions?
Do I approach the problem in the wrong way? Can I do it better somehow? Do you have some advice I could use?
You could replace all COPY, DEL or CALL with %COPY%, %DEL% ,...
So you can use the same file for production and also for the tests.
#echo off
if not defined UNITTEST (
set "COPY=COPY"
set "DEL=DEL"
set "CALL=CALL"
)
%COPY% src dest
%DEL% somefile.txt
%CALL% installer.exe
And from your unittest.bat, you could start it via
#echo off
set "COPY=>>trace.log ECHO COPY"
set "DEL=>>trace.log ECHO DEL"
set "CALL=>>trace.log CALL ECHO "
del trace.log
set "unittest=Active"
call production.bat
fc good.txt trace.log
I'm not an expert in Batch, but I have done my fair share of it. With that said, I can offer a few tips.
Forget trying to do it all at once. Batch is very hard to debug. Echoing out to a log file helps a lot, but it will not capture everything you need if something goes wrong.
Work on breaking out the exe and msiexec calls into self-contained scripts. It is much easier to test the small script for the functionality you desire. Once you have that working, it is simple to call that script from the "Master" script.
Establish a good protocol for passing args to, and return codes from the smaller scripts. If there are common settings needed to be used for all the scripts consider using a central settings file.
GOTOs are not the devil, unless they pass control all over the place. Normally there are two good reasons that I know of to use GOTO’s.
Skip past a block of code that does not need to run.
To SET values into variables. Note there is a bug that can prevent variables from having their value set from within an 'IF' statement block. That little bug caused a big headache for me at one time.
Calls to a label might be better option at times.
Depending on how far back the legacy support is required, consider using Powershell when possible. The power and debugging capabilities of Powershell far out way the benefits of simple scripting of Batch. Which at 3500 lines simplicity has already been lost. You are already looking at Python, so maybe that could be used instead.
If you need a break point, use Pause. ECHO all the settings you need to examine right before the pause. This is as close to a break point I have found for batch.
Echo the command you intend to run to a log file and actually run it.
Write small verification scripts to be used independently or with the “Master” script to confirm you are getting the results you are expecting.
Use the right tool for the job. I like to use EditPadPro, RegexBuddy, and BeyondCompare for batch editing and comparing differences. There free tools that can be used too NotePad++ and Windiff. Making many edits in a file of that size is best handled by a good editor. IE inserting an echo at the beginning of a line that calls a cmd.exe.
Remember it is scripting not programming. While there is a lot of overlap of the two, the same exact approach to a problem may not be viable between the two.
Always make a backup copy of the scripts as a whole before mucking around. A fallback position is greatly appreciated when there is one small bug that you can’t find.
If it ain't broke... well you wouldn't be working on it if everything was working just fine.
Always test changes. And when you are done test it again. After that have someone else test it.
Just my .02. I’m sure someone else can chime in with more advanced advice. My knowledge on Batch has been acquired from the school of hard knocks, supplemented by ss64.com
I have a simple script to move files between directories. Basically, it is:
mv /dir/* /dir/proc/
saved into a shell script "mvproc.sh".
For some reason, when I run the script (sh mvproc.sh) the file indeed gets moved, but it does not retain the filename and instead gets just an empty filename. When I run the same command at the bash prompt, it works fine however.
This script used to work fine on Debian but we had a hard drive failure and I am now migrating everything over to a Ubuntu machine.
Any idea why this is happening? It seems so simple yet I cannot figure it out.
Many thanks.
edit...
I think I found the solution. For some reason it was putting in carriage returns and maybe line breaks or something that I could not see while editing the sh script in either Notepad++ or even gedit. To solve this, when I open the scripts in gedit, I do a Save As, and select Unix/Linux in the drop down menu towards the bottom. This hopefully gets rid of the weird carriage returns even though I could not see them.
Hopefully this helps some poor soul like me in the future pulling their hair out over this!
Thanks!
Try: mv /dir/file /dir/proc/file
You are indeed moving the file, but aren't specifying a destination name. Other usages of mv:
Move and rename: mv /dir/filename /dir/proc/newfilename
Rename: mv /dir/filename /dir/newfilename
Question:
Can inotify be used to reliably record files in a [linux] system?
Details:
I am attempting to use inotifywait to track users movements (currently using bash, but it has been suggested that I migrate to a scripting language). Ultimately I want to add new files to a database upon creation (create, moved_from), update existing rows in a database upon file modification (modify, attrib, move_to), and finally remove a row upon file deletion (delete). I am, however, running into many problems as even an action as seemingly simple as save, generates many inotifywait messages. Observe the following commands and their output (note, the use of /home/user/ is purely for example purposes):
Examples:
Example 1: Listen for file creation:
$ inotifywait -mr /home/user/ -e create --format %w:%f:%e:%T --timefmt %T
Touch:
$touch test.txt
/home/user/:test.txt:CREATE:21:35:30
Open a new file with vim then issue :w command:
$vim test2.txt
/home/user/:test2.txt:CREATE:21:35:30
Open an existing file with vim then issue :w command:
$vim test2.txt
/home/user/:4913:CREATE:21:35:30
/home/user/:test2.txt:CREATE:21:35:30
Open a new file with gedit then click save:
$gedit test3.txt
/home/user/:test3.txt~:CREATE:21:35:30
Open an existing file with gedit then click save:
$gedit test3.txt
/home/user/:.goutputstream-HN3ZDW:CREATE:21:35:30
/home/user/:test3.txt~:CREATE:21:35:30
Note that not only are two new files displayed as having ben created (4913 and .goutputstream-HN3ZDW), but also that the only file being created is test3.txt~ and not test3.txt, even though the file test3.txt is created when checked with the ls command. For completeness, here is the above example, but with a few more options.
Example 1: Listen for file creation, modification, deltion, and movement:
$ inotifywait -mr /home/user/ -e create -e modify -e delete -e moved_to -e moved_from --format %w:%f:%e:%T --timefmt %T
Touch:
$touch test.txt
/home/user/:test.txt:CREATE:21:35:30
Open a new file with vim then issue :w command:
$vim test2.txt
/home/user/:test2.txt:CREATE:22:12:32
Open an existing file with vim then issue :w command:
$vim test2.txt
/home/user/:4913:CREATE:22:04:35
/home/user/:4913:DELETE:22:04:35
/home/user/:test2.txt:MOVED_FROM:22:04:35
/home/user/:test2.txt~:MOVED_TO:22:04:35
/home/user/:test2.txt:CREATE:22:04:35
/home/user/:test2.txt~:DELETE:22:04:35
Open a new file with gedit then click save:
$gedit test3.txt
/home/user/:test3.txt~:CREATE:21:35:30
Open an existing file with gedit then click save:
$gedit test3.txt
/home/user/:.goutputstream-0WQ2DW:CREATE:22:06:34
/home/user/:test3.txt~:CREATE:22:06:34
/home/user/:.goutputstream-0WQ2DW:MOVED_FROM:22:06:34
/home/user/:test3.txt:MOVED_TO:22:06:34
Basically my question is "is it possible to use inotify to update a file in a database"? For example, if a user edits a file and saves it, I want it to be reflected in the database as an update to that file, and not a brand new file replacing a completely different file. Any help would be greatly appreciated, even if it's a suggestion pointing me in a different direction.
inotify tells you what happens like it happens.
Gedit, like most editors, saves by first writing a temporary file then moving that file into place. This avoids overwriting the file with a half-written version in case the editor or the whole system crashes while the file is being written. Vim takes a different approach (this can be configured, I won't go into details here — see e.g. why inode value changes when we edit in “vi” editor?): it first creates a temporary backup file, then writes the new file.
If you want these to be recorded as a single editing event, you'll have to perform some pattern recognition on the even log. A create-write-move sequence that replaces an existing file and a create-move-create delete sequence like vim's would be the archetypal patterns. Note that the pattern might be interleaved with other events.
I have a suspicion that there's a better way to do what you want to do, but I don't understand what you're trying to do. If you're trying to log user actions, you have already found a way, but there are simpler ways: loggedfs or the audit subsystem. If you want to keep a backup of all file versions, either hook up the editor to a version control system (this lets users control what gets backed up) or use a versioning filesystem such as copyfs. You can even store the files in the database directly, by using a filesystem like mysqlfs or postgresqlfs (admittedly neither project looks maintained).
Note: For wont of a better word I call the fluff at the start of source files --
/* #(#) $Id: file.c,v 1.9 2011/01/05 11:55:00 user Exp $
**************************************************************************
* COPYRIGHT, 2005-2011 *
...
*/
-- Keyword Substitution comments, although I do not know if this is just a subversion term.
Anyway, now to the question: We have a 3rd party supplier that we get source code from. These c source all have these keyword subst comments, and every time we get a new version from the supplier, all (1000+) files are changed because they update these comments for every release they send us, even if no source code changes whatsover are made in these files, so the only change is the comments. Now, before we compile and use these sources, we would be interested in doing a cursory code review to see the areas that have been changed. (Never trust the release history). However, this is rather difficult, as doing a simple folder diff will obviously list all files.
What I'm looking for now is whether there already exist any simple tools to strip these special multi line comments from the source files. Maybe anyone has a link to a grep or sed script that will scratch that stuff from the files?
Something like:
perl -ne 'if(m+/\*.*\$Id: +) $c = 1; print unless $c; if($c && m+\*/+) $c = 0;'
Note that this will work only if
such comments are delimited with /*...*/
on the first line there is $Id:
there is nothing after the */
there is no */ before the /*
And that it will strip all lines that are between start of comment and end of comment.
I have not tested it!
First, I would try to convince them to review either their version control system (looks as if they use RCS, still?) or if that is not possible to have them hook up to a svn or git server for submitting their changes. But perhaps you already did?
If nothing in that sense is possible, I would try to set up a git repository to hold the versions that they supply to you. Git allows you to have filters when you are importing or exporting and also has support for ignoring such tags for deltas between versions.
I'm running Matlab 7.8.0 under Windows.
I am calling an external utility using dos() which creates a file in the current directory.
I the file is created correctly, but it cannot be seen by exist or fopen, which return 0 and -1 respectively. The filename is correct!
>> pwd
ans =
I:\
>> ls
file1.asc file2.asc file3.asc
>> exist('file1.asc') % this file was there before
ans =
2
>> exist('file2.asc') % this file is newly created
ans =
0
to confirm it's not an odd/problematic filename, I checked from a Cygwin shell:
/cygdrive/i/ $ if [ -f file2.asc ]; then echo "OK"; fi
OK
So the file is good. I tried renaming it
/cygdrive/i/ $ mv file2.asc test
and in Matlab
>> ls
file1.asc file3.asc test
>> exist('test')
ans =
0
If I exit and restart Matlab it works fine. But I need to dynamically create the file and then access it!
Very mysterious.
You could try:
The rehash command to see if that helps.
The two-argument version of exists: exist('foo.txt', 'file')
The Matlab exist() command is not a simple filesystem operation; it also looks at variables, functions, etc. Since you're on I:, I'm assuming that's a network drive, and you're probably running in to the dir contents caching problem that Jonas mentions.
Here are a couple other workarounds, in case nsanders' two-arg exist() or Jonas' change notification fixes don't work for you.
Try using absolute paths to the files, like "fopen('I:\file2.asc')", instead of relative paths and pwd. Matlab will treat unqualified filenames as "partial paths" for both exist() and fopen(), and that interacts with the directory info caching. Ls() does not work with partial paths, which may be why it can see the file and the other functions can't.
You can use Java from within Matlab to do a simpler file existence test.
java.io.File('file2.asc').exists()
Or since the ls() command is showing the file you want, you can just implement the file existence check on top of ls.
ismember({'file2.asc'}, ls())
The "{ }" is necessary to make ismember() operate at the string level instead of the char level.
If you're still having trouble reading it, try doing a lower level read with Java from within Matlab. That will tell you whether it's specifically Matlab's I/O functions that are having trouble, or of the process itself lacks access to the file. Try this. If you get a char out of it, that means your Matlab.exe process can see the file.
istr = java.io.FileInputStream('file2.asc')
c = char(istr.read())
On Windows, I used to get change handle notification warnings at startup until I turned the warnings off. I don't have 7.8 at hand right now, but the warning may be off by default.
As explained on the MathWorks site, if Windows runs out of change notification handles, it will not be able to properly "sense" whether the content of a directory has changed, which might be causing your problems.
Are you sure that MATLAB is running as the same user as explorer is? If MATLAB requires elevated permissions to run then the drive mappings may be different and you could find that the I:\ drive is not mapped.
To fix this you would need to somehow map the I: drive under elevated permissions.