Is there a program, or a script out there that can compare the md5 checksum of files I tried to create my own, but i'm having problems with any files that have a space in them, so I was wondering if it'd be easier to just use an application. md5deep is something I downloaded that returns the checksum.
rm md5mastervalue
for i in `ls /media/disk`; do md5deep -rb /media/disk/$i >> md5mastervalue; done
for d in 1 3 ; do cp -rf /media/disk/ /media/disk-$d & done
wait
rm md5valuet1
rm md5valuet3
for k in `ls /media/disk`
do
for f in 1 3; do md5deep -rb /media/disk-$f/$k >> md5valuet$f; done
done
for n in 1 3; do diff md5mastervalue md5valuet$n; done
echo Finished
are you on linux? if so, you can use md5sum, or sha512sum (better security). Example, create a base line of your files
$ sha512sum * > baseline.txt
then, the next time you want to check, just use -c option, eg
$ sha512sum -c baseline.txt
Related
youtube-dl can take some time parsing remote sites when called multiple times.
EDIT0 : I want to fetch multiple properties (here fileNames and remoteFileSizes) output by youtube-dl without having to run it multiple times.
I use those 2 properties to compare the local file size and ${remoteFileSizes[$i]} to tell if the file is finished downloading.
$ youtube-dl --restrict-filenames -o "%(title)s__%(format_id)s__%(id)s.%(ext)s" -f m4a,18,webm,251 -s -j https://www.youtube.com/watch?v=UnZbjvyzteo 2>errors_youtube-dl.log | jq -r ._filename,.filesize | paste - - > input_data.txt
$ cat input_data.txt
Alan_Jackson_-_I_Want_To_Stroll_Over_Heaven_With_You_Live__18__UnZbjvyzteo__youtube_com.mp4 8419513
Alan_Jackson_-_I_Want_To_Stroll_Over_Heaven_With_You_Live__250__UnZbjvyzteo__youtube_com.webm 1528955
Alan_Jackson_-_I_Want_To_Stroll_Over_Heaven_With_You_Live__140__UnZbjvyzteo__youtube_com.m4a 2797366
Alan_Jackson_-_I_Want_To_Stroll_Over_Heaven_With_You_Live__244__UnZbjvyzteo__youtube_com.webm 8171725
I want the first column in the fileNames array and the second column in the remoteFileSizes.
For the time being, I use a while read loop, but when this loop is finished my two arrays are lost :
$ fileNames=()
$ remoteFileSizes=()
$ cat input_data.txt | while read fileName remoteFileSize; do \
fileNames+=($fileName); \
remoteFileSizes+=($remoteFileSize); \
done
$ for fileNames in "${fileNames[#]}"; do \
echo PROCESSING....; \
done
$ echo "=> fileNames[0] = ${fileNames[0]}"
=> fileNames[0] =
$ echo "=> remoteFileSizes[0] = ${remoteFileSizes[0]}"
=> remoteFileSizes[0] =
$
Is it possible to assign two bash arrays with a single command ?
You assign variables in a subshell, so they are not visible in the parent shell. Read https://mywiki.wooledge.org/BashFAQ/024 . Remove the cat and do a redirection to solve your problem.
while IFS=$'\t' read -r fileName remoteFileSize; do
fileNames+=("$fileName")
remoteFileSizes+=("$remoteFileSize")
done < input_data.txt
You might also interest yourself in https://mywiki.wooledge.org/BashFAQ/001.
For what it's worth, if you're looking for specific/bespoke functionality from youtube-dl, I recommend creating your own python scripts using the 'embedded' approach: https://github.com/ytdl-org/youtube-dl/blob/master/README.md#embedding-youtube-dl
You can set your own signal for when a download is finished (text/chime/mail/whatever) and track downloads without having to compare file sizes.
I am working on embedded linux. I am trying to protect my rootfs by making it read only and mount fs file from the sdcard over the root.
I need both fs to be merged.
Any writes can be redirected to the fs file onto the sdcard.
Reading from the read only rootfs still possible.
I tried the following:
$ cd /media/sdcard
$ mount userfs /
$ cd /
$ echo a > a.txt
But I receive error:
-sh: a.txt: Read-only file system
Can any one help me to implement the needed functionality ?
To complete Ross answer, this is how I added overlayfs for /var/log:
add_overlayfs_mount() {
mkdir -p ${IMAGE_ROOTFS}/data/overlay/log
mkdir -p ${IMAGE_ROOTFS}/data/work/log
echo '/dev/sda4 /data ext4 defaults 0 0' >> ${IMAGE_ROOTFS}/etc/fstab
echo 'ofslog /var/log overlay defaults,x-systemd.requires=data,lowerdir=/var/log,upperdir=/data/overlay/log,workdir=/data/work/log 0 2' >> ${IMAGE_ROOTFS}/etc/fstab
}
ROOTFS_POSTPROCESS_COMMAND += "add_overlayfs_mount ; "
You can also use VOLATILE_BINDS in some situations:
VOLATILE_BINDS_append = " \
/data/etc/hostname /etc/hostname \n\
"
Yes, overlayfs is exactly what you want.
I try to write a wrapper script for songbook generation using lilypond, latex and sejda-console (for the pdf part). Everything works so far, but I have a problem with sejda that is giving me nuts.
Here is the relevant part of my code:
for %%i in (%f%) do (
sejda-console.bat extractbybookmarks -f ".\%%~ni.pdf" -o "export\%%~ni\%title%.pdf" -l 2 -p [BOOKMARK_NAME] -e "%title%" --overwrite
)
where f is a ";"-separated list of files. This command works for the first file, but fails for all others. I can't find any difference between the commands that sejda receives. Here is my console output:
make_sheet.bat -t "Live it up" --supress *.lytex
Configuring Sejda 3.2.30
Starting execution with arguments: 'extractbybookmarks -f .\book_drums.pdf -o export\book_drums\Live it up.pdf -l 2 -p [BOOKMARK_NAME] -e Live it up --overwrite'
Java version: '1.8.0_151'
Validating parameters.
Starting task (org.sejda.impl.sambox.ExtractByOutlineTask#28701274) execution.
Opening C:\Users\skr1_\Desktop\Tools\Songbook\Sample\out\.\book_drums.pdf
Retrieving outline information for level 2 and match regex Live it up
Starting extraction by outline, level 2 and match regex Live it up
Found 0 inherited images and 0 inherited fonts potentially unused
Starting extracting Live it up pages 9 9
Created output temporary buffer C:\Users\skr1_\Desktop\Tools\Songbook\Sample\out\export\book_drums\.sejdaTmp2789047920522272436.tmp
Appended relevant outline items
Filtering annotations
Skipped acroform merge, nothing to merge
Ending extracting Live it up
Task progress: 0% done
Moving C:\Users\skr1_\Desktop\Tools\Songbook\Sample\out\export\book_drums\.sejdaTmp2789047920522272436.tmp to C:\Users\skr1_\Desktop\Tools\Songbook\Sample\out\export\book_drums\Live it up.pdf.
Extraction completed and outputs written to org.sejda.model.output.FileOrDirectoryTaskOutput#478190fc[C:\Users\skr1_\Desktop\Tools\Songbook\Sample\out\export\book_drums\Live it up.pdf]
Task (org.sejda.impl.sambox.ExtractByOutlineTask#28701274) executed in 0 seconds
Completed execution
C:\Users\skr1_\Desktop\Tools\Songbook\Sample>(sejda-console.bat extractbybookmarks -f ".\book_general.pdf" -o "export\book_general\Live it up.pdf" -l 2 -p [BOOKMARK_NAME] -e "Live it up" --overwrite )
Configuring Sejda 3.2.30
Starting execution with arguments: 'extractbybookmarks -f .\book_general.pdf -o export\book_general\Live it up.pdf -l 2 -p [BOOKMARK_NAME] -e Live it up --overwrite'
Java version: '1.8.0_151'
Invalid value (File '.\book_general.pdf' does not exist): --files -f value... : pdf files to operate on. A list of existing pdf files (EX. -f /tmp/file1.pdf or -f /tmp/password_protected_file2.pdf:secret123) (required)
Invalid value (File '.\book_general.pdf' does not exist): --files -f value... : pdf files to operate on. A list of existing pdf files (EX. -f /tmp/file1.pdf or -f /tmp/password_protected_file2.pdf:secret123) (required)
C:\Users\skr1_\Desktop\Tools\Songbook\Sample>(sejda-console.bat extractbybookmarks -f ".\book_guitar.pdf" -o "export\book_guitar\Live it up.pdf" -l 2 -p [BOOKMARK_NAME] -e "Live it up" --overwrite )
Configuring Sejda 3.2.30
Starting execution with arguments: 'extractbybookmarks -f .\book_guitar.pdf -o export\book_guitar\Live it up.pdf -l 2 -p [BOOKMARK_NAME] -e Live it up --overwrite'
Java version: '1.8.0_151'
Invalid value (File '.\book_guitar.pdf' does not exist): --files -f value... : pdf files to operate on. A list of existing pdf files (EX. -f /tmp/file1.pdf or -f /tmp/password_protected_file2.pdf:secret123) (required)
Invalid value (File '.\book_guitar.pdf' does not exist): --files -f value... : pdf files to operate on. A list of existing pdf files (EX. -f /tmp/file1.pdf or -f /tmp/password_protected_file2.pdf:secret123) (required)
Even worse, if I copy the commands that sejda receives and paste them as arguments for a new command, everything works fine.
I suspect that something is happening with the working directory in between, but I don't get it.
Also, note that the output includes the command for subsequent passes of the for-loop (starting with "(sejda-console.bat ...") though echo is off. It is not included for the first run, however.
I'm not an expert with programming, especially not with batch, and any help would be very appreciated.
I'd suggest that sejda.bat is changing the current directory.
Try
pushd
call sejda.bat ...
popd
I have an executable that is used in a way such as the following:
executable -v -i inputFile.txt -o outputFile.eps
In order to be more efficient, I want to use a Bash variable in place of the input file. So, I want to do something like the following:
executable -v -i ["${inputData}"] -o outputFile.eps
Here, the square brackets represent some clever code.
Do you know of some trick that would allow me to pipe information into the described executable in this way?
Many thanks for your assistance
You can use the following construct:
<(command)
So, to have bash create a FIFO with the command as the output for you, instead of your attempted -i ["${inputData}"], you would do:
-i <(echo "$inputData")
Therefore, here is your final total command:
executable -v -i <(echo "$inputData") -o outputFile.eps
Echo is not safe to use for arbitrary input.
To correctly handle pathological cases like inputdata='\ntest' or inputdata='-e', you need
executable -v -i <(cat <<< "$inputData")
In zsh, the cat is not necessary
Edit: even this adds a trailing newline. To output the exact variable contents byte-by-byte, you need
executable -v -i <(printf "%s" "$inputData")
Note: zsh only:
To get a filename containing the contents of ${variable}, use:
<(<<<${variable})
Note:
<<<${variable} redirects STDIN to come from ${variable}
<<<${variable} is equivalent to (but faster than) cat <<<${variable}
So for the OP's case:
executable -v -i <(<<<${inputData}) -o outputFile.eps
I couldn't find a better solution than making temporary file and reference it into a variable. For example:
key=$(mktemp); # Creating a random file in the tmp folder
echo "some data" > $key; # Filling the file with content
ssh -i $key user#localhost
I assume when the bash session is close, the variable and temporary file is cleared
I have an external file that contains a list of patterns (pattern per line).
pattern1
foo bar
pattern_n
bar
bar foo
I would like to grep all files including the ones within sub-folders using those patterns, if the pattern matches, copy the file to some /tmp/mybackup/ and then delete it. What would be a good way of doing this?
If I understand your problem correctly, you need the following switches to grep:
-R to scan recursively
-l to print only matching filenames
-f to read the patterns from a file
-I to ignore binary files
so:
grep -RlIf patterns-file *
then feed this result to some other utility to perform the backup, eg xargs:
grep -RlIf patterns-file * | xargs -I {} mv {} /tmp/backup
or with a loop:
for afile in `grep -RlIf patterns-file *`; do
mv $afile /tmp/backup
done
Try
for x in `fgrep -f patternfile.txt -l -r .`; do cp $x /tmp/mybackup; rm $x; done