I upload my builds to S3 and have a latest folder which always contains the newest build. Alongside of the build I have a SHASUM file SHASUM256.txt.asc which looks like this:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
c05c2e459aab92a490bbbb498c20a86433be23a1 server.tar.gz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iF4EAREIAAYFAlja1qkACgkQb6kayJ97kRKmHgEArOhI8kWEBwddkSY0rmPvFjGh
WgYTK2BuPf3M0mL1n50BAJCj5cl6ZbOIfp5j+499xk/LLxDu1PcWDA+u8XhZ88Sf
=czUN
-----END PGP SIGNATURE-----
I'm trying to get Salt to download and extract the archive anytime that the SHASUM changes, but it never thinks that the archive has changed. The only solution I've found is to force it to always download and extract the file, but that kind of defeats the whole purpose.
Here's what I'm currently using:
server-build:
archive.extracted:
- name: /opt/server
- makedirs: True
- source: {{ branch }}/latest/build/server.tar.gz
- archive_format: tar
- source_hash: {{ branch }}/latest/build/SHASUMS256.txt.asc
- overwrite: True
What's the right way to tell Salt to only download an archive based on changes to the SHASUM file? I'm currently using salt 2016.11.3 (Carbon).
So, there is no way to make it only download the file if the source hash has changed, but you can set source_hash_update: True and the new file will be extracted if the hash changes.
This should make it so that the file only downloads if the hash is updated, but it might still be downloaded some other times, that I am not certain about.
Related
I have downloaded many gz files from an ftp address :
http://ftp.ebi.ac.uk/pub/databases/spot/eQTL/sumstats/
How can I check that whether the files have been truncated during the download (i.e. wget did not download the entire file because of network connection) ? Thanks.
As you can see in each directory you have file md5sum.txt.
You can use command like:
md5sum -c md5sum.txt
This will calculate the hashes and compare them with the values in the file.
How can I check that whether the files have been truncated during the
download (i.e. wget did not download the entire file because of
network connection) ?
You might use spider mode to get just headers of response, for example
wget --spider http://ftp.ebi.ac.uk/pub/databases/spot/eQTL/sumstats/Alasoo_2018/exon/Alasoo_2018_exon_macrophage_naive.permuted.tsv.gz
gives output
Spider mode enabled. Check if remote file exists.
--2022-05-30 09:38:55-- http://ftp.ebi.ac.uk/pub/databases/spot/eQTL/sumstats/Alasoo_2018/exon/Alasoo_2018_exon_macrophage_naive.permuted.tsv.gz
Resolving ftp.ebi.ac.uk (ftp.ebi.ac.uk)... 193.62.193.138
Connecting to ftp.ebi.ac.uk (ftp.ebi.ac.uk)|193.62.193.138|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 645718 (631K) [application/octet-stream]
Remote file exists.
Length is size of file (in bytes) so after comparing it with your local file you will be able to tell if it is complete or not.
If you want to download missing parts if any, rather than merely check for completeness, then take look at -c option, from wget man page
-c
--continue
Continue getting a partially-downloaded file. This is useful when you want to finish up a download started by a previous instance of
Wget, or by another program.(...)
Is there a way to batch convert Collada dae files to Scenekit scn files?
My project uses approx 50 models created in sketchup that are updated regularly, these are exported to DAE but also need to be converted to SCN files for usage in xCode. I know it can be done manually via xCode and "Convert to SceneKit scene file format (scn)" but this take to much manual labour.
Based on https://the-nerd.be/2014/11/07/dynamically-load-collada-files-in-scenekit-at-runtime/ I figured out that the scntool is able to convert it via the command line and write the following script:
find ./dae -name "*.dae" | while read f ; do
inputfilename=$(basename $f)
echo $inputfilename
./scntool --convert $f --format scn --output ./scn/$inputfilename
done
for file in ./scn/*.dae; do
mv "$file" "./scn/$(basename "$file" .dae).scn"
done
#HixField has a good shell script for invoking scntool. Another way to do that is to leverage Xcode's build system, which does the same thing for any .dae files you put in a project's scnassets folder. Even if you're not bundling those files in your app, you can create a dummy Xcode target or project that contains all the assets you want to convert, and it'll convert them all whenever you build the target. (Which you could then integrate into a CI system or other automation.)
I agree with #Hixfield About everything except you need to add one more option to the scntool to get your materials correctly without need to re add all manually
scntool --convert INPUT.dae --format scn --output OUT.scn --asset-catalog-path .
The dot at the end of the command line is very important it means you will set resources to same location
If you don’t set the —asset-catalog-path . You will have no materials
Getting cURL error: {"code":400,"error":"Cannot execute learning task. : no classifier name given"}
Getting the same result whether I use the beta GUI tool or a cURL entry:
curl -X POST \
-F "Airplanes_positive_examples=#Airplanes.zip" \
-F "Biking_positive_examples=#Biking.zip" \
-F "GolfPuttingGreens_positive_examples=#GolfPuttingGreens.zip" \
-F "name=AllJpegClassifier" \
"https://gateway-a.watsonplatform.net/visual-recognition/api/v3/classifiers?api_key={my-api-key}&version=2016-05-20"
I have read all previous SO questions for this problem and made sure of the following:
Classifier name is alphanumeric only
Zip filenames are alphanumeric only
Image filenames are alphanumeric with _ - . only
Zip files contain between 27 and 49 images each
All image files are the same format (JPEG)
All images conform to pixel size and file size limits
Your command looks fine, and when I try it with my API key and my own zip files, it works. So I suspect there is something in your zip files that the system is having trouble with. If you could provide the "owner" guid field (also called you instance-id) I could look into our logs to try to diagnose it. This is displayed when you do a GET /classifiers/{cid} of an existing classifier. Alternatively, you could let me know one of your other existing classifier_ids
Another way would be if you could open a Bluemix support ticket and include copies of the zip files which you're using in this example. Then we can reproduce the problem.
Question:
Can inotify be used to reliably record files in a [linux] system?
Details:
I am attempting to use inotifywait to track users movements (currently using bash, but it has been suggested that I migrate to a scripting language). Ultimately I want to add new files to a database upon creation (create, moved_from), update existing rows in a database upon file modification (modify, attrib, move_to), and finally remove a row upon file deletion (delete). I am, however, running into many problems as even an action as seemingly simple as save, generates many inotifywait messages. Observe the following commands and their output (note, the use of /home/user/ is purely for example purposes):
Examples:
Example 1: Listen for file creation:
$ inotifywait -mr /home/user/ -e create --format %w:%f:%e:%T --timefmt %T
Touch:
$touch test.txt
/home/user/:test.txt:CREATE:21:35:30
Open a new file with vim then issue :w command:
$vim test2.txt
/home/user/:test2.txt:CREATE:21:35:30
Open an existing file with vim then issue :w command:
$vim test2.txt
/home/user/:4913:CREATE:21:35:30
/home/user/:test2.txt:CREATE:21:35:30
Open a new file with gedit then click save:
$gedit test3.txt
/home/user/:test3.txt~:CREATE:21:35:30
Open an existing file with gedit then click save:
$gedit test3.txt
/home/user/:.goutputstream-HN3ZDW:CREATE:21:35:30
/home/user/:test3.txt~:CREATE:21:35:30
Note that not only are two new files displayed as having ben created (4913 and .goutputstream-HN3ZDW), but also that the only file being created is test3.txt~ and not test3.txt, even though the file test3.txt is created when checked with the ls command. For completeness, here is the above example, but with a few more options.
Example 1: Listen for file creation, modification, deltion, and movement:
$ inotifywait -mr /home/user/ -e create -e modify -e delete -e moved_to -e moved_from --format %w:%f:%e:%T --timefmt %T
Touch:
$touch test.txt
/home/user/:test.txt:CREATE:21:35:30
Open a new file with vim then issue :w command:
$vim test2.txt
/home/user/:test2.txt:CREATE:22:12:32
Open an existing file with vim then issue :w command:
$vim test2.txt
/home/user/:4913:CREATE:22:04:35
/home/user/:4913:DELETE:22:04:35
/home/user/:test2.txt:MOVED_FROM:22:04:35
/home/user/:test2.txt~:MOVED_TO:22:04:35
/home/user/:test2.txt:CREATE:22:04:35
/home/user/:test2.txt~:DELETE:22:04:35
Open a new file with gedit then click save:
$gedit test3.txt
/home/user/:test3.txt~:CREATE:21:35:30
Open an existing file with gedit then click save:
$gedit test3.txt
/home/user/:.goutputstream-0WQ2DW:CREATE:22:06:34
/home/user/:test3.txt~:CREATE:22:06:34
/home/user/:.goutputstream-0WQ2DW:MOVED_FROM:22:06:34
/home/user/:test3.txt:MOVED_TO:22:06:34
Basically my question is "is it possible to use inotify to update a file in a database"? For example, if a user edits a file and saves it, I want it to be reflected in the database as an update to that file, and not a brand new file replacing a completely different file. Any help would be greatly appreciated, even if it's a suggestion pointing me in a different direction.
inotify tells you what happens like it happens.
Gedit, like most editors, saves by first writing a temporary file then moving that file into place. This avoids overwriting the file with a half-written version in case the editor or the whole system crashes while the file is being written. Vim takes a different approach (this can be configured, I won't go into details here — see e.g. why inode value changes when we edit in “vi” editor?): it first creates a temporary backup file, then writes the new file.
If you want these to be recorded as a single editing event, you'll have to perform some pattern recognition on the even log. A create-write-move sequence that replaces an existing file and a create-move-create delete sequence like vim's would be the archetypal patterns. Note that the pattern might be interleaved with other events.
I have a suspicion that there's a better way to do what you want to do, but I don't understand what you're trying to do. If you're trying to log user actions, you have already found a way, but there are simpler ways: loggedfs or the audit subsystem. If you want to keep a backup of all file versions, either hook up the editor to a version control system (this lets users control what gets backed up) or use a versioning filesystem such as copyfs. You can even store the files in the database directly, by using a filesystem like mysqlfs or postgresqlfs (admittedly neither project looks maintained).
Sometimes I need to revert many files - near 20 - 50, but need to save the files with local changes - if I in the future will use something.
Project is big - more than 10 000 files.
Is it possible create a copy of only the files that not were committed?
Manually find changes and copy takes near 2 hours - tree of project has many nested folders.
You can create a diff with svn diff and then reapplying the diff with svn patch.
However this is not really how you should work with SVN. Better up if you can create branch with your changes, then you can later merge that branch and share the content with your peers.
Note that creating a branch is relatively cheep in SVN. On the server the files as linked to the original until actually changed. Only your changed files will take space on the server.
Note:
svn diff only saves the changed lines of your files, not the complete files. But that is enough if you need to reapply the patch.
If you really want copies of files (rather than use svn diff or do a branch), tne approach (a version of which we use for server configuration file backups) is to check which files are modified. The notes below assume you are at the top level of your repo.
For instance, if you run svn status you might get output like this:
? plans/software/intro_jan12.log
? plans/software/intro_jan12.dvi
? plans/software/data.txt
? plans/software/intro_jan12.nav
M plans/software/intro_jan12.pdf
M plans/software/jan12.tex
? plans/software/jan12/flowRoot9298.png
? plans/software/jan12/viewE_comments.pdf
? plans/software/jan12/team.ps
? plans/software/jan12/team.png
? plans/it/plan.log
(The ? shown unknown files, the M shows modified files.)
You can then easily extract modified files and do stuff with them by doing something like svn status | egrep '^M'.
Turning that into a short shell script that copies modified files elsewhere is pretty easy:
# step 1
svn status | egrep '^M' | awk '{ print $2 }' > recipe_file
# step 2
rsync -a --files-from=recipe_file <repo> <dest>
Naturally <dest> can be on a remote machine.
Presumably, once you have audited the copy files at you can then do svn revert -R.