In Spring-XD the file source detects new files in an input directory and streams their content through the pipeline.
Is there an analogous sink which creates separate result files in an output directory (e.g. with the original file names) and not a single file to which all results were appended, http://docs.spring.io/spring-xd/docs/current/reference/html/#file-sink: "The file sink uses the stream name as the default name for the file it creates, and places the file in the /tmp/xd/output/ directory."?
Scroll down to the options in that document you referenced.
Use --nameExpression=....
If you are using mode=contents; the original file name is available in the file_name header:
--nameExpression=headers[file_name]
mode=lines doesn't currently capture the file name (it will be fixed in the next release).
If you are using mode=ref, you need to set a header.
Minimal Working Example
In Spring-XD
stream create --name test --definition "file --mode=contents | b:file --binary=true --dirExpression='''/tmp/out''' --nameExpression=headers[file_name]" --deploy
than
echo "1111" > /tmp/xd/input/test/file1.txt
echo "2222" > /tmp/xd/input/test/file2.txt
results in
ll /tmp/out/
>
> -rw-rw-r-- 1 rmv rmv 5 Jul 7 10:19 file1.txt
> -rw-rw-r-- 1 rmv rmv 5 Jul 7 10:19 file2.txt
Related
I am writing a script in Lua 5.1 for use with a game engine (EDGE).
I need my script to copy about 20 files into a .miz file (which is really a zipped folder with a set structure) and navigate that structure and copy those files in from a non-zipped folder on the hard drive.
Because Windows 11 it the future I need to use NanaZip rather than 7z which isn't W11 supported.
However, all the examples I've found are for using LUA to zip up files, not insert non-zipped files INTO a zip file without unzipping it.
Is this even possible?
Similar to #koyaanisqatsi I tried it with 7z. You didn't comment on our question on why 7z should be avoided nor whether you are even allowed to use os.execute, but it should provide a good starting point:
os.execute("7z a yourZip.zip yourFile.png")
Where a is the flag for Add.
See the manual for other flags like compression: https://linux.die.net/man/1/7z
Windows 11 also have tar that have the option r and u
D:\temp>tar h
tar(bsdtar): manipulate archive files
First option must be a mode specifier:
-c Create -r Add/Replace -t List -u Update -x Extract
Common Options:
-b # Use # 512-byte records per I/O block
-f <filename> Location of archive (default \\.\tape0)
-v Verbose
-w Interactive
Create: tar -c [options] [<file> | <dir> | #<archive> | -C <dir> ]
<file>, <dir> add these items to archive
-z, -j, -J, --lzma Compress archive with gzip/bzip2/xz/lzma
--format {ustar|pax|cpio|shar} Select archive format
--exclude <pattern> Skip files that match pattern
-C <dir> Change to <dir> before processing remaining files
#<archive> Add entries from <archive> to output
List: tar -t [options] [<patterns>]
<patterns> If specified, list only entries that match
Extract: tar -x [options] [<patterns>]
<patterns> If specified, extract only entries that match
-k Keep (don't overwrite) existing files
-m Don't restore modification times
-O Write entries to stdout, don't restore to disk
-p Restore permissions (including ACLs, owner, file flags)
bsdtar 3.5.2 - libarchive 3.5.2 zlib/1.2.5.f-ipp bz2lib/1.0.6
( Above cmd.exe was opened from Lua with: os.execute('cmd') )
You can extract a ZIP with it but not creating one as far as i know.
(tar -xf archive.zip)
But is it a Problem for you to use TAR instead of ZIP?
does anyone knows how to use the Level option in IO::Compress::Zip?
I have the problem that I am trying to zip a DB Backup file. But after Zip the file is smaller then the original file and cant be used to importing the file on another server. The file is corupt ore something like this...
I am using perl to zip the file like this...
my $zipfile = zip['MYFILE'] => $zipFile, Zip64 => 1, Method => ZIP_CM_STORE
or die "Zip failed: $ZipError\n";
But no Success. For example orig file size is 13.910.216KB and when Zipped its only 13.909.298KB.
I dont know why but i think i need to set the Level option to Z_NO_COMPRESSION. How to do that?
Thanks
As has already been mentioned, by specifying the method ZIP_CM_STORE you are telling IO::Compress::Zip not to compress the file at all.
If you don't specify a Method at all, the code will use ZIP_CM_DEFLATE (which is the standard compression used in practically all zip files)
my $zipfile = zip['MYFILE'] => $zipFile, Zip64 => 1
or die "Zip failed: $ZipError\n";
If you want to change the compression level, use the Level option. By default it will uses Z_DEFAULT_COMPRESSION for the compression level. If you want the best compression, use Z_BEST_COMPRESSION
my $zipfile = zip['MYFILE'] => $zipFile, Zip64 => 1, Level => Z_BEST_COMPRESSION
or die "Zip failed: $ZipError\n";
Based on the conversation thread I tried to reproduce the use-case.
First create a file similar in size to yours
$ truncate -s 13910K test
$ ls -lh test
-rw-rw-r-- 1 xxx yyy 14M Apr 13 12:00 test
Add it to zip file using IO::Compress::Zip
$ perl -MIO::Compress::Zip=:all -e 'zip "test" => "test.zip", Method => ZIP_CM_STORE'
Check the sizes & CRC
$ crc32 test
49769d91
$ ls -l test
-rw-rw-r-- 1 xxx yyy 14243840 Apr 13 12:00 test
$ unzip -lv test.zip
Archive: test.zip
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
14243840 Stored 14243840 0% 2020-04-13 11:48 49769d91 test
-------- ------- --- -------
14243840 14243840 0% 1 file
All is as expected. The sizes & CRCs match.
Could you try this on your system please?
In the following path, i have the following logs:
When 10MB are reached for metricbeat file, it will go to write on metricbeat.1 and when this file size exceeds also, it will write to metricbeat.2 etc
[root# metricbeat]# ls -lorth
total 4.1M
-rwxrwxrwx 1 nifi 10 Aug 17 11:17 metricbeat.2
-rwxrwxrwx 1 nifi 10 Aug 17 11:17 metricbeat.1
-rwxrwxrwx 1 nifi 4.1M Aug 17 11:47 metricbeat
In NiFi (no cluster) I want to tail all the files that are stored on path
/logs/metricbeat/
I am using TailFile Processor with the following Configuration:
But, the main problem is that i am getting the following error:
'File to Tail' is invalid because There is no file to tail. Files must exist when starting this processor.
If i select the "Single Line" it gets the file "metricbeat"
Could you please tell me what i am doing wrong? Or how can i read all the "metricbeat" files from that path?
"Single File" mode does not require the file to exist before starting the processor, while "Multiple Files" mode does - hence the error you see.
When you do this:
cp file1 file2
(file2 already exists)
What actually happens behind the scene?
1) Does the content of file1 actually get copied to file2?
2) Or is a new file created with the name file2 (overriding the old one) which has same content of file1?
1) Since you're using "cp", I assume the OS is Linux.
2) On Linux, a "file" is referenced by "inodes". Here are two example files:
$ ls -li 1 2
245728 -rw-r--r-- 1 paulsm users 8 Aug 14 14:52 1
245729 -rw-r--r-- 1 paulsm users 8 Aug 14 14:52 2
$ cat 1
Hello 1
$ cat 2
Hello 2
3) Here is the result after "cp"
$ cp 1 2
$ ls -li 1 2
245728 -rw-r--r-- 1 paulsm users 8 Aug 14 14:52 1
245729 -rw-r--r-- 1 paulsm users 8 Aug 14 14:55 2
$ cat 2
Hello 1
You see:
a) the contents of "1" completely replace "2"
b) there is no "new file" - the inode for "2" remains unchanged from before the copy
c) the file date is changed along with the file contents
'Hope that helps .. PSM
Usually the first. Both an index-entry as well as the file's data are written.
Yet it would help to know on what (file-)system you are (guessing linux flavour).
You would probably be aware if you were creating a junction point or symbolic/hard LINK.
Think of it like this:
Hardlink is a pointer/name, that points to a data; i.e. it's just an alternative filename; it has same inode number as the file it was created from.
Copy obviously, copy of the data; point to a different direction that file it was copyed from; has different inode number.
Also difference is in system calls, but that`s somewhat deep-diving into issue
Ok, I have been working with Solaris for a 10+ years, and have never seen this...
I have a directory listing which includes both a file and subdirectory with the same name:
-rw-r--r-- 1 root other 15922214 Nov 29 2006 msheehan
drwxrwxrwx 12 msheehan sysadmin 2048 Mar 25 15:39 msheehan
I use file to discover contents of the file, and I get:
bash-2.03# file msheehan
msheehan: directory
bash-2.03# file msh*
msheehan: ascii text
msheehan: directory
I am not worried about the file, but I want to keep the directory, so I try rm:
bash-2.03# rm msheehan
rm: msheehan is a directory
So here is my two part question:
What's up with this?
How do I carefully delete the file?
Jonathan
Edit:
Thanks for the answers guys, both (so far) were helpful, but piping the listing to an editor did the trick, ala:
bash-2.03# ls -l > jb.txt
bash-2.03# vi jb.txt
Which contained:
-rw-r--r-- 1 root other 15922214 Nov 29 2006 msheehab^?n
drwxrwxrwx 12 msheehan sysadmin 2048 Mar 25 15:39 msheehan
Always be careful with the backspace key!
I would guess that these are in fact two different filenames that "look" the same, as the command file was able to distinguish them when the shell passed the expanded versions of the name in. Try piping ls into od or another hex/octal dump utility to see if they really have the same name, or if there are non-printing characters involved.
I'm wondering what could cause this. Aside from filesystem bugs, it could be caused by a non-ascii chararacter that got through somehow. In that case, use another language with easier string semantics to do the operation.
It would be interesting to see what would be the output of this ruby snippet:
ruby -e 'puts Dir["msheehan*"].inspect'
You can delete using the iNode
If you use the "-i" option in "ls"
$ ls -li
total 1
20801 -rw-r--r-- 1 root root 0 2010-11-08 01:55 a?
20802 -rw-r--r-- 1 root root 0 2010-11-08 01:55 a\?
$ find . -inum 20802 -exec rm {} \;
$ ls -li
total 1
20801 -rw-r--r-- 1 root root 0 2010-11-08 01:55 a?
I've an example (in Spanish) how you can delete a file using then iNode on Solaris
http://sparcki.blogspot.com/2010/03/como-eliminar-archivos-utilizando-su.html
Urko,
And a quick answer to part 2 of my own question...
I would imagine I could rename the directory, delete the file, and rename the directory back to it's original again.
... I would still be interested to see what other people come up with.
JB
I suspect that one of them has a strange character in the name. You could try using the shell wildcard expansion to see that: type
cat msh*
and press the wildcard expansion key (in my shell it's Ctrl-X *). You should get two names listed, perhaps one of which has an escape character in it.
To see if there are special characters in your file, Try the -b or -q options to ls,
assuming solaris 8 has those options.
As another solution to deleting the file you can bring up the graphical file browser
(gasp!) and drag and drop the unwanted file to the trash.
Another solution might be to move the one file to a different name (the one without the unknown special character), then delete the special character directory name with wildcards.
mv msheehan temp
rm mshee*
mv temp msheehan
Of course, you want to be sure that only the file you want to delete matches the wildcard.
And, for your particular case, since one was a directory and the other a file, this command might have solved it all:
rmdir msheeha*
One quick-and-easy way to see non-printing characters and whitespace is to pipe the output through cat -vet, e.g.:
# ls -l | cat -vet
Nice and easy to remember!
For part 2, since one name contains two extra characters, you can use:
mv sheehan abc
mv sheeha??n xyz
Once you've done that, you've got sane file names again, that you can fix up as you need.