./minigzip: my_file_name.gz: invalid distances set
Downloaded the code from https://github.com/madler/zlib, and tested with different versions of zlib with the command:
git reset --hard && git clean -df && git checkout vxxx && ./configure && make && make test && ./minigzip -d my_file_name.gz
Turns out v1.2.3.4 ~ v1.2.7 will throw this "invalid distances set" error, v1.2.3.3 and below, v1.2.7.1 and following versions are decompressing the file just OK.
This problem is extracted from our hadoop cluster. We have a job to write gzipped files and then a job to consume it. Thousands of files in the consuming job, randomly one of them will throw this error. We are using v1.2.7 of zlib native library.
However when using gzip (gunzip) commnad utility for this file, it can be normally decompressed. Hence the different versions of zlib testing above.
Is this just corrupted data or should I upgrade the zlib version?
With confirmation from Adler, there actually was a bug fixed in the decompressor side of zlib v1.2.7.1.
To reproduce the problem, I decompressed the .gz file with latest version of zlib, then compressed the decompressed file with v1.2.7, and got the exact same size compressed file:
~/workdir$ ll
-rw------- 1 xxx xxx 232917709 Mar 15 13:54 my_file_name.gz
~/zlib/$ git reset --hard && git clean -df && git checkout v1.2.11 && ./configure && make && ./minigzip -d ../workdir/my_file_name.gz
~/workdir$ ll
-rw-rw-r-- 1 xxx xxx 705650679 Mar 15 13:59 my_file_name
~/zlib/$ git reset --hard && git clean -df && git checkout v1.2.7 && ./configure && make && ./minigzip ../workdir/my_file_name
~/workdir$ ll
-rw-rw-r-- 1 xxx xxx 232917709 Mar 15 14:02 my_file_name.gz
After this, a decompression with v1.2.7 will throw the same error:
~/zlib/$ ./minigzip -d ../workdir/my_file_name.gz
./minigzip: ../workdir/my_file_name.gz: invalid distances set
That says it's not a data corruption.
Note: Some of the outputs were edited.
Related
I am writing a script in Lua 5.1 for use with a game engine (EDGE).
I need my script to copy about 20 files into a .miz file (which is really a zipped folder with a set structure) and navigate that structure and copy those files in from a non-zipped folder on the hard drive.
Because Windows 11 it the future I need to use NanaZip rather than 7z which isn't W11 supported.
However, all the examples I've found are for using LUA to zip up files, not insert non-zipped files INTO a zip file without unzipping it.
Is this even possible?
Similar to #koyaanisqatsi I tried it with 7z. You didn't comment on our question on why 7z should be avoided nor whether you are even allowed to use os.execute, but it should provide a good starting point:
os.execute("7z a yourZip.zip yourFile.png")
Where a is the flag for Add.
See the manual for other flags like compression: https://linux.die.net/man/1/7z
Windows 11 also have tar that have the option r and u
D:\temp>tar h
tar(bsdtar): manipulate archive files
First option must be a mode specifier:
-c Create -r Add/Replace -t List -u Update -x Extract
Common Options:
-b # Use # 512-byte records per I/O block
-f <filename> Location of archive (default \\.\tape0)
-v Verbose
-w Interactive
Create: tar -c [options] [<file> | <dir> | #<archive> | -C <dir> ]
<file>, <dir> add these items to archive
-z, -j, -J, --lzma Compress archive with gzip/bzip2/xz/lzma
--format {ustar|pax|cpio|shar} Select archive format
--exclude <pattern> Skip files that match pattern
-C <dir> Change to <dir> before processing remaining files
#<archive> Add entries from <archive> to output
List: tar -t [options] [<patterns>]
<patterns> If specified, list only entries that match
Extract: tar -x [options] [<patterns>]
<patterns> If specified, extract only entries that match
-k Keep (don't overwrite) existing files
-m Don't restore modification times
-O Write entries to stdout, don't restore to disk
-p Restore permissions (including ACLs, owner, file flags)
bsdtar 3.5.2 - libarchive 3.5.2 zlib/1.2.5.f-ipp bz2lib/1.0.6
( Above cmd.exe was opened from Lua with: os.execute('cmd') )
You can extract a ZIP with it but not creating one as far as i know.
(tar -xf archive.zip)
But is it a Problem for you to use TAR instead of ZIP?
$ /usr/pgsql-12/bin/pg_upgrade \
> -b /usr/pgsql-1
pgsql-10/ pgsql-12/
> -b /usr/pgsql-10/bin/ \
> -B /usr/pgsql-12/bin/ \
> -d /var/lib/pgsql/1
10/ 12/
> -d /var/lib/pgsql/10/data/ \
> -D /var/lib/pgsql/12/data/ \
> --check
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for system-defined composite types in user tables ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for tables WITH OIDS ok
Checking for invalid "sql_identifier" user columns ok
Checking for presence of required libraries fatal
Your installation references loadable libraries that are missing from the
new installation. You can add these libraries to the new installation,
or remove the functions using them from the old installation. A list of
problem libraries is in the file:
loadable_libraries.txt
Failure, exiting
[postgres#localhost ~]$ cat loadable_libraries.txt
could not load library "$libdir/ltree": ERROR: could not access file "$libdir/ltree": No such file or directory
Database: ___
Database: ___
could not load library "$libdir/pg_trgm": ERROR: could not access file "$libdir/pg_trgm": No such file or directory
Database: ___
Database: ___
could not load library "$libdir/uuid-ossp": ERROR: could not access file "$libdir/uuid-ossp": No such file or directory
Database: ___
Database: ___
Valid steps for upgrade from postgres 10 to 12 will be highly appreciated. As I didn't found any highly reviewed link that is complete.
I am currently following with this link: https://www.postgresql.r2schools.com/how-to-upgrade-from-postgresql-11-to-12/. Replaced 10 for 11 in every command.
Thanks in Advance.
You can run the command:
sudo dnf install postgresql12-contrib
I try to write a wrapper script for songbook generation using lilypond, latex and sejda-console (for the pdf part). Everything works so far, but I have a problem with sejda that is giving me nuts.
Here is the relevant part of my code:
for %%i in (%f%) do (
sejda-console.bat extractbybookmarks -f ".\%%~ni.pdf" -o "export\%%~ni\%title%.pdf" -l 2 -p [BOOKMARK_NAME] -e "%title%" --overwrite
)
where f is a ";"-separated list of files. This command works for the first file, but fails for all others. I can't find any difference between the commands that sejda receives. Here is my console output:
make_sheet.bat -t "Live it up" --supress *.lytex
Configuring Sejda 3.2.30
Starting execution with arguments: 'extractbybookmarks -f .\book_drums.pdf -o export\book_drums\Live it up.pdf -l 2 -p [BOOKMARK_NAME] -e Live it up --overwrite'
Java version: '1.8.0_151'
Validating parameters.
Starting task (org.sejda.impl.sambox.ExtractByOutlineTask#28701274) execution.
Opening C:\Users\skr1_\Desktop\Tools\Songbook\Sample\out\.\book_drums.pdf
Retrieving outline information for level 2 and match regex Live it up
Starting extraction by outline, level 2 and match regex Live it up
Found 0 inherited images and 0 inherited fonts potentially unused
Starting extracting Live it up pages 9 9
Created output temporary buffer C:\Users\skr1_\Desktop\Tools\Songbook\Sample\out\export\book_drums\.sejdaTmp2789047920522272436.tmp
Appended relevant outline items
Filtering annotations
Skipped acroform merge, nothing to merge
Ending extracting Live it up
Task progress: 0% done
Moving C:\Users\skr1_\Desktop\Tools\Songbook\Sample\out\export\book_drums\.sejdaTmp2789047920522272436.tmp to C:\Users\skr1_\Desktop\Tools\Songbook\Sample\out\export\book_drums\Live it up.pdf.
Extraction completed and outputs written to org.sejda.model.output.FileOrDirectoryTaskOutput#478190fc[C:\Users\skr1_\Desktop\Tools\Songbook\Sample\out\export\book_drums\Live it up.pdf]
Task (org.sejda.impl.sambox.ExtractByOutlineTask#28701274) executed in 0 seconds
Completed execution
C:\Users\skr1_\Desktop\Tools\Songbook\Sample>(sejda-console.bat extractbybookmarks -f ".\book_general.pdf" -o "export\book_general\Live it up.pdf" -l 2 -p [BOOKMARK_NAME] -e "Live it up" --overwrite )
Configuring Sejda 3.2.30
Starting execution with arguments: 'extractbybookmarks -f .\book_general.pdf -o export\book_general\Live it up.pdf -l 2 -p [BOOKMARK_NAME] -e Live it up --overwrite'
Java version: '1.8.0_151'
Invalid value (File '.\book_general.pdf' does not exist): --files -f value... : pdf files to operate on. A list of existing pdf files (EX. -f /tmp/file1.pdf or -f /tmp/password_protected_file2.pdf:secret123) (required)
Invalid value (File '.\book_general.pdf' does not exist): --files -f value... : pdf files to operate on. A list of existing pdf files (EX. -f /tmp/file1.pdf or -f /tmp/password_protected_file2.pdf:secret123) (required)
C:\Users\skr1_\Desktop\Tools\Songbook\Sample>(sejda-console.bat extractbybookmarks -f ".\book_guitar.pdf" -o "export\book_guitar\Live it up.pdf" -l 2 -p [BOOKMARK_NAME] -e "Live it up" --overwrite )
Configuring Sejda 3.2.30
Starting execution with arguments: 'extractbybookmarks -f .\book_guitar.pdf -o export\book_guitar\Live it up.pdf -l 2 -p [BOOKMARK_NAME] -e Live it up --overwrite'
Java version: '1.8.0_151'
Invalid value (File '.\book_guitar.pdf' does not exist): --files -f value... : pdf files to operate on. A list of existing pdf files (EX. -f /tmp/file1.pdf or -f /tmp/password_protected_file2.pdf:secret123) (required)
Invalid value (File '.\book_guitar.pdf' does not exist): --files -f value... : pdf files to operate on. A list of existing pdf files (EX. -f /tmp/file1.pdf or -f /tmp/password_protected_file2.pdf:secret123) (required)
Even worse, if I copy the commands that sejda receives and paste them as arguments for a new command, everything works fine.
I suspect that something is happening with the working directory in between, but I don't get it.
Also, note that the output includes the command for subsequent passes of the for-loop (starting with "(sejda-console.bat ...") though echo is off. It is not included for the first run, however.
I'm not an expert with programming, especially not with batch, and any help would be very appreciated.
I'd suggest that sejda.bat is changing the current directory.
Try
pushd
call sejda.bat ...
popd
I installed webalizer on my apache 2 webserver yesterday and came across the problem, that all the old access logs are not used. The directory list looks like that:
/var/log/apache2/
access.log
access.log1
access.log.10.gz
access.log.11.gz
...
How can I import all my files at once?
I tried several things, but it was telling me, that the records were ignored.
Hope somone can help. Thanks!
I ran into the same problem. I had just installed webalizer, and changed it to incremental mode (here are the relevant entries from my /etc/webalizer/webalizer.conf):
LogFile /var/log/apache2/access.log.1
OutputDir /var/www/htdocs/w
Incremental yes
IncrementalName webalizer.current
And then I ran webalizer by hand, which initialized the non-gz files in my logs directory. After that, any attempt to manually import an older gz logfile (by running webalizer /var/log/apache2/access.log.2.gz for instance) resulted in all of the entries being ignored.
I suspect this is because the entries found in the gz logs were older than the last import- I had to delete my webalizer.current file (really I cleared the whole dir- either way should work). Finally, in reverse order (oldest first), I could import the old gz files one at a time:
bhs128#home:~$ cd /var/log/apache2
bhs128#home:/var/log/apache2$ sudo rm -rf /var/www/htdocs/w/*
bhs128#home:/var/log/apache2$ ls -1t /var/log/apache2/access.log*gz | grep -o [0-9]* | tail -n1
52
bhs128#home:/var/log/apache2$ for i in {52..2}; do webalizer /var/log/apache2/access.log.$i.gz; done
I just had the same problem, and I took a look into the webalizer.current file:
$ head -n 2 webalizer.current
# Webalizer V2.21-02 Incremental Data - 11/05/2019 22:29:02
2019 11 5 22 29 2
The second line seems to contain the timestamp of the last run, so I just changed the year to 2018. After that, I was able to import older log files than the last imported ones, without having to delete all the data first.
I have a list of files in my current working copy that have been modified locally. There are about 50 files that have been changed.
I am using the following command to copy files that have been modified in subversion to a folder called /backup. Is there a way to do this but maintain the directories they are in? So it would do something similar to exporting a SVN diff of files. For example if I changed a file called /usr/lib/SPL/RFC.php then it would copy the usr/lib/SPL directory to backup also.
cp `svn st | ack '^M' | cut -b 8-` backup
It looks strange, but it is really easy to copy files with tar. E.g.
tar -cf - $( svn st | ack '^M' | cut -b 8- ) |
tar -C /backup -xf -
Why not create a patch of your changes? That way you have one file containing all of your changes which you can timestamp in the name - something like 2012-05-28-17-30-00-UnitTestChanges.patch, one per day.
Then you can roll up your changes to a fresh checkout once you're ready, and then commit them.
FYI: Subversion 1.8 should have checkpointing / shelving (which is what you seem to want to do), but that's a long way off, and might only be added in Subversion 1.9.