why minizip doesn't archive large file (larger 4 GB) - c

I am trying to use static minizip library on Windows 7 64-bit, using Visual Studio 2010.
The main goal is to archive files larger 4GB.
I build zlib using CMake 2.8, and linked in to my project.
It works for files less 4GB, but doesn't work proper for files larger 4GB.
Why I have a problem with archiving 5GB file using minizip?
Did I miss something on the build libraries stage?
Here are all my steps, library and project: https://github.com/koponomarenko/file_compression
Really need help. Thanks.
Updated:
"doesn't work proper for files larger 4GB" means that my test program archives 5GB txt file without any errors during this process. I checked zipWriteInFileInZip() returns ZIP_OK for all 5,368,709,120 bytes. zipCloseFileInZip() and zipClose() don't return any errors. But in created archive (I use 7-zip) file info is:
Uncompressed size: 4,294,967,295
Compressed size: 8,806,676
Attributes: empty field
CRC: 81970625
Method: Deflate
Host OS: FAT
Version: 45
And I get error message "Unsupported compression method" when try to unzip (using 7-zip) this archive.
I also zipped the same 5GB txt file with 7-zip. Here is file info from this archive:
Uncompressed size: 5,368,709,120
Compressed size: 9,608,471
Attributes: A
CRC: 81970625
Method: Deflate
Host OS: FAT
Version: 45
The first 3 fields are different from my archive.
Updated:
void Archive::create()
{
m_archiveHandle = zipOpen64(m_sName.c_str(), APPEND_STATUS_CREATE);
if ( !m_archiveHandle )
{
throw Error("Can't create the archive. Can be already created.");
}
}
void Archive::add(string sSrcFile, string sDstFile)
{
//////////////////////////// create a file
zip_fileinfo zfi;
memset(&zfi, 0, sizeof(zfi));
int zip64 = 1;
int ret = zipOpenNewFileInZip64(m_archiveHandle,
sDstFile.c_str(),
&zfi,
NULL, 0,
NULL, 0,
"my comment for this interior file",
Z_DEFLATED,
Z_DEFAULT_COMPRESSION,
zip64
);
if ( ret != 0 )
{
throw Error("Can't add file to a zip archive.");
}
//////////////////////////// write to the file
FILE * pFile = fopen( sSrcFile.c_str(), "rb" );
if ( !pFile )
{
throw Error("Can't open target file.");
}
char * buf = new char[BUF_SIZE];
int size_read = 0;
int err;
do
{
err = ZIP_OK;
size_read = (int)fread(buf, 1, BUF_SIZE, pFile);
if (size_read < BUF_SIZE)
if ( feof(pFile) == 0 )
{
err = ZIP_ERRNO;
}
if (size_read > 0)
{
err = zipWriteInFileInZip (m_archiveHandle, buf, size_read);
}
} while ((err == ZIP_OK) && (size_read>0));
delete [] buf;
//////////////////////////// close the file
if ( zipCloseFileInZip(m_archiveHandle) != 0 )
{
throw Error("Can't close added file.");
}
}
void Archive::close()
{
if ( !m_archiveHandle )
return;
if ( zipClose(m_archiveHandle, "my comment for exterior file") != 0)
{
throw Error( "Can't save changes and close." );
}
m_archiveHandle = 0;
}
I use this batch file to get static Zlib. I put it in zlib/build64 and run from "Visual Studio x64 Win64 Command Prompt".
rem ** go to the current script directory
cd %~dp0
cmake .. -G"Visual Studio 10 Win64" -DCMAKE_INSTALL_PREFIX="C:\devel\installed\zlib64"
msbuild /P:Configuration=Debug INSTALL.vcxproj
msbuild /P:Configuration=Release INSTALL.vcxproj
Updated:
Created archive can be unzipped by Total Commander and Windows Explorer. I compared a check-sum (md5, sha1) of the original file and unzipped file, and it was the same. But neither Total nor Explorer shows attributes of the compressed file. Meanwhile 7-zip says me "Unsupported compression method".
Updated December 09, 2013:
It seems that minizip has some problems with attributes when work with files larger 4GB on Windows. 7-Zip and WinZip can't unzip created archive.

The problem is in order of fields (extra field and file comment field) for Central Directory Record in ZIP. So if your file is bigger than 4GB and you added a comment for it you are going to face this bug.
It seems to me that the original author stopped maintenance this library, but I found a repo with some fresh changes on Github here https://github.com/nmoinvaz/minizip . So I wrote about this bug to the author of this repo.

Related

Opening files in C in Xcode

#include <stdio.h>
#include <math.h>
int main(void) {
int i=0;
int number =0;
float vector[100];
float sum=0., mean = 9., stdev=0.;
FILE *fp_in = NULL;
fp_in = fopen("stat_data.txt","r");
if(fp_in != NULL)
{
fscanf(fp_in,"%d",&number);
for (i=0; i < number; i++)
{
fscanf(fp_in, "%f", &vector[i]);
sum += vector[i];
}
mean = sum/number;
printf("Mean = %f\n",mean);
sum=0.0;
for(i=0; i<number;i++)
{
sum+= pow(vector[i]-mean,2);
}
stdev = sqrt(sum/(number - 1));
printf("standard deviation is %f\n", stdev);
}
else
{
printf("Opening of file stat_data.txt failed\n");
}
return 0;
}
I'm trying to open the text file "stat_data" in Xcode and it works on a linux computer, but it doesn't seem to work on Xcode. I think there's something wrong with the settings but I don't know what it could be.
Screenshot of the IDE showing where the stat_data.txt is placed -
This happened because the place where the Xcode created de binary file is different where your txt file is locate.
So I put my txt file in the same directory of my binary (readFile).
In the picture I list where my source and binary file are locate.
If you don't know where your binary file is. You can show it f you don't know where your binary file is. You can show it using the "show in finder".
So my result is it:
File opened.
Because you do not specify a path to the file name when you open it, the program expects the file to be in the current working directory which is set by Xcode when you run the program in Xcode and set by the shell when you run the program from the command line (as the directoy you are in when you run it).
There are two ways to get around this:
specify an absolute path when you open the file e.g.
/Users/thiagmarques/path/to/stat_data.txt
Edit the scheme to run from the directory containing the file. If you click on "Edit Scheme" and then select the Run section. The Options tab has a check box to "Use custom working directory". Check this and enter the directory containing your text file.

Cannot read network files in my OPC Server C++

A little background info:
PC is a Windows 7 64 bit machine with all SP's and windows updates.
I wrote an OPC server using LightOPC as the base and having issues reading network files as the title says. If I run the program manually it reads the files just fine.
There are no permissions issues obviously when running it manually but something else happens when the client launches it (FactoryTalk View 8.0).
The problem occurs when the program starts during the normal OPC connection launching. In the code std::strerror(errno) function returns
no such file or directory
which is obviously a lie. So hopefully someone can shed some light on what I can do to adjust the code (maybe some security level up or something like that) to let it read from these network files (about 10 total). I log data to the Windows Event viewer and I log the user name used to attach to the server, it is using NT AUTHORITY\SYSTEM at program launch.
Here is 1 of the pieces of code used to read 1 of the files (fin.is_open() is returning false so it falls into the else and I am writing the error to the OPC tag so I could debug it):
ifstream fin;
fin.open("\\\\SCS\\C\\Sorter Software\\Data\\TocStatus.dat"); // open a file | std::ios::binary | std::ios::app
if (fin.is_open())
{
char buf[512];
fin.getline(buf, 512);
char *theString = TrimCString(buf);
if (strlen(theString))
mbstowcs_s(NULL, V_BSTR(&tagValues[TAGS_MOTIONTL_STATUS_TOC].tvValue), strlen(theString)*2, theString, strlen(theString));
if (use_window)
ListView_SetItemText(TheListView, TAGS_MOTIONTL_STATUS_TOC-1, 1, theString);
if (tagValues[TAGS_MOTIONTL_STATUS_TOC].tvState.tsQuality == OPC_QUALITY_BAD)
{
tagValues[TAGS_MOTIONTL_STATUS_TOC].tvState.tsQuality = OPC_QUALITY_GOOD;
SetListViewItem(TAGS_MOTIONTL_STATUS_TOC, tagActivationFlags[TAGS_MOTIONTL_STATUS_TOC]);
}
if (tagValues[TAGS_MOTIONTL_STATUS_TOC].tvState.tsQuality != OPC_QUALITY_GOOD)
{
tagValues[TAGS_MOTIONTL_STATUS_TOC].tvState.tsQuality = OPC_QUALITY_GOOD;
SetListViewItem(TAGS_MOTIONTL_STATUS_TOC, 1);
}
}
else
{
V_BSTR(&tagValues[TAGS_MOTIONTL_STATUS_TOC].tvValue) = SysAllocString(L"UNKNOWN ERROR9 ");
mbstowcs_s(NULL, V_BSTR(&tagValues[TAGS_MOTIONTL_STATUS_TOCLOADED].tvValue), strlen(std::strerror(errno)) * 2, std::strerror(errno), strlen(std::strerror(errno)));
tagValues[TAGS_MOTIONTL_STATUS_TOC].tvState.tsQuality = OPC_QUALITY_GOOD;
if (use_window)
{
ListView_SetItemText(TheListView, TAGS_MOTIONTL_STATUS_TOC-1, 1, "UNKNOWN");
SetListViewItem(TAGS_MOTIONTL_STATUS_TOC, 2);
}
UpdateConnStatus(false);
}
fin.close();

Nagios plugin (written in C) unable to open file

I have a plugin written in c that will parse a .log file and determine the page hit count:
_xLogFileName = "./loging.log";
/* File operation starts here */
_xFile = fopen ( _xLogFileName, "r" );
if ( _xFile != NULL )
{
//read a line upto the end of the file
while ( fgets ( _xFileLine , sizeof _xFileLine, _xFile ) != NULL )
{
// #_xTiemInStr --> cur date in YYYY-MM-DD format to identify todays log
if(strstr(_xFileLine, _xTiemInStr) != NULL) {
if(strstr(_xFileLine, _xLoginHitString) != NULL) {
_xLoginPageCounter = _xLoginPageCounter + 1;
}
}
}
printf("Usage:Total Login Page Hit :%d\n",_xLoginPageCounter );
fclose ( _xFile );
return 0;
}
else
{
printf("error\n");
perror ( _xLogFileName );
return 3;
}
return 0;
Now I placed the a.out file in /usr/lib/nagios/plugin folder and placed the "loging.log" file in the same folder- for both chmod 777 done. I can run the plugin from command line but when I integrate the same with nagios then it is giving unknown status and printing the "error" from else part-- can anyone please help
2nd part
In addition I added the following code to determine from where the nagios is running?
char cwd[1024];
_xLogFileName1 = "loging.log";
if (getcwd(cwd, sizeof(cwd)) != NULL)
_xLogFileName = strcat( cwd,_xLogFileName1);
printf("FileName : %s\n", _xLogFileName);
and it is printing /loging.log in status information?
So where I have to actually place the file, my nagios is running from /etc/nagios3 and I placed the loging.log file there also but still it is not working.
UPDATE: Now it is working, as I print the pwd by the c program and find that it is running from my root (/) dir , so I placed the loging.log file there and now it is working fine.
You should be passing the full path to the file and storing that in '_xLogFileName1', and not just the file name alone. Strongly recommend that you do not copy files into the root directory.
Knowing web logs, it is very likely that the date string you're looking for is at the front of the line. So I'd recommend using 'strnstr' instead of 'strstr'. It would greatly improve your search speed.

How do I use Minizip (on Zlib)?

I'm trying to archive files for a cross-platform application, and it looks like Minizip (built on zlib) is about as portable as archivers come.
When I try to run the following dummy code, however, I get a system error [my executable] has stopped working. Windows can check online for a solution to the problem.
Can anyone help me see how to use this library? — (there's no doc or tutorial anywhere that I can find)
zip_fileinfo zfi;
int main()
{
zipFile zf = zipOpen("myarch.zip",APPEND_STATUS_ADDINZIP);
int ret = zipOpenNewFileInZip(zf,
"myfile.txt",
&zfi,
NULL, 0,
NULL, 0,
"my comment for this interior file",
Z_DEFLATED,
Z_NO_COMPRESSION
);
zipCloseFileInZip(zf);
zipClose(zf, "my comment for exterior file");
return 0;
}
Specs: Msys + MinGW, Windows 7, using zlibwapi.dll from zlib125dll.zip/dll32
Since I found this question via Google and it didn't contain any complete, working code, I am providing some here for future visitors.
int CreateZipFile (std::vector<wstring> paths)
{
zipFile zf = zipOpen(std::string(destinationPath.begin(), destinationPath.end()).c_str(), APPEND_STATUS_CREATE);
if (zf == NULL)
return 1;
bool _return = true;
for (size_t i = 0; i < paths.size(); i++)
{
std::fstream file(paths[i].c_str(), std::ios::binary | std::ios::in);
if (file.is_open())
{
file.seekg(0, std::ios::end);
long size = file.tellg();
file.seekg(0, std::ios::beg);
std::vector<char> buffer(size);
if (size == 0 || file.read(&buffer[0], size))
{
zip_fileinfo zfi = { 0 };
std::wstring fileName = paths[i].substr(paths[i].rfind('\\')+1);
if (S_OK == zipOpenNewFileInZip(zf, std::string(fileName.begin(), fileName.end()).c_str(), &zfi, NULL, 0, NULL, 0, NULL, Z_DEFLATED, Z_DEFAULT_COMPRESSION))
{
if (zipWriteInFileInZip(zf, size == 0 ? "" : &buffer[0], size))
_return = false;
if (zipCloseFileInZip(zf))
_return = false;
file.close();
continue;
}
}
file.close();
}
_return = false;
}
if (zipClose(zf, NULL))
return 3;
if (!_return)
return 4;
return S_OK;
}
The minizip library does come with examples; minizip.c for zipping and miniunz.c for unzipping. Both are command line utilities that show how to use the library. They are a mess though.
You also need to fill the zfi zip_fileinfo. At the very least you should initialize the structure to zero. zfi contains information about the file you want to store using zipOpenNewFileInZip. The structure should contain the date and attributes of "myfile.txt".
I recommend using PKWARE Desktop to diagnosis zip issues. It shows the structure/properties of the files in the ZIP and the ZIP file itself. When I opened the myarch.zip it told me there were errors. I drilled down into the file properties and found that the attributes were off.
The minizip lib is well documented. Just open the zip.h for details.
I can tell you here, you may have passed a wrong parameter for zipOpen. (APPEND_STATUS_ADDINZIP requires an existing zip file!)
Also, please check whether zipOpen returns a valid zipFile handle.

Create temporary file in C, MS Windows system

Basically, i have a program that is given a 4 meg compressed file, it has to decode this file into uncompressed ~ 100 meg, then compress it back into ~4 meg file. I need to store this intermediate 100 meg file somewhere on the drive (dont want to keep it in memory).
Program is written in C and will be executed on MS Windows 7. At the moment of uncompressing, no guaranteed folder (with write access) is given to the program (folder with source file might be read only and folder with target file might be not yet specified).
This has proven to be not an easy task:
1) I have read about a C function that creates a temp file that will disappear when closed or program is terminated. However, from what i understand it tries to make the file on disk C, in root directory, so this will obviously fail if user has no rights for that (which normal user doesnt)
2) I had an idea to use environmental/system variable TEMP and create a file there, BUT looking on a random Win7 PC which wasnt tweaked, i see that this variable points to c:/windows/temp, and that folder has specific rights for "users" - that is, they have rights to read, execute, create and write files, but not to delete them, check their attributes, etc. This means, i assume, that if program is ran with user privilleges, it will be able to make a file but not able to delete it, so the only way to "delete" it would be to open the file for writing and then close it, making it a 0 length file. This is also not desired, and i dont know how to query for system variables from C
3) So, basically, only idea i have right now is to make a function to open file that:
tries to create a temp file in the output dir, if possible
if failed, tries to create a temp file in input dir
if failed, tries to create a temp file in TEMP dir from system variable
if failed, tries to create a temp file in TMP dir from system variable
and a delete function that:
tries to remove() the file (by its name that is stored somewhere)
if failed, it tries to open the file for write, and close it, so it becomes a 0 byte file
Are there better ideas?
Any help is appreciated, thanks!
PS: Program must not use any external libraries like MFC or something, only built-in standart C functions
GetTempPath
Retrieves the path of the directory designated for temporary files.
GetTempFileName
Creates a name for a temporary file. If a unique file name is
generated, an empty file is created and the handle to it is released;
otherwise, only a file name is generated.
These two provide you easy way to obtain a location and name for a temporary file.
UPD: Code sample on MSDN: Creating and Using a Temporary File.
#include <windows.h>
#include <iostream>
#include <chrono>
#include <string>
#include <cstdio>
#include <chrono>
using namespace std;
int FileExists(string& filepath)
{
DWORD dwAttrib = GetFileAttributes(filepath.c_str());
return (dwAttrib != INVALID_FILE_ATTRIBUTES &&
!(dwAttrib & FILE_ATTRIBUTE_DIRECTORY));
}
int GetTemporaryFilePath(
string filePrefix,
string fileExt,
string& TmpFilePath /*return*/)
{
if (fileExt[0] == '.')
fileExt.erase(0,1);
char TempPath[MAX_PATH] = { 0 };
if (!GetTempPath(MAX_PATH, TempPath))
return -1;
uint16_t tickint = 0;
while(1) {
const int nowlen = 17; char nowstr[nowlen];
const int ticklen = 5; char tickstr[ticklen];
// Milliseconds since 1970
auto ms = chrono::duration_cast<chrono::milliseconds>(
chrono::system_clock::now().time_since_epoch()
);
__int64 nowint = ms.count();
snprintf(nowstr, nowlen, "%016" "I64" "x", nowint);
snprintf(tickstr, ticklen, "%04x", tickint);
TmpFilePath = string(TempPath)
+ filePrefix
+ "." + string(nowstr)
+ "." + string(tickstr)
+ "." + fileExt;
if (!FileExists(TmpFilePath)) {
//Touch File
FILE* w = fopen(TmpFilePath.c_str(), "w");
fclose(w);
break;
}
tickint++;
}
return 0;
}
int main()
{
string TmpFilePath;
GetTemporaryFilePath("MyFile", ".txt", TmpFilePath);
cout << "TmpFilePath: " << TmpFilePath << endl;
return 0;
}

Resources