I am trying to find a method to separate two files that have been concatenated together using
copy /b file1+file2 file3.
I know the mime type and file type of at least one of the two files.
With the following csharp code you can do the split based on the fact that the zip file has the signature of 4 bytes that indicates the local file header. This code will break if the EXE has the same 4 bytes some where. If you want to conquer that you have to dig through the PE/COFF header to add up all section sizes
And NO, it is not very efficient to copy a stream byte by byte...
using(var fs = new FileStream(#"exeandzip.screwed", FileMode.Open))
{
var lfh = new byte[] { 0x50, 0x4b, 0x03, 0x04 }; /* zip local file header signature */
var match = 0;
var splitAt = 0;
var keep = new Queue<int>();
var b = fs.ReadByte();
using(var exe = new FileStream(
#"exeandzip.screwed.exe",
FileMode.Create))
{
while((b != -1) && (match<lfh.Length))
{ splitAt++;
if (b==lfh[match])
{
match++;
keep.Enqueue(b);
}
else
{
while(keep.Count>0)
{
exe.WriteByte((byte) keep.Dequeue());
}
exe.WriteByte((byte)b);
match=0;
}
b = fs.ReadByte();
}
}
if (match==lfh.Length && b!=-1)
{
keep.Enqueue(b);
splitAt = splitAt-lfh.Length;
Console.WriteLine(splitAt);
using(var zip = new FileStream(
#"exeandzip.screwed.zip",
FileMode.Create))
{
while(keep.Count>0)
{
zip.WriteByte((byte) keep.Dequeue());
}
b = fs.ReadByte();
while(b != -1)
{
zip.WriteByte((byte)b);
b = fs.ReadByte();
}
}
}
}
Or u can use foremost -i <input file> -o <output directory>
I've even split the apple webarchive format file in this way
Related
I have an app with data files (some images and xml files) i have packed them up in a zip file.
I open the file with zipme and save the files. I used this code for that
private void save1( ) {
InputStream is;
FileChooser.showOpenDialog(".zip", new ActionListener() {
#Override
public void actionPerformed(ActionEvent e) {
if (e != null && e.getSource() != null) {
String file = (String)e.getSource();
FileSystemStorage fs = FileSystemStorage.getInstance();
try {
InputStream is = fs.openInputStream(file);
ZipInputStream zipStream = new ZipInputStream(is);
ZipEntry entry;
// create a buffer to improve copy performance later.
byte[] buffer = new byte[2048];
while ((entry = zipStream.getNextEntry()) != null) {
String s = entry.getName();
String outdir = FileSystemStorage.getInstance().getAppHomePath();
if (outdir.length() > 0) {
outdir = outdir ;
}
String outpath = outdir + "/" + entry.getName();
OutputStream output = null;
try {
output = FileSystemStorage.getInstance().openOutputStream(outpath);
int len = 0;
while ((len = zipStream.read(buffer)) > 0) {
output.write(buffer, 0, len);
}
} finally {
// we must always close the output file
if (output != null) {
output.close();
}
}
} } catch (IOException ex) {
Log.p(ex.getMessage(), 0); } } }});}
i see in netbeans that in the simulator the files are saved to
users/.cn1
So this works on the desktop
To fetch the image i use
String outdir = FileSystemStorage.getInstance().getAppHomePath();
Image uur1 = EncodedImage.create(outdir + "/West.jpg");
i also tried without outdir but also no luck.
What do i wrong.
This should work without the extra slash:
Image uur1 = EncodedImage.create(outdir + "West.jpg");.
Notice that this code is case sensitive so make sure the file has the right casing. Is this failing on the simulator, if so place a breakpoint on the loading code and make sure the file is physically there
The answer i found on my question is:
1 No extra slash as Shai Among suggested:
2 Make a inputstream for enecodedimage.create() instead of only a string with the path to the file
Without the second part the app doesn't run correctly in the simulation and on the device
FileSystemStorage fs = FileSystemStorage.getInstance();
String outdir = FileSystemStorage.getInstance().getAppHomePath();
String outpath = outdir + West.jpg;
InputStream isk = fs.openInputStream(outpath);
Image uur = EncodedImage.create(isk);
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
You have a big list of unique items (hundreds of thousands of lines). You want to see if those items exist in another set of data. That other set of data is just a file with items line by line, and are also a unique set of data. You can put any data in a db, use any programming language, etc.
What do you do to compare these the fastest? Only constraints are that the hardware is a normal server, not a db server. One spindle max.
C? Implementing sorting algorithms? DB for indexing etc?
Admins took out the answer I went with "because the question is too broad": Bloom filters in python. It's really easy to implement with python's bloom filter library.
If your "test" file has a resonable size, a quick solution is to build a hash map for every entry in that file. A C# solution (runs in Big O ( N )) is this:
public static bool SetIsPresentIn(string firstFileLocation, string secondFileLocation)
{
HashSet<string> set = new HashSet<string> ();
using (var sr = new FileStream(firstFileLocation, FileMode.Open, FileAccess.Read))
{
using (var reader = new StreamReader(sr))
{
while (reader.EndOfStream == false )
{
var text = reader.ReadLine();
set.Add(text);
}
}
}
// iterating through the first one!
using (var secondFile = new FileStream(secondFileLocation, FileMode.Open, FileAccess.Read))
{
using (var reader = new StreamReader(secondFile))
{
while (reader.EndOfStream == false)
{
var line = reader.ReadLine();
// perform a lookup!
if (set.Remove(line) && set.Count == 0)
return true;
}
}
}
return set.Count == 0;
}
Otherwise I would do a clever thing: split your "test" file in file partitions: each partition name matches a hash code for each line. When iterating over second file, just create a hash code and search inside the coresponding partition that was built from the first file!
Example:
public static bool SetIsPresentInUsingFilePartitions(string firstFileLocation, string secondFileLocation, string partitionsRootLocation)
{
Dictionary<int, StreamWriter> partitionWriters = new Dictionary<int, StreamWriter>();
Dictionary<int, string> locations = new Dictionary<int, string>();
using (var sr = new FileStream(secondFileLocation, FileMode.Open, FileAccess.Read))
{
using (var reader = new StreamReader(sr))
{
while (reader.EndOfStream == false)
{
var text = reader.ReadLine();
var hCode = text.GetHashCode();
var fileName = Path.Combine(partitionsRootLocation, hCode.ToString ());
if (false == partitionWriters.ContainsKey(hCode))
{
var fs = new FileStream(fileName, FileMode.Create, FileAccess.ReadWrite);
partitionWriters[hCode] = new StreamWriter(fs);
locations[hCode] = fileName;
}
partitionWriters[hCode].WriteLine(text);
}
}
}
// close writers
foreach (var item in partitionWriters)
item.Value.Dispose();
using (var sr = new FileStream(firstFileLocation, FileMode.Open, FileAccess.Read))
{
using (var reader = new StreamReader(sr))
{
while (reader.EndOfStream == false)
{
var line = reader.ReadLine();
var hCode = line.GetHashCode();
string location;
if (false == locations.TryGetValue(hCode, out location))
{
return false; // tere's a line that is not found in the second file!
}
var found = false;
using (var file = new FileStream(location, FileMode.Open, FileAccess.Read))
{
using (var fs = new StreamReader(file))
{
while (fs.EndOfStream == false)
{
var firstFileLine = fs.ReadLine();
if (line == firstFileLine)
{
found = true;
break;
}
}
}
}
if (!found)
return false;
}
}
}
return true;
}
You could use a simple bash script:
First sort the lists
$ sort list1.txt > list1.sorted.txt
$ sort list2.txt > list2.sorted.txt
Then do a join to find the common elements for both lists:
$ join -1 1 -2 1 list1.sorted.txt list2.sorted.txt
This should be relatively fast and has a low memory consumption.
I am trying to read a text file in my air project. It is actually the config file used by TinkerProxy. I have the following so far:
//Read settings from TinkerProxy Config File
var TextFileLoader:URLLoader = new URLLoader();
var ArrayOfLines:Array;
TextFileLoader.addEventListener(Event.COMPLETE, onLoaded);
TextFileLoader.load(new URLRequest("/tinkerproxy-2_0/serproxy.cfg"));
//TextFileLoader.dataFormat = URLLoaderDataFormat.VARIABLES;
function onLoaded(e:Event):void {
ArrayOfLines = e.target.data.split(/\r/);
trace(e.target.data);
}
trace(ArrayOfLines[0]);
What I'm really trying to do is find the 'net_port1=5331' entry and store '5331' in a variable.
Here is a sample of the text file:
# Generated by TinkerProxy Configurator
#
# Timeout in seconds
# 0 means infinite, no timeout
timeout=0
newlines_to_nils=false
comm_ports=1
serial_device1=COM1
net_port1=5331
comm_baud1=9600
comm_databits1=8
comm_stopbits1=1
comm_parity1=none
The file is autogenerated so I can not edit it (or rather I'd want to read it as it is generated.)
I'm able to see the data via trace(e.target.data) but I cannot access the data via trace(ArrayOfLines[0]); for instance.
What am I missing?
Thanks in advance.
You probably need to split on \n (Unix) or \r\n (Windows), not \r.
Usually when loading a text file from the filesystem and breaking into lines, I normalize line endings by doing this:
var lines:Array = text.replace(/\r\n/g, "\n").split("\n");
Then you can iterate over the lines and decode each line as desired. The file appears to be akin to .properties format, for which there is no built in parser in AS3 (like XML, JSON, or URLVariables) but it's a pretty simple format. For example, this:
var props:Object = {};
for each(var line:String in lines){
// skip blank lines and comment lines
if(line == "" || line.charAt(0) == "#")
continue;
var arr:Array = line.split("=");
if(arr.length == 2)
props[arr[0]] = arr[1];
}
trace(JSON.stringify(props, null, 2))
Outputs this:
{
"comm_parity1": "none",
"comm_ports": "1 ",
"newlines_to_nils": "false",
"comm_baud1": "9600",
"serial_device1": "COM1",
"comm_databits1": "8",
"timeout": "0",
"comm_stopbits1": "1",
"net_port1": "5331"
}
Which allows you to access properties by name:
trace(props.net_port1); // "5331"
(Note that all values are strings, so for example newlines_to_nils is not false, it is "false".)
Alternatively, you could search for the key you are looking for and extract just the data you want:
var key:String = "net_port1=";
var index:int = text.indexOf(key);
if(index != -1){
// extract text after the desired search key
var value:String = text.substring(index + key.length);
// parseInt will read until it hits a non-numeric character
var net_port1:int = parseInt(value);
trace(net_port1); // 5331
}
Here is the solution that worked for me. Thanks again to Aaron for his answer on properties. I may use that in the future.
//Read settings from TinkerProxy Config File
var TextFileLoader:URLLoader = new URLLoader();
var ArrayOfLines:Array;
var Port:int;
var COM:int;
TextFileLoader.addEventListener(Event.COMPLETE, onLoaded);
TextFileLoader.load(new URLRequest("/tinkerproxy-2_0/serproxy.cfg"));
function findSubstring(array:Array, string:String):int {
for(var i:int = 0; i < array.length; i++){
if(array[i].indexOf(string) > -1){
return i; //Index of Substring
}
}
return -1; //Not Found
}
function onLoaded(e:Event):void {
ArrayOfLines = e.target.data.split(String.fromCharCode(13));
if(findSubstring(ArrayOfLines, "net_port") > -1){
Port = Number(ArrayOfLines[findSubstring(ArrayOfLines, "net_port")].split("=")[1]);
}
else{
Port = 5331; //Default if not port is found.
}
if(findSubstring(ArrayOfLines, "serial_device1") > -1){
COM = Number(ArrayOfLines[findSubstring(ArrayOfLines, "serial_device1")].split("serial_device1=COM")[1]);
}
else{
COM = 1; //Default if not port is found.
}
trace("COM: " + COM + " Port: " + Port);
}
I am parsing big text files and it's working fine for some time but after few minutes it give me exception (An unhandled exception of type 'System.UnauthorizedAccessException' occurred in System.Core.dll
Additional information: Access to the path is denied.)
I get exception on below mention line.
accessor = MemoryMapped.CreateViewAccessor(offset, length, MemoryMappedFileAccess.Read);
Below is my function
public static void CityStateZipAndZip4(string FilePath,long offset,long length,string spName)
{
try
{
long indexBreak = offset;
string fileName = Path.GetFileName(FilePath);
if (fileName.Contains(".txt"))
fileName = fileName.Replace(".txt", "");
System.IO.FileStream file = new System.IO.FileStream(#FilePath, FileMode.Open,FileAccess.Read, FileShare.Read );
Int64 b = file.Length;
MemoryMappedFile MemoryMapped = MemoryMappedFile.CreateFromFile(file, fileName, b, MemoryMappedFileAccess.Read, null, HandleInheritability.Inheritable, false);
using (MemoryMapped)
{
//long offset = 182; // 256 megabytes
//long length = 364; // 512 megabytes
MemoryMappedViewAccessor accessor = MemoryMapped.CreateViewAccessor(offset, length, MemoryMappedFileAccess.Read);
byte byteValue;
int index = 0;
int count = 0;
StringBuilder message = new StringBuilder();
do
{
if (indexBreak == index)
{
count = count + 1;
accessor.Dispose();
string NewRecord = message.ToString();
offset = offset + indexBreak;
length = length + indexBreak;
if (NewRecord.IndexOf("'") != -1)
{ NewRecord = NewRecord.Replace("'", "''"); }
// string Sql = "insert into " + DBTableName + " (ID, DataString) values( " + count + ",'" + NewRecord + "')";
string Code = "";
if (spName == AppConfig.sp_CityStateZip)
{
Code = NewRecord.Trim().Substring(0, 1);
}
InsertUpdateAndDeleteDB(spName, NewRecord.Trim (), Code);
accessor = MemoryMapped.CreateViewAccessor(offset, length, MemoryMappedFileAccess.Read);
message = new StringBuilder();
index = 0;
//break;
}
byteValue = accessor.ReadByte(index);
if (byteValue != 0)
{
char asciiChar = (char)byteValue;
message.Append(asciiChar);
}
index++;
} while (byteValue != 0);
}
MemoryMapped.Dispose();
}
catch (FileNotFoundException)
{
Console.WriteLine("Memory-mapped file does not exist. Run Process A first.");
}
}
Somewhere deep in resource processing code we have something like this:
try {
// Try loading some strings here.
} catch {
// Oops, could not load strings, try another way.
}
Exception is thrown and handled already, it would never show up in your application. The only way to see it is to attach debugger and observe this message.
As you could see from the code, it has nothing to do with your problem. The real problem here is what debugger shows you something you should not see.
Run the solution without debugging mode and it works fine.
This exception means that your program does not get Read access to the file from Windows.
Have you made sure that this file is not locked when your program tries to read it ?
For example, it could be a file that your own program is currently using.
If not, try to run your program as an Administrator and see if it makes a difference.
I continue to get the (Could not find a part of the path 'C:\Users(user profile)\VirtualStore\Program Files (x86)\E!PC\Macros) exception. The directory is there on the drive but im not sure why i continue to get this exception.
Extra6DestPath = "C:\Users\(user profile)\VirtualStore\Program Files (x86)\E!PC\Macros\"
static void copyMacrosAndBitmaps(string ExtraSourcePath, string Extra6xDestPath )
{
//counter for total Macro count on network
int Count = 0;
//counter for total bitmap count on network
int iCount = 0;
//Get File information to use for copy
FileInfo[] macrosArray;
FileInfo[] iconArray;
//Get Directory information to use for copy
DirectoryInfo di = new DirectoryInfo(ExtraSourcePath);
DirectoryInfo diIcon = new DirectoryInfo(ExtraIconPath);
//set all macro paths as a string from directory into an array
macrosArray = di.GetFiles("*.ebm");
Count = macrosArray.Length;
//set all bitmaps from directory into an array
iconArray = diIcon.GetFiles("*.bmp");
iCount = iconArray.Length;
//copy macros into destination folder
if (Count == 0)
{
throw new FileNotFoundException("No Macros found to copy");
}
else
{
for (int i = 0; i < Count; i++)
{
File.Copy(Extra6xSourcePathW7 + macrosArray[i].ToString(), Extra6xDestPath + iconArray[i].Name, true);
}
//Copy the bitmaps into destination folder
if (iCount == 0)
{
throw new FileNotFoundException("No bitmaps found to copy");
}
else
{
for (int i = 0; i < Count; i++)
{
File.Copy(ExtraIconPath + iconArray[i].ToString(), Extra6xDestPath + iconArray[i].Name, true);
}
}
}
}
I would first try declaring the path with # symbol, to handle characters that need to be escaped:
Extra6DestPath = #"C:\Users\(user profile)\VirtualStore\Program Files (x86)\E!PC\Macros\"