Array to Stream in Delphi 10.4 - arrays

How can i save and load an 'Array[0..9] of TJpegImage' to a MemoryStream.
There are many samples to save one Image to a Stream but no one save an Array.
I need it to save the Stream to a Database Blob field and later load the BlobField back to my Array.
All Images in the Array has the same Size.
Maybe a Stream of TByte can do this ?
But how to convert a TByte-Stream back to an Array?
Update:
Thank you very much MBo for your example.
I tried to use it and get some Problems.
I use a Paintbox to Draw the Picture and get an access violation at 0x0062b9ab:read of address 0x000....
This is the Code i try:
unit PicToMemStream;
interface
uses
Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics,
Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Vcl.Imaging.jpeg, Vcl.ExtCtrls;
type
TForm1 = class(TForm)
Image1: TImage;
PaintBox: TPaintBox;
private
{ Private-Deklarationen }
public
{ Public-Deklarationen }
end;
var
Form1: TForm1;
implementation
{$R *.dfm}
var
ms, temp: TMemoryStream;
jps: array of TJpegImage;
i, n, sz: Integer;
ProgPath: String;
begin
ProgPath := ExtractFilePath(Application.ExeName);
n := 3;
SetLength(jps, n);
ms := TMemoryStream.Create;
temp := TMemoryStream.Create;
for i := 0 to n - 1 do
begin
jps[i] := TJpegImage.Create;
jps[i].LoadFromFile(Format(ProgPath + '%s.jpg', [Chr(Ord('a') + i)]));
// in my case loads files a.jpg, b.jpg, c.jpg
end;
// Save Pictures to Stream and later in BlobField (100kByte)
try
ms.Write(n, SizeOf(n));
for i := 0 to n - 1 do
begin
temp.Clear;
// Save Image Data first here
jps[i].SaveToStream(temp); // jpeg data here
sz := temp.Size;
// Save length second here
ms.Write(sz, SizeOf(sz)); // size of image
temp.Position := 0;
ms.CopyFrom(temp, sz); // image data to final destination
end;
ms.SaveToFile(ProgPath + 'base.dat'); // for control // later here Write Data to BLob
// Read Picture and show it in a Paintbox or Image
// revert to empty stream
ms.Clear;
ms.LoadFromFile(ProgPath + 'base.dat'); // load from "base" // later here Read Data from Blob
ms.Read(n, SizeOf(n));
// here some actions to setup array
for i := 0 to n - 1 do // Read 3 times a picture
begin
ms.Read(sz, SizeOf(sz)); // Read Lenth of Picturedata first
temp.Clear; // Read Picturedata in jps[i]
temp.CopyFrom(ms, sz);
temp.Position := 0;
jps[i].LoadFromStream(temp);
Form1.PaintBox.Canvas.Draw(0, 0, jps[i]); // Show Picture with Error
//Form1.Image1.Picture.Bitmap.LoadFromStream(temp);
Sleep(2000); // To see the Picture for 2 seconds (Test)
end;
finally
ms.Free;
temp.Free;
for i := 0 to n - 1 do
jps[i].Free;
end;
end.
Update2:
I put the code in a ButtomClick procedure and change:
temp.Position := 0;
jps[i].LoadFromStream(temp);
Form1.PaintBox.Canvas.Draw(0, 0, jps[i]); // Show Picture in Paintbox
Form1.Image1.Picture.Assign(jps[i]); // Show Picture in TImage
Application.ProcessMessages;
Sleep(500); // To see the Picture for 2 seconds (Test)
end;
I will also test the other Ideas with using Master-Detail Table or TFDMemTable or Encoding to Text, to see what is the best for me.
Thanks to all Helpers.

All Images in the Array has the same Size - perhaps dimensions (width/height) are the same, but size of jpeg image in bytes varies depending on picture and compression. So I assume you need to store both image data and data size.
Possible approach:
write integer value - length of array - to memory stream ms
for every array element save it to temporary memory stream temp_ms
write integer value sz = ms.size to main memory stream ms
reset temp_ms.Position to 0
copy (CopyFrom) sz bytes from temp_ms to ms
clear temp_ms
repeat for all array elements
Now ms contents looks like:
10 14212 FFD8....D9 31234 FFD8....D9
10 images
1st image size
1st image contents
2nd image size
2nd image contents
To retrieve image, perform reverse actions:
reset ms position (if needed)
read number of images n from ms
set arrray size (if array is dynamic)
n times:
read size of image sz
copy sz bytes to temp_ms
reset temp_ms.Position
load array[i] from temp_ms
working example:
var
ms, temp: TMemoryStream;
jps: array of TJpegImage;
i, n, sz: Integer;
begin
n := 3;
SetLength(jps, n);
ms := TMemoryStream.Create;
temp := TMemoryStream.Create;
for i := 0 to n - 1 do begin
jps[i] := TJpegImage.Create;
jps[i].LoadFromFile(Format('h:\%s.jpg', [Chr(Ord('a') + i)]));
//in my case loads files a.jpg, b.jpg, c.jpg
end;
try
ms.Write(n, SizeOf(n));
for i :=0 to n - 1 do begin
temp.Clear;
jps[i].SaveToStream(temp); //jpeg data here
sz := temp.Size;
ms.Write(sz, SizeOf(sz)); //size of image
temp.Position := 0;
ms.CopyFrom(temp, sz); //image data to final destination
end;
ms.SaveToFile('h:\base.dat'); //for control
//revert to empty stream
ms.Clear;
ms.LoadFromFile('h:\base.dat'); //load from "base"
ms.Read(n, SizeOf(n));
//here some actions to setup array
for i :=0 to n - 1 do begin
ms.Read(sz, SizeOf(sz));
temp.Clear;
temp.CopyFrom(ms, sz);
temp.Position := 0;
jps[i].LoadFromStream(temp);
Canvas.Draw(i * 200, 0, jps[i]); //control shot
end;
finally
ms.Free;
temp.Free;
for i := 0 to n - 1 do
jps[i].Free;
end;

Why don't you use a TFDMemTable?
Another option can be, TStringList using System.NetEncoding.TNetEncoding.Base64 encoding
since TStringList is very much like a superpowered vector

Related

Recording and seeking to CSV file positions in Golang

I need to read a CSV file and record the locations of lines with certain values into an array, then later go back and retrieve those lines in no particular order and with good performance, so random access.
My program uses csv.NewReader(file), but I see no way to get or set the file offset that it uses. I tried file.Seek(0,io.SeekCurrent) to return the file position, but it doesn't change between calls to reader.Read(). I also tried fmt.Println("+v +v\n",reader,file) to see if anything stores the reader's file position, but I don't see it. I also don't know the best way to use the file position if I do find it.
Here's what I need to do:
file,_ = os.Open("stuff.csv")
reader = csv.NewReader(file)
//read file and record locations
for {
line,_ = reader.Read()
if wantToRememberLocation(line) {
locations = append(locations, getLocation()) //need this function
}
}
//then revisit certain lines
for {
reader.GoToLine(locations[random]) //need this function
line,_ = reader.Read()
doStuff(line)
}
Is there even a way to do this with the csv library, or will I have to write my own using more primitive file io functions?
Here's a solution using TeeReader. This example just saves all the positions and goes back and rereads some of them.
//set up some vars and readers to record position and length of each line
type Record struct {
Pos int64
Len int
}
records := make([]Record,1)
var buf bytes.Buffer
var pos int64
file,_ := Open("stuff.csv")
tr := io.TeeReader(file, &buf)
cr := csv.NewReader(tr)
//read first row and get things started
data,_ := cr.Read()
dostuff(data)
//length of current row determines position of next
lineBytes,_ := buf.ReadBytes('\n')
length := len(lineBytes)
pos += int64(length)
records[0].Len = length
records = append(records, Record{ Pos: pos })
for i:=1;;i++ {
//read csv data
data,err = c.Read()
if err != nil {break}
dostuff(data)
//record length and position
lineBytes,_ = buf.ReadBytes('\n')
lenth = len(lineBytes)
pos += int64(length)
records[i].Len = length
records = append(records, Record{ Pos: pos })
}
//prepare individual line reader
line := make([]byte,1000)
lineReader := bytes.NewReader(line)
//read random lines from file
for {
i := someLineNumber()
//use original file reader to fill byte slice with line
file.ReadAt(line[:records[i].Len], records[i].Pos)
//need new lineParser to start at beginning every time
lineReader.Seek(0,0)
lineParser := csv.NewReader(lineReader)
data,_ = lineParser.Read()
doStuff(data)
}
os.Open returns a File, which implements io.Seeker.
So you can do this to rewind the stream to the beginning:
_, err = file.Seek(0, io.SeekStart)
https://golang.org/src/os/file.go

Get pos and extract string in TFileStream

I am trying to find posisition of mykeyword and extract strings from a loaded file (up to 200 MB).
procedure TForm5.Button4Click(Sender: TObject);
var
Stream: TFileStream;
Buffer: array [0 .. 1023] of AnsiChar;
i: Integer;
myKeyword: string;
pullStr: AnsiString;
begin
myKeyword :='anything';
Stream := TFileStream.Create(edtTarget.Text, fmOpenRead);
while Stream.Position < Stream.Size do
begin
Stream.Read(Buffer, 1024);
m1.Lines.Add(Buffer); // no need, just display to evaluate
(* 1. Get address of given keyword *)
// i := Stream.PositionOf(myKeyword); < how to do this?
(* 2. Stream Exract *)
// pullStr := Stream.copy(i,1000); < how to do this ?
end;
end;
I have read other topics regarding file and string. I found a very good answer from here. And i think i want to expand those features.
Something like
TFileSearchReplace.GetStrPos(const KeyWord: string) : Integer;
TFileSearchReplace.ExtractStr (const KeyWord: string; Len : Integer) ;
procedure TForm5.Button4Click(Sender: TObject);
var
Stream: TFileStream;
Buffer: AnsiString;
i, BytesRead, SearchPos: Integer;
myKeyword: string;
pullStr: AnsiString;
Found: Boolean;
begin
myKeyword :='anything';
Found := False;
SetLength(Buffer, 1024);
Stream := TFileStream.Create(edtTarget.Text, fmOpenRead);
while Stream.Position < Stream.Size do
begin
// read some bytes and remember, how many bytes been read actually
BytesRead := Stream.Read(Buffer[1], 1024);
// glue new bytes to the end of the pullStr
pullStr := pullStr + copy(Buffer, 1, BytesRead);
// file is divided to two parts: before myKeyword, and after
// if myKeyword alreay found, there is nothing to do, just repeat reading to pullStr
if Found then
continue;
// if myKeyword is not found yet, pullStr acts like temporary buffer
// search for myKeyword in buffer
SearchPos := Pos(myKeyword, pullStr);
if SearchPos > 0 then
begin //keyword was found, delete from beginning up to and icluding myKeyword
// from now on, pullStr is not tmp buffer, but result
Found := True;
Delete(pullStr, 1, SearchPos + Length(myKeyWord) - 1);
continue;
end;
// myKeyword still not found. Find last line end in buffer
SearchPos := LastDelimiter(#13#10, pullStr);
// and delete everything before it
if SearchPos > 0 then
Delete(pullStr, 1, SearchPos);
// so if myKeyword spans across two reads, it still will be found in next iteration
end;
// if there is no myKeyword in file, clear buffer
if not Found then
pullStr := '';
end;

"Operator is not overloaded" when checking for a byte in an array of bytes

I have an array[0..2] of byte. I need to check if a byte is in that array or not. However, when using if ($52 in byteArray) then, I run into the error "Operator is not overloaded". I've tried setting an additional variable as a byte and then using that in the statement, but still get the error. Here's an incredibly simple program showing this:
program overloaded;
var
byteArray: array[0..2] of Byte;
begin
byteArray[0] := $00;
byteArray[1] := $69;
byteArray[2] := $52;
if ($52 in byteArray) then
writeLn('We will not get to this point');
end.
This will fail to compile with the above error in FPC 3.0.2.
You have two alternatives, overloading the operator or using for.
program ByteArrayIteration;
var
ByteArray: array[0..2] of Byte = ($00, $69, $52);
SomeByte : Byte = $52;
begin
for SomeByte in byteArray do
if SomeByte = $52 then
begin
WriteLn('We will get to this point');
Break;
end;
end.
And the overload alternative:
program OverloadInOperator;
uses SysUtils;
operator in(const A: Byte; const B: TBytes): Boolean;
var
i: Integer;
begin
Result := True;
for i := Low(B) to High(B) do if A = B[i] then Exit;
Result := False;
end;
var
Bytes : TBytes;
AByte : Byte = $52;
begin
Bytes := TBytes.Create($41, $52, $90);
if AByte in Bytes then WriteLn('Done');
end.
If your use case is limited to 255 items per array, consider using sets instead.
program SetOfByte;
var
Bytes : set of Byte = [$41, $52, $90];
AByte : Byte = $52;
begin
if AByte in Bytes then WriteLn('Done');
end.

Convert JIntArray to Array of Object

function Java_com_erm_controller_ARMReports_S35(PEnv: PJNIEnv; Obj: JObject; ex_UserRowID, ex_BSID : Integer; ex_RevalDate : JString;
ex_AFS, ex_HTM, ex_HFT : Boolean;
ex_IsMcCaulay_PNL: Boolean;
ex_Maturity, ex_Scale : JIntArray
): Integer; stdcall; export;
var objRpt : TARMReports;
I : Integer;
Len : JInt; //just a renamed delphi integer
aMaturity:array of Integer;
aScale:array of Integer;
begin
DLLErrorLog('CASH -S35');
objRpt := TARMReports.Create; JVM := TJNIEnv.Create(PEnv); ex_RevalDate_J := JVM.JStringToString(ex_RevalDate);
Len:=PEnv^.GetArrayLength(PEnv, ex_Maturity);
SetLength(aMaturity, Len);
Len:=PEnv^.GetArrayLength(PEnv, ex_Scale);
SetLength(aScale, Len);
DLLErrorLog('ex_Maturity Length'+ intToStr(Len));
for I := 0 to Len-1 do
begin
PEnv^.GetIntArrayRegion(PEnv, ex_Maturity, I, Len, #aMaturity[I]);
DLLErrorLog('ex_Maturity '+ IntToStr(aMaturity[I]));
PEnv^.GetIntArrayRegion(PEnv, ex_Scale, I, Len, #aScale[I]);
DLLErrorLog('ex_Scale '+ IntToStr(aScale[I]));
end;
Result := objRpt.S35(ex_UserRowID, ex_BSID, ex_RevalDate_J,
ex_AFS, ex_HTM, ex_HFT ,
ex_IsMcCaulay_PNL,
aMaturity, aScale
);
DLLErrorLog('CASH2 Ends -S35');
JVM.Free; objRpt.Free;
end;
Need to convert ex_Maturity, ex_Scale to objects to Delphi's Array of Integer.
Now while calling from Java it throws java.lang.ArrayIndexOutOfBoundsException
While printing in Log array values are getting . Please suggest us to work for me.
There are a couple of ways, depending on what, exactly your JIntArray is.
Firstly, if its an array of int (as in the primitive java type) then a get the length of the array via JNI, allocate a delphi array of integers and then get JNI to copy the data from the java array
Uses
AndroidAPI.JNI;
Var
Len:JNIInt; //just a renamed delphi integer
aMaturity:array of integer;
begin
Len:=PEnv^.GetArrayLength(PEnv, ex_Maturity);
//allocate the receiving array
SetLength(aMaturity, Len);
//now get the array data - note we are passing the address of the first element
//not the address of the array itself!
PEnv^.GetIntArrayRegion(PEnv, ex_Maturity, 0, Len, #aMaturity[0]);
//do stuff
end;
If you are dealing with an array of Integer (thats the Java class "Integer") then you need to get the array of objects from JNI one element at a time and use TJNIResolver to get the raw value;
Uses
AndroidAPI.JNI, AndroidAPI.JNIBridge;
Var
Len:JNIInt; //just a renamed delphi integer
Count:Integer;
Current:JNIObject;
CurrentValue:integer;
aMaturity:array of integer;
begin
Len:=PEnv^.GetArrayLength(PEnv, ex_Maturity);
//allocate the receiving array
SetLength(aMaturity, Len);
For Count:=0 to Len-1 do
begin
Current:=PEnv^.GetObjectArrayElement(PEnv, ex_Maturity, Count);
if assigned(Current) then
begin
CurrentValue:=TJNIResolver.GetRawValueFromJInteger(Current);
//Yes, you can inline this but the point is, here you do stuff with
//the element
aMaturity[Count]:=CurrentValue;
end;
end;
end;
Obviously the first method is much faster as crossing the JNI barrier is slow and you are only doing it once, whereas with the array of Java Integers you are doing it multiple times for each element.
You should also watch out for errors - I'm not checking for Java exceptions at any point which could crash and burn your app if you don't deal with them.
Edit : The OP has ready my answer and tried to work with it, which is nice. They have gotten a out of bounds exception in their code.
function Java_com_erm_controller_ARMReports_S35(PEnv: PJNIEnv; Obj: JObject; ex_UserRowID, ex_BSID : Integer; ex_RevalDate : JString;
ex_AFS, ex_HTM, ex_HFT : Boolean;
ex_IsMcCaulay_PNL: Boolean;
ex_Maturity, ex_Scale : JIntArray
): Integer; stdcall; export;
var objRpt : TARMReports;
I : Integer;
Len : JInt; //just a renamed delphi integer
aMaturity:array of Integer;
aScale:array of Integer;
begin
DLLErrorLog('CASH -S35');
objRpt := TARMReports.Create; JVM := TJNIEnv.Create(PEnv); ex_RevalDate_J := JVM.JStringToString(ex_RevalDate);
//you only have 1 length defined and possibly different array lengths
//process arrays seperately
Len:=PEnv^.GetArrayLength(PEnv, ex_Maturity);
SetLength(aMaturity, Len);
DLLErrorLog('ex_Maturity Length'+ intToStr(Len));
//only call this once, also watch the parameters you are passing in
PEnv^.GetIntArrayRegion(PEnv, ex_Maturity, 0, Len, #aMaturity[0]);
Len:=PEnv^.GetArrayLength(PEnv, ex_Scale);
SetLength(aScale, Len);
DLLErrorLog('ex_Scale Length'+ intToStr(Len));
PEnv^.GetIntArrayRegion(PEnv, ex_Scale, 0, Len, #aScale[0]);
Result := objRpt.S35(ex_UserRowID, ex_BSID, ex_RevalDate_J,
ex_AFS, ex_HTM, ex_HFT ,
ex_IsMcCaulay_PNL,
aMaturity, aScale
);
DLLErrorLog('CASH2 Ends -S35');
JVM.Free; objRpt.Free;
end;
What you were doing was getting the length twice, setting the delphi arrays correctly but then looping over them both in the same loop not taking into account that they could be different lengths. Your call to getinarrayregion was also passing the complete length in for aScale on the second parameter for both calls - if you really want to get each one in a loop like that then you need to pass the count and a length of 1 to only return 1 element - this is most likely what was causing the exception.
If you want to report the contents then create a procedure to do it, rather than using a loop inside your current procedure, you would have to copy and paste the loop to do it otherwise which is, franky bad coding practice and we don't want that now do we?
Sarcasm on
Not that expecting someone who has tried to help you to correct the your code rather than actually understanding the problem is any better, but ho hum.
Sarcasm off

Sharing data array between two applications in Delphi

I want to share array data between two applications. In my mind, first program create the array and the second program can read the array from already allocated memory area. The array is not a dynamic array.
I found a way to share pointer using OpenFileMapping and MapViewOfFile. I have no luck to implement array sharing and I think i don't want to use IPC method yet.
Is it possible to plan a scheme like this (sharing array)? My purpose is to minimize memory usage and reading data quickly.
Scratched my head thinking of what a short-but-complete example of sharing memory between two applications might be. The only option is a console application, GUI applications require a minimum of 3 files (DPR + PAS + DFM). So I cooked up a small example where one integers array is shared using a memory mapped file (backed by the page file so I don't need to have a phisical file on disk for this to work). The console application responds to 3 commands:
EXIT
SET NUM VALUE Changes the value at index NUM in the array to VALUE
DUMP NUM displays the value in the array at index NUM
DUMP ALL displays the whole array
Of course, the command processing code takes up about 80% of the whole application. To test this compile the following console application, find the executable and start it twice. Go to the first window and enter:
SET 1 100
SET 2 50
Go to the second console and enter this:
DUMP 1
DUMP 2
DUMP 3
SET 1 150
Go to the first console and enter this:
DUMP 1
There you have it, you've just witnessed sharing memory between two applications.
program Project2;
{$APPTYPE CONSOLE}
uses
SysUtils, Windows, Classes;
type
TSharedArray = array[0..10] of Integer;
PSharedArray = ^TSharedArray;
var
hFileMapping: THandle; // Mapping handle obtained using CreateFileMapping
SharedArray: PSharedArray; // Pointer to the shared array
cmd, s: string;
num, value, i: Integer;
L_CMD: TStringList;
function ReadNextCommand: string;
begin
WriteLn('Please enter command (one of EXIT, SET NUM VALUE, DUMP NUM, DUMP ALL)');
WriteLn;
ReadLn(Result);
end;
begin
try
hFileMapping := CreateFileMapping(0, nil, PAGE_READWRITE, 0, SizeOf(TSharedArray), '{C616DDE6-23E2-425C-B871-9E0DA54D96DF}');
if hFileMapping = 0 then
RaiseLastOSError
else
try
SharedArray := MapViewOfFile(hFileMapping, FILE_MAP_READ or FILE_MAP_WRITE, 0, 0, SizeOf(TSharedArray));
if SharedArray = nil then
RaiseLastOSError
else
try
WriteLn('Connected to the shared view of the file.');
cmd := ReadNextCommand;
while UpperCase(cmd) <> 'EXIT' do
begin
L_CMD := TStringList.Create;
try
L_CMD.DelimitedText := cmd;
for i:=0 to L_CMD.Count-1 do
L_CMD[i] := UpperCase(L_CMD[i]);
if (L_CMD.Count = 2) and (L_CMD[0] = 'DUMP') and TryStrToInt(L_CMD[1], num) then
WriteLn('SharedArray[', num, ']=', SharedArray^[num])
else if (L_CMD.Count = 2) and (L_CMD[0] = 'DUMP') and (L_CMD[1] = 'ALL') then
begin
for i:= Low(SharedArray^) to High(SharedArray^) do
WriteLn('SharedArray[', i, ']=', SharedArray^[i]);
end
else if (L_CMD.Count = 3) and (L_CMD[0] = 'SET') and TryStrToInt(L_CMD[1], num) and TryStrToInt(L_CMD[2], value) then
begin
SharedArray^[num] := Value;
WriteLn('SharedArray[', num, ']=', SharedArray^[num]);
end
else
WriteLn('Error processing command: ' + cmd);
finally L_CMD.Free;
end;
// Requst next command
cmd := ReadNextCommand;
end;
finally UnmapViewOfFile(SharedArray);
end;
finally CloseHandle(hFileMapping);
end;
except
on E: Exception do
Writeln(E.ClassName, ': ', E.Message);
end;
end.
A Named File Mapping would be the easiest solution, here is some short example code.
In this sample there is a main program that writes some data and reader(s) that only read from it.
Main:
type
TSharedData = record
Handle: THandle;
end;
PSharedData = ^TSharedData;
const
BUF_SIZE = 256;
var
SharedData: PSharedData;
hFileMapping: THandle; // Don't forget to close when you're done
function CreateNamedFileMapping(const Name: String): THandle;
begin
Result := CreateFileMapping(INVALID_HANDLE_VALUE, nil, PAGE_READWRITE, 0,
BUF_SIZE, PChar(Name));
Win32Check(Result > 0);
SharedData := MapViewOfFile(Result, FILE_MAP_ALL_ACCESS, 0, 0, BUF_SIZE);
Win32Check(Assigned(SharedData));
end;
procedure TForm1.Button1Click(Sender: TObject);
begin
hFileMapping := CreateNamedFileMapping('MySharedMemory');
Win32Check(hFileMapping > 0);
SharedData^.Handle := CreateHiddenWindow;
end;
reader:
var
hMapFile: THandle; // Don't forget to close
function GetSharedData: PSharedData;
begin
hMapFile := OpenFileMapping(FILE_MAP_ALL_ACCESS, False, 'MySharedMemory');
Win32Check(hMapFile > 0);
Result := MapViewOfFile(hMapFile, FILE_MAP_ALL_ACCESS, 0, 0, BUF_SIZE);
Win32Check(Assigned(Result));
end;

Resources