Go loop through datastore cursor - google-app-engine

Is there a way to loop through datastore results via a cursor, until there are no results?
I need to pull every single record and iterate through it. But it's like 4 million records. So I wanted to work in chunks of 1000 to start with.
I know the below won't work. But it's just to show the sort of logic I'm hoping for.
q := datastore.NewQuery("Foo").Limit(1000)
while t, err := models.Client.GetAll(models.Ctx, q, &results) {
...
myCursor, err := t.Cursor()
q.Start(cursor)
}
EDIT: Here is the only way I found how to do it. I'm sure there is a more elegant way.
limit := 1000
offset := 0
resultCount := 1
for resultCount > 0 {
q := datastore.NewQuery("Foo").Limit(limit).Offset(offset)
_, err := client.GetAll(ctx, q, &foo)
checkErr(err)
resultCount = len(foo)
processPaths(foo)
offset += resultCount
}
It doesn't actually use the cursor.

Related

Special For-Loop in Pascal

I work with a current software for the simulation of power plant processes. Smaller scripts can be written within the software for automation, these scripts are based on Pascal and own function libraries. Was simply retained after the initial release 20 years ago.
My simple script transfers values from one element to another and has this structure:
var f: integer;
S13Be.MXTSTO.data(1,1) := 22;
S12Be.MXTSTO.data(1,S12Be.NFLOW) := 22;
S11Be.MXTSTO.data(1,1) := S12Be.MXTSTO.data(1,S12Be.NFLOW);
S10Be.MXTSTO.data(1,S10Be.NFLOW) := 22;
S9Be.MXTSTO.data(1,1) := S10Be.MXTSTO.data(1,S10Be.NFLOW);
S8Be.MXTSTO.data(1,S8Be.NFLOW) := 22;
S7Be.MXTSTO.data(1,1) := S8Be.MXTSTO.data(1,S8Be.NFLOW);
S5Be.MXTSTO.data(1,S5Be.NFLOW) := 22;
S4Be.MXTSTO.data(1,1) := S5Be.MXTSTO.data(1,S4Be.NFLOW);
S2Be.MXTSTO.data(1,S2Be.NFLOW) := 22;
S1Be.MXTSTO.data(1,1) := S2Be.MXTSTO.data(1,S4Be.NFLOW);
for f := 1 to S13Be.NFLOW+1 do begin
S13Be.MXTSTO.data(1,f) := S13Be.MXTSTO.data(1,1);
end;
for f := 1 to S12Be.NFLOW+1 do begin
S12Be.MXTSTO.data(1,f) := S12Be.MXTSTO.data(1,1);
end;
for f := 1 to S11Be.NFLOW+1 do begin
S11Be.MXTSTO.data(1,f) := S11Be.MXTSTO.data(1,1);
end;
.
.
.
for f := 1 to S2Be.NFLOW+1 do begin
S2Be.MXTSTO.data(1,f) := S2Be.MXTSTO.data(1,1);
end;
for f := 1 to S1Be.NFLOW+1 do begin
S1Be.MXTSTO.data(1,f) := S1Be.MXTSTO.data(1,1);
end;
I would like to put another loop around the outside so that the elements are automatically selected.
The names of the elements are S1Be, S2Be.... S13Be and S1Ent, S2Ent, S3Ent...S13Ent
.MXSTO.data accesses a matrix in the respective element
(1,f) defines the position in the matrix (currently there are only 1x5 and 1x10 matrices; the value .NFLOW specifies which matrix is involved.)
I would be very grateful for a tip, a book recommendation and of course a code.
With best regards
Felix
Translated with www.DeepL.com/Translator (free version)
Names generally no meaning in a compiled program, it doesn't contain them.
If the identifiers are the same type you might be able to define an array of pointers to them, and then iterate using that array.
This can be handled with an enumerated type. The solution will require a little reorganization of your date, but I think it will be worth it. Like most languages variables are independent of each other. If you want to deal with a list of variables they will need to be in a structure of some sort. An array in your case makes sense. However, associating a numeric index with each variable is a bother and requires pointers or some difficult to maintain parsing system.
Nicklaus Wirth created a mechanism to deal with this sort of problem. It is called an enumerated type. For example:
type
BeName = (S1Be, S2Be, S3Be, S4Be);
var
NFLOW: array[BeName] of integer;
MXTSTOdata: array[BeName,1:5,1:10] of integer;
SnBe: BeName;
begin
… Initialization here, the following is how to change your code.
MXTSTOdata(S2Be,1,NFLOW[S2Be]) := 22;
MXTSTOdata(S1Be,1,1) :=MXTSTOdata(S2Be,1,NFLOW[S4Be]);
… Here is just one for loop:
for f := 1 to NFLOW[S1Be]+1 do
MXTSTOdata[S1Be,1,f] := MXTSTOdata[S1Be,1,1];
… Here is a loop of for loops:
for SnBe := S1Be to S4Be do
for f := 1 to NFLOW[SnBe]+1 do
MXTSTOdata[SnBe,1,f] := MXTSTOdata[SnBe,1,1];
Note how the for loop doesn’t need a start and end index. But that depends on which Pascal you are using. Delphi, FreePascal, and standard Pascal differ. You might have to use the first and last element like I showed. You might have a Low and High function available.
var
NFLOW: array[BeName] of integer;
SnBe: BeName;
for SnBe := Low(BeName) to High(BeName) do
for f := 1 to NFLOW[SnBe]+1 do
MXTSTOdata[SnBe,1,f] := MXTSTOdata[SnBe,1,1];
And I might have the syntax for the array declaration wrong. I’ve seen var NFLOW: array of [BeName] of integer; documented on the web, built I haven’t fired up my pascal compiler to check this fragment. However, the enumeration type would help you a lot. Also, there is a for-in construct in FreePascal
for SnBe in BeName do
for f := 1 to NFLOW[SnBe]+1 do
MXTSTOdata[SnBe,1,f] := MXTSTOdata[SnBe,1,1];
The enumeration type is useful in preventing bothersome minor spelling errors from messing up the program, has nice for loop options, and can change the ordering of the values in the enumeration. Changing the ordering may be needed for handling what order assignments are made in, at the cost of program fragility, as you might imagine.
The pred and succ operators are implemented. If you ever wondered what the need were for pred and succ you have now found out. A while loop:
SnBe := S1Be;
repeat
… something
SnBe := succ(SnBe)
until SnBe = S4Be;
Of course that doesn’t get easily to the last value in the enumeration. You could add guard values, but that adds some confusion and messes up the for-in loop.
SnBe := S1Be;
repeat
SomefunctionF(SnBe);
SnBe := succ(SnBe)
until SnBe = S4Be;
SomefunctionF(S4Be);
Is probably the cleanest way to deal with the problem of running in a repeat loop. The reason for adding these examples is you may have two enumerations running in parallel:
type
ToBeName = (S1Be, S2Be, S3Be, S4Be);
NotToBeName = (Bob, Carol, Ted, Alice);
var
NFLOW: array[BeName] of integer;
MXTSTOdata: array[BeName,1:5,1:10] of integer;
Romeo: NotToBeName;
SnBe: BeName;
begin
SnBe:=S1Be;
Romeo:=Bob;
Repeat
ActionFunction(SnBe,Romeo);
SnBe := succ(SnBe);
Romeo := succ(Romeo)
until SnBe = Alice;
Also, this idea might be helpful for your program:
type
EType = (S1Be, S2Be, S3Be, S4Be, Bob, Carol, Ted, Alice);
var
Romeo: EType;
SnBe: EType;
begin
Romeo:= Bob;
SnBe:=S1Be;
repeat
SomeFn(Romeo,SnBe);
SnBe:=succ(SnBe);
until SnBe>S4Be;
The range check applies to pred and succ. For example, the statement
succ(Alice)
would produce an error because there is no element after Alice in the enumerations above.
Lastly, if you need to do things in reverse order you can do:
for SnBe := S4Be downto S1Be do

How to check array length of dynamic records in Delphi?

In an array of type TArrayParams that stores records of type TParams, according to the code below, there is a string field that can vary from 1 to 1000. How to check the length (in bytes) of memory that TArrayParams will have?
TParams = record
S: String; // Length may vary
I: Integer;
end;
TArrayParams = TArray<TParams>;
var arr: TArrayParams;
i: Integer;
begin
SetLength(arr, 2);
for i := 0 to 1 do
begin
arr[i].S := 'my random string.... xyz up to (maybe) 1000';
arr[i].I := i;
end;
// What is the length (in bytes) of the "arr" array's memory space?
SizeOf(arr) ??? It Doesn't work...
Length(arr) ??? It Doesn't work...
ByteLength(arr) ?? It Doesn't work...
end;
In the comments you were already told that strings are not stored in the records, so you can't simply use stream.write(record, SizeOf(Record)) to save such a record because that would only write a pointer to the stream that is not valid outside the context of your program.
So, you need some other way of storing the data. One option would be text:
Since the record only contains a number and a string, you could simply write them as text:
var
sl: tstringlist;
begin
sl := tstringlist.create;
try
for i := Low(arr) to High(arr) do begin
sl.Add(Format('%d'#9'%s', [arr[i].I, arr[i].s]));
end;
sl.SaveToFile('c:\path\to\filename.txt');
finally
sl.Free;
end;
end;
Please note that this code is completely untested!
The resulting file could be loaded into a text editor or e.g. OpenOffice Calc for display purposes.
If text (actually csv with tab as column separator) is not what you want, you need to be more specific in your question.

Plotting multiple arrays in maple

I'm trying to plot 4 data-sets in a single coordinate-system. I can plot every single one of them by themselves. However when I try to plot 2 or more with the same plot function I get an error. I can't use lists since I want to scale the arrays up to 2*10000, which a list can't. I'm using Maple 18.
Can anybody please help me solve this?
This is my code:
Here is a plotted data-set:
Here is the error I get, when trying to plot multiple sets(note I have tried using {} instead of []):
Your problem is that your use of pair and zip is not going to produce Arrays P[i] whose layout is valid for plotting. (Perhaps you cribbed that bit of code from something which was intended to produce a list of lists instead of an Array.)
Instead, you could construct the P[i] as iterator-by-2 Matrices (ie. 2-column Matrices).
One way:
restart;
mu := 2.5:
iterator := 25:
fcn := proc(a)
local F,i;
F := Array(1..iterator);
F[1] := a;
for i from 2 by 1 to iterator do
F[i] := mu*F[i-1]*(1-F[i-1]);
end do;
F;
end proc:
f[1]:=fcn(0.01):
f[2]:=fcn(0.1):
f[3]:=fcn(0.3):
f[4]:=fcn(0.5):
x := Array([seq(i,i=1..iterator)]):
for i from 1 to 4 do
P[i] := <x,f[i]>^%T:
end do:
plot([P[1],P[2],P[3],P[4]]);
Another (similar) way:
restart;
mu := 2.5:
iterator := 25:
fcn := proc(a)
local F,i;
F := Vector(1..iterator);
F[1] := a;
for i from 2 by 1 to iterator do
F[i] := mu*F[i-1]*(1-F[i-1]);
end do;
F;
end proc:
f[1]:=fcn(0.01):
f[2]:=fcn(0.1):
f[3]:=fcn(0.3):
f[4]:=fcn(0.5):
x := Vector([seq(i,i=1..iterator)]):
for i from 1 to 4 do
P[i] := <x|f[i]>:
end do:
plot([P[1],P[2],P[3],P[4]]);
You could alternatively use the command plots:-listplot directly on your f[i], though you'd likely way to also specify different colors for each so that it looked nice when you used plots:-display to render them all together.
I leave aside considerations of performance. There are ways to do all this and get faster computation. I deliberately keep your basic methodology.

Reorder array in pascal, descending

couldn't find similar posts, so posting my own question.
I got variable array of real:
price = array([58.9],[38.7],[8.95],[28.3])
I need to order it descending, with my code everything works well until last value of the array, I know even why, but can't find solution on my own. Anyway here's the code:
Procedure orderarray;
Var i,dz, j: Integer;
c :real;
v :string[25];
Begin
dz := 1;
For i := 1 to 3 do
Begin
For j:=i+1 to 4 do
if price[j]>price[dz] //searches for highest value in the array
then dz:=j;
c:=price[i]; price[i] := price[dz]; price[dz] := c; //switches current value with highest
End;
I've found solution to my own problem. Posting it just in case anyone will need it. I needed to reset dz to i, not j
Procedure orderarray;
Var i,dz, j: Integer;
c :real;
v :string[25];
Begin
For i := 1 to 3 do
Begin
dz:=i;
For j:=i+1 to 4 do
if price[j]>price[dz] //searches for highest value in the array
then dz:=j;
c:=price[i]; price[i] := price[dz]; price[dz] := c; //switches current value with highest
End;
Thank you everybody for your help, wouldn't get to the solution without you, anyway.
you need to reset dz to j everytime
for j:=i+1 to 4 do
begin
dz := j;
if price[j]>price[dz]
...;
c:=price[i]; ...;
end
In your sample dz will remain always 1, last element will never be moved since it is less then the first.

Best way to swap two records in TDataset Delphi?

New to delphi and database programming in general but am curious if there is a better way to swap records in a TDataset? I have read through some help and cant find any obvious methods. Currently I have a procedure implemented to move records down the dataset until they hit the Eof marker. However I am getting some odd errors when I get to the last record in my data. All I have is implemented a standard array-style swap routine trying to preserve data and whatnot while juggling active records.
Code So Far
procedure TForm2.btnDownClick(Sender: TObject);
var
sTmp,sTmp2 : string;
iTmp,iTmp2 : integer;
begin
tblMatched.DisableControls;
if ( tblMatched.Eof <> true ) then
begin
// Grab data to swap
tblMatched.GotoBookmark( tblMatched.GetBookmark );
iTmp := tblMatched.Fields[0].AsInteger;
sTmp := tblMatched.Fields[1].AsString;
tblMatched.Next;
iTmp2 := tblMatched.Fields[0].AsInteger;
sTmp2 := tblMatched.Fields[1].AsString;
// Swap data
tblMatched.Prior;
tblMatched.Edit;
tblMatched.Fields[0].Value := iTmp2;
tblMatched.Fields[1].Value := sTmp2;
tblMatched.Next;
tblMatched.Edit;
tblMatched.Fields[0].AsInteger := iTmp;
tblMatched.Fields[1].AsString := sTmp;
end;
tblMatched.EnableControls;
end;
It looks like you're using an in-memory dataset, such as TClientDataset. If you simply put an index on the dataset, it will keep things ordered for you so you don't have to rearrange them manually. Just set up the index based on whatever criteria you want to have it use.

Resources