I'm trying to iterate over 1000 keys of an entity at a time using this code but unlike Query, QueryKeys does not have a startAt() method for me to set it's cursor:
QueryKeys<Car> queryKeys = ofy().load().type(Car.class)
.limit(1000)
.keys();
// queryKeys has no startAt()
queryKeys = queryKeys.startAt(cursor)
Is there a way to loop through keys with QueryKeys just like Query?
The reason I want to loop through keys is that I need to delete those entities from the Datastore. I might be deleting 100k - 1 million entities of a single entity type and wanted to do it in chunks because I am afraid that loading that many keys may slow things down too much or maybe error out somehow.
You'll want to set the cursor before calling keys, e.g. objectify.get().load().type(Car.class).setStartCursor(...).limit(1000).keys().
Also, you are best to use the cursor from the previous query instead of relying on starting the query again to avoid skipping over tombstones as noted at https://cloud.google.com/datastore/docs/best-practices#deletions .
If you delete a page synchronously, there's no need for pagination:
public void deleteAllEntities() {
boolean continueDeletionLoop = true;
while (continueDeletionLoop) {
QueryKeys<Car> queryKeys = objectify.get().load().type(Car.class)
.limit(1000)
.keys();
if (queryKeys.list().isEmpty()) {
continueDeletionLoop = false;
} else {
objectify.get().delete().keys(queryKeys).now();
}
}
}
I am new to Apex and I’m struggling with creating a class to help me with some data analysis. I have data from a 3rd party (transactions__C) that has a field (fin_acct_txt__c) that is the pointer to another object (fin_accounts__C). I want to updated transactions__c with the id from fin_accounts__C into the lookup field transactions__c.fin_acct__c.
I want to do this in a class versus a trigger as there would be thousands of records loaded from the 3rd party on a monthly basis. I think doing this in bulk would be more efficient.
My thought is I create a list for transactions__c and a map for fin_accounts__c. Using the fin_acct_txt__c=fin_accounts__c.name I would be able to get the fin_accounts__c.id and update the transactions__c.fin_acct__c with that data.
But being new to Apex seems to be causing me some problems that I’m unsure how to resolve.
Here’s a copy of what I’ve done to date:
public class updateTxnFinAcctID {
// Build map of financial accts since that is unique
map<string ,fin_acct__c> finAccts = new map<string, fin_acct__c>
([select id,name from fin_acct__c where name!=null]);
//Iterate through the map to find the id to update the transactions
{
for(fin_acct__c finAcct: finAccts.values())
{
if (finAcct.name != Null)
{
finAccts.put(finAcct.name, finAcct);
}
// Find all records in transaction__c where fin_acct__c is null
//and the pointer is the name in the map
list<Transaction__c> txns =[
select id,fin_acct_txt__c from Transaction__c where fin_acct__c = null
and fin_acct_txt__c=:finaccts[0].name];
//create the list that will be used to update the transaction__c
list <Transaction__c> txnUpdate = new list <Transaction__c>();
{
//Find the id from fin_acct__c where name = fin_acct_txt__c
for (Transaction__c txn: txns){
finacct[0].Id =txn.fin_acct__c;
txnUpdate.add(txn);
}
//3. Update transaction with ID
}
}
// if (txnUpdate.size()>0 { update txnUpdate};
system.debug(txnUpdate.size());
}
}
I seem to be in a doom loop. The error I get is “Expression must be a list type: Map” pointing to the list txns = [ …]. But as that is not unique, it must be list. But I would believe I’ve got something structurally wrong here and that is a symptom of a larger issue.
Thanks.
I tried to understand what should to do your code, and I have a few tips, possibly they help to solve your issue:
1) In first loop over values of map finAccts you really don't need validation with if (finAcct.name != Null), because you already add it in SOQL query (where name!=null).
2) It's a bad practice - to put as a key to map different entities (for example, Ids and Names). I mean that when you queried fin_acct__c into the map finAccts, keys of the map are Ids of fin_acct__c. And then in the first loop you put in the same map the same objects only using their names as a key. If you really need such map with names as a keys is better to create new map and put the data there.
3) You execute SOQL query to Transaction__c object into the loop. It is likely to be the cause of an exception related to the SF limits (Especially if you are sure that the code will handle large amounts of data). Better collect all fin_acct__c names in list and move SOQL query out from the loop, using IN instead of = in where condition.
If I understood correctly that fin_acct_txt__c field contains names, not Ids, your class should looks something like:
public class updateTxnFinAcctID {
Map<String ,fin_acct__c> finAccts = new Map<String, fin_acct__c>
([select id,name from fin_acct__c where Name != null]);
Map<String, fin_acct__c> finAcctByNames = new Map<String, fin_acct__c>();
for(fin_acct__c finAcct: finAccts.values()){
finAcctByNames.put(finAcct.Name, finAcct);
}
List<Transaction__c> txns =[select id, fin_acct_txt__c, fin_acct__c
from Transaction__c where fin_acct__c = null
and fin_acct_txt__c IN finAcctByNames.keySet()];
List <Transaction__c> txnUpdate = new List<Transaction__c>();
for (Transaction__c txn: txns){
fin_acct__c relatedFinAcct = finAcctByNames.get(txn.fin_acct_txt__c);
if(relatedFinAcct != null){
txn.fin_acct__c = relatedFinAcct.Id;
txnUpdate.add(txn);
}
}
if(!txnUpdate.isEmpty()){
update txnUpdate;
system.debug(txnUpdate.size());
}
}
It possibly can contains some spelling mistakes, but it's a common idea.
In an architecture where objects have many complex relationships, what are some maintainable approaches to dealing with
Resolving Dependencies
Optimistic Updates
in react applications?
For example, given this type of schema:
```
type Foo {
...
otherFooID: String,
bars: List<Bar>
}
type Bar {
...
bizID: String,
}
type Biz {
...
}
```
A user might want to save the following ->
firstBiz = Biz();
secondBiz = Biz();
firstFoo = Foo({bars: [Bar({biz: firstBiz})]
secondFoo = Foo({bars: [Bar({biz: secondBiz})] otherFooId: firstFooId.id})
First Problem: Choosing real ids
The first problem with above is having the correct id. i.e in order for secondFoo to save, it needs to know the actual id of firstFoo.
To solve this, we could make the tradeoff, of letting the client choose the id, using something like a uuid. I don't see anything terribly wrong this this, so we can say this can work
Second Problem: Saving in order
Even if we determine id's from the frontend, the server still needs to receive these save requests in order.
```
- save firstFoo
// okay. now firstFoo.id is valid
- save secondFoo
// okay, it was able to resolve otherFooID to firstFoo
```
The reasoning here is that the backend must guarantee that any id that is being referenced is valid.
```
- save secondFoo
// backend throws an error otherFooId is invalid
- save firstfoo
// okay
```
I am unsure what the best way to attack this problem is
The current approaches that come to mind
Have custom actions, that do the coordination via promises
save(biz).then(_ => save(Bar).then(_ => save(firstFoo)).then(_ => save(second)
The downside here is that it is quite complex, and the number of these kinds of combinations will continue to grow
Create a pending / resolve helper
const pending = {}
const resolve = (obj, refFn) => {
return Promise.all(obj, refFn(obj));
}
const fooRefs = (foo) => {
return foo.bars.map(bar => bar.id).concat(foo.otherFooId);
}
pending[firstFoo].id = resolve(firstFoo, fooRefs).then(_ => save(firstFoo))
```
The problem with 2. is that it can cause a bunch of errors easily, if we forget to resolve or to add to pending.
Potential Solutions
It seems like Relay or Om next can solve these issues, but i would like something less high power. Perhaps something that can work in with redux, or maybe it's some concept I am missing.
Thoughts much appreciated
I have a JS/PHP implementation of such a system
My approach is to serialize records both on the client and server using a reference system
For example unsaved Foo1 has GUID eeffa3, and a second Foo references its id key as {otherFooId: '#Foo#eeffa3[id]' }
Similarily you can reference a whole object like this
Foo#eefa3:{bars['#Baz#ffg4', '#Baz#ffg5']}
Now the client-side serializer would build a tree of relations and model attributes like this
{
modelsToSave:[
'Foo#effe3':{
attribs:{name:'John', title:'Mr.'},
relations:{bars:['#Bar#ffg4']}
},
'Bar#ffg4':{
attribs:{id:5}
relations:{parentFoo:'#Foo#effe3'}
},
]
}
As you can see in this example I have described circular relations between unsaved objects in pure JSON.
The key here is to hold these "record" objects in client-side memory and never mutate their GUID
The server can figure out the order of saving by saving first records without "parent" dependencies, then records which depend on those parents
After saving, the server wil return the same reference map, but now the attribs will also include primary keys and foreign keys
JS walks the received map twice (first pass just update server-received attributes, second pass substitute record references and attribute references to real records and attributes).
So there are 2 mechanisms for referencing a record, a client-side GUID and a server-side PK
When receiving a server JSON, you match your GUID with the server primary key
In my db scheme, I need a autoincrement primary key. How I can realize this feature?
PS For access to DynamoDB, I use dynode, module for Node.js.
Disclaimer: I am the maintainer of the Dynamodb-mapper project
Intuitive workflow of an auto-increment key:
get the last counter position
add 1
use the new number as the index of the object
save the new counter value
save the object
This is just to explain the underlying idea. Never do it this way because it's not atomic. Under certain workload, you may allocate the same ID to 2+ different objects because it's not atomic. This would result in a data loss.
The solution is to use the atomic ADD operation along with ALL_NEW of UpdateItem:
atomically generate an ID
use the new number as the index of the object
save the object
In the worst case scenario, the application crashes before the object is saved but never risk to allocate the same ID twice.
There is one remaining problem: where to store the last ID value ? We chose:
{
"hash_key"=-1, #0 was judged too risky as it is the default value for integers.
"__max_hash_key__y"=N
}
Of course, to work reliably, all applications inserting data MUST be aware of this system otherwise you might (again) overwrite data.
the last step is to automate the process. For example:
When hash_key is 0:
atomically_allocate_ID()
actual_save()
For implementation details (Python, sorry), see https://bitbucket.org/Ludia/dynamodb-mapper/src/8173d0e8b55d/dynamodb_mapper/model.py#cl-67
To tell you the truth, my company does not use it in production because, most of the time it is better to find another key like, for the user, an ID, for a transaction, a datetime, ...
I wrote some examples in dynamodb-mapper's documentation and it can easily be extrapolate to Node.JS
If you have any question, feel free to ask.
Another approach is to use a UUID generator for primary keys, as these are highly unlikely to clash.
IMO you are more likely to experience errors consolidating primary key counters across highly available DynamoDB tables than from clashes in generated UUIDs.
For example, in Node:
npm install uuid
var uuid = require('uuid');
// Generate a v1 (time-based) id
uuid.v1(); // -> '6c84fb90-12c4-11e1-840d-7b25c5ee775a'
// Generate a v4 (random) id
uuid.v4(); // -> '110ec58a-a0f2-4ac4-8393-c866d813b8d1'
Taken from SO answer.
If you're okay with gaps in your incrementing id, and you're okay with it only roughly corresponding to the order in which the rows were added, you can roll your own: Create a separate table called NextIdTable, with one primary key (numeric), call it Counter.
Each time you want to generate a new id, you would do the following:
Do a GetItem on NextIdTable to read the current value of Counter --> curValue
Do a PutItem on NextIdTable to set the value of Counter to curValue + 1. Make this a conditional PutItem so that it will fail if the value of Counter has changed.
If that conditional PutItem failed, it means someone else was doing this at the same time as you were. Start over.
If it succeeded, then curValue is your new unique ID.
Of course, if your process crashes before actually applying that ID anywhere, you'll "leak" it and have a gap in your sequence of IDs. And if you're doing this concurrently with some other process, one of you will get value 39 and one of you will get value 40, and there are no guarantees about which order they will actually be applied in your data table; the guy who got 40 might write it before the guy who got 39. But it does give you a rough ordering.
Parameters for a conditional PutItem in node.js are detailed here. http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/frames.html#!AWS/DynamoDB.html. If you had previously read a value of 38 from Counter, your conditional PutItem request might look like this.
var conditionalPutParams = {
TableName: 'NextIdTable',
Item: {
Counter: {
N: '39'
}
},
Expected: {
Counter: {
AttributeValueList: [
{
N: '38'
}
],
ComparisonOperator: 'EQ'
}
}
};
For those coding in Java, DynamoDBMapper can now generate unique UUIDs on your behalf.
DynamoDBAutoGeneratedKey
Marks a partition key or sort key property as being auto-generated.
DynamoDBMapper will generate a random UUID when saving these
attributes. Only String properties can be marked as auto-generated
keys.
Use the DynamoDBAutoGeneratedKey annotation like this
#DynamoDBTable(tableName="AutoGeneratedKeysExample")
public class AutoGeneratedKeys {
private String id;
#DynamoDBHashKey(attributeName = "Id")
#DynamoDBAutoGeneratedKey
public String getId() { return id; }
public void setId(String id) { this.id = id; }
As you can see in the example above, you can apply both the DynamoDBAutoGeneratedKey and DynamoDBHashKey annotation to the same attribute to generate a unique hash key.
Addition to #yadutaf's answer
AWS supports Atomic Counters.
Create a separate table (order_id) with a row holding the latest order_number:
+----+--------------+
| id | order_number |
+----+--------------+
| 0 | 5000 |
+----+--------------+
This will allow to increment order_number by 1 and get the incremented result in a callback from AWS DynamoDB:
config={
region: 'us-east-1',
endpoint: "http://localhost:8000"
};
const docClient = new AWS.DynamoDB.DocumentClient(config);
let param = {
TableName: 'order_id',
Key: {
"id": 0
},
UpdateExpression: "set order_number = order_number + :val",
ExpressionAttributeValues:{
":val": 1
},
ReturnValues: "UPDATED_NEW"
};
docClient.update(params, function(err, data) {
if (err) {
console.log("Unable to update the table. Error JSON:", JSON.stringify(err, null, 2));
} else {
console.log(data);
console.log(data.Attributes.order_number); // <= here is our incremented result
}
});
🛈 Be aware that in some rare cases their might be problems with the connection between your caller point and AWS API. It will result in the dynamodb row being incremented, while you will get a connection error. Thus, there might appear some unused incremented values.
You can use incremented data.Attributes.order_number in your table, e.g. to insert {id: data.Attributes.order_number, otherfields:{}} into order table.
I don't believe it is possible to to a SQL style auto-increment because the tables are partitioned across multiple machines. I generate my own UUID in PHP which does the job, I'm sure you could come up with something similar like this in javascript.
I've had the same problem and created a small web service just for this purpose. See this blog post, that explains how I'm using stateful.co with DynamoDB in order to simulate auto-increment functionality: http://www.yegor256.com/2014/05/18/cloud-autoincrement-counters.html
Basically, you register an atomic counter at stateful.co and increment it every time you need a new value, through RESTful API. The service is free.
Auto Increment is not good from performance perspective as it will overload specific shards while keeping others idle, It doesn't make even distribution if you're storing data to Dynamodb.
awsRequestId looks like its actually V.4 UUID (Random), code snippet below to try it:
exports.handler = function(event, context, callback) {
console.log('remaining time =', context.getRemainingTimeInMillis());
console.log('functionName =', context.functionName);
console.log('AWSrequestID =', context.awsRequestId);
callback(null, context.functionName);
};
In case you want to generate this yourself, you can use https://www.npmjs.com/package/uuid or Ulide to generate different versions of UUID based on RFC-4122
V1 (timestamp based)
V3 (Namespace)
V4 (Random)
For Go developers, you can use these packages from Google's UUID, Pborman, or Satori. Pborman is better in performance, check these articles and benchmarks for more details.
More Info on Universal Unique Identifier Specification could be found here.
Create the new file.js and put this code:
exports.guid = function () {
function _p8(s) {
var p = (Math.random().toString(16)+"000000000").substr(2,8);
return s ? "-" + p.substr(0,4) + "-" + p.substr(4,4) : p ;
}
return (_p8() + _p8(true) + _p8(true)+new Date().toISOString().slice(0,10)).replace(/-/g,"");
}
Then you can apply this function to the primary key id. It will generate the UUID.
Incase you are using NoSQL DynamoDB then using Dynamoose ORM, you can easily set default unique id. Here is the simple user creation example
// User.modal.js
const dynamoose = require("dynamoose");
const userSchema = new dynamoose.Schema(
{
id: {
type: String,
hashKey: true,
},
displayName: String,
firstName: String,
lastName: String,
},
{ timestamps: true },
);
const User = dynamoose.model("User", userSchema);
module.exports = User;
// User.controller.js
const { v4: uuidv4 } = require("uuid");
const User = require("./user.model");
exports.create = async (req, res) => {
const user = new User({ id: uuidv4(), ...req.body }); // set unique id
const [err, response] = await to(user.save());
if (err) {
return badRes(res, err);
}
return goodRes(res, reponse);
};
Instead of using UUID use KSUID for ids. Naturally ordered by generation time.
https://www.npmjs.com/package/ksuid?activeTab=readme
I have a simple textfiled that I use for searching in my grid by getting the inserted value and adding filter to the store like this:
instructionSearchField: function(field, e) {
if (e.getKey() == e.ENTER) {
var searchTxt = this.getValue();
this.recordsStore = Ext.data.StoreManager.lookup('Instructions');
this.recordsStore.clearFilter(true);
this.recordsStore.filter({
filterFn: function(item) {
return item.get('title').toLowerCase().indexOf(searchTxt.toLowerCase()) != -1;
}
});
this.recordsStore.load();
}
},
And thus the user can serch as long as he wants - it works just as it has to. The problem is that at some point I want to return the state of the grid as it was before filtering/serching. Loading it from the server is not that easy so I wonder is there a way in ExtJS4 to return the begining state of a store, when local filter like the one above was used multiple times on it? The simple way for me would be to just return the basic state of the store adn then just load it, but I haven't found nothing like this in the documentation so - is there a way to do this?
Thanks
Leron
If you don't use remote filtering (as in your server filters records and returns only matching set) you don't need to do load() after applying filters. Here is basic workflow:
store.load() - after that your local copy has all records from the database
store.filter(...) - you store still has all records in the memory but you have access to only ones that match your filter
store.clearFilter() - you store again shows all records and you can apply different filter again (note that load() operation was performed only once)