I have a mongoose object which contains an array of ObjectIds, being used for population from another table. I want to be able to dedupe these. eg I have
[ '61e34f3293d9361bbb5883c7' ,'61e34f3293d9361bbb5883c7', '61e34f3293d9361bbb5883c7' ]
When i print and iterate through these they look like strings.
But they also have an _id property, so I think they're somehow "populated" or at least contain references to the child table.
What's the best way to do this? I tried:
const uniqueTokens = _.uniqBy(tokens, '_id') which doesn't seem to work as _id is some kind of Object.
converting to a string will allow me to dedupe:
const tokens = this.tokens || []
let newTokens: string[] = []
for (let t of tokens) {
const text = t.toString()
// clog.info('t', t, t._id, typeof t._id)
if (!newTokens.includes(text)) {
newTokens.push(text)
}
}
but then these aren't real Objects I can assign back to the original parent object.
// this.tokens = newTokens
await this.save()
I could maybe go through and re-find the objects, but that seems to be digging deeper into the hole!
Seems there must be a better way to handle these type of types...
related searches
How to compare mongoDB ObjectIds & remove duplicates in an array of documents using node.js?
I also tried using lean() on the tokens array to try and convert it back to a simple list of references, in case somehow the 'population' could be undone to help.
I'm down to creating a unique signature field for the referenced items and de-duping based on that.
I've a REST api that returns the list of locales as dictionary:
{
"en-uk": "English UK",
"en-us": "English USA",
...
}
This dictionary is correctly ordered alphabetically by value.
When AngularJS receives it via HTTP, the dictionary gets automatically re-sorted by key, so when I bind to a select element the list of options is ordered by key, and the alphabetical order by key doesn't match the one by value, I get a wrong sorting.
The problem I suppose is due to the fact that such dictionary becomes basically one object with 800+ properties. How do I sort it by value?
First: You have to find all keys.
Second: Iterate all the keys.
Third: Then sort the array with values.
Please use the following:
let obj = {
"en-us": "English USA",
"en-uk": "English UK"
};
// Get an array of the keys:
let keys = Object.keys(obj);
// Then sort by using the keys to lookup the values in the original object:
keys.sort(function(a, b) { return obj[a] > obj[b] });
console.log(keys);
console.log(obj[keys[0]]);
You can modify the way you send the response from the server. Instead of sending the response as an object, send the stringified object.
The problem is indeed you cannot sort the values of the properties of an object. So I convert it to an array before binding it:
So,
languageResource.getCultures().then(function(cultures) {
vm.availableCultures = cultures;
});
becomes
languageResource.getCultures().then(function (culturesDictionary) {
var cultures = [];
angular.forEach(culturesDictionary, function (value, key) {
cultures.push({
lcid: key,
name: value
});
});
vm.availableCultures = cultures;
});
Seen this when the key is numerical. If the key's data type is string than it would keep its sorted state after an API call. If the key's data type is numerical, than, you would need set the key's value as a string and even add single quotes before and after the key's value, before the API sends it back.
I haven't tried the approach to stringfy the dictionary in the API. After the call you would parse the string back to an object with something like JSON.parse(string) might be your best bet.
I am trying to make a Meteor app to let users push a value to the database. It works ok, but there a small issue. As soon a certain user has pushed his information, i don't want to let the same user create another entry. Or this must be blocked, or the value the user is pushing must be overwritten for the value he is posting the second time. Now I get multiple entry's of the same user.
Here is my code. Hope you can help me here. Thanks in advance.
Estimations.update(userstory._id, {
$addToSet: {
estimations: [
{name: Meteor.user().username, estimation: this.value}
]
}
});
From the mongo docs
The $addToSet operator adds a value to an array unless the value is
already present, in which case $addToSet does nothing to that array.
Since your array elements are objects the value is the entire object, not just the username key. This means a single user can create multiple name, estimation pairs as long as the estimation value is different.
What you can do is remove any value for the user first, then reinsert:
var username = Meteor.user().username;
Estimations.update({ userstory._id },
{ $pull: { estimations: { name: username }}}); // if it doesn't exist this will no-op
Estimations.update({userstory._id },
{ $push: { estimations: { name: username, estimation: this.value }}});
By way of commentary, you've got a collection called Estimations that contains an array called estimations that contains objects with keys estimation. This might confuse future developers on the project ;) Also if your Estimations collection is 1:1 with UserStorys then perhaps the array could just be a key inside the UserStory document?
Suppose I have 2 Schema's in Mongoose that look like this:
var movieSchema = mongoose.Schema({
name: String,
type: String
});
var moviePlaylistSchema = mongoose.Schema({
name: String,
movies: [{type: mongoose.Schema.Types.ObjectId, ref: 'Movie'}]
});
var Movie = mongoose.model('Movie', movieSchema);
var MoviePlaylist = mongoose.model('MoviePlaylist', moviePlaylistSchema);
If a query was made along the following lines:
MoviePlaylist.find({}).populate('movies').exec(function(err, res) {
if (err) console.log('err', err);
else {
console.log('res', res);
res.forEach(function(elem, index) {
console.log('elem.name', elem.name);
});
}
});
Would the order of the elements in the array be maintained? The objective here is to allow the user to maintain a playlist order of their movies. If, when the "populate" method fires, the array order of Movie object Ids is not maintained, then this will not serve my purpose. Hence thought I'd ask someone who is more knowledgeable in this area.
If this works, then I have another task which is allowing the user to change the order of movies in the playlist, which should be straight forward by allowing the movie object id index to be swapped in the array.
Thanks for your help in advance.
MongoDB will keep the order of the array, much like an array in any programming language.
You can view the BSON/JSON spec for reference which highlights that the array must contain integer values for keys, and be maintained in ascending numerical order.
Additionally, the Mongoose populate on an array works by calling Model.populate via forEach on each element of the array. This modifies the array in place, hence the order is preserved. You can see the relevant source code here.
In my db scheme, I need a autoincrement primary key. How I can realize this feature?
PS For access to DynamoDB, I use dynode, module for Node.js.
Disclaimer: I am the maintainer of the Dynamodb-mapper project
Intuitive workflow of an auto-increment key:
get the last counter position
add 1
use the new number as the index of the object
save the new counter value
save the object
This is just to explain the underlying idea. Never do it this way because it's not atomic. Under certain workload, you may allocate the same ID to 2+ different objects because it's not atomic. This would result in a data loss.
The solution is to use the atomic ADD operation along with ALL_NEW of UpdateItem:
atomically generate an ID
use the new number as the index of the object
save the object
In the worst case scenario, the application crashes before the object is saved but never risk to allocate the same ID twice.
There is one remaining problem: where to store the last ID value ? We chose:
{
"hash_key"=-1, #0 was judged too risky as it is the default value for integers.
"__max_hash_key__y"=N
}
Of course, to work reliably, all applications inserting data MUST be aware of this system otherwise you might (again) overwrite data.
the last step is to automate the process. For example:
When hash_key is 0:
atomically_allocate_ID()
actual_save()
For implementation details (Python, sorry), see https://bitbucket.org/Ludia/dynamodb-mapper/src/8173d0e8b55d/dynamodb_mapper/model.py#cl-67
To tell you the truth, my company does not use it in production because, most of the time it is better to find another key like, for the user, an ID, for a transaction, a datetime, ...
I wrote some examples in dynamodb-mapper's documentation and it can easily be extrapolate to Node.JS
If you have any question, feel free to ask.
Another approach is to use a UUID generator for primary keys, as these are highly unlikely to clash.
IMO you are more likely to experience errors consolidating primary key counters across highly available DynamoDB tables than from clashes in generated UUIDs.
For example, in Node:
npm install uuid
var uuid = require('uuid');
// Generate a v1 (time-based) id
uuid.v1(); // -> '6c84fb90-12c4-11e1-840d-7b25c5ee775a'
// Generate a v4 (random) id
uuid.v4(); // -> '110ec58a-a0f2-4ac4-8393-c866d813b8d1'
Taken from SO answer.
If you're okay with gaps in your incrementing id, and you're okay with it only roughly corresponding to the order in which the rows were added, you can roll your own: Create a separate table called NextIdTable, with one primary key (numeric), call it Counter.
Each time you want to generate a new id, you would do the following:
Do a GetItem on NextIdTable to read the current value of Counter --> curValue
Do a PutItem on NextIdTable to set the value of Counter to curValue + 1. Make this a conditional PutItem so that it will fail if the value of Counter has changed.
If that conditional PutItem failed, it means someone else was doing this at the same time as you were. Start over.
If it succeeded, then curValue is your new unique ID.
Of course, if your process crashes before actually applying that ID anywhere, you'll "leak" it and have a gap in your sequence of IDs. And if you're doing this concurrently with some other process, one of you will get value 39 and one of you will get value 40, and there are no guarantees about which order they will actually be applied in your data table; the guy who got 40 might write it before the guy who got 39. But it does give you a rough ordering.
Parameters for a conditional PutItem in node.js are detailed here. http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/frames.html#!AWS/DynamoDB.html. If you had previously read a value of 38 from Counter, your conditional PutItem request might look like this.
var conditionalPutParams = {
TableName: 'NextIdTable',
Item: {
Counter: {
N: '39'
}
},
Expected: {
Counter: {
AttributeValueList: [
{
N: '38'
}
],
ComparisonOperator: 'EQ'
}
}
};
For those coding in Java, DynamoDBMapper can now generate unique UUIDs on your behalf.
DynamoDBAutoGeneratedKey
Marks a partition key or sort key property as being auto-generated.
DynamoDBMapper will generate a random UUID when saving these
attributes. Only String properties can be marked as auto-generated
keys.
Use the DynamoDBAutoGeneratedKey annotation like this
#DynamoDBTable(tableName="AutoGeneratedKeysExample")
public class AutoGeneratedKeys {
private String id;
#DynamoDBHashKey(attributeName = "Id")
#DynamoDBAutoGeneratedKey
public String getId() { return id; }
public void setId(String id) { this.id = id; }
As you can see in the example above, you can apply both the DynamoDBAutoGeneratedKey and DynamoDBHashKey annotation to the same attribute to generate a unique hash key.
Addition to #yadutaf's answer
AWS supports Atomic Counters.
Create a separate table (order_id) with a row holding the latest order_number:
+----+--------------+
| id | order_number |
+----+--------------+
| 0 | 5000 |
+----+--------------+
This will allow to increment order_number by 1 and get the incremented result in a callback from AWS DynamoDB:
config={
region: 'us-east-1',
endpoint: "http://localhost:8000"
};
const docClient = new AWS.DynamoDB.DocumentClient(config);
let param = {
TableName: 'order_id',
Key: {
"id": 0
},
UpdateExpression: "set order_number = order_number + :val",
ExpressionAttributeValues:{
":val": 1
},
ReturnValues: "UPDATED_NEW"
};
docClient.update(params, function(err, data) {
if (err) {
console.log("Unable to update the table. Error JSON:", JSON.stringify(err, null, 2));
} else {
console.log(data);
console.log(data.Attributes.order_number); // <= here is our incremented result
}
});
🛈 Be aware that in some rare cases their might be problems with the connection between your caller point and AWS API. It will result in the dynamodb row being incremented, while you will get a connection error. Thus, there might appear some unused incremented values.
You can use incremented data.Attributes.order_number in your table, e.g. to insert {id: data.Attributes.order_number, otherfields:{}} into order table.
I don't believe it is possible to to a SQL style auto-increment because the tables are partitioned across multiple machines. I generate my own UUID in PHP which does the job, I'm sure you could come up with something similar like this in javascript.
I've had the same problem and created a small web service just for this purpose. See this blog post, that explains how I'm using stateful.co with DynamoDB in order to simulate auto-increment functionality: http://www.yegor256.com/2014/05/18/cloud-autoincrement-counters.html
Basically, you register an atomic counter at stateful.co and increment it every time you need a new value, through RESTful API. The service is free.
Auto Increment is not good from performance perspective as it will overload specific shards while keeping others idle, It doesn't make even distribution if you're storing data to Dynamodb.
awsRequestId looks like its actually V.4 UUID (Random), code snippet below to try it:
exports.handler = function(event, context, callback) {
console.log('remaining time =', context.getRemainingTimeInMillis());
console.log('functionName =', context.functionName);
console.log('AWSrequestID =', context.awsRequestId);
callback(null, context.functionName);
};
In case you want to generate this yourself, you can use https://www.npmjs.com/package/uuid or Ulide to generate different versions of UUID based on RFC-4122
V1 (timestamp based)
V3 (Namespace)
V4 (Random)
For Go developers, you can use these packages from Google's UUID, Pborman, or Satori. Pborman is better in performance, check these articles and benchmarks for more details.
More Info on Universal Unique Identifier Specification could be found here.
Create the new file.js and put this code:
exports.guid = function () {
function _p8(s) {
var p = (Math.random().toString(16)+"000000000").substr(2,8);
return s ? "-" + p.substr(0,4) + "-" + p.substr(4,4) : p ;
}
return (_p8() + _p8(true) + _p8(true)+new Date().toISOString().slice(0,10)).replace(/-/g,"");
}
Then you can apply this function to the primary key id. It will generate the UUID.
Incase you are using NoSQL DynamoDB then using Dynamoose ORM, you can easily set default unique id. Here is the simple user creation example
// User.modal.js
const dynamoose = require("dynamoose");
const userSchema = new dynamoose.Schema(
{
id: {
type: String,
hashKey: true,
},
displayName: String,
firstName: String,
lastName: String,
},
{ timestamps: true },
);
const User = dynamoose.model("User", userSchema);
module.exports = User;
// User.controller.js
const { v4: uuidv4 } = require("uuid");
const User = require("./user.model");
exports.create = async (req, res) => {
const user = new User({ id: uuidv4(), ...req.body }); // set unique id
const [err, response] = await to(user.save());
if (err) {
return badRes(res, err);
}
return goodRes(res, reponse);
};
Instead of using UUID use KSUID for ids. Naturally ordered by generation time.
https://www.npmjs.com/package/ksuid?activeTab=readme