I have a mongodb database that is storing time-series data. Let's assume this is the data that is already in there.
{
asset: "BTC/EUR",
timeframe: "5m",
stream: {O,H,L,C,V},
date: "2020-06-18T11:35:00+00:00"
}
{
asset: "BTC/EUR",
timeframe: "5m",
stream: {O,H,L,C,V},
date: "2020-06-18T11:40:00+00:00"
}
{
asset: "BTC/EUR",
timeframe: "5m",
stream: {O,H,L,C,V},
date: "2020-06-18T11:45:00+00:00"
}
As you can see, what matters here is the date field.
After my API call, I receive some new data, that might look like that.
{
asset: "BTC/EUR",
timeframe: "5m",
stream: {O,H,L,C,V},
date: "2020-06-18T11:35:00+00:00"
}
{
asset: "BTC/EUR",
timeframe: "5m",
stream: {O,H,L,C,V},
date: "2020-06-18T11:40:00+00:00"
}
{
asset: "BTC/EUR",
timeframe: "5m",
stream: {O,H,L,C,V},
date: "2020-06-18T11:45:00+00:00"
}
{
asset: "BTC/EUR",
timeframe: "5m",
stream: {O,H,L,C,V},
date: "2020-06-18T11:50:00+00:00"
}
So, all the other 3 entries are already in the db, I just would like to add the data that is not in the db, in our case, the last entry dated 2020-06-18T11:50:00+00:00 - or potentially fill in any gaps in the previous records.
Do I need to pull all the previous entries from the db and compare them to my latest payload, then filter out the existing ones and then make my db entry?
This is how my schema looks:
Related
We have two different approaches for an appointment scheduling system, using mongodb.
1st approach:
appointments:
{
resourceId: "string",
resourceType: "doc"/"nut"...,
userId: "string",
userName: "string",
startDate: "2020-05-18T16:00:00Z",
endDate: 2020-05-18T17:00:00Z
title: "string",
description: "string",
type: "string"/"off"
},
{
resourceId: "string",
resourceType: "doc"/"nut"...,
userId: "string",
userName: "string",
startDate: "2020-05-21T12:00:00Z",
endDate: 2020-05-21T12:30:00Z,
title: "string",
description: "string",
type: "string"/"off"
},
...
resources:
{
resourceId: "string",
resourceName: "string"
resourceType: "doc"/"nut"/"room",
autoApprove: true/false,
constantDaysOff: [sunnday]
},
{
resourceId: "string",
resourceName: "string"
resourceType: "doc"/"nut"/"room",
autoApprove: true/false,
constantDaysOff: [sunnday]
},
{
resourceId: "string",
resourceName: "string"
resourceType: "doc"/"nut"/"room",
autoApprove: true/false,
constantDaysOff: [sunnday]
}
Here appointments and resources are different collections, with sample documents in each collection.
2nd approach:
resources:
{
resourceId: "string",
resourceName: "string",
resourceType: "doc"/"nut"...,
constantDaysOff: [sunday],
2020-05-21: [
{
startDate: "2020-05-21T12:00:00Z",
endDate: 2020-05-21T12:30:00Z,
userId: "string",
userName: "string",
title: "string",
description: "string",
type: "string"/"off"
},
{
startDate: "2020-05-21T14:00:00Z",
endDate: 2020-05-21T14:30:00Z,
userId: "string",
userName: "string",
title: "string",
description: "string",
type: "string"/"off"
}
],
2020-05-22: [
{
startDate: "2020-05-22T12:00:00Z",
endDate: 2020-05-22T12:30:00Z,
userId: "string",
userName: "string",
title: "string",
description: "string",
type: "string"/"off"
},
{
startDate: "2020-05-22T14:00:00Z",
endDate: 2020-05-22T14:30:00Z,
userId: "string",
userName: "string",
title: "string",
description: "string",
type: "string"/"off"
}
]
...
}
Here we only have one collection, and appointment dates are keys in the collection. Each date key would contain multiple json objects representing different appointments on the same day.
NOTE:
There are no appointments longer than 1 day, the reason we have startDate and endDate is to calculate the length of the appointment, its start and end time.
We need to be able to perform queries, most efficiently, along the lines of:
get all appointments for a specific resource id
get appointments for a resource between two different dates
get appointments for a user between different dates
cancel/remove appointments
A resource could be anything, for example a doctor, coach, room...
So my question is which one would be more efficient/feasible when it comes to mongodb queries?
most articles said not to use start-Date and end-Date, but Date and start/end-Time (not even in Date format) - you mostly select (query) by day/days, then you will handle time
Time is more flexible, especially when you need some time-step between appointments. Also, you can easily edit the time long without changing the date or change the date without changing the time.
IMHO: according to the previous point, it's much better to use your first variant and the second one I dislike.
In the second variant, it will be hard to change the date. You will need to handle the whole array, upsert by date, and then upsert the whole Set of those.
I'm trying to mock a YAML API using JSON Wiremock for a PUT multipart/form-data. The multipart contains two arrays of metadata. How can I match a set of specific values (or regex) in each array, disregarding of the order.
We are bound to use YAML 2.0 (if you wonder), which is why we have these two arrays for the metadata. I've been able to successfully match specific values for the array (for example, for fileMetadataName, I can match "permissions,owner"), but I haven't found how to match the full set of potential values (5 values with all possible permutations, see YAML below).
Here is the JSON Wiremock file that can match one case of the array:
{
"request": {
"method": "PUT",
"urlPath": "/files",
"headers": {
"Content-Type": { "contains": "multipart/form-data"},
"Source": { "matches": "POC(.+)"}
},
"multipartPatterns": [
{
"headers": {
"Content-Disposition": {
"contains": "name=\"typeOfFile\""
}
},
"bodyPatterns": [ {
"matches": "PDF"
} ]
},
{
"headers": {
"Content-Disposition": {
"contains": "name=\"fileMetadataName\""
}
},
"bodyPatterns": [ {
"matches": "permissions,owner"
} ]
}
]
},
"response": {
"status": 201,
"jsonBody": {
"DocumentId": "123456789-123456789"
}
}
}
And here is an extract of the YAML that describes the multipart input:
paths:
'/files':
put:
tags:
- ProofOfConcept
summary: Upload a file in the files repository
description: Do the job
operationId: putFile
consumes:
- multipart/form-data
produces:
- application/json
parameters:
- name: Source
description: ID of the sender
in: header
type: string
required: true
- name: theFile
description: The file to be uploaded
in: formData
required: true
type: file
- name: typeOfFile
description: 'File type: PDF, JPG...'
in: formData
required: true
type: string
- name: fileMetadataName
description: 'Metadata name. Possible values are: permissions, owner, group, creationDate, appGeneratedId (format: <app-name>;<id>)'
in: formData
type: array
items:
type: string
- name: fileMetadataValue
description: Value associated to the corresponding metadata name
in: formData
type: array
items:
type: string
responses:
'201':
description: Created
schema:
$ref: '#/definitions/DocumentId'
I expect to be able to match for fileMetadataName, for example, all permutations of :
permissions, owner, group, creationDate, appGeneratedId
And in the case of fileMetadataValue, I expect to be able to match regex values for all permutations (e.g. ([0-9]{3,3}) for permissions).
In my sample document, I have a campaign document that contains the _id of the document and an importData array. importData is an array of objects containing a unique date and source value.
My goal is to have an object updated with a unique date/source pair. I would like to have the new object replace any matching object. In the example below, Fred may have originally donated a TV, but I want my application to update the object to reflect he donated both a TV and a radio.
// Events (sample document)
{
"_id" : "Junky Joe's Jubilee",
"importData" : [
{
"date": "2015-05-31",
"source": "Fred",
"items": [
{item: "TV", value: 20.00},
{item: "radio", value: 5.34}
]
},
{
"date": "2015-05-31",
"source": "Mary",
"items": [
{item: "Dresser", value: 225.00}
]
}
]
}
My original thought was to do something like the code below, but not only am I updating importData with Fred's donations, I'm also blowing away anything else in the importData array:
var collection = db.collection("events");
collection.update(
{_id: "Junky Joe's Jubilee",
importData: {
date: "2015-05-31",
source: 'Fred'
},
}, // See if we can find a campaign object with this name
{
$set:
{"importData":
{
date: "2015-05-31",
source: 'Fred',
items: [
{item: "TV", value: 20.00},
{item: "radio", value: 5.34}
]
}
}
},
{upsert: true}); // Create a document if one does not exist for this campaign
When I tried pushing (instead of $set), I was getting multiple entries for the date/source combos (e.g. Fred would appear to have donated two items multiple times on "2015-05-31").
How would I go about doing that with the MongoDB native driver and NodeJS?
Try this
var collection = db.collection("events");
collection.update(
{_id: "Junky Joe's Jubilee",
importData: {
date: "2015-05-31",
source: 'Fred'
},
}, // See if we can find a campaign object with this name
{
$set:
{"importData.$":
{
date: "2015-05-31",
source: 'Fred',
items: [
{item: "TV", value: 20.00},
{item: "radio", value: 5.34}
]
}
}
},
{upsert: true}); // Create a document if one does not exist for this campaign
According to the documentation under Array update operators this should only modify the first element in the array, which matches the query.
I want make a filter tag system with mongoose.
The principle is simple, I have a model that has an array of string and an array in input.
I want select elements which contains tags in the input array.
Example:
game_console [{
name: "Playsation",
tags: ["old", "sony"]
}, {
name: "xBox",
tags: ["nintendo", "powerfull"]
}, {
name: "megadrive",
tags: ["old", "sega"]
}, {
name: "Gameboy",
tags: ["prehistory", "sega"]
}];
If I put in array input ["old", "sega"], I must get the elements: Playstation and Megadrive.
After searching I found $in and I tested that:
Preambule.find({
'tags': {
$in: tags_array_input
}
}).sort('-commentaires_size')
.exec(function(err, things) {
// [...]
});
It works but only if my field tags was not an array of String..
I don't know what to search in documentation.
My model :
var PreambuleSchema = new Schema({
titre: String,
description: String,
content: String,
auteur: String,
tags: [String],
votes: [],
commentaires: [{
"authorMail": String,
"content": String
}],
commentaires_size: Number,
date: String
});
I'm using yeoman fullstacks, Angularjs, mangoose, nodeJs.
I am using angularjs for my page. I want to filter the values from the JSON object, So that no redundancy exists. But I didn't find any way to get the unique values from angular ng-repeat. Is there anyway to do that?
Ok, Here is some description about the question. I have a JSON in this format. This JSON I am getting from a service. So we cant expect how the repeated data occure.
result = [
{
_id: "500004",
subject: "Complete the task",
date: "25 Aug, 2013"
},
{
_id: "500004",
subject: "Complete the task",
date: "25 Aug, 2013"
},
{
_id: "500005",
subject: "Attend to the event",
date: "2 Jan, 2013"
},
{
_id: "500065",
subject: "Some task deadline",
date: "20 Sep, 2013"
},
{
_id: "500004",
subject: "Complete the task",
date: "25 Aug, 2013"
}
]
I want the output JSON to be with no repeated elements, So that my output will be something like this
result = [
{
_id: "500004",
subject: "Complete the task",
date: "25 Aug, 2013"
},
{
_id: "500005",
subject: "Attend to the event",
date: "2 Jan, 2013"
},
{
_id: "500065",
subject: "Some task deadline",
date: "20 Sep, 2013"
}
]
You can make use of Angular UI which has the unique filter defined.
The source can be found here.
Basically, you can then make use of the filter as follows:
<div ng-repeat="item in result | unique:'_id'">
//Body here
</div>
You can use 'unique'(aliases: uniq) filter in angular.filter module (https://github.com/a8m/angular-filter)
usage: colection | uniq: 'property'
you can filter by nested properties to : colection | uniq: 'property.nested_property'
So you can do something like that..
function MainController ($scope) {
$scope.orders = [
{ id:1, customer: { name: 'foo', id: 10 } },
{ id:2, customer: { name: 'bar', id: 20 } },
{ id:3, customer: { name: 'foo', id: 10 } },
{ id:4, customer: { name: 'bar', id: 20 } },
{ id:5, customer: { name: 'baz', id: 30 } },
];
}
HTML: We filters by customer id, i.e remove duplicate customers
<th>All customers list: </th>
<tr ng-repeat="order in orders | unique: 'customer.id'" >
<td> {{ order.customer.name }} , {{ order.customer.id }} </td>
</tr>
result:
All customers list:
foo 10
bar 20
baz 30
var data = [
{
_id: "500004",
subject: "Complete the task",
date: "25 Aug, 2013"
},
{
_id: "500004",
subject: "Complete the task",
date: "25 Aug, 2013"
},
{
_id: "500005",
subject: "Attend to the event",
date: "2 Jan, 2013"
},
{
_id: "500065",
subject: "Some task deadline",
date: "20 Sep, 2013"
},
{
_id: "500004",
subject: "Complete the task",
date: "25 Aug, 2013"
}
]
var uniqueNames = [];
var uniqueObj = [];
for(i = 0; i< data.length; i++){
if(uniqueNames.indexOf(data[i]._id) === -1){
uniqueObj.push(data[i])
uniqueNames.push(data[i]._id);
}
}
console.log('uniqueObj',uniqueObj)
http://jsfiddle.net/C97DJ/165/