Following the guidance provided here (https://cloud.google.com/datastore/docs/concepts/transactions), I'm creating an entity with below schema
data: [
{
name: 'created',
value: new Date().toJSON()
},
{
name: 'name',
value: templateObj.name,
excludeFromIndexes: true
}
]
I see this goes thru as expected and notice that the column "name" is not indexed.
Now, I use the transaction and update the entity. Below is the payload
[
{
"property": "active",
"value": false
},
{
"property":"name",
"value":"new updated template"
}
]
Code is exactly the same as documented.
While updating the data, why does it update the schema?
There is really no schema when it comes to Cloud Datastore. It is schemaless. Any time you save (Insert or Update) an entity (regardless of its Kind), you have to specify whether or not the properties in the entity should be indexed. If you do not explicitly set the excludeFromIndexes, the property will be indexed. So, when you create an entity or update an existing entity, make sure to set the excludeFromIndexes to false if you want to keep the property unindexed.
Related
I have Azure Cognitive Search running, and my index is working as expected.
We are trying to add a security filter into the search, based on the current users permissions.
The users permissions are coming to me in as IEnumerable, but I am currently selecting just a string[] and passing that into my filter, then do a string.join, which looks like this.
permission1, permission2, permission3, permission4
In our SQL database, we have a view that is where the index is getting it's data from. There is a column on the view called RequiredPermissions, it is a Collection(Edm.string) in the index, and the data looks like this.
[ 'permission1', 'permission2', 'permission3' ]
The requirement is that for a record to return in the results, a user's permissions must contain all of the RequiredPermissions for that record.
So if we have a user with the following permissions
permission1, permission3, permission5
And we have the following records
Id, SearchText, Type, Permissions
1, abc, User, [ 'permission1', 'permission2' ]
2, abc.pdf, Document, [ 'permission1' ]
3, abc, Thing, [ 'permission1', 'permission3' ]
4, abc, Stuff, [ 'permission3', 'permission4' ]
If the user searched for 'abc' and these four results would come back, I need to $filter results that do not have the proper permissions. So I would expect the following results
Id, Returned, Reason
1, no, the user does not have permission2
2, yes, the user has permission1 and nothing else is needed
3, yes, the user has both permission1 and permission3
4, no, the user does not have permission4
If I run the following filter, then I get back anything that has permission1 or permission3, which is not acceptable, since the user should not see items Id 1 or 4
RequiredPermissions/any(role: search.in(role, 'permission1, permission3', ','))
If I run this filter, then I get nothing back, everything is rejected, because no records have permission5, and the user has it
RequiredPermissions/all(role: not search.in(role, 'permission1, permission3', ','))
If I try to run the search using 'all' and without the 'not' I get the following error
RequiredPermissions/all(role: search.in(role, 'permission1, permission3', ','))
Invalid expression: Invalid lambda expression. Found a test for equality or inequality where the opposite was expected in a lambda expression that iterates over a field of type Collection(Edm.String). For 'any', please use expressions of the form 'x eq y' or 'search.in(...)'. For 'all', please use expressions of the form 'x ne y', 'not (x eq y)', or 'not search.in(...)'.\r\nParameter name: $filter
So it seems that I cannot use the 'not' with 'any', and I must use the 'not' with 'all'
What I wish for is a way to say that a user has all the permissions in their list that is in the RequiredPermissions column.
I am currently just working in Postman using the RestApi to solve this, but I will eventually move this into .Net.
Your scenario can't be implemented with Collection(Edm.String) due to the limitations on how all and any work on such collections (documented here).
Fortunately, there is an alternative. You can model permissions as a collection of complex types, which allows you to use all the way that you need to implement your permissions model. Here is a JSON example of how the field would be defined:
{
"name": "test",
"fields": [
{ "name": "Id", "type": "Edm.String", "key": true },
{ "name": "RequiredPermissions", "type": "Collection(Edm.ComplexType)", "fields": [{ "name": "Name", "type": "Edm.String" }] }
]
}
Here is a JSON example of what a document would look like with its permissions defined:
{ "#search.action": "upload", "Id": "1", "RequiredPermissions": [{"Name": "permission1"}, {"Name": "permission2"}] }
Here is how you could construct a filter that has the desired effect:
RequiredPermissions/all(perm: search.in(perm/Name, 'permission1,permission3,permission5'))
While this works, you are strongly advised to test the performance of this solution with a realistic set of data. Under the hood, all is executed as a negated any, and negated queries can sometimes perform poorly with the type of inverted indexes used by a search engine.
Also, please be aware that there is currently a limit on the number of elements in all complex collections across a document. This limit is currently 3000. So if RequiredPermissions were the only complex collection in your index, this means you could have at most 3000 permissions defined per document.
I have a multiple Cosmos DB document types which are aggregated into a single Azure Search index.
For each document type I have a data source + indexer pair. It is an autogenerated pair that maps from Cosmos DB document fields into the index.
I faced with an issue: I generated a new indexer + data source for a new document type. I see that indexer completed the indexing. But my Index does not contain new fields. And I can't even add a new field manually, which will contain the data added by indexer.
Basing on docs it is possible to add a new field without rebuilding the index. But it's not clear how exactly this can be done
What problem are you seeing when you try to add the new index field manually? You should be able to add a new field to the index definition with a new field via REST or the portal interface.
Via portal: Navigate to your search service, select appropriate index from list, click on the "Fields" tab and click "Add field". A new field should show up at the bottom of the index grid and you should be able to edit field attributes/name/type.
Via REST: You have the right link to the update operation. You can do a GET on the existing index definition and add your new field to the "fields" array before using PUT for update. Your new index field JSON should be similar to the other fields in the index, for example:
"fields": [
...,
{
"name": "new field",
"type": "Edm.String",
"facetable": false,
"filterable": false,
"key": false,
"retrievable": true,
"searchable": true,
"sortable": false,
"analyzer": "standard.lucene",
"indexAnalyzer": null,
"searchAnalyzer": null,
"synonymMaps": [],
"fields": []
}
]
Here's a doc reference on the index definition JSON.
Be sure to set index field attributes (Filterable, Sortable, Searchable...) on new field creation as not all attributes can be added without rebuilding the index. Only retrievable, searchAnalyzer, and synonymMaps can be modified to an existing field.
I currently have the following schema:
const mongoose = require('mongoose');
const Schema = mongoose.Schema;
const Comments = new Schema({
authorId: {
type: Schema.Types.ObjectId,
unique: false
},
commentBody: {
type: String
}
});
//Create schema
const GroupSchema = new Schema({
name:{
type: String,
required: true
},
comments: [Comments]
});
module.exports = Group = mongoose.model('group', GroupSchema);
I can insert the first group with no errors, but as soon as i try to add the second one i get this error:
{
"error": {
"name": "MongoError",
"message": "E11000 duplicate key error collection: app-idiomas.groups index: comments.authorId_1 dup key: { : null }",
"driver": true,
"index": 0,
"code": 11000,
"errmsg": "E11000 duplicate key error collection: app-idiomas.groups index: comments.authorId_1 dup key: { : null }"
}
}
So, as i suppose, the problem is with the "Comments" schema and i have searched about nested schema to see if i can solve this problem. I have tried to use "default" and "sparse" attribute, but no success.
Finally i decided to make a test and remove the Comments schema and it's reference in Groups Schema just to confirm that everything is OK. I erased all my database and added the first group (now without any comments attribute) and i still get the error whenever i try to add the second one. I just don't know why.
Any help ?
UPDATE
I didn't have time in these last days, but finally i found what the problem was. Probably when i was creating the Comments Schema i may have set the authorId to be unique and because of that MongoDB created an index for this attribute. I don't know much about MongoDB and that's why i hadn't checked before. But here is where i found the index and removed it:
Location of "indexes" tab in MongoDB
This problem has to be with record in the database.
But as you said you cleared the database, the problem looks weird.
Just make a check another time whether you deleted the correct database i.e the one you are using in your api.
I'm using couchdb to store large documents, which is causing some trouble when fetching them to memory. I do realize the database is not meant to be used this way. As a fallback solution, is it possible to fetch partial documents from the database, without creating a view?
In example, if a document has the fields id, content and extra_content, I would like to retrieve only the first two.
Thank you in advance.
If you are using CouchDB 2.x, you can use /db/_find endpoint as a mechanism to retrieve part of the doc.
POST /db/_find
{
"selector": {
"_id": "a-doc-id"
},
"fields": [
"_id",
"content"
]
}
You'll get only the set of fields you have specified in the query
This is not possible prior to CouchDB 2.x. For CouchDB 2.x or greater, see JuanjoRodriguez's answer.
But one possible work around for any version of CouchDB would be to take advantage of file attachments, which by default are excluded from a fetch. If some of your data isn't always needed, and doesn't need to be included in indexes, you could potentially store it as (JSON) attachments, rather than as part of the document directly:
{
"id": "foo",
"content": "stuff",
"extra_content": "other stuff"
}
becomes:
{
"id": "foo",
"content": "stuff",
"_attachments": {
"extra_content": {
"content_type": "application/json",
"data": "ZXh0cmEgc3R1ZmYK"
}
}
}
Let's say I have the following document schema in a collection called 'users':
{
name: 'John',
items: [ {}, {}, {}, ... ]
}
The 'items' array contains objects in the following format:
{
item_id: "1234",
name: "some item"
}
Each user can have multiple items embedded in the 'items' array.
Now, I want to be able to fetch an item by an item_id for a given user.
For example, I want to get the item with id "1234" that belong to the user with name "John".
Can I do this with mongoDB? I'd like to utilize its powerful array indexing, but I'm not sure if you can run queries on embedded arrays and return objects from the array instead of the document that contains it.
I know I can fetch users that have a certain item using {users.items.item_id: "1234"}. But I want to fetch the actual item from the array, not the user.
Alternatively, is there maybe a better way to organize this data so that I can easily get what I want? I'm still fairly new to mongodb.
Thanks for any help or advice you can provide.
The question is old, but the response has changed since the time. With MongoDB >= 2.2, you can do :
db.users.find( { name: "John"}, { items: { $elemMatch: { item_id: "1234" } } })
You will have :
{
name: "John",
items:
[
{
item_id: "1234",
name: "some item"
}
]
}
See Documentation of $elemMatch
There are a couple of things to note about this:
1) I find that the hardest thing for folks learning MongoDB is UN-learning the relational thinking that they're used to. Your data model looks to be the right one.
2) Normally, what you do with MongoDB is return the entire document into the client program, and then search for the portion of the document that you want on the client side using your client programming language.
In your example, you'd fetch the entire 'user' document and then iterate through the 'items[]' array on the client side.
3) If you want to return just the 'items[]' array, you can do so by using the 'Field Selection' syntax. See http://www.mongodb.org/display/DOCS/Querying#Querying-FieldSelection for details. Unfortunately, it will return the entire 'items[]' array, and not just one element of the array.
4) There is an existing Jira ticket to add this functionality: it is https://jira.mongodb.org/browse/SERVER-828 SERVER-828. It looks like it's been added to the latest 2.1 (development) branch: that means it will be available for production use when release 2.2 ships.
If this is an embedded array, then you can't retrieve its elements directly. The retrieved document will have form of a user (root document), although not all fields may be filled (depending on your query).
If you want to retrieve just that element, then you have to store it as a separate document in a separate collection. It will have one additional field, user_id (can be part of _id). Then it's trivial to do what you want.
A sample document might look like this:
{
_id: {user_id: ObjectId, item_id: "1234"},
name: "some item"
}
Note that this structure ensures uniqueness of item_id per user (I'm not sure you want this or not).