I have created a Kind called User in Google App Engine datastore, and I am trying to add an index for this kind.
Firstly, I followed https://cloud.google.com/appengine/docs/standard/java/config/indexconfig to create index by adding datastore-indexes.xml inside war/WEB-INF, but it doesn't work, no index is created after I deploy to app engine.
Then I followed https://cloud.google.com/appengine/docs/standard/python/config/indexref, I created a index.yaml and run gcloud app deploy index.yaml, this time I can see the index created in GCP console as seen below.
Index seen in GCP console
But I still got an exception as below:
Uncaught exception from servlet
com.google.appengine.api.datastore.DatastoreNeedIndexException:
no matching index found. recommended index is:
- kind: User
properties:
- name: area
- name: coins_balance
The suggested index for this query is:
<datastore-index kind="User" ancestor="false" source="manual">
<property name="area" direction="asc"/>
<property name="coins_balance" direction="asc"/>
</datastore-index>
I searched google with "index created but still got DatastoreNeedIndexException", but didn't find any helpful information.
Please can someone help, thanks.
The order of the properties in the index matters. The screenshot shows an index for
- kind: User
properties:
- name: coins_balance
- name: area
You actually want:
- kind: User
properties:
- name: area
- name: coins_balance
So you can either change your query to match your existing index, or you can build the index that the recommended index.
Related
Playing with Bookshelf tutorial and the Auth package.
The ListBooksCreatedBy query is failing with this error:
could not list books: datastoredb: could not list books: rpc error: code = FailedPrecondition desc = no matching index found. recommended index is:
- kind: Book
properties:
- name: CreatedByID
- name: Title
However, index.yaml does exist and is in the same directory as app.yaml :
index YAML content:
indexes:
# This index enables filtering by "CreatedByID" and sort by "Title".
- kind: Book
properties:
- name: CreatedByID
direction: asc
- name: Title
direction: asc
Why would the index not be imported?
Where in the Golang App Engine code is the index.yaml imported?
I searched the src and found nothing.
The index.yaml file is not intended to be imported/used in your application code, it is intended to be deployed to the Datastore service (separately/independently from your app deployments!) to configure its indexing activities. From Updating indexes:
You upload your index.yaml configuration file to Cloud Datastore
with the gcloud command. If the index.yaml file defines any indexes
that don't exist in Cloud Datastore, those new indexes are built.
It can take a while for Cloud Datastore to create all the indexes and
therefore, those indexes won't be immediately available to App Engine.
If your app is already configured to receive traffic, then exceptions
can occur for queries that require an index that is still in the
process of being built.
To avoid exceptions, you must allow time for all the indexes to build.
Only after proper deployment and, if needed, the completion of the index update operation (i.e. the index reaches the Serving state) the app's queries needing that index can work.
The index.yaml file is also used by the local development server in its datastore emulation. By default it automatically updates the file with the necessary indexes for the queries it encounters, but the file isn't automatically deployed to the Datastore. But you can disable this behaviour using the --require_indexes option. From Local Development Server Options:
--require_indexes=yes|no
Disables automatic generation of entries in the index.yaml file.
Instead, when the application makes a query that requires that its
index be defined in the file and the index definition is not found, an
exception will be raised, similar to what would happen when running on
App Engine. The default value is no.
Is anyone else having an issue with a lot of their datastore entities being "un-editable" in the Developers Console?
I always get this error when clicking on an entity ID:
We could not find an entity with namespace: "{NAMESPACE}", kind: "{KIND}", and identifier "id={ID}"
Google Issue Reported
Consider the following situation:
I have two AppEngine projects: A and B
I have a Cloud Storage bucket with the following ACL:
<?xml version="1.0" ?>
<AccessControlList>
<Owner>
<ID>id-of-the-user-who-created-the-bucket</ID>
</Owner>
<Entries>
<Entry>
<Scope type="UserByEmail">
<EmailAddress>app-A-service-account-name</EmailAddress>
</Scope>
<Permission>FULL_CONTROL</Permission>
</Entry>
<Entry>
<Scope type="UserByEmail">
<EmailAddress>app-B-service-account-name</EmailAddress>
</Scope>
<Permission>FULL_CONTROL</Permission>
</Entry>
</Entries>
</AccessControlList>
My GAE applications are written in Python and they are using GCS Client Library
Now, here is what I want to achieve: I want application A to create files inside the bucket and then application B to read them.
At first I tried to simply create a file with cloudstorage.open(file_name, 'w') and then read its status with cloudstorage.stat(file_name, 'r'), but this way I end up with the following error while reading:
ForbiddenError at /.../
Expect status [200] from Google Storage. But got status 403.
(The error message provides also request/response information: path, headers, body and extra info. Please let me know if you think they may be helpful in solving this case)
Then I started experimenting with ACLs by setting the x-googl-acl option while creating a file, for example:
cloudstorage.open(file_name, 'w', options={'x-goog-acl': 'authenticated-read'})
Although ACLs work as intended, none of the available options seem to fit my requirements:
private - only the bucket owner has the access, B cannot read
public-read - file is accessible by anonymous users, unacceptable
public-read-write - same as above
authenticated-read - everyone with authenticated account is able to read (even people who are not part of the project), so it's no different than the previous option
bucket-owner-read - seems perfect, but it turns out that "the bucket owner" is NOT the user who was set as "owner" through the Cloud Console, but the user who created the bucket
bucket-owner-full-control - same as above
It looks like I ran out of options, but I can't believe that such a simple thing cannot be achieved with the Cloud Storage. The only solution that comes to my mind is changing system's architecture, but I would like to avoid it. Any other suggestions?
Add the accessor Service Accounts (e.g. app1#appspot.gserviceaccount.com or 1234567890-compute#developer.gserviceaccount.com for compute engine) as member with 'Editor' permission on project with the GCS bucket to use. This can be done in IAM page of the project that owns the bucket:
https://console.developers.google.com/iam-admin/iam/project?project=app1
How do I reload it into Solr again? If I try to launch the Solr Admin UI, I get the following message:
There are no SolrCores running.
Using the Solr Admin UI currently requires at least one SolrCore.
You will need to modify the <cores> entry in the solr.xml file that is in the root of where your Solr instance is running and add at least one <core> entry. Below is the example that comes with the Solr distribution.
<cores adminPath="/admin/cores" defaultCoreName="collection1">
<core name="collection1" instanceDir="." />
</cores>
You will need to modify this according to your local settings. You can reference more information on Cores and their settings in the CoreAdmin Solr Wiki page.
You can just run: http://127.0.0.1:8080/solr/admin/cores?action=RELOAD&core=collection1
Please note that the path and port to your solr server are correct.
I'm following this tutorial on setting up django-haystack and solr: http://django-haystack.readthedocs.org/en/latest/tutorial.html
I hit a stumbling block here:
If you’re using the Solr backend, you have an extra step. Solr’s
configuration is XML-based, so you’ll need to manually regenerate the
schema. You should run ./manage.py build_solr_schema first, drop the
XML output in your Solr’s schema.xml file and restart your Solr
server.
Where is my schema.xml file located? It says it should in the Solr home directory and the .conf folder. But where is the Solr home directory, and/or how do I configure its location?
The solr home is the place where you can find your schema.xml and solrconfig.xml, as well as some other files depending on the text analysis you're using (dictionaries for stemming, stopwords etc.), and where your index gets created by default.
There are a couple of ways to configure the solr home, since it is located outside of the servlet container:
solr.solr.home java system property (most used one)
java:comp/env/solr/home for JNDI lookup
You can either check your servlet container configuration or go to the Solr admin page http://host:port/solr/admin, which prints out the actual solr home location together with other information about the solr instance running.
First check whether your Solr instance is working.
Got to -> http://localhost:8983/solr
If you can see a Solr web panel you have a live Solr instance.
Now go to Java Properties
Here you will see the the variables. This is where you can find the home DIRs
Note schema is now managed. If you want to override this you will have to hack it a bit. check here