I was trying to find a step by step guide to migrate an existing project working with Datastore to the new Firebase in Datastore mode. If you do not want to wait for the Automatic migration to Firestore here is my experience with the migration.
Documentation page "Exporting and Importing Entities" has the steps, but I struggled a bit figuring things out. So I thought I'd share my experience.
Some notes to consider:
Be ware of cost/time if you have a huge database!
Yet to figure out the challenges of moving the application from Python 2.7 to Python 3 (not entirely a datastore issue, this doc page may help!)!
From Google Console, create a new project
gcloud config set project <your-old-project-id>
Create a new bucket on Google Storage (on your old project)
Export your database to the newly created bucket gcloud datastore export gs://<your-new-bucket-name> (on your old project)
Waiting for
[projects/[your-old-project-id]/operations/AvcsdafSDFasdfI3MDQJGnRsdWFmZWQHEmVwb3J1Z
S1zYm9qLW5asdfcsopEg] to finish...done.
From Google Console go to IAM page (still on your old project)
Add the new project service account account and grant "Cloud
Datastore Import Export Admin" & "Storage Admin" permissions (NOTE: for some reason when adding the "Storage viewer" only permission I got the below error, so I used the "Storage Admin" instead!)
"details: [new-project-service-account]#appspot.gserviceaccount.com does not have storage.buckets.get access to [new-project-id].appspot.com."
Move to your new project gcloud config set project <your-NEW-project-id>
Import the data from your storage bucket gcloud datastore import gs://<your-bucket-path>/[FILENAME].overall_export_metadata, when done you should see a message similar to export
Waiting for
[projects/[your-old-project-id]/operations/AaredafSDFa2otbmltZGEQCigSFmZWQHEmVwb3J1Z
S1z2otbmltZGEQCigS] to finish...done.
Optional: feel free to remove the new project service account
permission from the old project IAM page
If you browser to your new project Datastore page, you
should see your migrated Entities
Related
I have built this website using reactjs, firebase. I would like to know if someone wants to contribute to this project how should he/she get the data that is stored in my firebase account for the development of the website? https://www.codingspace.codes/
I know they need to create a new firebase project but how do they get the exact data that is on my website.
I don’t have any idea, anyone please help me with this.
This is a very open-ended question, typically though if it's for local development I'd just give them access to the firebase project as a collaborating member.
Alternatively, you could be looking into the import/export features of Firebase.
Use the firestore export command to export all the documents in your database, replacing [BUCKET_NAME] with the name of your Cloud Storage bucket. Add the --async flag to prevent the gcloud tool from waiting for the operation to complete.
gcloud firestore export gs://[BUCKET_NAME]
And importing it on their project in a similar manner:
Use the firestore import command to import documents from a previous export operation.
gcloud firestore import gs://[BUCKET_NAME]/[EXPORT_PREFIX]/
where [BUCKET_NAME] and [EXPORT_PREFIX] point to the location of your export files.
See the documentation for further details https://firebase.google.com/docs/firestore/manage-data/export-import
Our production app (python 2.7 standard environment) running on Google App Engine suddenly lost permissions to write (create objects) on Google Cloud Storage, without any change in the code on our side.
The code is able to create new buckets, but not new objects within them.
It seems that the default app engine service account is not granted the permission.
Needless to say, the service account has the Storage Object Creator role, as well as the Editor role on the project level.
Strangely, the exact same code running on the test environment project, continues to work perfectly.
We are using the api client library to obtain credentials, like so:
from oauth2client.appengine import AppAssertionCredentials
from apiclient.discovery import build as discovery_build
credentials = AppAssertionCredentials(scope=scope)
http = credentials.authorize(httplib2.Http())
service = discovery_build('storage', 'v1', http=http)
And then using the service to make the api call.
All calls to create objects are suddenly failing with the message:
"Anonymous caller does not have storage.objects.create access to /"
Any ideas what could suddenly have gone wrong ??
This turned up to be an issue with Google Cloud Storage (GCS).
A payed support ticket was opened, after approximately 90 hours, a rollback was made by Google GCS engineers which solved the issue, however, the root cause of the issue was not found or reported.
Very troubling that a production app can be affected this way for such a long time and eventually there is no explanation.
I want to create a Firestore in Native mode in an existing project.
I don't have any data in Cloud Datastore, but it blocks me, saying
This project uses another database service
Your current project is set up to use Cloud Datastore or Cloud Firestore in Datastore mode. You can access data for this project from the Cloud Datastore console.
when going through https://console.cloud.google.com/firestore/
and
Cannot enable Firestore for this project
Currently Firestore cannot be enabled in projects already using Cloud Datastore or App Engine
when going through https://console.firebase.google.com/
I've tried it with writes to datastore enabled and disabled
I just want to completely purge the Cloud Datastore product from my project.
Unfortunately, there is no way to purge the previous existence of a Cloud Datastore database to try either Cloud Firestore in native or Datastore mode. You'll have to use a new project to try Cloud Firestore in either native or Datastore mode.
You can switch using the below command
$ gcloud alpha firestore databases update --type=firestore-native
This is the response I got from Google Cloud Support today (February 16th, 2021):
Generally, we do recommend to create a new project and enable Firestore therein. Nonetheless I may submit a request to delete your existing database which may allow you to change the database to the desired mode. However, please keep in mind that I’m unable to guarantee it’s approval.
I just tried it on my end. I deleted all my entities (test project) and disabled my Datastore API, same issue as you when I visit the console.firebase.google.com page.
This is likely an issue that needs to be reported either through support (if you have a support package for Google Cloud Platform), or through our issue tracker.
Still the same as of 10 June 2020.
If you have an empty Datastore database and you never executed a write to the database, you can upgrade to Firestore in Datastore mode or Native mode.
If you do not receive this option, then your database instance will be automatically upgraded at a future date.
If you upgrade from Datastore to Firestore in Datastore mode or from Datastore mode to Native mode, you cannot undo the operation.
See this page for additional details:
https://cloud.google.com/datastore/docs/upgrade-to-firestore
If you just want to lock down use the following security rule it will not delete but lock down your firestore so no one can write to it. Its not the answer you are looking for but in the same spirit.
// Deny read/write access to all users under any conditions
service cloud.firestore {
match /databases/{database}/documents {
match /{document=**} {
allow read, write: if false;
}
}
}
Just visit https://console.cloud.google.com/....... by clicking on the button, if you have not added any data it will show option to switch to native
I read about the DataStore admin, but can't find it in the console. Also, the doc says the feature is experimental?
More clarification:
We have an enterprise application, and we need to be able to take backups of our data at a certain frequency. Since we are using Google App Engine, we need to be able to take a backup off the App Engine DataStore. When I go into the Admin console, I can see the option to view the entities, I see "Dashboard","Query" and "Index" but nothing to take a backup of the DataStore data, say to something like CloudStorage. Also, when I go to localhost:8000/datastore, I see the same.
Datastore Admin is only available to Python applications - or rather, applications with a Python version. If your app is Java, you can deploy an empty/trivial app to a non-default version.
There's a note on this in the docs, here.
At the time of your request it was only available in the old GAE console.
Since few weeks it has been integrated in the new console as well.
Go to Storage/Cloud Datastore/Settings, then the first time you'll have to "Enable Datastore Admin", then you'll have a new button "Open Datastore Admin"
I recall seeing an option on the google app engine admin page that would allow me to duplicate/copy a deployment to a different appid. Was I imagining this? I do have a copy of all the files needed for deployment but I do not currently have access to a computer with admin privileges in order to install the GAE SDK.
Not relevant to this question, but FYI I need to do this simply because there is a major bug that's preventing me from logging in to a particular part of the admin interface and google tech support has failed to resolve the issue, so I'm just trying a workaround.
Copy old settings to new app via the admin console admin>application settings>Duplicate Application Settings. (you'll choose new appid here too)
Then upload a version of your code to the new appId (so you'll need gae sdk or use online sdk devtable).
Then go to datastore admin in the old app, select all the entities and choose "copy to another app" You'll need datastore admin to be enabled on the old app (and new app I think).