i'm trying to create a datastore emulator with docker ,
and followed the instructions here
https://cloud.google.com/datastore/docs/tools/datastore-emulator
also i used the cloud-sdk
https://hub.docker.com/r/google/cloud-sdk/
I was able to create the emulator and authenticate with it, but when i attempt to access it throght localhost:8000/datastore
it gives me "Not found" response,
How could i access the datastore data?
the command i used to create it is:
docker run -p 8000:8000 google/cloud-sdk gcloud beta emulators datastore start --project=pname --host-port localhost:8000 --no-store-on-disk
Datastore emulator does supports only HTTP/2. This means that you should access the data in the emulator using support client i.e google-cloud-python, google-cloud-java e.t.c This official libraries support setting the emulator host using the environmental variables DATASTORE_EMULATOR_HOST and DATASTORE_PROJECT_ID.
sample setting variables
export DATASTORE_EMULATOR_HOST=localhost:8000
export DATASTORE_PROJECT_ID=project-id-in-google
this should allow the client to access the emulator instead, when done unset the variable above to access the live datastore using:
unset DATASTORE_EMULATOR_HOST
unset DATASTORE_PROJECT_ID
Notes:
the DATASTORE_EMULATOR_HOST can be retrieved from the running docker container as shown below
[datastore] API endpoint: http://localhost:8000
[datastore] If you are using a library that supports the DATASTORE_EMULATOR_HOST environment variable, run:
[datastore]
[datastore] export DATASTORE_EMULATOR_HOST=localhost:8000
[datastore]
[datastore] Dev App Server is now running.
DATASTORE_PROJECT_ID should match the project online
tid bits
check emulator status GET request to http://localhost:8000, it should return ok if the emulator is running
reset the emulator to remove all data by POST request to http://localhost:8000/reset
Related
it shows such a message in my console when I run my express
$ firebase emulators:start
i emulators: Starting emulators: functions, database, hosting
! functions: The following emulators are not running, calls to these services from the Functions emulator will affect production: auth, firestore, pubsub, storage
+ functions: Using node#16 from host.
i database: Database Emulator logging to database-debug.log
! database: Fatal error occurred:
Database Emulator has exited because java is not installed, you can install it from https://openjdk.java.net/install/,
stopping all running emulators
i functions: Stopping Functions Emulator
i database: Stopping Database Emulator
! database: Error stopping Database Emulator
i hub: Stopping emulator hu
Have you noticed the error message in your output?
Database Emulator has exited because java is not installed,
you can install it from https://openjdk.java.net/install/
I would recommend following the instructions to install java.
Also check the requirements here https://firebase.google.com/docs/emulator-suite/install_and_configure
Is it possible to deploy Elasticsearch on App engine flex environment using a docker image.
I have tried the following
My files on the local machine
Folder : elasticsearch
app.yaml
Dockerfile
docker-entrypoint.sh
config folder(containing elasticsearch.yml)file
Contents of app.yaml
runtime: custom
env: flex
Dockerfile and docker-entrypoint.sh copied from https://github.com/GoogleCloudPlatform/elasticsearch-docker/tree/master/5/5.2.0
Modifications to the Dockerfile
replaced EXPOSE 9200 9300 to EXPOSE 8080
Modification to the elasticsearch.yml
cluster.name: "beaconinside-docker-cluster"
path.data: /usr/share/elasticsearch/data
http.host: 0.0.0.0
http.port: 8080
discovery.zen.minimum_master_nodes: 1
I build a container using the docker file on my local machine
docker build -t elasticdemo .
Then, I run the container
docker run -p 8080:8080 elasticdemo
I am able to access elasticsearch on 0.0.0.0:8080
Problem:
I am trying to deploy elasticsearch as an app to Google app engine flex environment
gcloud app deploy app.yaml --version elasticdocker --project myproject
The deployment fails with the following error
Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [9]
I was expected elasticsearch to deploy as an app and be available on the deployed url.
Could you please provide pointers/help/suggestions with this approach?
While you can deploy ES to App Engine Flexible environment it's not particularly useful. The VMs hosting GAE Flexible containers are restarted regularly as part of maintenance and whatever data is stored on the local disk will be lost on restart. If you want to use local disk for long term storage, I'd suggest to deploy the GCE VM's (or alternatively use a solution from the GCP Marketplace) or deploy to GKE which supports persistent disks
As for the actual question: you probably don't have a health check handler and therefore App Engine Flexible environment doesn't consider your app healthy after deploying it. The error message is useless, I agree.
From the GAE Flexible docs for building custom images:
"A health check is an HTTP request to the URL /_ah/health. A healthy application should respond with status code 200."
Alternatively you can turn off health checks by adding into app.yaml
enable_health_check: False
I've been developing some REST service using Flask and other third party libraries and I want to deploy it to GAE in the flexible environment. I usually deploy to the GAE standard environment but I wanted to try the new flexible environment. At the moment I wish to deploy to flexible environment without enabling billing, and the Google support assured me that it was possible to deploy over GAE flexible environment without enabling billing.
Running my code locally works fine, and have the following yaml file:
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT whereismybus230.starter:app
runtime_config:
python_version: 3
So I created a new project on through the Google cloud console web page (as usual), and created a new gcloud profile on my local machine so I deploy it to this new project.
Then I run:
gcloud app deploy --verbosity=info
I get that a docker image is being build and at some point it will be pushed to a Compute Engine but it fails after a few minutes here:
Successfully built sophiabus230 aniso8601 future docopt itsdangerous MarkupSafe
Installing collected packages: Werkzeug, click, MarkupSafe, Jinja2, itsdangerous, Flask, jsonschema, pytz, six, python-dateutil, aniso8601, flask-restplus, beautifulsoup4, future, sophiabus230, coverage, requests, docopt, coveralls
Successfully installed Flask-0.12 Jinja2-2.9.4 MarkupSafe-0.23 Werkzeug-0.11.15 aniso8601-1.2.0 beautifulsoup4-4.5.3 click-6.7 coverage-4.3.4 coveralls-1.1 docopt-0.6.2 flask-restplus-0.9.2 future-0.16.0 itsdangerous-0.24 jsonschema-2.5.1 python-dateutil-2.6.0 pytz-2016.10 requests-2.12.5 six-1.10.0 sophiabus230-0.4
---> 3e3438680079
Removing intermediate container bd9f8ccb6f4a
Step 8 : ADD . /app/
---> bde0915f6720
Removing intermediate container e3193eb4ef70
Step 9 : CMD gunicorn -b :$PORT whereismybus230.starter:app
---> Running in 022d38d769f8
---> 36893d0a549a
Removing intermediate container 022d38d769f8
Successfully built 36893d0a549a
PUSH
The push refers to a repository [us.gcr.io/whereismy230/appengine/default.20170120t131841]
e5f488ee94c5: Preparing
8d27ce27f03c: Preparing
3d5800d45c36: Preparing
06ba8a2a8ec3: Preparing
c0fb81dae3c6: Preparing
2e4eabdbeed3: Preparing
b5d474284f52: Preparing
c307273999be: Preparing
d73750730c30: Preparing
63bbaf04cf0b: Preparing
badb9b2d625b: Preparing
40c928fd4dcc: Preparing
dfcf8dbe47e1: Preparing
6d820e13990c: Preparing
2e4eabdbeed3: Waiting
b5d474284f52: Waiting
c307273999be: Waiting
d73750730c30: Waiting
63bbaf04cf0b: Waiting
badb9b2d625b: Waiting
40c928fd4dcc: Waiting
dfcf8dbe47e1: Waiting
6d820e13990c: Waiting
denied: Unable to create the repository, please check that you have access to do so.
The push refers to a repository [us.gcr.io/whereismy230/appengine/default.20170120t131841]
...
ERROR: (gcloud.app.deploy) Error Response: [2] Build failed; check build logs for details
Using the IAM service, I made sure my account was the owner of the project, and even checked all permissions.
Since the flexible environment relies on the Compute Engines (VMs), I tried to check from the web page and it's telling me that I need to enable billing to be able to use this functionality.
Am I doing something wrong ?
Thanks !
From App Engine Pricing:
Instances within the standard environment have access to a daily
limit of resource usage that is provided at no charge defined by a set
of quotas. Beyond that level, applications will incur charges as
outlined below. To control your application costs, you can set a
spending limit. To estimate costs for the standard environment,
use the pricing calculator.
Go to the pricing calculator
For instances within the flexible environment, services and APIs are
priced as described below.
And from Flexible environment instances:
Applications running in the App Engine flexible environment are
deployed to virtual machine types that you specify. This table
summarizes the hourly billing rates of the various computing
resources:
US
Resource Unit Unit cost
vCPU per core hour $0.0526
Memory per GB hour $0.0071
Persistent disk per GB per month $0.0400
Unlike the standard env, the flex env has no free quota. Which is inline with your observation that the developer console requires billing to be enabled to run GAE flex instances.
Without billing enabled you might be able to deploy your app (but without actually launching a GAE instance for it, so unsure of its usefulness, since you want to try it) by using the --no-promote option:
--promote
Promote the deployed version to receive all traffic.
True by default. To change the default behavior for your current
environment, run:
$ gcloud config set app/promote_by_default false
Overrides the default promote_by_default property value for this
command invocation. Use --no-promote to disable.
Side note: when you encounter problems you may also want to use --verbosity=debug to potentially get more relevant info about the failures.
I run the Datastore emulator, and I get the following apparently correct console log
> gcloud beta emulators datastore start --project=myproj --data-dir "./gcloud_datastore"
[datastore] If you are using a library that supports the DATASTORE_EMULATOR_HOST environment variable, run:
[datastore]
[datastore] export DATASTORE_EMULATOR_HOST=localhost:8232
[datastore]
[datastore] Dev App Server is now running.
In another cmd window, while the emulator is running, I run the following, or alternatively I stop the emulator first. Either way, it produces a different port and a different project:
> gcloud beta emulators datastore env-init
set DATASTORE_DATASET=anotherproj
set DATASTORE_EMULATOR_HOST=localhost:8297
set DATASTORE_EMULATOR_HOST_PATH=localhost:8297/datastore
set DATASTORE_HOST=http://localhost:8297
set DATASTORE_PROJECT_ID=anotherproj
What is going wrong? How do I fix this?
Use the same data directory when running the env-init command. For example:
gcloud beta emulators datastore env-init --data-dir=./gcloud_datastore
When you start the Emulator using the start command, it creates/updates a file env.yaml in the specified data directory. This file contains the environment info (such as port, project name), which I believe is used by the env-init command.
I'm now able to run a Dart app using
gcloud --verbosity debug preview app run app.yaml
and also to deploy and run on AppEngine
gcloud --verbosity debug preview app deploy app.yaml
but I haven't found a way to connect a debugger to the Dart app running on the development server.
I found http://dartbug.com/21067 but still couldn't find a way to make it work.
See also https://groups.google.com/a/dartlang.org/forum/#!topic/cloud/OK1nJtg7AjQ
Update 2015-02-27
The app can be run without Docker and then be debugged like any Dart command line application:
Source. https://groups.google.com/a/dartlang.org/d/msg/cloud/zrxgOHFz_lA/q5CdLLQPBAgJ
The API server is part of the App Engine SDK, and we are using it for
running tests in the appengine package. If you look at
https://github.com/dart-lang/appengine/blob/master/tool/run_tests.sh
you will see that it expects the environment variable
APPENGINE_API_SERVER.
The API server is in /platform/google_appengine/api_server.py
and takes a number of arguments. I just tested running it like this:
$ $CLOUD_SDK/platform/google_appengine/api_server.py \ -A
dev~test-application \ --api_port 4444 \ --high_replication \
--datastore_path /tmp/datastore
To run an app engine application outside the normal development server
requires that a number of environment variables are set. This worked
for my application:
$ GAE_LONG_APP_ID=test-application \ GAE_MODULE_NAME=default \
GAE_MODULE_VERSION=version \ GAE_PARTITION=dev \ API_PORT=4444 \
API_HOST=127.0.0.1 \ dart bin/server.dart
In the Dart Editor you cannot set environment variables for each
launch configuration, so they have to be set globally before starting
the Dart Editor. In WebStorm it is possible to have run configuration
specific environment variables.
This simple setup will of cause not support everything the normal
development server support. Some of the issues are:
Only one application at the time as it is always listening on port
8080 (can easily be made configurable) * The users API (mocking this
shouldn't be that difficult) * The modules API * No health-checks
(should not be a problem) * All HTTP headers are direct from the
client (no x-appengine- headers) * The admin web interface is not
available * Probably other stuff as well
This is all experimental, but it is one solution for a simpler
developer setup, which of cause does not match the deployment
environment as closely as the development server.
Running the API Server using Docker is also possible as the image
google/cloud-sdk with the Cloud SDK is on hub.docker.com.
Use the following Dockerfile
FROM google/cloud-sdk EXPOSE 4444 ENTRYPOINT
["/google-cloud-sdk/platform/google_appengine/api_server.py", \
"-A", "dev~test-application", \ "--api_port", "4444", \
"--high_replication", \ "--datastore_path", "/tmp/datastore"]
Build and run
$ docker build -t api_server . $ docker run -d -p 4444:4444 api_server
Change API_HOST above to 192.166.59.103 (of wherever your Docker
containers are) and run.
Regards, Søren Gjesse
Update 2014-11-27
Debugging from DartEditors debugger started working with the bleeding Dart build 1.8.0.edge_042017.
I assume that the next dev build (probably 1.9.0-dev1.0) will include the related fixes as well?
Detailed steps how this works can be found here: https://groups.google.com/a/dartlang.org/d/msg/cloud/OK1nJtg7AjQ/u-GzUDI-0VIJ
Build a custom Docker image with the latest Dart dev build 1.8.0-dev.4.6.
The Dart team is actually preparing an easy way to do this yourself (see https://github.com/dart-lang/dart_docker)
Installe the latest bleeding_edge on the host system (using this script https://gist.github.com/zoechi/d240f56a32ed5649797f or manual download from http://gsdview.appspot.com/dart-archive/channels/be/raw/latest/editor/darteditor-linux-x64.zip)
Add this to the app.yaml file
env_variables:
DBG_ENABLE: 'true'
# disable health-checking because this is so annoying during debugging
vm_health_check:
enable_health_check: False
See How to disable health checking for `gcloud preview app run` for more details about customizing health checking.
Launch the server code of your app with glcoud --verbosity debug app run app.yaml or glcoud --verbosity debug app run app.yaml index.yaml
Wait until the Docker container is ready (check with docker ps if the Command column shows a value starting with /dart_runtime/dart_
Open DartEditor
Open Menu Run > Remote Connection...
Connect to: Command-line VM
Host: localhost if you dont use boot2dockeror the IP address returned by the commandboot2docker ip`
Port: 5005
Select Folder... select the directory which contains the source code of your project.
Click OK
Set breakpoints and continue as usual.
Old
A first step is using the Observatory which includes a browser based debugger UI.
To make this work add the following lines to the app.yaml file
network:
forwarded_ports: ["8181"]
This might be useful as well to make the server.dart wait until we had the chance to set breakpoints using the observatory.
env_variables:
DART_VM_OPTIONS: '--pause-isolates-on-start'
boot2docker gives us the Docker ip (192.168.59.103) and after starting with gcloud preview app run app.yaml we can connect to http://192.168.59.103:8181 which should open the Observatory GUI.