I'm deploying an app in Google Cloud that has multiple microservices. For one of them, I wish to use a push queue. I configured my queue.yaml file as follows:
queue:
- name: equeue
rate: 10/s
retry_parameters:
task_retry_limit: 1
task_age_limit: 2d
I spin up the SDK and the queue is working fine, just as it should. But then I go to push it to my app on Google Cloud using:
gcloud app deploy --project=<projectname> --promote --stop-previous-version
The code all loads correctly, but the queue never gets created. I am getting errors from my code as follows:
API error 1 (taskqueue: UNKNOWN_QUEUE)
And when I look in the console, on https://console.cloud.google.com/appengine/taskqueues, the queue has not been created.
I've read the documentation and it seems to indicate that all I need to do is create the queue.yaml file and deploy the app (and that has worked for me in the past, albeit with an older version of the SDK.)
In case it helps, here are the components I'm running locally:
Google Cloud SDK 164.0.0
app-engine-go
app-engine-python 1.9.57
bq 2.0.24
core 2017.07.25
gcloud
gsutil 4.27
Thanks!
Dan
The queue.yaml is an app-level configuration file, not a service-specific one. See (marginally related) Why do I need to deploy a "default" app before I can deploy multiple services in GCP?
While in a single service app case deploying the default service might also update one or more of these app-level configs that is not guaranteed in a multi-service app case. Which is why each of these configs also comes with an individual dedicated method of deployment which can be performed independently of and/or without updating any of the app's services.
For queue.yaml in particular deployment can be done using gcloud app deploy:
gcloud app deploy [...] queues.yaml
or, if you're using the GAE SDK, using appcfg.py update_queues:
appcfg.py update_queues -A <app_id>
Related
There are many answers to this question already, but they no longer work here in January 2021. All those answers come in 3 flavors:
Set local machine timeout with something like gcloud config set app/cloud_build_timeout 1600 then deploy with gcloud app deploy ...
Use a cloudbuild.yaml file instead with timeout: 1600s bits in the gcloud app deploy buildstep and the global configuration, then deploy with gcloud builds submit ...
Per google's own docs, don't set timeout: 1600s in the cloudbuild file, but rather do a mashup of the previous 2 flavors with a build step including args: ['-c', 'gcloud config set app/cloud_build_timeout 1600 && gcloud app deploy']
None of them have any impact on the app deploy build - it's stuck at 10mins. When using gcloud builds submit, it results in 2 Cloud Builds being kicked off: one for the cloudbuild.yaml, and one for the app engine deployment using buildpack. The above solutions can impact the first build, but once that first build kicks off the second build (gcloud app deploy, you can see at https://console.cloud.google.com/cloud-build in the Execution Details tab that the Timeout is still 10m.
IMO, solutions 2-3 are hacks since 1 doesn't work, but now that 2-3 don't work either, I'm looking for another hack. Does anyone have a solution which works in 2021? Since my app is using GAE Standard Environment, I can't prebuild an image - I'm stuck with Buildpack building my ruby app and pulling all the Gems every time, and this runs out the seemingly immutable 10m clock.
You cannot change the timeout property in App Engine standard and it's always 10min. The workaround is to use App Engine flex and this way you can use the gcloud config set app/cloud_build_timeout TIME_SECONDS.
There is a feature request to enable timeout edit for App Engine standard but seems still in progress.
We are deploying our Go app to Google app engine with the command: gcloud app deploy
This creates a cloud build, which normally works fine. However, sometimes there are some dependencies which are flaky (sometimes unavailable or down) or gone (revision no longer existing).
These issues break our complete deploy and build and we can only get rid of the dependency or wait until it is fixed on that side.
Solution for this is using a proxy for the go dependencies (like https://proxy.golang.org/), but I don't have any way of setting this setting for our app engine deployments.
Does anyone know how I can use a proxy for google app engine deploys to make our builds more stable?
I have 2 json files, I want to deploy on GCloud
With this command:
gcloud endpoints services deploy first_file.json
gcloud endpoints services deploy second_file.json
But after second file deploys the first one is removed.
Why?
If both files are for the same service you should check this documentation on how to deploy configuration files. You might be also having some issues with naming, as described in the description of the gcloud endpoints services deploy page.
Besides that, here you'll find tutorials on how to configure Endpoints on App Engine Standard.
If this is not the case, please add more information to the question, such as what is your use case and some code if possible.
I am trying to run an app engine spring project on localhost using eclipse. I am getting the following error in the console.
Apr 23, 2018 4:01:04 PM com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
WARNING: Failed to detect whether we are running on Google Compute Engine.
java.net.SocketException: Network is unreachable: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:104)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
at com.google.auth.oauth2.ComputeEngineCredentials.runningOnComputeEngine(ComputeEngineCredentials.java:191)
at com.google.auth.oauth2.DefaultCredentialsProvider.tryGetComputeCredentials(DefaultCredentialsProvider.java:270)
at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentialsUnsynchronized(DefaultCredentialsProvider.java:194)
at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:112)
at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:127)
at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:100)
at com.google.cloud.ServiceOptions.defaultCredentials(ServiceOptions.java:298)
at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:272)
at com.google.cloud.datastore.DatastoreOptions.<init>(DatastoreOptions.java:102)
at com.google.cloud.datastore.DatastoreOptions.<init>(DatastoreOptions.java:35)
...
How do I provide credentials to use the datastore in localhost?
I get the following on the console when i access the datastore.
2018-04-23 17:22:05.849:WARN:oejs.ServletHandler:qtp152005629-14: org.spring.framework.web.util.NestedServletException: Request processing failed; nested exception is com.google.cloud.datastore.DatastoreException: Unauthenticated.
Caused by:
com.google.cloud.datastore.DatastoreException: Unauthenticated
Objectify v6 uses the Datastore SDK for Google Cloud Platform (which you could think of as GCP's "general purpose" SDK). That SDK makes actual API calls to the cloud service, even when it's being run locally. If you want to run tests/develop against a local datastore, you need to do one of two things.
Option One
Instead of using the GCP Datastore SDK, use the App Engine SDK’s Datastore bindings. Objectify has committed in v6 to using the GCP SDK, but v5 used the App Engine SDK. You can downgrade to v5 and everything will "just work".
Option Two
If you want to keep using Objectify v6, you need to install an extra gcloud component to run a local datastore emulator and you'll need to set some environment variables. More details here, but the short version goes:
Install and start the datastore emulator:
$ gcloud components install cloud-datastore-emulator
$ gcloud beta emulators datastore start
Then tell your application where to find it:
$ (gcloud beta emulators datastore env-init)
Or you can set the relevant env vars manually. On the machine that runs your application, set the environment variables and values as directed by the output of the env-init command. For example:
gcloud beta emulators datastore env-init
export DATASTORE_EMULATOR_HOST=localhost:8432
export DATASTORE_PROJECT_ID=my-project-id
according to the official documentation for Google App Engine Standard environment (Go API) the "preferred tooling to deploy a project" is now the Cloud SDK and so we moved to gcloud from goapp.
We are unable to deploy Go projects to GAE because all the sub-packages of every given project can't be found at "deploy time".
The typical folder structure that we have been using for every GAE project was as follows:
-project-name
--app.yaml
--main.go
--assets
---package1
---package2
When global libraries were put in the system GOPATH everything worked smoothly.
Running gcloud app deploy we now get this:
You are about to deploy the following services:
- yourproject/default/123456789 (from [/Path/to/app.yaml])
Deploying to URL: [https://yourproject.appspot.com]
Do you want to continue (Y/n)? Y
Beginning deployment of service [default]...
ERROR: (gcloud.app.deploy) Staging command [/path/to/yourproject/app.yaml /var/folders/b6/5ydn0wdn64jd32sxzzz48b7c0000gn/T/tmpbd4oiG] failed with return code [1].
------------------------------------ STDOUT ------------------------------------
------------------------------------ STDERR ------------------------------------
2017/03/24 10:25:58 failed analyzing /path/to/yourproject: cannot find package "yourpackage" in any of:
($GOROOT not set)
/path/to/gopath/src/yourpackage (from $GOPATH)
GOPATH: /path/to/gopath
--------------------------------------------------------------------------------
while dev_appserver.py works perfectly keeping the same folder structure.
Did we miss something?
How can we deploy to Google App Engine Standard environment using gcloud?
If the project structure needs to be changed: how? Is there official documentation about it?
Thanks in advance,
Edit -- Further infos:
Luigi-Mac-Pro:path/to/yourproject distudio$ gcloud version
Google Cloud SDK 148.0.0
app-engine-go
app-engine-go-darwin-x86_64 1.9.50
app-engine-python 1.9.50
bq 2.0.24
bq-nix 2.0.24
core 2017.03.17
core-nix 2016.11.07
gcloud
gcloud-deps 2017.03.17
gcloud-deps-darwin-x86_64 2017.02.21
gsutil 4.23
gsutil-nix 4.22
Google recommends keeping your dependencies outside of the app directory, and using GOPATH to refer them. In your case, that would mean doing the following:
-project-name
--app.yaml
--main.go
where main.go contains
import (
"package1"
"package2"
)
And somewhere else:
-my_packages
--src
---package1
---package2
And then set GOPATH environment variable to path/to/my_packages prior to running dev_appserver and gcloud app deploy.
For the future
We are working out the long term solution for properly vendoring packages inside your app directory – likely using Go's future native package manager. I'm sorry to say that we don't have a good way of supporting sub-packages for gcloud app deploy. This was an unfortunate side effect of compatibility with the App Engine Flexible environment.