My problem is that I want to create dev, stage, prod environments by using different GCP projects.
Basically they are running the same code, just running them in different isolated environments.
I'm using gcloud app deploy in command line to deploy app right now.
How can I efficiently deploy an app to different project?
Do I have to do gcloud init to change my configuration of default project every time?
There must be some better practices.
Or, is there a better way for me to set up dev... environments in the context of app engine?
Thanks.
Instead of using gcloud init to change your configuration each time, use this code to deploy to multiple projects faster:
gcloud app deploy -q --project [YOUR_PROJECT_ID]
If your projects have similar IDs, let's say: test1, test2, test3, test4, You can deploy to the four projects with one command. Use this code:
for i in {1..4}; do gcloud app deploy -q --project test${i}; done
The "standard" approach is to use versions, e.g.
qa.myApp.appspot.com
Once a version is ready for next step, you deploy it with a different version id.
One problem with using multiple projects is that you have to maintain a different data set for each project.
My preference is to have the different environments managed via the same version control as the code - one branch for each environment, keeping the deployments perfectly aligned with the natural flow of code changes, promoted via branch merges: dev -> stage -> production.
To minimize the risk of human error I try as much as possible to keep the deployment configs in the code itself (i.e. - have the app IDs, versions, etc. picked up from the .yaml files, not passed to the deploy cmd as args). The deployment cmds themselves are kept in a cheat-sheet file (too simple to warrant a full-blown script at this time), also git-controlled. Illustrated in this answer: https://stackoverflow.com/a/34111170/4495081
Deployments are done from separate, dedicated workspaces - one for each environment, based on the corresponding git branch (I never switch the branches in these workspaces). I just update the workspace corresponding to the desired environment to the version needed and copy-paste the deployment cmd from the workspace's cheat-sheet.
This model is IMHO CI/CD-ready and can easily be entirely automated.
For Python applications, you can set application in the app.yaml file. This allows you to use different data for each project. This is when you deploy using the appcfg.py command.
application: myproject
version: alpha-001
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /
script: home.app
If you don't want to change the application value in this file for each project, you can run the following:
appcfg.py -A <YOUR_PROJECT_ID> -V v1 update myapp/
https://cloud.google.com/appengine/docs/python/config/appref
If you do not specify the application in the file, use the --application option in the appcfg command when you deploy. This element is ignored when you deploy using the gcloud app deploy command.
Related
Recently tried to update my Gaelyk project (yes, it's old, but it works well and I still use it), but Google App Engine will no longer accept the update. The error message returned is "Deployments using appcfg are no longer supported. See https://cloud.google.com/appengine/docs/deprecations". The thing is, I never used appcfg to deploy my application; I used Gaelyk and Gradle. But obviously Gaelyk must have used appcfg under the covers.
I did download the replacement Google Cloud SDK, but this new tool is not similar at all to how Gaelyk and Gradle worked. Is there anything I can do to get Gaelyk to work anymore? Or is Gaelyk just dead and I need to rewrite my application (like in Node.js or something instead of Groovy).
This will be hard, however I will try to help you as possible. I think you may try to migrate it somehow to app.yaml configuration of GAE.
I am not sure what plugins are used in the project. From Gaelyk temple project I can see that it's using appengine-geb which, according to the documentation, behind the scenes, is using gradle-appengine-plugin (there is wrong link on this doc, but proper is bellow).
On the github of gradle-appengine-plugin I have found following.
There is a note:
NOTE: All App Engine users are encouraged to transition to the new
gradle plugin for their projects.
And in FAQ part there is following information:
How do I deploy with gcloud?
If you're using gcloud to deploy your application, the newest version of app deploy > doesn't support war
directories, you will need to provide it with an app.yaml OR you can
use the appengineStage task to create a directory that is deployable
in /build/staged-app
$ ./gradlew appengineStage
$ gcloud app deploy build/staged-app/app.yaml --project [app id]
--version [some version]
NOTES:
You must explicitly define all config files your want to upload
(cron.yaml, etc)
This does not work with EAR formatted projects.
I think the best option will be to migrate to new appenine plugin or if not possible try to implement is with gcloud app deploy command crating the config files manually (at least app.yaml). And for this migration I can provide you this document.
I hope you will manage somehow...
I can confirm that Serge's answer on the Gaelyk Groups site works; the same procedure that he figured out also worked for me. To summarize:
Run gradlew appengineRun as run previously with Gaelyk.
Copy all jar files inside the build\exploded-app\WEB-INF\lib folder into a \src\main\webapp\web-inf\lib folder (for me the new lib folder did not exist previously).
To deploy, use the new required gcloud tool, and instead of running gradlew appengineUpdate (which fails now), instead run
gcloud app deploy appengine-web.xml where that XML file can be found in your webapp/WEB-INF directory. I navigated to that directory to run the gcloud command, but you can use a relative path there if your working directory is elsewhere. (There are a number of optional flags associated with the gcloud app deploy command, but I didn't need any of them.)
Serge needed to use these instructions to convert datastore-indexes.xml to index.yaml and run gcloud app deploy index.yaml, however, I didn't need to do this because I had no datastores.
I want to test Python 3.7 on Google App Engine Standard.
I am afraid if I upload application with python37 old application will stop working and will be no revert to previous python27.
Can you share some experience if it possible to keep both runtime versions?
It is not clear for me.
I read Python 3.7 on Google App Engine - documentation but not found information.
You have 3 main options, ranked from easier to 'harder' to do:
Just deploy this test you want to do in a new project, and you don't have to worry about your old application getting replaced.
Deploy this test to a new service. Just add the line service: whatevername to the app.yaml file of your test, and deploy it as usual with $gcloud app deploy. Voilà, you have a new service.
You can deploy this test in your default service, but to avoid stopping the previous one, add the --no-stop-previous-version flag to your $ gcloud app deploy command. After that, you can split traffic between this test and your actual app.
To be honest, I would go with options 1 or 2, as they are far more simple to deploy, and you can later on delete the test project or delete the new service if you wish to.
I'm using Google AppEngine Flexible with python environment. Right now I have two services: default and worker that share the same codebase, configured by app.yaml and worker.yaml. Now I need to install native C++ library, so I had to switch to Custom runtime and added Dockerfile.
Here is the Dockerfile generated by gcloud beta app gen-config --custom command
FROM gcr.io/google-appengine/python
LABEL python_version=python3.6
RUN virtualenv --no-download /env -p python3.6
# Set virtualenv environment variables. This is equivalent to running
# source /env/bin/activate
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH
ADD requirements.txt /app/
RUN pip install -r requirements.txt
ADD . /app/
CMD exec gunicorn --workers=3 --threads=3 --bind=:$PORT aces.wsgi
Previously my app.yaml and worker.yaml each had it's own entrypoint: config that specified the command needed to be run to start the service.
So, my question is how can I use two different commands to start the services?
EDIT 1
So far I was able to solve this by rewriting CMD line in dockerfile for each deploy of each service. However, I'm not quite satisfied with this solution.
gcloud app deploy command has --image-url flag that allows to set image url from GCR. I haven't researched that yet, but it seems that I can just upload images to GCR and use the urls since don't change that often
Yes, as you mentioned, I think using the --image-url flag, is a good option here.
Specify a custom runtime.
Build the image locally, tag it, and push it to Google Container Registry (GCR)
then, deploy your service, specifying a custom service file, and specifying the remote image on GCR using the --image-url option.
Here's an example that accomplishes different entrypoints in 2 services that share the same code:
...this is assuming that the "flex" and not "standard" app engine offering is being used.
lets say you have a: project called my-proj
with a default service that is not important
and a second service called queue-processor which is using much of the same code from the same directory.
Create a separate dockerfile for it called QueueProcessorDockerfile
and a separate app.yaml called queue-processor-app.yaml to tell google app engine what i want to happen.
QueueProcessorDockerfile
FROM node:10
# Create app directory
WORKDIR /usr/src/app
COPY package.json ./
COPY yarn.lock ./
RUN npm install -g yarn
RUN yarn
# Bundle app source
COPY . .
CMD [ "yarn", "process-queue" ]
*of course i have a "process-queue" script in my package.json
queue-processor-app.yaml
runtime: custom
env: flex
... other stuff...
...
build and tag the docker image
Check out googles guide here -> https://cloud.google.com/container-registry/docs/pushing-and-pulling
docker build -t eu.gcr.io/my-proj/queue-processor -f QueueProcessorDockerfile .
push it to GCR
docker push eu.gcr.io/my-proj/queue-processor
deploy the service, specifying which yaml config file google should use, as well as the image url you have pushed
gcloud app deploy queue-processor-app.yaml --image-url eu.gcr.io/my-proj/queue-processor
Since the Dockerfile name cannot be changed, the only way to not have to modify the Dockerfile would be to store each service in its own, separate directory. Clean separation, each service has its own Dockerfile and/or startup configuration.
But this raises a question: how to deal with the code shared by multiple services? Using symlinks (which works great for sharing code across standard env services) doesn't work for the flexible env services, see Sharing code between flexible environment modules in a GAE project.
I see a few possible approaches, none really ideal, but maybe more appealing than what you currently have:
hard-link each and every shared source code file (since hardlinking directories is not possible). A bit tedious and error-prone, but you only have to do that once per file
package and publish your shared code as an external library, added to the requirements.txt file of each service using it
split the shared code in a separate repository and have a copy of that repository in each service using it (maybe as a git submodule if using git?). You just need to ensure at the service deployment time that the shared repository is pulled at the proper version - can be quite reliably done through automation. A bit more complicated if you have uncommited changes in this repo - you'd have to patch the same changes in all services.
have multiple copies of the Dockerfiles with different names which you simply copy over instead of always editing the same file. Symlinking instead of copying might work as well, since the symlink doesn't need to be followed outside of the service directory, if it's just replicated as a symlink it'll work.
So i had a very similar issue with my Java applications. We were looking to migrate from Heroku to GAE and were attempting to simulate the Heroku Procfile with GAE services. Effectively what we did was to create separate directories in our application src/main/appengine/web and src/main/appengine/worker where each directory conainted the app.yaml and Dockerfile specific to the process. Then using the mvn appengine:deploy capabilities, we specified the -Dapp.stage.dockerDirectory and -Dapp.stage.appEngineDirecory respectively for each service we wanted to deploy. Then using just some parameters we were able to basically script out parallel deployments of each service from the same code base. Not sure if this works in your situation, but it was very useful for us: Here are the two example commands in their entirety:
Web Process:
mvn appengine:deploy -Dapp.stage.dockerDirectory=src/main/appengine/web -Dapp.stage.appEngineDirectory=src/main/appengine/web -Dapp.stage.stagingDirectory=target/appengine-web -Dapp.deploy.projectId=${project-id} -Dapp.deploy.version=${project-version}
Worker Process:
mvn appengine:deploy -Dapp.stage.dockerDirectory=src/main/appengine/worker -Dapp.stage.appEngineDirectory=src/main/appengine/worker -Dapp.stage.stagingDirectory=target/appengine-worker -Dapp.deploy.projectId=${project-id} -Dapp.deploy.version=${project-version}
I have an app that uses the following command to deploy a static app generated by create-react-app:
gcloud app deploy --project MY_PROJECT -v dev
After that, my app is available on myproject.appstop.com.
But I don't figure out how to deploy this app using a diff URL for each environment. Like.: dev.myproject.appspot.com, stg.myproject.com and so on.
If you know or have other ideas about how to solve this please share your thoughts.
It depends on what you understand by environment. I would definitely recommend you to create different projects for the different environments (my-project-dev, my-project-staging, my-project-test, my-project-prod...)
If you just want to have a different url for your deployment, you are doing ok by using the -v (version parameter).
Once you've deployed again:
gcloud app deploy --project MY_PROJECT -v test
you'll have both versions accessible at:
dev.myproject.appspot.com
test.myproject.appspot.com
Also, check Dan's answer below as it contains very relevant info.
To complement #MarCialR's answer, you can use SSL as well, but with *-dot-* URLs like https://test-dot-myproject.appspot.com, see Targeted routing.
But personally I'm not a big fan of a version-based environment, it brings trouble, see Continuous integration/deployment/delivery on Google App Engine, too risky?
I chose to implement environments at app level - each environment as a different app, each with its own URL. See Advantages of implementing CI/CD environments at GAE project/app level vs service/module level?
Since I'm using a wildcard custom-domain SSL certificate I'm simply mapping the different apps (environments) to different hosts/subdomains in my custom domain - thus ensuring everything works as expected with a custom domain at every point in the CI/CD pipeline, no surprises in production.
I have a Google App Engine app (Go lang, if that matters) that I would like to deploy more than once, with slightly different setup. Think production vs. QA.
env_variables in app.yaml seemed promising, but it seems I can only have one such file. For example, I don't see a way to call "goapp deploy" with app-qa.yaml.
How can I tweak deployment configuration? Is it possible to have more than one app.yaml, without custom script that copies files to a directory and manipulates app.yaml? Any other way to configure this?
My preference is to have the delta between the staging/QA and production reflected in (and controlled via) the VCS (git in my case):
my main development happens on the main branch, where the staging environment config is reflected in the .yaml files' contents
my production branch is pulled off the main branch and contains the production environment config - the .yaml files are modified accordingly
When I perform a deployment I do it from a workspace based on the appropriate branch depending on which config I need to deploy.
The delta between the production branch and the main branch is pretty much just deployment config changes. Whenever I'm happy with the staging results and I'm ready to deploy in production I just sync the production branch to the main refpoint corresponding to the OK'd staging deployment and deploy.
Another possible approach is to directly use the SDK's appcfg.py tool, which is what goapp deploy ultimately invokes anyways:
goapp deploy wraps the appcfg.py python tool provided in the SDK. You
can also invoke this tool directly if you need greater control over
the deployment:
The appcfg.py tool allows you to actually have and use alternate .yaml files residing in the same app/module directory (you'd probably have to use that anyways if you go with multiple or non-standard module configurations, since auto-detection from the app's dir won't work anymore):
appcfg.py update app-qa.yaml
App Engine Modules could be used for such purposes.