I have around 10 microservices. Usually I use a script that loops through their folders (each of which contain an app.yaml for that service) and use gcloud app deploy to deploy it. However, is it possible to speed this up by using gcloud app deploy in say, 10 different terminals, one for each microservice? Will GAE processs the deploys in parallel or will this cause any issues?
Note: I am using App Engine flex, and from what I've been told, Standard environment deploys are a lot faster than deploys to Flexible environment. Regardless, switching to standard is not possible.
I tried deploying a couple of services on different tabs of the Cloud Shell Terminal and it works, so it appears to be possible but you should test that with all your microservices to see if this is not a bigger load than app engine could handle (which shoudn't be, but just in case).
However, it felt not worth the effort, since you have to lose time by opening up a new tab, waiting for it to be setup and typing the commands all over again. I did this manually, so if you are able to automate that opening of tabs into a script, it could be worth it.
Also, as you can see on this community post, another way to do a faster deploy is to deploy to another "version" of your app, then when all modules are deployed, do a very fast version switch.
NOTE: Not sure if you are doing this already, but an alternative is to stack the deployment of your services into a single command on Cloud Shell like this:
gcloud app deploy ~/my_app/app.yaml ~/my_app/another_service.yaml
Turns out it does work. I deployed 6 microservices in parallel, and it took < 10 mins for all of them to get deployed, compared to ~60 mins (average of 10 mins each for each service)!
For parallelizing it, I used Azure Devops pipelines, and since each job runs in parallel, all I had to do was create a job for each microservice, then run gcloud app deploy in each of the jobs to deploy them.
Related
It takes so long to deploy an app to App Engine by this command
gcloud app deploy
After I updated my source code, even if it's just one line of change, I had to issue this command again to upload the changes to App Engine. Again it will take a long time to deploy for just one line of code change. Is there a way to quickly update the code on App Engine instead of doing this time consuming deploy again?
I am looking for options that can speed up the code update after it's first deployment to App Engine. Since I am using flex, would using standard make it faster to update the code? Is it true that there is no way around it on App Engine and the only way to do it is to use another service such as Compute Engine?
Yes, standard env app deployment in such context is much faster, because it only involves uploading the modified files and restarting the instances.
But using standard env is often not an option for other reasons. The potential duplicate post I referenced contains itself references to both explanations and suggestions for potential speed improvements. I don't see a point in duplicating them.
The most common reasons for choosing flex env:
language support
restrictions of the standard env sandbox
See also: Google App Engine Standard or Flexible environment?
(Sep 2021): First of all, the issue you describe for App Engine (flexible) does not apply to App Engine (standard), which is almost a completely different product. (For those who are new, here is a comparison b/w both products.) This question is a few years old, and there are a few more options available to you now:
App Engine standard: If your app can run in Python 3.7+, Java 11, Go 1.12+, PHP 7, Node.js 10+, or Ruby 2.5+, then this is the obvious place to go. A deploy should take less than 1 minute.
Cloud Functions: If you don't have an entire app or your app is made up of one function/route or is otherwise short & simple, consider turning it into a microservice for Cloud Functions. Its deployment time is also less than a minute.
Cloud Run: If you're using App Engine flexible because your app is containerized, consider Cloud Run, which runs containers in a serverless way. A big bonus is that a Dockerfile is optional (due to Cloud Buildpacks). It has taken many GAE flexible use cases. Deployment is also less than a minute. If interested, I produced a pair of videos on how to migrate from App Engine standard to Cloud Run, using Docker as well as no Docker. At some point, I'll do one on transitioning from GAE flexible.
App Engine flexible: If none of the options above work, requiring you to stay on GAE flexible, then yes, you'll have to deal with its longer deployment time (due to the complexities of the system and all the resources it needs to spin-up & manage on your behalf). If you have a container and Cloud Run isn't an option, also consider GKE Autopilot or GCE MIGs (managed [VM] instance groups).
Trying out new flexible app engine runtime. In this case a custom Ruby on Rails runtime based on the google provided ruby runtime.
When firing of gcloud preview app deploy the whole process takes ~8 minutes, most of which is "updating service". Is this normal? And more importantly, how can I speed it up?
Regards,
Ward
Yes, that is totally normal. Most of the deployment steps happen away from your computer and are independent of your codebase size, so there's not a lot you can do to speed up the process.
Various steps that are involved in deploying an app on App Engine can be categorized as follows:
Gather info from app.yaml to understand overall deployment
Collect code and use the docker image specified in app.yaml to build a docker image with your code
Provision Compute Instances, networking/firewall rules, install docker related tools on instance, push docker image to instance and start it
Make sure all deployments were successful, start health-checks and if required, transfer/balance out the load.
The only process which takes most of time is the last part where it does all the necessary checks to make sure deployment was successful and start ingesting traffic. Depending upon your code size (uploading code to create container) and requirements for resources (provisioning custom resources), step 2 and 3 might take a bit more time.
If you do an analysis you will find that about 70% of time is consumed in last step, where we have least visibility into, yet the essential process which gives app-engine the ability to do all the heavy lifting.
Deploying to the same version got me from 6 minutes to 3 minutes in subsequent deploys.
Example:
$ gcloud app deploy app.yaml --version=test
Make sure you check what is in the zip it's uploading (it tells you the location of this on deploy), and make sure your yaml skip_files is set to include things like your .git directory if you have one, and node_modules
Note that the subsequent deploys should be much faster than 8 mins. It's usually 1 minute or less in my tests with Node.js on App Engine Flex.
As suggested above by #ludo you could use in the meantime Google App Engine Standard instead of Flex. Which, takes approximately ~30-50 seconds after the first deployment.
You can test GAE Standard by running this tutorial, which doesn't require a billing account:
https://codelabs.developers.google.com/codelabs/cloud-app-engine-springboot/index.html#0
And agreed. this doesn't address GAE Flex but gives some options to accelerate during development.
Just fire this command from root directory of app.yaml
From shell visit directory of app.yaml then run gcloud app deploy
It will be uploaded within few seconds.
We have a development,uat and production environment for our angular/java/mongodb app. At the moment we are moving our deploying the same build to dev and uat. The problem with that is that some things are not finished and basically not ready to be deployed to uat. What is a good way of managing/selecting which commits/changesets are to be deployed to uat/prod? If I know which commits I want is how can I accomplish this?
The use case we have is that some builds need to be tested (by other jobs) before we know if they are good enough to deploy / release.
We use the "Promoted Builds Plugin" for that. There is a manual promotion process as well.
I have developed an app in Twilio which I would like to run from the cloud. I tried learning about AWS and Google App Engine but am quite confused at this stage:
I have 2 questions which I hope to get your help on:
1) How can I store my scripts and database in the cloud? Right now, everything is running out of my local machine but I would like to transfer the scripts and db to another server and run my app at a predetermined time of day. What would be the best way to do this?
2) How can I write a batch file to run my app at a predetermined time of day in the cloud?
I understand this does not have code, but I really hope someone can point me to the right direction. I have spent lots of time trying to understand this myself but still am unsure. Tks in adv.
Update: The application is a Twilio app that makes calls to people, the script simply applies an algorithm to make calls in a certain fashion and the database is a mysql db that provides the details of people to be called.
This is quite difficult to provide an exact answer without understanding what is the application, what is the DB or what is the script that you wish to run.
I can give you a couple of ideas that might be helpful in such cases.
OpsWorks (http://aws.amazon.com/opsworks/) is a managed service for managing applications. You can define your stack (multiple layers like web, workers, DB...) and what are the chef recipes that should run in various points in the life of the instances in each layer (startup, shutdown, app deployment or stack modification..). Then you can use the ability to add instances to each layer in specific days and hours, to implement the functionality of running at predetermined times as you requested.
In such a solution you can either have some of your instances (like DB) always on, or even to bootstrap them using the chef recipes every day, with restore from snapshot on start and create snapshot on shutdown.
Another AWS service that you use is Data Pipeline (http://aws.amazon.com/datapipeline/). It is designed to move data periodically between data sources, for example from a MySQL database to Amazon Redshift, the Data warehouse service. But you can use it to trigger scripts and run random shell scripts that you wish (http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-shellcommandactivity.html), and schedule it to run in various conditions like every hour/day or specific times (http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-concepts-schedules.html).
A simple path here would be just to create an EC2 instance in AWS, and put the components needed to run your app there. A thorough walk through is here:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/get-set-up-for-amazon-ec2.html
Essentially you will create an EC2 virtual machine, which you can for most purposes treat just like any other Linux server. You can install MySQL on it, copy your script there, and run it. Of course whatever container or support libraries your code requires will need to be installed as well.
You don't say what OS you are using locally, but if it is Mac or Linux, you should be able to follow almost the same process to get your script running on an EC2 instance that you used on your local machine.
As you get to know AWS, there are sophisticated services you can use for deployment, infrastructure orchestration, database services, and so on. But just to get started running a script from a virtual machine should be pretty straightforward.
I recently developed a Twilio application using Ruby on Rails for the backend and found Heroku extremely simple to setup and launch. While Heroku does cost more than AWS, I found that the time I saved using Heroku more than made up this. As an early stage startup, we wanted to spend our time developing important features, and not "wasting" time optimizing our AWS cloud.
However, while I believe Heroku is ideal for early-stage websites/startups I do believe hosting should be reevaluated once a company reaches a certain size. At some point it becomes economically viable to devote resources into optimizing an AWS cloud solution because it will be cheaper than Heroku in the long run.
I have this application that will be run in a local network where a number of devices should interact with a database. I could use xampp and go for CherryPy or any other Python framework (Python is usually my choice) but it is the sum of a lot of different things: Python, Apache, MySQL... With GAE, which I have previously used in a number of applications successfully, I feel everything is neatly packed in a single box. Thay may not be true, but using the Google App Engine Launcher to create a local working copy of an app couldn't be easier.
But is it reliable? Should it be used like that? I know it's intended for development, so I'm unsure about using it as a local server in production. A few versions ago there even was this nasty bug that flushed the local datastore from time to time. But it seems that they fixed it and now data persists.
Would you recommend GAE for an application running in a local network or should I stick to LAMP (P for Python)?
Other alternative is http://code.google.com/p/appscale/.
May be you can check the the project TyphoonAE. I think it is exactly what you need.
The TyphoonAE project aims at providing a full-featured and productive
serving environment to run Google App Engine (Python) applications. It
delivers the parts for building your own scalable App Engine while
staying compatible with Google's API.