I want to make sure that when I update my GAE site that it does not have any bad side effects for users who are currently using my site.
Does Google automatically make this a safe process, e.g. by:
complete all pending requests
temporarily delay new requests
update code
process new requests
Or does Google just terminate whatever is running and update the server code?
If the latter, any recommendations for doing a safe update? Any other precautions for tasks in queues?
I think that for some (usually quite short) time, your two versions are serving side-by-side. So some requests are on the new version, but some are still finishing up on the old version. Be sure to consider that when thinking about how your data structures will update.
Occasionally, I have received errors from a small number of users while deploying.
I think the safest way to do it is to deploy to a new 'version' that isn't the active one, then to change your default 'version' in the admin console. This ensures the quickest, cleanest, changeover.
Related
We have a service running on Google App Engine.
If that service does not receive a traffic for some time then all instances are killed and the next call takes a few additional seconds to start the application.
We are thinking about specifying a min_instances option in app.yaml to always keep at least one instance alive.
We deploy new versions of that service quite frequently and keeping old versions for some time. Those old versions are not serving traffic and kept just in case.
What we would like to do is to always keep at least one instance of default service version alive and leave all other non-default versions with default behavior – we want them to be scaled automatically to 0 instances if they do not receive any traffic.
I didn't find such option in the documentation (https://cloud.google.com/appengine/docs/standard/python3/config/appref#scaling_elements) and didn't come to any workarounds.
I am thinking about creating a cron job (https://cloud.google.com/appengine/docs/flexible/python/scheduling-jobs-with-cron-yaml) which will periodically "ping" only default version of my application periodically thus making it always asleep. But I am not sure if it is good solution.
Are there any better solutions to such case?
Thanks!
min_idle_instances config option seems to solve my problem.
Note following in the documentation: "This setting only applies to the version that receives most of the traffic" which is almost exactly my case:
automatic_scaling:
min_idle_instances: 1
I am having trouble with using Push Queues on Google App Engine's Flexible Environment (formally named, their Managed VM Environment). I am receiving numerous 404 Instance Unavailable (see picture below).
After a bit of sleuthing, I believe these errors may be because I am adding a task to a task queue, then deploying a new version of the Flexible VM instance. The taskqueue that I previously pushed is locked to the older instance, and can no longer run. Is this how taskqueues work with Flexible VM? If so, how does one use push taskqueues with the Flexible VM?
I was 90% done migrating to flexible env when I came across this same problem. After extensive research, I concluded there are three options:
REST API (experimental)
Use the beta REST API for task queues (this, as all other google APIs from flexible env, is external, so you need to deal with auth appropriately).
REST API reference: https://cloud.google.com/appengine/docs/python/taskqueue/rest/
Note, this is external and experimental. Find e.g. a java sdk without any meaningful documentation here: https://developers.google.com/api-client-library/java/apis/ (current version: https://developers.google.com/api-client-library/java/apis/taskqueue/v1beta2)
Compat runtime
Build your own flexible environment, based off a -compat runtime. This offers the old appengine api in a container suitable for the flexible env:
https://cloud.google.com/appengine/docs/flexible/custom-runtimes/build (look for images with "YES" in the last column)
e.g.: https://cloud.google.com/appengine/docs/flexible/java/dev-jetty9-and-apis
https://cloud.google.com/appengine/docs/flexible/java/migrating-an-existing-app
Note: I spent two weeks in blistered frustration pleading every God almighty help me get this to work, following container rabbit holes into the depths of Lucifer's soul and across unexplored dimensions. I eventually had to give in. I just can't get this to work to a satisfying degree.
Proxy service
Kind of a hacky alternative, but it gets the job done: create a very thin standard environment wrapper service which proxies tasks onto / off your queue. Pass them to your own app however you want. ¯\_(ツ)_/¯
Downside is you are now spinning up extra instances and burning extra minutes.
I ended up with a variation of this, where I'm using a proxy service in standard env, but just ported my eventual task handler to AWS Lambda (so it's completely off GAE). It's a different disaster, but a more manageable one.
Good luck!
I'm the tech lead and manager for this product.
There are two distinct answers to your question.
For the first, it seems like you have a version routing issue -- as you say, tasks cannot run against a VM because you launched a new version. By default, tasks are assigned to run on the version from which they were enqueued to avoid version mismatches. You should be able to override the version by re-configuring the target in your queue.yaml (or queue.xml). Documentation for that can be found here. You might also need to look at your
From a broader perspective, building a migration path away from standard/MVM-only support for task queues is currently our highest priority.
The replacement is Cloud Tasks, which exposes the same interface but can be used fully independently from App Engine. It exists in the same universe as AppEngine Task Queues, so you will be able to add tasks to existing queues (both push and pull). It is currently available in closed alpha. You can sign up to join the alpha here.
We strongly recommend against writing new code against the REST API. It is unsupported and the cloud tasks alpha is already substantially more feature complete.
I'm upvoting hraban's answer (he did wrestle with the devil after all) but providing an additional answer here.
Keep in mind that the Flexible Environment (Managed VMs) is still just a compute engine instance... with Google doing a good job of extracting features from AppEngine to make them reachable in a transparent manner. TaskQueues didn't quite make it. Keep a sharp eye on the cloud library--that's the mechanism by which the DataStore becomes usable (for Java go to http://googlecloudplatform.github.io/google-cloud-java/0.3.0/index.html). If you go to that link you can also select other languages. I have it on official word that TaskQueues are still on the roadmap (but no ETA).
As of now you can't use the REST api to enqueue onto PUSH queues. Now the way that I decided to tackle this problem was to use the REST API and create a PULL queue to put tasks in. Then I poll that queue inside an AppEngine service (i.e. module) and put it into a PUSH task queue. Why do I go to all that trouble? Because I need scheduled execution... which is a feature that TaskQueues alone can give you on AppEngine. So I package my task in an envelope and then unpack and re-push it into a task queue. Sounds crazy? It's been working reliably for me. Don't be scared off by the fact that the REST api is "alpha".
I will say if you're starting something new take a good look at the Pub/Sub API.
I have a restful angular app that is hosted on a AWS and I'm looking for a clean and quick deployment solution to put the new site live without taking down the previous. I don't have much DevOps experience so any advice would be great. The site is full RESTFUL so its just static pages.
I was looking at setting up a dokku with AWS plugin solution but was pretty sure its overkill and may not be able to detect my app because its just static pages (no node, rails, etc).
The best way to do this is to reconfigure the web server on the fly to point to the new application.
Install the new version of the app to a new location, update the web server config files to point to the new location, and reload the server.
For inflight requests, they will be satisfied by the old application, and all the new requests will hit the new application, with no down time between them save for the trivial delay when refreshing the web server (don't restart it, just tickle it to reload it's configuration files).
Similarly, you can do this solely at the file system, by installing the new app in a new directory parallel to the old one. Then:
mv appdir appdir.bak
mv appdir.new appdir
But this is not zero downtime, but it is a very, very short down time as the two inodes are renamed. Just ensure that both the new and old directories are on the same filesystem, and the mv will be instantaneous. The advantage is that you can trivially "undo" the operation in the same way.
There IS a window where you have no app at all. For a fraction of a second there will be no appdir, and you will serve up 404's for those few microseconds. So, do it when the system is quiet. But it's trivial to instrument and do.
We ended up going with TeamCity for our build/tests and deploying via Shipit.
https://github.com/shipitjs/grunt-shipit
https://www.jetbrains.com/teamcity/
Try to use git repo for live deployment https://danbarber.me/using-git-for-deployment/
A simple solution is to use a ELB. This will enable you to deploy a new instance, deploy the code, test it, update the ELB to switch traffic to the new instance and then you can then remove the old instance.
An easy solution to this is to always be running two instances, a production and a staging. These guys should be identical and interchangeable (because they are going to switch. Assign an elastic ip to your production. When it's time to update, copy the code onto the staging, make sure it's working, and then attach the elastic ip to staging. It is now production and production is now staging. This is not an ideal solution but it is very easy and the same principals apply to better solutions.
A better solution involves an elastic load balancer. Make sure you have 2 instances attached. When it is time to update, detach an instance, perform your update, make sure it is working and reattach it. Now you will have a brief point in time where the client could get either your new website or your old website. Detach the other old note, perform the update and reattach.
The fact of the matter is even if you just overwrite files on the live server there will only be a 10ms window or so where the client could get a new version of one file (e.g. the html) and the old version of another (e.g. the css). After that it will be perfect again.
I have a couple of questions about the App Engine Map Reduce API. First of all there's a mapreduce package in the SDK, and there's a separate mapreduce bundle here:
https://developers.google.com/appengine/downloads
Which one should I be using? Should I be using the bundle, or is the documentation out of date and I should actually use the SDK version?
Second I'd like to be able to run mapreduce's on a non-default version to make sure that the requests from the mapreduce don't interfere with user requests.
What's the best way to do this? Can I start the pipeline with a task queue, and set the target version of that queue to be my non-default version?
We recommend using the open source version of Map Reduce for GAE at http://code.google.com/p/appengine-mapreduce/
The stale bundle link in the docs is a bug. That'll get cleaned up soon.
A few of our SDKs have bits of MapReduce (for historic reasons), but the open source version is the way to go for now.
As for using a separate version, this is kind of "it depends". If you're thinking of interference in terms of competition for the processor, that's not likely to be a noticeable issue. Depending on queue processing rates you've set up, more instances of your app will be spun up to handle mapping tasks as needed. I'd try some experiments first. Make sure you have a problem before you invest time and effort solving it.
mapreduce can be start on a not default version. And after it starts, it will continue run on that version automatically.
In my case I just deploy the code on a non default version and trigger the mapreduce with version_id.app_id.appspot.com/path_to_start_a_job.
cron job can also trigger the mapreduce on non default version without problem.
I am interesting in hearing if others have addressed release management for Silverlight applications.
I have a business application that is to be released shortly andam concerned about how to "release" updates to this application. Typically this application's users will leave the application open all day (and potentially all night) without reloading it.
What if there is is need to release an change that includes an web service interface change? How can this be deployed w/o causing errors on the client side?
We have grown so used to deploying ASP.Net apps by just dropping the latest code on the server. My only idea currently involves a client version number and a periodic timer on to check for updates.
I would love to know what others have done before implementing this.
Thanks,
Mike
I just answered a question on how to make sure that .xap files are not cached by the browser, which might be of some help:
Prevent Silverlight xap from being cached by proxy server
But that's no use if the users never reload your application. In my own application this is not a problem since users will be automatically thrown out whenever we deploy an update to the web service. But I like your idea with the timer, I would go with that.
Stating the obvious but don't do anything to annoy your users. E.g. could they spend twenty minutes entering data, nip off to the coffee machine and return to click Submit to find the timer has expired, noticed an update and their work is lost due to a forced restart?
If so, and I admit this hasn't had a lot of thought, if e.g. you have to make changes to the web service that break the current release, could you have the new web service version side-by-side such that users don't get thrown out until the timer has expired and the unit of work is complete? Or is this also stating the obvious?
For server code, i.e. endpoints just do as per normal. for the xap's I think you have a few options depending upon how you handle communications. You could have request contain a version number and if the server has been updated then force some code to reload the client, bit lame, messy but do-able. Perhaps a cleaner solution would be to control the clients session, which presumably is part and parcel with requests to the backedn. When you deploy a new version you could invalidate the client session, perhaps forcing a page refresh with custom logic. If your protocol is push base you could send a command to the client to do what ever you want, for many systems that are on all day its likely that this infrastructure would exist (if u've build it nicely :)). For instance our service layer is abstracted away from the repositories models and view models, in our case we'd could send a logout or perhaps a specific command to kick in some custom logic on the client informing the application is being updated and to refresh your browser when done. Our shell is light weight so our modules (basically other xap's) can be updated in time for the refresh.
I would recommend you to use a solution like mentioned in App Arch Guide:
The Guide Chapter I mean see Deployment considerations.
Divide the application into logical
modules that can be cached
separately, and that can be replaced
easily without requiring the user to
download the entire application
again.
Version your components.
Have you considered keeping a WCF polling duplex channel going that alerts the app when it needs to reload? In addition, you can have your WCF calls direct to a virtual directory that contains 'interfaced' calls. For example:
Silverlight app hosted at "x.x.x.x\Default.aspx"
Silverlight talks to WCF at "x.x.x.x\Version2\DataPortal.svc"
DataPortal.svc talks to a GAC (or otherwise base) assembly that can identify what version can handle what calls.
This way, if you upgrade to "x.x.x.x\Version3\DataPortal.svc", you can still make calls against Version2, assuming those calls have code to convert them to a Version3 concept.
This helps in cases where your line of business app has dynamic xap downloading ('main', 'customer', 'inventory', etc.) and you want to release them independently.