We are developing a small e-commerce project for which we are using GAE as our web service backend but it's draining our financial resources. We studied our current infrastructure and how it was billed. That shows us that it's because of the instances we are using.
We have 7 services running which is using 12 instances as a whole.
Want to understand how to stop Google App Engine instances when not being used as it's the key contributor to our billing.
Services
UpdateFeed.yaml
application: ...
module: updatecategoryfeed
version: uno
runtime: python27
api_version: 1
instance_class: F1
threadsafe: true
automatic_scaling:
min_idle_instances: 6
max_idle_instances: automatic # default value
min_pending_latency: 30ms # default value
max_pending_latency: automatic
max_concurrent_requests: 50
handlers:
- url: /.*
script: FeedModule.FeedBuilder.update_category_feed
libraries:
- name: endpoints
version: 1.0
- name: webapp2
version: "latest"
- name: ssl
version: latest
All other services following the same structure. We have a total of 7 active services.
UPDATE 2
We decreased the auto scaling values as per our project requirements which min idle instances to 0 and max idle instance to be 1. The price dropped drastically. But we are still looking for the answer on how to stop an instance when not being used. For ex. the below graph shows an instance started on its own without any activity and is being billed.
Drop your min_idle_instances configuration in the automatic_scaling section - those configs effectively represent instances running at all times - also called resident instances for that reason.
Their role is not to handle traffic regularly. They only handle overflow traffic for short periods of time when the traffic exceeds the capacity of the running dynamic instances to keep the latencies low, while GAE spins up additional dynamic instances (which are the ones actually handling the bulk of the traffic).
Just in case it's not clear - resident instances aren't going away - it is their job to remain alive even if there is no activity on the site.
You can start and stop GAE services, configuring to manual_scaling and use GAE API to start and stop.
const proj = '' // project id here
const serv = 'default' // or your service id
const vers = '' // version id here
const {google} = require('googleapis')
const appengine = google.appengine ({version: 'v1', auth: auth})
// To retrieve Status:
appengine.apps.services.versions.get ({
appsId: proj,
servicesId: serv,
versionsId: vers,
view: 'BASIC',
})
.then ()
.catch ()
// To Change Status:
newStatus = 'STOPPED' // or 'SERVING'
appengine.apps.services.versions.patch ({
appsId: proj,
servicesId: serv,
versionsId: vers,
updateMask: 'serving_status',
requestBody: {
serving_status: newStatus
}
})
.then ()
.catch ()
Related
I am trying to get the sessionID of a user from express-session when the user opens up a WebSocket connection. The code below shows how this is done. The problem is that when I deploy my server to google app engine, I can no longer see the session cookie when authorizing the socket connection. It works fine when running locally, so google app engine must be doing something to remove the cookie? I'm not too familiar with the google app engine so any help would be greatly appreciated.
I've tried many different solutions I found online such as passport.socketio from npm, many of which I believe implement the same solution that I ended up writing myself.
Edit 1:
To be more specific, data.headers.cookies contains an io, connect.sid and express.sid cookie when running from localhost. When running on the google app engine, it only contains the io cookie. (End of edit)
Edit 2:
I thought it could be an issue with having different instances of my server due to google's automatic scaling, so I've changed my setting in app.yaml to ensure that wont happen:
network:
session_affinity: true
manual_scaling:
instances: 1
(End of edit)
Code:
var session = require('express-session');
var MemoryStore = require('memorystore')(session);
var store = new MemoryStore({
checkPeriod: 86400000
}); // Likely to switch to RedisStore
app.use(session({
store: store,
secret: 'jpcs-0001080900DRXPXL',
saveUninitialized: false,
resave: true,
key: 'express.sid'
}));
// ...
io.set('authorization', (data, accept) => {
if (data && data.headers && data.headers.cookie) {
console.log('COOKIES:');
console.log(data.headers.cookie);
cookies_str = data.headers.cookie;
cookies_arr = cookies_str.split(';');
cookies = {};
for (index in cookies_arr) {
cookie = cookies_arr[index].split('=');
if (cookie.length != 2) continue;
key = cookie[0].replace(/ /g,'');
val = cookie[1];
cookies[key] = val;
}
if (!cookies['express.sid']) accept('User not signed in', null);
// HERE IS MY PROBLEM:
// When running from google app engine cookies['express.sid'] is null
sessionId = cookies['express.sid'].split('.')[0].substring(4);
data.sessionId = sessionId;
store.get(sessionId, (err, session) => {
if (!err && session) {
data.session = session;
accept(null, true);
}
else if (err) accept('Could not get session', false);
else if (!session) accept('Could not get session', false)
});
}
accept('Could not get session', false);
});
So I expect cookies['express.sid'] to contain the sessionID in it, but on google app engine it is null.
App.yaml
runtime: nodejs10 # For Node.js 8, use runtime: nodejs8
#instance_class: F2
#env_variables:
#BUCKET_NAME: "example-gcs-bucket"
handlers:
- url: .*
secure: always
redirect_http_response_code: 301
script: auto
network:
session_affinity: true
manual_scaling:
instances: 1
It's more likely that your application cannot retrieve the user from the session store when running on GAE due to a network or app configuration. Some more info on which session store you are using, the configuration for it in express, and the app.yaml would be more helpful in solving this. Its unlikely that GAE would be removing the cookie from the HTTP response headers.
I found that in order to use redis with express/express-session I needed to set the network name in app.yaml.
network:
name: default
Unfortunately it doesn't seem like you can use this for standard environments, so I also had to change:
runtime: nodejs
env: flex
And also needed to specify an engine in package.json when switching to a flexible environment.
"engines": {
"node": "10.x"
}
I set a cache-control on my server of 1 year.
How to say to the AppEngine "clear !" to take a new version from the server ?
The configuration is Flex custom environment
runtime: custom
env: flex
env_variables:
writecontrolEnv: 'prod'
handlers:
- url: /.*
script: this field is required, but ignored
service: gateway-prod
automatic_scaling:
min_num_instances: 1
max_num_instances: 2
resources:
cpu: 1
memory_gb: 2
disk_size_gb: 10
skip_files:
- node_modules/
network:
instance_tag: gateway
Assuming that your app is the one serving the static files then the cache parameters sent by the server are controlled by your application code. Which means that once you deploy a new version with updates parameters the server will send the updated values.
But the problem is that caching is actually performed by the client (or some middle-man network device), so the end user will not reach to the server until the (very long in your case) cache expiration time is reached, so it won't see the update until then.
You can try to clear your browser cache, hoping that the browser was the one doing the cache-ing.
To prevent such occurrences in the future you may want to choose a shorter cache expiration time or use some cache busting technique like this one.
Using Google App Engine, I have an application myapp as a default service that adds a task to a task queue and launch a background service worker called optimize. Although myapp is running fine, unfortunately I always see a POST 404 error in the myapp log when the AppEngine task queue tries to launch the URL /optimize-dot-myapp.appspot.com/index.php/optimize. Of course because of the 404 error the task queue keeps retrying. My current optimize.yaml file contains the following. Any thoughts?
# optimize.yaml configuration for Google App Engine
# Full details at: https://cloud.google.com/appengine/docs/php/config/appref
runtime: php55
api_version: 1
service: optimize
handlers:
# Serve php scripts.
- url: /index.php/optimize
script: index.php/optimize
The default app.yaml file contains the following:
# app.yaml configuration for Google App Engine
# Full details at: https://cloud.google.com/appengine/docs/php/config/appref
runtime: php55
api_version: 1
handlers:
# Serve php scripts.
- url: /(.+\.php).*
script: \1
- url: /
script: index.php
# All URLs beginning with /assets are treated as paths to
# static files in the assets/ directory.
- url: /assets
static_dir: assets
In case it's useful, the optimize worker is started in the default task queue with the following PHP:
// Start the background worker
// API details: https://cloud.google.com/appengine/docs/php/refdocs/classes/google.appengine.api.taskqueue.PushTask
$url = '/optimize-dot-myapp.appspot.com/index.php/optimize';
$task = new PushTask($url, $param);
$task_name = $task->add();
You have collisions in the URL path patterns. The /index.php/optimize path matches both the /index.php/optimize url pattern from optimize.yaml and the /(.+\.php).* pattern from app.yaml. Probably the request ends up in the default service instead of the optimize one. Easy to confirm: check the app logs, you can select a specific service and you'll see which service got the request.
I would add a dispatch.yaml file to clarify things and eliminate the possibility of ambiguous routing (no need to specify the default module, anything not matching dispatch rules is sent to the default module):
application: my_app
dispatch:
- url: "*/optimize/*"
module: optimize
Then adjust the url patterns accordingly in optimize.yaml (they should all start with /optimize):
- url: /optimize/index.php
script: index.php
Note: the index.php file mentioned above would be in the optimize service dir, not in the default service one. Assuming here that each service has its own dir, as mentioned in Can a default service/module in a Google App Engine app be a sibling of a non-default one in terms of folder structure?
And in the task enqueueing code the url should only contain the request path, not the hostname (which is interpreted as part of the path, thus causing a mismatch with the handler's url pattern). You want:
$url = '/optimize/index.php';
In Google App Engine, applications define task queues in a configuration file called queue.yaml. You can use queue.yaml to configure both push queues and pull queues.
The following a basic example that defines a named queue and overrides the default processing rate:
queue:
- name: my-push-queue
rate: 1/s
The following is a more complex example of a queue.yaml configuration that demonstrates setting up task retries and modifying the default processing rate.
queue:
- name: fooqueue
rate: 1/s
retry_parameters:
task_retry_limit: 7
task_age_limit: 2d
- name: barqueue
rate: 1/s
retry_parameters:
min_backoff_seconds: 10
max_backoff_seconds: 200
max_doublings: 0
- name: bazqueue
rate: 1/s
retry_parameters:
min_backoff_seconds: 10
max_backoff_seconds: 200
max_doublings: 3
Please read this documentation for further details
I'm refactoring an existing codebase. I switched from using the appcfg.py to using the gcloud command, which seemed to go fine. Our entire codebase was running on one default frontend instance, which I'm now trying to break into services. To start, I created one "worker" backend service, and I'm using a cron job to test.
I can see the worker in the console, but no instance is started. The logs for that service are rapidly flooded with 404's to /_ah/start. I've tried manual and basic scaling. The documentation states that it's okay not to have a startup script, and that a 404 at that endpoint is considered success. However, the instance is not starting.
Logs
worker.yaml
service: worker
runtime: python27
api_version: 1
instance_class: B2
manual_scaling:
instances: 1
threadsafe: false
handlers:
- url: /work/.*
script: worker.app
secure: always
login: admin
worker.py
import webapp2
import handlers
config = {
#...
}
app = webapp2.WSGIApplication([
webapp2.Route(
'/work/test<:/?>',
handlers.Test,
methods=['GET'],
),
], debug=True, config=config)
dispatch.yaml
dispatch:
- url: "*/work/*"
module: worker
I have a module that works fine when I push it to app engine. When it works it logs stuff nicely and the logs are accessible in the console logs viewer. But then after a while it just stops working and when I try to access any url give me 500 server errors with no info (it just says waiting 30 seconds might be a good idea). When this happens nothing gets logged for requests.
If I restart the module (by pushing my code to app engine) then it works for a little while again.
The module is running a Pyramid app and the configuration file looks a little something like:
application: my_app
module: my_module
version: dev
runtime: python27
api_version: 1
threadsafe: false
instance_class: B2
basic_scaling:
max_instances: 2
idle_timeout: 10m
handlers:
- url: /actions/.*
script: my_module.application
login: admin
- url: /.*
script: my_module.application
builtins:
- appstats: off
libraries:
- name: webob
version: latest
- name: setuptools
version: latest
includes:
- mapreduce/include.yaml
I think what is happening is that it's hitting the idle timeout and shutting down. I need requests to the module to turn it back on again. How do I do that?
Let me know if you need more info, I'm an app engine noob at this stage. Any help would be greatly appreciated.
When a module start, App Engine call the url /_ah/start. You mustn't handle this request.
In you my_module.application you need to add in the handler who match this request :
def get(self):
# Let module start
if "X-Appengine-Cron" in self.request.headers or "X-AppEngine-TaskName" in self.request.headers or "X-Appengine-Failfast" in self.request.headers:
return
Even if you hit the idle timeout, AppEngine will spin a new instance when new request is coming in.
Use Cloud Debugger to inspect the state of your application. The debugger makes it easier to view the application state and understand what happens after your app has been running for while.
Check the docs on startup state https://cloud.google.com/appengine/docs/python/modules/#Python_Instance_states
you current default handler /.* should be able deal with a /_ah/start if youe handler can gracefully deal with a 404.
Thats how I handle startups. Goes through the main handler which can deal with non existent url requests using the default pyramid not found.
I have a config.add_notfound_view(notfound) registered.