We are starting a new project that requires two main components:
Backend for task management, e.g retrieve a task from a queue and according to some specific logic validate it.
Run a real compiler on that specific task and create an executable that an end user should receive.
We love app engine, however the second part will require a concrete instance where an actual compiler will have to be installed, app engine is not capable here. We were thinking to mix both app engine and aws instances to accomplish the task (part 1 will be app engine and part 2 will be aws).
All of our senses say it's a bad idea:
unneeded traffic between the two providers, someone needs to pay for that unfortunately.
We'll have to deal with two systems, two deployments process, each system has its own quirks --> double the work.
But we love app engine.
Does anyone has any experience in combining the two systems? any recommendations ?
There's no reason why what you suggest won't work, especially if you separate your concerns well, by exposing a clean 'compiler' interface on AWS or a similar service. Yes, you will have to pay for traffic between the two services, but this is unlikely to be substantial. If you are serving up the end result to the user, you can link them directly to AWS, rather than fetching it with your app first.
AWS's EC2s are literally just vanilla linux boxes in the sky. I would also throw out the suggestion of just moving to it completely. Porting your system over may be easier than it sounds if you're unix savvy.
Related
I'm new to cloud programming but it's my understanding that the server I'm building can be run in multiple regions as multiple "instances" to improve user experience. In other words, my server code is just running on several different machines at once, all independently of each other. However, things like a database server for example should only be run as a single instance no matter how many server instances there are.
Is there a way to do this using Google App Engine? More specifically, is there a way to categorize portions of the project as scalable and others as non-scalable? My initial plan was simply to make two different projects - one which scales automatically and one which does not scale - and have them communicate through network requests. This could (potentially?) have the added benefit of spreading the resources used by my project across multiple cloud projects, reducing the per-project usage for billing purposes.
I'd love to know if I'm on the right track, or if what I'm doing is over-complicating things.
Yes, this is possible, you will need to create 2 separate App Engine service and set one with automatic scaling and the other with manual scaling and setup how many instances you want for that service up and running all the time. You can read this documentation for more details on the types of scaling and this documentation on how to set this up in your app.yaml file.
That being said I don't think this will reduce your cost. In fact the opposite might happen, as App Engine is designed to reduce wasted resources as much as possible with auto scaling, and if you were to use Manual Scaling with a bigger load of instances than what you actually need you will be charged more for that, so you need to consider this into the design of your infra, I also recomment taking a look at the App Engine pricing documentation.
Sorry, if this is a naive question, but i've watched bunch of talks from google's staff and still don't understand why on earth i would use AE instead of CF?
If i understood it correctly, the whole concept of both of these services is to build "microservice architecture".
both CF and AE are stateless
both suppose to execute during limited period of time
both can interact with dbs and other gcp apis.
Though, AE must be wrapped into own server. Basically it utilizes a lot of complexities on top of the same capabilities as CF. So, when should i use it instead of CF?
Cloud Functions (CFs) and Google App Engine (GAE) are different tools for different jobs. Using the right tool for the job is usually a good idea.
Driving a nail using pliers might be possible, but it won't be as convenient as using a hammer. Similarly building a complex app using CFs might be possible, but building it using GAE would definitely be more convenient.
CFs have several disadvantages compared to GAE (in the context of building more complex applications, of course):
they're limited to Node.js, Python, Go, Java, .NET Core, and Ruby. GAE supports several other popular programming languages
they're really designed for lightweight, standalone pieces of functionality, attempting to build complex applications using such components quickly becomes "awkward". Yes, the inter-relationship context for every individual request must be restored on GAE just as well, only GAE benefits from more convenient means of doing that which aren't available on CFs. For example user session management, as discussed in other comments
GAE apps have an app context that survives across individual requests, CFs don't have that. Such context makes access to certain Google services more efficient/performant (or even plain possible) for GAE apps, but not for CFs. For example memcached.
the availability of the app context for GAE apps can support more efficient/performant client libraries for other services which can't operate on CFs. For example accessing the datastore using the ndb client library (only available for standard env GAE python apps) can be more efficient/performant than using the generic datastore client library.
GAE can be more cost effective as it's "wholesale" priced (based on instance-hours, regardless of how many requests a particular instance serves) compared to "retail" pricing of CFs (where each invocation is charged separately)
response times might be typically shorter for GAE apps than CFs since typically the app instance handling the request is already running, thus:
the GAE app context doesn't need to be loaded/restored, it's already available, CFs need to load/restore it
(most of the time) the handling code is already loaded; CFs' code still needs to be loaded. Not too sure about this one; I guess it depends on the underlying implementation.
App Engine is better suited to applications, which have numerous pieces of functionality behaving in various inter-related (or even unrelated) ways, while cloud functions are more specifically single-purpose functions that respond to some event and perform some specific action.
App Engine offers numerous choices of language, and more management options, while cloud functions are limited in those areas.
You could easily replicate Cloud Functions on App Engine, but replicating a large scale App Engine application using a bunch of discrete Could Functions would be complicated. For example, the backend of Spotify is App Engine based.
Another way to put this is that for a significantly large application, starting with a more complex system like App Engine can lead to a codebase which is less complex, or at least, easier to manage or understand.
Ultimately these both run on similar underlying infrastructure at Google, and it's up to you to decide which one works for the task at hand. Furthermore, There is nothing stopping you from mixing elements of both in a single project.
Google Cloud Functions are simple , single purpose functions which are fired while watching event(s).
These function will remove need to build your own application servers to handle light weight APIs.
Main use cases :
Data processing / ETL : Listen and respond to Cloud Storage events, e.g. File created , changed or removed )
Webhooks : Via a simple HTTP trigger, respond to events originating from 3rd party systems like GitHub)
Lightweight APIs : Compose applications from lightweight, loosely coupled bits of logic
Mobile backend: Listen and respond to events from Firebase Analytics, Realtime Database, Authentication, and Storage
IoT: Thousands of devices streaming events and which in-turn calls google cloud functions to transform and store data
App Engine is meant for building highly scalable applications on a fully managed serverless platform. It will help you to focus more on code. Infrastructure and security will be provided by AE
It will support many popular programming languages. You can bring any framework to app engine by supplying docker container.
Use cases:
Modern web application to quickly reach customers with zero config deployment and zero server management.
Scalable mobile backends : Seamless integration with Firebase provides an easy-to-use frontend mobile platform along with the scalable and reliable back end.
Refer to official documentation pages of Cloud functions and App Engine
As both Cloud Functions and App Engine are serverless services, this is what I feel.
For Microservices - We can go either with CF's or App Engine. I prefer CF's though.
For Monolithic Apps - App engine suits well.
Main differentiator as #Cameron points out, is that cloud functions reliably respond to events. E.g. if you want to execute a script on a change in a cloud storage bucket, there is a dedicated trigger for cloud functions. Replicating this logic would be much more cumbersome in GAE. Same for Firestore collection changes.
Additionally, GAE’s B-machines (backend machines for basic or manual scaling) have conveniently longer run times of up to 24 hours. Cloud functions currently only run for 9 minutes top. Further, GAE allows you to encapsulate cron jobs as yamls next to your application code. This makes developing a server less event driven service much more clean.
Of course, the other answers covered these aspects better than mine. But I wanted to point out the main advantages of Cloud Functions being the trigger options. If you want functions or services to communicate with each other, GAE is probably the better choice.
I'm trying to build a chat application on GAE in JAVA . I have the need to keep count of all online users and their networks (chat rooms of some sort) , and this info needs to be updated constantly .
I have (wrongly?) assumed that I can just use Java's SerlvetContext and Set/Get Attribute methods to update online\offline users and share that information with all servlets . As I have come to know(with lovely bugs) , since GAE is distributed\cloud service , it doesn't effectively implement ServletContext.setAttribute - meaning is that my app probably runs on more than one JVM , and information on ServletContext is shared only between servlets belonging to the same JVM.
This is a huge problem for me , of course .
Several Questions -
1)Does ServletContext indeed won't work properly on GAE?
2)Is GAE a bad choice for beginner web developpers like myself ? It seems to me that I always find new problems and things that don't correspond with Servlet\JSP rules. Since it is hard enough for a beginner to learn Servlets , maybe GAE isn't the right choice?
3)How then can I share information between Servlets ?
If you're really just trying to learn Java EE for your own purposes I would probably avoid GAE for the reasons you mention. It's a perfectly good service, but yes, it has its own set of caveats that might get in the way of your learning. You might be better off just spinning up an EC2 instance for your purposes.
That said - you are correct, AppEngine will spin up and down instances to serve requests. If you want shared state you should use memcache which is shared across instances, but you have to manage access to the memcache objects for the possibility of multiple users writing to it at the same time.
In google app engine, application state is usually shared between instances using the datastore. As your requirement is more real-time and might not behave nicely using polling you should use the Channel API (perhaps in addition to the datastore):
https://developers.google.com/appengine/docs/java/channel/
Quoting from that page:
The Channel API creates a persistent connection between your
application and Google servers, allowing your application to send
messages to JavaScript clients in real time without the use of
polling. This is useful for applications designed to update users
about new information immediately.
We're going to develop a near-realtime event-driven application (backend and bunch of mobile clients).
I think Akka (http://akka.io) is more than suitable for this. However, my collegaue wants to use Google App Engine and its async features. I'm not convinced it's the best approach to take, I wonder if we can somehow meld those two things together. I can't find any solid contemporary info via Google.
The Channel API may be useful. However, the main limitation you may face using app engine are transactional writes to the datastore, because an entity group (parent entity and its children) can only support one to ten writes per second.
It is notable that the new Go support for app engine supports actor-style programming with goroutines. When datastore or other ops block then other goroutines run. It would be nice if someone could do this for scala and one of the actor variants. The new backend system allows this style to be used in a long running way I presume.
Unrelatedly, on the issue of writing to an entity group. I write a record that might be already there (same key_name), and now I'm wondering if I should read it first to check.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
What alternatives are there to GAE, given that I already have a good bit of code working that I would like to keep. In other words, I'm digging python. However, my use case is more of a low number of requests, higher CPU usage type use case, and I'm worried that I may not be able to stay with App Engine forever. I have heard a lot of people talking about Amazon Web Services and other sorts of cloud providers, but I am having a hard time seeing where most of these other offerings provide the range of services (data querying, user authentication, automatic scaling) that App Engine provides. What are my options here?
AppScale
AppScale is a platform that allows users to deploy and host their own Google App Engine applications. It executes automatically over Amazon EC2 and Eucalyptus as well as Xen and KVM. It has been developed and is maintained by AppScale Systems. It supports the Python, Go, PHP, and Java Google App Engine platforms.
http://github.com/AppScale/appscale
In the mean time...
...it is amost 2015 and it seems that containers are the way to go forward. Alternatives to GAE are emerging:
Google has released Kubernetes, container scheduling software developed by them to manage GCE containers, but can be used on other clusters as well.
There are some upcoming PaaS on Docker such as
http://deis.io/
http://www.tsuru.io/
even Appscale themselves are supporting Docker
Interesting stuff to keep an eye on.
I don't think there is another alternative (with regards to code portability) to GAE right now since GAE is in a class of its own. Sure GAE is cloud computing, but I see GAE as a subset of cloud computing. Amazon's EC2 is also cloud computing (as well as Joyent Accelerators, Slicehost Slices), but obviously they are two different beasts as well. So right now you're in a situation that requires rethinking your architecture depending on your needs.
The immediate benefits of GAE is that its essentially maintenance free as it relates to infrastructure (scalable web server and database administration). GAE is more tailored to those developers that only want to focus on their applications and not the underlying system.In a way you can consider that developer friendly. Now it should also be said that these other cloud computing solutions also try to allow you to only worry about your application as much as you like by providing VM images/templates. Ultimately your needs will dictate the approach you should take.
Now with all this in mind we can also construct hybrid solutions and workarounds that might fulfill our needs as well. For example, GAE doesn't seem directly suited to this specific app needs you describe. In other words, GAE offers relatively high number of requests, low number of cpu cycles (not sure if paid version will be any different).
However, one way to tackle this challenge is by building a customized solution involving GAE as the front end and Amazon AWS (EC2, S3, and SQS) as the backend. Some will say you might as well build your entire stack on AWS, but that may involve rewriting lots of existing code as well. Furthermore, as a workaround a previous stackoverflow post describes a method of simulating background tasks in GAE. Furthermore, you can look into HTTP Map/Reduce to distribute workload as well.
As of 2016, if you're willing to lump PaaS (platform as a service) and FaaS (function as a service) in the same serverless computing category, then you have a few FaaS options.
Proprietary
AWS Lambda
AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume - there is no charge when your code is not running. With Lambda, you can run code for virtually any type of application or backend service - all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.
AWS Step Functions complements AWS Lambda.
AWS Step Functions makes it easy to coordinate the components of distributed applications and microservices using visual workflows. Building applications from individual components that each perform a discrete function lets you scale and change applications quickly. Step Functions is a reliable way to coordinate components and step through the functions of your application. Step Functions provides a graphical console to arrange and visualize the components of your application as a series of steps. This makes it simple to build and run multi-step applications. Step Functions automatically triggers and tracks each step, and retries when there are errors, so your application executes in order and as expected. Step Functions logs the state of each step, so when things do go wrong, you can diagnose and debug problems quickly. You can change and add steps without even writing code
Google Cloud Functions
As of 2016 it is in alpha.
Google Cloud Functions is a lightweight, event-based, asynchronous compute solution that allows you to create small, single-purpose functions that respond to cloud events without the need to manage a server or a runtime environment. Events from Google Cloud Storage and Google Cloud Pub/Sub can trigger Cloud Functions asynchronously, or you can use HTTP invocation for synchronous execution.
Azure Functions
An event-based serverless compute experience to accelerate your development. It can scale based on demand and you pay only for the resources you consume.
Open
Serverless
The Serverless Framework allows you to deploy auto-scaling, pay-per-execution, event-driven functions to any cloud. We currently support Amazon Web Service's Lambda, and are expanding to support other cloud providers.
IronFunctions
IronFunctions is an open source serverless computing platform for any cloud - private, public, or hybrid.
It remains to seen how well FaaS competes with CaaS (container as a service). The former seems more lightweight. Both seem suited to microservices architectures.
I anticipate that functions (as in FaaS) are not the end of the line, and that many years forward we'll see further service abstractions, e.g. test-only development, followed by plain-language scenarios.
Alternatives:
1. AppScale
2. Heroku.
Ref: Alternative for Google AppEngine?
Amazon's Elastic Compute Cloud or EC2 is a good option. You basically run Linux VMs on their servers that you can control via a web interface (for powering up and down) and of course access via SSH or whatever you normally set up...
And as it's a linux install that you control, you can of course run python if you wish.
Microsoft Windows Azure might be worth consideration. I'm afraid I haven't used it so can't say if it's any good and you should bear in mind that it's a CTP at the moment.
Check it out here.
A bit late, but I would give Heroku a go:
Heroku is a polyglot cloud application platform. With Heroku, you
don’t need to think about servers at all. You can write apps using
modern development practices in the programming language of your
choice, back it with add-on resources such as SQL and NoSQL databases,
Memcached, and many others. You manage your app using the Heroku
command-line tool and you deploy code using the Git revision control
system, all running on the Heroku infrastructure.
https://www.heroku.com/about
You may also want to take a look at AWS Elastic Beanstalk - it has a closer equivalence to GAE functionality, in that it is designed to be PaaS, rather than an IaaS (i.e. EC2)
If you're interested in the cloud, and maybe want to create your own for production and/or testing you have to look at Eucalyptus. It's allegedly code compatible with EC2 but open source.
I'd be more interested in seeing how App Engine can be easily coupled with another server used for CPU intensive requests.
TyphoonAE is trying to do this. I haven't tested it, but while it is still in beta, it looks like it's atleast in active development.
The shift to cloud computing is happening so rapidly that you have no time to waste for testing different platforms.
I suggest you trying out Jelastic if you are interested in Java as well.
One of the greatest things about Jelastic is that you do not need to make any changes in the code of your application, except the changes for your application functionality, but not for the reason the chosen platform demands this. With reference to this you do not actually waste your time.The deployment process is just flawless, and you can deploy your .war file anywhere further.Using GAE requires you to modify the app around their system needs. In case if you happen to get working with Java and start looking for a more flexible platform, Jelastic is a compatible alternative.
You can also use Red Hat's Cape Dwarf project, to run GAE apps on top of the Wildfly appserver (previously JBoss) without modification.
You can check it out here:
http://capedwarf.org/