I'm one man shop, creating a social site. If my site becomes popular, will the tech gods (i.e. techcrunch, mashable, etc.) look down on me for using Google App Engine instead of creating and developing my own cutting edge infrastructure from scratch?
Answer #1: No, TechCrunch et al often report favorably on sites that use fashionable cloud computing services as their back end.
Answer #2: Who the heck cares what the so-called "Tech Gods" think about your implementation details? What matters is whether you can efficiently build a service that customers want at a profitable margin. The opinions of wannabe and never-were journalists are irrelevant to a business plan and in any case nothing impresses like actual success.
Techcrunch, mashable et al. are not tech gods.
It's about being cost-effective. If GAE is right for you, then go for it.
You shouldn't worry about what others might think about the infrastructure of your site. Focus on the content/service that you are going to provide and what others might think about that instead.
No. "Don't reinvent the wheel" is one of the core tenets of programming; this goes for infrastructure too - if GAE fits your infrastructure needs, then go for it.
There are a number of sites using Google App Engine that get pretty good buzz. One is WalkScore.com
Perhaps if you considered it from a different angle? If you develop your site with GAE you should end up with an application that can scale quickly to meet increasing traffic. If your site becomes popular, you will certainly be 'looked down upon' by the tech gods if the site starts falling over.
As everyone else has said, if GAE is a good fit for your site/app, use it. 99% of your users won't even know the difference.
Why would it matter? unless you are getting the sort of traffic that facebook is getting why do you want to waste so much time trying to redo what could be done sucessfully. And if it flops no big loss.
And TechCrunch = not TechGods. What a Heretic!
Related
We are in the process of deciding a route to take for a new CRM system. We've had Salesforce come in and give us their pitch and the developers have had a little play with it, made it do a few things we need etc...
It's hard for us to get a good idea of the pros and cons until we start to develop with it and if you start, you are tied in to a year contract for X number of users and it's pretty expensive as it is..
So, my question. Who has developed for sales force platform? how did you find the experience? would you recommend it as a good solution? Should we just continue with our ruby/rails/mongo systems?
Thanks!
The good news is the amount of customization you can do via configuration is amazing. The out-of-box functionality is very strong and you get a pretty nice security model and reporting system included.
Having said that, when you do need to do custom development beyond what the configuration can support, the pain can start;
-APEX is the most frustrating (modern?) language I have ever worked with.
-Deployment/Migration can be slow and painful (some things cannot be migrated, e.g. Approval processes)
-APEX is a rather immature language missing much of the concepts of .net or java
-Debugging is messy (log actually gets truncated at a certain length, no stepping)
Having said all that, SalesForce.com is a very strong CRM - 90% of the custom work you'll want to do will be really smooth and fast, the remainder will be extremely painful.
I'm going to create a fairly large (from my point of view anyway) web project with a friend. We will create a site with roads and other road related info.
Our calculations is that we will have around 100k items in our database. Each item will contain some information like location, name etc. (about 30 thing each). We are counting on having a few hundred thousand unique visitors per month.
The 100k items and their locations (that will be searchable) will be the main part of the page but we will also have some articles, comments, news and later on some more social functions (accounts, forums, picture uploads etc.).
We were going to use Google AppEngine to develop our project since it is really scalable and free (at least for a while). But I'm actually starting to doubt that AppEngine is right for us. It seems to be for webbapps and not sites like ours.
Which system (language/framework etc.) would you guys recommend us to use? It doesn't really mater if we know the language since before (we like learning new stuff) but it would be good if it's something that is future proof.
I think that GAE can do the job. Google claims that Google App Engine is able to handle 5 million visitors for free and you will have to start paying only if you exceed their free quota.
It's also pretty easy to get started. If you don't have experience on administrating websites and choose a regular hosting service, you will have to worry about several things that you don't even imagine now.
My only concern would be with respect of the kind of data and queries you will have to do, since it does not have a relational database. Anyway, there is an open source project for GAE, called GeoModel that gives GAE the ability to do complex geo spacial queries, like proximity fetch. Have a look at their tutorial and the demo app.
About your impression that GAE was intended only for small web apps, there are a couple of CMS that run on it.
Good luck!
If once of your concerns is scalability, and you don't want to depend on expensive or commercial tools, I would recommend that you take a look at this tech stack:
Erlang - A programming language designed for concurrency and distribution.
Nitrogen - An Erlang web framework with a lot of cool stuff, like transparent AJAX.
NoSQL scalable databases, such as CouchDB or Riak - Save the the hassle of SQL code and are more scalable than plain MySQL. Both has direct native Erlang API.
To be honest, I don't know if this tool set is your cup of tea; These are not mainstream solutions. I just suggest these to everyone who ask about size-sensitive web applications.
All serious web frameworks will provide you with what you need. The real issues (for example scalability) might be tackled in a different way depending on what you use, but you wont be limited if you choose a well-known one. The choice of database system might be more important for that (sql vs nosql), even if both of those will do fine too.
It's all about
knowing how to use
enjoying to use
the tool(s) you've chosen.
In either case, name-dropping some suggestions:
Rails (Ruby)
Django (Python)
Nitrogen (Erlang)
ASP.NET MVC (C#)
And please note, if you really want to learn everything from the bottom, you'd be fine with any of these (or one of the other gazillion out there). But if you want to perform your best, choose one that supports a language you know well or uses techniques/tools you have experience of etc. Think twice about how you value this is fun and we learn a lot against we want to be productive and do a really good job.
I am experimenting with App engine. One of my stumbling blocks has been the support for managed relations or lack there off, this is further compounded by the lack of join support.
Without getting into details of the issues I have run into (which I will post under different topic), I would like to ask two things.
1. Has any one of you used managed relations in something substantial. If so if you can share some best practices that will help.
2. Is there any good comprehensive example(s) that you have come across which you can point me at.
Thanks in advance.
I think this answer might disappoint you, but before you develop on the app engine you should read it anyway, and confirm this in the docs.
No. No one on the app engine has used managed relations for anything 'substantial', simply because Bigtable is not built for managed relations. It is a sharded and sorted array, and as such is a very different kind of data structure than what you would normally use.
Now there are attempts to set up managed relationships - the GAE/Java team is pushing JDO features that come close to this, and there's more info on this blog, but this simply isn't the natural state of things on the app engine, and you'll very quickly run into problems if you decide to spend too much time wrapping yourself in a leaky abstraction.
Its a lot easier to actually look at what bigtable really is - there are a ton of videos on the google i/o pages for 2010 and 2009 that do a fantastic job of explaining that, and then figure out ways to map your problem according to the capabilities of the datastore. It may sound unreasonable, but think about it... the GAE is a tool that can do certain things exceedingly well, and if you can figure out your problem in terms of ideas like object stores, sets, merge joins, task queues, pre-computation and caching, then you can use this tool to kick ass.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 9 years ago.
Improve this question
We are about to start working on new commercial web project and considering Google App Engine as a potential platform.
Questions:
Does Google App Engine is really
scalable and may be considered as a
production platform for commercial
project?
Is it more expensive (or
cheaper) than good hosting company
service in long run?
Is it
possible (and pretty cheap) to move
the app from Google App Engine to
independent server/farm (e.g. to use it
as a private system, to exploit our
own hardware etc.)?
Is there some
mechanism to deal with DDoS attacks?
Can I make a full backup of the
app data?
Sorry for such silly questions.
I'll answer question 1:
I'm in the pilot phase of a new web application on app engine. We've spent about a month writing code and getting things ready for our first customer. They went live last week. They love the software but a couple of days ago I started to get random deadline exceeded errors in the application. You look up a record or a list and it would come back in miliseconds. The next go it would take 30 seconds and come back with a deadline exceeded error.
The stack traces in the dashboard give random results. I've tried everything, even stripping the app down to a hello world. I put a log message into our django process request middleware, the first bit of our code that gets executed. It was showing that on the timeout requests it took 25 seconds from google getting the request to running our process_request code. I posted to the google forum and got nothing. I contacted someone at google and they answered back quickly but only said they would contact the team. Nothing since.
It is possible there's something I'm doing to cause this but I really doubt it. Google doesn't provide support so I'm basically out of luck.
If this was a full blown commercial application I'd be out of business.
tl;dr: google app engine has great promise but needs to mature and is not yet suitable for comercial production
Watch google IO (Whre among other they say that: "yep it's scallable".
That depends ... It can even be free for you (you pay for load that you've got).
You can move to Amazon for example using appdrop. It's also a good idea to use app-engine-patch.
... Good question. I realy do not know.
Use GAEBar.
It all depends on your needs.
For a project that has the need to scale from very few users to possible millions of users in short time, google app engine might be exactly what you're looking for.
However, note that you might be surprised by the limitations that GAE comes with. Datastore can amongst others not do full-text search or queries using the IN statement.
So be carefull to specify what needs your application will have, and what data you're going to store and search for.
This also means that moving your application from GAE to a separate server might be troublesome, since the database architecture will most likely be different.
My answers:
BuddyPoke runs on gae (probably the biggest app), check their millions numbers.
You don't pay until your app grows quite a bit
If you are familiar with python, web2py offers this feature with some limitations
Dos protection (java, python)
Gaebar, here a great article.
You're question #3 raises a red flag. If this is an important issue, I'd caution against App Engine at this time. I love the platform, and don't doubt that their will be viable migration paths to a self-hosted solution at some point, but not now. Things like appdrop prove it would be possible to do, but would the effort and investment be worth it? That's the question I'd ask. I'd love to know if somebody has successfully ported a real-world production app engine app to another host.
Backups should be easily scripted or there are tools like GAEbar as Bolotov mentioned.
Regarding cost, you can probably get tens (maybe hundreds) of thousands of objects (records) and decent traffic/use for free. Beyond that, I'm not sure about comparative hosting costs, sounds like a good area to do some research in (note to self).
Finally, Silfverstrom is right about limitations, especially around full-text search. There are some projects underway to tackle this, but probably nothing as robust as a mature RDBMS.
To update with some more recent info (2013), GAE now has a text search API. You can't search data in the database directly; you create searchable documents from your data, and add those to a searchable index. It's not terribly hard to do, but it's a hassle. In particular, whenever your data changes, you need to re-regenerate the changed documents and update them in the index.
It's also fairly easy to export data into Google Big Query, which makes it easy to do reporting.
I am trying to scrape some website and republish the data as a RSS feed. How hard is this to setup with Google App Engine? Disadvantages and Advantages using GAE. Any recommendations and guidelines greatly appreciated!
Google AppEngine offers much more functionality (and complexity) than you will need if truly all you will want to do is republish some structured data as RSS.
Personally, I would use something like Yahoo pipes for a task like this.
That being said... if you want/need to get your feet wet with GAE, go for it!
Working with Google App Engine is pretty straight forward. I would recommend going through the Getting Started guide. It's short and simple and touches on essential GAE topics. There are more pros and cons than I will list here.
Pros:
In general, App Engine is designed for high traffic web applications that need to scale. Furthermore, it is designed from a programmer's perspective. Much of the scalability issues (database optimization, server administration, etc) are dealt with by Google. Having said that, I find it to be a nice platform. It is still being actively developed by Google engineers, and scheduling of tasks (a feature that has been long requested) is in the current road map.
Cons:
Perhaps the biggest downside right now is again the lack of official scheduling support and the quota limits currently set for free accounts. However you can't complain much if its free. Currently it only supports Python as a programming interface (although a new language [Java I predict] is coming soon). Furthermore, Python 2.6 (and 3.0 for that matter) are not yet supported. In addition, Django 1.0 is not officially supported in App Engine (although you can package Django 1.0 with your application).
Harder than it would be in most other technologies.
GAE can sort of do scheduled batch stuff like this now, but it's really not intended for that type of thing. Pick pretty much any other language and platform for this particular task, and you'll make your life a lot easier.
I think BeautifulSoup could run on GAE, so all your scraping needs are handled :D
Also, GAE has a geturl thingy. The only problem I think you might have is not having enough time to get the data (30 secs limitation).
I am working on a same project and I've decided that it's easier to prepare the data on another server and push them to GAE.
You might also want to look into Yahoo! Query Language (YQL)