Such questions have been asked before - but all of the answers are outdated now.
I am looking forward to work on a Scala based webapp. I understand this question can be split into two, but I am posting them as one because they rely on same context, there being a dependency on the hosting platform and frameworks used.
I have read multiple (awesome) debates on Play! and Lift, but cannot find a good comparison between Play! 2.1 and Lift. How do I decide which one is better for my scenario (a social network website) ?
Similarly, this discussion has some very good arguments as to which platform to use for if I go with Lift, but it's from 2010 and seems outdated. The recommended provider (stax.net) is dead (or I guess it's merged with cloudbees.com). I am personally inclined towards GAE, as they are quick to start with, but unsure if the issues still prevail :
Support for actors (I am not sure if Akka helps us solve this problem)
Requests for a given session being served by different JVMs without notice to running app
Quoting David Pollak (lead author of Lift) :
GAE is slow and non-scalable, despite Google's claims (everyone I've
spoken with that have tried to scale GAE apps have failed and gone
elsewhere). GAE locks you into a tremendously suboptimal storage
mechanism. GAE is free, but so is Stax and there are many inexpensive
options including SliceHost. Next up, you've got Amazon EC2 and
RackSpace. So, I haven't found a good reason for anyone to use GAE.
And if there's no good reason to use GAE, devoting a pile of resources
to code around the GAE JVM incompatibilities (e.g., no new threads)
seems like a waste.
Another issue if I go with GAE is lack of Play! 2.1 support. I still don't see a module for that. Another issue is difficulty to migrate to other databases (although I hear migrating to MongoDB should be relatively easier) in the future. Worst case would be to move out of GAE and use AppScale.
Personally I use Lift, Cloudbees, and MongoLab as my first choice for most of my projects. I tried several cloud hosting services to no avail (Heroku and RedHat in particular. I don't think I tried GAE due to the post from David Pollak that you have already referenced). To use cloudbees, you just need an sbt plugin. Then it is as easy as running the cloudbees-deploy target. Within a minute, your code is up and running. I was floored by how easy it was. I went with Mongo primarily because of this excellent g8 template (note, there is now an SQL equivalent)
Another thing I really like about Cloudbees and MongoLab is they both have free services. It's great for me because I only work on these projects in my free time, so I don't want to spend any money while my ideas are half-baked.
As for Lift, I can't compare it much to Play. I downloaded/installed play and was immediately turned off by how MVC it is. I felt that the view-first approach, albeit foreign to me, seemed to be a much more intuitive and powerful way to build web applications. I love how Lift doesn't obscure from me the fact that I am indeed developing a web application. I often feel that MVC frameworks try to keep all of the HTML/CSS/JS etc at an arms-length.
The question is quite open so I will share my experience and opinion regarding Scala web app development as it might help you with your decision.
I built my first scala web application using Scalatra and Scalate using Jetty as the server. The app is hosted on an Amazon EC2 instance and I've had no problems with this... it's been running since the end of 2011 with only one small blip that took 10 mins to resolve. I found it a good experience for learning to use Scala in web applications.
http://www.scalatra.org/
Typesafe (http://typesafe.com) appear to have opted for the Play Framework and so for my next scala based web app I am likely to go for Play. A book I have been reading on the Play Framework is "Play for Scala". It has just been published this month (Oct 2013).
http://www.manning.com/hilton/
My impression is that Lift was the go-to framework in the past but that this has shifted to the Play Framework.
I'm interested in web development and by that I mean the bigger projects like facebook or twitter. I know the basics of java, css, php and mysql. I know there is a lot more out there. I read about it. But I don't know what the purpose is and how to put in place.
Things like: Scribe, thrift, casandra, Unix/Linux, shell/perl/python scripting, PostgreSQL, MongoDB, non-relational NoSQL datastores, JVM, nginx
I want to know why they need it, how they use it and what te purpose is.
What I need is a book like technical background of facebook for dummies or so.
Are there any books or websites that explain this from scratch?
Thank you!
EDIT:
Thank you for your answers! You have been very helpful. I was in the assumption, experienced programmers know almost anything about the technology there's used today. But as I read, you can only know so much and I need to figure out which technology to use. I take on the encouragement to start building small. And will take on php and improve my skills from there.
Thanks again!
http://highscalability.com/
This is one of the best sites out there. There are several case studies describing what and why many websites use, and pointers to further references. I would also look at the Google Scalability Conference 2007 talks
http://www.google.com/search?q=Google+Scalability&hl=en&client=firefox-a&hs=YUg&rls=org.mozilla:en-US:official&prmd=v&source=univ&tbs=vid:1&tbo=u&ei=fl4OTPUkorIwueCQxQw&sa=X&oi=video_result_group&ct=title&resnum=4&ved=0CDIQqwQwAw
It's all about choosing the right tool for the job in my eyes. There is so much technology out there it's impossible to learn it all. Just choose the subset that will work for you.
The best place to start is by building small simple websites, and as you come accross problems that you need solved you research the tools needed to solve those problems.
If you attack all of the areas at once, it's going to be overwhelming and you will not get anywhere.
For a general overview on what each of the technologies does, Wikipedia gives a good overview on most technologies.
If you are interested in database content which it seems like you are, a good place to start is reading up on normalisation.
Scribe, thrift, casandra, Unix/Linux, shell/perl/python scripting, PostgreSQL, MongoDB, non-relational NoSQL datastores, JVM, nginx
Those I would search on Wikipedia for to get a quick overview. Facebook is written in PHP/MySQL. There are some books on the subject of creating social networking sites, and some books have gotten decent reviews on Amazon.com, however, I have not read any of them myself.
If I were you, I'd start with PHP/MySQL and sit down and write a simple social network. Break the project down into components and tasks and Google for each challenge you encounter such as sessions, database structure, security, friend structure, and processing POST and GET requests.
You'll learn a lot and you get the big picture. Once you see the big picture, you can take another look at different technologies that are available and then decide which component you could have developed better with other tools. I personally don't think that looking too much into the technology available is good for someone who is still in the beginning stages. Start doing, learn from it, and then your questions become much more specific and a lot of things will make more sense.
The problem you're having is you're looking at smaller, specialty products, and not at larger, more mature technologies. Wikipedia will actually give you a decent overview of most of the medium-and-large projects out there.
Cassandra, Hadoop, Mongo, and NoSQL are all lovely... but they're specialty tools. SQL is a general purpose solution that works for 99% of the sites on the net.
Unix/Linux isn't a specialty tool; you might want to try going to Ubuntu's website and installing Linux, and just using it day-to-day, the way you'd use Windows. When you need to figure out something new, like setting up a webserver, do it on the Linux box and a Windows box, and you'll eventually learn linux pretty darn well.
As far as scripting, O'Reilly makes a great line of books on Bash, Perl, and Python.
JVM is a Java Virtual Machine, which is a core of getting Java code to go. Sun's website has a great set of tutorials on learning Java.
It might be much, much easier to pick a project (or three) that you'd like to learn, and learn some of these by doing. I'd probably suggest learning some SQL before learning the newly established alternatives; that lets you learn the rest of the system, as SQL is pretty easy. Once you've got the rest of the thing solid, try swapping in a NoSQL solution at that point.
There are a lot of frameworks that do a lot of different things. You've named a lot of different things from a lot of different areas. The best way to think of these things is to group them by category. Here's an example:
Suppose you have a laptop and you want to host a website. You'll need the following at a minimum:
1) Web Server software. Two popular options are Microsoft's IIS and Apache Web Server.
That's really all you need. You can set up your www_root folder and load files into it. Assuming everything is configured properly, you can now load HTML pages into that folder and access them through your IP address. Every page you view in your web browser is in HTML format. CSS is a stylesheet language that defines how your HTML will be formatted. You can also start writing Javascript, as most modern browsers support the client-side scripting language.
Chances are you'll want the following as well:
2) Database software. Two popular options are Microsoft's SQL Server and MySQL
3) Server-side scripting. PHP is very popular, as is ASP. You'll need the runtime deployed on your server. Python, Ruby, Perl, etc all fall under this category.
4) Web Application Framework(s). This will provide you with libraries for your language of choice to help develop web applications and websites. CakePHP, Ruby on Rails, and the Google Web Toolkit are examples of web application frameworks.
Additionally, you may want to utilize:
5) Additional libraries. JQuery, for example, is quickly becoming a popular library for Javascript that handles a lot of common tasks for you. Instead of writing complex effects code and what-not yourself, just use the pre-written code in the JQuery library.
6) Data interchange technology. If you are passing a lot of information back and forth, you will likely want to encapsulate this data in a logical format. Ideally, this format would describe the data and allow your applications to easily read/process it following a standard. This is where XML and JSON come into play.
I can't recommend a good book for you to learn this stuff, but I feel that the collective replies to your question here should be more than enough to get you started.
Ultimately, what you need to do is determine what technologies you need, and then choose the right one for the job. Don't go building an application using Ruby on Rails just because it's what Twitter used, but rather choose it because it provides some advantage to you over the other options.
I am experimenting with App engine. One of my stumbling blocks has been the support for managed relations or lack there off, this is further compounded by the lack of join support.
Without getting into details of the issues I have run into (which I will post under different topic), I would like to ask two things.
1. Has any one of you used managed relations in something substantial. If so if you can share some best practices that will help.
2. Is there any good comprehensive example(s) that you have come across which you can point me at.
Thanks in advance.
I think this answer might disappoint you, but before you develop on the app engine you should read it anyway, and confirm this in the docs.
No. No one on the app engine has used managed relations for anything 'substantial', simply because Bigtable is not built for managed relations. It is a sharded and sorted array, and as such is a very different kind of data structure than what you would normally use.
Now there are attempts to set up managed relationships - the GAE/Java team is pushing JDO features that come close to this, and there's more info on this blog, but this simply isn't the natural state of things on the app engine, and you'll very quickly run into problems if you decide to spend too much time wrapping yourself in a leaky abstraction.
Its a lot easier to actually look at what bigtable really is - there are a ton of videos on the google i/o pages for 2010 and 2009 that do a fantastic job of explaining that, and then figure out ways to map your problem according to the capabilities of the datastore. It may sound unreasonable, but think about it... the GAE is a tool that can do certain things exceedingly well, and if you can figure out your problem in terms of ideas like object stores, sets, merge joins, task queues, pre-computation and caching, then you can use this tool to kick ass.
I am trying to scrape some website and republish the data as a RSS feed. How hard is this to setup with Google App Engine? Disadvantages and Advantages using GAE. Any recommendations and guidelines greatly appreciated!
Google AppEngine offers much more functionality (and complexity) than you will need if truly all you will want to do is republish some structured data as RSS.
Personally, I would use something like Yahoo pipes for a task like this.
That being said... if you want/need to get your feet wet with GAE, go for it!
Working with Google App Engine is pretty straight forward. I would recommend going through the Getting Started guide. It's short and simple and touches on essential GAE topics. There are more pros and cons than I will list here.
Pros:
In general, App Engine is designed for high traffic web applications that need to scale. Furthermore, it is designed from a programmer's perspective. Much of the scalability issues (database optimization, server administration, etc) are dealt with by Google. Having said that, I find it to be a nice platform. It is still being actively developed by Google engineers, and scheduling of tasks (a feature that has been long requested) is in the current road map.
Cons:
Perhaps the biggest downside right now is again the lack of official scheduling support and the quota limits currently set for free accounts. However you can't complain much if its free. Currently it only supports Python as a programming interface (although a new language [Java I predict] is coming soon). Furthermore, Python 2.6 (and 3.0 for that matter) are not yet supported. In addition, Django 1.0 is not officially supported in App Engine (although you can package Django 1.0 with your application).
Harder than it would be in most other technologies.
GAE can sort of do scheduled batch stuff like this now, but it's really not intended for that type of thing. Pick pretty much any other language and platform for this particular task, and you'll make your life a lot easier.
I think BeautifulSoup could run on GAE, so all your scraping needs are handled :D
Also, GAE has a geturl thingy. The only problem I think you might have is not having enough time to get the data (30 secs limitation).
I am working on a same project and I've decided that it's easier to prepare the data on another server and push them to GAE.
You might also want to look into Yahoo! Query Language (YQL)
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
Is anyone out there* using web2py?
Specifically:
In production?
With what database?
With Google Application Engine?
by "out there" I mean at stackoverflow.
You are welcome to ask the same question on the google group. You will find more than 500 users there and some of them are development companies building projects for their clients.
My impression is that most of them use postgresql (that's what I do to) and some others use the Google App Engine. In fact web2py is the only framework that allows you to write code once and the same code will run on GAE, SQLite, MySQL, PostgreSQL, Oracle, MSSQL and FireBird (with the limitations imposed by GAE).
You can find the Reddish (reddit clone) appliance with source code for GAE here
Here you can find links to some productions app. Some are running on GAE.
#Armin:
Nothing is wrong with Django or Pylons. They are excellent frameworks. I have used them before developing web2py. There are a few things you can do with web2py that you cannot with them. For example:
web2py does distributed transactions with Postgresql, Armin requested this feature.
the Django ORM does not do migrations natively (see South), web2py does.
the Django ORM does not allow partial sums (count(field)) and group by, web2py does.
web2py can connect to multiple databases at once, Django and Pylons need to be hacked to do that, and
web2py has a configuration file at the app, not at the project level, like them.
webp2y logs all tracebacks server side for the administrator, Django and Pylons do not.
web2py programs often run on GAE unmodified.
web2py has built-in xmlrpc web services.
web2py comes with jQuery.
There are many things that web2py does better (using a more coherent API) and faster (processing templates and generating SQL for example). web2py is also very compact (all modules fit in 265K bytes) and therefore it is much easier to maintain than those competing projects.
You only have to learn Python and 81 new function/classes (50 of which have the same names and attributes as corresponding HTML tags, BR, DIV, SPAN, etc. and 19 are validators, IS_IN_SET, IS_INT_IN_RANGE, etc.).
Anyway, the most important issue is that web2py is easier than Django, Pylons, PHP and Rails.
You will also notice that web2py is hosted on both Google Code and Launchpad and there are not open tickets. All past issues have been resolved in less than 24 hours.
You can also check on the google mailing list that all threads (10056 messages today) ended up with an answer from me or one of the other developers within 24 hours.
You can find a book on web2py on Amazon.
Armin, I know you are the developer of Jinja. I like Jinja but have different design philosophies. Both Django and Jinja define their own template languages (and Jinja in particular has excellent documentation) but I do prefer to use pure Python in templates so that my users do no need to learn a template language at all. I am well aware of the pros and cons of each approach. Let's the users decide what they prefer. No need to criticize each other.
#Andre: db.table.field refers to the field object. 'table.field' is a field name. You can always pass a field object when a field name is required because str(db.table.field) is 'table.field'. The only case you are required to use a string instead of an object is when you need to reference by name a field that has not already been defined... perhaps we should move this discussion to the proper place. ;-)
I hope you will decide to give web2py a try and, whether you like it or not, I would love to hear your opinion.
I'm using web2py for a small web app. It's running the HITs on a Mechanical Turk project, and giving me an interface to control and visualize them. I started on Google App Engine, but then got sick of the little annoyances of not having direct database access and having to wait forever each time I want to upload my code, and moved to a local server with postgres. GAE makes most things harder in order to make a few scaling things easier... stay away from it unless you really need their scaling help.
I like web2py a lot. Compared to Django and Ruby on Rails, it's WAY easier to learn and get going. Everything is simple. You get stuff done fast. Massimo is everywhere solving your problems (even on this board haha).
I started using web2py about 6 month ago. I choose it, because I wanted to move from PHP to Python, to have a more object-oriented approch because of the language featrues of python.
The all-in-one approach of web2py is really amazing and makes the start very fast.
As a former symfony user I soon started to miss Components and Forms that aren't dependend on table structure.
Just with a simple registration form, I could not find a way to do the Form DRY. For me the real bugger was the form validation. I forgot the details, but I ended up with having form validation in the forms itself. Because some thing just didn't work else.
Also the naming concept of capitalised words with that lot of repeated chars is just not my thing.
dba.users.name.requires=IS_NOT_EMPTY()
dba.users.email.requires=[IS_EMAIL(), IS_NOT_IN_DB(dba,'users.email')]
dba.dogs.owner_id.requires=IS_IN_DB(dba,'users.id','users.name')
dba.dogs.name.requires=IS_NOT_EMPTY()
dba.dogs.type.requires=IS_IN_SET(['small','medium','large'])
dba.purchases.buyer_id.requires=IS_IN_DB(dba,'users.id','users.name')
dba.purchases.product_id.requires=IS_IN_DB(dba,'products.id','products.name')
dba.purchases.quantity.requires=IS_INT_IN_RANGE(0,10)
Sometimes the names have to be in quotes, sometimes not ... and if I looked at the examples or sites already made with web2py, I really didn't see that big step forward from using php.
I recommend you: Look if web2py works for you. It would be nice, because the community and especially massimo (the creator) are very helpful and nice.
Also you have a much quicker start, than with django, easier deployment and less hassle if you change your database models.
As Massimo points out above, the team at tenthrow uses web2py for tenthrow.com
We did most of our development work during 2009. Our stack uses cherokee, web2py, postgresql, and amazon s3. We had done many python web implementations prior to this on a variety of frameworks and backends. To say that we simply could not have done tenthrow so quickly and easily without web2py is an understatement. It's the best kept secret in web development.
I am evaluating web frameworks for a long time now. I wrote my own (not open) frameworks in Perl and in PHP. Well, PHP has a builtin deadend and the whole infrastructure is still quite poor, but I did not want to go back to Perl, so I checked Python and the Python Web Frameworks like Django, Turbogears, Pylon and web2py. There are many things to think about, if you want to choose a codestack that is not your own and you will often scratch your head because there is still no "right way" to program things. However, web2py is my current favourite, because the author, despite beeing a "real programmer", keeps things easy! Just look at the comparison on web2py site - I was wondering why python frameworks like django or turbogears had to introduce such redundance and complicated syntax in their code - web2py shows, that it IS in fact possible to keep your syntax clean and easy!
#Armin: could you please specify you criticism? Where exactly do you see web2py "bypassing Python semantics"? I can not understand, what you mean. I must admit that I am not that deep into python right now, but I see no problem with the web2py code - in fact, I think it is brilliant and one of the best frameworks available today.
I use web2py for academic purposes. About a year ago I published on pythonanywhere a digital text book for german grammar.
The resource requires authentication and looks like a little LMS with roles, activities and grades. It was my first experience of this kind. And it was a success because PHP was to difficult for me, and only web2py could provide a clear way to handle a database. With Python I could easily solve my problems as e. g. text analysis and downloading reports. As for database so SQLite was completely enough.
My students like the design and the way everything is functioning. So I am very satisfied with the results and going to develop other interesting applications for my university.
I think web2py is very good for applied linguists and L2 teachers, who are not as experienced in computer science as programmers. So that was my humble opinion.
There are some users listed here: http://mdp.cti.depaul.edu/who.
I'm starting to use it with Postgresql. But a long way off production... I've also played with Zope V2 and Ruby on Rails and really love the approach of web2py.
I vote for Web2py. I only have time to develop small but useful stuff for my own use.
Hopefully next month I will have an oppty to create an app that is perfect for Web2py and run on Google app engine.
Web2py = breath of fresh air!
We are using it with our website that teachers Chinese, www.dominochinese.com. Our host is pythonanywhere.com and we love the simplicity of it. I work on building stuff instead of wishing I could get stuff working. I worked with django for 1.5 years and I hated it. In a sense I feel web2py is the PHP but in python. It makes people quickly do stuff without going into object oriented programming, which can be really confusing for beginners to intermediate programmers.
I am not using web2py. But I had a look at the source code and it's horrible for so many reasons. For one the database definitions as well as the views and models and I don't know what, are evaluated against a global dictionary of values. It feels like PHP in that regard (it's bypassing Python semantics in name behaviour) and is very inefficient and I could imagine that it's hard to maintain.
I have no idea where all that fuzz about web2py is coming from lately, but I really can't see a reason why anyone would want to use it.
What's wrong with Django or Pylons? What does web2py do that you can't do with Django in a few lines of code with a better performance, code that's easier to read and on an established platform where tons of developers will jump in and fix problems in no time if they appear. (Well, there are exceptions I must admit, but in general the developers fix problems quickly)
I am using web2py for 2 years, this web frameworks is great and unique. Easy to use, accept a lot of DB's but the best DB supported is postgres.
I have created 2 projects with web2py and a really like how easy it is. 1 project is a financial management and other a mail tracker, both in production systems (4 linux lpar with postgres) running fine.
web2py is a good choice
[small application created with web2py 2.5.1]
updated
http://freitasmgustavo.pythonanywhere.com/calculoST/
Actually it's using MySQL, but it could switch to postgresql at a moments notice as web2py is so diverse :)
I have been evaluating web frameworks for sometime now. I prefer web2py because it's easy to follow, compact yet powerful.
I like it because it is so tiny that I can easily distribute with my application.
We started to use Web2py 7 months ago. We already have one application in production in El Prado (National Museum in Spain). We developed a app to check and automate all the systems, to make servers statistics, access statistics, etc.
I use it in production on Google Appengine for www.crowdgrader.org.
I store data as follows:
The core metadata, where I need ACID, is stored in Google Cloud SQL, which is working very well for me. For big text fields, I store in Google Cloud SQL the key, and in the Datastore the key-value.
The text typed by users is stored in the Google Datastore, see above, with the key stored in Cloud SQL.
File uploads go in the blobstore.
I am slowly migrating more storage to the Datastore, to get more write bandwidth for things that do not require complex queries and can deal with a bit of eventual consistency.
I am very happy about web2py + appengine + Google Cloud SQL + Datastore + Blobstore.
I am using web2py in production with postgres on webfaction, and also on the GAE.
I used web2py for small projects so far, but I hope to introduce it in my company. It's my favorite web framework.
My blog is running on GAE with web2py.
I also have a facebook apps running on top of web2py: My Top 10 Gift
I use Web2py with Google App Engine in production. See https://www.nittiolearn.com.
For storing data, Google Datastore (accessed via web2py DAL) is used except for storing large resources where Google Cloud Storage is used. I have done multiple web2py version upgrades on the production environment in the last 5 years without any major issues.
Google app engine has also been mostly friction-free over the years.
But neither Web2py nor Google app engine has been adopted widely as I had thought 5-6 years ago. If I'm starting a new project, I'm unlikely to go with web2py or app engine as the number of developers who will be excited to work on these technologies is limited.
I am using web2py with gae and google datastore in production of custom application , it is a very good framework.
I did made some minor fixes for work good on GAE, work fast and stable, i have published the Web2Py version changes uses on my github soyharso.
The uploads to GAE are fast, the version control app engine is secure, the free tier offer google for tuning your code is excellent, the monthly cost is adequate
Well, I am using Web2Py professionally, with PostgreSQL, and on linux. I am working on my Social network named "Ourway". You may like some features of it like "Blog" part.
http://www.noobmusic.com is using the Google App Engine.
I am using web2py in production. Currently while in early production we are developing with SQLite because it is easy and it comes out of the box but later we will probably switch to MySQL. I don' think there are any plans to use Google App Engine.
This are quite old responses but I will chip in anyway. In the year 2008 maybe it was excellent choice, as well as Django/Flask. And it still might be good.
But this days people want instant results, with a way less learning curve.
The web2py is not that intuitive to be fair.
Do I need to study MVC concepts for working with MS Access? I could not care less for URL routing, just need to display a few tables on the web, preferably with some validation. Plus some authentication.
This is where framework like http://jam-py.com/ shines! Not only that you wont be lost, but it does remind of Access which ruled the offices for like decades. And still rules in 2019. Why? Almost no learning curve.