while starting to learn the google app engine datastore API I noticed the tutorial said:
"the rate at which you can write to the same entity group is limited to 1 write to the entity group per second". Here
I cant seem to understand how can you store a lot of user's information that needs to be written more than 1 time per second like: a simple app that let a user change a value on their profile or something like a comment or chat app that must write rapidly into the datastore
how can this be achieved? what have I missed here?
if there's any samples or tutorials for a real application with the datastore it can be of great reference to me. (preferably in golang but anything will do)
Thanks!
The key part of the bit you quoted is "entity group". Have you properly understood what that means?
A user updating their own profile is unlikely to happen more than once per second - that would require some very fast typing and clicking. As long as the profiles aren't in the same entity group then there's no reason, say, 10k users can't all update their profiles at the same time. For a chat, storing an entire conversation/room as a single entity would be tricky, but storing each message as its own entity wouldn't hit this limit.
You have not missed anything, it's a way High Replication Datastore (HRD) works. It could be strange at first, but if you read about the benefits of HRD, you may find that it makes a sense.
If it's not suitable for you, you can use native MySQL for GAE.
Also, there is a closed question about this subject.
Related
I am in the process of developing a Google App Engine application where users can read (and edit but it's not important atm) an address book. More specifically the Google Apps Directory address book.
Reading per se is not an issue since Google provides Directory API which I am familiar with. The problem is that my address book has thousands of entries and I need (almost) 0ms performance when searching. Also, I must be able to make a full text search on the address book. In the best scenario users want the entire data to be loaded at once.
Basically I come up with a couple of solutions.
1) Load the entire database in an Angular JS array on page load and then use ng-model to search. Performance are not good at all, meaning that sometimes the tab crashes or takes seconds to load results. It may be related to the fact that I am using angular-datatable to organize results but it's unlikely.
2) Use Google Datastore to make a replica of the address book but then I am stuck with the NO SQL schema. I heard Big Query can help but I've never used it, can I make full text search?
3) Use Google Cloud SQL but performance are not as fast as I'd like them to be.
4) Directly query the Google address book but from the reference the only operators allowed are '=' and ':' and performance are not reliable. Not to mention along the address book I must read Shared Contacts as well.
5) ???
Do you have any suggestions? Many thanks.
I am new to Google Cloud Datastore. I have read that there is a write limit on 1 write per second on an entity group. Does it means that the main "guestbook" tutorial on app engine cannot scale to thousands of very active users?
Indeed.
The tutorial is just a showcase. The writes per second limitation is due to the strong consistency guarantee for entities in the same group or ancestor. This limit can be exceed at the price of changing strong consistency by eventual consistency, meaning all datastore queries will show the same information at some point. This is due to App engine distributed design.
Please, have a look at https://cloud.google.com/appengine/articles/scaling/contention to avoid datastore contention issues. Hope it helps.
Yes, I think it does mean that.
It might not be a problem if the greetings are all added to different guestbooks, but quickly adding Greetings to the same Guestbook is definitely not gonna scale. However, in practice it's often much faster than 1 write per second.
Perhaps you could work around this by using a taskqueue to add Greetings, but that might be overkill.
That guestbook tutorial is not a very good example in general. You shouldn't put logic in your jsp's like in that example (you probably shouldn't use jsp's at all). It's also not very practical to use the datastore at such a low level. Just use Objectify.
Ok, so about a year ago I wrote a web app that helps organize appointments for my dads company. He now "couldn't do business without it". I have decided that I want to build a SAAS subscription model out of it and open it up to the public.
It's currently built on codeigniter and php which I do not think is a good fit for a SAAS version. I am planning on rebuiling it from scratch in laravel 4 and using stripe as a payment gateway.
My concern is how best to handle the database / application structure for more than one client. Currently, it just serves the one business and is very un-abstract and is specific to my dads companies needs. I need it to be able to handle different data depending on what the business who uses it does.
I have looked into multi-tenancy but i'm not sure this is right for this. I am thinking that a 'gmail' style approach would be better. One app / domain that after login the user will see their customised dashboard and only their data.
Before I get stuck in with the coding I need to work out how best to handle multiple 'accounts' on the one database. I do not want to create a table for each user, nor a database for each user.
I guess my question is can anybody point me in the right direction for how best to handle a monthly payment subscription in Laravel? It's not so much the code that I'm stuggling with, rather what exactly I would need to build to handle charging the customer each month and denying them access if billing failed etc.
Thanks
You are in for a lot of reading and a ton of work!
First of all, let's completely ignore the billing aspect of this for now — at the end of the day that portion of the application is really fairly trivial. Take a page out of 37signals Rework (page 93 and 94) and launch your product with a 30 day free trial before you even begin implementing it (you should know how to implement it by then).
Second, why do you think that "gmail" doesn't use multi-tenancy, URI structure tells nothing about the underlying database structure. I'm fairly confident they aren't cloning a database schema for every one of their customers. Therefore you've probably answered your own question — you want to implement multi-tenancy.
You're going to want to abstract your database (and application architecture), and honestly there is no better resource to help you on your way to doing that than Taylor Otwell's (creator of Laravel) book Laravel: From Apprentice To Artisan. His book is not for beginners, and by the time you're done reading it you should probably be able to answer this question for yourself.
You are not going to be creating a table or a database for each user, you aren't even going to be creating one for each organization. Instead you'll be creating abstract database structure in code, which will pull your users data out of the database.
Think about checking for permission to access an organization as another layer of user authentication. On every request you'll be checking to see if that user can access a particular organization. You'll likely also check to ensure that organization is still active (did it expire because they didn't pay?) this will again happen on every request and likely with a filter within laravel.
This really leads to the next very important factor of developing a SaaS application.
I don't know about you, but I'm paranoid, and I couldn't sleep well at night if I wasn't sure that user number 4506 couldn't see the data of an organization that he doesn't belong to. The only really good way to ensure this is through unit testing, which I'd highly suggest learning if you haven't already.
The best way to do this within Laravel 4 is to read Jeffrey Way's book Laravel Testing Decoded. This book is extremely advanced, but still easy to understand if you have a good grasp of the fundamentals.
Last but not least, the number one thing is get involved in the community — the easiest way I'd suggest doing that is idling on the #laravel IRC channel (freenode). Ask some questions, maybe answer some questions, everyone in the channel is very nice and responsive.
You are definitely in for an adventure, don't be afraid to ask questions and make mistakes. Good luck.
As a rough overview, I would have a clients table, and a subscriptions table. Any other data that needs storing such as contacts, or appointments, can be associated using foreign keys to the client table.
In laravel, you can use the ORM to get the currently logged in client, and then through a relationship, fetch appointments and contacts belonging to them.
There are some useful tools for laravel at cartalyst.com, including sentry and sentry-social for user auth, and integrating user accounts with facebook/google/twitter, etc.
Stripe will allow you to configure recurring payments, and will notify you via web hooks each time there is a payment attempt. you can log these in the payments table, and associate them with a user/client. you can use this to keep track of who has paid, and how recently.
Also, bear in mind that you may not want to cancel the account immediately on failed payment.
Stripe will reattempt, and it may be that your best response is after it is two or three days late, or you get an invalid card notification,to get in touch with the client and prompt them to update their payment details.
It may also be an opportunity to check when they last logged in.
If it was over a month ago you can credit them with a free month, and remind them of how much your app can do for them.
By doing this, you may be able to get people to continue using (and paying) for something they had forgotten they had subscribed to.
My concern is that anyone in Google can have some access to my uploaded (and published) source codes in plain text. And, what about data stored in DataStore? Are they protected?
I'm using Python 2.7 runtime and, of course, I can make it 'undownloadable' if that helps in some ways.
It's certainly possible that some employee might be able to look at your code and/or data by trying really hard, breaking their employment contract with Google and possibly becoming a law-breaker. I think that's highly unlikely, though.
If you're concern is more around the intellectual property with regards to your code+data, Google's stance regarding that is explicit:
Customer owns all Intellectual Property Rights in any Application and Customer Content, and Google owns all Intellectual Property Rights in the Service.
So at no point of time are they going to look at your code+data and copy anything. Doing so will open the door to your taking them to court immediately.
I think Google takes measures to solve this problem on a much larger scale then what relates to your and my applications on GAE. Compared to the data that it needs to protect from all human eyes, we're fairly unimportant :D
I don't see why this is particularly a problem with AppEngine. Any hosting provider is always going to have access to your code, simply because they own the disks that it lives on. You simply have to believe their code of conduct, if they have one, or that they really don't care about your code.
Take a look at the Confidential Information section of the Google App Engine Terms of Service.
In practice, both your code and your data are probably lost in the noise as far as Google are concerned.
I´m thinking about to write an application will have to store a small amount of records per user (<300) but hopefully will have a lot of users (>>1000).
I did some research for a platform that allows starting small and scale if there is a need to do so and got stuck with App Engine, but I´m not sure if it is the right tool for it, especially the datastore.
How will I get it to scale if I have a User entity and a Message entity and store all users and messages in that entities? I think the amount of records in the entities will grow really big and filtering i.e. for all messages of a user will get expensive. Is that a problem or will Google handle that ? Do I have to introduce multitenancy and create a namespace for each user so I only see the records in the entities that relates to the user? Is there a limit for the number of namespaces ? What would be the right approach for modeling the data in the datastore?
I do not really have a clue how to handle the App Engine datastore and if its the right tool for me.
The App Engine datastore is explicitly designed to handle this kind of scalability. Queries execute in time proportional to the number of records returned, so fetching all a user's messages will take the same time regardless of how many users there are in the system.
I think with those kind of numbers you are probably ok in terms of scalability. anywhere from 300,000 to millions of records is easily handled by any serious datastore.
It is not advisable to think of scaling during the infancy of your project.. Your first step should always be to build an app/product and launch it... Scaling comes afterwords Most of the app/products that are launched these days never make it to the level where they need to scale.. even if you do make or launch such a website/product/app that gets hit by large amount of traffic and you need to scale, then rejoice!!! because you've made it to that level.. But how to get to that level should always be the first question...
I'm not trying to de-moralise you, rather trying to help you focus where you should be... Thanks for reading and good luck with your App! May you do need to scale and as Toby said, even the most basic App Engine configuration is good enough to handle a couple of hundred thousands of records...