Let's say we are building a product catalog using Google app engine and trying to incorporate full text search. What are the factors to consider to guarantee latency in 200 ms .
As the document index grows , how do you configure app engine to linearly scale without compromising latency.
Neo
Related
Our project was running in GCP compute engine. For scaling purpose, it is moved to app engine. We had rabbitmq implemented for push messages and chatbots in compute engine. In app engine it is not feasible to implement rabbitmq. So I was going through alternate options. There I found cloud task option. But I have doubts in certain areas even after reading their documentation
In my understanding, we need an app engine instance for cloud tasks. In that case, can I implement it in same project itself as a different service? Will this affect the performance of the existing project?
Is there any better solution than cloud tasks in this case?
You can implement additional services under your app in the App Engine as shown in this diagram.
By default, App Engine scales your app to match the load. Your apps will scale up the number of instances that are running to provide consistent performance, or scale down to minimize idle instances and reduces costs.
You can consider running a RabbitMQ Cluster on Google Kubernetes Engine. You can find more information in the following documentation: rabbitmq.
I discovered that my pure App-Engine project (no Compute Engine Instances at all) is heavily using Google Compute Engine API (26,165 requests) but that is not visible as API use on the dashboard. Additionally, in Quotas I find strange items:
Google Compute Engine Networks Global 1 / 15
Google Compute Engine Firewall rules Global 2 / 200
Google Compute Engine Routes 2 / 250
I could disable the Compute Engine API, but I fear consequences if I don't understand what I am doing here.
Does anyone know further details or experience something similar?
PS: I also noticed that old deployed versions of my appengine-code from 2013 showed up, which I have deleted again.
My app is a standard env python one.
I do see the quotas you mentioned in the IAM & Admin Quotas page, but with different limits:
Google Compute Engine Networks Global 1 / 5
Google Compute Engine Firewall rules Global 4 / 100
Google Compute Engine Routes 2 / 200
I suspect that these are automatically driven by the standard env GAE infra (as opposed to being configured by the app itself for the flex env). Their visibility might not be intentional, tho.
I do see a few GCE API calls as well (but then my app is not very active) on the API Manager page:
Google Compute Engine API 17
I accidentally clicked on this API's Disable button and got this warning (potentially hinting at what this API is used for):
Disable Google Compute Engine API
Code that uses this project's keys, client IDs, or service accounts
will not be able to call this API.
Do you want to disable this API: Google Compute Engine API?
By clicking on the actual Google Compute Engine API text/link and then to the Quota tab I can see:
Queries per day Unlimited
Queries per 100 seconds 2,000
Queries per 100 seconds per user 2,000
My peak over the last 30 days is 15, so I won't worry about it :)
I do not see any unexpected older app version.
I've created a social networking website with the following:
FrontEnd: Java for web site and hybrid app for mobile.
Web Service: Java web service which returns JSON object.
BackEnd: Neo4j(nosql, graph based database).
I've decided to host this application in any cloud server. After all research I've decided to go with google compute engine. I'm expecting to receive 10 million concurrent users. Since this is a social networking website, the user will upload photos, Like, comments, blogs, chat and etc. My region is going to be Asia/Pacific. Google does provide pricing calculator. Please find the link below:
Click Here for Compute Engine pricing calculator
However I do not know what configuration is required to handing these much of traffic. Please if any one of you have used google compute engine, please help me in selecting the right Servers, Persistent Disk, Load Balancing and GCE Network Bandwidth in Compute Engine Section from the above link. So that I can estimate the cost I need to spend per month.
Thanks for your help in advance.
Nobody can estimate the cost but you. We do not know how well the application was developped and how you use your resources.
Deploy your application on the Google Cloud, build a performance test plan (see tools like JMeter, LoadRunner etc) and test your different assumptions (type of VM, type of disk).
Use monitoring tools (Google Compute Engine now comes with one) to measure anything you need to know (bandwidth, CPU usage and whatnot).This will allow you to estimate the cost.
That way, you will know how much each option costs and will be able to make your own informed decision.
What is the benefit of using luceneappengine over the search api provided by google app engine?
The luceneappengine project was born first of all to make easier the migration of an existing application to google app engine, and next to allow you to deploy a standard lucene application in google app engine environment.
The search api must be your first choice if you are developing a new google app engine application, unless if you think to move your application into another servlet container that will lack of search api.
Other considerations can be done on costs: luceneappengine uses the default datastore to save index (it means that will use datastore read and write), search api has a dedicated cost definition. You can find the cost details here. Unfortunately it's hard to predict the cost/operations of a luceneappengine application because it depends on indexed files (quantity, size, indexed fields etc..).
I have a BerkeleyDB key/value datastore embedded in my Google App Engine project, which does not use the Google Datastore.
When I upload the application to Google App Engine, the database will be less than 1MB.
But is it possible that, as it grows, over time it will eventually exceed an App Engine file-size limitation?
What would be the consequence of that to my running production application?
App Engine DataStore entity is limited to 1MB. You won't be able to save a larger entity.
Free quota for DataStore is 1 GB. If you exceed it with billing disabled, you won't be able to save new entities or enlarge existing entities. If you have billing enabled, you will be charged for additional storage.
App Engine static file limit is 10 MB. Static file can't be changed by your application, only re-uploaded or deleted by appcfg.py tool. You won't be able to upload a file larger than 10 MB.