I need advice if it is worthwhile to explore Google APP Engine option, so if learned and experienced user could comment, it would really help (I do not need code)
Present Scenario:
I have a website, where the data need to be updated every second ? it is built on .NET, and a user need to have updated data every time they visit, the data changes every second. The users have bookmarked the URLs so the data is changed and URL remains the same.
We also have a lot static data, which users access for researching and reading.
Experience with cloud:
We had tried using the Website with one of the Big Players (not with the original cloud company, with their nearest competitor ;) we had problems the file getting stuck at times (essentially some users are seeing update, some not), and they had 'Modified Trust' rights level implemented, which was restricting us at multiple places (Auto Generating files in directory)
My Questions:
(a) You think in above scenario, Google App Engine could help ?
(b) URL re-writing more specifically generating 200 server return instead of 404 would that be possible or the 404 being trapped and coverted into 302 and redirected ?
(c) We had a hole in the pocket on hosting fees when we moved from traditional to cloud and now we are back on traditional server with Load Balancer, do you think on heavy traffic site do we stick with traditional or look at google app to lower our costs ?
I look forward in hearing comments..
Thanking everyone in advance.
(a) You think in above scenario, Google App Engine could help ?
The problem with users not seeing data is a factor of caching or eventual consistency in your database. That's not going to be "solved" by moving to a new cloud provider. The appengine datastore uses eventual consistency, but you can solve that problem by using memcache to store data that changes frequently. That said, Appengine doesn't give you complete control over memcache so you may still have problems solving that issue.
(b) URL re-writing more specifically generating 200 server return instead of 404 would that be possible or the 404 being trapped and coverted into 302 and redirected ?
Not really sure what you mean here. You can certainly return 302 or 200 responses instead of 404s using any web framework worth its salt
(c) When designed well, appengine can be very cost effective, but when not optimized it can be a money sink... there are a lot of good papers out there about how to effectively optimize it, but if you are talking about a lot of users hitting the site every second you are going to pay for it.
Related
I have been working with Google Cloud Platform & Flask for the first time - my client wanted me to deliver a solution on it in 2 weeks.
I have been successful so far in creating a Flask application and wanted to productionize it through Google App Engine. However, the runtime of my function is a little over 2.5 minutes and I get a "504 Gateway Timeout" error. In the code piece, I am accessing bigQuery, google spreadsheets and GCS buckets. Should I switch to a new GCP service or can some tweaks in my code/yaml file suffice? My yaml config is -
runtime: python37
liveness_check:
check_interval_sec: 300
timeout_sec: 299
failure_threshold: 10
success_threshold: 10
initial_delay_sec: 500
readiness_check:
app_start_timeout_sec: 1800
I would be very, very grateful to anyone who can help me resolve this issue.
Thank you!
Edit: Just to give a brief about the application - this is a forecasting application that reads data from bigquery, GCS buckets, and Google Spreadsheets, processes it and runs ML models on it. The results are written back to Google Spreadsheets within the application itself (i.e. no response needed from the application per see). I'm triggering the application using google AppsScript.
The best option is to restructure how your website works. You can then stick with GAE standard. Even if you switch to GAE flexible, it will work better this way.
Here is the sequence of operations:
A user submits a request from web page.
Your website returns immediately with a page that indicates that you are working on it.
The page you return includes Javascript that will poll your website to ask if the task is complete.
When the task is complete, the Javascript will update the page and present the results to the user.
On the server side, you can use cloud tasks for doing the processing which I believe has a time limit of 10 minutes.
This way, the user is always viewing a web page and isn't sitting there waiting and looking at a blank screen while waiting for the server to return something.
I have created a social media app using Parse backend. I don't have any user at this point but I am anticipating to get lots of users in the future. I have several problems with Parse:
1- Parse limits API calls and my app needs to work on million users' phone and 600 call / sec is not enough to support millions of users
2- Auto Scalability is a critical requirement for social media and Parse doesn't provide it
3-There is no tech support from Parse team and they are not even available to solve our issues. I don't know why they do not provide paid consulting services to help their customers
So I decided to move the backend to Google App engine
My Questions:
1- Is there any way to solve the three issues mentioned above Using Parse so that I stay on Parse without migration to another backend? and How?
2- If the answer is No, what's the best mobile backend to accommodate my requirements?
3- what are the challenges moving from Parse to Google App Engine (GAE)
4- Parse keeps everything in MangoDb - what happens if I create another MangoDb on GAE and replicate the same DB schema? how about queries and cloud code that we have written on MangoDb and Parse? do we need to rewrite them again?
I would truly appreciate it if you guys would give me some advise - I am ready to hire someone who has done this kind of work in the past and has a good experience on Parse.
Thank you
Ben
I have a front-end angular app using firebase to store user data.
I currently do not have a backend set up, such as a node.js server.
I would like to use the Google Docs API to upload files from my app.
Since the Great Firewall of China does not (or makes unstable) the use of Google services, is it possible to place those services on the backend server and still use them reliably?
Perhaps after they have uploaded the document to firebase, a backend script retrieves it, uploads it to google docs, and then removes the record from firebase? Just trying to see if Google or similar services are even feasible for this use case.
I suppose the crux of my question is whether or not the calling of the Google API would be taking place on the user's computer, in which case would it become unstable?
** Updates for clarity:
I am deciding whether my firebase-backed app needs a more traditional backend like a node server to do things like: upload images and documents, send mail via Mandrill, etc... It would be helpful to me if I knew whether, after putting in the time to create a server, some of the services I am after (aka APIs) are any more resilient to the GFW than they would be if they ran on the client side. So if any one has had success in such a task, I would like to know.
** Technical update:
So, for example, if I run the Google Maps API on the client side, if the user is in China and is not running a VPN, accessing the API calls will either lag or time out or (rarely) success in returning the scripts. If I was somehow able to able to process the map query "off-site" aka on the server, could I then return with a static image of the map to a Chinese user without fail?
If I was somehow able to able to process the map query "off-site" aka
on the server, could I then return with a static image of the map to a
Chinese user without fail?
Yes, of course. What you are going to miss this way is all the front-end interactive functionality Google Maps offers. But if that's ok in your use case, sure.
I have never tried it with the GCF, but what I would do is this:
Google Maps <-> Your Reverse proxy <-> User
So, instead of the user visitng the real google maps site, it will be visiting your maps.mydomain.com site, that will be sitting in between, proxying everything.
Nginx is an excellent choice for a reverse proxy. If you need more control, there are good node.js reverse proxying packages that you an use to rewrite the content extensively before serving it (perhaps to obfuscate it in case the GCF blacklists content based on pattern matching, or to change the script names/links again to avoid pattern matching).
You are misunderstanding about the great firewall of China. I consulted for a couple of Chinese companies after the dot com crash so I can say this from personal experience, not hearsay.
It is mostly high-end Cisco hardware behind gateways behind their government telecom infrastructure. Nowadays they knock off what hardware they can, every chance they can, and spend money on specialized hardware to monitor cell phones systems.
There was a brief mention of the street-level surveillance hardware on 20/20 before the crash if you are interested in looking it up.
Not to discourage you, but I say set up whatever open servers you want with whatever frontends or backends you want, but the reality is the traffic is not going to be there.
That is why they call it an oppressive regime, they do not get to decide for themselves, remember?
I have a small question with possibly a complex answer. I have tried to research around, but I think I may not know the keywords.
I want to build a web service that will send a JSON response, which would be used for another application. My goal is having the App Engine server crawl a set of webpages and store the relevant values so the second application (client) would not need to query everything. It will only go to my server with the already condensed information.
I know, it's pretty common, but how can I defend from attackers who wish to exhaust my App Engine resources/quota?
I have been thinking on limiting the amount of requests by IP (say.. 200 requests / 5 minutes), but is that feasible? Or is there a better, and more clever way of doing it?
First, you need to cache the JSON. don't hit the datastore for every request. use memcache or possibly, depending on your requirements, you can cache the JSON in a static file in Cloud Storage. This simple is the best defender against DDOS, since every request adds minimal overhead.
Also, take a look in the DDOS protection service offered by app engine:
https://developers.google.com/appengine/docs/java/config/dos
You could require users to log-in then generate and send an auth key to the client app that must accompany any requests to the app engine service.
I put one app on google app engine.
My app has one cronjob which is parse data from Internet and store into my db.
When user using my app, it will extract data from db, and show data to users.
I found that is too time consuming and too many request from db.
I want to revise each page when the cronjob running daily.
Then user can see the page without query my database.
How can I do that in GAE ?
Thank you for your reply.
Not nearly enough info in the question to help you. For example, what does "too many request from db" mean? Is it because you have a lot of traffic? Or you are querying the db too much?
Possible solutions are:
Edge cache your page: https://groups.google.com/forum/#!topic/google-appengine/6xAV2Q5x8AU/discussion
Store your page in memcache.
Optimize your database accesses. Most likely you're doing something very inefficiently.
Use a cronjob to generate your page, store it in the blobstore, and redirect your fetches to the blobstore. You can do this, but this is a pretty dumb way to go about it, given that there are better options.
I'm afraid it's a limitation of GAE, according to this post, no matter how much caching and tricky solutions you find. They just worsen google's policies.
You can't have what you need if you don't write directly inside the html file and store it on the server every time, which is far more resource consuming and, in my opinion, just pointless. Since GAE is a free service, with the purpose of testing, you should acquaint with what you have, or pay.