I have put in about 16000 products on google app engine search index.
The search operation seems to be very slow now.On a lot of occassions,I get timeout error exception(deadline exceeded exception).It is not possible for me to migrate the entire database now.Kindly provide me some means to fasten up the search results.
To increase the time needed for uploading 16000 products to Google App Engine search index, you can try following things:
Run on VPS with very fast Internet connection (like Amazon EC)
Separate your task to many threads
Separate whole task to at least 2 kinds of thread: loading product and uploading product.
Related
I have been working with Google Cloud Platform & Flask for the first time - my client wanted me to deliver a solution on it in 2 weeks.
I have been successful so far in creating a Flask application and wanted to productionize it through Google App Engine. However, the runtime of my function is a little over 2.5 minutes and I get a "504 Gateway Timeout" error. In the code piece, I am accessing bigQuery, google spreadsheets and GCS buckets. Should I switch to a new GCP service or can some tweaks in my code/yaml file suffice? My yaml config is -
runtime: python37
liveness_check:
check_interval_sec: 300
timeout_sec: 299
failure_threshold: 10
success_threshold: 10
initial_delay_sec: 500
readiness_check:
app_start_timeout_sec: 1800
I would be very, very grateful to anyone who can help me resolve this issue.
Thank you!
Edit: Just to give a brief about the application - this is a forecasting application that reads data from bigquery, GCS buckets, and Google Spreadsheets, processes it and runs ML models on it. The results are written back to Google Spreadsheets within the application itself (i.e. no response needed from the application per see). I'm triggering the application using google AppsScript.
The best option is to restructure how your website works. You can then stick with GAE standard. Even if you switch to GAE flexible, it will work better this way.
Here is the sequence of operations:
A user submits a request from web page.
Your website returns immediately with a page that indicates that you are working on it.
The page you return includes Javascript that will poll your website to ask if the task is complete.
When the task is complete, the Javascript will update the page and present the results to the user.
On the server side, you can use cloud tasks for doing the processing which I believe has a time limit of 10 minutes.
This way, the user is always viewing a web page and isn't sitting there waiting and looking at a blank screen while waiting for the server to return something.
Im a little confused about this because the docs say I can use stackdriver for "Request logs and application logs for App Engine applications" so does that mean like web requests? Does that mean like millions of web requests?
Stackdriver's pricing is per resource so does that mean I can log all of my web servers web request logs (which would be HUGE) for no extra cost (meaning I would not be charged by the volume of storage the logs use)?
Does stackdriver use GCP cloud storage as a backend and do I have to pay for the storage? It just looks like I can get hundreds of gigabytes of log aggregation for virtually no money just want to make sure Im understanding this.
I bring up ELK because elastic just partnered with google so it must not do everything elasticsearch does (for almost no money) otherwise it would be a competitor?
Things definitely seem to be moving quickly at Google's cloud division and documentation does seem to suffer a bit.
Having said that, the document you linked to also details the limitations -
The request and application logs for your app are collected by a Cloud
Logging agent and are kept for a maximum of 90 days, up to a maximum
size of 1GB. If you want to store your logs for a longer period or
store a larger size than 1GB, you can export your logs to Cloud
Storage. You can also export your logs to BigQuery and Pub/Sub for
further processing.
It should work out of the box for small to medium sized projects. The built in log viewer is also pretty basic.
From your description, it sounds like you may have specific needs, so you should not assume this will be free. You should factor in costs for Cloud Storage for the logs you want to retain and BigQuery depending on your needs to crunch the logs.
I am running the phpMyAdmin app on my Google App Engine project.
My SQL properties:
Here is the Google App Engine instance summary:
This might be of relevance?
I am at a complete loss, why it takes 5 seconds to get the page loaded on average. Is GAE even a feasible option for someone trying to run a website. I know I'm using the shared CPU but I am the only one currently (trying to) using it.
I was handed an assignment but I don't know where to start.
The aim is to have 2 piece of code running. One will run in Open stack private cloud and perform the task of indexing two sets of text, with another running in EC2 with the task of matching the two indexed tests.
I want to access them via google app engine.
Ideally, I would like to click a button or perform an action on Google app engine, which then sends a request to Openstack to run the code and retrieve the output of a txt file.
That outputted text files will then be forwarded onto EC2 where the matching will occur and the results sent back to Google App Engine.
My question is, how can I send the files between the systems using REST requests?
FrankN --
EC2, GAE and OpenStack are disparate cloud computing platforms. To integrate them might include, say, using one platform while saving backups to another.
CloudU.Rackspace.com is a vendor-neutral education site about cloud computing (note: It'll take six or so hours to finish it all). This might help.
Disclaimer: I work for Rackspace.
Firstly, not really sure what your requirements are, why your code does or why are you trying to mix cloud providers in that way.
That said, I would suggest taking the upload from GAE and push it to AWS S3 where you can then retrieve and use as you please from EC2.
Not sure what functionality you are trying to get out of OpenStack that is not present in AWS; however, I would suggest building whatever you are building in EC2 first, then duplicate in on OpenStack services to avoid future vendor lock in.
My application is currently on app engine server. My application writes the records(for logging and reporting) continuously.
Scenario: Views count in the website. When we open the website it hits the server to add the record with time and type of view. Showing these counts in the users dashboard.
Seems these requests are huge now. For now 40/sec. Google App Engine writes are going heavy and cost is increasing like anything.
Is there any way to reduce this or any other db to log the views?
Google App Engine's Datastore is NOT suitable for such a requirement where you have to continuously write to datastore and read less often.
You need to offload this task to a third party service (either you write one or use existing one)
Better option for user tracking and analytics is Google Analytics (Although you wont be directly able to show the hit counters on website using analytics).
If you want to show your user page hit count use a page hit counter: https://www.google.com/search?q=hit+counter
In this case you should avoid Datastore.
For this kind of analytics it's best to do the following:
Dump data to GAE log (yes, this sounds counter-intuitive, but it's actually advice from google engineers). GAE log is persistent and is guaranteed to not loose data you write to it.
Periodically parse the log for your data and then export it to BigQuery.
BigQuery has a quite powerful query language so it's capable of doing complex analytics reports.
Luckily this was already done before: see the Mache framework. Also see related video.
Note: there is now a new BigQuery feature called streaming inserts, which could potentially replace the cumbersome middle step (files on Cloud Storage) used in Mache.