Is there a way to access task queues and the datastore from a Google Compute Engine application? If so, please point me in the right direction, if not, then I guess I'll need to build an API on the Google Compute Engine side and call the API from the Google App Engine side (less ideal)
What I'm trying to do is to have a Google App Engine pop an ID of an entity on the Task Queue, once the application on Google Compute Engine receives it, it will do a lookup and retrieve that record from the datastore, do some heavy computation on it (which is not suitable for App Engine due to its various limitations) and then store the result directly into the entity.
Yes, it's possible. You need to use a PULL queue and the Task Queue REST API to read the message inside GCE. More information here: https://developers.google.com/appengine/docs/java/taskqueue/rest/libraries
If you need to read from datastore directly, last year Google announced the Datastore API, so it should also be possible, although I have never done that. Anyway, perhaps it's possible to put the entity itself in the PULL queue to avoid another operation coming from the GCE.
Cheers!
Related
After reading available answers/comments on similar questions on this forum, it is now evident that GAE app is not straight-forward ready to be deployed on Compute engine. I fully understand that what all the managed services(mostly as APIs be it, datastore, document/index Search, memcache, cloud storage, task queues, cron jobs etc.), App Engine offered being a platform, won't be the same-fashioned accessible/integration-ready if available on Compute engine at all.
We have a 5 years old fully-grown App engine app now.
I am considering a scenario to support high-level of customization/control and adding third party softwares/middlewares to our server environment which is not possible with App engine. So if we have all solutions(Compute Engines, Container Engines etc.) other than App engine, to migrate our application to meet such requirements, what is the cost of such migration?
Need of server provisioning and configuration at Compute engines with different pricing model[Understood, should not be a problem :)]
Full or partial code rewrite to continue using the same APIs esp. Datastore, Cloud Storage, Task Queues, Cron jobs, Document Search, Memcache etc.[Need confirmation here and any reference/link to migration guide would be help!!]
Does this lead to risk of losing any managed service/API offered from App Engine? Document Search, Memcache, Task Queues, Cron jobs seem the possible candidates. Please confirm.
As per my reading, Big Query, Cloud storage, Pub-Sub APIs integration should not be much affected with such migration(Client-libraries or Rest APIs should still help!). Please confirm.
In nut-shell, We wanted it fully managed in the beginning so PaaS seemed the right choice 5 years ago. Now we want App minus platform-managed plus customized/flexible to our choice. How complicated this transition is going to be?
Full or partial code rewrite to continue using the same APIs esp. Datastore, Cloud Storage, Task Queues, Cron jobs, Document Search, Memcache etc.[Need confirmation here and any reference/link to migration guide would be help!!]
Unfortunately, some of those service only be provided on GAE, such as Document Search. But most of service can be use directly for GCP, such as Datastore, Cloud Storage. GAE Flexible Environment is much like GCP environment, so you can read this article first Migration to GAE Flexible Environment
In following articles also have some answer：
How to migrate Google App Engine Project to Compute Engine completely
Google App Engine Blobstore to Google Cloud Storage Migration Tool
Does this lead to risk of losing any managed service/API offered from App Engine? Document Search, Memcache, Task Queues, Cron jobs seem the possible candidates. Please confirm.
Yes, Document Search only available on GAE.
As per my reading, Big Query, Cloud storage, Pub-Sub APIs integration should not be much affected with such migration(Client-libraries or Rest APIs should still help!). Please confirm.
Yes, but you may need to change SDK or library. It dependency on your language and how to call those service by Rest API directly or SDK.
Can google app engine be used running a web crawler?
I have a crawler process that starts with a specific id (or ASIN number) of an amazon web store product. The process then uses amazon product advertising API to get similar products. After the similar products are fetched, their ASIN numbers are used to call the API for other similar products. This process is suppose to get the information of about 25,000 products which roughly takes 3-6 hours.
Now my client wants to run multiple instances of this process with different initial ASIN numbers. I gave him a proposal on how the whole system can be made using google compute engines, but he is eager to use google app engine. He thinks that app engine will somehow automatically make crawler instances.
Since google app engine is used for backends and APIs, I have concerns that deploying such a crawler system on app engine would have its limitations.
Yes. You could fetch the base URL, scrape the data, get the urls for related products, then spin up taskqueues to fetch each subsequent url. Rinse and repeat.
It seems like you can get events from Google Cloud Platform, but not from Google App Engine services. Why is that?
Thanks
Cloud Functions is a new service on GCP, whereas Cloud Datastore is a high query per second (QPS) service that is more sensitive to latency changes.
This means it's still early and integration with Cloud Datastore is more complex than most other systems. It's been looked into, but nothing to announce yet.
Have you looked at Datastore Callbacks? They're somewhat analogous to triggers in an RDBMS environment. For example, you could write callback methods that get invoked every time an entity of a particular Kind is put or deleted.
https://cloud.google.com/appengine/docs/standard/java/datastore/callbacks
I know most likely the answer will be no but I would still like to ask even if I searched.
Is there any way to be able to update the memcache of app engine from a compute engine? I am using dedicated memcache for my appengine but I would like to be able to edit it also from a compute engine instance.
Write a simple REST API on the AppEngine to GET/POST/PUT/DELETE entries in memcache. Then call that API from your Compute Engine. You could probably write this in Go or Python.
I have made a few stand alone apps that don't pull too much information from the internet. However, I want to make an app that allows users to post data which is then used by every user who has the app installed, like an online communal DB/datastore.
From what I've read it looks like Google App Engine is the solution and can be used in this sort of manner (looking at the "guestbook" demo app").
Can I use the Google app engine to provide data for an app?
If you're talking about an Android application, I think you can.
Depending on what kind of data you wish to store on your GAE datastore,
you can make a RESTful way of retreiving data from your GAE datastore.
For one example, you may be able to retrieve that data as a JSON object
Application sends GET request to the server, and the server responds with a JSON object.