After reading available answers/comments on similar questions on this forum, it is now evident that GAE app is not straight-forward ready to be deployed on Compute engine. I fully understand that what all the managed services(mostly as APIs be it, datastore, document/index Search, memcache, cloud storage, task queues, cron jobs etc.), App Engine offered being a platform, won't be the same-fashioned accessible/integration-ready if available on Compute engine at all.
We have a 5 years old fully-grown App engine app now.
I am considering a scenario to support high-level of customization/control and adding third party softwares/middlewares to our server environment which is not possible with App engine. So if we have all solutions(Compute Engines, Container Engines etc.) other than App engine, to migrate our application to meet such requirements, what is the cost of such migration?
Need of server provisioning and configuration at Compute engines with different pricing model[Understood, should not be a problem :)]
Full or partial code rewrite to continue using the same APIs esp. Datastore, Cloud Storage, Task Queues, Cron jobs, Document Search, Memcache etc.[Need confirmation here and any reference/link to migration guide would be help!!]
Does this lead to risk of losing any managed service/API offered from App Engine? Document Search, Memcache, Task Queues, Cron jobs seem the possible candidates. Please confirm.
As per my reading, Big Query, Cloud storage, Pub-Sub APIs integration should not be much affected with such migration(Client-libraries or Rest APIs should still help!). Please confirm.
In nut-shell, We wanted it fully managed in the beginning so PaaS seemed the right choice 5 years ago. Now we want App minus platform-managed plus customized/flexible to our choice. How complicated this transition is going to be?
Full or partial code rewrite to continue using the same APIs esp. Datastore, Cloud Storage, Task Queues, Cron jobs, Document Search, Memcache etc.[Need confirmation here and any reference/link to migration guide would be help!!]
Unfortunately, some of those service only be provided on GAE, such as Document Search. But most of service can be use directly for GCP, such as Datastore, Cloud Storage. GAE Flexible Environment is much like GCP environment, so you can read this article first Migration to GAE Flexible Environment
In following articles also have some answer：
How to migrate Google App Engine Project to Compute Engine completely
Google App Engine Blobstore to Google Cloud Storage Migration Tool
Does this lead to risk of losing any managed service/API offered from App Engine? Document Search, Memcache, Task Queues, Cron jobs seem the possible candidates. Please confirm.
Yes, Document Search only available on GAE.
As per my reading, Big Query, Cloud storage, Pub-Sub APIs integration should not be much affected with such migration(Client-libraries or Rest APIs should still help!). Please confirm.
Yes, but you may need to change SDK or library. It dependency on your language and how to call those service by Rest API directly or SDK.
Related
I can't find any documentation on how gcp scheduler works under the hood. An App Engine is needed in the project, so I assume that the Http calls or Pub/Sub messages are started from the App Engine.
Currently I can use a cloud scheduler even without an App Engine in the project. Apparently a compute engine that also contains a permanently running VM is also sufficient. Could someone confirm my assumptions please or does anyone have sources on this?
I can't tell you how work Cloud Scheduler under the hood. I can just tell you that works well!
I'm sure there is a VM, or a cluster of VM, on Google serverless environment, and your Cloud Scheduler job is set on it. It's serverless, the under the hood doesn't matter, it works, and it's what I want!
Now, the relation with App Engine can be confusing. In fact, there is no longer relation between the product now, but you need the App Engine API activated on your project to use Cloud Scheduler. This strange things is normal if you have been using Google Cloud for a while. At the beginning, only App Engine existed, and Datastore, Cloud Task, Cloud Scheduler was all "modules" of App Engine. Years, after years, google has refactored and extracted these modules to create independent products, as you can see them today. However, some relations are still present, like the API activation.
Our project was running in GCP compute engine. For scaling purpose, it is moved to app engine. We had rabbitmq implemented for push messages and chatbots in compute engine. In app engine it is not feasible to implement rabbitmq. So I was going through alternate options. There I found cloud task option. But I have doubts in certain areas even after reading their documentation
In my understanding, we need an app engine instance for cloud tasks. In that case, can I implement it in same project itself as a different service? Will this affect the performance of the existing project?
Is there any better solution than cloud tasks in this case?
You can implement additional services under your app in the App Engine as shown in this diagram.
By default, App Engine scales your app to match the load. Your apps will scale up the number of instances that are running to provide consistent performance, or scale down to minimize idle instances and reduces costs.
You can consider running a RabbitMQ Cluster on Google Kubernetes Engine. You can find more information in the following documentation: rabbitmq.
I have a machine learning project and for this project, I have to get data from a website in evey 15 minutes. And I decided to use google cloud platform to do it. I've coded a python script to do the process(get the data from website and write down to a csv file) and when I run this script on my computer, it works well. I need to run this script for a couple weeks. So it should be running in google cloud's computers and it should continue running when I close my computer. How can I do this?
I can also use another cloud service if it's required to but google cloud would be better.
Disclaimer: I'm with Google Cloud Platform Support
Google Cloud Compute Engine is defined as an Infrastructure as a Service. It basically provides access to Virtual Machines (VMs), Disks and Networking functionalities. By using this product, you are able to configure your resources from scratch, defining one or multiple VM instances, configuring your work environment, etc. It might require more configuration and boiler plating than needed, but it offers the most control. You can always use some resources for free but in my opinion it is a lot of scratch to start from.
Google Cloud App Engine is defined as a Platform as a Service. It is basically a managed app platform. The management can be automatised to certain degrees. It is based on Compute Engine, in the sense that it provides functionalities, a platform, on top of the infrastructure defined by Compute Engine VMs. You can thus deploy your python script in an App Engine Flexible Python Environment. You can define your whole application as a collection of interrelated microservices, i.e. one service gets the data from a website, maybe another writes csv files and another might trigger ML jobs.
App Engine also provides the possibility to schedule jobs as cron jobs. So if your application needs to run periodical jobs or at a specific time, this is the tool to use. App Engine pricing is correlated with the used resources, but you can estimate eventual budgets by using the Google Cloud Platform Pricing Calculator.
You can store the csv files in Google Cloud Storage as objects in buckets or as data in Datastore, Cloud SQL or BigQuery. Components of Google Cloud Platform can communicate with each other via service acounts. This allows your App Engine deployment, for example, to perform CRUD operations in your Cloud SQL instance, programatically. Or... to trigger a Cloud Machine Learning job.
Your question is very broad and can be addressed in multiple, various ways. I would initially deploy the python script in App Engine Flexible. I would deploy a cron job if needed, to fetch data every 15 minutes. I would upload the csv files in Google Cloud Storage Buckets. I would then use the Cloud Machine Learning python client to trigger Machine Learning jobs programatically.
There are other products that might interest you:
Cloud Dataflow - configure stream/batch data processing
Cloud Dataprerp - transform/clean raw data
Cloud Pub/Sub - global real-time messaging.
All the products/components and sub-products/sub-components can communicate with each other and processes can easily be automated in the Cloud. So the whole project can run in Google's Cloud infrastructure when you close your computer. But, of course, you have to configure it beforehand, in your Google Cloud Platform Project(s).
I am aware that I met your broad question with a broad answer. For any specific issues along your path of implementing the project in the Cloud, the community will be here to provide support.
Good luck!
Im creating a Node.js website that probably won't have loads of traffic, and was looking into cheap solutions to host the site. Came across Google cloud services offering free usage for their services with limits. A f1-mirco is more than enough for my needs, but I will happily pay for some usage if it goes over by any chance.
I wanted to setup a linux centOS 7 on GCE (which I already did), and run my application and REST API on it. Now here comes the problem.
I tried to use Google's datastore service, but it sprung an app engine instance and without it datastore won't work.
Is datastore entirely relying on app engine to function?? In the docs, it said if you use any of the client API, it requires app engine. What can I do to not use the client api and query data then? Don't want to use the app engine at the moment or datastore is just not for me then?
Thanks for any help!
Some of the underlying infrastructure of Cloud Datastore and App Engine are still tied together for creation, etc. So while creating an Cloud Datastore database also defines an App Engine instance for the project, it doesn't require you to use it. You don't get charged for App Engine either, unless you decide to deploy an App using it.
You should be totally fine use the Google Cloud Node client library on the f1 micro instance.
What is better to code with php in google app engine or in amazon-ec2.
I think it is better in amazon-ec2 because they support datastore with php and google app engine doesn't, what do you think ?
While its not possible to access the appengine datastore, google has a new preview service for "cloud datastore", google "cloud datastore from php" and you will see how to use it.
You really cant compare AWS with appengine as one requires managing servers and scalability manually and their noSQL solution (dynamoDB) is a joke compared to google's datastore, for example in dynamoDB you must provision your writes beforehand and even if you are all day changing provisioning, it takes sometimes hours to propagate the new setting.
I had never use Google App Engine, but several times AWS systems, and sure, as AWS EC2 could be used as Linux Server Instance, I recommend you that provider. And coz' it seems that you use PHP, they have strong API for this langage. Have fun with AWS.