Is there a way to run app engine and neo4j in one application locally? What I want to do is fetch app engine datastore entities and create my neo4j graph database using it. I checked app engine documentation where they have specified what all external libraries it supports. Right now, I am running into an issue where my csv stream response is more than 32 MB. Also, I can't use Compute Engine. Pricing is the issue and I am making too many requests!
Related
I have a machine learning project and for this project, I have to get data from a website in evey 15 minutes. And I decided to use google cloud platform to do it. I've coded a python script to do the process(get the data from website and write down to a csv file) and when I run this script on my computer, it works well. I need to run this script for a couple weeks. So it should be running in google cloud's computers and it should continue running when I close my computer. How can I do this?
I can also use another cloud service if it's required to but google cloud would be better.
Disclaimer: I'm with Google Cloud Platform Support
Google Cloud Compute Engine is defined as an Infrastructure as a Service. It basically provides access to Virtual Machines (VMs), Disks and Networking functionalities. By using this product, you are able to configure your resources from scratch, defining one or multiple VM instances, configuring your work environment, etc. It might require more configuration and boiler plating than needed, but it offers the most control. You can always use some resources for free but in my opinion it is a lot of scratch to start from.
Google Cloud App Engine is defined as a Platform as a Service. It is basically a managed app platform. The management can be automatised to certain degrees. It is based on Compute Engine, in the sense that it provides functionalities, a platform, on top of the infrastructure defined by Compute Engine VMs. You can thus deploy your python script in an App Engine Flexible Python Environment. You can define your whole application as a collection of interrelated microservices, i.e. one service gets the data from a website, maybe another writes csv files and another might trigger ML jobs.
App Engine also provides the possibility to schedule jobs as cron jobs. So if your application needs to run periodical jobs or at a specific time, this is the tool to use. App Engine pricing is correlated with the used resources, but you can estimate eventual budgets by using the Google Cloud Platform Pricing Calculator.
You can store the csv files in Google Cloud Storage as objects in buckets or as data in Datastore, Cloud SQL or BigQuery. Components of Google Cloud Platform can communicate with each other via service acounts. This allows your App Engine deployment, for example, to perform CRUD operations in your Cloud SQL instance, programatically. Or... to trigger a Cloud Machine Learning job.
Your question is very broad and can be addressed in multiple, various ways. I would initially deploy the python script in App Engine Flexible. I would deploy a cron job if needed, to fetch data every 15 minutes. I would upload the csv files in Google Cloud Storage Buckets. I would then use the Cloud Machine Learning python client to trigger Machine Learning jobs programatically.
There are other products that might interest you:
Cloud Dataflow - configure stream/batch data processing
Cloud Dataprerp - transform/clean raw data
Cloud Pub/Sub - global real-time messaging.
All the products/components and sub-products/sub-components can communicate with each other and processes can easily be automated in the Cloud. So the whole project can run in Google's Cloud infrastructure when you close your computer. But, of course, you have to configure it beforehand, in your Google Cloud Platform Project(s).
I am aware that I met your broad question with a broad answer. For any specific issues along your path of implementing the project in the Cloud, the community will be here to provide support.
Good luck!
Im creating a Node.js website that probably won't have loads of traffic, and was looking into cheap solutions to host the site. Came across Google cloud services offering free usage for their services with limits. A f1-mirco is more than enough for my needs, but I will happily pay for some usage if it goes over by any chance.
I wanted to setup a linux centOS 7 on GCE (which I already did), and run my application and REST API on it. Now here comes the problem.
I tried to use Google's datastore service, but it sprung an app engine instance and without it datastore won't work.
Is datastore entirely relying on app engine to function?? In the docs, it said if you use any of the client API, it requires app engine. What can I do to not use the client api and query data then? Don't want to use the app engine at the moment or datastore is just not for me then?
Thanks for any help!
Some of the underlying infrastructure of Cloud Datastore and App Engine are still tied together for creation, etc. So while creating an Cloud Datastore database also defines an App Engine instance for the project, it doesn't require you to use it. You don't get charged for App Engine either, unless you decide to deploy an App using it.
You should be totally fine use the Google Cloud Node client library on the f1 micro instance.
We got a couple millions data in the current GAE project using Google Cloud store. Mostly GPS point information. We want to be able to use all these GPS points in another demo instance, which is hosted in another GAE instance. Anyway we can do it?
Using Golang + Google App Engine
There is a Google Cloud Datastore API that you can use to access your Datastore data from any other deployment, including a different App Engine app. It's not available in Go, so you will have to mix in some Python or Java.
MySQL has this Sample Sakila DB where we can start playing around with bunch of data already for our application, how about for Google App Engine/GAEJ is there something like this for the datastore?
I started recently to experiment with the Google App Engine and I was confronted with the same question. I was interested in a REST based app engine backbone which I could easily load/unload with data but couldn't find something to play around.
So I started to build up two projects on github which supports me in such kind of work.
clb-appEngineTemplate is a skeleton application for a Google App Engine Jave REST backend. It provides some sample code for a standardized REST API based persistency layer on Business Object level and can be easily extended (using Objectify and GSON).
clb-test which is a utility class which allows to load Test Data from Excel CSV file into your Google App Engine REST backend.
Both projects are maven based and allow me easily to define data objects which I can upload into the App Engine. Mainly I'm run them against the local test server, which serves me for initial testing.
I just released a first version and will incrementally extend over the next weeks.
AFAIK, there is no sample DB for GAE, probably because datastore write operations are expensive. There are demos bundled with GAE SDK. If you are using Eclipse you can import the samples to your workspace. Some of them involve datastore so you can run the application and add data yourself.
Another way is to use bulkloader to upload data at once using CSV files. But you can quickly run out of free quota for datastore writes.
I was having a look at the new Google Cloud Datastore and looks great. But there is something I could not understand... is it supposed to substitute Google App Engine Datastore? How can I use it inside GAE? What are the differences between both of them?
I have a GAE app in Java that uses 3 entities with thousands of rows each one, and I need to do joins quite often...
The cloud datastore is the App Engine datastore, for use outside App Engine. You won't get any benefit trying to use it with your App Engine app, unless you need other external apps to also have access to the data.
You certainly won't get more efficient joins. If you really need that, perhaps you should look into Cloud SQL, which is basically a version of MySQL you can use from App Engine.
They both are the same, in fact Google App Engine can use Google Cloud DataStore as one of its way to store data, the other options include Google Cloud SQL, Google Cloud Storage. You may select among these three according to the type of data you want to store and the way you want to access them.
From your question its clear that Google Cloud SQL would be right choice as no other options provide joins for retrieving results.
Cloud SQL is nothing but MySQL(the popular open source database) running on google platform. So you can perform your regular SQL-like queries to get your results.