I am using Google cloud to run some services. speifically, I have code running in firebase, as well as Cloud Run and Cloud SQL.
For some reason, however, I am seeing constant traffic to a compute engine API, which I have no idea why? I don't have any VM's setup (outside of what Cloud SQL is running), but I'm assuming that's a separate service.
Why would I be seeing this traffic against this API? I could "disable" it, but the warning says any resources created with it could soon be deleted and I don't want to bring down my Cloud SQL or Cloud Run instances. There's nothing I can taht would generate 418,000+ requests this month....unless this is just internal Google stuff to support my cloud SQL and Cloud run instances?
Any ideas what this could be? Should it be safe to disable this?
You will see traffic on the Compute Engine API because you are actually using the service.
CE API is not just related for GCE instances, it handles much more of that, for example: Disk, Firewall, Snapshots etc.
You can find more information about the CE API here
So I recommend not disable the CE API because it can cause errors and other problems.
Related
I have a job that involves continuously listening to one or more websocket/mqtt feeds and forwarding this data to an event queue. This job is written in javascript and would run 24/7 in a continuous loop.
The most obvious solution is to run this job on a VM with Compute Engine, but I was wondering is there is a more elegant solution. Azure, for example, has WebJobs that's well-suited to this kind of task. It even restarts the script if there is an error.
Is there some other component on GCP that can run this job in a "managed" way?
Google Cloud does not have a product similar to Azure WebJobs at the moment. Both the standard and flexible environments of Google Cloud App Engine do not currently support websockets. In order to use websockets you can use Compute Engine or Kubernetes Engine.
My Django app is working on GAE with Cloud SQL（MySQL）.
By using New Relic Monitoring, I tracked requests done by my app to www.googleapis.com.
I don't know those requests. Can somebody explain this behavior?
(app connecting with cloud proxy)
As merely described here, there are various processes running in the instance along side (with?) your application... scaling, billing, logging, etc. Communication with other Google Cloud Services (like the ones I previously mentioned) is mainly done through Google Cloud APIs. Since App Engine is a managed Platform as a Service, this management needs to be done somehow... a myriad of REST API requests and responses do this. Partly through the www.googleapis.com endpoint.
You don't have to worry, though... your application's performance is not affected by this, nor your billing account.
I have a machine learning project and for this project, I have to get data from a website in evey 15 minutes. And I decided to use google cloud platform to do it. I've coded a python script to do the process(get the data from website and write down to a csv file) and when I run this script on my computer, it works well. I need to run this script for a couple weeks. So it should be running in google cloud's computers and it should continue running when I close my computer. How can I do this?
I can also use another cloud service if it's required to but google cloud would be better.
Disclaimer: I'm with Google Cloud Platform Support
Google Cloud Compute Engine is defined as an Infrastructure as a Service. It basically provides access to Virtual Machines (VMs), Disks and Networking functionalities. By using this product, you are able to configure your resources from scratch, defining one or multiple VM instances, configuring your work environment, etc. It might require more configuration and boiler plating than needed, but it offers the most control. You can always use some resources for free but in my opinion it is a lot of scratch to start from.
Google Cloud App Engine is defined as a Platform as a Service. It is basically a managed app platform. The management can be automatised to certain degrees. It is based on Compute Engine, in the sense that it provides functionalities, a platform, on top of the infrastructure defined by Compute Engine VMs. You can thus deploy your python script in an App Engine Flexible Python Environment. You can define your whole application as a collection of interrelated microservices, i.e. one service gets the data from a website, maybe another writes csv files and another might trigger ML jobs.
App Engine also provides the possibility to schedule jobs as cron jobs. So if your application needs to run periodical jobs or at a specific time, this is the tool to use. App Engine pricing is correlated with the used resources, but you can estimate eventual budgets by using the Google Cloud Platform Pricing Calculator.
You can store the csv files in Google Cloud Storage as objects in buckets or as data in Datastore, Cloud SQL or BigQuery. Components of Google Cloud Platform can communicate with each other via service acounts. This allows your App Engine deployment, for example, to perform CRUD operations in your Cloud SQL instance, programatically. Or... to trigger a Cloud Machine Learning job.
Your question is very broad and can be addressed in multiple, various ways. I would initially deploy the python script in App Engine Flexible. I would deploy a cron job if needed, to fetch data every 15 minutes. I would upload the csv files in Google Cloud Storage Buckets. I would then use the Cloud Machine Learning python client to trigger Machine Learning jobs programatically.
There are other products that might interest you:
Cloud Dataflow - configure stream/batch data processing
Cloud Dataprerp - transform/clean raw data
Cloud Pub/Sub - global real-time messaging.
All the products/components and sub-products/sub-components can communicate with each other and processes can easily be automated in the Cloud. So the whole project can run in Google's Cloud infrastructure when you close your computer. But, of course, you have to configure it beforehand, in your Google Cloud Platform Project(s).
I am aware that I met your broad question with a broad answer. For any specific issues along your path of implementing the project in the Cloud, the community will be here to provide support.
Good luck!
I want to deploy a multi-service application on GAE using custom runtimes.
I've everything set, my service expects some environment variable to know when to connect to the database and other services.
But I can't see anywhere in the doc where it explains how to connect to other Google Cloud service from a custom runtime.
If it is explained, thanks for pointing me there. if not, can someone who has been successful on doing that explain to me how he/she achieved it.
The documentation for connecting from Flexible Environment apps is found in Cloud SQL > Documentation > MySQL > Connecting from App Engine. Some of the languages (for example, for python) make use of the Cloud SQL Proxy, while others (such as java) don't.
As for Pub/Sub, you can read about client libraries in the documentation.
We are currently running a combined AppEngine / GCE app and thus far have kept all of our datastore access on the AppEngine side of things. Now we are exploring also allowing our GCE instance to make some queries into the (shared) datastore. To start, I'm trying to figure out how to run things locally. What we have so far:
A Go devappserver running
A Go standalone binary that wants to issues queries to the devappserver datastore.
We installed ('go get') google-api-go-client/datastore/v1beta2 so that we can use an API instead of issuing direct HTTP calls. However we are definitely willing to issue direct HTTP calls if this API library won't work in development.
We have service accounts set up (we already access GCS from GCE) but I doubt that's relevant for running locally...
I've seen some docs but they (a) only talk about Python & Java, and (b) discuss connecting to the (standalone) development datastore server, as opposed to the datastore embedded in AppEngine's devappserver (if those are even different?). There is also the following answer here on StackOverflow, but again it discusses connecting to the standalone development datastore server:
How to connect to the local google cloud Datastore db?
Any pointers would be much appreciated!
Ian
Currently this is not possible in the development environment for several reasons. The Google Cloud Datastore tool (gcd.sh) uses the java development server. However when developing go for App Engine you use the python development server, which has different underlying storage. There is a bug to track this issue on the github page.
You can still develop a Google Cloud Datastore application in go however there are many bugs in the current go client library. Unfortunately, the development server does not currently support the JSON API, which the go library uses (see the note at the top of the page).
Update: I wanted to make sure proppy's comment was seen as part of the answer. His suggestion does provide a way to use the protocol version of the API, which is probably more stable than the go client library above. It could also let you use the gcd.sh tool to test this in the development server. You will have to craft the HTTP requests yourself though, and you won't be able to share the data in the datastore between your application and the Cloud Datastore in development. However it is definitely a good workaround and lets you use the Cloud Datastore API, which as it develops will be easier to work with than other workarounds.
From proppy:
Note that you can still use Cloud Datastore Protobuf HTTP API with Go. The protobuf definition is available on GitHub, you can compile it to Go code using the Go protobuf compiler plugin and then send POST HTTP requests to /datastore/{version}/datasets/{datasetId}/{method}.
If the use case from your "GO" app server is straight forward enough, you may want to implement access by using an API call to your GAE service (perhaps extending the service to receive the API calls).
This has the added benefit of only having to make changes in one place if your datastore definitions or functions change.