We are using Spring Cloud Gateway [JavaDsl] as our API Gateway. For the proxies we have multiple microservices [running on different ip:port] as target's. Would like to know either we can configure multiple targets to spring cloud gateway proxies, similar to apache camel load balancer eip.
camel.apache.org/manual/latest/loadBalance-eip.html
We are looking for software load balancing with in spring cloud gateway [similar to netflix/apache-camel] instead of another dedicated LB.
Able to get Spring Cloud Gateway load balanced Route working using spring-cloud-starter-netflix-ribbon. However when one of the server instance is down, load-balancing-fails. Code Snippets below.
Version :
spring-cloud-gateway : 2.1.1.BUILD-SNAPSHOT
Gateway Route
.route(
r ->
r.path("/res/security/")
.filters( f -> f
.preserveHostHeader()
.rewritePath("/res/security/", "/targetContext/security/")
.filter(new LoggingFilter())
.uri("lb://target-service1-endpoints")
)
application.yml
ribbon:
eureka:
enabled: false
target-service1-endpoints:
ribbon:
listOfServers: 172.xx.xx.s1:80, 172.xx.xx.s2:80
ServerListRefreshInterval: 1000
retryableStatusCodes: 404, 500
MaxAutoRetriesNextServer: 1
management:
endpoint:
health:
enabled: true
Here is the response from Spring Cloud Team.
What you've described, indeed, happens. However, it is not gateway-specific. If you just use Ribbon in a Spring Cloud project with listOfServers, the same thing will happen. This is because, unlike for eureka, the IPing for non-discovery-service scenario is not instrumented (a DummyPing instance is used).
You could probably change this behaviour by providing your own IPing, IRule or ServerListFilter implementation and overriding the setup we provide in the autoconfiguration in this way.
https://github.com/spring-cloud/spring-cloud-gateway/issues/1482
I've got auto scaling on in my app engine flexible (PHP) app.yaml file. Is there a gcloud or console way to figure out the machine class of the instances that have been assigned to my service?
I know that I can set the resources I need in terms of disk and RAM in my yaml file, but this is a minimum; actual allocated machines may be different. I'm looking to determine what was allocated after the machine is running.
There's a public feature request already opened for this feature, however it isn't yet available.
As a workaround, you can use the Google API Explorer and use the Stackdriver Logging API v2 > logging.entries.list endpoint. You should get relevant information by entering the following details:
fields: entries/protoPayload
Request body: filter: resource.type="gce_instance_template" AND protoPayload.methodName="v1.compute.instanceTemplates.insert" AND protoPayload.resourceName:"/global/instanceTemplates/aef-"
projectIds: <YOUR_PROJECT_IDS>
You'll see the required information under "machineType".
Some functions in the Google Developers Console, like the Analytics API, are free until you reach a quota. Other functions, like Google Cloud Storage, create costs from the first click.
When I upload a file under https://console.developers.google.com/ > Storage > Cloud Storage > Storage Browser and I make this file publicly available, I pay about $0.12 per GB traffic.
But theoretically the traffic to this link could explode, e.g. because of sudden popularity. Therefore I would like to set something like a daily or monthly cost limit.
Q: How do I protect myself from overly high costs in the Google Developers Console?
You cannot. I asked Google about this, here's their response, from May 7 2016:
(GCE = Google cloud engine. No spending limits.
GAE = Google app engine — yes it has spending limits.)
... you are eligible for support on ... only ...
... [various helpful links] ...
That been said, at the moment there is no a feature that allows you to
configure a limited budget on GCE. This feature is certainly available
for GAE [1]. As you mentioned in your comments, you either can totally
shut down your VMs (will depend on your use case) or set the VMs to
send you alerts if they reach a certain traffic limit [2].
Sincerely,
Someone's first name
Technical Solutions Representative
Google Cloud Platform
[1] https://cloud.google.com/appengine/docs/quotas
[2] https://cloud.google.com/monitoring/support/notification-options
#wmdry, you wrote: "traffic to this link could explode" — I'm afraid of this too. That's why I asked Google about this. And I'm planning to avoid Google's CDN because of this, and use another CDN provider instead, which has spending limits. Because, unlike Nginx, I don't see any way for me to rate limit / throttle Google's CDN.
I do plan to use GCE (Google Cloud Engine) though. Therefore, right now I'm reading about how to rate limit my Nginx server. Because if I just configure Nginx correctly, then those $0.12 / GB you mentioned, cannot possible explode to ... like $10k in a month? What if Google sends a $10k bill when I'm back from an a few week's vacation, just because of my hobby project and a few people downloading a 1 MB movie over and over again forever (because: evil). Hmm, & the bigger & faster my servers, the higher the risk.
I hope Google will add spending limits, because I did want to use Google's CDN.
Update 2020: Apparently this does bite people from time to time — look here:
"Burnt $72k testing Firebase and Cloud Run and almost went bankrupt", Dec 08, 2020, https://news.ycombinator.com/item?id=25372336,
In that case, they could contact Google and in the end didn't need to pay.
As of July 2017 you can set budgets that send notifications via email but do not cap spending:
To set an alert-only budget, which will not cap spending:
Go to the Cloud Platform Console.
Open the console left side menu and click Billing
If you have more than one billing account, click the billing account name.
On the left, click Budgets & alerts.
Official help page: https://support.google.com/cloud/answer/6293540?hl=en
I found that Google's documentation now provides two methods to actually limit the cost of a GCP project. It involves the following setup:
Create a Cloud Function that checks the cost against the budget, and carries out a certain action if the cost exceeds the budget. Google's Documentation provides a sample code snip that can either shutdown all VM instances in a Project or disable the billing for a project. Shutting down all VMs would stop all VM-related cost but you get to keep your data (and still have to pay for the storage). Disabling the billing for a project would effectively zap all cost-related activities and you could lose data. You can name the Cloud Function "budget-enforcer".
The Google code snip as provided above has a hard coded ZONE variable. Remember to change it to match your zone!
Create a Service Account to run the Cloud Function "budget-enforcer". For shutting down VMs, the Service Account would need role "Compute Instance Admin (v1)". For disabling billing on a project, the Service Account would need role "Project Billing Manager".
Set a Topic for the Cloud Function (I call mine "proj-name-stop-vm" and "proj-name-disable-bill").
Set up a budget alert as usual, and connect it to one of the Pub/Sub topic above.
Please be noted that Google's documentation did mention that there could be a delay between the cost exceeds a budget and the function is triggered, so you should build in a buffer if you have an absolute hard cost limit. I use 90% of the budget as the trigger line for shutting down my instances.
The API usage can be limited with a hard limit:
Depending on the API, you can explicitly cap requests in a variety of
ways, including: requests per day, requests per 100 seconds, and
requests per 100 seconds per user. You might want to limit the
billable usage by setting caps. For example, to prevent getting billed
for usage beyond the free courtesy usage limits, you can set requests
per day caps
Source
You can combine budget pub/sub alerts with a cloud function that can disable billing on your entire account if a threshold is met.
Full Tutorial Here:
https://www.youtube.com/watch?v=KiTg8RPpGG4
GitHub Repo Here: https://github.com/aioverlords/Google-Cloud-Platform-Killswitch
To Disable Billing
const _disableBillingForProject = async projectName => {
const res = await billing.updateBillingInfo({
name: projectName,
resource: {
billingAccountName: ''
}, // Disable billing
});
console.log(res);
console.log("Billing Disabled");
return `Billing disabled: ${JSON.stringify(res.data)}`;
};
Simply go to the developer console:
https://console.developers.google.com/project
Select your project.
Select "billings & settings"
Enable billing.
Then go to Compute/AppEngine/Settings and set a daily budget.
Go to Google Cloud console, and then to Billing / Budgets and Alerts and create a new budget for one or all your projects. You can select which services should be included in the limit and set a monthly amount that should not be exceeded.
I want to put a scraping service using Apache HttpClient to the Cloud. I read problems are possible with Google App Engine, as it's direct network access and threads creation are prohibited. What's about other cloud hosting providers? Have anyone experince with Apache HttpClient + cloud?
AppEngine has threads and direct network access (HTTP only). There is a workaround to make it work with HttpClient.
Also, if you plan to use many parse tasks in parallel, you might check out Task Queue or even mapreduce.
Btw, there is a "misfeature" in GAE that you can not fully set custom User-agent header on your requests - GAE always adds "AppEngine" to the end of it (this breaks requests to certain sites - most notably iTunes).
It's certainly possible to create threads and access other websites from CloudFoundry, you're just time limited for each process. For example, if you take a look at http://rack-scrape.cloudfoundry.com/, it's a simple rack application that inspects the 'a' tags from Google.com;
require 'rubygems'
require 'open-uri'
require 'hpricot'
run Proc.new { |env|
doc = Hpricot(open("http://www.google.com"))
anchors = (doc/"a")
[200, {"Content-Type" => "text/html"}, [anchors.inspect]]
}
As for Apache HttpClient, I have no experience of this but I understand it isn't maintained any more.
i tried to use the custom search api ( http://code.google.com/intl/de-DE/apis/websearch/docs ) with java. it works perfectly on eclipse on my local machine.
when i try to do the same from google app engine the reply is: {"responseData": null, "responseDetails": "Quota Exceeded. Please see http://code.google.com/apis/websearch", "responseStatus": 403}
i do not understand. isn't it possible to call search api from GAE apps?
If you look at the very top of that page you linked to, they note that the API has been deprecated and the number of search queries you can make is limited.
However, if you absolutely NEED to use that API instead of the Custom Search API as Google suggests, there are a few troubleshooting steps you can take:
1) Check that your API key is unique to the project, and the limited number of queries you're allowed isn't being consumed by some other application.
2) Google does (did?) hostname filtering so that one computer doesn't use up all the API requests. You may be able to move the queries to Javascript instead of Java -- essentially move the request from the server to the client.
3) Try using a named backend (Java Backends)