Some time ago I had Composite C1 installed on a public url to test it (for example http://c1.mydomain.com). But I did remove it already some time ago.
I checked my firewall logs recently and I discovered requests for http://c1.mydomain.com/Composite/top.aspx every single night from IP address 109.238.52.32. (Composite.net's ip address is 109.238.52.24, which is almost the same, so I assume the requests are comming from Composite.net.)
So the question is: Why is Composite.net requesting my admin page every single day?
What you are seeing is a process that check long-term usage of the software in order to generate statistics. The daily URL request looks at the HTTP status code (HTTP 200 vs other) to determine if your website is still online and using the CMS. Since the URL is unique and likely not used by any other system it is a good indication.
When that is said it is probably silly to keep on requesting the same URL once the check starts reporting 404 or similar.
Related
I am trying to use google search engine via python script. It was working fine at the beginning but later I have got "HTTP Error 403: Forbidden". I thing it is due to 100 requests per day limitation. But I have payed account and billing support told me that everyting is ok. Is it possible somehow to solve this problesm?
Thanks, Ilia
There are lot of reasons for which API will throw Error 403. You can find them all here : GOOGLE DOC FOR 403 Forbidden
Your 403 error might indicate that the API key you are using may have lost it's permissions, or that the quota of your API requests may have been exceeded and the new quota (The one you have when you already have a billing account) might not have been updated.
I assume you are using this API; If not, don't worry, the following advice might also help you.
You first need to see if your API key is still working or you might want to generate new ones. You can do this here. Your code should make reference to the new API Key file.
If it is not the case then you should wait a little bit until your Billing account is updated and the quota gets readjusted. You can read a bit more about how to monitor the requests to your APIs here.
You can always find more help with your particular issue if you describe a little the symptoms and your attempts or hypothesis to solve it.
At last you may find more info on the community page of the API.
My instance has little to no traffic but I have a min-idle instance set to 1. What I notice is that whenever there is a random url (via some bot) that doesn't exist is accessed, it is considered a dynamic request since my catch all handler is auto. This is fine, except I see these 404 errors (404 because there are no http handlers associated with these url patterns even though the yaml defines a catch all pattern) resulting in instance restarts. Why should the instance restart if it runs into 404 errors?
I have all my dynamic handlers follow "/api" pattern and then a few that don't. So, I can explicitly list all valid patterns and map them to the auto handler. Would that then consider these random links as static but not present and throw 404 error (which I am fine with)? I want to make sure the instance doesn't keep running just because of some rouge requests.
I just did a local experiment (I don't presently have any quickly deployable play app) and it looks like your quite interesting idea could work.
I replaced the .* pattern previously catching all stragglers and routing them to my default service script (I'm using the python runtime) with specific patterns, then added this handler after all others:
- url: /(.*)$
static_files: images/\1
upload: images/.*
My images directory is real, holding static images (but for which I already have another handler with a more specific pattern).
With this in place I made a request to /crap and got, as expected (there is no images/crap file):
INFO 2019-11-08 03:06:02,463 module.py:861] default: "GET /crap
HTTP/1.1" 404 -
I added logging calls in my script handler's get() and dispatch() calls to confirm they're not actually getting invoked (the development server request logging casts a bit of doubt).
I also checked on an already deployed GAE app that requesting an image that matches a static handler pattern but which doesn't actually exist gets the 404 answer without causing a service's instance to be started (no instance was running at the time), i.e. it comes directly from the GAE's static content CDN.
So I think it's well worth a try with the go runtime, this could save some significant instance time for an app without a lot of activity faced with random bot traffic.
As for the instance restarts, I suspect what you see is just a symptom of your min-idle instance set to 1. Unlike a dynamic instance the idle (aka resident) instance is not normally meant to handle traffic, it's just ready to do it if/when needed. Only when there is no dynamic instance running (and able to handle incoming traffic efficiently) and a new request comes in that request is immediately routed to the idle instance. At that moment:
the idle instance becomes a dynamic one and will continue to serve traffic until it shuts due to inactivity or dies
a fresh idle instance is started to meet the min-idle configuration, it will remain idle until another similar event occurs
Note: your idea will help with the instance hours portion used by the dynamic instances, but not with the idle instance portion.
According to the documentation which quotes the following:
"When an instance responds to the request /_ah/startwith an HTTP status code of 200–299 or 404, it is considered to have started correctly and that it can handle additional requests. Otherwise, App Engine cancels the instance. Instances with manual scale adjustment restart immediately, while instances with basic scale adjustment restart only when necessary to deliver traffic."
You can find more detail about how instances are managed for Standard App Engine environment for Go 1.12 on the link: https://cloud.google.com/appengine/docs/standard/go112/how-instances-are-managed
As well, I recommend you to read the document "How instances are managed", on which quotes the following:
"Secondary routing
If a request matches the part [YOUR_PROJECT_ID].appspot.comof the host name, but includes the name of a service, version, or instance that does not exist, the service is routed default. Secondary routing does not apply to custom domains; requests sent to these domains will show an HTTP status code 404if the hostname is not valid."
https://cloud.google.com/appengine/docs/standard/go112/how-instances-are-managed
I am using watson assistant with flask framework. Once the session expires i get the following error
ApiException(response.status_code, error_message, http_response=response)
ibm_cloud_sdk_core.api_exception.ApiException: Error: NotFound: session id ba62bf14-87e-4c7b071b591e for agent instance 0e5c653c-9-b4e0-5a597c3e541e, Code: 404 , X-global-transaction-id: ffea409d54977b49
127.0.0.1 - - [03/Jul/2019 11:27:47] "GET / HTTP/1.1" 200
Refreshing the page does not create a new session
How to create a new session on this exception, so that I need not restart the local server ?
You mention a 5 minute session timeout.
Watson Plus trial has this set for just testing, and is not intended to be production level. It cannot be changed.
Standard version and lite can use session but both were originally stateless. You should use the V1 API for these. You will have to manage the state, but you will no longer have a time out.
If you are using the paid version of Watson Plus, or premium you can change the time out within the assistant settings.
You need to provide code, as any response will be blind and based on guesswork on what errors you have made, and there are many. But based on the fact that you failed to provide code, and still raised the question expecting someone to tell you what you did wrong then let's go with a rookie error.
I think you are creating the session as a global in your flask app, and maybe saving it it the app object. You probably have a route for the dialog, which makes use of the session. The code in this route, needs to catch the error, or test if the session is still valid, and recreate it. If you are catching the error, and trying to recreate the session, then you may have fallen foul of Python's global variable restrictions. Essentially if in a function you have a variable as the right side of an assignment, then its scope will be limited to the function.
Refreshing the web page, will not force the flask app to re-create the session. As you probably have that logic outside of any routes.
I noticed this:
(Note that I deployed this service just 2 days ago and no one know it!!)
The only valid request here is the one to "/extract" at 4:10, because it is correct and it is at the scheduled time (of the cron job).
(You can also notice that the user agent is "AppEngine")
I created some routing for the "hacking try" paths to reply with a "F*** Y**".
Then I decided to use a dos.yaml file to define a blacklist.
I was taking note of the IPs and I noticed that the IP of the valid request is also used in this request (and probably other "malicious" ones):
I noticed that every execution of the my scheduled "cron job" (automated one or using "Run now") has a different IP. Also all the malicious requests have different IPs but you can see that some of them are executed programmatically:
How is it possible?
My explanation is that Google Cloud Platform is hosting a lot of malicious services.
Also, the documentation says that cron jobs are issued by the IP 0.1.0.1.
Someone can explain me this unexpected behavior or where is the fault in my argument?
Alex
I have a fairly big application which went over a major overhaul.
The newer version uses lot of JSONP calls and I notice 500 server errors. Nothing is logged in the logs section to determine the error cause. It happens on JS, png and even jersey (servlets) too.
Searching SO and groups suggested that these errors are common during deployment. But it happens even after hours after deployment.
BTW, the application has become slightly bigger and it even causes deadline exception while starting few instances in few rare cases. Sometimes, it starts & serves within 6-10secs. Sometimes it goes to more than 75secs thereby causing a timeout for the similar request. I see the same behavior for warmup requests too. Nothing custom is loaded during app warmup.
I feel like you should be seeing the errors in your logs. Are you exceeding quotas or having deadline errors? Perhaps you have an error in your error handler like your file cannot be found, or the path to the error handler overlaps with another static file route?
To troubleshoot, I would implement custom error pages so you could determine the actual error code. I'm assuming Python since you never specified what language you are using. Add the following to your app.yaml and create static html pages that will give the recipient some idea of what's going on and then report back with your findings:
error_handlers:
- file: default_error.html
- error_code: over_quota
file: over_quota.html
- error_code: dos_api_denial
file: dos_api_denial.html
- error_code: timeout
file: timeout.html
If you already have custom error handlers, can you provide some of your app.yaml so we can help you?
Some 500s are not logged in your application logs. They are failures at the front-end of GAE. If, for some reason, you have a spike in requests and new instances of your application cannot be started fast enough to serve those requests, your client may see 500s even though those 500s do not appear in your application's logs. GAE team is working to provide visibility into those front-end logs.
I just saw this myself... I was researching some logs of visitors who only loaded half of the graphics files on a page. I tried clicking on the same link on a blog that they did to get to our site. In my case, I saw a 500 error in the chrome browser developer console for a js file. Yet when I looked at the GAE logs it said it served the file correctly with a 200 status. That js file loads other images which were not. In my case, it was an https request.
It is really important for us to know our customer experience (obviously). I wanted to let you know that this problem is still occurring. Just having it show up in the logs would be great, even attach a warm-up error to it or something so we know it is an unavoidable artefact of a complex server system (totally understandable). I just need to know if I should be adding instances or something else. This error did not wait for 60 seconds, maybe 5 to 10 seconds. It is like the round trip for SSL handshaking failed in the middle but the logs showed it as success.
So can I increase any timeout for the handshake or is that done on the browser side?