Am researching activity audits for the last couple days using an asp.net MVC project. I was using contentType=Audit.Exchange and contentType=Azure.ActiveDirectory successfully since the last half of yesterday and this morning up until about two hours ago.
I made no changes to my authorization/authentication code and the tokens look good. Also no changes to the calls themselves. I added some json handling for the response to list subscriptions and when I ran the app to test that code, suddenly I am getting an InternalServerError response to start subscription, list subscriptions and stop subscription. The error is returned after a long timeout (in fact I had to increase the default timeout value).
So as of about two hours ago all the APIs are returning InternalServerError after a long timeout. This is happening on the following APIs:
/activity/feed/subscriptions/start
/activity/feed/subscriptions/list
/activity/feed/subscriptions/stop
The body of the response message is empty. So does not include any error info as described in https://msdn.microsoft.com/office-365/office-365-management-activity-api-reference.
Seems crazy this could be a service outage, so I must be missing something really elemental?
Hmmm. With no further changes to the code, now am getting HTTP 200 responses. If that was a service outage, that was a heck of a long outage for 99.9% uptime.
Related
We're working on a client-server application, where the client is Angular + AngularJS hybrid, running on Chrome. The server is a spring MVC running on Tomcat.
The communication between the client and server is HTTPS.
Our client sends different REST requests to the server using the AngularJS $http service. Some of them repeat on predefined intervals.
Over the past week we started noticing that every now and then some of the repeating requests fail with ERR_EMPTY_RESPONSE (status is "failed"). For the same request it can fail and 20 seconds later (when the interval is reached) succeed.
It seems like this could potentially happen with every REST request but we mainly notice it on the repeating ones.
The failing requests seem to not reach the server at all as they don't appear in the localhost_access_log.txt file and there is a gap between the repeating requests that did reach the server that indicate the failing requests never reached the server.
The application has been working for quite some time now and without any issues regarding $http requests until last week (around March 25th 2019).
The code that sends these REST requests is not new and has not been changed in years.
It also doesn't look related to the latest Chrome update, as the issue reproduces with Chrome 63, and with the newest Chrome version (73).
We would appreciate your help with this issue.
Thanks.
I am seeing an odd intermittent authorization failure on publish. My publisher is running on App Engine Standard (Python). Because of that, I am using the "old" python client library. So the code looks like this:
from googleapiclient.discovery import build
build('pubsub','v1').projects().topics().publish(topic=topic,body=body).execute()
This works just fine. The identity gets picked up and everything is authenticated. However, again intermittently, it will stop working and I get 403 forbidden errors. Then later it will start working again (even with the same topic and body). In the meantime, no code changes, no deployments.
I have had to wrap the publish to catch this error, throw it on a task queue and have the request repeat with decaying frequency until it finally starts working again a few hours later. This is OK in the very short term, but obviously this will not work for us.
To summarize, this is on the publish side, GAE Standard ... it works, then stops working, then works again.
Thanks for any insight or help.
It turns out, of course, that in fact there were deployments when I wasn't aware. So I thought, "no code change - no deployments", but there were deployments. And the issue was that the person making these deployments had an old library (or other dependency) for google_api_python_client. Once corrected, pubsub is working just fine.
I have a small service discovery service running on Google App Engine free tier. It queries Google Cloud Datastore with simplest queries on the data that is virtually static. Recently we had incidents that the service was returning empty results. That was going on for 12 hours. After that results came back to normal. We only recently noticed it. In the logs I see at least 3 incidents like that.
I logged into a console and saw:
0 app errors
0 server errors
all green GCE status
max used quota is just 5%
intervals of time where every response is 204
I see absolutely no reason to receive an empty response and yet they are sometimes empty. I see no way to notify Google that there is cloud-side problems, since this is a free tier account with no support.
So, is there anything I might have missed?
UPD: As I look into the code, the only way to get 204+empty body is to have exception in getting an instance of javax.jdo.PersistenceManager or a new instance of javax.jdo.Query. So it is less likely that Cloud Datastore is at fault, because if the result would be empty, app should answer 200+[], if there was an error during a query, app should answer 204+<h1>Exception</h1>....
But again, I don't see how request could work now, but not work 5 minutes ago.
UPD2: The app was stable for more than two years.
It was working fine till last day and suddenly stopped pushing to endpoint. Checked all settings including endpoint URL and found everything remains unchanged. Can you guys suggest possible causes.
Not receiving a message on a push endpoint could happen for many reasons. The first thing to do would be to go to Stackdriver and create a graph for the subscription/push_request_count metric. You can break this down by response_code to see how many requests Cloud Pub/Sub is sending to your push endpoint and what response codes it is returning. If there are requests being delivered that are returning errors, this graph will show that.
It might also be worth checking the publish side to ensure messages are still being published as expected. You can look at the topic/send_message_operation_count metric, which can also be broken down by response_code, to make sure the publish requests are all returning success.
You should also check to ensure the subscription still exists using the Pub/Sub Subscriptions page in the Cloud console. After 30 days of inactivity (including inability to successfully deliver a message to a push endpoint), subscriptions are potentially deleted.
If the issue still unsolved after those steps, it is best to contact Google Cloud support with your project ID and subscription name so that things can be investigated for your specific case.
Everytime I ping amazon to get _GET_FBA_MYI_UNSUPPRESSED_INVENTORY_DATA_ I can make the request and it SUBMITTED for about 10 seconds then when I ping again it says cancel I have tried making the request 3 times and all of the times it seems to get cancelled.
You are querying too much. For that particular report I request once an hour (the most you can) and if you get a cancel message have it wait an hour and then query again. It is initially hard to debug so I would keep around the exports to test your other portions of your code while keep the report download code separate.