I am using fetch in a NodeJS application. Technically, I have a ReactJS front-end calling the NodeJS backend (as a proxy), and then the proxy calls out to backend services on a different domain.
However, from logging errors from consumers (I haven't been able to reproduce this issue myself) I see that a lot of these proxy calls (using fetch) throw an error that just says Network Request Failed, which is of no help. Some context:
This only occurs on a subset of all total calls (lets say 5% of traffic)
Users that encounter this error can often make the same call again some time later (next couple minutes/hours/days) and it will go through
From Application Insights, I can see no correlation between browsers, locations, etc
Calls often return fast, like < 100 ms
All calls are HTTPS, non are HTTP
We have a fetch polyfill from fetch-ponyfill that will take over if fetch is not available (Internet Explorer). I did test this package itself and the calls went through fine. I also mentioned that this error does occur on browsers that do support fetch, so I don't think this is the error.
Fetch settings for all requests
Method is set per request, but I've seen it fail on different types (GET, POST, etc)
Mode is set to 'same-origin'. I thought this was odd, since we were sending a request from one domain to another, but I tried to set it differently and it didn't affect anything. Also, why would some requests work for some, but not for others?
Body is set per request, based on the data being sent.
Headers is usually just Accept and Content-Type, both set to JSON.
I have tried researching this topic before, but most posts I found referenced React native applications running on iOS, where you have to set some security permissions in the plist file to allow HTTP requests or something to do with transport security.
I have implement logging specific points for the data in Application Insights, and I can see that fetch() was called, but then() was never reached; it went straight to the .catch(). So it's not even reaching code that parses the request, because apparently no request came back (we then parse the JSON response and call other functions, but like I said, it doesn't even reach this point).
Which is also odd, since the request never comes back, but it fails (often) within 100 ms.
My suspicions:
Some consumers have some sort of add-on for there browser that is messing with the request. Although, I run with uBlock Origin and HTTPS Everywhere and I have not seen this error. I'm not sure what else could be modifying requests that would cause it to immediately fail.
The call goes through, which then reaches an Azure Application Gateway, which might fail for some reason (too many connected clients, not enough ports, etc) and returns a response that immediately fails the fetch call without running the .then() on the response.
For #2, I remember I had traced a network call that failed and returned Network Request Failed: Made it through the proxy -> made it through the Application Gateway -> hit the backend services -> backend services sent a response. I am currently requesting access to backend service logs in order to verify this on some more recent calls (last time I did this, I did it through a screenshare with a backend developer), and hopefully clear up the path back to the client (the ReactJS application). I do remember though that it made it to the backend services successfully.
So I'm honestly not sure what's going on here. Does anyone have any insight?
Based on your excellent description and detective work, it's clear that the problem is between your Node app and the other domain. The other domain is throwing an error and your proxy has no choice but to say that there's an error on the server. That's why it's always throwing a 500-series error, the Network Request Failed error that you're seeing.
It's an intermittent problem, so the error is inconsistent. It's a waste of your time to continue to look at the browser because the problem will have been created beyond that, either in your proxy translating that request or on the remote server. You have to find that error.
Here's what I'd do...
Implement brute-force logging in your Node app. You can use Bunyan, or Winston or just require(fs) and write out to some file when an error occurs. Then look at the results. Only log it out when the response code from the other server is in the 400 or 500 ranges. Log the request object and the response object.
Something like this with Bunyan:
fetch(urlToRemoteServer)
.then(res => res.json())
.then(res => whateverElseYoureDoing(res))
.catch(err => {
// Get the request & response to the remote server
log.info({request: req, response: res, err: err});
});
where the res in this case is the response we just got from the other domain and req is our request to them.
The logs on your Azure server will then have the entire request and response. From this you can find commonalities. and (🤞) the cause of the problem.
Related
I added additional API to the Duende IdentityServer 6.2 as described here. Then I tried to access it from a sample App, using typed httpClient using their own library called AccessTokenManagement (aka Identity.Model) pretty much following their simple example. I use Authorization Code flow, everything pretty much simple and default.
It works well until both server and client are on the same dev machine under localhost. As soon as I publish IdentityServer to IIS, the API stops to work, while the rest still works well (I can be authenticated, and I see in the Fiddler that token exchanges work normally).
The call to API consists from two calls:
Calling to /connect/token using refresh token. Server returns access token.
Calling my endpoint using this new access token.
The flow fails on the step 1. Call to /connect/token is already unauthorized and I can't understand why. The "good" and "bad" calls looks the same, I cannot see any differences. Previous call moment ago to /connect/userinfo consists of the same two steps and it works. Logs on both server and client give no clues.
No reverse proxies, just good plain simple URI. Automatic key management is enabled and the keys are in the SQL table, common for dev and published server. Asp.Net Core Data Protection is enabled and keys are also common.
Relevant parts of logs are below. I noticed that "No endpoint entry found for request path" is specific to IdentityServer and it doesn't actually mean that endpoint was not found. It was found but not processed. I also noticed reacher response headers from bad request and log entry about "Cookie signed-in" in good request but not sure what does it mean and whether it's relevant.
I'm running out of ideas.
Bad response from IIS while trying to get new Access Token:
Proper response while developing:
///////Relevant part of log for BAD request
|Duende.AccessTokenManagement.OpenIdConnect.UserAccessAccessTokenManagementService|Token for user test#test.com needs refreshing.
|Microsoft.AspNetCore.Authentication.Cookies.CookieAuthenticationHandler|AuthenticationScheme: cookie was successfully authenticated.
|Duende.AccessTokenManagement.OpenIdConnect.UserTokenEndpointService|refresh token request to: https://auth.mysite.org/connect/token
|Duende.AccessTokenManagement.OpenIdConnect.UserAccessAccessTokenManagementService|Error refreshing access token. Error = Unauthorized
|System.Net.Http.HttpClient.IdsService.ClientHandler|Sending HTTP request POST https://auth.mysite.org/mycontroller/myaction
|System.Net.Http.HttpClient.IdsService.ClientHandler|Received HTTP response headers after 117.7278ms - 401
///////Same part of GOOD request
|Duende.AccessTokenManagement.OpenIdConnect.UserAccessAccessTokenManagementService|Token for user test#test.com needs refreshing.
|Microsoft.AspNetCore.Authentication.Cookies.CookieAuthenticationHandler|AuthenticationScheme: Cookies was successfully authenticated.
|Duende.AccessTokenManagement.OpenIdConnect.UserTokenEndpointService|refresh token request to: https://localhost:5001/connect/token
|Microsoft.AspNetCore.Authentication.Cookies.CookieAuthenticationHandler|AuthenticationScheme: Cookies signed in.
|System.Net.Http.HttpClient.IdsService.ClientHandler|Sending HTTP request POST https://localhost:5001/mycontroller/myaction
|System.Net.Http.HttpClient.IdsService.ClientHandler|Received HTTP response headers after 1994.9611ms - 200
///////Server log during BAD request
Duende.IdentityServer.Hosting.EndpointRouter No endpoint entry found for request path: "/mycontroller/myaction"
Duende.IdentityServer.Hosting.LocalApiAuthentication.LocalApiAuthenticationHandler HandleAuthenticateAsync called
Duende.IdentityServer.Hosting.LocalApiAuthentication.LocalApiAuthenticationHandler AuthenticationScheme: "IdentityServerAccessToken" was not authenticated.
Duende.IdentityServer.Hosting.LocalApiAuthentication.LocalApiAuthenticationHandler AuthenticationScheme: "IdentityServerAccessToken" was challenged.
Okay, found it. Thankfully, looked at Fiddler's WebView and had seen familiar picture!
Then, found this topic. The solution was disabling Basic authentication in IIS settings. Access token request has basic authentication header and it seems like IIS intercepts it. Still a bit unclear why other parts of flow worked.
I am using mule-4. I am trying to integrate a third-party API(confidential). It is working from the postman and is returning responses within 1 second.
When I wrote a request connector for the same in mule, The API kept giving a timeout exception.
I increased the response timeout to 2 minutes then also got the same error i.e. timeout exceeded.
Please help.
EDIT 1:
I was able to reproduce this issue on postman. SO postman is adding Connection:keep-alive header by default and when this particular header is added then the API gives response within seconds but when this header is missing then the API gives a timeout error.
You are not really providing too much details of the issue. I can give some generic guidelines:
Ensure that there is network connectivity. If you are testing Postman and Mule from the same computer it is probably not an issue.
Ensure host, port and certificates (if using TLS) are the same. Pointing an HTTPS request to port 80 could cause a timeout sometimes.
Enable HTTP Wire logging in Mule and compare with Postman code as HTTP to identify any significant difference between the requests. For example: headers, URI, body should be the same, except probably for a few headers. Depends on the API. This is usually the main cause for differences between Postman and Mule, when the request are different.
I'm trying to use ITfoxtec.Identity.Saml2.MvcCore on a .NET Core 3.1 web application using an in-house IdP.
It works great on our test server (Windows Server 2012, hosted in the IIS) but I can't get it to work on any other server.
This is what happens:
The initial call to the website is correctly identified as a non authenticated call and the user is being sent to the IdP where the user logs in as usual. The SAML-token is then posted back to the web applications assertion consumer service where everything seems like it does what its supposed to, saml2AuthnResponse.Status has statuscode Saml2StatusCodes.Success and the logfile says "AuthenticationScheme: saml2 signed in". Then it reads the ReturnUrl-parameter and log something like "Executing RedirectResult" but then it just stops. Nothing in the logfile, nothing in the IIS-logs. The user is met by the message
This site can’t be reached
...
ERR_HTTP2_PROTOCOL_ERROR
In short, every controller that has the [Authorize]-attribute gives the ERR_HTTP2_PROTOCOL_ERROR-error. When I remove all [Authorize]-attributes the application works great, although without authentication.
I've also tried the example TestWebAppCore-application from ITfoxtec.Identity.Saml2's github-page and it gives the same error. It works on our 2012 test-server but nowhere else.
Any ideas that I can try?
I think you need to trace the calls to see the actual http request and responses send between the browser and server. I usually use Fiddler for tracing the requests/response. Remember to enable Fiddler for https tracing.
My first thought is that the problem can have something to do with cookies. But it is only a guess...
You might be on to something, we disabled http/2 on the server and was greeted instead by this message:
Bad Request - Request Too Long
HTTP Error 400. The size of the request headers is too long.
It uses 5 cookie-chunks for the SAML-data for a total of 19941 bytes which is a bit to much. I've tried to make the application save the sessiondata in classic session objects instead but I cant seem to get it to work.
This is what I added to StartUp.cs:
In ConfigureService:
services.AddMvc()
.AddSessionStateTempDataProvider();
services.AddSession(options =>
options.Cookie.IsEssential = true
);
services.Configure<CookiePolicyOptions>(options =>
{
options.CheckConsentNeeded = context => false;
options.MinimumSameSitePolicy = SameSiteMode.None;
});
In Configure:
app.UseSession();
But it still fills up the header with cookies. What am I doing wrong? Is there a another way to make the session cookies smaller?
I'm using Etags in the response headers for dynamic resources, and my client is sending future requests for the same resource with the If-None-Match header.
The client and server are interacting perfectly, and I get 304 responses if the data was not modified. I can see this on the nginx logs on my server, and the browser network tab on Firefox. However, when i log the exact same response from the whatwg-fetch response status, I'm getting 200.
Although react doesn't re-render if there is not diff in the data, all the processing till that point (dispatching actions, running custom data manipulation etc) are being executed, which is consuming too much memory.
I'm using react-boilerplate, which is internally whatwg-fetch, and too far out to make a U-turn and start using xhrHttpRequests, which apparently don't have this problem from the scant resources I found in other parts of the web.
Please help; I can attach logs and notes if need be. Thanks :)
I'm using Ionic platform for my mobile application. Using angular
$http for sending requests to server.
Intermittently when Mobile app tries to access server $http goes to it's errorCallback with response status -1 only no other
data.
When I check log on server, not able to see any hit.
I've changed timeout of application to 2 minutes using interceptors.
I have used chrome debugger but it won't show anything apart from
request it forms, shows nothing in response and preview columns.
I got that in Ionic we use pre-flight to check if server is alive
before sending actual request. But it's for CORS; we have enabled
CORS on server and thats why app is working good since last 15 days.
Thought of using network packet tracer tool but if call not logged on
server no use of it. as Status -1 says $http aborted the request.
My Question is why it's aborting when I click once and do send
when I click same button again.
Please me help to figure out an issue.
After lots of debugging and surfing over internet for issue.
I guess that an issue was like mobile app sending pre-flight messages and so $http aborting the request and even some time Server played a culprit here how will tell you;
We have server hosted on AWS in where we had Load balancer in different zone and actual API server is in different zone. After changing them to same zone ask, production people to test now they are not getting this issue.
The another reason was we were using unstable mobile networks to test.
If any one have any thing else on this please let me know.