I have an API on AWS API Gateway that takes a POST request and forwards headers and body to a third party API and returns the result. This process works fine when I send the request from Postman, but doesn't work either via cURL or JavaScript.
NB - all URIs, auth tokens etc below are modified so may be inconsistent between screengrabs.
The request in Postman is below
Postman console for this looks like
POST https://dsvdvsdvsdrc.execute-api.eu-west-1.amazonaws.com/Prod/
200
957 ms
Network
Request Headers
Authorization: Basic <myauthtoken>NTAwYTQxNDdmYzcyLWFkZDgtNDZmMy05ZWU0LWQzYWM=
Content-Type: application/x-www-form-urlencoded
User-Agent: PostmanRuntime/7.26.8
Accept: */*
Cache-Control: no-cache
Postman-Token: 9dab6f01-67bf-4611-8d8e-c3d5fe725067
Host: tsfsfsdrc.execute-api.eu-west-1.amazonaws.com
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 82
Request Body
grant_type: "client_credentials"
scope: "https://api.ebay.com/oauth/api_scope"
In my JavaScript app I have the following code:
var data = qs.stringify({
'grant_type': 'client_credentials',
'scope': 'https://api.ebay.com/oauth/api_scope'
});
var config = {
method: 'post',
url: 'https://fddgdgddrc.execute-api.eu-west-1.amazonaws.com/Prod/',
headers: {
'Authorization': 'Basic sssscyLWFkZDgtNDZmMy05ZWU0LWQzYWM=',
'Content-Type': 'application/x-www-form-urlencoded'
},
data : data
};
console.log("******Here is ebayData REQUEST***** "+ JSON.stringify(config));
axios(config)
.then(function (response) {
console.log("******Here is ebayData***** "+ JSON.stringify(response.data));
})
.catch(function (error) {
console.log( "******Here is ebay Error***** "+ error);
});
However when the application runs I get a 500 response. Below are the request headers and body that is being sent in the request
I've enabled cloudwatch logs on the API and below is an example of a successful request via Postman
and here is an example of an unsuccessful request from the browser
Looking further into the response headers for a failed and a successful response I see the headers with comments against them are different
Failed request
(d360923b-eff2-433f-8f76-a9038547dcdf) Endpoint response headers: {rlogid=t6ldssk67%3D9whhldssk67*qc1qr%28rbpv6710-17dd35648ce-0x129,
x-ebay-c-version=1.0.0,
x-frame-options=SAMEORIGIN,
x-content-type-options=nosniff,
x-xss-protection=1; mode=block,
set-cookie=ebay=%5Esbf%3D%23%5E;Domain=.ebay.com;Path=/; Secure,dp1=bu1p/QEBfX0BAX19AQA**6581b87b^;Domain=.ebay.com;Expires=Tue, 19-Dec-2023 15:36:27 GMT;Path=/; Secure,
content-encoding=gzip,
cache-control=private, <--- doesn't appear in successful response
pragma=no-cache, <--- doesn't appear in successful response
date=Sun, 19 Dec 2021 15:36:26 GMT,
server=ebay-proxy-server,
x-envoy-upstream-service-time=19,
x-ebay-pop-id=UFES2-RNOAZ03-api,
transfer-encoding=chunked}
Successful request
(fe565553-3283-4593-8b07-b4e2d58dd2a6) Endpoint response headers: {rlogid=t6ldssk67%3D9vjdldssk67*5cddm%28rbpv6775-17dd23fa53c-0x124,
x-ebay-c-version=1.0.0,
x-ebay-client-tls-version=TLSv1.2,<--- doesn't appear in failed response
x-frame-options=SAMEORIGIN,
x-content-type-options=nosniff,
x-xss-protection=1; mode=block,
set-cookie=ebay=%5Esbf%3D%23%5E;Domain=.ebay.com;Path=/; Secure,dp1=bu1p/QEBfX0BAX19AQA**65817126^;Domain=.ebay.com;Expires=Tue, 19-Dec-2023 10:32:06 GMT;Path=/; Secure,
content-encoding=gzip,
content-type=application/json,<--- doesn't appear in failed response
date=Sun, 19 Dec 2021 10:32:06 GMT,
server=ebay-proxy-server,
x-envoy-upstream-service-time=96,
x-ebay-pop-id=UFES2-SLCAZ01-api,
transfer-encoding=chunked}
I think I've been looking at this for too long and am probably missing something obvious, but headers and body etc all seem to be consistent across the app and Postman, so I'm confused why the request from one is successful and the other is failing. Any advice much appreciated.
Add all other headers as in Postman , some application rejects requests without proper user-agent header or some other required headers.
Related
soo i have the following fetch api
const loginUser = async (e) => {
e.preventDefault()
const URL = "http://localhost:8000/auth/login/"
let response = await fetch(URL, {
// credentials: "include",
method: "POST",
headers: {
"Content-Type": "application/json",
},
body: JSON.stringify({
username: e.target.username.value,
password: e.target.password.value,
}),
})
if (response.status === 200) {
console.log(response)
} else {
setLoginStatus("Invalid credentials")
loginStatusRef.current.style.visibility = "visible"
}
}
This is the response of the above fetch
Access-Control-Allow-Origin http://localhost:3000
Allow POST, OPTIONS
Content-Length 37
Content-Type application/json
Cross-Origin-Opener-Policy same-origin
Date Thu, 25 Aug 2022 16:44:07 GMT
Referrer-Policy same-origin
Server WSGIServer/0.2 CPython/3.10.5
Set-Cookie
csrftoken=Ou23mE46QwuuJi37bVgsZq19NbEAzaFh; expires=Thu, 24 Aug 2023 16:44:07 GMT; Max-Age=31449600; Path=/; SameSite=Lax
Set-Cookie
sessionid=hyvv9uwwdi0m0blhb3ks6vzgtnyimxa6; expires=Thu, 08 Sep 2022 16:44:07 GMT; HttpOnly; Max-Age=1209600; Path=/; SameSite=Lax
Vary Accept, Cookie, Origin
X-Content-Type-Options nosniff
X-Frame-Options DENY
when i try to enable credentials:'include',i get the following error:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:8000/auth/login/. (Reason: CORS request did not succeed). Status code: (null)
i have already enabled cors in django
CORS_ALLOWED_ORIGINS = ['http://localhost:3000', ]
INSTALLED_APPS = [
. . .
"corsheaders"
]
as you can see, a cookie sessionid is being returned but its not saved on browser. kindly help
Currently you have default behaviour that CORS-preflight requests must never include credentials.
Specify Access-Control-Allow-Credentials:true in response to indicate that the actual request can be made with credentials that should solve it.
The Access-Control-Allow-Credentials header indicates whether or not the response to the request can be exposed when the credentials flag is true. When used as part of a response to a preflight request, this indicates whether or not the actual request can be made using credentials.so if a request is made for a resource with credentials, if this header is not returned with the resource, the response is ignored by the browser and not returned to web content.
Note that simple GET requests are not preflighted.
I'm building a REST API with Flask (using Flask-RESTful and Flask-JWT-Extended) and a front end React application that consumes its data.
To log users in, after getting their credentials through the login endpoint, the server sends back an access JWT and a refresh JWT (both in the form of httpOnly cookies) and a CSRF cookie for each JWT, the idea being that for further requests to any API endpoint the client has to send the access JWT and the CSRF cookies.
As I understand, those cookies should be sent automatically, but in reality they're not being sent. The login works fine (I receive the 4 cookies I expect), but on further requests my React front end is not sending the cookies back, even including the authorization: 'include' and credentials: 'include' options in the request header.
I think it's worth to note that the React server runs on port 3000 and my Flask API on port 5000. I think it may cause a problem, but its just a hunch as I don't know how it might affect the communication between the two.
These are the response headers for the login endpoint (which works fine):
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 29
Set-Cookie: access_token_cookie=<very_long_access_JWT>; HttpOnly; Path=/
Set-Cookie: csrf_access_token=<csrf_access_token>; Path=/
Set-Cookie: refresh_token_cookie= <very_long_refresh_JWT>; HttpOnly; Path=/
Set-Cookie: csrf_refresh_token= <csrf_refresh_token>; Path=/
Access-Control-Allow-Origin: http://localhost:3000
Vary: Origin
Server: Werkzeug/2.0.2 Python/3.10.2
Date: Fri, 18 Mar 2022 21:07:03 GMT
These are the request headers for trying to send a POST request to the test endpoint:
POST /api/test HTTP/1.1
Host: 127.0.0.1:5000
Connection: keep-alive
Content-Length: 0
Pragma: no-cache
Cache-Control: no-cache
sec-ch-ua: "Chromium";v="98", " Not A;Brand";v="99"
authorization: include
DNT: 1
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.136 Safari/537.36
credentials: include
sec-ch-ua-platform: "Linux"
Accept: */*
Origin: http://localhost:3000
Sec-Fetch-Site: cross-site
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: http://localhost:3000/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,es;q=0.8
And this is a "condensed" version of the fetch function that I use on every View on my React application.In reality, the fetchResource() function is saved in its own module and is imported and called by differente Views as needed. I have tested this function before to fetch data without login in and it works.
/**
* Multipurpose fetch function
* #param {string} queryURL URL string to fetch from.
* #param {Object} options request options. Defaults to null
* #returns {Object} returns jsonified data if fetch was successfull, error object if not
*/
const fetchResource = async (queryURL, options = null) => {
try {
const response = await fetch(queryURL, options);
const data = await response.json();
if (!response.ok) {
return { code: response.status, data };
}
return data;
} catch (error) {
return `Looks like there was a problem: ${error}`;
}
};
fetchResource('http://127.0.0.1:5000/api/test', {
method: 'POST',
headers: {
authorization: 'include',
credentials: 'include'
}
});
What could be causing this issue? I'm completely new to cookies and I'm at a loss here.
I am using a React frontend to log into a nodejs server running express-session. Frontend is running on localhost:3000, server is on localhost:5000.
Everything is working properly using postman from localhost (session cookie is sent from server when user is properly authenticated and received/stored by postman. Subsequent postman api request to different path on server uses the session cookie and correctly retrieves the data it should based on the session contents). I can also is login using the browser directly to the server (http://localhost:5000/api/authenticate). The server generates the session, sends the cookie to the browser and it stores the cookie locally.
What doesn't work is when I make the api request from within the React app. The server is returning the session cookie but the browser is not storing it. After researching this for the last few days (there are a lot of questions on this general subject), it seems to be an issue with cross site request but I can't seem to find the right set of app and server settings to get it working properly. The cookie is being sent by the server but the browser won't store it when the request from the app.
*** after some additional troubleshooting and research, I've made some updates. My initial XHR request requires a pre-flight and the request and response headers appear to be correct now but still no cookie being stored in browser. More details below the setup ****
Server Setup
var corsOptions = {
origin: 'http://localhost:3000',
credentials: true
};
app.options('*', cors(corsOptions)) // for pre-flight
app.use(cors(corsOptions));
app.use(session({
genid: (req) => {
console.log('Inside the session middleware');
console.log(req.sessionID);
return uuidv4();
},
store: new FileStore(),
secret: 'abc987',
resave: false,
saveUninitialized: true,
cookie: { httpOnly: false, sameSite: 'Lax', hostOnly: false }
}));
app.use( bodyParser.json() );
app.use(bodyParser.urlencoded({
extended: true
}));
app.use(function(req, res, next) {
res.header('Access-Control-Allow-Headers', 'Origin, X-Requested-With, Content-Type, Accept, withCredentials, credentials');
next();
});
app.post('/api/authenticate', function(req, res) {
const usernameLower = req.body.username.toLowerCase();
const passwordHash = md5(req.body.password);
connection.query('select USERID from USERS where LOWER(USERNAME)=? && PASSWORD=? ', [usernameLower, passwordHash], function (error, results, fields) {
if (error) {
console.log(error);
req.session.destroy();
res.status(500)
.json({
error: 'Internal error please try again'
});
} else if (results[0]) {
const userId = results[0].USERID;
// setup session data
mySession = req.session;
mySession.user = {};
mySession.user.userId = userId;
res.json(mySession.user);
} else {
console.log('auth failed');
req.session.destroy();
res.status(401)
.json({
error: 'Incorrect email or password'
});
}
});
});
Client setup -- the request is triggered by clicking a submit button in a form
handleSubmit(event) {
event.preventDefault();
axios.defaults.withCreditials = true;
axios.defaults.credentials = 'include';
axios({
credentials: 'include',
method: 'post',
url: 'http://localhost:5000/api/authenticate/',
headers: {'Content-Type': 'application/json' },
data: {
username: this.state.username,
password: this.state.password
}
})
.then((response) => {
if (response.status === 200) {
this.props.setLoggedIn(true);
console.log('userId: '+response.data.userId);
} else {
console.log("login error");
}
})
.catch(error => console.log(error))
}
Below is the response cookie sent to the browser but the browser is not storing it.
{"connect.sid":{"path":"/","samesite":"Lax","value":"s:447935ac-fc08-47c6-9b66-4fa30b355021.Yo5H3XVz3Ux3GjTPVhy8i2ZPJm2RM2RzUnznxU9wBvo"}}
Request headers from XHR request (pre-flight):
OPTIONS /api/authenticate/ HTTP/1.1
Host: localhost:5000
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Access-Control-Request-Method: POST
Access-Control-Request-Headers: content-type
Referer: http://localhost:3000/
Origin: http://localhost:3000
DNT: 1
Connection: keep-alive
Pre-flight server response headers
HTTP/1.1 204 No Content
X-Powered-By: Express
Access-Control-Allow-Origin: http://localhost:3000
Vary: Origin, Access-Control-Request-Headers
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: GET,HEAD,PUT,PATCH,POST,DELETE
Access-Control-Allow-Headers: content-type
Content-Length: 0
Date: Fri, 10 Jul 2020 21:35:05 GMT
Connection: keep-alive
POST request header
POST /api/authenticate/ HTTP/1.1
Host: localhost:5000
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/json
Content-Length: 45
Origin: http://localhost:3000
DNT: 1
Connection: keep-alive
Referer: http://localhost:3000/
Server response headers
HTTP/1.1 200 OK
X-Powered-By: Express
Access-Control-Allow-Origin: http://localhost:3000
Vary: Origin
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Content-Type: application/json; charset=utf-8
Content-Length: 95
ETag: W/"5f-Iu5VYnDYPKfn7WPrRi2d2Q168ds"
Set-Cookie: connect.sid=s%3A447935ac-fc08-47c6-9b66-4fa30b355021.Yo5H3XVz3Ux3GjTPVhy8i2ZPJm2RM2RzUnznxU9wBvo; Path=/; SameSite=Lax
Date: Fri, 10 Jul 2020 21:35:05 GMT
Connection: keep-alive
I used the "Will it CORS" tool at https://httptoolkit.tech/will-it-cors/ and my request/response headers all seem to be correct but still no cookie stored.
Pre-flight request contains the correct origin
Pre-flight response contains the correct allow-origin and allow-credentials
POST request contains the correct origin and allow-credentials
POST response contains the correct
Appreciate any help to unravel this....
I solved my issues and wanted to post the solution in case others come across this.
To recap, the backend server is nodejs using express. The following setup allows the front-end to accept the cookies which were created on the nodejs server.
app.use(function (req, res, next) {
res.header("Access-Control-Allow-Origin", "https://frontendserverdomain.com:3000"); // update to match the domain you will make the request from
res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
res.header("Access-Control-Allow-Credentials", true); // allows cookie to be sent
res.header("Access-Control-Allow-Methods", "GET, POST, PUT, HEAD, DELETE"); // you must specify the methods used with credentials. "*" will not work.
next();
});
The front-end app is based on React and uses axios to make http request. It is hosted at "https://frontendserverdomain.com:3000" which is added to the "Access-Control-Allow-Origin" header in the nodejs setup (see above).
On the front-end, Axios needs the "withCreditials" setting applied.
axios.defaults.withCredentials = true;
With these settings, your app will be able to exchange cookies with the back-end server.
One gotcha for me getting CORS working was to make sure the front-end host is properly added to the back-end servers header "Access-Control-Allow-Origin". This includes the port number if it's specified in your URL when accessing the front-end.
Inn terms of cookie exchange, the "Access-Control-Allow-Credentials" and "Access-Control-Allow-Methods" headers must be set correctly as shown above. Using a wildcard on "Access-Control-Allow-Methods" will not work.
This does not look right:
axios.defaults.headers.common = {
credentials: "include",
withCredentials: true
}
There are no such request headers. Instead credentials is controlled via XHR request.
Use this instead to make sure your client accepts cookies:
axios.defaults.withCredentials = true;
Previously I used the following type of headers in my function to make the request from the application:
Error
OPTIONS http://API_URL 405 (Method Not Allowed)
XMLHttpRequest cannot load http://API_URL. Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://192.168.1.9:8100' is therefore not allowed access. The response had HTTP status code 405.
Data Login
datos = {
Usuario: $scope.usuariotxt,
Password: $scope.passwordtxt
};
'Content-Type': 'application/x-www-form-urlencoded'
function
function Autenticacion(datos) {
var url = 'http://API_URL';
return $http.post(url, $httpParamSerializer(datos), {
headers: {
'Content-Type': 'application/x-www-form-urlencoded'
}
});
};
Now the content-type must be different from how you submitted the request in advance
'Content-Type': 'application/json'
function Autenticacion(datos) {
//var url = 'API_URL';
return $http.post(url, $httpParamSerializer(datos), {
headers: {
'Content-Type': 'application/json'
}
});
};
But using the ARC tool, the request works perfectly
The server is in Azure, this the information
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 270
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/8.0
X-Aspnet-Version: 4.0.30319
X-Powered-By: ASP.NET
Set-Cookie: ARRAffinity=0a3517ba6ed8bb14ffe517099672a3eb4ea3c4b710ad8c6e0edaa70c2d244335;Path=/;Domain=apipedroupc20170125045931.azurewebsites.net
Date: Tue, 28 Feb 2017 20:53:09 GMT
Seems this is a cross domain issue. For security reasons, browsers restrict cross-origin HTTP requests initiated from within scripts. No others implement this restriction, so it's ok for the ARC tool.
You can configure CORS in the Azure portal.
For more info, please see my earlier post from Stack Overflow: CORS Headers in Azure App Service running Express JS, or Azure's documentation: https://learn.microsoft.com/en-us/azure/app-service-api/app-service-api-cors-consume-javascript.
I have an angularjs 1.x application which communicates with a rest service using jsons.
Since the service sits under a different domain, I'm using CORS.
My angular application has a response interceptor that logs the response headers:
app.factory('responseInterceptor', ["$q", function ($q) {
return {
response: function (response) {
console.log(response.headers()); // log the headers!!!
return response || $q.when(response);
},
responseError: function (rejection) {
return $q.reject(rejection);
}
}
}]).
config(['$httpProvider', function ($httpProvider) {
$httpProvider.interceptors.push('responseInterceptor');
}]);
Now everything works ok, but I'm not able to access custom http response headers
I'm using a http sniffer tool and I can see the headers in the servers response.
I'm pretty sure that this has something to do with CORS but I can't nail it.
The response headers (including CORS) look like this:
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
MY-CUSTOM-HEADER: Go Home This Works
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: GET, POST, OPTIONS, PUT, PATCH, DELETE
Access-Control-Allow-Headers: MY-CUSTOM-HEADER, CONTENT-TYPE
Date: Tue, 21 Jun 2016 13:18:20 GMT
Content-Length: 14112
This is what I see in my console from the interceptor:
Object {pragma: "no-cache", content-type: "application/json; charset=utf-8", cache-control: "no-cache", expires: "-1"}
Questions:
What's the reason I cannot see the MY-CUSTOM-HEADER when I log the response headers? Or other http response header that I can see on my sniffing tool. Who blocks me from accessing them on the responseInterceptor?
Edit:
I've checked the same application under the same domain and COULD access the header with the same code. It does have something to do with CORS.
Ok, after identifying that the issue was CORS, I've found I was missing an CORS http header called Access-Control-Expose-Headers
After setting Access-Control-Expose-Headers="MY-CUSTOM-HEADER", I was able to access that header.