I'm testing an application on Google App Engine.
I use the Flexible environment with a custom python runtime.
In my application I need a "session.id" header on HTTP requests.
My web application code extracts the value of session.id and validates it, if it's invalid or missing the web application returns HTTP 401.
When I issue a HTTP POST to my GAE web application using curl and setting the header, e.g.:
curl -v -X POST --data "echo" -H "session.id: someweirdandlargestring" https://*****.appspot.com/echo
it seems GAE proxy removes the "session.id" header, then the web app returns 401. If I send "session.id" as a Cookie everything works fine and the server returns 200. I've debugged the application and the header does not reach the web application.
I've read the docs (https://cloud.google.com/appengine/docs/flexible/python/how-requests-are-handled) and they describe headers that are expected to be removed or added from requests before they reach our actual server. But they say:
For security purposes, some headers are sanitized or amended by intermediate proxies before they reach the application.
What makes believe me GAE is removing the HTTP header I've set.
My questions:
Is this an expected behavior from GAE?
What would you suggest to fix it? In order words: how to make my header entry to reach the web server application code?
Note that using a cookie is not an option in short term.
As #viz has suggested I've tested a header name without the dot. It works.
If I use, let's say, "sessionid", then the header reaches the server.
GAE web servers are discarding malformed http headers.
The best I've found about HTTP header conventions are on NGINX docs http://nginx.org/en/docs/http/ngx_http_core_module.html#ignore_invalid_headers:
Valid names are composed of English letters, digits, hyphens, and possibly underscores
If this can help, please let me bring this September 21 update, as I bumped intot he same issue for PHP7, which GAE was seemingly discarding my "api_secret" header for (not really RFC-ish, it is?). I deployed a little debugging tool which dumped the headers, and carried on experiments.
Outcome:
headers are forced to camel case;
apisecret became Apisecret;
api-secret became Api-Secret;
api.secret was discarded;
I finally chose X-Api-Secret, to be in line with RFC2047 recommendation for user-defined headers.
Related
I'm building a ReactJS frontend that has to gather some data from AWS Lambdas using a JS fetch.
I cannot make it work, no mater what CORS technique I apply. I've looked into other answers here to no avail.
I am definitely adding Access-Control-Allow-Origin with "*" value in my response (verified this using postman to call the endpoint). Also, Chrome complains about the preflight with Response to preflight request doesn't pass access control check, but no preflight request (OPTIONS method) is ever actually fired by chrome, all I see is the GET I'm trying to make on the first place, which is really confusing.
What am I missing? Why is chrome complaining about preflight when no OPTIONS preflight request is made? Why adding Access-Control-Allow-Origin with "*" in my response is not enough?
Thanks!
TL;DR: There was a preflight request happening, it just wasn't showing on chrome (there's a way to make them show up). Also, there's a tweak to make if you use custom headers for authorization tokens for example.
Summary
Well, after looking into this for a day and checking several other answers I'm posting this because none quite fit my problem, with the hope it will help anyone else facing this.
First, I'll summarize the several parts involved in the error and then how to fix it, without resorting to any "hackish" solution like bypassing CORS with a chrome extension, or using any 3rd party service, like many posts suggest. My setup looked like this:
ReactJS frontend, try to make a GET request using fetch javascript method, running on http://localhost:3000 for development
AWS Lambda Backend, that answers the GET with a JSON payload (coded in python, not important but anyways). This lambda is adding Access-Control-Allow-Origin:"*" header to its response.
The above AWS Lambda is behind what AWS calls an "Authorizer", which is a function that runs before your lambda to check whatever authorization header you want to use to protect access to your API. This is important as we'll see later, because due to some nonsense on AWS inner workings, sometimes you cannot use the standard HTTP Authorization header, and it defaults to using authorizationToken as they suggest in their documentation and samples (and changing it not always works, there are plenty of users reporting this in their forums). We'll keep a note on this for later.
Both API methods (actual API and its Authorizer) routed and published on the internet using AWS API Gateway, which in short is a way to pair your lambda with a public URL to call it from elsewhere.
Google Chrome used as browser (with its developer tools enabled to monitor things)
The error
When trying to call the lambda, chrome blocks the GET request with this error showing on the console: Response to preflight request doesn't pass access control check. My lambda is already answering with the correct Access-Control-Allow-Origin header so, what's wrong? Also, no preflight OPTIONS request are being made anyways, so this was confusing.
Some debugging
AWS Lambda is great but their debugging tools are not as fluid as I would like, so I replaced the lambda with a local expressjs server, implementing just two methods: GET /foo and OPTIONS /foo. To my surprise, when from my ReactJS frontend I fetched /foo, it did call OPTIONS /foo first (I confirmed this by adding logs to my endpoint, etc, something you can also do in lambdas but its not as easy).
What was actually happening
A "preflight" request is an OPTIONS request to validate what is actually allowed when doing the following GET, but the Network tab in Chrome was not showing any OPTIONS request actually happening (I remember they used to show up here). Well, they changed it at some point, and now they are hidden by default. If you want them to show again (as a developer, I do), you can re-enable that by changing the out-of-blink-cors flag to disabled as explained here.
After changing this flag, now the OPTIONS request does show on the network tab. From there I could craft the OPTIONS response so it would enable the required GET afterwards. There are other considerations when using credentials and other cases (I found this article from Mozilla helpful with that), but in short my OPTIONS response headers look like this:
Access-Control-Allow-Origin: "http://localhost:3000"
Access-Control-Allow-Methods: "GET, POST, OPTIONS"
Access-Control-Allow-Headers: "authorizationToken"
(That last one, Access-Control-Allow-Headers, comes into play when dealing with AWS Lambdas Authorizers. If you are using that custom header to send your tokens, you need to allow it here).
After making CORS work locally, to solve it for my lambdas I did two things:
You need your API Gateway to be able to answer OPTIONS request. There are a number of ways to achieve this, from writing your own lambda to answer it to having AWS to mock-response it for you. More info on that here.
You need to make sure your GET lambda adds the Access-Control-Allow-Origin header, pointing to the same value your OPTIONS response did (In my case, http://localhost:3000).
After that, all worked as expected.
Final Notes
I wrote this answer because I found the conjunction of "React-CORS-AWS-Authorization" was not actually covered by any questions I found around.
Also there are a number of problems that may arise from using localhost for development on chrome, leading to suggestions of using an external service like lvh.me, but this was not the case, and some answers misleadingly relate this CORS problem to that. Moreover, some answers suggest disabling CORS checks altogether with some chrome extension, which is really bad security advice.
Finally, I found the idea of making a simple expressJS server to debug the server-side of things pretty helpful in understanding what was happening, because sometimes you simply cannot access what's happening on the other side, so maybe this suggestion might help people shorten the time dealing with things like this.
I have a nodejs express server running on app engine.
If i make a GET request to https://astral-pursuit-252600.appspot.com/users in the browser it works fine to say unauthorized (401).
If I do the same GET request in postman it returns 400 bad request.
Is there any obvious reason why this is occurring?
This is a known issue with postman. This tool sends certain headers by default that you cannot remove. App Engine does not like them for some reason. I had to use the Insomnia tool instead which does not include default headers.
The first thing that I can think about is that, in order to do an API call, you need to use an API key in your request. You should create one, after that you need to obtain an access token. Your requests should be send to an address like https://astral-pursuit-252600.appspot.com/users?key=YOUR_API_KEY and include in your request a header to contain the access token. Something like this : --header 'authorization: Bearer YOUR_ACCESS_TOKEN'.
In order to do that I do not think you need to change manually each request, but you need to change some POSTMAN settings. You can find here a guide with exactly what setting should be changed for this use case.
You can see more details about this topic and a more detailed guide for doing an API calls here.
In case this was not the issue, could you please provide me your POSTMAN settings? I am pretty sure this is about the way POSTMAN does the requests anyway.
Alright, first of all, I am absolutely aware that we have a bunch of answers on this and there is a plethora of articles on the topic. I just read these answers a second before typing this:
Why is CORS without credentials forbidden?.
Is CORS considered bad practice?
Etc. My particular situation is this - I just set up WebAPI2 for my practice project, the front end for which is running via gulp browser-sync. I have no idea how these ports get picked, but lets say the Web API is running on port http://localhost:1234/ and browser-sync generates the website on http://localhost:4321/. So I hit the API via angular's $http and get the famous CORS error (API controller method does get hit), so I am guessing it's the API returning not allowed. Edit: I fixed this via installing a CORS for Web API package via NuGet (Article Here) before asking this Q, just referencing for anyone who might need it later.
So, I was thinking, if I deployed this, ANY request would get denied, unless I am missing something. Or would it not be denied because of something I don't understand? Is disallowing CORS just a throwback from the MVC days? Or is there some purpose to it with APIs?
Maybe I am just ranting, but this confuses the **** out of me.
CORS is based on the response headers returned from the API. It is not the API that rejects responding to the request, the web browser explicitly disallows handling the response. The API will process the request as normal.
When dealing with anything other than a GET, CORS also requires a "preflight" request to the API first, to ensure subsequent requests are allowed. This amongst sending the headers back is what the Web API nuget package provides.
CORS is off by default for security purposes.
I originally wrote an REST API to work with a previously written mobile app. The mobile programmer requested from me to generate an auth_token on login that he will pass as a header on each request that needed authentication. This API runs at api.example.com.
Later on, I was commissioned to write an AngularJS app that communicates with this API, so I had to use Access-Control-Allow headers on the backend for OPTIONS requests to be CORS compatible CORS so my browser allows the connection (looks like iOS does not look for this headers). This app runs at one.example.com.
Now, I have to write a second AngularJS app that will run at two.example.com and there's a third being planned for the near future at three.example.com.
My problem is that my Access-Control-Allow-Origin header looks like this:
Access-Control-Allow-Origin: http://one.example.com:80
* is not allowed, nor I'm able to set this header to more than one origin. So as far as I can see I have two solutions:
Implement token-based authentication in parallel to the current cookie-based one. I'm thinking on this. This will of course take some time I'm willing to save.
Send the requester a header or param to the API endpoint identifying the app on the OPTIONS request and server-side, produce the CORS headers accordingly. I don't even know if it's possible and this looks nasty for even thinking it.
Any better ideas?
If they have the same origin, example the same domain (example.com) or the same subdomain (1.ex.example.com and 2.ex.example.com) they can share the same cookie. Because cookie is based on the domain itself.
In the APP Engine API, it is mentioned that, if the request comes with "Accept-Encoding" set, then it will automatically compress the response.
But when I look at the request, the header is not there. but at the browser, it is set. when I try to explicitly set the header(with JQuery ajax function), there is a message:
Refused to set unsafe header "Accept-Encoding"
But this situation is not occurring when working in local host - request has the "Accept-Encoding" header. this happens only after publishing. but not allowing to set the "Accept-Encoding" explicitly happens always.
I searched everywhere, but couldn't find a explanation to the problem. It would be really helpful if someone can explain...
You have two different problems:
App Engine does not compress reply. GAE uses a number of factors to determine if response needs to be compressed. It takes content type and user agent into account when deciding. See the answer by Nick Johnson (from GAE team).
jQuery refuses to set "Accept-Encoding" header. Note that this is a jQuery issue and has nothing to do with GAE. See this: Is it possible to force jQuery to make AJAX calls for URLs with gzip/deflate enabled?
I have a similar problem as in the HTTPRequest header, "Accept-Encoding" is null. As GAE has explained it looks for Accept-Encoding and User-Agent headers, if it wants to compress. but in my case there is no way the GAE can recognize whether to compress.
From the browser, then header is set, but in the request header, it is not.