EDIT: Looks like this post is much ado about nothing, as the OPTIONS preflighting was introduced late last fall; my problems have been more user error than anything. See comments for more details (I'll leave it all here for documentation's sake). --jlm
It's a known issue that the Youtube API does not support the OPTIONS request method, so any web app that tries to do a Cross-Origin preflight will fail, even if (as is the case with the Youtube API) the actual CORS request would succeed. As we've been facing this issue yet again this week, we've come up with three workarounds; however, each has it's own plusses and minuses.
POSSIBLE WORKAROUND #1: Forget the preflight, and just make the simple cross-origin request.
Benefits -- A) it works. B) this is all the specification requires when doing most GET or POST requests.
Drawbacks -- A) The specification states that any PATCH, PUT, or DELETE requests, along with POST requests that use a content-type other than form-data, url-encoded, or text/plain, need to be preflighted. So while it would work, it would be breaking the spec (which we'd like to avoid if possible). B) Preflighting is also necessary when setting custom headers; so, for example, when I use AngularJS's $http method in the 1.0 branch, it sets a custom header and thus triggers preflighting of even GET requests. In this case, of course, I could write my own $http service or move to the 1.1 branch (as the problem would really be on Angluar's end).
POSSIBLE WORKAROUND #2: Use JSON-P
Benefits -- A) it works, too (in some cases, anyway). B) It's fairly simple to set up.
Drawbacks -- A) An older technology, and it isn't clear if the Youtube API will continue to support JSONP. B) Requires a callback. C) Is limited to GET requests only.
POSSIBLE WORKAROUND #3: Set up a server-side proxy on the same domain, use it to communicate with the Youtube API.
Benefits -- A) Avoids the need to do CORS requests at all, since the client works on the same origin and the server-side proxy has no need to preflight anything.
Drawbacks -- A) Can get complicated to set up, especially if trying to work with credentials (oAuth2 through a proxy can be quite the beast).
So that this post has an actual question (or several, actually)
What take do you have (for better for worse) on any or all of the workarounds above?
Has anyone implemented other solutions?
Is there any information as to if/when the Youtube servers might support the OPTIONS method?
Any and all comments are welcome -- and I apologize in advance if this isn't the best forum for such a question (although I'm hoping that by putting it here on Stack Overflow, it'll prove useful to others facing the same issue)
Youtube upload API(uploads.gdata.youtube.com) doesn't support CORS.
You may vote the issues here and here . There are also several forum posts about it. CORS is not working !
It seems Google is still working to fix it , though started working on it about an year ago so most likely the API will be deprecated before to have the issue fixed(as we already have a V3).
The only workaround is to use a proxy(which is not cool). Basically upload the video on your server and then send it to youtube.
Related
I have a blogging app built on top of the MERN Stack. I am fetching my blog posts on the react front-end, however, I feel anyone can use my blog posts on his own site by hitting the same end-point. I want to protect this behaviour. Is there a way?
If, for some reason, it isn't enabled already, make sure your endpoints have standard Access-Control-Allow-Origin restrictions - that is, that they only permit direct connections from your domain, not from other sites. This will make it slightly more difficult for other sites to scrape yours, because they won't be able to make requests directly from the frontend.
You could also change your application structure so that the blog data gets sent with the initial HTML response. For a small example, you could have
<script type="application/json" class="blog-data">
[{"title":"some post title", "content":"some content"}]
</script>
const blogData = JSON.parse(document.querySelector('.blog-data').textContent);
This will also make it a bit harder for a scraper to work - they won't have an endpoint ready to serve the plain blog data, they'll have to parse through your HTML response first.
You could also frequently change up the DOM structure of the data in the HTML response to make it harder.
But web scraping is fundamentally nearly impossible to stop, for someone who's determined enough.
Basically, you can use CORS on your backend to protected fetching your endpoints from any browsers origins except allowed ones.
Anyway it will not help you to protect from calling API from such things like mobile apps, Postman etc.
If you worry about loading to the server you can add something like rate limiting.
But keep in mind if your API is public it will be public for all, you can't restrict to use it from your site only.
Here are a few ideas:
Maybe add some authentication to protect your endpoints.
If you are using CORS, only accept requests from a certain URL.
In your package.json, add a proxy.
I'm still very new to React so forgive me if the question is too naive. To my understand, React usually requires an API to do XHR requests. So someone with very basic tech background can easily figure out what the api looks like by looking at the network tab in web browser's debug console.
For example, people might find a page that calls to api https://www.example.com/product/1, then they can just do brute force scraping on product id 1 - 10000 to get data for all products.
https://www.example.com/api/v1/product/1
https://www.example.com/api/v1/product/2
https://www.example.com/api/v1/product/3
https://www.example.com/api/v1/product/4
https://www.example.com/api/v1/product/5
https://www.example.com/api/v1/product/6
...
Even with user auth, one can just use same cookie or token when they login to make the call and get the data.
So what is the best way to prevent scraping on React app? or maybe the api shouldn't be designed as such, hence I'm just asking the wrong question?
Here are some suggestions to address the issue you're facing:
This is a common problem. You need to solve it by using id's that are GUID's and not sequentially generated integers.
Restricting to the same-origin won't work because someone can make a request through Postman or Insomnia or Curl.
You can also introduce rate-limiting
In addition, you can invalidate your token after a certain number of requests or require it to be renewed after every 10 requests
I think no matter what you do to the JavaScript code, reading your API endpoint is the easiest thing in the world(Wireshark is an easy, bad example), once it is called upon from the browser. Expect it to be public, with that said, protecting it it is easier than you might anticipate.
Access-Control-Allow-Origin is your friend
Only allow requests to come from given urls. This may or may not allow GET requests but it will always allow direct access on GET routes. Keep that in mind.
PHP Example
$origin = $_SERVER['HTTP_ORIGIN'];
$allowed_domains = [
'http://mysite1.com',
'https://www.mysite2.com',
'http://www.mysite2.com',
];
if (in_array($origin, $allowed_domains)) {
header('Access-Control-Allow-Origin: ' . $origin);
}
Use some form of token that can be validated
This is another conventional approach, and you can find more about this here: https://www.owasp.org/index.php/REST_Security_Cheat_Sheet
Cheers!
I am trying to implement a REST protocol and have realized in trying to debug that my web server is disallowing the PUT request.
I have tested and confirmed this further by running:
curl -X PUT http://www.mywebserver.com/testpage
Which for my web server gives back a 403 - Forbidden error.
The same happens for DELETE, where as for POST and GET everything is fine.
I am wondering if this is a common issue that those who use REST encounter and what the work-around might be?
Could I make a simple change to an .htaccess file? Or do I need to modify the protocol to set a hidden variable "_method" in the POST query string?
Often web servers will be configured to block anything except GET and POST since
99% of the time they're all that are needed and there have been problems in the
past with applications assuming the requests were one of those two.
You don't say which server it is but, for example, you can tell Apache which
methods to allow with the directive:
eg:
<Limit POST PUT DELETE>
Require valid-user
</Limit>
It sounds like maybe some helpful sysadmin has used this to block non GET/POST
You could try an .htaccess with
<Limit GET POST PUT DELETE>
Allow from all
</Limit>
(I'm not an expert at apache, this may not be exactly correct)
I'm trying to send JSON HTTP request from Google App Engine application and retrieve response, and while this works great locally, it suddenly breaks when I deploy it to GAE.
To be more precise, the HTTP response body that is returned to my application ends up looking like this instead of being simple JSON:
�\bD�[��8��ʖϣ�M�M$ �\\�bA` #!r���~pvk�cR]�_7E�
I did find one set of circumstances where I get correct response on GAE which might give some insight at this behavior - if the response doesn't have content-type header it goes through fine, but as soon as there is content-encoding header set to "gzip" present I get the incorrect garbage above as a response body.
Unfortunately I don't have control over the service I'm calling. So the only choice I have is to fix this somehow on my side, but to fix it I'm trying to understand the difference between what Google does to response. Does anyone know?
I understand that Google does some things to HTTP traffic. Is it forcing gzip on my responses as well?
I've also tried playing with encodings, trying to read response as utf-8, and setting utf-8 as default encoding for my GAE application as recommended here, but with no effect. I've ruled out incorrect processing of response in my code or anything I'm using, at least I think so, otherwise I'd have the same problems locally. I'm trying to understand what exactly is happening with hope that this will give me some idea how to prevent it.
EDIT: I figured it out and did a workaround but it's still a workaround, not a solution. So my GAE app calls another web service from outside GAE which sometimes gzips response and sometimes doesn't. If it does, GAE strips away content-type header from response, thus preventing my app from correctly decoding response body. My workaround so far is to get response bytes and test if response is valid JSON, unzipping it manually if it isn't. Would still want to know if stripping content-type can be prevented...
As explained at https://cloud.google.com/appengine/kb/#compression , the application should not supply the content-encoding header: "Google App Engine does its best to serve gzipped content to browsers that support it. Taking advantage of this scheme is automatic and requires no modifications to applications".
I believe that the origin of this architecture (that content encoding is not controlled by the application side of things -- on App Engine, your application code -- but by the server/gateway side) originated with WSGI, the Python standard interface of applications to web servers/gateways (which App Engine uses on the Python runtime), but the architecture makes enough sense that we generalized it -- as the above page puts it, "This approach avoids some well-known bugs with gzipped content in popular browsers.".
The client is far from powerless, and in fact, if it so chooses, can control the content encoding -- still quoting, "To force gzipped content to be served, clients may supply 'gzip' as the value of both the Accept-Encoding and User-Agent request headers. Content will never be gzipped if no Accept-Encoding header is present.".
Can anyone point me in the right direction of how I can use SSL client-side certificates with Silverlight to access a restful web service?
I can't seem to find anything on how to handle them, or even whether they are supported.
Cheers.
Slipjig mentioned this:
"The browser stack does, and pretty much automatically, if you're willing to live with its other limitations (lack of support for all HTTP verbs, coercion of response status codes, etc.)."
If that is acceptable to you, look at how Microsoft themselves deal with this in some of their APIs using the custom X-HTTP-Method header, like how they do it for WCF and OData:
http://www.odata.org/developers/protocols/operations
In MSDN, Microsoft also mentions this about using REST in conjunction with SharePoint 2010's WCF based REST API:
msdn.microsoft.com/en-us/library/ff798339.aspx
"In practice, many firewalls and other network intermediaries block HTTP verbs other than GET and POST. To work around this issue, WCF Data Services (and the OData standard) support a technique known as "verb tunneling." In this technique, PUT, DELETE, and MERGE requests are submitted as a POST request, and an X-HTTP-Method header specifies the actual verb that the recipient should apply to the request. For more information, see X-HTTP-Method on MSDN and OData: Operations (the Method Tunneling through POST section) on the OData Web site."
Don Box's also had some words about this, but regarding GData specifically:
www.pluralsight-training.net/community/blogs/dbox/archive/2007/01/16/45725.aspx
"If I were building a GData client, I honestly wonder why I'd bother using DELETE and PUT methods at all given that X-HTTP-Method-Override is going to work in more cases/deployments."
There's an article about Silverlight and Java interop which also addresses this limitation of Silverlight by giving the same advice:
www.infoq.com/articles/silverlight-java-interop
"Silverlight supports only the GET and POST HTTP methods. Some firewalls restrict the use of PUT and DELETE HTTP methods.
It is important to point out that true RESTful service can be created (conforming to all the REST principles listed above) only using the GET and POST HTTP methods, in other words the REST architecture does not require a specific mapping to HTTP. Google’s GData X-Http-Method-Override header is an example of this approach.
The following HTTP methods overrides may be set in the header to accomplish the PUT and DELETE actions if the web services interpret the X-HTTP-Method-Override header on a POST:
* X-HTTP-Method-Override: PUT
* X-HTTP-Method-Override: DELETE"
Hope this helps
-Josh
It depends on whether you're using the browser HTTP stack or the client HTTP stack. The client stack does not support client certificates, period. The browser stack does, and pretty much automatically, if you're willing to live with its other limitations (lack of support for all HTTP verbs, coercion of response status codes, etc.).
I have however been running into a problem using the browser stack with client certificates in an OOB scenario. Prism module loading fails under these conditions - the request gets to IIS, but causes a 500 server error for no apparent reason. If I set IIS to ignore client certs, or if I run the app in-browser, it works fine :-/
take a look at this.
http://support.microsoft.com/kb/307267
just change your urls to https
hope this helps
Dim url As Uri = New Uri(Application.Current.Host.Source, "../WebService.asmx")
Dim binding As New System.ServiceModel.BasicHttpBinding
If url.Scheme = "https" Then
binding.Security.Mode = ServiceModel.BasicHttpSecurityMode.Transport
End If
binding.MaxBufferSize = 2147483647 'this value set to override a bug,
binding.MaxReceivedMessageSize = 2147483647 'this value set to override a bug,
Dim proxy As New ServiceReference1.WebServiceSoapClient(binding, New ServiceModel.EndpointAddress(url))
proxy.InnerChannel.OperationTimeout = New TimeSpan(0, 10, 0)