Since requesting the Inventory from https://api.steampowered.com/IEconItems_730/GetPlayerItems/v1/ is permanently disabled I have to use https://steamcommunity.com/profiles/<steam_id>/inventory/json/2
Because of CORS I have to use my backend to provide the requested data. But when I do requests too often, my requests get rejected and my app cannot work on a large scale.
So the question should be simple: How can I avoid the rejection of my requests?
Any ideas and suggestions welcome.
Steam inventory endpoints are pretty heavily rate-limited, but there are a few different endpoints that you can use.
Trade offer endpoint
https://steamcommunity.com/tradeoffer/new/partnerinventory
This is the endpoint that's used when you open a trade offer with someone. It can be used to fetch both your own inventory as well as a trade partners. Required parameters are partner which is the user's Steam 64 Id, appid which is 730 in the case of CS:GO, and contextid which is 2 for most valve games. I don't know the exact limit, but I've been hitting this endpoint about once a second for a month with minimal 429 responses. To use this endpoint, you need to have a valid steam session and send the proper cookies along with the request. This will also only return tradable items.
Inventory endpoint #1
http://steamcommunity.com/inventory/STEAM64ID/APPID/CONTEXTID
Another inventory endpoint which has the same parameters but in the URL. I use this endpoint as a fallback to the first, but I've found that if the first endpoint is rate limited then this one will be too. That said, this one becomes limited much faster so it's best to use the first one instead.
Inventory endpoint #2
http://steamcommunity.com/profiles/STEAM64ID/inventory/json/APPID/CONTEXTID
The endpoint you're using. I don't use this one at all, but it could be worth knowing as another fallback.
Not all of them return the same data format, so be mindful. One inventory a second is a pretty solid rate for any decently sized site, especially if you limit user's ability to refresh inventories. If you need more though, you'll have to start looking into proxies.
Related
I make a web service and I'm going to use a React. A data for the service will be fetch from my API.
But there is a simple way to find out which endpoints I'm using, and what data I'm sending. This knowledge gives a lot options to make bots for my service.
Is there any option to prevent this?
I know, I can require a signing all requests, but it's also easy to get to know.
This cannot be done. Whatever is done in client-side JavaScript, can be reverse-engineered and simulated.
Efforts should be focused on preventing API from being abused, i.e. throttling or blacklisting clients based on their activity or available information (user agent, suspicious request, generated traffic). If the use of API allows captcha, suspicious clients can be asked for proving their humaneness.
There are half-measures that can be applied to client side application and make it less advantageous for abuse (and also for development).
Prevent unauthorized access to unminified/unobfuscated JS AND source maps. There may be a need to authorize them on per user basis. This will make debugging and bug reporting more difficult
Hard-code parts that are involved in request signing to browser APIs, e.g.:
apiKey = hash(NOT_SO_SECRET_KEY + document.querySelector('.varyingBlock').innerHTML)
This requires bots to emulate browser environment and makes their work much less efficient. This also affects the design of the application in negative way. Obviously, there will be additional difficulties with SSR and it won't translate to native platforms easily.
here two basic preventive measures that you can use.
Captcha
Use a captcha service like recaptcha. so that user can use your website only after passing the captcha test. Its highly difficult for bots to pass the captchas.
Rate Limit Api usage.
Add rate limiting to your api. so that a logged in user can only make 100 requests in 10 minutes, the numbers will depend on you use case
Hi I am currently using channel API for my project. My client is a signage player which receives data from app engine server only when user changes a media content. Appengine sends data to client only ones or twice a day. Do you think channel api is a over kill for this? what are some other alternatives?
Overall, I'd think not. How many clients will be connected?
Per https://cloud.google.com/appengine/docs/quotas?hl=en#Channel the free quota is 200 channel-hours/day, so if you have no more than 8 clients connected you'll be within the free quota -- no "overkill".
Even beyond that, per https://cloud.google.com/appengine/pricing , there's "no additional charge" beyond the computational resources keeping the channel open entails -- I don't have exact numbers but I don't think those resources would be "overkill" compared with alternatives such as reasonably frequent polling by the clients.
According to the Channel API documentation (https://cloud.google.com/appengine/features/#channel), "The Channel API creates a persistent connection between an application and its users, allowing the application to send real time messages without the use of polling.". IMHO, yours might not the best use case for it.
You may want to take a look into the TaskQueue API (https://cloud.google.com/appengine/features/#taskqueue) as an alternative of sending data from AppEngine to the client.
I am making an application based on Amazon Product API (or could be MWS), I will need to fetch data huge information again and again. So, in order to avoid Throttling limits, i would like to use Proxy ips.
Is this a valid thing to do.
1. Is the throttling limit apply per MWS account or per IP address
2. There are several proxy hosts avaialble for free/commerical. Is it ok to use them.
thanks
Valid? not really.
Amazon has some sharp people working there. I'm pretty sure this scheme wouldn't last very long. Your api access would be revoked pretty quickly.
I've read quite a few SO threads about authentication and authorization with REST and Angular, but I'm still not feeling like I have a great solution for what I'm hoping to do. For some background, I'm planning to building an app in AngularJS where I want to support:
Limited guest access
Role-based access to the application once authenticated
Authentication via APIs
All of the calls to the REST API will be required to occur over SSL. I'd like to do build the app without breaking RESTful principles, namely not keeping session state stored on the server. Of course, whatever is done vis-a-vis authorization on the client-side has to be reinforced on the server side. Since we need to pass the entire state with each request, I know I need to pass some sort of token so that the backend server receiving the REST request can both authenticate and authorize the call.
With that said, my main question is around authentication - what are the best practices here? It seems there are lots of different approaches discussed, here's just a few that I've found:
http://broadcast.oreilly.com/2009/12/principles-for-standardized-rest-authentication.html
http://frederiknakstad.com/2013/01/21/authentication-in-single-page-applications-with-angular-js/
http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html
There was a similar question asked (AngularJS best practice application authentication), but unless I'm misunderstanding the answer, it seems to imply that a server session should be used, which is breaking RESTful principles.
My main concern with the Amazon AWS and the George Reese article is it seems to assume that the consumer is a program, rather than an end user. A shared secret can be issued to a programmer in advance, who can then use it to encode calls here. This isn't the case here - I need to call the REST API from the app on behalf of the user.
Would this approach be enough? Let's say I have a session resource:
POST /api/session
Create a new session for a user
To create a session, you need to POST a JSON object containing the "username" and "password".
{
"email" : "austen#example.com",
"password" : "password"
}
Curl Example
curl -v -X POST --data '{"username":"austen#example.com","password":"password"}' "https://app.example.com/api/session" --header "Content-Type:application/json"
Response
HTTP/1.1 201 Created {
"session": {
"id":"520138ccfa4634be08000000",
"expires":"2014-03-20T17:56:28+0000"
}
}
Status Codes
201 - Created, new session established
400 - Bad Request, the JSON object is not valid or required information is missing
401 - Unauthorized, Check email/password combo
403 - Access Denied, disabled account or license invalid
I'm leaving out the HATEOAS details for clarity. On the backend, there would be a new, limited duration session key created and associated with the user. On subsequent requests, I could pass this as part of the HTTP headers:
Authorization: MyScheme 520138ccfa4634be08000000
Then the backend servers would be responsible for digesting this out of the request, finding the associated user and enforcing authorization rules for the request. It should probably update the expiration for the session as well.
If all this is happening over SSL, am I leaving the door open to any kind of attacks that I should be protecting against? You could try to guess session keys and place them in the header, so I suppose I could additionally append a user GUID to the session key to further prevent brute force attacks.
It's been a few years since I've actively programmed and I'm just getting back into the swing here. Apologies if I'm being obtuse or unnecessarily reinventing the wheel, just hoping to run my ideas by the community here based on my reading thus far and see if they pass the litmus test.
When someone asks about REST authentication, I defer to the Amazon Web Services and basically suggest "do that". Why? Because, from a "wisdom of the crowds" point of view, AWS solves the problem, is heavily used, heavily analyzed, and vetted by people that know and care far more than most about what makes a secure request than most. And security is a good place to "not reinvent the wheel". In terms of "shoulders to stand on", you can do worse than AWS.
Now, AWS does not use a token technique, rather it uses a secure hash based on shared secrets and the payload. It is arguably a more complicated implementation (with all of its normalization processes, etc.).
But it works.
The downside is that it requires your application to retain the persons shared secret (i.e. the password), and it also requires the server to have access to that a plain text version of the password. That typically means that the password is stored encrypted, and it then decrypted as appropriate. And that invite yet more complexity of key management and other things on the server side vs secure hashing technique.
The biggest issue, of course, with any token passing technique is Man in the Middle attacks, and replay attacks. SSL mitigates these mostly, naturally.
Of course, you should also consider the OAuth family, which have their own issues, notably with interoperability, but if that's not a primary goal, then the techniques are certainly valid.
For you application, the token lease is not a big deal. Your application will still need to operate within the time frame of the lease, or be able to renew it. In order to do that it will need to either retain the user credential or re-prompt them for it. Just treat the token as a first class resource, like anything else. If practical, try and associate some other information with the request and bundle it in to the token (browser signature, IP address), just to enforce some locality.
You are still open to (potential) replay problems, where the same request can be sent twice. With a typical hash implementation, a timestamp is part of the signature which can bracket the life span of the request. That's solved differently in this case. For example, each request can be sent with a serial ID or a GUID and you can record that the request has already been played to prevent it from happening again. Different techniques for that.
Here is an incredible article about authentication and login services built with angular.
https://medium.com/opinionated-angularjs/7bbf0346acec
This SO question do a good job of summing up my understanding of REST
Do sessions really violate RESTfulness?
If you store a token in a session you are still creating state on the server side (this is an issue since that session is typically only stored on the one server, this can be mitigated with sticky sessions or other solutions).
I'd like to know what your reasoning is for creating a RESTful service though because perhaps this isn't really a large concern.
If you send a token in the body along with every request (since everything is encrypted with SSL this is okay) then you can have any number of servers (load balanced) servicing the request without any previously knowledge of state.
Long story short I think aiming for RESTful implementations is a good goal but being purely stateless certainly creates an extra layer of complexity when it comes to authentication and verifying authorization.
Thus far I've started building my back-ends with REST in mind, making URIs that make sense and using the correct HTTP verbs, but still use a token in a session for the simplicity of authentication (when not using multiple servers).
I read through the links you posted, the AngularJS one seems to focus just on the client and doesn't seem to explicitly address the server in that article, he does link to another one (I'm not a Node user so forgive me if my interpretation is wrong here) but it appears the server is relying on the client to tell it what level of authorization it has which is clearly not a good idea.
I am developing a web app that will be hit frequently by mobile browsers. I am wondering if there is a way to get enough information from the browser request to lookup position data (triangulation or GPS) Not from the request directly, of course. A colleague suggested there some carriers supply a unique identifier in the request header that can be sent to a web service exposed by said provider that will return position data if the customer has enabled that. Can anyone point me in the right direction for this or any other method for gleaning position data, even very approximate. Obviously this is app candy, e.g. if the data is not available the app doesn't really care...
Or perhaps a web service by carrier that will provide triangulated data by IP?
Google has ClientLocation as part of their AJAX APIs. You'll need to load Google's AJAX API (requires an API key) and it'll try to resolve the user's location data for you.
I've got blackberry gps to javascript working OK in a GMaps mashup. Pretty simple, actually. http://www.saefern.org/tickets/test4.php -- help yrself to view source.
(I don't currently have a bb. A user emailed me with "... it seems to be polling every 15 seconds or so, so it keeps adding new locations ... ".)
I'm looking for javascript gps info on an iPhone equivalent. And Nokia, and ... .
Any information appreciated.
I have used this javascript library sucessfully:
http://code.google.com/p/geo-location-javascript/
The examples work great. The user will always be prompted to share their location--don't know a way to avoid that.
Use the source IP address to approximate a network location. No, you won't get latitude and longitude in an HTTP request from an iPhone. Not unless you write a 3rd party app and ask them to run it.
You might be better off just running a poll on your website.
I know that some providers in Japan have a tracking service for location of cellphones.
I also know that the information is not public. I think you need to have a very good reason before the provider gives that information free as it is in my opinion sensitive personal data. Of course they will give the information to police officers but not to the general public.