I am making an application based on Amazon Product API (or could be MWS), I will need to fetch data huge information again and again. So, in order to avoid Throttling limits, i would like to use Proxy ips.
Is this a valid thing to do.
1. Is the throttling limit apply per MWS account or per IP address
2. There are several proxy hosts avaialble for free/commerical. Is it ok to use them.
thanks
Valid? not really.
Amazon has some sharp people working there. I'm pretty sure this scheme wouldn't last very long. Your api access would be revoked pretty quickly.
Related
Since requesting the Inventory from https://api.steampowered.com/IEconItems_730/GetPlayerItems/v1/ is permanently disabled I have to use https://steamcommunity.com/profiles/<steam_id>/inventory/json/2
Because of CORS I have to use my backend to provide the requested data. But when I do requests too often, my requests get rejected and my app cannot work on a large scale.
So the question should be simple: How can I avoid the rejection of my requests?
Any ideas and suggestions welcome.
Steam inventory endpoints are pretty heavily rate-limited, but there are a few different endpoints that you can use.
Trade offer endpoint
https://steamcommunity.com/tradeoffer/new/partnerinventory
This is the endpoint that's used when you open a trade offer with someone. It can be used to fetch both your own inventory as well as a trade partners. Required parameters are partner which is the user's Steam 64 Id, appid which is 730 in the case of CS:GO, and contextid which is 2 for most valve games. I don't know the exact limit, but I've been hitting this endpoint about once a second for a month with minimal 429 responses. To use this endpoint, you need to have a valid steam session and send the proper cookies along with the request. This will also only return tradable items.
Inventory endpoint #1
http://steamcommunity.com/inventory/STEAM64ID/APPID/CONTEXTID
Another inventory endpoint which has the same parameters but in the URL. I use this endpoint as a fallback to the first, but I've found that if the first endpoint is rate limited then this one will be too. That said, this one becomes limited much faster so it's best to use the first one instead.
Inventory endpoint #2
http://steamcommunity.com/profiles/STEAM64ID/inventory/json/APPID/CONTEXTID
The endpoint you're using. I don't use this one at all, but it could be worth knowing as another fallback.
Not all of them return the same data format, so be mindful. One inventory a second is a pretty solid rate for any decently sized site, especially if you limit user's ability to refresh inventories. If you need more though, you'll have to start looking into proxies.
I make a web service and I'm going to use a React. A data for the service will be fetch from my API.
But there is a simple way to find out which endpoints I'm using, and what data I'm sending. This knowledge gives a lot options to make bots for my service.
Is there any option to prevent this?
I know, I can require a signing all requests, but it's also easy to get to know.
This cannot be done. Whatever is done in client-side JavaScript, can be reverse-engineered and simulated.
Efforts should be focused on preventing API from being abused, i.e. throttling or blacklisting clients based on their activity or available information (user agent, suspicious request, generated traffic). If the use of API allows captcha, suspicious clients can be asked for proving their humaneness.
There are half-measures that can be applied to client side application and make it less advantageous for abuse (and also for development).
Prevent unauthorized access to unminified/unobfuscated JS AND source maps. There may be a need to authorize them on per user basis. This will make debugging and bug reporting more difficult
Hard-code parts that are involved in request signing to browser APIs, e.g.:
apiKey = hash(NOT_SO_SECRET_KEY + document.querySelector('.varyingBlock').innerHTML)
This requires bots to emulate browser environment and makes their work much less efficient. This also affects the design of the application in negative way. Obviously, there will be additional difficulties with SSR and it won't translate to native platforms easily.
here two basic preventive measures that you can use.
Captcha
Use a captcha service like recaptcha. so that user can use your website only after passing the captcha test. Its highly difficult for bots to pass the captchas.
Rate Limit Api usage.
Add rate limiting to your api. so that a logged in user can only make 100 requests in 10 minutes, the numbers will depend on you use case
Hi I am currently using channel API for my project. My client is a signage player which receives data from app engine server only when user changes a media content. Appengine sends data to client only ones or twice a day. Do you think channel api is a over kill for this? what are some other alternatives?
Overall, I'd think not. How many clients will be connected?
Per https://cloud.google.com/appengine/docs/quotas?hl=en#Channel the free quota is 200 channel-hours/day, so if you have no more than 8 clients connected you'll be within the free quota -- no "overkill".
Even beyond that, per https://cloud.google.com/appengine/pricing , there's "no additional charge" beyond the computational resources keeping the channel open entails -- I don't have exact numbers but I don't think those resources would be "overkill" compared with alternatives such as reasonably frequent polling by the clients.
According to the Channel API documentation (https://cloud.google.com/appengine/features/#channel), "The Channel API creates a persistent connection between an application and its users, allowing the application to send real time messages without the use of polling.". IMHO, yours might not the best use case for it.
You may want to take a look into the TaskQueue API (https://cloud.google.com/appengine/features/#taskqueue) as an alternative of sending data from AppEngine to the client.
I'm writing a RESTful Database Server called Phoenix. Being an easy interface into an entire application's data, security is quite an important issue, and I'm interested in what authentication solutions any of you could suggest.
It needs to be:
Secure - it's got to be very hard to break. Signing requests could be a good way of doing this, but considering it's REST there aren't many parameters that are sent so I don't know what good singing would do.
Minimal - I'd rather it didn't take four requests to compare six tokens in HMAC-signed requests - the USP of the server is it's simplicity, so authentication from clients has got to be easy.
Implementable - it has to fit the system, which is a database server. So, for instance, oAuth wouldn't work here.
I'd love to hear your suggestions - thank you!
Jamie
Not much information here about what your security or implementation needs are. The quick answers are Basic or Digest over SSL, or signed requests. Are there reasons not to use these?
Signing requests typically adds a timestamp and/or a nonce, so any request can be authenticated. See the Amazon AWS authentication documentation for a description and libraries.
I have a similar server. I choose to use OAuth signing for its simplicity,
http://oauth.net/core/1.0#signing_process
We don't enforce the nonce, just limit the timestamp to a short window (30 seconds) to thwart replay.
The OAuth library is available on many platforms so you don't have to write much code to implement it. Don't know why you think OAuth is not implementable.
For each client allowed to access the data, it's assigned a consumer_key and a consumer_secret. All the requests are signed with consumer_secret so only client knowing the secret can get access.
We also considered other options. HTTTP Basic Auth over SSL is too expensive. HTTP Digest Auth is too slow because it needs to wait for a challenge.
I am developing a web app that will be hit frequently by mobile browsers. I am wondering if there is a way to get enough information from the browser request to lookup position data (triangulation or GPS) Not from the request directly, of course. A colleague suggested there some carriers supply a unique identifier in the request header that can be sent to a web service exposed by said provider that will return position data if the customer has enabled that. Can anyone point me in the right direction for this or any other method for gleaning position data, even very approximate. Obviously this is app candy, e.g. if the data is not available the app doesn't really care...
Or perhaps a web service by carrier that will provide triangulated data by IP?
Google has ClientLocation as part of their AJAX APIs. You'll need to load Google's AJAX API (requires an API key) and it'll try to resolve the user's location data for you.
I've got blackberry gps to javascript working OK in a GMaps mashup. Pretty simple, actually. http://www.saefern.org/tickets/test4.php -- help yrself to view source.
(I don't currently have a bb. A user emailed me with "... it seems to be polling every 15 seconds or so, so it keeps adding new locations ... ".)
I'm looking for javascript gps info on an iPhone equivalent. And Nokia, and ... .
Any information appreciated.
I have used this javascript library sucessfully:
http://code.google.com/p/geo-location-javascript/
The examples work great. The user will always be prompted to share their location--don't know a way to avoid that.
Use the source IP address to approximate a network location. No, you won't get latitude and longitude in an HTTP request from an iPhone. Not unless you write a 3rd party app and ask them to run it.
You might be better off just running a poll on your website.
I know that some providers in Japan have a tracking service for location of cellphones.
I also know that the information is not public. I think you need to have a very good reason before the provider gives that information free as it is in my opinion sensitive personal data. Of course they will give the information to police officers but not to the general public.