I have many sub-domains and I want to event data for each sub-domain.
what I did is execute HTTP request like this, for each sub-domain:
/?
idSite=6&
token_auth=out-outh-token&
format=json&
date=2014-10-13%2C2014-10-14&
expanded=1&
segment=eventCategory%3D%3Dgoals%3BpageUrl%3D%40some-sub-domain&
method=Events.getCategory&
module=API&
period=range
which works fine for development, but as I execute many HTTP requests, they (the requests) start to time out.
so I added waits every time it encountered a timeout, what made it slow...
is there a way to get all this data in one request?
is there a way to better handle this problem?
How many subdomains do you have? Please consider adding those segments in the segment editor and set them to "pre-archived" (I assume you have cron archiving set up).
It's normal that such queries timeout. Piwik by default processes such requests (needs to iterate through all the reports with segment applied in the "where" clause).
Related
This is more like a question about the right approach:
We have an single page web application in angularjs that is loading a view that contains multiple diagrams. Each diagram fetch the data that needs to be displayed through the REST service. There is a limitation in chrome with 6 connection simultaneously. As we have views with more than 10 diagrams the data fetch results in queuing the calls untils previous one are resolved. This appears to the user as if the data fetch is slow.
Is there a way to execute all calls in parallel (same server, different REST endpoints)?
What where the single page solution that would not be limited by the browser but provide faster throughput?
Caching in frontend is only partially applicable, due to the active filtering of data by the user.
One solution will be combining multiple request to one request, by that the overhead of multiple connection establishment time will be gone.
You can make a proxy api which can take care of them.
The problem with combining endpoints is, if any of your endpoint has higher processing time then the other combined endpoints response has to wait for it.
Best solution is, make the endpoints first enough so 6 connections are enough
I'm using OSB 12c.I have an OSB proxy, which takes 15 minutes to complete each request on an average.
Lets say that I have five request now in running state.
Is there a way to see these running requests just like we can see the requests of bpel in EM console ?
Is there a way to terminate one of the requests without any impact on rest of the running requests ?
Is it possible to kill all request in case point-2 is not possible ?
Thanks !
I don't think so, not without changing things.
If you were open to changing the service to e.g. decompose the request into separate internal JMS messages, you should be able to use JMX to interrogate the MDBs and discover what they're up to. Then again, if you were to switch to JMS you could probably just look at the queue and get an idea about what it's doing just from the number and content of the messages created.
I'm not aware of the ability to cancel individual requests in OSB, sorry.
I think you can not terminate OSB threads directly.
You can configure your Weblogic to deal with stuck threads. (Thread that were running for a certain period of time)
You can configure a Dispatch Policy in your proxys using Work Manager to handle Stuck Threads and minimize impact on server.
Every minute or so my app creates some data and needs to send it out to more than 1000 remote servers via URL Fetch callbacks. The callback URL for each server is stored on separate entities. The time lag between creating the data and sending it to the remote servers should be roughly less than 5 seconds.
My initial thought is to use the Pipeline API to fan out URL Fetch requests to different task queues.
Unfortunately task queues are not guaranteed to be executed in a timely fashion. Therefore from requesting a task queue start to it actually executing could take minutes to hours. From previous experience this gap is regularly over a minute so is not necessarily appropriate.
Is there any way from within App Engine to achieve what I want? Maybe you know of an outside service that can do the fan out in a timely fashion?
Well, there's probably no good solution for the gae here.
You could keep a backend running; hammering the datastore/memcache
every second for new data to send out, and then spawn dozens of async url-fetches.
But thats really inefficient...
If you want a 3rd party service, pubnub.com is capable of doing fan-out, however i don't know if it could fit in your setup.
How about using the async API? You could then do a large number of simultaneous URL calls, all from a single location.
If the performance is particularly sensitive, you could do them from a backend and use a B8 instance.
We're using sessions in our GAE/J application. Over the weekend, we had a large spike in our datastore writes that appears to have been caused by a large number of _ah_SESSION entities being created (about 100-200 every minute). Near as we can tell, there was a rogue task queue creating them because they stopped when we purged the queue. The task was part of a mapper process we run hourly.
We don't need sessions in that hourly mapper (or indeed in any of our task queues or cron jobs or many other requests). Is there a way to disable creating a session for selected URLs?
Unfortunately that can not be done.
This is particularly nasty when you have a non-browser clients (devices via REST or mapreduce jobs) where every request generates a new _ah_SESSION entity in the database.
The only way to avoid this is to write your own session handler: e.g. a servlet filter that sets/checks cookies and set it so that it ignores certain paths.
EDIT:
I just realized that there could be another way: make sure your client (mapreduce job) sets a dummy cookie with a proper name. GAE uses cookies named ACSID in production and dev_appserver_login on dev server. Just use always the same cookie value, so all requests will be treated as one user/session.
There will still be overhead of looking-up/saving session objects, but at least it will not create countless _ah_SESSION entities.
I'm building a simple "play against a random opponent" back-end using Goole App Engine. So far I'm adding each user that wants to play into a "table" in the Datastore. As soon as there are more than 1 player in the Datastore, I can start to match them.
The Schedule Tasks with Cron looked promising for this work until I saw that the lowest resolution seems to be minutely. If there are plenty of players signing up I want them to be matched quickly and not have to wait a whole minute (worst case).
I thought about having the servlet that recives the "play against random opponent" request POST to a Task Queue that would do the match making, but I think this will lead to a lot of contention when reading from the Datastore and deleting the enteties from the "random" table after they have been matched?
Basically I want a single worker that will do the matching, and I want to signal this worker from time to time that now is a good time to try to match opponents.
Any suggestions on what would be the right course of action here?
You can guarantee exclusive access via transactions:
Receive a request to play via REST. Check (within a transaction) if there is any request in database.
If there is, notify both users to start the play and delete request (transactionaly) from database.
If there isn't, add it to the database and wait for the next request.
Update:
Alternativelly you can achieve what you need via pull queue. Same scenario as above, just instead of datastore you'd check if there is a task in the pull queue, retrieve if there is or create a new one if there isn't one.