Due to the async nature of http access via WebClient or HttpWebRequest in Silverlight 4, when I want to do multiple http get/posts serially, I find myself writing code that looks like this:
doFirstGet(someParams, () =>
{
doSecondGet(someParams, () =>
{
doThirdGet(...
}
});
Or something similar. I'll end up nesting subsequent calls within callbacks usually implemented using lambdas of some sort. Even if I break things out into Actions or separate methods, it still ends up being hard to read.
Does anyone have a clean solution to executing multiple http requests in SL 4 serially?
I don't need the code that actually kicks all of this off to be synchronous, but I need the requests to happen serially, so each request needs to be effectively synchronous.
Take look at a couple of my blog posts on this:-
Simple Asynchronous Operation Runner – Part 1
Simple Asynchronous Operation Runner – Part 2
The articles are a little in depth because they focus on the actual implementation, the idea is not have to include fancy frameworks just to get this done. The only code you need is actually there in the articles, no additional dlls or zip files to download.
However note in part 2 the approach of imagining what your code would look like if synchronous coding were possible. In your case your code would look like this:-
void StuffToDo()
{
doFirstGet(someParams);
doSecondGet(someParams);
doThirdGet(...);
}
The next step is modify the content of the "do" methods to return an AsyncOperation instead. Currently they probably look something like this:-
void doFirst(someParams, Action callback)
{
SomeAsyncObj thing = new SomeAsyncObj();
thing.OnCompleted += (s, args) { callback() };
thing.DoSomethingAsync();
}
You would convert it to:-
AsyncOperation doFirst(someParams)
{
return (completed) =>
{
SomeAsyncObj thing = new SomeAsyncObj();
thing.OnCompleted += (s, args) =>
{
try
{
completed(null);
}
catch (Exception err)
{
completed(err);
}
};
thing.DoSomethingAsync(source);
};
}
The third step is modify your imaginary synchronous code like this:-
IEnumerable<AsyncOperation> StuffToDo()
{
yield return doFirstGet(someParams);
// Do some other synchronous stuff here, this code won't run until doFirstGet has completed.
yield return doSecondGet(someParams);
// Do some other synchronous stuff here, this code won't run until doSecondGethas completed.
yield return doThirdGet(...);
// Do some final synchronous stuff here, this code won't run until doThirdGethas completed.
}
Finally the call to StuffToDo changes to:-
StuffToDo().Run((err) =>
{
// report any error in err sensibly
});
I have encountered this problem. I found coroutines to be very helpful, I based my ideas on work by Jeremy Likness and Rob Eisenberg.
This link will give more details.
Encapsulate each request into a class
Create a collection or array of the requests
Iterate through the collection making and processing each request.
Related
Suppose I need to fetch data to create a card. What is the quickest way to get this data using promises? This is the current way I'm doing it:
async function getCards() {
const promises = []
for (let i = 0; i < 10; i++) {
promises.push(getCard(i))
}
const cards = await Promise.allSettled(promises)
setCards(cards)
}
async function getCard(i) {
const property1 = await getProperty1(i)
const property2 = await getProperty2(i)
const property3 = await getProperty3(i)
const card = <div>
<div>Property 1: {property1}</div>
<div>Property 2: {property2}</div>
<div>Property 3: {property3}</div>
</div>
return card
}
For my purposes, I don't need Promise.allSettled, since I don't need to wait for all 10 cards to finish awaiting (I may just create a component), I can render each one as they complete. But I'd still like it to be parallel/execute as fast as possible. What other options do I have there? And is there a better way to handle what I'm doing in getCard?
If getPropertyN() are indeed an asynchronous operation (such as a networking request), then getCards() will run all the calls in your for loop in parallel, such that they are all in-flight at the same time and it will generally reduce the end-to-end time vs. run them serially.
There are some other factors in play, such as what the receiving host does when it receives a bunch of requests at once. If it only handles them one at a time, then you may not gain a whole lot. But, if the host has any parallelism, then you will definitely see a speedup by putting multiple requests in flight at the same time.
Note that your getCard(i) implementation is serializing the three calls to getProperty1(), getProperty2() and getProperty3() which perhaps could also be done in parallel with something like:
const [property1, property2, property3] = await Promise.all([
getProperty1(i),
getProperty2(i),
getProperty3(i)
]);
Instead of this:
const property1 = await getProperty1(i)
const property2 = await getProperty2(i)
const property3 = await getProperty3(i)
Another thing to keep in mind is that a browser (such as a fetch() call) will only make N simultaneous requests to the same host (where N is around 6). Once you exceed that number of requests to the same host that are all in-flight at the same time, then the browser will queue the rest of the requests until one of the previous ones finishes. The way it's implemented, it doesn't slow things down to do more than the max requests, but you don't gain any more parallelism after the browser's limit. If you were running this code from a different Javascript environment such as nodejs, then that limit would not apply as this is a browser-specific thing.
Note, the key thing to achieving the parallelism is launching multiple requests to be in-flight at the same time. There is no requirement that you use Promise.allSettled() before acting on any results unless you need to get all the results in order before you can process the results.
If the results can be processed individually as they finish and can be processed in any order, you can also write the code that way without using Promise.allSettled() such as:
getProperty(1).then(processResult).catch(processErr);
getProperty(2).then(processResult).catch(processErr);
getProperty(3).then(processResult).catch(processErr);
Note: I also don't see any error handling in your code. Any outside network request can fail and you must have some handler for rejected promises.
I'm trying to implement short-term caching in my Angular service -- a bunch of sub-components get created in rapid succession, and each one has an HTTP call. I want to cache them while the page is loading, but not forever.
I've tried the following two methods, neither of which have worked. In both cases, the HTTP URL is hit once for each instance of the component that is created; I want to avoid that -- ideally, the URL would be hit once when the grid is created, then the cache expires and the next time I need to create the component it hits the URL all over again. I pulled both techniques from other threads on StackOverflow.
share() (in service)
getData(id: number): Observable<MyClass[]> {
return this._http.get(this.URL)
.map((response: Response) => <MyClass[]>response.json())
.share();
}
ReplaySubject (in service)
private replaySubject = new ReplaySubject(1, 10000);
getData(id: number): Observable<MyClass[]> {
if (this.replaySubject.observers.length) {
return this.replaySubject;
} else {
return this._http.get(this.URL)
.map((response: Response) => {
let data = <MyClass[]>response.json();
this.replaySubject.next(data);
return data;
});
}
}
Caller (in component)
ngOnInit() {
this.myService.getData(this.id)
.subscribe((resultData: MyClass[]) => {
this.data = resultData;
},
(error: any) => {
alert(error);
});
}
There's really no need to hit the URL each time the component is created -- they return the same data, and in a grid of rows that contain the component, the data will be the same. I could call it once when the grid itself is created, and pass that data into the component. But I want to avoid that, for two reasons: first, the component should be relatively self-sufficient. If I use the component elsewhere, I don't want to the parent component to have to cache data there, too. Second, I want to find a short-term caching pattern that can be applied elsewhere in the application. I'm not the only person working on this, and I want to keep the code clean.
Most importantly, if you want to make something persistent even when creating/destroying Angular components it can't be created in that component but in a service that is shared among your components.
Regarding RxJS, you usually don't have to use ReplaySubject directly and use just publishReplay(1, 10000)->refCount() instead.
The share() operator is just a shorthand for publish()->refCount() that uses Subject internally which means it doesn't replay cached values.
I have an Angular 1 app that I am trying to increase the performance of a particular service that makes a lot of calculations (and probably is not optimized but that's besides the point for now, running it in another thread is the goal right now to increase animation performance)
The App
The app runs calculations on your GPA, Terms, Courses Assignments etc. The service name is calc. Inside Calc there are user, term, course and assign namespaces. Each namespace is an object in the following form
{
//Times for the calculations (for development only)
times:{
//an array of calculation times for logging and average calculation
array: []
//Print out the min, max average and total calculation times
report: function(){...}
},
//Hashes the object (with service.hash()) and checks to see if we have cached calculations for the item, if not calls runAllCalculations()
refresh: function(item){...},
//Runs calculations, saves it in the cache (service.calculations array) and returns the calculation object
runAllCalculations: function(item){...}
}
Here is a screenshot from the very nice structure tab of IntelliJ to help visualization
What Needs To Be Done?
Detect Web Worker Compatibility (MDN)
Build the service depending on Web Worker compatibility
a. Structure it the exact same as it is now
b. Replace with a Web Worker "proxy" (Correct terminology?) service
The Problem
The problem is how to create the Web Worker "Proxy" to maintain the same service behavior from the rest of the code.
Requirements/Wants
A few things that I would like:
Most importantly, as stated above, keep the service behavior unchanged
To keep one code base for the service, keep it DRY, not having to modify two spots. I have looked at WebWorkify for this, but I am unsure how to implement it best.
Use Promises while waiting for the worker to finish
Use Angular and possibly other services inside the worker (if its possible) again WebWorkify seems to address this
The Question
...I guess there hasn't really been a question thus far, it's just been an explanation of the problem...So without further ado...
What is the best way to use an Angular service factory to detect Web Worker compatibility, conditionally implement the service as a Web Worker, while keeping the same service behavior, keeping DRY code and maintaining support for non Web Worker compatible browsers?
Other Notes
I have also looked at VKThread, which may be able to help with my situation, but I am unsure how to implement it the best.
Some more resources:
How to use a Web Worker in AngularJS?
http://www.html5rocks.com/en/tutorials/workers/basics/
https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers#Worker_feature_detection
In general, good way to make a manageable code that works in worker - and especially one that also can run in the same window (eg. when worker is not supported) is to make the code event-driven and then use simple proxy to drive the events through the communication channel - in this case worker.
I first created abstract "class" that didn't really define a way of sending events to the other side.
function EventProxy() {
// Object that will receive events that come from the other side
this.eventSink = null;
// This is just a trick I learned to simulate real OOP for methods that
// are used as callbacks
// It also gives you refference to remove callback
this.eventFromObject = this.eventFromObject.bind(this);
}
// Object get this as all events callback
// typically, you will extract event parameters from "arguments" variable
EventProxy.prototype.eventFromObject = (name)=>{
// This is not implemented. We should have WorkerProxy inherited class.
throw new Error("This is abstract method. Object dispatched an event "+
"but this class doesn't do anything with events.";
}
EventProxy.prototype.setObject = (object)=> {
// If object is already set, remove event listener from old object
if(this.eventSink!=null)
//do it depending on your framework
... something ...
this.eventSink = object;
// Listen on all events. Obviously, your event framework must support this
object.addListener("*", this.eventFromObject);
}
// Child classes will call this when they receive
// events from other side (eg. worker)
EventProxy.prototype.eventReceived = (name, args)=> {
// put event name as first parameter
args.unshift(name);
// Run the event on the object
this.eventSink.dispatchEvent.apply(this.eventSink, args);
}
Then you implement this for worker for example:
function WorkerProxy(worker) {
// call superconstructor
EventProxy.call(this);
// worker
this.worker = worker;
worker.addEventListener("message", this.eventFromWorker = this.eventFromWorker.bind(this));
}
WorkerProxy.prototype = Object.create(EventProxy.prototype);
// Object get this as all events callback
// typically, you will extract event parameters from "arguments" variable
EventProxy.prototype.eventFromObject = (name)=>{
// include event args but skip the first one, the name
var args = [];
args.push.apply(args, arguments);
args.splice(0, 1);
// Send the event to the script in worker
// You could use additional parameter to use different proxies for different objects
this.worker.postMessage({type: "proxyEvent", name:name, arguments:args});
}
EventProxy.prototype.eventFromWorker = (event)=>{
if(event.data.type=="proxyEvent") {
// Use superclass method to handle the event
this.eventReceived(event.data.name, event.data.arguments);
}
}
The usage then would be that you have some service and some interface and in the page code you do:
// Or other proxy type, eg socket.IO, same window, shared worker...
var proxy = new WorkerProxy(new Worker("runServiceInWorker.js"));
//eg user clicks something to start calculation
var interface = new ProgramInterface();
// join them
proxy.setObject(interface);
And in the runServiceInWorker.js you do almost the same:
importScripts("myservice.js", "eventproxy.js");
// Here we're of course really lucky that web worker API is symethric
var proxy = new WorkerProxy(self);
// 1. make a service
// 2. assign to proxy
proxy.setObject(new MyService());
// 3. profit ...
In my experience, eventually sometimes I had to detect on which side am I but that was with web sockets, which are not symmetric (there's server and many clients). You could run into similar problems with shared worker.
You mentioned Promises - I think the approach with promises would be similar, though maybe more complicated as you would need to store the callbacks somewhere and index them by ID of the request. But surely doable, and if you're invoking worker functions from different sources, maybe better.
I am the author of the vkThread plugin which was mentioned in the question. And yes, I developed Angular version of vkThread plugin which allows you to execute a function in a separate thread.
Function can be defined directly in the main thread or called from an external javascript file.
Function can be:
Regular functions
Object's methods
Functions with dependencies
Functions with context
Anonymous functions
Basic usage:
/* function to execute in a thread */
function foo(n, m){
return n + m;
}
// to execute this function in a thread: //
/* create an object, which you pass to vkThread as an argument*/
var param = {
fn: foo // <-- function to execute
args: [1, 2] // <-- arguments for this function
};
/* run thread */
vkThread.exec(param).then(
function (data) {
console.log(data); // <-- thread returns 3
}
);
Examples and API doc: http://www.eslinstructor.net/ng-vkthread/demo/
Hope this helps,
--Vadim
I have been reading about $q and promises for days now and I seem to understand it...somewhat. I have the following situation in practice:
An $http request is made and checks whether a subsequent call can be made.
If the first call fails, return "no data", if it succeeds and says a call can be made, the second call is made, if not - "no data" again. If the second call succeeds, it returns data, if not - "no data". It looks like this (approximately, I simplified for general idea, so don't worry about the minor mistakes here):
return $http.get (something)
.then(function(allowedAccess){
if(allowedAccess){
return $http.get (somethingElse)
.then( function(result){return {data:result} },
function(error){return {data:"n0pe"} }
)
} else {
return {data:"n0pe"}
}
},
function(){ return {data:"n0pe"} });
I was told to use $q here. I don't really understand how or why I would. The $http calls are promises already.
If there is a way to make this cleaner, I don't see it. Just got done re-reading this post on the subject. Essentially, am I missing something / is there a better way to do this?
Edit: Also just re-read a tutorial on chaining promises - it doesn't handle call failures at all. Basically posting this as due diligence.
Edit 2: This is more of an elaborate on the theory I am asking about, excerpt from the first article:
This is a simple example though. It becomes really powerful if your then() callback returns another promise. In that case, the next then() will only be executed once that promise resolves. This pattern can be used for serial HTTP requests, for example (where a request depends on the result of a previous one):
This seems to be talking about chains like this:
asyncFn1(1)
.then(function(data){return asyncFn2(data)})
.then(function(data){return asyncFn3(data)})
So, if I understand correctly this a). Doesn't apply to me because I don't have a 3rd function. b). Would apply to me if I had three functions because while I run an if statement inside the first $http request, and only inside the if statement do I return another promise. So, theoretically, if I had three async functions to chain, I would need to put my if statement inside a promise?
Promises really help with code composition of making async calls. In other words, they allow you to compose your code in a similar manner to how you would compose a synchronous set of calls (with the use of chained .thens) and as if it the sync code was in a try/catch block (with .catch).
So, imagine that your HTTP calls were blocking - the logic you have would look like so:
var allowedAccess, data;
try {
allowedAccess = $http.get(something);
if (allowedAccess){
try{
var result = $http.get(somethingElse);
data = {data: result};
} catch (){
data = {data: "n0pe"};
}
} else {
data = {data: "n0pe"};
}
} catch (){
data = {data: "n0pe"};
}
return data;
You could simplify it a bit:
var allowedAccess, result;
try {
allowedAccess = $http.get(something);
var result;
if (allowedAccess) {
result = $http.get(somethingElse);
} else {
throw;
}
data = {data: result};
} catch () {
data = {data: "n0pe"};
}
return data;
And that would translate to the async version of:
return $http
.get(something)
.then(function(allowedAccess){
if (allowedAccess){
return $http.get(somethingElse);
} else {
return $q.reject(); // this is the "throw;" from above
}
})
.then(function(result){
return {data: result};
})
.catch(function(){
return {data: "n0pe"};
})
At least, this is the reasoning you could apply when composing code with branches and async calls.
I'm not saying that the version I presented is optimal or shorter - it is, however, more DRY because of a single error handling. But just realize that when you do .then(success, error) it is equivalent to try/catch over the previous async operation - this may or may not be needed depending on your specific circumstance.
This is how I would code this sort of problem:
// returns a promise that resolves some endpoint if allowed
function getDataWithAccess(allowed){
return allowed ? $http.get(someEndpoint) : $q.reject();
}
// do something with data
function handleData(data){
// do stuff with your data
}
// main chain
$http.get(accessCredEndpoint)
.then(getDataWithAccess)
.then(handleData)
.catch(function(err){
return { data: "n0pe" };
});
Yes, this is very much like New Dev's answer, however I wanted to make a point of extracting the functions into their own blocks. This makes the overall code much more readable.
$q will help reduce pyramid of calls like this:
async-call1.then(...
aysnc-call2.then(...
This blog post - http://chariotsolutions.com/blog/post/angularjs-corner-using-promises-q-handle-asynchronous-calls/ - offers a clean way of making multiple HTTP requests. Notice the cleaner approach using $q. In case you were hitting a single HTTP endpoint, using your method would have been just fine. I'd say, what you have is fine also; $q might allow greater flexibility in the future.
The blog post describes a service while using $q and the code looks cleaner.
service('asyncService', function($http, $q) {
return {
loadDataFromUrls: function(urls) {
var deferred = $q.defer();
var urlCalls = [];
angular.forEach(urls, function(url) {
urlCalls.push($http.get(url.url));
});
// they may, in fact, all be done, but this
// executes the callbacks in then, once they are
// completely finished.
$q.all(urlCalls)
.then(...
I am a beginner with promises also, so take this with a grain of salt.
By following the AngularFire guide, I have synchronized a scope variable with a Firebase array. My code is basically the same as the tutorial (Step 5):
https://www.firebase.com/docs/web/libraries/angular/quickstart.html
Everything in my app is working, but I am really confused about how to properly mock the Firebase call in my Karma unit tests. I guess something along the lines of using $provide to mock the data? But then $add wouldn't work in my controller methods. Help?
Copied from this gist which discusses this topic at length.
The discussion is all inlined in the comments. There is a lot to consider here.
What goes wrong with mocks
Consider this data structure, used for getting a list of names based on membership
/users/<user id>/name
/rooms/members/<user id>/true
Now let's create a couple simple classes without any real consideration for testing structures, assuming we'll use a mock of Firebase to test them. Note that I see these sorts of errors constantly in the wild; this example is not far fetched or exaggerated (it's rather tame in comparison)
class User {
// Accepts a Firebase snapshot to create the user instance
constructor(snapshot) {
this.id = snapshot.key;
this.name = snapshot.val().name;
}
getName() { return this.name; }
// Load a user based on their id
static load(id) {
return firebase.database().ref('users').child(uid)
.once('value').then(snap => new User(snap));
}
}
class MembersList {
// construct a list of members from a Firebase ref
constructor(memberListRef) {
this.users = [];
// This coupling to the Firebase SDK and the nuances of how realtime is handled will
// make our tests pretty difficult later.
this.ref = memberListRef;
this.ref.on('child_added', this._addUser, this);
}
// Get a list of the member names
// Assume we need this for UI methods that accept an array, so it can't be async
//
// It may not be obvious that we've introduced an odd coupling here, since we
// need to know that this is loaded asynchronously before we can use it.
getNames() {
return this.users.map(user => user.getName());
}
// So this kind of stuff shows up incessantly when we couple Firebase into classes and
// it has a big impact on unit testing (shown below)
ready() {
// note that we can't just use this.ref.once() here, because we have to wait for all the
// individual user objects to be loaded, adding another coupling on the User class's internal design :(
return Promise.all(this.promises);
}
// Asynchronously find the user based on the uid
_addUser(memberSnap) {
let promise = User.load(memberSnap.key).then(user => this.users.push(user));
// note that this weird coupling is needed so that we can know when the list of users is available
this.promises.push(promise);
}
destroy() {
this.ref.off('child_added', this._addUser, this);
}
}
/*****
Okay, now on to the actual unit test for list names.
****/
const mockRef = mockFirebase.database(/** some sort of mock data */).ref();
// Note how much coupling and code (i.e. bugs and timing issues we might introduce) is needed
// to make this work, even with a mock
function testGetNames() {
const memberList = new MemberList(mockRef);
// We need to abstract the correct list of names from the DB, so we need to reconstruct
// the internal queries used by the MemberList and User classes (more opportunities for bugs
// and def. not keeping our unit tests simple)
//
// One important note here is that our test unit has introduced a side effect. It has actually cached the data
// locally from Firebase (assuming our mock works like the real thing; if it doesn't we have other problems)
// and may inadvertently change the timing of async/sync events and therefore the results of the test!
mockRef.child('users').once('value').then(userListSnap => {
const userNamesExpected = [];
userListSnap.forEach(userSnap => userNamesExpected.push(userSnap.val().name));
// Okay, so now we're ready to test our method for getting a list of names
// Note how we can't just test the getNames() method, we also have to rely on .ready()
// here, breaking our effective isolation of a single point of logic.
memberList.ready().then(() => assertEqual(memberList.getNames(), userNamesExpected));
// Another really important note here: We need to call .off() on the Firebase
// listeners, or other test units will fail in weird ways, since callbacks here will continue
// to get invoked when the underlying data changes.
//
// But we'll likely introduce an unexpected bug here. If assertEqual() throws, which many testing
// libs do, then we won't reach this code! Yet another strange, intermittent failure point in our tests
// that will take forever to isolate and fix. This happens frequently; I've been a victim of this bug. :(
memberList.destroy(); // (or maybe something like mockFirebase.cleanUpConnections()
});
}
A better approach through proper encapsulation and TDD
Okay, now let's reverse this by starting with an effective test unit design and see if we can design our classes with less coupling to accommodate.
function testGetNames() {
const userList = new UserList();
userList.add( new User('kato', 'Kato Richardson') );
userList.add( new User('chuck', 'Chuck Norris') );
assertEqual( userList.getNames(), ['Kato Richardson', 'Chuck Norris']);
// Ah, this is looking good! No complexities, no async madness. No chance of bugs in my unit test!
}
/**
Note how our classes will be simpler and better designed just by using a good TDD
*/
class User {
constructor(userId, name) {
this.id = userId;
this.name = name;
}
getName() {
return this.name;
}
}
class UserList {
constructor() {
this.users = [];
}
getNames() {
return this.users.map(user => user.getName());
}
addUser(user) {
this.users.push(user);
}
// note how we don't need .destroy() and .ready() methods here just to know
// when the user list is resolved, yay!
}
// This all looks great and wonderful, and the tests are going to run wonderfully.
// But how do we populate the list from Firebase now?? The answer is an isolated
// service that handles this.
class MemberListManager {
constructor( memberListRef ) {
this.ref = memberListRef;
this.ref.on('child_added', this._addUser, this);
this.userList = new UserList();
}
getUserList() {
return this.userList;
}
_addUser(snap) {
const user = new User(snap.key, snap.val().name);
this.userList.push(user);
}
destroy() {
this.ref.off('child_added', this._addUser, this);
}
}
// But now we need to test MemberListManager, too, right? And wouldn't a mock help here? Possibly. Yes and no.
//
// More importantly, it's just one small service that deals with the async and external libs.
// We don't have to depend on mocking Firebase to do this either. Mocking the parts used in isolation
// is much, much simpler than trying to deal with coupled third party dependencies across classes.
//
// Additionally, it's often better to move third party calls like these
// into end-to-end tests instead of unit tests (since we are actually testing
// across third party libs and not just isolated logic)
//
// For more alternatives to a mock sdk, check out AngularFire.
// We often just used the real Firebase Database with set() or push():
// https://github.com/firebase/angularfire/blob/master/tests/unit/FirebaseObject.spec.js#L804
//
// An rely on spies to stub some fake snapshots or refs with results we want, which is much simpler than
// trying to coax a mock or SDK to create error conditions or specific outputs:
// https://github.com/firebase/angularfire/blob/master/tests/unit/FirebaseObject.spec.js#L344