Angular Service and Web Workers - angularjs

I have an Angular 1 app that I am trying to increase the performance of a particular service that makes a lot of calculations (and probably is not optimized but that's besides the point for now, running it in another thread is the goal right now to increase animation performance)
The App
The app runs calculations on your GPA, Terms, Courses Assignments etc. The service name is calc. Inside Calc there are user, term, course and assign namespaces. Each namespace is an object in the following form
{
//Times for the calculations (for development only)
times:{
//an array of calculation times for logging and average calculation
array: []
//Print out the min, max average and total calculation times
report: function(){...}
},
//Hashes the object (with service.hash()) and checks to see if we have cached calculations for the item, if not calls runAllCalculations()
refresh: function(item){...},
//Runs calculations, saves it in the cache (service.calculations array) and returns the calculation object
runAllCalculations: function(item){...}
}
Here is a screenshot from the very nice structure tab of IntelliJ to help visualization
What Needs To Be Done?
Detect Web Worker Compatibility (MDN)
Build the service depending on Web Worker compatibility
a. Structure it the exact same as it is now
b. Replace with a Web Worker "proxy" (Correct terminology?) service
The Problem
The problem is how to create the Web Worker "Proxy" to maintain the same service behavior from the rest of the code.
Requirements/Wants
A few things that I would like:
Most importantly, as stated above, keep the service behavior unchanged
To keep one code base for the service, keep it DRY, not having to modify two spots. I have looked at WebWorkify for this, but I am unsure how to implement it best.
Use Promises while waiting for the worker to finish
Use Angular and possibly other services inside the worker (if its possible) again WebWorkify seems to address this
The Question
...I guess there hasn't really been a question thus far, it's just been an explanation of the problem...So without further ado...
What is the best way to use an Angular service factory to detect Web Worker compatibility, conditionally implement the service as a Web Worker, while keeping the same service behavior, keeping DRY code and maintaining support for non Web Worker compatible browsers?
Other Notes
I have also looked at VKThread, which may be able to help with my situation, but I am unsure how to implement it the best.
Some more resources:
How to use a Web Worker in AngularJS?
http://www.html5rocks.com/en/tutorials/workers/basics/
https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers#Worker_feature_detection

In general, good way to make a manageable code that works in worker - and especially one that also can run in the same window (eg. when worker is not supported) is to make the code event-driven and then use simple proxy to drive the events through the communication channel - in this case worker.
I first created abstract "class" that didn't really define a way of sending events to the other side.
function EventProxy() {
// Object that will receive events that come from the other side
this.eventSink = null;
// This is just a trick I learned to simulate real OOP for methods that
// are used as callbacks
// It also gives you refference to remove callback
this.eventFromObject = this.eventFromObject.bind(this);
}
// Object get this as all events callback
// typically, you will extract event parameters from "arguments" variable
EventProxy.prototype.eventFromObject = (name)=>{
// This is not implemented. We should have WorkerProxy inherited class.
throw new Error("This is abstract method. Object dispatched an event "+
"but this class doesn't do anything with events.";
}
EventProxy.prototype.setObject = (object)=> {
// If object is already set, remove event listener from old object
if(this.eventSink!=null)
//do it depending on your framework
... something ...
this.eventSink = object;
// Listen on all events. Obviously, your event framework must support this
object.addListener("*", this.eventFromObject);
}
// Child classes will call this when they receive
// events from other side (eg. worker)
EventProxy.prototype.eventReceived = (name, args)=> {
// put event name as first parameter
args.unshift(name);
// Run the event on the object
this.eventSink.dispatchEvent.apply(this.eventSink, args);
}
Then you implement this for worker for example:
function WorkerProxy(worker) {
// call superconstructor
EventProxy.call(this);
// worker
this.worker = worker;
worker.addEventListener("message", this.eventFromWorker = this.eventFromWorker.bind(this));
}
WorkerProxy.prototype = Object.create(EventProxy.prototype);
// Object get this as all events callback
// typically, you will extract event parameters from "arguments" variable
EventProxy.prototype.eventFromObject = (name)=>{
// include event args but skip the first one, the name
var args = [];
args.push.apply(args, arguments);
args.splice(0, 1);
// Send the event to the script in worker
// You could use additional parameter to use different proxies for different objects
this.worker.postMessage({type: "proxyEvent", name:name, arguments:args});
}
EventProxy.prototype.eventFromWorker = (event)=>{
if(event.data.type=="proxyEvent") {
// Use superclass method to handle the event
this.eventReceived(event.data.name, event.data.arguments);
}
}
The usage then would be that you have some service and some interface and in the page code you do:
// Or other proxy type, eg socket.IO, same window, shared worker...
var proxy = new WorkerProxy(new Worker("runServiceInWorker.js"));
//eg user clicks something to start calculation
var interface = new ProgramInterface();
// join them
proxy.setObject(interface);
And in the runServiceInWorker.js you do almost the same:
importScripts("myservice.js", "eventproxy.js");
// Here we're of course really lucky that web worker API is symethric
var proxy = new WorkerProxy(self);
// 1. make a service
// 2. assign to proxy
proxy.setObject(new MyService());
// 3. profit ...
In my experience, eventually sometimes I had to detect on which side am I but that was with web sockets, which are not symmetric (there's server and many clients). You could run into similar problems with shared worker.
You mentioned Promises - I think the approach with promises would be similar, though maybe more complicated as you would need to store the callbacks somewhere and index them by ID of the request. But surely doable, and if you're invoking worker functions from different sources, maybe better.

I am the author of the vkThread plugin which was mentioned in the question. And yes, I developed Angular version of vkThread plugin which allows you to execute a function in a separate thread.
Function can be defined directly in the main thread or called from an external javascript file.
Function can be:
Regular functions
Object's methods
Functions with dependencies
Functions with context
Anonymous functions
Basic usage:
/* function to execute in a thread */
function foo(n, m){
return n + m;
}
// to execute this function in a thread: //
/* create an object, which you pass to vkThread as an argument*/
var param = {
fn: foo // <-- function to execute
args: [1, 2] // <-- arguments for this function
};
/* run thread */
vkThread.exec(param).then(
function (data) {
console.log(data); // <-- thread returns 3
}
);
Examples and API doc: http://www.eslinstructor.net/ng-vkthread/demo/
Hope this helps,
--Vadim

Related

Akka.net - Streams with parallelism, backpressure and ActorRef

Tying to learn how use Akka.net Streams to process items in parallel from a Source.Queue, with the processing done in an Actor.
I've been able to get it to work with calling a function with Sink.ForEachParallel, and it works as expected.
Is it possible to process items in parallel with Sink.ActorRefWithAck (as I would prefer it utilize back-pressure)?
About to press Post, when tried to combine previous attempts and viola!
Previous attempts with ForEachParallel failed when I tried to create the actor within, but couldn't do so in an async function. If I use an single actor previous declared, then the Tell would work, but I couldn't get the parallelism I desired.
I got it to work with a router with roundrobin configuration.
var props = new RoundRobinPool(5).Props(Props.Create<MyActor>());
var actor = Context.ActorOf(props);
flow = Source.Queue<Element>(2000,OverflowStrategy.Backpressure)
.Select(x => {
return new Wrapper() { Element = x, Request = ++cnt };
})
.To(Sink.ForEachParallel<Wrapper>(5, (s) => { actor.Tell(s); }))
.Run(materializer);
The Request ++cnt is for console output to verify the requests are being processed as desired.
MyActor has a long delay on every 10th request to verify the backpressure was working.

angularjs chain http post sequentially

In my application, I am storing data in local storage and trigger async http post in the background. Once successfully posted, the posted data gets removed from local storage. When http post is in process, there may be more data added to local storage so I need to queue up the post and sequentially process it because, I need to wait for the local storage to be cleared from the successful posts. The task should be called recursively until there is data in local storage.
taskQueue: function () {
var deferred = $q.defer();
queue.push(deferred);
var promise = deferred.promise;
if (!saveInProgress) {
// get post data from storage
var promises = queue.map(function(){$http.post(<post url>, <post data>).then(function(result){
// clear successful data
deferred.resolve(result);
}, function(error){
deferred.reject(error);
})
})
return $q.all(promises);
}
As angular newbie, I am having problems with the above code not being sequential. How can I achieve what I intend to? The queue length is unknown and also the queue length increases as the process is in progress. Please point me to other answers if this is a duplicate.
Async.js sounds a good idea but if you don't want to use a library ...
$q.all will batch up an array of requests and fire them at the same time and will resolve when ALL promises in the array resolve - WHICH IS NOT WHAT YOU WANT.
to make $http calls SEQUENTIALLY from an array do this ....
var request0 = $http.get('/my/path0');
var request1 = $http.post('/my/path1', {data:'fred'});
var request2 = $http.get('/my/path2');
var requestArray = [];
then ...
requestArray.push(request0);
requestArray.push(request1);
requestArray.push(request2);
then ...
requestArray[0].then(function(response0) {
// do something with response0
return requestArray[1];
}).then(function(response1) {
// do something with response1
return requestArray[2];
}).then(function(response2) {
// do something with response2
}).catch(function(failedResponse) {
console.log("i will be displayed when a request fails (if ever)", failedResponse)
});
While having a library solution would be great (per #nstoitsev's answer), you can do this without it.
sequential requests of unknown length
Just to recap:
we do not know the number of requests
each response may enqueue another request
A few assumptions:
all requests will be working on a common data object (local storage in your case)
all requests are promises
running the queue
function postMyData (data){
return $http.post(<url>, data)
}
var rqsts = []
function executeQueue (){
if(!rqsts.length)
//we're done
return
var rqst = rqsts.shift()
rqst()
.then(function(rsp){
//based on how you're determining if you need to do another request...
if(keepGoing)
rqsts.push(postMyData(<more data>))
})
}
codepen - http://codepen.io/jusopi/pen/VaYRXR?editors=1010
I intentionally left this vague because I don't understand what the conditions for failure are and if you wanted to vary up the requests to use more than the same $http.post call, you could pass it back in some way.
and just a suggestion
As angular newbie...
Many things are progressing towards this whole functional, reactive programming paradigm. Since you're relatively new to Angular and NG2 already has some of this built in, it might be worthy of your attention. I think rxjs is already in many NG2 example bundles.
The easies way to achieve this is by using Async.js. There you can find a method called mapSeries. You can run it over the queue and it will sequentially process all elements of the array one by one, and will continue to the next element only when the correct callback is called.

Block or extend Breeze metadata auto fetching

Current application - Angular application with Breeze. Application has ~7 entity managers and different data domains (metadata). When application runs we trying to fetch entity managers, like:
app.run(['$rootScope', 'datacontext1', ... ], function($rootScope, datacontext1, ...) {
datacontext1.loadMetadata();
...
datacontext7.loadMetadata();
}
Every datacontext has its own entity manager and loadMetadata is:
function loadMetadata() {
manager.fetchMetadata().then(function(mdata) {
if (mdata === 'already fetched') {
return;
}
...
applyCustomMetadata(); // Do some custom job with metadata/entity types
});
}
Metadata comes from server asynchronously. Few module has really big metadata, like 200Kb and takes some time for loading and apply to entity manager. Its possible that first Breeze data request executed in same entity manager will be started before this loadMetadata operation finished and as I understand Breeze automatically fetch metadata again. Usually its not a problem, Metadata end point cached on server, but sometimes it produces very strange behavior of Breeze - EntityManager.fetchMetadata resolve promise as "already fetched" and in this case applyCustomMetadata() operation can not be executed.
As I understand problem is inside Breeze and approach its used to resolve metadata promise (seems to be http adapter is singleton and second request override metadata with "already fetched" string and applyCustomMetadata() operation never executes).
Need to figure out some way to resolve issue without significant changes in application.
Logically need to delay full application from using entity managers while loadMetadata done. Looking for any way on Breeze level to disable auto fetch metadata if its already in progress (but not interrupt request, just wait and try again after some time). Any other ideas are fine as well.
Why are you allowing queries to execute before the metadata is loaded? Therein lies your problem.
I have an application bootstrapper that I expose through a global variable; none of my application activities depending on the Entity Manager are started until preliminary processes complete:
var bootstrapper = {
pageReady: ko.observable(false)
};
initBootstrapper();
return bootstrapper;
function initBootstrapper() {
window.MyApp.entityManagerProvider.initialize() // load metadata, lookups, etc
.then(function () {
window.MyApp.router.initialize(); // setup page routes, home ViewModel, etc
bootstrapper.pageReady(true); // show homepage
});
};
Additionally, depending on the frequency of database changes occurring in your organization, you may wish to deliver the metadata to the client synchronously on page_load. See this documentation for further details:
http://breeze.github.io/doc-js/metadata-load-from-script.html

How to mock out Firebase in Karma tests for Angular app

By following the AngularFire guide, I have synchronized a scope variable with a Firebase array. My code is basically the same as the tutorial (Step 5):
https://www.firebase.com/docs/web/libraries/angular/quickstart.html
Everything in my app is working, but I am really confused about how to properly mock the Firebase call in my Karma unit tests. I guess something along the lines of using $provide to mock the data? But then $add wouldn't work in my controller methods. Help?
Copied from this gist which discusses this topic at length.
The discussion is all inlined in the comments. There is a lot to consider here.
What goes wrong with mocks
Consider this data structure, used for getting a list of names based on membership
/users/<user id>/name
/rooms/members/<user id>/true
Now let's create a couple simple classes without any real consideration for testing structures, assuming we'll use a mock of Firebase to test them. Note that I see these sorts of errors constantly in the wild; this example is not far fetched or exaggerated (it's rather tame in comparison)
class User {
// Accepts a Firebase snapshot to create the user instance
constructor(snapshot) {
this.id = snapshot.key;
this.name = snapshot.val().name;
}
getName() { return this.name; }
// Load a user based on their id
static load(id) {
return firebase.database().ref('users').child(uid)
.once('value').then(snap => new User(snap));
}
}
class MembersList {
// construct a list of members from a Firebase ref
constructor(memberListRef) {
this.users = [];
// This coupling to the Firebase SDK and the nuances of how realtime is handled will
// make our tests pretty difficult later.
this.ref = memberListRef;
this.ref.on('child_added', this._addUser, this);
}
// Get a list of the member names
// Assume we need this for UI methods that accept an array, so it can't be async
//
// It may not be obvious that we've introduced an odd coupling here, since we
// need to know that this is loaded asynchronously before we can use it.
getNames() {
return this.users.map(user => user.getName());
}
// So this kind of stuff shows up incessantly when we couple Firebase into classes and
// it has a big impact on unit testing (shown below)
ready() {
// note that we can't just use this.ref.once() here, because we have to wait for all the
// individual user objects to be loaded, adding another coupling on the User class's internal design :(
return Promise.all(this.promises);
}
// Asynchronously find the user based on the uid
_addUser(memberSnap) {
let promise = User.load(memberSnap.key).then(user => this.users.push(user));
// note that this weird coupling is needed so that we can know when the list of users is available
this.promises.push(promise);
}
destroy() {
this.ref.off('child_added', this._addUser, this);
}
}
/*****
Okay, now on to the actual unit test for list names.
****/
const mockRef = mockFirebase.database(/** some sort of mock data */).ref();
// Note how much coupling and code (i.e. bugs and timing issues we might introduce) is needed
// to make this work, even with a mock
function testGetNames() {
const memberList = new MemberList(mockRef);
// We need to abstract the correct list of names from the DB, so we need to reconstruct
// the internal queries used by the MemberList and User classes (more opportunities for bugs
// and def. not keeping our unit tests simple)
//
// One important note here is that our test unit has introduced a side effect. It has actually cached the data
// locally from Firebase (assuming our mock works like the real thing; if it doesn't we have other problems)
// and may inadvertently change the timing of async/sync events and therefore the results of the test!
mockRef.child('users').once('value').then(userListSnap => {
const userNamesExpected = [];
userListSnap.forEach(userSnap => userNamesExpected.push(userSnap.val().name));
// Okay, so now we're ready to test our method for getting a list of names
// Note how we can't just test the getNames() method, we also have to rely on .ready()
// here, breaking our effective isolation of a single point of logic.
memberList.ready().then(() => assertEqual(memberList.getNames(), userNamesExpected));
// Another really important note here: We need to call .off() on the Firebase
// listeners, or other test units will fail in weird ways, since callbacks here will continue
// to get invoked when the underlying data changes.
//
// But we'll likely introduce an unexpected bug here. If assertEqual() throws, which many testing
// libs do, then we won't reach this code! Yet another strange, intermittent failure point in our tests
// that will take forever to isolate and fix. This happens frequently; I've been a victim of this bug. :(
memberList.destroy(); // (or maybe something like mockFirebase.cleanUpConnections()
});
}
A better approach through proper encapsulation and TDD
Okay, now let's reverse this by starting with an effective test unit design and see if we can design our classes with less coupling to accommodate.
function testGetNames() {
const userList = new UserList();
userList.add( new User('kato', 'Kato Richardson') );
userList.add( new User('chuck', 'Chuck Norris') );
assertEqual( userList.getNames(), ['Kato Richardson', 'Chuck Norris']);
// Ah, this is looking good! No complexities, no async madness. No chance of bugs in my unit test!
}
/**
Note how our classes will be simpler and better designed just by using a good TDD
*/
class User {
constructor(userId, name) {
this.id = userId;
this.name = name;
}
getName() {
return this.name;
}
}
class UserList {
constructor() {
this.users = [];
}
getNames() {
return this.users.map(user => user.getName());
}
addUser(user) {
this.users.push(user);
}
// note how we don't need .destroy() and .ready() methods here just to know
// when the user list is resolved, yay!
}
// This all looks great and wonderful, and the tests are going to run wonderfully.
// But how do we populate the list from Firebase now?? The answer is an isolated
// service that handles this.
class MemberListManager {
constructor( memberListRef ) {
this.ref = memberListRef;
this.ref.on('child_added', this._addUser, this);
this.userList = new UserList();
}
getUserList() {
return this.userList;
}
_addUser(snap) {
const user = new User(snap.key, snap.val().name);
this.userList.push(user);
}
destroy() {
this.ref.off('child_added', this._addUser, this);
}
}
// But now we need to test MemberListManager, too, right? And wouldn't a mock help here? Possibly. Yes and no.
//
// More importantly, it's just one small service that deals with the async and external libs.
// We don't have to depend on mocking Firebase to do this either. Mocking the parts used in isolation
// is much, much simpler than trying to deal with coupled third party dependencies across classes.
//
// Additionally, it's often better to move third party calls like these
// into end-to-end tests instead of unit tests (since we are actually testing
// across third party libs and not just isolated logic)
//
// For more alternatives to a mock sdk, check out AngularFire.
// We often just used the real Firebase Database with set() or push():
// https://github.com/firebase/angularfire/blob/master/tests/unit/FirebaseObject.spec.js#L804
//
// An rely on spies to stub some fake snapshots or refs with results we want, which is much simpler than
// trying to coax a mock or SDK to create error conditions or specific outputs:
// https://github.com/firebase/angularfire/blob/master/tests/unit/FirebaseObject.spec.js#L344

Calling Service's method multiple times holds single data instance

I am using services for my AngularJS project and trying to call a service.method using a for loop, like this :
for( key in URLs) {
Service.fetchXML(key);
}
Service description:
[..]
fetchXML : function(resource) {
var prop = SomeVar[resource]; //SomeVar is declared within the service
$.get('xmlURL.xml?resource='+prop, function(data) {
//adds data to the IndexedDB after properly parsing it.
console.log(resource)
dbAdd();
})
Problem is when I try resource inside fetchXML() method; its set permanently, means if the loop runs for five times, only one instance of fetchXML() is created and console.log(resource) returns the same for all five iterations.
Please tell me what am I doing wrong here.
for( key in URLs) {
Service.fetchXML();
}
Should be passing parameter to function since it is used as resource to create prop.
for( key in URLs) {
Service.fetchXML(key);
}
This should have been fairly easy to troubleshoot. First it would be apparent in the request url inspected in browser console/dev tools.
Also using some simple degugger or console.log() statements in function would have helped. Or setting breakpoint on the function and stepping through it to see variable values

Resources