I'm currently working on a project that requires me to make an API call. It only allows me to make 500 requests / 10 mins but the data returned (object with ~800 properties) only changes every few months so I rather just cache it somewhere.
I'm very new to this whole thing and I'm wondering how can I make the call every few months and store the data somewhere so that I could retrieve it from the client whenever needed?
Thanks in advance!
Since you want to store your object for a longer period of time, I would suggest writing it to disk rather than caching it in memory (in case your node app crashes).
You didn't mention it precisely, but I assume you are referring to a simple javascript object, which you want to store? To store such an object to disk, you can do the following:
var fs = require("fs");
// with your object being stored in the variable "myObject", after your API call:
var myObject = ....
fs.writeFile( "myFilename.json", JSON.stringify(myObject), "utf8", function(err) {
if(err) {
return console.log(err);
}
// do whatever you want to do after file has been saved...
});
To read the object from disk, simply do:
myObject = require("./filename.json");
Related
I have a typical ngrx-data arrangement of 'User' entities linked to db.
I implement the standard service to handle the data:
#Injectable({providedIn: 'root'})
export class UserService extends EntityCollectionServiceBase<UserEntity> {
constructor(serviceElementsFactory: EntityCollectionServiceElementsFactory) {
super('User', serviceElementsFactory);
}
}
I read the data using:
this.data$ = this.userService.getAll();
this.data$.subscribe(d => { this.data = d; ... }
Data arrives fine. Now, I have a GUI / HTML form where user can make changes and update them. It also works fine. Any changes user makes in the form are updated via:
this.data[fieldName] = newValue;
This updates the data and ngrx-data automatically updates the entity cache.
I want to implement an option, where user can decide to cancel all changes before they are written to the db, and get the initial data before he made any adjustments. However, I am somehow unable to overwrite the cached changes.
I tried:
this.userService.clearCache();
this.userService.load();
also tried to re-call:
this.data$ = this.userService.getAll();
but I am constantly getting the data from the cache that has been changed by the user, not the data from the db. In the db I see the data not modified. No steps were taken to write the data to db.
I am not able to find the approach to discard my entity cache and reload the original db data to replace the cached values.
Any input is appreciated.
You will need to subscribe to the reassigned observable when you change this.data$, but it will be a bit messy.
First you bind this.data$ via this.data$ = this.userService.entities$, then no matter you use load() or getAll(), as long as the entities$ changed, it fire to this.data$.subscribe(). You can even skip the load and getAll if you already did that in other process.
You can then use the clearCache() then load() to reset the cache.
But I strongly recommand you to keep the entity data pure. If the user exit in the middle without save or reset, the data is changed everywhere you use this entity.
My recommand alternatives:
(1) Use angular FormGroup to set the form data with entity data value, then use this setting function to reset the form.
(2) Make a function to copy the data, then use this copy function as reset.
For example using _.cloneDeep.
(2.1) Using rxjs BehaviourSubject:
resetTrigger$ = new BehaviourSubject<boolean>(false);
ngOnInit(){
this.data$ = combineLastest([
this.resetTrigger$,
this.userService.entities$
]).subscribe([trigger, data])=>{
this.data = _.cloneDeep(data)
});
// can skip if already loaded before
this.userService.load();
}
When you want to reset the data, set a new value to the trigger
resetForm(){
this.resetTrigger$.next(!this.resetTrigger$.value)
}
(2.2) Using native function (need to store the original):
this.data$ = this.userService.entities$.pipe(
tap(d=>{
this.originData = d;
resetForm();
})
).subscribe()
resetForm:
resetForm:()=>{
this.data = _.cloneDeep(this.originData);
}
I would like to load a local file (client-side) with papaparse into my React application. Unfortunately, it only loads the first chunk but never the whole file. My file contains about 500 rows and there are never more than 300 rows loaded. It seems like the complete functions is already called after the first chunk.
Since I need to navigate to another page when the file is loaded completely, this is bothering me since I need the complete file for further functions.
The code I use at the moment:
async getData() {
const self = this;
let dataList = [];
Papa.parse(await this.fetchCsv(),
{
delimiter: ',',
header: true,
chunk: function (result, parser) {
parser.pause();
dataList = dataList.concat(result.data)
parser.resume();
},
complete: function () {
self.updateData(dataList);
}
});
}
async fetchCsv() {
const response = await fetch(this.props.location.state.filename);
const reader = response.body.getReader();
const result = await reader.read();
const decoder = new TextDecoder('utf-8');
return decoder.decode(result.value);
}
What I've also tried is using step instead of chunk but this did not change anything.
Can anyone tell me what I'm doing wrong here and why papaparse does not load the whole file?
You may be able to let papaparse do more. It can read local File or stream data from a remote server.
If you only have about 500 records, you may not need to add the complexity associated with streaming. This is especially true if you're just accumulating the data (which it appears you are). Use streaming primarily to process the data 1 record at a time.
If you want to stream, I'd recommend using the "step" callback instead of the "chunk" callback so you can process each row of data.
If you use the step or chunk callbacks, then you don't need the complete callback. If it's called, it won't have the data.
Okay. I'm kinda new to react and I'm having a #1 mayor issue. Can't really find any solution out there.
I've built an app that renders a list of objects. The list comes from my mock API for now. The list of objects is stored inside a store. The store action to fetch the objects is done by the components.
My issue is when showing these objects. When a user clicks show, it renders a page with details on the object. Store-wise this means firing a getSpecific function that retrieves the object, from the store, based on an ID.
This is all fine, the store still has the objects. Until I reload the page. That is when the store gets wiped, a new instance is created (this is my guess). The store is now empty, and getting that specific object is now impossible (in my current implementation).
So, I read somewhere that this is by design. Is the solutions to:
Save the store in local storage, to keep the data?
Make the API call again and get all the objects once again?
And in case 2, when/where is this supposed to happen?
How should a store make sure it always has the expected data?
Any hints?
Some if the implementation:
//List.js
componentDidMount() {
//The fetch offers function will trigger a change event
//which will trigger the listener in componentWillMount
OfferActions.fetchOffers();
}
componentWillMount() {
//Listen for changes in the store
offerStore.addChangeListener(this.retriveOffers);
}
retrieveOffers() {
this.setState({
offers: offerStore.getAll()
});
}
.
//OfferActions.js
fetchOffers(){
let url = 'http://localhost:3001/offers';
axios.get(url).then(function (data) {
dispatch({
actionType: OfferConstants.RECIVE_OFFERS,
payload: data.data
});
});
}
.
//OfferStore.js
var _offers = [];
receiveOffers(payload) {
_offers = payload || [];
this.emitChange();
}
handleActions(action) {
switch (action.actionType) {
case OfferConstants.RECIVE_OFFERS:
{
this.receiveOffers(action.payload);
}
}
}
getAll() {
return _offers;
}
getOffer(requested_id) {
var result = this.getAll().filter(function (offer) {
return offer.id == requested_id;
});
}
.
//Show.js
componentWillMount() {
this.state = {
offer: offerStore.getOffer(this.props.params.id)
};
}
That is correct, redux stores, like any other javascript objects, do not survive a refresh. During a refresh you are resetting the memory of the browser window.
Both of your approaches would work, however I would suggest the following:
Save to local storage only information that is semi persistent such as authentication token, user first name/last name, ui settings, etc.
During app start (or component load), load any auxiliary information such as sales figures, message feeds, and offers. This information generally changes quickly and it makes little sense to cache it in local storage.
For 1. you can utilize the redux-persist middleware. It let's you save to and retrieve from your browser's local storage during app start. (This is just one of many ways to accomplish this).
For 2. your approach makes sense. Load the required data on componentWillMount asynchronously.
Furthermore, regarding being "up-to-date" with data: this entirely depends on your application needs. A few ideas to help you get started exploring your problem domain:
With each request to get offers, also send or save a time stamp. Have the application decide when a time stamp is "too old" and request again.
Implement real time communication, for example socket.io which pushes the data to the client instead of the client requesting it.
Request the data at an interval suitable to your application. You could pass along the last time you requested the information and the server could decide if there is new data available or return an empty response in which case you display the existing data.
In my application, I am storing data in local storage and trigger async http post in the background. Once successfully posted, the posted data gets removed from local storage. When http post is in process, there may be more data added to local storage so I need to queue up the post and sequentially process it because, I need to wait for the local storage to be cleared from the successful posts. The task should be called recursively until there is data in local storage.
taskQueue: function () {
var deferred = $q.defer();
queue.push(deferred);
var promise = deferred.promise;
if (!saveInProgress) {
// get post data from storage
var promises = queue.map(function(){$http.post(<post url>, <post data>).then(function(result){
// clear successful data
deferred.resolve(result);
}, function(error){
deferred.reject(error);
})
})
return $q.all(promises);
}
As angular newbie, I am having problems with the above code not being sequential. How can I achieve what I intend to? The queue length is unknown and also the queue length increases as the process is in progress. Please point me to other answers if this is a duplicate.
Async.js sounds a good idea but if you don't want to use a library ...
$q.all will batch up an array of requests and fire them at the same time and will resolve when ALL promises in the array resolve - WHICH IS NOT WHAT YOU WANT.
to make $http calls SEQUENTIALLY from an array do this ....
var request0 = $http.get('/my/path0');
var request1 = $http.post('/my/path1', {data:'fred'});
var request2 = $http.get('/my/path2');
var requestArray = [];
then ...
requestArray.push(request0);
requestArray.push(request1);
requestArray.push(request2);
then ...
requestArray[0].then(function(response0) {
// do something with response0
return requestArray[1];
}).then(function(response1) {
// do something with response1
return requestArray[2];
}).then(function(response2) {
// do something with response2
}).catch(function(failedResponse) {
console.log("i will be displayed when a request fails (if ever)", failedResponse)
});
While having a library solution would be great (per #nstoitsev's answer), you can do this without it.
sequential requests of unknown length
Just to recap:
we do not know the number of requests
each response may enqueue another request
A few assumptions:
all requests will be working on a common data object (local storage in your case)
all requests are promises
running the queue
function postMyData (data){
return $http.post(<url>, data)
}
var rqsts = []
function executeQueue (){
if(!rqsts.length)
//we're done
return
var rqst = rqsts.shift()
rqst()
.then(function(rsp){
//based on how you're determining if you need to do another request...
if(keepGoing)
rqsts.push(postMyData(<more data>))
})
}
codepen - http://codepen.io/jusopi/pen/VaYRXR?editors=1010
I intentionally left this vague because I don't understand what the conditions for failure are and if you wanted to vary up the requests to use more than the same $http.post call, you could pass it back in some way.
and just a suggestion
As angular newbie...
Many things are progressing towards this whole functional, reactive programming paradigm. Since you're relatively new to Angular and NG2 already has some of this built in, it might be worthy of your attention. I think rxjs is already in many NG2 example bundles.
The easies way to achieve this is by using Async.js. There you can find a method called mapSeries. You can run it over the queue and it will sequentially process all elements of the array one by one, and will continue to the next element only when the correct callback is called.
I want to know the relation between velocity and IS. If a request is satisfied by velocity, then will it going to use worker process. Or what happen I’m confused. ?
Also I want to store some data like country, state and city for auto suggest in velocity. This database could be on 3 gb. Now how velocity will work. And how IS will work. Is this going to effect IS. Basically my requirements is that I want to save all country, state and city data in velocity and don’t want to hit database and don’t want to make IS busy. What is the solution?
Please help
Velocity was the codename for Microsoft's AppFabric distributed caching technology. Very similar to memcache, it is used for caching objects across multiple computers.
This has no real bearing on how IIS processes requests. All requests are satisfied by IIS, AppFabric is a mechanism for storing data, not processing requests.
In answer to your second question; You can use AppFabric is a first-call check for data. If the data does not exist in the cache, call the database to populate the cache, and then return the data.
var factory = DataCacheFactory();
var cache = factory.GetCache("AutoSuggest");
List<Region> regions = cache.Get("Regions") as List<Region>;
if (regions == null) {
regions = // Get regions from database.
cache.Add("Regions", regions);
}
return regions;
Checking the cache first enables the app to get a faster response, as the database is only hit on the first instance (ideally), and the result data is pushed back into the cache.
You could wrap this up a bit more:
public T Get<T>(string cacheName, string keyName, Func<T> itemFactory)
where T : class
{
var cache = dataFactory.GetCache(cacheName);
T value = cache.Get(keyName) as T;
if (value == null) {
value = itemFactory();
cache.Add(keyName, value);
}
return value;
}
That way you can change your lookup calls to something similar to:
var regions = Get<List<Region>>("AutoSuggest", "Regions", () => GetRegions());