I'm trying cacheing query, condition is after first request and if query object is not empty I want to cache. I have tried like below
public function getCategories()
{
$categoriesTable = TableRegistry::getTableLocator()->get( 'Categories' );
if(\Cake\Cache\Cache::read('categories', 'redis_cache') == null)
{
$categories = $categoriesTable->find();
if(!$categories->isEmpty())
{
print("hello");
$categories->cache('categories','redis_cache');
debug(\Cake\Cache\Cache::read('categories', 'redis_cache'));
}
$this->set('categories',$categories);
}else{
$this->set('categories',\Cake\Cache\Cache::read('categories', 'redis_cache'));
}
$this->viewBuilder()->setOption('serialize', ['categories']);
}
Output I got in postman
1st time hit
hello
null
2nd time hit
hello
null
But after comment if(!$categories->isEmpty()) condition
1st time hit
{
"categories": [
{
"id": 2,
...
}
Also if I write the condition like below it's also working fine.
if(!$categoriesTable->find('all')->isEmpty()). // it's working
What's the wrong I'm doing here ?
Queries are lazy evaluated, meaning they are not being executed until the data is actually being requested, however isEmpty() is one of the methods that will cause the query to be executed, it needs the results to tell whether there are any.
Your code sets the query cacher config afterwards, which will have no effect, as further evaluation/iteration of the query object will not execute the query again (which is when it would normally cache its results), but access the runtime buffered results instead, hence no data is being cached.
If you need more control over how caching is being handled, then you maybe want to avoid the built-in query caching, and do it completely manually, eg read and also write the cache as neccessary.
As far as your use case is concerned, if you have a script running that deletes and regenerates the data, you might want to consider using it to also clear the cache afterwards (bin/cake cache clear redis_cache). Then, if you implement the "try again until there's data" logic in the frontend, your backend code can be simplified to just read and cache whatever data there is, and if there is none yet, your frontend will let the user know that the data isn't ready yet and poll again. While websockets or server sent events might be better, long polling is certainly still preferable over having a script run minutes on end.
Related
When I run a .fetch() command, it first returns null, then say suppose I have 100 documents and it will keep on loading from 1 to 100 and the counter keeps updating from 1 to 100 progressively. I don't want that to happen. I want all the results to be displayed at once after the fetch process has been completed.
Also, how can I display a relevant message to the user if no documents exist? The fetch method doesn't work for me since it returns 0 at first and hence "No document found" flashes for a second.
dbName.find({userID:"234234"}).fetch()
Even though the above has 100 docs, it first shows null and then keep loading the documents one by one. I want it load all at once or just return something if no docs exist.
I don't want that to happen. I want all the results to be displayed at once after the fetch process has been completed
To really obtain all documents at once on the client you will have to write a Meteor Method that returns all the documents:
Meteor.methods({
'allDocs' () {
return dbName.find({userID:"234234"}).fetch()
}
})
Note, that you have to call fetch on the cursor to return the documents, otherwise you will face an "unhandled promise rejection error".
Then call it from the client as usually. You can even add the documents to your client side local collection without affecting allow/deny (which should be off / deny all by default):
Meteor.call('allDocs', (err, documents) => {
// ... handle err
// all client collections have a local collection accessible via ._collection
const localCollection = dbName._collection
documents.forEach(doc => localCollection.insert(doc))
})
Advantages:
Returns all documents immediately
Less resources consumed (no publication observers required)
Works with caching tools, such as ground:db, to create offline-first applications
Disadvantages:
You should limit the query and access to your collections using Methods as much as possible, (using mdg:validated-method) which can require much more effort than shown in this examples
Not reactive! If you need reactivity on the client you need to include Tracker and reactive data-sources (ReactiveVar etc.) to provide some decent reactive user experience
Manual syncing can become frustrating and is error prone
Your question is actually about the subscription and it's state of readiness. While it is not yet ready, you can show a loading page, and once it is you can run the .fetch() to get the whole array. This logic could be put in your withTracker call, e.g.:
export default withTracker((props) => {
const sub = Meteor.subscribe('users');
return {
ready: sub.ready(),
users: sub.ready() && Users.find({userID: props.userID}).fetch()
};
})(UserComponent);
Then, in your component, you can decide whether to render a spinner (while ready == false), or the users.
Your question is not entirely clear to me in terms of tools etc (please state which database connector lib are you using), but firstly, given you're doing a database access, most likely, your ".fetch()" call is not a sync function but async, and also most likely, handled by a promise.
Secondly, given that you're using react, you want to set a new state only after you get all the results back.
If fetch is a promise then just do:
dbName.find({userID:"234234"}).fetch().then(results =>
setState({elements:results.data}) // do your processing accordingly
}
By only calling setState inside the promise, you'll always have all the results fetched at that instant and only with that do you update your component state with the setState function - either using your react component class this.setState or with hooks like useState (much cleaner).
I'm currently using the Google memcache API in my Google Appsengine application to store a large amount of data in cache, however this needs to happen asynchronously as I need to return a result before it is done.
I already found an answer here stating how it is done, however I still can't get my code to work.
I tried using this, however it simply causes memcache to be store the values synchronously:
client = memcache.Client()
rpc = client.set_multi_async(values)
rpc.get_result()
return values[id]
I also tried this, but it causes memcache to never save the values:
client = memcache.Client()
client.set_multi_async(values, rpc=memcache.create_rpc())
return values[id]
Is there any way to store the values asynchronously and return a value at the same time? Thanks
rpc.get_result() causes the rpc object to wait until the result is available before returning, thus making your code synchronous. In order to make your code asynchronous, you would need to return the rpc object(s) and then use the Future class to wait for and handle results when they are ready.
I am working on a project where we were asked to "patch" (they don't want a lot of time spent on development as they soon will replace the system) a system implemented under ExtJS 4.1.0.
That system is used under a very slow and non-stable network connection. So sometimes the stores don't get the expected data.
First two things that come to my mind as patches are:
1. Every time a store is loaded for the first time, wait 5 seconds and try again. Most times, a page refresh fix the problem of stores not loading.
Somehow, check detect that no data was received after loading a store and, try to get it again.
This patches should be executed only once to avoid infinite loops or unnecessary recursivity, given that it's ok that some times, it's ok that stores don't get any data back.
I don't like this kind of solutions but it was requested by the client.
This link should help with your question.
One of the posters suggests adding the below in an overrides.js file which is loaded in between the ExtJs source code and your applications code.
Ext.util.Observable.observe(Ext.data.Connection);
Ext.data.Connection.on('requestexception', function(dataconn, response, options){
if (response.responseText != null) {
window.document.body.innerHTML = response.responseText;
}
});
Using this example, on any error instead of echoing the error in the example you could log the error details for debugging later and try to load again. I would suggest adding some additional logic into this so that it will only retry a certain number of times otherwise it could run indefinitely while the browser window is open and more than likely crash the browser and put additional load on your server.
Obviously the root cause of the issue is not the code itself, rather your slow connection. I'd try to address this issue rather than any other.
I've run into a bit of a dilemma with Cake 2.4's caching system and I was wondering if anyone can think of a simply solution.
I have made a function within a component which I use for connecting to an API to obtain data. I also want to to cache data in order to make it more efficient.
The problem is that exactly when the cache expires depends on a field in the $result array (from the API) called $result['expiry']. So, I've done this:
class ApiComponent extends Component {
public function getRecords($data) {
$cacheName = $data['Model'] . "_" . $data['action'];
// IMPOSSIBLE because we don't have $result:
// Cache::set(array('duration' => '+' . $result['expiry'] . ' days'));
$result = Cache::read($cacheName);
if (!$result) {
$result = $this->connect('getRecords',$data);
Cache::set(array('duration' => '+' . $result['expiry'] . ' days'));
Cache::write($cacheName, $result);
}
return $result;
}
public function connect($endpoint,$data) {
// Code for connecting to the API
}
}
The problem is that, for some reason, Cake needs me to specify the expiry for a write AND a read, despite the expiry timestamp being present in the cache file itself.
I've commended the necessary line as IMPOSSIBLE in my example above. According to the book, this is how it's supposed to work. Unfortunately it doesn't make much sense to me.
I understand having to set the expiry before a write but why do I need to set the expiry before a read when the cache file itself has a UNIX timestamp specifying exactly when the cache expires, based on what I did when I set the write.
Please note that accessing the API is only costing me 100ms so there wouldn't be any point saving the expiry to the database and accessing that every time unless I also cached the local database output but that seems like a truly bizzare thing to have to do: caching a cache expiry!
Theoretically you don't necessarily need the duration when reading from the cache. The reason for having it available on read, is to be able to invalidate the cache in case the duration changed ever since the data was cached.
https://github.com/cakephp/cakephp/commit/caa7bb621871706baf664b5e2e4f562353f3671f
Unfortunately there is no clean way of overcoming this auto-magic as a duration option value will be used no matter what, when not defining a custom one, the one from the default configuration will be used, and if that one is lower than the one used on write, then your cache is going to be invalidated prematurely.
A (kinda ugly) workaround would be to set an extraordinary high duration dummy value for the read operation, you're losing the auto-magic, but your cache is not going to be invalidated prematurely:
Cache::set(array('duration' => '+10 years'));
$result = Cache::read($cacheName);
Suppose that one needs to send the same collection of 10,000 documents down to every client for a Meteor app.
At a high level, I'm aware that the server does some bookkeeping for every client subscription - namely, it tracks the state of the subscription so that it can send the appropriate changes for the client. However, this is horribly inefficient if each client has the same large data set where each document has many fields.
It seems that there used to be a way to send a "static" publish down the wire, where the initial query was published and never changed again. This seems like a much more efficient way to do this.
Is there a correct way to do this in the current version of Meteor (0.6.5.1)?
EDIT: As a clarification, this question isn't about client-side reactivity. It's about reducing the overhead of server-side tracking of client collections.
A related question: Is there a way to tell meteor a collection is static (will never change)?
Update: It turns out that doing this in Meteor 0.7 or earlier will incur some serious performance issues. See https://stackoverflow.com/a/21835534/586086 for how we got around this.
http://docs.meteor.com/#find:
Statics.find({}, {reactive: false} )
Edited to reflect comment:
Do you have some information that the reactive: false param is only client side? You may be right, it's a reasonable, maybe likely interpretation. I don't have time to check, but I thought this may also be a server side directive, saying not to poll the mongo result set. Willing to learn...
You say
However, this is horribly inefficient if each client has the same large data set where each document has many fields.
Now we are possibly discussing the efficiency of the server code, and its polling of the mongo source for updates that happen outside of from the server. Please make that another question, which is far above my ability to answer! I doubt that is happening once per connected client, more likely is a sync between app server info and mongo server.
The client requests you issue, including sorting, should all be labelled non-reactive. That is separate from whether you can issue them with sorting instructions, or whether they can be retriggered through other reactivity, but which need not include a trip to the server. Once each document reaches the client side, it is cached. You can still do whatever minimongo does, no loss in ability. There is no client asking server if there are updates, you don't need to shut that off. The server pushes only when needed.
I think using the manual publish ( this.added ) still works to get rid of overhead created by the server observing data for changes. The observers either need to be added manually or are created by returning a Collection.curser.
If the data set is big you might also be concerned about the overhead of a merge box holding a copy of the data for each client. To get rid of that you could copy the collection locally and stop the subscription.
var staticData = new Meteor.Collection( "staticData" );
if (Meteor.isServer ){
var dataToPublish = staticData.find().fetch(); // query mongo when server starts
Meteor.publish( "publishOnce" , function () {
var self = this;
dataToPublish.forEach(function (doc) {
self.added("staticData", doc._id, doc); //sends data to client and will not continue to observe collection
});
});
}
if ( Meteor.isClient ){
var subHandle = Meteor.subscribe( "publishOnce" ); // fills client 'staticData' collection but also leave merge box copy of data on server
var staticDataLocal = new Meteor.Collection( null ); // to store data after subscription stops
Deps.autorun( function(){
if ( subHandle.ready() ){
staticData.find( {} ).forEach( function ( doc ){
staticDataLocal.insert( doc ); // move all data to local copy
});
subHandle.stop(); // removes 'publishOnce' data from merge box on server but leaves 'staticData' collection empty on client
}
});
}
update: I added comments to the code to make my approach more clear. The meteor docs for stop() on the subscribe handle say "This will typically result in the server directing the client to remove the subscription's data from the client's cache" so maybe there is a way to stop the subscription ( remove from merge box ) that leaves the data on the client. That would be ideal and avoid the copying overhead on the client.
Anyway the original approach with set and flush would also have left the data in merge box so maybe that is alright.
As you've already pointed out yourself in googlegroups, you should use a Meteor Method for sending static data to the client.
And there is this neat package for working with Methods without async headaches.
Also, you could script out the data to a js file, as either an array or an object, minimize it, then link to it as a distinct resource. See
http://developer.yahoo.com/performance/rules.html for Add an Expires or a Cache-Control Header. You probably don't want meteor to bundle it for you.
This would be the least traffic, and could make subsequent loads of your site much swifter.
as a response to a Meteor call, return an array of documents (use fetch()) No reactivity or logging. On client, create a dep when you do a query, or retrieve the key from the session, and it is reactive on the client.
Mini mongo just does js array/object manipulation with an syntax interpreting dsl between you and your data.
The new fast-render package makes one time publish to a client collection possible.
var staticData = new Meteor.Collection ('staticData');
if ( Meteor.isServer ){
FastRender.onAllRoutes( function(){
this.find( staticData, {} );
});
}