Problem: what I'm sometimes seeing is that very old data is being returned from my various databases when I do SELECT operations (findAll).
I have a VueJS app in which I'm using AxiosJS to call my backend ExpressJS app which uses Sequelize to connect to several (4) SQL Server databases. I'm also using the VueJS Developer Tools so I can see the variables and Vuex store change in real time on the frontend. I'm also using console.log on the backend so I can see some things there too.
In searching for solutions, I have found that:
I'm not using transactions; all of my queries are single queries that return results to the application, so I don't (shouldn't) have issues with commit timing; i.e. I UPDATE or ADD a row and only after it returns a result do I then make a SELECT
Sequelize keeps connections open, and so old connections show as "Sleeping" in the DB; I see these do get reused over time
I'm using the default isolation level in SQL Server which is READ COMMITTED and which should return committed data; since I'm waiting for the first query to return a result before launching the second query, shouldn't it be committed and ready to get me the latest values?
I see that SQL Server CAN store old copies of rows, sometimes making SNAPSHOTS, but I think they require higher isolation levels - but I'm not sure about that; maybe it IS keeping old versions?
I've been unsuccessful in figuring out how to close and reopen Sequelize connections. I'd like to close things at logout and reconnect there too since the app is still running in the tab (if this will solve my old data problem), thereby allowing someone to relogin and open all of the DB connections to be available again. I can't figure it out based on the available docs.
I think Sequelize is using old connections and somehow it is that which is causing SELECT results to be stale - this is even more likely if SQL Server is keeping old versions of rows
For some reason, if I logout and do a hard refresh of my app (CTRL+F5), all data is fresh; I don't understand why this would be at all. What could the browser be holding on to? Note that the console logs on the server app are always consistent with whatever the frontend shows. In other words, if the backend console log is stale, the frontend is stale; if backend is fresh, frontend is too.
I am unable to duplicate the getting of stale data using SSMS or Postman
Question: So what are the possible ways I can be getting stale data when I'm using Axios on the frontend and Sequelize on the backend?
Related
I'm trying to figure out how to create an offline / online approch to use within a huge application.
Right now, each part of the application has its own model and datalayer, who directly read / write data from / to SQL. My boss is asking me to create a kind of buffer that, in case of connectivity failure, might be used to store data until the connection to SQL return active.
What I'm trying to create is something like this: move all datalayers into a servicestack service. Each "GET" method should query the database and store the result into a cache to be reused once the connection to SQL is not available. Each "POST" and "PUT" method must execute their actions or store the request into a cache if the connection fail. this cache must be cleared once the connection to SQL is restored.
How can I achieve this? Mine is a WPF application running on Windows 10.
Best regards
Enrico
Maintaining caches on the server is not going to help create an offline Application given the client wouldn't have access to the server in order to retrieve those caches. What you'd need instead is to maintain state on the client so in the event that network access is lost the client is loading from its own local caches.
Architecturally this is easiest achieved with a Web App using a Single Page App framework like Vue (+ Vuex) or React (+ Redux or MobX). The ServiceStack TechStacks and Gistlyn Apps are good (well documented) examples of this where they store client state in a Vuex store (for TechStacks created in Vue) or Redux Store (for Gistlyn created in React), or the Old TechStacks (created with AngularJS).
For good examples of this checkout Gistlyn's snapshots feature where the entire client state can be restored from a single serialized JSON object or approach used the Real Time Network Traveler example where an initial client state and delta's can be serialized across the network to enable real-time remote control of multiple connected clients.
They weren't developed with offline in mind, but their architecture naturally leads to being offline capable, courtesy of each page being first loaded from its local store then it fires off a Request to update its local cache which thanks to the reactivity of JS SPA fx's, the page is automatically updated with the latest version of the server.
Messaging APIs
HTTP has synchronous tight coupling which isn't ideal for offline communication, what you want instead is to design your write APIs so they're One Way/Asynchronous so you can implement a message queue on the client which queues up Request DTOs and sends them reliably to the server by resending them (using an exponential backoff) until the succeed without error. Then for cases where the client needs to be notified that their request has been processed they can either be done via Server Events or via the client long-polling the server checking to see if their request has been processed.
I'm facing a challenge on using heroku-connect with Salesforce. I'm inserting a record into a parent object (order) on PG db and I get the PG id when I do the insert, then I have to insert the child (order lines) but heroku-connect hasn't done the insert into Salesforce and I don't have the SFID to be able to insert it.
What would you guys recommend I do? Do I do a requery of the field that tells me if it's synched and refresh the $digest in NG? Or do I do it on the API backend with a interval. I'm a little lost on what route to take.
I'm using streaming API but still can't get the SFID from the callback when I do the insert.
rows: [ { id: 85, sfid: null } ],
EDIT
Got this from Heroku support, works great.
https://devcenter.heroku.com/articles/herokuconnect#relationships-between-objects
We ran into this issue before, when we were evaluating Heroku Connect. We ended up with writing our own sync due several limitations of our data model.
In your case, before saving record into Postgres database, I would suggest to send API REST call to SFDC in order to get SFDC Id - only after that persist a record into Postgres. It will keep data model on SFDC/Postgres side consistent. At the same time API call limit utilization will be relatively low as you will be using API only for record creation.
As you are using Heroku Connect(not stand alone sync app) I would not recommend to put backend scheduler into your web app to control refresh of populated Ids. Logic will be highly coupled and might be painful in future to support you system
I am using sessions saved in the database. Works well. Lot of data relating to pagination, browsing history etc is stored perfectly within the database.
However, I notice that data sent to a controller using Ajax is not being stored successfully.
If I debug the session within the controller called by ajax, right after I have set the session vars, I see the values appear to be stored correctly in the session, but on subsequent requests, it transpires that the session vars have NOT been saved.
I have done some testing and have found that the problem disappears if I change back to using "php" for the session instead of "database".
I have eliminated pretty much everything from the mix - and it boils down to Cake not saving session data that is sent by ajax. Again, simply switching back to using "php" for sessions, and everything works perfectly.
I wonder if anyone has experienced anything similar?
CakePHP 2.4
Many thanks.
Well, in case anyone is interested, it turns out that the issue I was having was not strictly related to storing sessions in the database.
My application was making 2 ajax calls at the same time, both attempting to update the session. This was a bug / error on my part, and was causing other session-related issues too, such as returning 403 error status.
I removed the offending bug and all is now well.
i have a scenario where i have to load data from SQL server when i start running a web service. Later i have to use this data for my application, instead of accessing it every time from Database. In addition to this this data should be refreshed every one hour without affecting the website operation on the back end.If any of you has came across such scenario please let me know the solution. By the way i am using asp.net web services, SQL server database, and DNN for my front end.Thanks in advance.
In Global Asax,Application start event you can load all your data in the Dataset.
And by using Sql Cache dependency, You can refresh the data for each hour.But loading the
Entire data is not advisable.By making so you memory will be full.There will
be Performance degrade.
http://www.codeproject.com/Articles/14976/ASP-NET-Caching-Dependencies
Pre-loading all of your data is not a good practice because the database loses its purpose then. It’s probably ok for some data that is very rarely updated but needed very frequently but most definitely not for all the data you have in database.
As for the loading of data you can use app start event as others have already suggested.
Regarding caching – use Application object to make this data available to all parts of application and add a proprety to it that will keep the time of the last update. Then just create separate service that will check the last update time every X minutes and refresh the data when the time comes.
Is this possible to prevent database connection while I requests concrete action in Zend Framework 2?
As Sam stated in a comment; if you do not wish to establish a connection to the database, then simply don't. If you do not have any code within your controller action that uses the database, then there won't be a database connection (provided that you are not doing funky stuff in a bootstrap method or similar).
If you are building a database connection in the standard way, then the actual connection will be lazily loaded, meaning that there won't be an actual connection before you try to use it.
In your comment you state that you believe the problem is caused by many database connections. I just want to clarify that there will only be one database connection per request (provided that you make use of the database).
If you have no code that calls a database, then your web server won't actually make a connection to your database. Either way, if you have 150+ images per page, then that would be a bigger concern and is probably the root cause for slow page loads. Perhaps consider pagination or if you do not display the pictures in their full sizes, then avoid scaling them in HTML as you would then be sending lots of unnecessary data from your web server to your visitors. You could resize the pictures in PHP when they are added, for instance. After that, you could even consider using a Content Delivery Network (CDN), but that is a different discussion...