My question is about how to store data which once was received online and still can be processed after the mobile device got offline and/or was restarted.
I'm using AngularJS with Ionic (PhoneGap) for building apps. But my question is not explicitly adressing these technologies.
Best practices, patterns or algorythms would be very helpful to me or even some useful articles or key words.
1) The most simple challenge is to make my app more user-friendly by making its functionality usable not only if the device is online but also in offline mode. In my case this implies that I have to make the last fetched online data available for later use (while device is offline and also after restarting the device!).
2) A bit more difficult is to reduce the communication costs by only synchronizing the server side changed data when the device reconnects to the internet.
3) Entities can also be produced on client side while the device is offline and they must get synchronized to the server too. There are no potential risks of conflicts because the users don't share Entities with write access.
4) I use Googles and Apples push services to inform the devices about newer entity versions, which should get updated on client side. So polling isn't needed.
Client side technologies: Javascript, AngularJS Framework, Ionic Framework, SQLite (WebSQL) or IndexedDB, PhoneGap (Cordova)
Server side technologies: Java EE, JPA, MySQL
Data Format and communication: JSON over REST / http, Googles and Apples push services for server-to-client messaging
1) Store the needed data inside a local SQLite database, and pull it out when the app starts/resumes.
2)In the MySQL database you need a table that creates new entries when you update/change/create content. You would need to store an id and a timestamp (maybe a boolean value if the content was deleted).
On the device you would make a request to the server to send the data from that table and compare it with the locally stored data. If there is a new id or the timestamp has change make a new request to pull the updated data.
3)Store the created data locally with a flag that it isn't synced with the server. When the device then goes online again check for not synced flags and send the data to the server with an identifier to know which device it comes from and where to save it.
4)See 2)
You could make a Java script which checks every x minutes for updated entries and send an automatic push Notification with it. What you would need is 2 tables, one with the newest updates and one with the updates that got pulled by the device(just id's and timestamps, not all the data).
I hope this was helpful, if something new comes to my mind I will update this answer.
Related
I'm trying to figure out how to create an offline / online approch to use within a huge application.
Right now, each part of the application has its own model and datalayer, who directly read / write data from / to SQL. My boss is asking me to create a kind of buffer that, in case of connectivity failure, might be used to store data until the connection to SQL return active.
What I'm trying to create is something like this: move all datalayers into a servicestack service. Each "GET" method should query the database and store the result into a cache to be reused once the connection to SQL is not available. Each "POST" and "PUT" method must execute their actions or store the request into a cache if the connection fail. this cache must be cleared once the connection to SQL is restored.
How can I achieve this? Mine is a WPF application running on Windows 10.
Best regards
Enrico
Maintaining caches on the server is not going to help create an offline Application given the client wouldn't have access to the server in order to retrieve those caches. What you'd need instead is to maintain state on the client so in the event that network access is lost the client is loading from its own local caches.
Architecturally this is easiest achieved with a Web App using a Single Page App framework like Vue (+ Vuex) or React (+ Redux or MobX). The ServiceStack TechStacks and Gistlyn Apps are good (well documented) examples of this where they store client state in a Vuex store (for TechStacks created in Vue) or Redux Store (for Gistlyn created in React), or the Old TechStacks (created with AngularJS).
For good examples of this checkout Gistlyn's snapshots feature where the entire client state can be restored from a single serialized JSON object or approach used the Real Time Network Traveler example where an initial client state and delta's can be serialized across the network to enable real-time remote control of multiple connected clients.
They weren't developed with offline in mind, but their architecture naturally leads to being offline capable, courtesy of each page being first loaded from its local store then it fires off a Request to update its local cache which thanks to the reactivity of JS SPA fx's, the page is automatically updated with the latest version of the server.
Messaging APIs
HTTP has synchronous tight coupling which isn't ideal for offline communication, what you want instead is to design your write APIs so they're One Way/Asynchronous so you can implement a message queue on the client which queues up Request DTOs and sends them reliably to the server by resending them (using an exponential backoff) until the succeed without error. Then for cases where the client needs to be notified that their request has been processed they can either be done via Server Events or via the client long-polling the server checking to see if their request has been processed.
I am starting to work with angular and am fascinated by the bi-directional data-binding capabilities and by its $http method, which lets me save changes in to my mysql database, without refreshing the page.
Another thing I am currently fascinated by is the real time capability across multiple clients using firebase. Here all clients are updated in REAL TIME, when the database receives any changes. I'd probably like to use firebase, but I would have to drop Laravel and MySql as a persistence layer entirely, which I would like to keep for the moment, since my application is already working in Laravel, just not in real time.
How would I go about having a Real Time application, which updates every client, without refreshing the view, in Laravel using MySQL and Angular?
If I am not mistaken, Pusher and PubNub, are providing this necessary open connection with the server using websockets, so when the server has something to share, angular will now and render it.
Since I would like to use Laravel and MySQL as a persistence layer, I am not sure, what the best way would be. I am not even sure, if I understood everything correctly, which I wrote above, since I am new to angular and real-time applications.
What would be the next necessary steps, to get some Real-Time capability into a PHP/MySQL application?
The solution for your problem is:
1º - open websocket connection with the websocket-server and subscribe a channel, after this send the data to your serve using ajax
tutorial angular pusher
2º - In server side, you get the data, saves to your database and send a 'PUBLISH' to the respective channel into websocket server
lib useful for this
3º - Through the subscribe gets the data in real time
Pusher.subscribe('channel', 'event', function (item) {
// code
});
I had a similar problem recently and I finally ended up using Redis publish/subscribe Redis. You can store data in the channel and then subscribe to any changes. When something changes you can send it to Pusher which will send it then to the clients.
I also recommend considering Node.js and Socket.io since you can achieve very good performance without third party service, and even if you don't have experience with node you can find very good examples on Socket.IO how to write an application.
For Redis there is a good library for PHP called Predis and there is Redis Node client as well, so you can mix it all together.
Have an Angular web app that track status of many objects. Now, I have a directive for each object to poll a nodejs server for status. This works but it is quite chatty and I am a bit worried about scalability. However, the backend does not support subscription so it could not use a pusher. One option is to let the nodejs server do the polling and push status back. Then nodejs need to keep a list of objects to poll and client probably need to check the message to determine whether a status needs update. Really appreciate your suggestion for a better strategy.
You can use firebase and load the data from a third party source into the firebase database then use angular-fire to get that data into the app... alternatively you could use a socket connection to DIY. In either scenario, I assume like you said the NodeJS portion on the server will need to handle polling the third party system for data and populating the firebase instance which would be a read-only layer as far as the system is concerned... changes would be sent directly to the third party and NodeJS would just be responsible for updating the firebase DB.
I'm currently using angularJS and phonegap to build a test application for Android / iOS.
The app use only text data stored in a Firebase database. I want the app to have its own local database (used when the device is offline) and sometime (when the device is online)
sync with a Firebase database.
The offline mode uses the storage API of phonegap/cordova. Could I just check the device's online state and backup the online database periodically ?
Any clues on how I can achieve this ? Last time a similar question was asked, the answer was "not yet"... (here)... because it focused on a hypothetical Firebase feature.
If Firebase is online at the start and loses its connection temporarily, then reconnects later, it will sync the local data then. So in many cases, once Firebase is online, you can simply keep pushing to Firebase during an outage.
For true offline usage, you will probably want to monitor the device's state, and also watch .info/connected to know when Firebase connects.
new Firebase('URL/.info/connected').on('value', function(ss) {
if( ss.val() === null ) /* firebase disconnected */
else /* firebase reconnected */
});
The way to achieve this with the current Firebase toolset, until it supports true offline storage, would
keep the local data simple and small
when the device comes online, convert the locally stored data to JSON
use set() to save the data into Firebase at the appropriate path
Additionally, if the app loads while the device is offline, for some reason, you can "prime" Firebase by calling set() to "initialize" the data. Then you can use Firebase as normal (just as if it were online) until it comes online at some point in the future (you would also want to store your local copy to handle the case where it never does).
Obviously, the simpler the better. Concurrent modifications, limits of local storage size, and many other factors will quickly accumulate to make any offline storage solution complex and time consuming.
After some time, I would like to add $0.03 to #Kato's answer:
Opt to call snapshot.exists() instead of calling snapshot.val() === null. As the documentation points out, exists() is slightly more efficient than comparing snapshot.val() to null.
And if you want to update data prefer to use the update() method rather then set(), as the last will overwrite your Firebase data. You can read more here.
I am building a website (probably in Wordpress) which takes data from a number of different sources for display on various pages.
The sources:
A Twitter feed
A Flickr feed
A database on a remote server
A local database
From each source I will mainly retrieve
A short string, e.g. for Twitter, the Tweet, and from the local database the title of a blog page.
An associated image, if one exists
A link identifying the content at its source
My question is:
What is the best way to a) store the data and b) retrieve the data
My thinking is:
i) Write a script that is run every 2 or so minutes on a cron job
ii) the script retrieves data from all sources and stores it in the local database
iii) application code can then retrieve all data from the one source, the local database
This should make application code easier to manage - we only ever draw data from one source in application code - and that's the main appeal. But is it overkill for a relatively small site?
I would recommend putting the twitter feed and flickr feed in JavaScript. Both flickr and twitter have REST APIs. By putting it on the client you free up resources on your server, create less complexity, your users won't be waiting around for your server to fetch the data, and you can let twitter and flickr cache the data for you.
This assumes you know JavaScript. Once you get past JavaScript quirks, it's not a bad language. Give Jquery a try. JQuery Twitter plugin Flickery JQuery plugin. There are others, that's just the first results from Google.
As for your data on the local server and remote server, that will depend more on the data that is being fetched. I would go with whatever you can develop the fastest and gives acceptable results. If that means making a REST call from server to sever, then go for it. IF the remote server is slow to respond, I would go the AJAX REST API method.
And for the local database, you are going to have to write server side code for that, so I would do that inside the Wordpress "framework".
Hope that helps.