I'd like to be able to do the following in a HTML5 (iPad) web app:
upload data to an online database (which would be probably <50Mb in size if I was to build the online database in something like SQLite)
extract either a subset or a full copy of data to an offline webdatabase
(travel out of 3G network coverage range)
perform a bunch of analytic-type calculations on the downloaded data
save parameters for my calculations to the offline webdatabase
repeat, saving different parameter sets for several different offline analytic-type calculation sessions over an extended period
(head back into areas with 3G network coverage)
sync the saved parameters from my offline webdatabase to the central, online database
I'm comfortable with every step up till the last one...
I'm trying to find information on whether it's possible to sync an offline webdatabase with a central database, but can't find anything covering the topic. Is it possible to do this? If so, could you please supply link/s to information on it, OR describe how it would work in enough detail to implement it for my specific app?
Thanks in advance
I haven't worked specifically with HTML5 local databases, but I have worked with mobile devices that require offline updates and resyncing to a central data store.
Whether the dataset is created on the server or on the offline client, I make sure its primary key is a UUID. I also make sure to timestamp the record each time its updated.
I also make not of when the last time the offline client was synced.
So, when resyncing to the central database, I first query the offline client for records that have changed since the last sync. I then query the central database to determine if any of those records have changed since the last sync.
If they haven't changed on the central database, I update them with the data from the offline client. If the records on the server have changed since last sync, I update them to the client.
If the UUID does not exist on the central server but does on the offline client, I insert it, and vice versa.
To purge records, I create a "purge" column, and when the sysnc query is run, I delete the record from each database (or mark it as inactive, depending on application requirements).
If both records have changed since last update, I have to either rely on user input to reconcile or a rule that specifies which record "wins".
I usually don't trust built-in database import functions, unless I'm importing into a completely empty database.
Steps:
Keep a list of changes on the local database.
When connected to remote database, check for any changes since last sync on remote.
If changes on remote side has conflicts with local changes, inform the user on what to do.
For all other changes, proceed with sync:
download all online changes which did not change locally.
upload all local changes which did not change remotely.
This method can actually work on any combination of databases, provided there is a data convertor on one side.
It looks to me, from a few sites I visited, that (as long as you are using SQLite for your Server db) it should be possible.
HTML5 webdatabases also use SQLite (although not all browsers support it and W3C seems to have dropped support for it)
so...
If you export the data using the .dump command and then import the data into the webdatabase using the $sqlite mydb.db < mydump.sql syntax you should be able to do this with some fidgeting with a php or java backend?
Then when you want to sync the "offline" data to your server just do the opposite dump from the webdatabase to a dump.sql file and import to the server database
This site explains exporting to and importing from SQLite dumps
SOURCE: dumping and restoring an SQLite DB
HTML5 supports browser db SQLite , I have tried in Mozilla and chrome, and it works fine.
I also have a requirement where I have some offline form, user punches the data and click on save, it saves in local browser db, and later when user syncs with the server or comes online, he can click on sync button which actually sync data from browser db any other datasource.
Related
I am developing a flutter app that should also be able to run in offline mode. Because I am using flutter I want to also offer the use of the web version of my application. The application I want to build is data reliant therefore to make it work offline I need to have some kind of local database, but for the web version to work, I also need to store the data on a remote database so it can be accessed from the web. The problem that this proposes is how do I makes sure that the local and remote databases are always on the same page. If something is changed from the web it needs to also affect the local database on the device and if something is changed locally it also has to affect the remote database. I simply need a tip on how to generally achieve what I am looking for.
This can be easily achieved using Firebase Firestore which supports offline.
Or
If you plan to do this from scratch one way to do this is to keep a separate local database and execute every transaction on it. If online do the same transactions on the remote database. If offline keep a record of transactions pending on the local database ( preferably in a separate DB from the main DB) and execute them when the remote database is connected.
You can use Hive or sqflite for local DB
Or
Keep track of records, which the transactions were performed on. Before synchronize merge these records from both local(done offline) and remote(done on the web, phone not connected). If multiple transactions were performed on the same record, Update both remote & local DB records to the latest record state from DB where the latest transaction was performed (if the latest transaction was performed on the remote DB, update the local DB& vice-versa)
I am in the process of developing an app for picking up orders for billing, in which I have determined that it works offline because users arrive at certain places where there is no signal, therefore the app will work with a local database SQLite, which I want when it connects to the internet, it synchronizes the data in a bidirectional way, it is a MERGE type replica, between SQL Server and SQLite, it should be noted that the app works in Xamarin Forms, I wanted to know if there is any information about this.
Thank you for your attention.
It's a manual process, you read from your SQLite db and write in SQL Server and vis versa, keep track of the data you already synced using flags and date columns. ( By manual I mean Write Code)
Currently i working the project same as you, using Dotmim.sync tool
for data around 10k to sync with no problem but after 100k data it need take time to sync, maybe that sync tool will help you
We are building a web application and plan to run it on AWS. Created a RDS instance with MySQL. The proposed architecture is as follows:
Data is being uploaded from company data mart to Core DB in RDS. On the other side, user is sending data through our Rest API to post data. This user input data will be saved in a separate DB within the same RDS, as one of our architects suggested. The data will then be periodically copied to a table inside Core DB. We will have a rule engine running based Core DB. Whenever an exception is detected, notification will be sent to customers.
The overall structure seems fine. One thing I would change though is instead of having two separate DBs, we can just have one DB and have user input data in a table in the same Database. The logic behind separate DBs, according to our architect, is for security concerns. Since Core DB will have data from our company, it is better to be on its own. So the http requests from clients will only affect the user input DB.
While it makes sense, I am not sure it is really necessary. First all the user input is authenticated. Secondly the web api provides another protection layer against database since it only allows certain requests, which in this case couple of endpoints for post request. Besides if someone can somehow still hack into the User Input DB in RDS, since the it resides on the same RDS instance plus there is data transfer between DBs, it is not impossible they can't get to Core.
That said, do we really need separate DBs? And if this is the way to go, what is best way to sync from User Input DB to a User Input TB in Core DB?
In terms of security reason, separating the db are not magically make it true. My suggestion :
Restrict the API layer, such as only have write access ( just in case to avoiding accidentally deleting data)
For credentials data, don't put it on source code, you can put it as environment variables, example on ElasticBeanstalk Environment Variables
For RDS itself, put it under VPC
In term of synchronizing data if you have to go with 2 db.
if your two database are exactly same on the schema, you can use db replication capability (such as mysql replication)
if not, you can send it to message broker service (SQS) then create a worker to pulling it then save it to target database
or you can use another service such as datapipeline
I have got some tabular data that due to unrelated issues is proving too slow to get out of SQL Server in realtime. As we get more users this will only get worse so I am thinking of using Redis as a front-end cache to store users' tabular pageable data.
This data could become stale after about 10 minutes after which time I would like to get the record set again and put in in Redis.
The app is an .NET MVC app. I was thinking that when the user logs into the app this data gets pulled out of the database (takes around 10 seconds) and put into Redis ready to be consumed by the MVC client. I would put an expiry on that data and then when it becomes stale it will get refetched from the SQL Server database.
Does this all sound reasonable? I'm a little bit scared that:
The user could get to the page before the data is in Redis
If Redis goes down or does not respond I need to ensure that the ViewModel can get filled direct from SQL SErver without Redis being there
I will go for Service stack redis implementation, here are all the details required. Redis is particularly good when doing caching in compare to other nosql. But if you are having high read - write application, I will insist to check out nosql database as database combined with SQL server. That will help in case of scalability.
Please let me know if any further details required. You just need to fire nuget command and you are almost up and running.
You could use something like MemcacheD to store cached pages in memory.
You can set a validity of 10 minutes on a cached object. After that the cache will automatically remove the object.
Your actual repository would have to do these steps:
1. Check the cache for the data you want, if it is there, great use it
2. If the cached data doesn't exist, go to SQL server to retrieve it
3. Update the cache with data returned from SQL server
I've used the Enyim client before. It works great. https://github.com/enyim/EnyimMemcached
I might also use something like Quartz to schedule a background task to prime the cache. http://quartznet.sourceforge.net/
In a Firebird database driven Delphi application we need to bring some data online, so we can add to our application online-reporting capabilities.
Current approach is: whenever data is changed or added send them to the online server(php + mysql), if it fails, add it to the queue and try again. Then the server having the data is able to create it's own reports.
So, to conclude: what is a good way to bring that data online.
At the moment I know these two different strategies:
event based: whenever changes are detected, push them to the web server / mysql db. As you wrote, this requires queueing in case the destination system does not receive the messages.
snapshot based: extract the relevant data in intervals (for example every hour) and transfer it to the web server / mysql db.
The snapshot based strategy allows to preprocess the data in a way that if fits nicely in the wb / mysql db data structure, which can help to decouple the systems better and keep more business logic on the side of the sending system (Delphi). It also generates a more continuous load, as it does not care about mass data changes.
One other way can be to use replication but I don't know system who make replication between Firebird and MySQL database.
For adding reporting tools capability on-line : you can also check fast report server