Context
I'm developing a hybrid/mobile app using Cordova. This app displays content, such as news, magazine articles and weather info that is dynamically generated and delivered via a Web App (self-hosted developed with PHP). In order to decide which mobile devices "get" which content, I need to constantly keep track of the history of displays on each device, and gather information such as : article name, display date/time, location(lat/lon), etc.
The BIG problem I have is...
Now, because the data that I'm gathering is unstructured, I first tried with Firebase (Google), which allowed me to easily gather all the data, in real-time using a key:value schema and then retrieve my data on the webapp running another JS script. The problems with this solution are:
Querying possibilities are too limited.
I'm not comfortable leaving all my data handled by Google (Nothing against Google, butI'd prefer to own my data).
The Question
What are the storage strategies that I could use ? I've been reading a lot about Redis, Parse-server, PouchDB+CouchDB, and MongoDB, but I got lost in a world of information about NoSql databases, and now I really do not know what to do :'(
I've always used relational databases for my projects (Mysql, MariaDB), but I'm not afraid in getting my hands dirty with NoSQL databases, however I'd appreciate if anyone can point me to the easiest-to-implement solution.
EDIT (17/04/18)
I think I narrowed down my question to 2 options
PouchDB + CouchDB, or
Couchbase Mobile = Couchbase Lite + Sync Gateway
Can anyone, please give me some feedback with regards to these two options? Ease of implementation, Querying, maintenance, etc ?
Thanks!
Figured, I should probably explain my earlier comment better.
Couchbase Mobile = Couchbase Lite (runs on mobile device) + Sync Gateway + Couchbase Server (hosted in backend)
W.r.t your first concern
Querying possibilities are too limited.
Couchbase Server supports N1QL - which is a superset of SQL for NoSQL. So you should be able to do the kind of extensive querying that you want w/ Couchbase Server.This link discusses how you can do geospatial queries (since you mentioned that as a use case)
W.r.t your second concern
I'm not comfortable leaving all my data handled by Google (Nothing
against Google, butI'd prefer to own my data).
With Couchbase Mobile, you have control over where the data is hosted - you have the option of persisting the data in Couchbase servers that you can host in private/public cloud (AWS, Azure, GCP).
Note on Cordova support in Couchbase Lite:
It is supported as a community project in Couchbase Lite 1.4. However, in the latest 2.0 release, you will have to develop your Cordova plugin on top of the native API .
Related
I am developing an app, which is in android.
In my app I want to use couchdb database for storing and sync purpose.
which will work offline too.
so, how to create couchDB database first and then insert data in that database, so that my application will use the couchDB database as local storage like SQLite.
I have aleready went through many links but they are not satisfying my need.
So Please provide some guide line or an help or comment or sample code to performed that operation.
CouchDb and Couchbase Lite are very close in terms of usage. If you are looking for an easy way to sync and store data locally on your android device I would recommend looking at Couchbase Lite for Android.
https://github.com/couchbase/couchbase-lite-android
Alrighty I have a cross platform mobile app built using xamarin and the mvvmcross framework. Part of the app is grabbing some data from a remote database. This data will be updated/deleted/created occasionally and i want to keep my users up-to-date. What's a good way to achieve this, preferably using cross platform code?
I can always set some kind of timer but I would much prefer a more elegant way.
Couchbase, as mentioned by Artur, is a great option - especially if doing disconnected editing. It does have a bit of a learning curve picking it up from scratch (which is totally worth it if you press on).
Consider Akavache (https://github.com/akavache/Akavache) as an alternative. It's well used and simpler but provides the basic capabilities you are asking about.
I recommend combining either approach with using ModernHttpClient (https://components.xamarin.com/view/modernhttpclient) which is basically just a drop-in component that gets you a performance boost.
Check out this excellent article by Rob Gibbens that covers using Akavache, ModernHttpClient, and other x-platform wholesome goodness: http://arteksoftware.com/resilient-network-services-with-xamarin/.
Check out Couchbase Mobile
http://www.couchbase.com/nosql-databases/couchbase-mobile
There is a wrapper for Xamarin
https://components.xamarin.com/view/couchbase-lite-net
With Couchbase you can save/update/synchronize data between your mobile device and cloud server using NoSQL DB.
In the context of Couchbase Mobile for your use-case where you are updating/deleting and creating your data occasionally, you can replicate the data based on per method invocation. This would happen once as oppose to replication that is on a continuous basis.
You would do this by defining the logic on when your PUSH and PULL replication would occur on a particular database endpoint.
---PUSH Replication would be from your mobile app/device to Couchbase Server(Cloud component)
The library, Couchbase Lite (app component), on your app/device would flag new data or changes on existing documents(database) and will PUSH replicate to the Sync Gateway (data orchestration component).
---PULL Replication happens from the Couchbase Server(Cloud component) to mobile device. The Sync Gateway (data orchestration component) has advanced listeners that are observing for changes in the Couchbase Server and upon remote changes will pull data to the Sync Gateway and then update your mobile app/device.
I also suggest taking a look at the mini-hack quick tutorial, specifically the Xamarin one on how to get a Xamarin app working with Couchbase Mobile. And the mobile Developer Portal is a another great resource too where there are Xamarin Android & iOS guides.
I'm a web developer and used to using FireBase for a lot of my projects. Much of the work is already done out-of-the box there regarding realtime synchronization etc. In my next project, however, I'm not allowed to use an external provider for storing data, and as there's no way to host Firebase on a local server, it's out of the question.
For that project, currently a Redis key-value store is used. I know Redis also supports a pub-sub model, and I'm searching for a good way to use this Redis key-value store in a realtime web application. I've found a Meteor library (https://github.com/meteor/redis-livedata), which looks promising. I'm used to using AngularJS, however. And the majority of the solutions I found only use Redis for the pub-sub model, and not for the datastore (maybe for good reason). I've found an alternative which uses Django as the datastore (https://github.com/mburst/django-realtime-tutorial). I've also come across Firehose (http://firehose.io), but this doesn't use the Redis key-value store either.
Does anyone know a good way to emulate the same functionality Firebase provides, using a Redis key-value store, preferably compatible with AngularJS?
Correction - Django is a python based web framework, not a datastore .
For your requirement , tools like deployd and loopback suit alot.
They are opensource versions of services like Firebase or Parsed.
Both of them work with javascript(nodejs) and hence would support AngularJS or any other framework you would want to use.
For datastore backend, most preferred pluggable backend is mongodb as it supports fast writes, rich querying, and persistence of data.
Are there any cloud hosting solutions for geospatial data? I am currently writing a directory style app where businesses can sign up and then users can find nearby ones.
I am considering Google App Engine for this, but from what I can tell the GeoModel code is quite expensive (up to tens of thousands of dollars a year) to run since Google updated the pricing of App Engine. It doesn't seem like App Engine's database is really suited to this kind of query (though the SQL solution may be an answer).
I was hoping to find a service where I could send off a HTTP request to add data (a business' id, name and icon url) to a database, and then another one to find a list of businesses that are nearby to a given point. A service is preferable as this is work done for a client and we would like the solution to be managed with as little interaction from us or the client needed as possible.
EDIT:
I just found cartodb.com which uses PostgreSQL and is reasonably priced. Are the any other alternatives?
The App Engine Search API (currently in Experimental) supports GeoPoints and geosearch, and is great for exactly the kind of query that you describe.
See the Google Developers Academy (GDA) App Engine Search API classes for a bit more info and an example as well.
http://www.iriscouch.com/ is a cloud-based host for CouchDB and they support the geocouch extensions for CouchDB to store geoJSON data and perform spatial queries.
We have decided to go with cartodb.com because it looks like they have a good price to ease of use ratio.
You mentioned going with CartoDB, which is a good choice with a nice UI.
Just adding, if you were just looking for a scalable backend, you could use StormDB. It is a cloud hosted SQL database with geospatial extensions. You data is automatically distributed amongst multiple nodes for write, read, and parallel query scalability.
I was just wondering how many db queries might facebook be issuing to render a user's home page. Does anybody have some idea on how the facebook DB is designed. I've heard it runs MySql and there are thousands of replica plus more memcache server than DB Servers.
Is the facebook data shard-ed?
If it is does it go to every shard and search for the latest update of my friend. In worst case if I've 100 friends and suppose facebook has 101 shards, there is a possibility that each of my friend is in a different shard. How might facebook be handling this?
I'll be highly grateful if somebody can provide me seom hints or pointers towards something like "How to Design a DB for Social Networking Website". I'm just curious!
Facebook is using LAMP structure. Facebook’s backend services are written in a variety of different programming languages including C++, Java, Python, and Erlang and they are used according to requirement. With LAMP Facebook uses some technologies ,to support large number of requests, like
Memcache - It is a memory caching system that is used to speed up dynamic database-driven websites (like Facebook) by caching data and objects in RAM to reduce reading time. Memcache is Facebook’s primary form of caching and helps alleviate the database load. Having a caching system allows Facebook to be as fast as it is at recalling your data.
Thrift (protocol) - It is a lightweight remote procedure call framework for scalable cross-language services development. Thrift supports C++, PHP, Python, Perl, Java, Ruby, Erlang, and others.
Cassandra (database) - It is a database management system designed to handle large amounts of data spread out across many servers.
HipHop for PHP - It is a source code transformer for PHP script code and was created to save server resources. HipHop transforms PHP source code into optimized C++. After doing this, it uses g++ to compile it to machine code.
If we go into more detail, then answer to this question go longer. We can understand more from following posts:
How Does Facebook Work?
Data Management, Facebook-style
Facebook database design?
Facebook wall's database structure
Facebook "like" data structure
At this website you find lots of details about those big internet companies and their technical structures:
http://highscalability.com/
Adding to #Somnath Muluk's answer - Facebook uses few other technologies like Hadoop etc.
Refer to the following links for more details:
http://www.quora.com/Facebook-Engineering/What-is-Facebooks-architecture
Facebook Architecture
Hope it helps.
Zero. On average, that is. A highly interconnected network with a large number of users such as Facebook can only run effectively if it runs fully out of ram for the pages that are shown often. Nearly all data should already be in the memcache.