I am building a blog on a website and I want to use JSON objects to store the blog post data because it's nicer than documents.
I read somewhere that Firebase Real-Time Data storage (uses a nice JSON format) is optimal for a limited geographic area. Does this mean that at further distances it will still work but with more latency, or will it not work at all?
Or is it recommended for me to just use the document style Cloud Firestore or MongoDB?
Realtime Database will work anywhere there is an internet connection. The servers are currently hosted in the United States, so the further away you get from there, the lower latency you will probably observe. It all depends on the quality of the network connection between the client and server.
Firestore has several regions to select from, if that helps your users. You should benchmark with actual data though.
Related
I am just starting to set up a project to keep track of some open, home devices that are enabled for an at home network. I have a program that saves this data, and am putting together a process to upload that data to Snowflake automatically. I would like to know what you would recommend so I can easily access the home device information from anywhere.
The two options I am considering are aws's and snowflake's auto ingest option using the snowpipe rest api, which I have tested with only a few devices.
I am considering these two factors - which method can I set up to upload and select data quickly from a mobile app written in python or ruby depending on the device.
Any advice or resources you can point me to on this?
Thank you!
Your question is a pretty open question, so details from you might make this answer a bit more detailed, as well. However, in general, I would suggest that if your IoT data can be stored directly to Blob Storage (S3 in the case of AWS), then you should leverage Snowflake's Snowpipe for continuous ingestion. Also, look into Tasks and Streams to automate moving that data through whatever processes you'll setup once the data is in Snowflake.
A good reference for you:
https://docs.snowflake.net/manuals/user-guide/data-pipelines-intro.html
Can we use hazel-cast database to link and design the data according to tracker with bar graph, below are the points which I need to confirm to build the application for hardware:
- I am using temperature sensor interfacing with Arduino Yun and wanted to upload the data given by temperature sensor on hazel-cast server.
By using single database output uploaded in hazelcast server, reads the data through database through Arduino MKR1000.
Link the data to different development tools to design different types of dashboards like Pie chart, Bar chart, Line chart etc.
Please suggest how the best way to link to create the database in data-grid
How you want to use data on your dashboard will basically depend on how you have modelled your data - one map or multiple maps etc. Then you can retrieve data through single key-based lookups or by running queries and use that for your dashboard. You can define the lifetime of the data - be it few minutes or hours or days. See eviction: http://docs.hazelcast.org/docs/3.10.1/manual/html-single/index.html#map-eviction
If you decide to use a visualisation tool for dashboard that can use JMX then you can latch on to Hazelcast exposed JMX beans that would give you information about data stored in the cluster and lot more. Check out this: http://docs.hazelcast.org/docs/3.10.1/manual/html-single/index.html#monitoring-with-jmx
You can configure Hazelcast to use a MapLoader - MapStore to persist the cached data to any back-end persistence mechanism – relational or no-sql databases may be good choices.
On your first point, I wouldn’t expect anything running on the Arduino to update the database directly, but the MKR1000 is going to get you connectivity, so you can use Kafka/MQTT/… - take a look at https://blog.hazelcast.com/hazelcast-backbone-iot-internet-things/.
If you choose this route, you’d set up a database that is accessible to all cluster members, create the MapLoader/MapStore class (see the example code, for help) and configure the cluster to read/write.
Once the data is in the cluster, access is easy and you can use a dashboard tool of your choice to present the data.
(edit) - to your question about presenting historical data on your dashboard:
Rahul’s blog post describes a very cool implementation of near/real-time data management in a Hazelcast RingBuffer. In that post, I think he mentioned collecting data every second and buffering two minutes worth.
The ring buffer has a configured capacity, but note that he is over-writing, on add - this is kind of a given for real-time systems; given the choice is losing older data or crashing.
For a generalized query-tool approach, I think you’d augment this. Off the top of my head, I could see using the ring-buffer in conjunction with a distributed map. You could (but wouldn’t need to) populate the map, using an map-event interceptor to populate the ring buffer. That should leave the existing functionality intact. The map, though, would allow you to configure a map-store/map-loader, so that your data is saved in a backing store. The map would support queries - but keep in mind that IMDG queries do not read through to the backing store.
This would give you flexibility, at the cost of some complexity. The real-time data in the ring buffer would be always available, quickly and easily. Data returned from querying the map would be very quick, too. For ‘historical’ data, you can query your backing-store - which is slower, but will probably have relatively great storage capacity. The trick here is to know when to query each. The most recent data is a given, with it’s fixed capacity. You need to know how much is in the cluster - i.e. how far back your in-memory history goes. I think it best to configure the expiry to a useful limit and provision the storage so that data leaves the map by expiration - not eviction. In this way, you can know what the beginning of the in-memory history is. Monitoring eviction events would tell you that your cluster has a complete view of data back to a known time.
I have an offline ready application that I am currently building in electron.
The core requirements are that all data is restricted (have to be a user to read or write) and that within that data some data is further restricted to a user, (account information, messages, etc...)
Now I do not want to replicate any data offline that a user should not have access to (this is because all the data can be seen using the devtools regardless of restriction) so essentially I only want to sync data to PouchDB's offline store if that user has access to it as well as all the data all users have access to.
Now I have read the following posts/guides but I am still a little confused.
https://pouchdb.com/2015/04/05/filtered-replication.html
https://www.joshmorony.com/creating-a-multiple-user-app-with-pouchdb-couchdb/
Restricting Access to local PouchDB
From my understanding filtering is a bad choice performance wise even though it could do what I want.
Setting up a proxy would work but it then essentially becomes a REST api and the data synchronization falls apart.
And the final option which I think is what I want is to have a database for every user that would contain their private information and then additional databases to hold the information that is available to every user.
The only real question I have with this approach is how is data handled that is private but shared between two users (messages, etc...)
I am more after an overarching view of how the data should be stored as opposed to code examples, just really struggling with the conceptual architecture of the application.
There are many solutions to your problem. One solution looks very promising: IBM Cloudant has started work on Cloudant Envoy, a proxy simulating the CouchDB interface instead of a simple REST API. You can read more about it on the site for Envoy over at ibm.com. A custom replicator for PouchDB is also available on Github.
There's also a blog post on Medium.com on this.
The idea is the same as the much older Couchbase Sync Gateway. Although Couchbase has common roots with CouchDB, I have not tracked if they still support replication with CouchDB.
The easiest way to start would be to create a single database per user on the server, and a common database that you just pull the shared data from. Let me know if you need more info on this solution.
Can data be permanently stored in Persistent storage, such that anytime the applications loads I can get the last inserted item?
If yes, can I get a code sample or a link to some examples.
Of course! That's how modern databases work.
My advice is to consider storing your data as JSON in MongoDB for maximum portability:
MongoDB is an open-source document database that provides high
performance, high availability, and automatic scaling.
I highly recommend mlab, who offer MongoDB as a service - you can host up to 500MB on one or more databases. Follow the steps in their getting started documentation (5-10 mins). Then download Robomongo and configure it with the connection settings from mlab; you'll be able to view all of your databases, collections (aka tables) and documents (aka datums), and interact with them using either basic point and click, or programmatically using its Mongo Shell interface.
See the developer guide section on storage where the various types of storage options are discussed in depth. There are many samples in that section but you will need to clarify a more exact use case of what you are trying to accomplish.
If you just want to save one variable you can use something like:
Preferences.set("var", val);
String var = Preferences.get("var", defaultValue);
Guys just a quick question,
I am creating a WEB application in my company. But it's a on the local intranet. So we don't have access to the Internet.
Is there any way of having maps (google, bing, etc) without internet?
Thank you
UPDATE
I found this:
https://www.mapbox.com/mapbox.js/api/v2.1.2/
I'll try and then let you know
There are a couple of offline mapping platforms like MapPoint and Microsoft Streets and Trips but they have recently been retired. Mapping data is large. A single copy of the data in Bing Maps is measured in Petabytes. Offline is just not an option in most cases.
It is possible to create an offline mapping platform and I've helped companies do this in the past, but in most cases limits had to be set, such as limiting to a specific area, or a limited number of zoom levels.