our company is seeking for the time-series data resolution , I don't understand the specific advantages between TDengine and ClickHouse. does anyone have the performance benchmark ?
hope to get a specific comparison between these two database , I need to choose a database for my company in IOT scenario .
Related
How to plan hardware resources for TDengine database
does it have a Common configuration?
I have 300000 tables around.
You may consider about below questions:
Cloud service or local server is your preferred?
The machine hardware configuration for the server and the cost.
High avalibity is optional.
Actually, there's not a regular answer for this specific question. But 30000 tables will always be well handled by TDengine database.
It is 300000 but not 30000 tables ,haha
Anyway ,for this level amounts of tables , TDengine database could handle it easily .
but it also depends on the structure of your table and requirement of using it ,and the performance you expect
It would be better if if you could provide those information above.
I'm seeking for a solution for a time-series database that will hold measurement data of IoT devices used by multiple tenants.
Each tenant have their own device types that sends different measurement data (for example pressure, temperature, etc.)
After some research I arrived an SQL solution with TimeScaleDB for postgreSQL and dynamically creating a new SQL table for each device type - each with it's own set of columns.
Assuming I might have 1000 tenants, each with a few device types, is it a good design to have a database with thousands of tables?
As far as I understand, postgreSQL has a limitation of 8KB page size - so even in minimal cases where only 1 device is sending data for a month, I will most probably exceed that - which is a good thing for the multi-table design.
Thanks.
I am working on a project which is highly performance dashboard where results are mostly aggregated mixed with non-aggregated data. First page is loaded by 8 different complex queries, getting mixed data. Dashboard is served by a centralized database (Oracle 11g) which is receiving data from many systems in realtime ( using replication tool). Data which is shown is realized through very complex queries ( multiple join, count, group by and many where conditions).
The issue is that as data is increasing, DB queries are taking more time than defined/agreed. I am thinking to move aggregated functionality to Columnar database say HBase ( all the counts), and rest linear data will be fetched from Oracle. Both the data will be merged based on a key on App layer. Need experts opinion if this is correct approach.
There are few things which are not clear to me:
1. Will Sqoop be able to load data based on query/view or only tables? on continuous basis or one time?
2. If a record is modified ( e.g. status is changed), how will HBase get to know?
My two cents. HBase is a NoSQL database build for fast lookup queries, not to make aggregated, ad-hoc queries.
If you are planning to use a hadoop cluster, you can try hive with parquet storage formart. If you need near real-time queries, you can go with MPP database. A commercial option is Vertica or maybe Redshift from Amazon. For an open-source solution, you can use InfoBrigth.
These columnar options is going to give you a greate aggregate query performance.
I intend to create a 'data ware house' that integrates several sql server 2008 databases. This integration solution will only have a life time of 12 months or so. Hence, I do not want to spend lots of time creating a star schema etc. Still the 'data ware house' database will contain some conforming dimensions. Is it worth replicating the source databases in the 'data ware house' or shall I just create some cross database views + stored procedures + conformed dimensions in the 'data ware house'? Any feedback would be appreciated.
it depends what you're planning to do with the warehouse, usually warehouses created to look for historical data and run analytics on them to get the trends and problems that occured historically and to get these analysis out of it, it runs big queries which does lot of aggregation on lots of data. If you just create some across database views than it will work on your OLTP data and it might degrade the performance of your main running system.
Pro's for DataWarehouse:
1. you can view historical data without affecting main system
2. you can generate trends out of it with out degrading main system performance
3. Since all historical data is in warehouse, you can lighten your main DB which will increase performance on the main app
4. you can merge all db's into one
cons for Datawarehouse
1. It take some hit on your main db when you copying the records to the warehouse and if there are frequent updates on the historical records, then you have to keep up the warehouse with that
2. It is not supposed to be used for real time queries
Given a .NET environment with Windows CE, can you persist thousands of records per second in a local database (SQL Server 2008 - standard or CE).
What are the performance issues with persisting realtime instrument data in a database versus a log file?
SQL Server 2008 standard is more than capable of those insertion rates PROVIDED you have hardware capable of supporting it.
The question you really need to be asking is do I require the ability to search the captured data quickly?
This SO answer might be of interest: What does database query and insert speed depend on?
The number (and width) of indexes on a table will obviously have an impact on insertion rate.
If you are considering open-source, then MySQL is often cited as being able to handle high volumes.