We know that Snowflake is a compressed columnar storage database and tuned to run queries with MPP and auto scaling. We also know that for creating data marts and DW, Kimball and Dimensional modelling(Star Schema) has been in market and practice from decades now. This was a success due to the massive Row store DBs we used to have for our DWs.
So the question here is for creating Data Marts and DW in Snowflake, do we have to follow Kimball ? Does it add any value to the performance, in fact I read that it add overhead for the engine that is already tuned to work on columnar compressed data? Do we still need to use surrogate keys for columns and force to create Facts and Dimensions and star schema, where we can simply join the flat denormalized tables to get the similar or better performance ?
What does superpower DBs like Snowflake recommend from a best practice point of view for Modelling? Is Kimball must to have or redundant as it defeats the purpose of columnar storage benefits ?
I think SAP HANA / Redshift / Big Query or even Azure SQL Datawarehouse, no one recommend this and I could not find anywhere a single line that recommend to use Kimball or star schema. Few does mention that, “It also works for Star schema” which does not mean that Star schema has to be used ?
One thing to keep in mind: Snowflake is a row-oriented, columnar store. That's an important distinction. This means that Snowflake takes advantage of all of the significant compression gains associated with columnar storage but still maintains the row-oriented approach to storing data.
Why does this matter?
With the micro-partition approach, it means we can still eliminate large numbers of rows using query predicates and then interrogate only the column stores within those row groupings that met the query's criteria. So you really get the best of both worlds.
In my opinion, Snowflake can support just about any data model (or partial/hybrid implementation).
Also - "redundant" values in a row-oriented, column store tend to lead to very, VERY good compression.
i have a question why operational database are not fulfilling business challenges as data warehouse?
in operational database i can create reports in details about any product or any thing and i can issue statistical reports with charts and diagrams, so why the operational database can not use as data warehouse?
Best Regards
Usually an operational database only keeps track of the current state of each record.
The purpose of a data warehouse is two-fold:
- Keep track of historic events without overwhelming the operational database;
- Isolate OLAP queries so that they don't impact the load on the operational datastore.
If you try to query your operational data store for sales per product line per month for the past year the amount of joins required, as well as the amount of information you need to read from storage may cause performance degradation on your operational database.
A data warehouse tries to avoid this by 1) keeping things separated and 2) denormalising the data model (Kimball approach) so that query plans are simpler.
I suggest reading The Data Warehouse Toolkit, by Ralph Kimball. The first chapter deals precisely with this question: why do we need a data warehouse if we already have an operational data store?
i can create reports in details about any product or any thing
and i can issue statistical reports with charts and diagrams
Yes you can but a business user can not as they don't know SQL. And you it's very difficult to put a BI tool (for a business users to use) over the top of an operational database for many reasons:
The data model is not built for an end user to understand. A data warehouse data model is (i.e. there is ONE table for customers that has everything about a customer in it rather than being split into addresses, accounts etc.)
The operational data store is probably missing important reporting reference data such as grouping levels and hierarchies
A slowly changing dimension is a method of 'transparently' modelling changes to for example, customers. An operational data model generally doesn't do this very well. You need to understand all the table and join them correctly, if this information is even stored
There are many other reasons but these just serve to address your points
When you get to the point that you are too busy to service business users requests, and you are issuing reports that don't match from one day to the other, you'll start to see the value of a data warehouse.
Can an enterprise use both data warehouse and database in one Head Office? Is it just OK to use only one of these or is it necessary to use both in the same place?
Yes, an enterprise can use both data warehouse and database in one office. They need not be in the same physical data-center. It all depends on the needs of the organization. Generally, databases are used to support transactions as they happen and data warehouses are used to support business intelligence or the like.
Database
Transactions in an enterprise is most likely to happen in a relational database management system (aka database, aka RDBMS). Reporting can happen using the same database, but it is also possible that reporting is done off of a mirror of the RDBMS. Now, an enterprise may have more than one RDBMS - one running SQL Server, one running Oracle, one running MySQL etc. All this is great for recording activities and reporting.
Warehouse
Additionally, enterprises seek to do data analysis on a regular basis. Business Intelligence, data science, big data - regardless of the term, we are talking about data analysis overall. Doing number crunching on large amounts of data stored in an RDBMS can be hard on the RDBMS. So, organizations decide to de-normalize data to some extent and store data in a warehouse. When data is extracted, transformed and loaded (ETL) from one or more RDBMS (and other sources of data) and stored in a data warehouse, it is available for some research.
Organizations may choose to move the warehouse to a different office location, or may have multiple-warehouses. For example, a headquarter with 5 satellite facilities may choose to bring data from all those facilities to the warehouse at the headquarter every night, or it may choose to have a warehouse in a different datacenter. In contrast to that, a company with hundreds of satellite facilities may choose to have a warehouse with high-level summarized data at their headquarter and regionalize their warehouses; one warehouse in each continent, so that target markets are better served by satellite units in that particular continent.
Database (or databases) to Warehouse journey
Business Intelligence
Cognos, QlikView, Tableau, Microstrategy etc. are some business intelligence/data analytic tools among many that reach out to the data warehouse and present data for analytics. They are great for presentation and reporting - data visualization, in general. These tools can also get data from RDBMS, but it's convenient to get it from data warehouse since they are architected in a way to make it easier to showcase that data on a business intelligence dashboard
Example of a dashboard:
Big data
The buzzword around big data is interesting. Many of us may take a subset of data from a large pool of data, do analysis and assume that the results from the subset applies to the large pool of data. What if all the data was used for analysis? And even better - what if we took related data from elsewhere (outside of our dataset) and included it in our analysis? Yeah, you would have a giant pile of data and if you had means to analyze them all, you'd be doing big data. We are talking several hundred GB or even PB of data. Although Hadoop and the like are used in big data analysis, they could derive that data from the warehouse.
I've been reviewing a client's architecture, particularly their OLAP system, which is just a regular old snowflake schema on SQL Server. The facts and dimensions are ETL'd in from other transactional systems such as ERP.
One thing that jumped out at me was several additional tables, in the same database, for multiple additional OLTP applications. These tables have FK relationships to dimension tables in the snowflake schema.
There are a lot of joins into the dimension data from the OLTP system, so performance is not the best.
I am not an OLAP expert at all; but this just feels wrong. I've done some searching but can't find much about this on the internet either pro or con. What are the benefits of doing this? Are there any? What about potential problems?
I would try to avoid any explicit foreign keys between OLTP and OLAP data. Having foreign keys from OLTP to OLAP prevents you from adding new entities in the business and may require to define entities in OLAP first, while the standard is to run the ETL processes one-directional only - always from OLTP o OLAP. And having foreign keys from OLAP to OLTP prevents you from keeping historical data in the data warehouse that is not relevant for the current business, but may be interesting for analysis.
Of course, there are always situations where you break rules for a reason. Maybe there is one. Does someone at the client's side have an explanation why this was implemented the way you describe?
It is not common to share a dimension table between OLTP and OALP. There are at least 2 reasons: (1) the attributes interesting in OLTP and in OLAP may be quite different. (2) the contention and consequent performance problem.
On the other hand, it is not uncommon (but is somewhat advanced) for OLTP and an ODS to share exactly the same copy of a dimension. This is often called a "golden copy". I often call an ODS designed like this to be an active ODS. When there are multiple copies of the dimension, I call it a passive ODS. It may be that the OLAP you refer to is not true OLAP but just some form of tactical reporting, in which case sharing the same dimension table is not uncommon.
What is the difference between a database and a data warehouse?
Aren't they the same thing, or at least written in the same thing (ie. Oracle RDBMS)?
Check out this for more information.
From a previous link:
Database
Used for Online Transactional Processing (OLTP) but can be used for other purposes such as Data Warehousing. This records the data from the user for history.
The tables and joins are complex since they are normalized (for RDMS). This is done to reduce redundant data and to save storage space.
Entity – Relational modeling techniques are used for RDMS database design.
Optimized for write operation.
Performance is low for analysis queries.
Data Warehouse
Used for Online Analytical Processing (OLAP). This reads the historical data for the Users for business decisions.
The Tables and joins are simple since they are de-normalized. This is done to reduce the response time for analytical queries.
Data – Modeling techniques are used for the Data Warehouse design.
Optimized for read operations.
High performance for analytical queries.
Is usually a Database.
It's important to note as well that Data Warehouses could be sourced from zero to many databases.
From a Non-Technical View:
A database is constrained to a particular applications or set of applications.
A data warehouse is an enterprise level data repository. It's going to contain data from all/many segments of the business. It's going to share this information to provide a global picture of the business. It is also critical to integration between the different segments of the business.
From a Technical view:
The word "Data Warehouse" has been given no recognized definition. Personally, I define a data warehouse as a collection of data-marts. Where each data-mart consists of one or more databases where the database is specific to a specific problem set (application, data-set or process).
Simply put a database is a component of a data-warehouse. There are many places to explore this concept, but because there is no "definition", you will find challenges with any answer you give.
A data warehouse is a TYPE of database.
In addition to what folks have already said, data warehouses tend to be OLAP, with indexes, etc. tuned for reading, not writing, and the data is de-normalized / transformed into forms that are easier to read & analyze.
Some folks have said "databases" are the same as OLTP -- this isn't true. OLTP, again, is a TYPE of database.
Other types of "databases": Text files, XML, Excel, CSV..., Flat Files :-)
The simplest way to explain it would be to say that a data warehouse consists of more than just a database. A database is an collection of data organized in some way, but a data warehouse is organized specifically to "facilitate reporting and analysis". This however is not the entire story as data warehousing also contains "the means to retrieve and analyze data, to extract, transform and load data, and to manage the data dictionary are also considered essential components of a data warehousing system".
Data Warehouse
Data Warehouse vs Database: A data warehouse is specially designed for data analytics, which involves reading large amounts of data to understand relationships and trends across the data. A database is used to capture and store data, such as recording details of a transaction.
Data Warehouse:
Suitable workloads - Analytics, reporting, big data.
Data source - Data collected and normalized from many sources.
Data capture - Bulk write operations typically on a predetermined batch schedule.
Data normalization - Denormalized schemas, such as the Star schema or Snowflake schema.
Data storage - Optimized for simplicity of access and high-speed query. performance using columnar storage.
Data access - Optimized to minimize I/O and maximize data throughput.
Transactional Database:
Suitable workloads - Transaction processing.
Data source - Data captured as-is from a single source, such as a transactional system.
Data capture - Optimized for continuous write operations as new data is available to maximize transaction throughput.
Data normalization - Highly normalized, static schemas.
Data storage - Optimized for high throughout write operations to a single row-oriented physical block.
Data access - High volumes of small read operations.
DataBase :-
OLTP(online transaction process)
It is current data, up-to-date detailed data, flat relational
isolated data.
Entity relationship is used to design the database
DB size 100MB-GB simple transaction or quires
Datawarehouse
OLAP(Online Analytical process)
It is about Historical data Star schema,snow flexed schema and galaxy
schema is used to design the
data warehouse
DB size 100GB-TB Improved query performance foundation
for DATA MINING DATA VISUALIZATION
Enables users to gain a deeper understanding and knowledge about various
aspects of their corporate data through fast, consistent, interactive access
to a wide variety of possible views of the data
Any data storage for application generally uses the database. It could be relational database or no sql databases which are currently trending.
Data warehouse is also database. We can call data warehouse database as specialized data storage for the analytical reporting purposes for the company.
This data used for key business decision.
The organized data helps is reporting and taking business decision effectively.
Database:
Used for Online Transactional Processing (OLTP).
Transaction-oriented.
Application oriented.
Current data.
Detailed data.
Scalable data.
Many Users, Administrators / Operational.
Execution time: short.
Data Warehouse:
Used for Online Analytical Processing (OLAP).
Oriented analysis.
Subject oriented.
Historical data.
Aggregated data.
Static data.
Not many users, manager.
Execution time: long.
A Data Warehousing (DW) is process for collecting and managing data from varied sources to provide meaningful business insights. A Data warehouse is typically used to connect and analyze business data from heterogeneous sources. The data warehouse is the core of the BI system which is built for data analysis and reporting.
Source for the Data warehouse can be cluster of Databases, because databases are used for Online Transaction process like keeping the current records..but in Data warehouse it stores historical data which are for Online analytical process.
A Data Warehouse is a type of Data Structure usually housed on a Database. The Data Warehouse refers the the data model and what type of data is stored there - data that is modeled (data model) to server an analytical purpose.
A Database can be classified as any structure that houses data. Traditionally that would be an RDBMS like Oracle, SQL Server, or MySQL. However a Database can also be a NoSQL Database like Apache Cassandra, or an columnar MPP like AWS RedShift.
You see a database is simply a place to store data; a data warehouse is a specific way to store data and serves a specific purpose, which is to serve analytical queries.
OLTP vs OLAP does not tell you the difference between a DW and a Database, both OLTP and OLAP reside on databases. They just store data in a different fashion (different data model methodologies) and serve different purposes (OLTP - record transactions, optimized for updates; OLAP - analyze information, optimized for reads).
See in simple words :
Dataware --> Huge data using for Analytical/storage/ copy and Analysis .
Database --> CRUD operation with Frequently used data .
Dataware house is Kind of storage which u are not using on daily basis & Database is something which your dealing frequently .
Eg. If we are asking statement of bank then it gives us for last 3/4/6/more months bcoz it is in database. If you want more than that it stores on Dataware house.
Example: A house is worth $100,000, and it is appreciating at $1000 per year.
To keep track of the current house value, you would use a database as the value would change every year.
Three years later, you would be able to see the value of the house which is $103,000.
To keep track of the historical house value, you would use a data warehouse as the value of the house should be
$100,000 on year 0,
$101,000 on year 1,
$102,000 on year 2,
$103,000 on year 3.