I have to connect multiple tables that are part of single or multiple databases. Approximately 10-15 tables in each query have to be connected to generate data for the analysis in SQL Server 2014.
I don't have access to the database diagram or architecture and these reports are to be sent out weekly. I want to understand the approach on how to begin writing these kind of queries which are of basic and advanced level and identify the relationship between tables and what kind of advanced level queries I can learn or utilize like CTE, Rank Partition, Subqueries etc.
Anybody who can provide a rough flow diagram or structure about the approach will be really helpful.
It's very unlikely that owners of those source systems want to be directly queried every time someone runs a report. Since you already have access to SQL Server, I would suggest building a data warehouse with that.
You haven't provided a whole lot of information to go on, but SSIS packages could be created to connect to the source systems and load into your data warehouse. And furthermore, those packages can be scheduled through Agent.
As for modeling... Again it is difficult with the lack of information, but generally the star model works great for reporting, which is a fact table surrounded by dimension (or attribute) tables.
As for figuring out relationships without a diagram, this will have to be done via experimentation and tieing to existing reports to make sure your joins aren't dropping records or cascading.
Good luck.
Two different relational databases.
Your task is to write a code to transfer the data from the first database to the second database.
Some tables in the database you are transferring to are of the same structure as the table you are transferring from, the transfer of these tables is as simple as "INSERT INTO DbA.TableA (...) VALUES SELECT * FROM DbB.TableB".
Some tables in the database you are transferring to have different structures and different purposes. After proper analysis, you understand the relations and you understand the right transformation you need to code.
My question is: how do you express such knowledge? How do you express the transformational relations between two databases? Are there any tools or diagrams?
The best way I know right now is writting the list of tables of the first database and for each table describing how it is to be transformed into the second database. Is it possible to make this more formal/concise/cool?
If you are wanting a toolset and work in the Microsoft database stack then this is exactly what SQL Server Integration Services (or SSIS) is used for.
If you are wanting to document the process then you would typically write an interface definition document (IDD). There are many examples on Google but here is something to get you started.
I have a question about data warehousing and column oriented databases. In my project the company use a warehouse solution in visual studio SQL server, they have troubles with the performance when querying complex questions on large amount of data. I want to try to replace the database with a columnar based database. I know that you can "transform" a row oriented database in to more column based or use an open source database such as Vertica or Sybase IQ, i just wondering how it would fit in the warehouse? Do you have to have a star join schema in a warehouse or can you use the columnar approach instead, i realize this is kind of a stupid question but im just trying to understand it all before i start to explore the different databases and solutions.
I know that SQL Server 2012 have a column store but i would like to try the other open source databases as well.
Thanks in advance!
Do you have to have a star join schema in a warehouse or can you use the columnar approach instead?
The star join schema consists of the table definitions of your data warehouse. The star schema, and similar schema, trade query performance for query flexibility. Usually, query flexibility is more important than query performance in a data warehouse.
Based on the Wikipedia article you linked to in your comments, a column oriented database engine stores the actual database bytes in column order, rather than the traditional row order of relational databases.
As the article says, this can improve disk access performance.
The star schema is how you define tables. A column oriented database engine is concerned with how the database information is written to disk. The two concepts have nothing to do with one another, except that they both apply to a data warehouse.
Keep your present data warehouse schema, and see if a column oriented database engine will improve query performance.
What is the difference between Database and Data Source?
A data source is simply something your program relies on to get data. A database is a kind of data source that persists data to some digitized form. Other data sources include files, services, etc — these all provide data to your programs.
Further to BoltClock's answer, here are example of Databases (or Database Servers) vs various Data Sources.
Databases
SQL
Oracle
MySQL
Data Sources
All of the databases above
XML Files
CSV Files
Web Services
and many many more..
Data Source may not be connected to DB, it can be just file system or any source of data.
To quote the description from Techopedia:
A data source, in the context of computer science and computer
applications, is the location where data that is being used come from.
In a database management system, the primary data source is the
database, which can be located in a disk or a remote server. The data
source for a computer program can be a file, a data sheet, a
spreadsheet, an XML file or even hard-coded data within the program.
In a simple word, I will try to answer this question.
First, we need to understand data sources, Data source is something from where we get data to analyze or a place where data is stored. Kinds of different data sources are:
1. Databases
2. Flat files, Excel sheets, Spread Sheets
3. Web Services, Etc.
Now come to the database as in the above examples we can see there is mentioned database, which is one of the kinds of data sources. In databases companies stores their collections of records, responses, survey, etc. In databases there are two types:
1. DBMS: Database management system
2. RDBMS: Relational Database Management system
Data --- image, video, file , pdf,msg, name, age, height, weight, etc
Database is a collection of data (contact number, best friend names, shopping list is a collection of data) stored in a formet Systematically that can be easily accessed.
Example of database: attendance register
Attendance register of employees in office
Attendance register of students in school
Attandance register is a database
Database store in computer , mobile , tables, excel, folders etc
Types of database: Network Database, Object-Oriented Database, Relational Database, Hierarchical Database like attendance register, attendance page, attendance darry.
Database Management System (DBMS)------- A database management system (DBMS) is a software package create, manipulate, retrieve and manage data in a database .A DBMS generally manipulates the data itself, the data format, field names, record structure and file structure. It also defines rules to validate and manipulate this data. example of DBMS---> HR, Teacher.
Hr, Teacher maintain register same as Dbms maintain database ...
________database create ________
| |
MANUALLY Electrically
(Hands) ( computer,mobilephones etc)
using pen, paper using DBMS software, File system etc
A DATABASE store the data and provide a method to access it, a DBMS actually converts the queries into a meaningful command, to invoke the method used to access the database.
Some other DBMS (teacher, HR) examples include:
MySQL ( example ->eng teacher)
SQL Server ( hindi teacher)
Oracle ( evs teacher)
dBASE ( senior teacher)
FoxPro ( math teacher)
All teacher maintain attendance register same as all this DBMS maintain database..
Principal or head decide which teacher is create and maintain register same as developer decide which DBMS( my sql, oracle) is create and maintain database and which is best.
Find rohan total attendance?
Find rohan total attendance --------> HR/Teacher -----------> Attandance register
Find rohan total attendance------> DBMS(my sql) --------->database
SQL (Structured query language)
NOSQL
SQL--------- SQL stands for Structured Query Language. SQL is used to communicate with a database
it is the standard language for relational database management systems. SQL statements are used to perform tasks such as update data on a
database, or retrieve data from a database. Some common DBMS( Oracle, Sybase, Microsoft SQL Server, Access, Ingres, My sql, Oracle,Sqlite) that use
Sql . Sql database is tablebase database like Ms excel, vertically data store, relational database.
In NOSQL data is store in key value, pair like json...NOSQL used by---Redis , MongoDB. example ofsql , sql is class monitor or head student of class
that is help to the DBMS( Teacher, Hr) to manage Database (Register)... one class monitor helps all teacher same as one SQL Used by many dbms
(My sql , oracle )
Purpose .. To query and operate database system.
SQL use by NOSQL use by
1.My sql 1.Redis
2.Ms-sql 2.MongoDB etc
3. oracle etc
| |
|_______________DBMS__________________|
What is the best method to transfer data from sales table to sales history table in sql server 2005. sales history table will be used for reporting.
Take a look at SSAS. OLAP is built for reporting and is easy to query with tools like excel pivot tables.
Bulkcopy is fast and it will not use the transaction log. One batch run at the end of the day.
Deleting the copied records from your production server is a different situation that needs to be planed on that server's maintenance approach/plans. Your reporting server solution should not interfere with or affect the production server.
Keep in mind that your reporting server is not meant to be a backup of the data but rather a copy made exclusively for reporting purposes.
Also check on the server settings of your reporting server to be on Simple recovery model.
Most solutions will require 2 steps;
-copy the records from source to target
-delete records from source.
It is essential that your source table have a primary key.
The "best" method depends on a lot of things.
How many records?
Is this a production environment?
What tools do you have?
Unless you are moving a large amount of data, a simple stored procedure should do the trick.
A sql server job can manage the timing of when to call the proc.
if you just want to move the data to another table, use BulkCopy/BulkInsert. if you want to build reporting I would suggest a BI solution such as MS Analysis Service (OLAP).
It is difficult and in my opinion ugly to maintain two or more history/archive tables in the same database. For a reporting solution you will be considering all the tables for that piece of information anyway. History/Archive tables should only be used if you are going to put the data away and not touch it for a long period of time, ie. archive it away outside the operational DB.