What is the difference between Database and Data Source?
A data source is simply something your program relies on to get data. A database is a kind of data source that persists data to some digitized form. Other data sources include files, services, etc — these all provide data to your programs.
Further to BoltClock's answer, here are example of Databases (or Database Servers) vs various Data Sources.
Databases
SQL
Oracle
MySQL
Data Sources
All of the databases above
XML Files
CSV Files
Web Services
and many many more..
Data Source may not be connected to DB, it can be just file system or any source of data.
To quote the description from Techopedia:
A data source, in the context of computer science and computer
applications, is the location where data that is being used come from.
In a database management system, the primary data source is the
database, which can be located in a disk or a remote server. The data
source for a computer program can be a file, a data sheet, a
spreadsheet, an XML file or even hard-coded data within the program.
In a simple word, I will try to answer this question.
First, we need to understand data sources, Data source is something from where we get data to analyze or a place where data is stored. Kinds of different data sources are:
1. Databases
2. Flat files, Excel sheets, Spread Sheets
3. Web Services, Etc.
Now come to the database as in the above examples we can see there is mentioned database, which is one of the kinds of data sources. In databases companies stores their collections of records, responses, survey, etc. In databases there are two types:
1. DBMS: Database management system
2. RDBMS: Relational Database Management system
Data --- image, video, file , pdf,msg, name, age, height, weight, etc
Database is a collection of data (contact number, best friend names, shopping list is a collection of data) stored in a formet Systematically that can be easily accessed.
Example of database: attendance register
Attendance register of employees in office
Attendance register of students in school
Attandance register is a database
Database store in computer , mobile , tables, excel, folders etc
Types of database: Network Database, Object-Oriented Database, Relational Database, Hierarchical Database like attendance register, attendance page, attendance darry.
Database Management System (DBMS)------- A database management system (DBMS) is a software package create, manipulate, retrieve and manage data in a database .A DBMS generally manipulates the data itself, the data format, field names, record structure and file structure. It also defines rules to validate and manipulate this data. example of DBMS---> HR, Teacher.
Hr, Teacher maintain register same as Dbms maintain database ...
________database create ________
| |
MANUALLY Electrically
(Hands) ( computer,mobilephones etc)
using pen, paper using DBMS software, File system etc
A DATABASE store the data and provide a method to access it, a DBMS actually converts the queries into a meaningful command, to invoke the method used to access the database.
Some other DBMS (teacher, HR) examples include:
MySQL ( example ->eng teacher)
SQL Server ( hindi teacher)
Oracle ( evs teacher)
dBASE ( senior teacher)
FoxPro ( math teacher)
All teacher maintain attendance register same as all this DBMS maintain database..
Principal or head decide which teacher is create and maintain register same as developer decide which DBMS( my sql, oracle) is create and maintain database and which is best.
Find rohan total attendance?
Find rohan total attendance --------> HR/Teacher -----------> Attandance register
Find rohan total attendance------> DBMS(my sql) --------->database
SQL (Structured query language)
NOSQL
SQL--------- SQL stands for Structured Query Language. SQL is used to communicate with a database
it is the standard language for relational database management systems. SQL statements are used to perform tasks such as update data on a
database, or retrieve data from a database. Some common DBMS( Oracle, Sybase, Microsoft SQL Server, Access, Ingres, My sql, Oracle,Sqlite) that use
Sql . Sql database is tablebase database like Ms excel, vertically data store, relational database.
In NOSQL data is store in key value, pair like json...NOSQL used by---Redis , MongoDB. example ofsql , sql is class monitor or head student of class
that is help to the DBMS( Teacher, Hr) to manage Database (Register)... one class monitor helps all teacher same as one SQL Used by many dbms
(My sql , oracle )
Purpose .. To query and operate database system.
SQL use by NOSQL use by
1.My sql 1.Redis
2.Ms-sql 2.MongoDB etc
3. oracle etc
| |
|_______________DBMS__________________|
Related
I have a database in SQL Developer which pull data from an ERP tool and I would like to create a Data warehouse in order to connect it then to PowerBI.
It's my first time that I am doing all this process from the beginning so I am not so experienced.
So where are you suggesting to create the Data Warehouse (I was thinking on SSMS) and how can I connect it with PowerBI ?
My Data Warehouse will consist from some View of my tables and some Joins to get some data in the structure that I want since it is not possible to change anything in the DB.
Thanks in advance.
A "data warehouse" is just a database. The distinction is really more about the commonly used schema design, in the sense that a warehouse is often built along the lines of a star or snowflake design.
So if you already have a database that is extracting data from your ERP, there is nothing to stop you from pointing PowerBI directly at that and performing some analytics etc. If your intention is to start with this database, and then clone/extract/load this data into a new database which is a star/snowflake schema, then that's a much bigger exercises.
I am currently working on a project which involves the following:
The application I am working on is connected to a SQlserver
database.
SAP loads information into multiple tables (in a daily
and also hourly basis) into a MASTER database
There are 5 other databases(hosted on the same server) that access this information via synonyms and stored procedure calls to the MASTER database
The MASTER database purely used for storing the data and routing it to the other databases)
Master Database -
Tables:
MASTER_TABLE1 <------- SAP inserts data into this table.Triggers are used to process the valid data & insert into secondary staging tables -say MASTER_TABLE1_SEC
MASTER_TABLE1_SEC -- Holds processed data coming into MASTER_TABLE1
FIVE other databases ( for each manufacturing facility) are present in the same server. My application is connected to the facility databases ( not the Master)
FACILITY1
Facility2
....
FACILITY5
Synonyms of MASTER_TABLE1_SEC are created in each of these 5 facility databases
Stored procedures are again called from the Facility databases- in order to load data from the MASTER_TABLE1_SEC into the respective tables( within EACH facility) based on the business logic.
Is there a better architecture to handle this kind of a project? I am a beginner when it comes to advanced data management. Can anyone suggest a better architecture or tools to handle this?
There are a lot of patterns that would actually meet the needs described here. It serves that you are working with a type of Data Warehouse. I use Data Vault for my Enterprise Data Warehouses. It is an Ensemble Modeling technique designed for integration and master data preparation. You can think of it as a way to house all data from all time. You would then generate Data Marts (Kimball Method) for each of the Facilities containing only thei or whatever is required for their needs.
I have a mini account software. In this software I can store multiple company data. The data is stored in SQL Server 2008 R2 database.
In current database I have a User table which stores all user names, a Company Master table which stores company details like name,address, session etc. and user ID as FK with user table. Next is tran table which link with company Master and stores vouchers details and others table link to tran tabel like bill, payment etc.
The app is build for small companies and professionals who keep & maintain there their client data. In that scenario all data is separate and mutually independedent. In case of the small company they maintain all subsidiary company's account related data in a single app. Some time they receipt or send any one subsidiary company data to that company or any government body or Audit firms. like mobile phone contacts, I can send all contacts or any selected contact.
Users used to select his/her company first form company Master and then add/edit reference data or view report on the basis of selected company ID.
Now my problem is the data volume is become very high on some client places because of 50 to 60 companies data are stored in a single database and how I get company ID wise backup or restore the data. Is filegroup of sql server can help on this matter? I have no knowledge of filegroup.
Please help me.
Do not split your SQL database into multiple SQL databases (either do not create more filegroups etc.) just because you need to get data filtered by the CompanyId. Everytime when your Client would need to create a new Company, your application would have to create a new database for it. This would also quite complicate things like app updates.
If you do not face any grave performance problems - like when using SQL Express and your client database is 9 GB (max. database size for Express is 10 GB) - leave 1 database for 1 client.
Be sure all your related tables are well indexed by the CompanyID column. Then you can provide means to export data by CompanyID from your application - custom reports, exports to csv files, Excel etc.
Database backup file is usually not used for passing data to other applications. Its goal is to assure disaster recovery - when the disk fails etc. then your client will be able to recover easily. On contrary when he would have 50 database files in place of just 1 he would have hard time restoring all those databases properly.
Two different relational databases.
Your task is to write a code to transfer the data from the first database to the second database.
Some tables in the database you are transferring to are of the same structure as the table you are transferring from, the transfer of these tables is as simple as "INSERT INTO DbA.TableA (...) VALUES SELECT * FROM DbB.TableB".
Some tables in the database you are transferring to have different structures and different purposes. After proper analysis, you understand the relations and you understand the right transformation you need to code.
My question is: how do you express such knowledge? How do you express the transformational relations between two databases? Are there any tools or diagrams?
The best way I know right now is writting the list of tables of the first database and for each table describing how it is to be transformed into the second database. Is it possible to make this more formal/concise/cool?
If you are wanting a toolset and work in the Microsoft database stack then this is exactly what SQL Server Integration Services (or SSIS) is used for.
If you are wanting to document the process then you would typically write an interface definition document (IDD). There are many examples on Google but here is something to get you started.
This involves data replication, kind of:
We have many sites with SQL Express installed, there is an 'audit' database on each site that has one table in 1st normal form (to make life simple :)
Now I need to get this table from each site, and copy the contents (say, with a Date Time Value > 1/1/200 00:00, but this will change obviously) and copy it to a big 'super table' in sql server proper, that also has the primary key as the Site Name (That needs injecting in) and the current primary key from the SQL Express table)
e.g. Many SQL Express DBs with the following table columns
ID, Definition Name, Definition Type, DateTime, Success, NvarChar1, NvarChar2 etc etc etc
And the big super table needs to have:
SiteName, ID, Definition Name, Definition Type, DateTime, Success, NvarChar1, NvarChar2 etc etc etc
Where items in bold are the primary key(s)
Is there a Microsoft (or non MS I suppose) app/tool/thing to manager copying all this data accross already, or do we need to write our own?
Many thanks.
You can use SSIS (which comes with SQL Server) to populate, it can be set up with variables to change the connection string to the various databases. I have one that loops through the whole list and does the same process using three differnt files from three differnt vendors. You could so something simliar to loop through the different site databases. Put the whole list of database you want to copy the audit data from in a table and loop through it changing the connection string each time.
However, why on earth would you want one mega audit table per site? If every table in the database populates the audit table as changes happen, then the audit table eventually becomes a huge problem for performance. Every insert, update and delete has to hit this table and then you are proposing to add an export on top of that. This seems to me to be a guaranteed structure for locking and deadlocks and all sorts of nastiness. Do yourself a favor and limit each audit table to the table it is auditing.
Things to consider:
Linked servers and sp_msforeachdb as part of a do-it-yourself solution.
SQL Server Replication (by Microsoft) (which I believe can pull data from SQL Server Express)
SQL Server Integration Services which can pull data from SQL Server Express instances.
Personally, I would investigate Integration Services first.
Good luck.
You could do this with SymmetricDS. SymmetricDS is open source, web-enabled, database independent, data synchronization/replication software. It uses web and database technologies to replicate tables between relational databases in near real time. The software was designed to scale for a large number of databases, work across low-bandwidth connections, and withstand periods of network outage.
As of right now, however, you would need to implement a custom IDataLoaderFilter extension point (in Java) to add the extra column. The metadata would be available though because your SiteName would be the external_id.