I wanted to ask whether it is possible to compare the complete database structure of two huge databases.
We have two databases, the one is a development database, the other a production database.
I've sometimes forgotten to make changes in to the production database, before we released some parts of our code, which results that the production database doesn't have the same structure, so if we release something we got some errors.
Is there a way to compare the two, or synchronize?
For MySQL database you can compare view and tables (column name and column type) using this query:
SET #firstDatabaseName = '[first database name]';
SET #secondDatabaseName = '[second database name]';
SELECT * FROM
(SELECT
CONCAT(cl.TABLE_NAME, ' [', cl.COLUMN_NAME, ', ', cl.COLUMN_TYPE, ']') tableRowType
FROM information_schema.columns cl, information_schema.TABLES ss
WHERE
cl.TABLE_NAME = ss.TABLE_NAME AND
cl.TABLE_SCHEMA = #firstDatabaseName AND
ss.TABLE_TYPE IN('BASE TABLE', 'VIEW')
ORDER BY
cl.table_name ) AS t1
LEFT JOIN
(SELECT
CONCAT(cl.TABLE_NAME, ' [', cl.COLUMN_NAME, ', ', cl.COLUMN_TYPE, ']') tableRowType
FROM information_schema.columns cl, information_schema.TABLES ss
WHERE
cl.TABLE_NAME = ss.TABLE_NAME AND
cl.TABLE_SCHEMA = #secondDatabaseName AND
ss.TABLE_TYPE IN('BASE TABLE', 'VIEW')
ORDER BY
cl.table_name ) AS t2 ON t1.tableRowType = t2.tableRowType
WHERE
t2.tableRowType IS NULL
UNION
SELECT * FROM
(SELECT
CONCAT(cl.TABLE_NAME, ' [', cl.COLUMN_NAME, ', ', cl.COLUMN_TYPE, ']') tableRowType
FROM information_schema.columns cl, information_schema.TABLES ss
WHERE
cl.TABLE_NAME = ss.TABLE_NAME AND
cl.TABLE_SCHEMA = #firstDatabaseName AND
ss.TABLE_TYPE IN('BASE TABLE', 'VIEW')
ORDER BY
cl.table_name ) AS t1
RIGHT JOIN
(SELECT
CONCAT(cl.TABLE_NAME, ' [', cl.COLUMN_NAME, ', ', cl.COLUMN_TYPE, ']') tableRowType
FROM information_schema.columns cl, information_schema.TABLES ss
WHERE
cl.TABLE_NAME = ss.TABLE_NAME AND
cl.TABLE_SCHEMA = #secondDatabaseName AND
ss.TABLE_TYPE IN('BASE TABLE', 'VIEW')
ORDER BY
cl.table_name ) AS t2 ON t1.tableRowType = t2.tableRowType
WHERE
t1.tableRowType IS NULL;
If you prefer using tool with UI you can also use this script
https://github.com/dlevsha/compalex
which can compare tables, views, keys etc.
Compalex is a lightweight script to compare two database schemas. It
supports MySQL, MS SQL Server and PostgreSQL.
Screenshot (compare tables)
You can use the command line:
mysqldump --skip-comments --skip-extended-insert -d --no-data -u root -p dbName1>file1.sql
mysqldump --skip-comments --skip-extended-insert -d --no-data -u root -p dbName2>file2.sql
diff file1.sql file2.sql
You can just dump them with --no-data and compare the files.
Remember to use the --lock-tables=0 option on your production database to avoid the big nasty global lock.
If you use the same mysqldump version (your dev and production systems should have the same software, right?) then you'll expect to get more-or-less identical files out. The tables will be in alpha order so a simple diff will show discrepancies up easily.
To answer this kind of question currently, I've made a script that uses information_schema content to compare column, datatype, and table
SET #database_current = '<production>';
SET #database_dev = '<development>';
-- column and datatype comparison
SELECT a.TABLE_NAME, a.COLUMN_NAME, a.DATA_TYPE, a.CHARACTER_MAXIMUM_LENGTH,
b.COLUMN_NAME, b.DATA_TYPE, b.CHARACTER_MAXIMUM_LENGTH
FROM information_schema.COLUMNS a
LEFT JOIN information_schema.COLUMNS b ON b.COLUMN_NAME = a.COLUMN_NAME
AND b.TABLE_NAME = a.TABLE_NAME
AND b.TABLE_SCHEMA = #database_current
WHERE a.TABLE_SCHEMA = #database_dev
AND (
b.COLUMN_NAME IS NULL
OR b.COLUMN_NAME != a.COLUMN_NAME
OR b.DATA_TYPE != a.DATA_TYPE
OR b.CHARACTER_MAXIMUM_LENGTH != a.CHARACTER_MAXIMUM_LENGTH
);
-- table comparison
SELECT a.TABLE_SCHEMA, a.TABLE_NAME, b.TABLE_NAME
FROM information_schema.TABLES a
LEFT JOIN information_schema.TABLES b ON b.TABLE_NAME = a.TABLE_NAME
AND b.TABLE_SCHEMA = #database_current
WHERE a.TABLE_SCHEMA = #database_dev
AND (
b.TABLE_NAME IS NULL
OR b.TABLE_NAME != a.TABLE_NAME
);
Hope this script can also help people that looks for a non-application solution, but the usage of script. Cheers
I tried mysqldiff without success, so I would like to enrich the future readers by drawing attention to mysqlworkbench's compare function. http://dev.mysql.com/doc/workbench/en/wb-database-diff-report.html#c13030
if you open a model tab, and select the databases menu, you get a compare schemas option, which you can use to compare two different schemas on two different servers, or two schemas on the same server, or a schema and a model, or a lot of other options i haven't tried yet.
For mysql on Linux, it is possible via phpmyadmin to export the databases without 'data' and only structure.
Scrolling through the export options for the entire database, just deselect 'data' and set the output to text. Export both databases you wish to compare.
Then in file compare in your preferred program / site, compare the two text file outputs of the databases. Synchronization is still manual in this solution, but this is effective for comparing and finding the structural differences.
Depending on your database, the tools available vary.
I use Embarcadero's ER/Studio for this. It has a Compare and Merge feature.
There are plenty others, such as Toad for MySQL, that also have compare. Also agree on the Red-Gate suggestion, but never used it for MySQL.
Check out Gemini Delta - SQL Difference Manager for .NET. A free beta version is available for download, but the full version is only a few days away from public release.
It doesn't compare row-level data differences, but it compares tables, functions, sprocs, etc... and it is lightning fast. (The new version, 1.4, loads and compares 1k Sprocs in under 4 seconds, compared with other tools I've tested which took over 10 seconds.)
Everyone is right though, RedGate does make great tools.
Related
Can anyone help me with creating a replica in EXASOL i.e. I need to copy all the tables including Views,Functions and Scripts from one schema to another schema in the same server.
For Eg.: I want all the data from Schema A to be copied not moved to Schema B.
Many thanks.
Thank you wildraid for your suggestion :)
In-order to copy DDL of all the tables in schema, I've got a simple way that will give us the DDLs for all the tables :
select t1.CREATE_STATEMENT||t2.PK||');' from
(Select C.COLUMN_TABLE,‘CREATE TABLE ’ || C.COLUMN_TABLE ||'(' || group_concat( ‘“’||C.COLUMN_NAME||'“' || ' ' || COLUMN_TYPE || case when (C.COLUMN_DEFAULT is not null
and C.COLUMN_IS_NULLABLE=‘true’) or(C.COLUMN_DEFAULT<>‘NULL’ and C.COLUMN_IS_NULLABLE=‘false’) then
' DEFAULT ' || C.COLUMN_DEFAULT end || case when C.COLUMN_IS_NULLABLE=‘false’ then ' NOT NULL ' end
order by column_ordinal_position) CREATE_STATEMENT
from EXA_ALL_COLUMNS C
where
upper(C.COLUMN_SCHEMA)=upper(‘Source_Schema’) and column_object_type=‘TABLE’
group by C.COLUMN_SCHEMA, C.COLUMN_TABLE order by C.COLUMN_TABLE ) t1 left join
(select CONSTRAINT_TABLE,‘, PRIMARY KEY (’ ||group_concat(‘“’||COLUMN_NAME||'“' order by ordinal_position) || ‘)’ PK
from EXA_ALL_CONSTRAINT_COLUMNS where
constraint_type=‘PRIMARY KEY’ and upper(COnstraint_SCHEMA)=upper(‘Source_Schema’) group by CONSTRAINT_TABLE ) t2
on t1.COLUMN_TABLE=t2.constraint_table
order by 1;
Replace the Source_Schema with your schema name and it will generate the Create statement that you can run on the EXAplus.
For copying the data, I have used the same way that you've mentioned in step 2.
Ok, this question consists of two smaller problems.
1) How to copy DDL of all objects in schema
If you need to copy only small amount of schemas, the fastest way is to use ExaPlus client. Right click on schema name and select "CREATE DDL". It will provide you with SQL to create all objects. You may simply run this SQL in context of new schema.
If you have to automate it, you may take a look at this official script: https://www.exasol.com/support/browse/SOL-231
It creates DDL for all schemas, but it can be adapted to use single schema only.
2) How to copy data
This is easier. Just run the following SQL to generate INSERT ... SELECT statements for every table:
SELECT 'INSERT INTO <new_schema>.' || table_name || ' SELECT * FROM <old_schema>.' || table_name || ';'
FROM EXA_ALL_TABLES
WHERE table_schema='<old_schema>';
Copy-paste result and run it to make the actual copy.
I have a fairly large amount of sql server objects that I would like to batch deploy. Each object (table, view, procedure, etc) is in it's own file.
The problem is, many of the objects have interdependencies, so the order of creation is important, otherwise errors will occur.
Currently, I am deploying using a dos batch file that calls a controlling script into which I have manually specified the order of script execution, like so:
BATCH FILE:
SQLCMD -S %SERVER_NAME% -d %DATABASE_NAME% -i "DeployProcedures.sql"
SQL Script (DeployProcedures.sql):
:r view1.sql
:r view2.sql
:r view3.sql
etc
:r proc1.sql
:r proc2.sql
:r proc1.sql
etc
This works, but it is cumbersome to have to constantly keep this up to date.
Is there any other way of doing this? I think I would even be happy with running the deploy 4 times, with suppressed or "do not fail" on errors for the first 3 iterations, and then only enable terminate on errors for the final iteration.
I would rather something self-authored rather than a commercial product like: http://solutioncenter.apexsql.com/re-order-script-to-avoid-dependency-based-errors/
EDIT: downvotes on a question regarding a problem for which someone actually bothered to go through the trouble to write a commercial application - sigh.
sys.sql_expression_dependencies is your friend.
An example:
-- Show all the things that all objects depend on
SELECT o.name AS [Object], o.[type] AS [ObjectType], t.name AS [DependsOn], t.type AS [DependentObjectType]
FROM sys.objects o
INNER JOIN sys.sql_expression_dependencies d ON o.object_id = d.referencing_id
INNER JOIN sys.objects t ON d.referenced_id = t.object_id
ORDER BY o.name, t.name
Basically, you can generate your file that runs the script in order based on the dependencies you find from this query. It'll take some massaging for your application, but this should be a good start. You can always add a filter to limit things to specific types, of course. You can pretty much create all the tables in any order you want. Views, Functions, and Stored Procedures (in that order) are a bit trickier... views especially can have recursive dependencies that can be difficult to deal with, but it's certainly possible (recursive CTEs can auto-generate order here as well).
This looks like a "crude" but extremely straightforward way to handle it:
http://inedo.com/blog/using-source-control-to-manage-stored-procedures
Key excerpt:
The rules of database objects in source control are simple: one file
per object (named after the object) that will DROP and then CREATE the
object. You can use a tool like Scriptio to extract existing objects
into files. Throw in a little file organization, and you're ready for
check-in:
1.FUNCTIONS\
1.FormatOrderNumber.sql
2.VIEWS\
1.OrdersWithTotals.sql
1.SalesReport.sql
2.OrdersPastDue.sql
3.PROCS\
1.AddItemToOrder.sql
1.ValidateOrder.sql
2.ProcessOrder.sql
Note that the numeric prefixes ensure proper execution order.
Procedures can use Views, which in turn can use functions... but
usually not the other way around. And since a handful database objects
depend on other objects, those can prefixed with “2.” instead of “1.”.
Execution of these scripts is just as easy, and can be accomplished
with a simple batch file:
FOR /R %%f IN (*.sql) DO (
ECHO Running %%f
OSQL -E -i "%%f" -n -b -d myDatabase -S myServer
)
With scripts and a batch script set-up in source control, building
your database code is as easy as grabbing from source control and
clicking “execute scripts”.
Another possible approach is outlined here:
https://www.simple-talk.com/sql/t-sql-programming/dependencies-and-references-in-sql-server/
(I'll paste in one script but there are others in that article worth checking out.)
Here is a routine that shows you the soft dependency order of the objects in your database, and lists the external dependencies of any objects. (note that a lot of entities in a database aren’t classed as objects. )
IF object_id (N'DependencyOrder') IS NOT NULL
DROP FUNCTION dbo.DependencyOrder
GO
CREATE FUNCTION dbo.DependencyOrder()
/*
summary: >
This table-valued function is designed to give you the order in which
database objects should be created in order for a build to succeed
without errors. It uses the sys.sql_expression_dependencies table
for the information on this.
it actually only gives the level 1,,n so within the level the order
is irrelevant so could, i suppose be done in parallel!
It works by putting in successive passes, on each pass adding in objects
who, if they refer to objects, only refer to those already in the table
or whose parent object is already in the table. It goes on until no more
objects can be added or it has run out of breath. If it does more than
ten iterations it gives up because there must be a circular reference
(I think that's impossible)
Revisions:
- Author: Phil Factor
Version: 1.0
Modification: First cut
date: 3rd Sept 2015
example:
- code: Select * from dbo.DependencyOrder() order by theorder desc
returns: >
a table, giving the order in which database objects must be built
*/
RETURNS #DependencyOrder TABLE
(TheSchema VARCHAR(120) null, TheName VARCHAR(120) NOT null, Object_id INT PRIMARY KEY, TheOrder INT NOT null,iterations INT null,ExternalDependency VARCHAR(2000) null )
AS
-- body of the function
BEGIN
DECLARE #ii INT,#EndlessLoop INT,#Rowcount INT
SELECT #ii=1, #EndlessLoop=10, #RowCount=1
WHILE #rowcount>0 AND #endlessLoop>0
BEGIN
;WITH candidates (object_ID, Parent_object_id) AS
(SELECT sys.objects.Object_ID,sys.objects.parent_Object_id
FROM sys.objects
LEFT OUTER JOIN #DependencyOrder Dep --not in the dependency table already
ON dep.Object_id = objects.object_id
WHERE dep.Object_id IS NULL
AND type NOT IN ('s', 'sq','it')
)
INSERT INTO #DependencyOrder (TheSchema,TheName,Object_id,TheOrder)
SELECT object_schema_name(c.Object_ID),object_name(c.Object_id),c.object_id, #ii
FROM candidates c INNER JOIN #DependencyOrder parent
ON c.Parent_object_ID= parent.Object_ID
UNION
SELECT object_schema_name(Object_ID),object_name(Object_id),object_id, #ii
FROM candidates c
WHERE Parent_object_id=0
AND object_id NOT IN (
SELECT c.object_id
FROM candidates c
INNER JOIN sys.sql_expression_dependencies
ON object_ID=referencing_ID
LEFT OUTER JOIN #DependencyOrder ReferedTo
ON ReferedTo.object_id =referenced_ID
WHERE ReferedTo.object_id IS NULL
AND referenced_id IS NOT NULL--not a cross-database dependency
)
SET #rowcount=##rowcount
SELECT #ii=#ii+1,#EndlessLoop=#EndlessLoop-1
END
UPDATE #dependencyOrder SET iterations= #ii-1
UPDATE #dependencyOrder SET ExternalDependency= ListOfDependencies
FROM
(SELECT Object_ID, STUFF(
(SELECT ', '+coalesce(referenced_server_name+'.','')
+coalesce(referenced_database_name+'.','')
+coalesce(referenced_schema_name+'.','')
+referenced_entity_name
FROM sys.sql_expression_dependencies sed
WHERE sed.referencing_ID=externalRefs.object_ID
and referenced_database_name IS NOT NULL AND is_ambiguous=0
FOR Xml PATH (''), ROOT('i'), TYPE).value('/i[1]', 'varchar(max)')
,1,2,'') ListOfDependencies
FROM #dependencyOrder externalRefs
)f
INNER JOIN #dependencyOrder d ON f.object_id=d.Object_ID
RETURN
END
GO
Using the stored procedure sp_msforeachtable it's possible to execute a script for all tables in a database.
However, there are system tables which I'd like to exclude from that. Instinctively, I would check the properties IsSystemTable or IsMSShipped. These don't work like I expect - I have for example a table called __RefactorLog:
But when I query if this is a system or MS Shipped table, SQL Server reports none of my tables are system tables:
exec (N'EXEC Database..sp_msforeachtable "PRINT ''? = '' + CAST(ObjectProperty(Object_ID(''?''), ''IsSystemTable'') AS VARCHAR(MAX))"') AS LOGIN = 'MyETLUser'
-- Results of IsSystemTable:
[dbo].[__RefactorLog] = 0
[schema].[myUserTable] = 0
and
exec (N'EXEC Database..sp_msforeachtable "PRINT ''? = '' + CAST(ObjectProperty(Object_ID(''?''), ''IsMSShipped'') AS VARCHAR(MAX))"') AS LOGIN = 'MyETLUser'
-- Results of IsMSShipped:
[dbo].[__RefactorLog] = 0
[schema].[myUserTable] = 0
When I look into the properties of the table (inside SSMS), the table is marked as a system object. An object property like IsSystemObject doesn't exist though (AFAIK).
How do I check if a table is a system object, apart from the object property? How does SSMS check if a table is a system object?
Management studio 2008 seems to run some quite ugly following code when opening the "System Tables" folder in the object explorer, the key bit seems to be:
CAST(
case
when tbl.is_ms_shipped = 1 then 1
when (
select
major_id
from
sys.extended_properties
where
major_id = tbl.object_id and
minor_id = 0 and
class = 1 and
name = N''microsoft_database_tools_support'')
is not null then 1
else 0
end
AS bit) AS [IsSystemObject]
(Where tbl is an alias for sys.tables)
So it seems that it's a combination - either is_ms_shipped from sys.tables being 1, or having a particular extended property set.
__refactorlog is, in contrast to what SSMS suggests, a user table. It is used during deployment to track schema changes that cannot be deduced from the current database state, for example renaming a table.
If all your other user tables are in a custom (non-dbo) schema, you can use a combination of the isMSshipped/isSystemTable attributes and the schema name to decide if a table is 'in scope' for your script.
In the past I've worked on the assumption that, in the sys.objects table, column is_ms_shipped indicates whether an object is or is not a system object. (This column gets inherited by other system tables, such as sys.tables.)
This flag can be set by procedure sp_ms_markSystemObject. This, however, is an undocumented procedure, is not supported by Microsoft, I don't think we're supposed to know about it, so I didn't tell you about it.
Am I missing something?
However, there are system tables which I'd like to exclude from that
At least on SQL Server 2008, sp_MSforeachtable already excludes system tables, as this excerpt from it shows:
+ N' where OBJECTPROPERTY(o.id, N''IsUserTable'') = 1 ' + N' and o.category & ' + #mscat + N' = 0 '
I have some offline databases on a SQL server. I would like to know which files on disc are related to these databases. Is it possible to retrieve the file list of offline databases without taking them online first?
This will give you a list of all physical file paths related to any offline databases, along with database name and file type:
SELECT
'DB_NAME' = db.name,
'FILE_NAME' = mf.name,
'FILE_TYPE' = mf.type_desc,
'FILE_PATH' = mf.physical_name
FROM
sys.databases db
INNER JOIN sys.master_files mf
ON db.database_id = mf.database_id
WHERE
db.state = 6 -- OFFLINE
Or simply
select * from sys.databases where state_desc='OFFLINE'
List all the available, but offline SQL server database files
The following statement will list all the files associated with offline SQL server databases
SELECT
m.physical_name + '\' + m.name AS [file_path]
FROM
sys.databases AS d
INNER JOIN sys.master_files AS m ON d.database_id = m.database_id
WHERE
d.state_desc = 'OFFLINE'
--AND m.type_desc = 'ROWS'
GROUP BY
m.physical_name + '\' + m.name
Note: Uncomment the line AND m.type_desc = 'ROWS' (delete the --) to filter the list further to include only database files. Otherwise, log files will also be listed.
The GROUP BY clause is there to prevent entries appearing more than once.
I have an old database that I am inheriting. The access rights are not clearly defined anywhere and I'm looking for a quick way to get them for everyone. Let's say I have a user in my database that does not belong to any membership roles. However, they have been given access to do specific things to specific tables. For example, they can run select queries on table X and run update queries on table Y. I know I can find out what they have by going to the properties for each user. I would imagine, however, that there has to be a system table somewhere that has all of this defined in it and makes it easily queryable. What would this query look like.
FYI: I am working with SQL Server 2005
Update: Is there also a way to do this for all databases on the server?
Take a look at the Security Catalog Views, then check out MrDenny's answer here which gives a query to list a user's rights. I reproduce it here (tidied up to my liking)..
SELECT [Schema] = sys.schemas.name
, [Object] = sys.objects.name
, username = sys.database_principals.name
, permissions_type = sys.database_permissions.type
, permission_name = sys.database_permissions.permission_name
, permission_state = sys.database_permissions.state
, state_desc = sys.database_permissions.state_desc
, permissionsql = state_desc + ' ' + permission_name
+ ' on ['+ sys.schemas.name + '].[' + sys.objects.name
+ '] to [' + sys.database_principals.name + ']'
COLLATE LATIN1_General_CI_AS
FROM sys.database_permissions
INNER JOIN sys.objects ON sys.database_permissions.major_id = sys.objects.object_id
INNER JOIN sys.schemas ON sys.objects.schema_id = sys.schemas.schema_id
INNER JOIN sys.database_principals ON sys.database_permissions.grantee_principal_id = sys.database_principals.principal_id
ORDER BY 1, 2, 3, 5
Things are a bit trickier actually. The effective permissions are a combination of internal database permissions (queryable as Denny's query showed above by doza) and windows group membership. Th later unfortunately is stored outside SQL, in the AD schema so you can't realy query it.
So if your goal is to display 'Access to the table X is given to domain\someuser and domain\somegroup and denied to domain\someothergroup' then you can use the catalog metadata and query it, as showed in doza's post.
However if your goal is to answer 'Does user domain\someuser have access to table X?' you can't get the answer from the query above. That's right, despite the fact that you see a record saying the domain\someuser is granted access, you cannot answer if it has effective access. Remember that a single deny trumps all grants, and if domain\user is member of domain\someothergroup group then domain\someuser is effectively denied access.
To answer the later question you must use a different mechanism, namely you have to impersonate the user at SQL level and check the permission via HAS_PERM_BY_NAME:
EXECUTE AS USER = 'domain\someuser';
SELECT HAS_PERMS_BY_NAME('X','TABLE','SELECT');
REVERT;
Is worth noting that the first question can be answered by anyone with view privileges on the security catalogs, while the later requires impersonate permission, a much more powerful privilege.