I have a lot of parquet and Delta files sitting in ADLS Gen2. These files have SQL views pointing against them that are defined in a Synapse serverless SQL pool (across several DBs). I recently discovered that some views are unable to select some of the data in the files due to rows with "bad data". The "bad data" are 0001-01-01 date / timestamp values. This is bad data because Synapse has a bug that prevents selection of these values, as discussed here. I want to programmatically identify all SQL views (or, equivalently, all offending parquet / Delta files in ADLS) that are failing to return data when queried due to this bad data. This is easy to do manually: simply run a query that selects all rows from the given view; if the query errors out, then it is due to bad data. (I can guarantee that any query failures are due to this bad data root cause.) However, how can I programmatically do this? If it matters, I have Azure Databricks available to me, and I have a list of all SQL views that have this potential.
Maybe this...
select schema_name(schema_id) as schema_name,
name as view_name
from sys.views
order by schema_name,
view_name;
Related
I need to create an hourly .SQB backup file of some specific tables, each filtered with a WHERE clause, from a SQL Server database. As an example, I need this data:
SELECT * FROM table1 WHERE pk_id IN (2,5,7)
SELECT * FROM table2 WHERE pk_id IN (2,5,7)
SELECT * FROM table3 WHERE pk_id IN (2,5,7)
SELECT * FROM table4 WHERE pk_id IN (2,5,7)
The structure of the tables on the source database may change over time, e.g. columns may be added or removed, indexes added, etc.
One option is to do some kind of export, script generation, etc. into a staging database on the same instance of SQL Server. Efficiency aside, I have no problem dropping or truncating the tables on the destination database each time. In short, I'm looking to have both the schema and data of the tables duplicated to the destination database. That's completely acceptable.
Another is to just create a .SQB backup from the source database. Being that the .SQB file is all that I really need (it's going to be sent SFTP) - that would be fine, too.
What's the recommended approach in this scenario?
Well if I understand your requirement correctly, you want data from some tables from your database to be shipped over to somewhere else periodically.
Thing that is not possible in SQL server is taking a backup of a subset of tables from your database. So, this is not an option.
Since you have mentioned you will be using SFTP to send the data, using BCP command to extract data is one option, but BCP command may or may not perform very well and it definitely will not scale-out very well.
Instead of using BCP, I would prefer an SSIS package, you will be able to do all (extract files, add where clauses, drop files on SFTP, tune your queries, logging, monitoring etc) in your SSIS package.
Finally, SQL Server Replication can be used to create a subscriber, only publish the articles (tables) you are interested in, you can also add where clauses in your publications.
Again there are a few options with the replication subscriber database.
Give access to your data clients to your subscriber database, no need
for extracts.
Use BCP on the subscriber database to extract data,
without putting load on your production server.
Use SSIS Package to
extract data from the subscriber database.
Finally create a backup of
this subscriber database and ship the whole backup (.bak) file to
SFPT.
I short there is more than one way to skin the cat, now you have to decide which one suits your requirements best.
I'm on microsoft sql server management studio 2008 10.0.1600.22
I have a number of databases (the things that have a can icon in Object Explorer), Each of these Databases contains several folders with generic dames like; Databases, Security, Server Objects, Replication, Management, and lastly SQL Server Agent. Inside the "Databases" folder, there are a large number of databases which each have their own columns/rows etc...
So for example
BLOCK1 (SQL Server ##.#.####)
[-]Databases
.[+]System Databases
.[+]Database Snapshots
.[-]GROUP1
..[+]Database Diagrams
..[-]Tables
...[+]System Tables
...[+]dbo.Table1
...[+]dbo.Table2
...[+]...
...[+]dbo.TableN
.[-]GROUP2
..[+]Database Diagrams
..[-]Tables
...[+]System Tables
...[+]dbo.Table1
...[+]dbo.Table2
...[+]...
...[+]dbo.TableN
...
.[-]GROUPN
Is there any high level built-in command or utility I can use to print out the schema for each entire 'Block'?
I want a directory or 'map' of some sort I can use to see what Tables are within each Databases within each block. Expanding each database to see the contained tables often causes Microsoft SQL Server Management Studio to freeze so having a printed secondary 'map' would be very useful in figuring out where things need to be done in a relatively LARGE network where finding out table names can take a very long time (loading/crash/freeze) or be functionally untenable.
I don't need column names; I can query the table to get that; I just need an easy way to see all the table names quickly to navigate and target queries. This is for work purposes; This RDBMs is extremely large and queries function fine against it; but expanding it to see where to target queries is impractical in object explorer due the extremely large number of components which clicking the [+] next to a database name is more likely to crash SQL server than return useful information if not taking a minute or so to load all the table names.
I know the simple but time consuming way to get this information; I'm just wondering if MS SQL Mgmt Studio 2008 might happen to have some built in function to just generate this list in plain text so I can print it out?
There are several "Object Catalog Views" in SQL Server that contain the names and types for the system objects (as well as other info). The one you may be most interested in is sys.objects, although there are others (sys.tables, sys.columns, etc.).
Here is a very generic example query:
use AdventureWorks2008
go
select *
from sys.objects as o
order by o.type
For more info, see the documentation.
nvm just did it by looping
SELECT TABLE_NAME
FROM INFORMATION_SCHEMA.TABLES (nolock)
WHERE TABLE_TYPE = 'BASE TABLE'
order by TABLE_NAME asc
and putting it in charts. this is more an issue of finding a better file format to store it in because the excel schema is 175MB large and would be 313k+ pages if physically printed. This initiative needs to re-examined from another angle. question closed
I am populating a table in a local database (localdb, SQL Server) with data pulled from a remote database (remotedb, Oracle).
The remote database is across the internets, and of course, local database is local to my db server.
I need to pull data from remotedb, and in order to pull the data from remotedb, I need to only pull data from specific key values (e.g. the user_id field on a table on remotedb).
The set of keys is large, approx. 10,000.
Oh, by the way, remotedb is read-only access, and I don't have access to things like imp/exp or batch or access to the server (it is with an outside vendor).
Currently, I am using a query like this:
SELECT
<my data>
FROM
<remotedb>.<remote_table> join <remotedb>.<remote_table> ...
WHERE
<remotedb>.<remote_table>.<remote_field> in
( <select that returns my 20,000 IDs> )
This seems like a brute force solution to me, but I can't think of any other way to do it.
Has anyone out there solved this problem?
If I'm reading your statement correctly, we had a similar issue with this at a past company and it created a performance hit. Unless the vendor can pull all the information for you beforehand and you perform a query off that, I don't see another way around it.
I have many SQL Server databases, each with a few tables containing important (from my point of view) information. I check the data (retrieving for example maximum or minimum value) within those tables using T-SQL queries.
Since I don't want to create views for each of the databases, I'm thinking about most convenient, easier and simply the best way to prepare summary which will be updating each time when opened.
The output file (or web page) will be used internally within technical team. All members can log into database using Windows authentication.
My idea was:
Excel + dynamic T-SQL --> I want to connect to database and execute T-SQL (cursor will go through all database names)
PowerShell (showing table using Out-GridView cmdlet)
PHP - first I will ask about all database names (executing select name fromsys.databases` and then execute query for each DB)
What is in your opinion best way? Do you have any better (from programmers point of view) way of getting such report/data?
You can use SSRS Reports .You have the options of exporting the report data to several formats such as pdf ,excel ,word .You can create a dataset for all your database .Since you are interested in showing aggregation and summation of values ,SSRS reports will be pretty useful in these cases .
I've inherited a clunky and horribly un-documented site from a bad developer and am trying to get a look at the database schema. Unfortunately the web host is the worst I've ever dealt with and has no control panel capability for viewing the db schema or even exporting tables.
Is there any way that I can get a look at the schema via a SQL query (this would be with ASP + SQL Server)? My end goal here is to see what tables exist, possibly get a SQL dump of the vital tables, and then recreate the whole thing the right way.
The INFORMATION_SCHEMA schema is a good place to start:
SELECT * FROM INFORMATION_SCHEMA.TABLES
SELECT * FROM INFORMATION_SCHEMA.VIEWS
...and so on.
You might also want to have a look at using SMO, an API to get at the metadata in SQL Server.
I'm not sure if simple queries like
SHOW TABLES;
DESCRIBE table_name;
SHOW TABLE STATUS from table_name;
are valid in MS SQL. They would also be useful
SchemaSpy http://schemaspy.sourceforge.net/ is a great tool for analyzing existing databases. It generates html lists of table and constraints as well as a graphical representation of relationships