Currently we're identifying if a column is virtual in Snowflake by running a SHOW COLUMN query and checking the KIND field for VIRTUAL_COLUMN. Unfortunately, there's a 10k limit on entries returned from SHOW queries in Snowflake and we'd like to be able to run this query at the schema level on schemas ~25k tables.
According to this post there's no way to identify virtual columns in the information_schema.columns view and we'd like to avoid having to run a SHOW COLUMNS query at the table level or having to run a desc table on every table.
Is there some other way we can identify virtual columns at scale?
Unfortunately, not aware of any native capability. I would consider writing a script using the get_ddl() function and run it against all objects in a schema.
Related
I was given a task to develop a report that would present the following details (as separate columns in ALV):
1) Name of the DB table (like MSEG, EKPO etc.)
2) Size of the DB table in megabytes
3) Number of records
4) Number of read requests performed on the table
5) Number of write requests performed on the table
There are DB* tables that contain such kind of info. Specifically I am referring to DB6PMHST and DB6HISTBS. When I try to view them via SE11 or SE16, system reports that these tables do not hold any records. I tried all three development, testing and production landscapes.
Please may you provide a guidance on what I am doing wrong? Maybe there are some other system tables that would contain the necessary info?
P.S. I tried to debug ST04 transaction in order to see the tables from which the report selects data, but wasn't able to find those lines of the source code.
I would deeply appreciate your kind assistance.
P.S.S. Checked the table MSSDBSTATT - it is empty as well (our enterprise uses MS SQL Database)
Go to SE38 and run this report RSTABLESIZE, enter table ID and see the magic.
The number of reads and writes on a table is a subject specific to the type of database (MSSQL) -> please tag your question accordingly.
If you get an answer by an MSSQL expert, which says that the data is stored in some MSSQL tables, then you cannot query those tables using "Open SQL" but you may query them using "native SQL" (i.e. EXEC SQL or ADBC for instance).
Do you know how to transfer only new records between two different databases (ie. Oracle and MSSQL) using SSIS? There is no problem transfering new data only between two tables in the same database and server, but is this possible to do such operation between completely different servers and databases?
Ps. I know about solution using Lookup but it is not very efficient if anybody needs to check and add a lot of records (50k and more) several times per day. I would like to operate with new data only.
You have several options:
Timestamp based solution
If you have a column which stores the insertation time in the source system, you can select only the new records created since the last load. With the same logic, you can transfer modified records too, just mark the records with the timestamp value when it change.
Sequence based solution
If there is a sequence in the source table, you can load the new records based on that sequence. Query the last value from the destination system, then load avarything which is larger than that value.
CDC based solution
If you have CDC (Change Data Capture) in your source system, you can track the changes and you can load them based on the CDC entries.
Full load
This is the most resource hungry solution: you have to copy all data from the source to the destination. If you do not have any column which marks the new records, you should use this solution.
You have several options to achieve this:
TRUNCATE the destination table and reload it from source
Use a Lookup component to determine which records are missing
Load all data from source to a temporary table and write a query which retrieves the new/changed records.
Summary
If you have at least one column, which marks the new/modified records, you can use it to implement a differential/incremental load with SSIS. If you do not have any clue, which columns/rows are changed, you have to load (or at least query) all of them.
There is no solution which enables a one-query (INSERT .. SELECT) solution using multiple servers without transferring all data. (Please note, that a multi-server query using Linked Servers are transfers the data from the source system).
What about variables? Is it possible to use the same variable between different databases and servers in SSIS?
I would like to transfer last id number from a destination table and transfer it to the source table (different server!).
I can set a variable in a database scope like this:
DECLARE #Last int
SET #Last = (SELECT TOP 1 Id FROM dbo.Table_1 ORDER BY Id DESC)
SELECT *
FROM dbo.Table_2
WHERE ID > #Last;
However it works between two tables in the same database (as a SQL command) only. I can create a variable for a entire SSIS package in Variables --> Add variable, but I don't know it is possible to use the variable in a similar way as above - to keep an information about last id in a destination table and pass it to another table on a source server as data limit.
The setup
I have the following database setup:
CentralDB
Table: Stores
Table: Users
Store1DB
Table: Orders
Store2DB
Table: Orders
Store3DB
Table: Orders
Store4DB
Table: Orders
... etc
CentralDB contains the users, logging and a Stores table with the name of each store database and general information about each store such as address, name, description, image, etc...
All the StoreDB's use the same structure just different data.
It is important to know that the list of stores will shrink and increase in the future.
The main client communicating with this setup is an API REST Service which gets passed a STOREID in the Header of each request telling it which database to connect to. This works flawlessly so far.
The reasoning
Whenever we need to do database maintenance on one store, we don't want all other stores to be down.
Backup management should be per store
Not having to write the WHERE storeID=x every time and for every table
Performance: each store could run on its own database server if the need arises
The goal
I need my REST API Service to somehow get all orders from all stores in one query.
Will you help me figure out a way to do this without hardcoding all storedb names? I was thinking about a stored procedure on the CentralDB but I was hoping there would be other solutions. In any case it has to be very efficient.
One option would be to have a list of databases stored in a "system" table in CentralDB.
Then you could create a stored procedure that would read the database names from the table, loop through them with cursor and generate a dynamic SQL that would UNION the results from all the databases. This way you would get a single recordset of results.
However, this database design is IMHO flawed. There is no reason for using multiple databases to store data that belongs to the same "domain". All the reasons that you have mentioned can be solved by using a single database with proper database design. Having multiple databases will create multiple problems on the long term:
you will need to change structure of all the DBs when you modify your database model
you will need to create/drop new databases when new stores are added/removed from your system
you will need to have items and other entities that are "common" to all the stores duplicated in all the DBs
what about reporting requirements (e.g. get sales data for stores 1 and 2 together, etc.) - this will require creating complex union queries...
etc...
On the long term, managing and maintaining this model will be a big pain.
I'd maintain a set of views that UNION ALL all the data. Every time a store is added or deleted those views must be updated. This can be automated.
The views provide an illusion to the application that there is only one database.
What I would not do is have each SQL query or procedure query all the database names and create dynamic SQL. That would entail lots of code duplication and an unnecessary loss of performance. This approach is error prone. Better generate code once in a central place and have all other SQL code reference that generated code.
I setup a test-bed application vulnerable to mssql injection and i wondered, how do i extract column data from another database? To extract column data from current database we do:
convert(int,(select columnnamegoeshere from tablenamegoeshere))--
and then to enumerate the other column data we do:
convert(int,(select columnnamegoeshere from tablenamegoeshere where columnnamegoeshere not in ('firstentryfromcolumn')))--
But if it's not inside the default database and we want to extract column data from another database, how do we do that? Thanks.
I would do a join... but, to keep it simple for you, here's your code using a different database column:
convert(int,(select columnnamegoeshere from tablenamegoeshere where columnnamegoeshere not in (select top 1 firstentryfromcolumn from otherdb.dbo.otherdbtable)))
It would be better to join the 2 tables together and exclude the records if that's possible... subqueries are usually slower and not the best way to go about it.
I need to use the changed table tracking feature of sql server 2008. I have enabled this on many tables. Now i have to write a sync program to transfer this data to another location.
My problem is how do i get only those tables whose data has changed without having to loop through all the changed tables list and checking each of them?
Try sys.CHANGE_TRACKING_TABLES (documented on MSDN here).
You'll have to use OBJECT_NAME to get the table name from the first column.