While using Hive 2.3, we ran into an issue where a table that was meant to be external was created as a managed table and populated with a significant amount of data due to human error.
We're looking to find a way to change this table to an external one without data movement.
The standard way of accomplishing this would be
ALTER TABLE <name> SET TBLPROPERTIES('EXTERNAL'='TRUE')
However since our table is non-native (data is also stored in a MapRDB table using a json storage handler) this query fails.
Is there any way to drop the managed table while keeping the data, or any other way to change this table into an external one?
I have a requirement where I have to create tables with the date/datetime in the table name when they were created dynamically.Wondering if this option is possible in Snowflake?
Eg:
I would need somethinglike this.
CREATE TABLE someNewTable_YYYYMMDD
Thank you for your responses;
Best,
AB
You can achieve this using SQL variables and the IDENTIFIER keyword.
Here's an example that adds the current date into the table-name:
SET table_name=(SELECT 'someNewTable_' || TO_VARCHAR(CURRENT_DATE(), 'YYYYMMDD'));
CREATE TABLE IDENTIFIER($table_name) (col STRING);
For more complicated tasks where using IDENTIFIER keyword is inadequate, you can also use stored procedures as shown in this answer.
In some Snowflake databases I can run SELECT GET_DDL('Database', 'MyDatabase'); without problem but in another one, I get the following error message
SQL compilation error: Invalid object type: 'EXTERNAL_TABLE'
There might be an external defined in that database but I would still like to extract the DDL recursively on the database...
Is there any trick I could use?
You could try using DESCRIBE EXTERNAL TABLE which describes the virtual columns on the table.
DESCRIBE EXTERNAL TABLE
5.8 Release Notes - March 15-18, 2021
GET_DDL Function: External Table Support
With this release, the GET_DDL function supports returning a DDL statement that can be used to recreate a specified external table.
Note that currently, the GET_DDL output for a specified database or schema does not include any external tables contained in the schema.
My team is creating a high volume data processing tool. The idea is to take a 30,000 line batch file and bulk load it into a table and then process the records use parallel processing.
The part I'm stuck on is creating dynamic tables. We want to create a new physical table for each file that we receive. The tables will be purged from our system by a separate process after they are completed.
The part I'm stuck on is creating dynamic tables. For each batch file we receive I need to create a new physical file with a unique table name.
I have the base structure for the table and I intend to create unique table names using a combination of date/time stamp and a guid (dashes converted to underscore characters).
I could do this easily enough in a stored procedure but I'm wondering if there is a better way.
Here is what I have considered...
Templates in SQL Server Management Studio. This is a GUI tool built into Management Studio (from Management Studio Ctrl+Alt+T) that allows you to define different sql objects including a table and specify parameters. This seems like it would work, however it appears that this is a GUI tool and not something that I could call from a stored procedure.
Stored Procedure. I could put everything into a stored procedure and build my file name and schema into a nvarchar(max) string and use sp_executesql to create the table. This might be the way to accomplish my goal but I wonder if there is a better way.
Stored Procedure with an existing table as a template. I could define a base table and then query sys.columns & sys.dataypes to create a string representing the new table. This would allow me to add columns to the base table without having to update my stored procedure. I'm not sure if this is a better approach.
I'm wondering if any Stack Overflow folks have solved a similar requirements. What are your recommendations.
I just came across the system view sys.sql_modules today. What is a module versus a DB object? The view returns, most prominently, a column containing the definition text, as returned by sys.syscomments.
A module, in SQL-Server-speak is a stand-alone object that contains sql batches, such as a view, table valued function, stored procedure, trigger or scalar function. A SQL object is a more all-encompassing term that includes some that contain SQL Expressions, such as check or default constraints. A module used to be referred to as a 'routine' before SQL Server 2005, but I think the two terms are used interchangeably.
The table build script is not stored in SQL Server because of the ease with which components of a table can be altered separately. Therefore it is treated as an object but not a module.
Typical objects that aren't also considered to be modules are system tables, default constraints, foreign key constraints, service queues, check constraints, user tables, primary key constraints, internal tables and unique constraints.
Columns aren't considered to be objects. Neither are indexes.
Yes, it is all more complex than one might first think.
It's the blocks of T-SQL statements that make up a stored procedure, a stored function, a trigger or a view definition.
From "Books Online" in the "CREATE PROCEDURE" section:
Getting Information About Stored
Procedures To display the definition
of a Transact-SQL stored procedure,
use the sys.sql_modules catalog view
in the database in which the procedure
exists.
In sys.sql_modules, you'll find the actual T-SQL code.
Marc
A modules are functions, procedures, queues, and triggers. These Modules call Objects. That's what I understood from this article which describes "EXECUTE AS":
EXECUTE AS
I came across to this question when I wanted to get detailed explanation of EXECUTE command. I wanted to have deep understanding how to read t-sql help of command and realized that I haven't known what module means.
I just want to add these these to the list of sql modules:
system stored procedure,
user-defined stored procedure,
CLR stored procedure,
scalar-valued user-defined function,
or extended stored procedure.
Executes a command string or character string within a Transact-SQL batch, or one of the following modules: system stored procedure, user-defined stored procedure, CLR stored procedure, scalar-valued user-defined function, or extended stored procedure. The EXECUTE statement can be used to send pass-through commands to linked servers. Additionally, the context in which a string or command is executed can be explicitly set. Metadata for the result set can be defined by using the WITH RESULT SETS options.
https://learn.microsoft.com/en-us/sql/t-sql/language-elements/execute-transact-sql?view=sql-server-2017