How do I use a path stored in table with OPENROWSET? - sql-server

I have just recently began using OPENROWSET to insert images into a table. Previously, I would specify the path to each image (1 image = 1 INSERT statement), and use PHP to generate the image's binary string:
INSERT INTO nopCommerce..Picture (PictureBinary, MimeType, SeoFilename, AltAttribute, TitleAttribute, IsNew)
VALUES (
(
SELECT *
FROM OPENROWSET(BULK '" . $this->image['path'] . "', SINGLE_BLOB) AS Binary
),
'" . $this->image['mime_type'] . "',
'" . $this->image['seo_filename'] . "',
'" . $this->image['alt'] . "',
'',
0
)
However, I am trying to insert all images with a single query. So, I have began storing the path to each image into a table, and now I need to insert each one as I did before (just using the table's path field instead of a PHP string). But, when I attempt the following:
INSERT INTO nopCommerce..Picture (PictureBinary, MimeType, SeoFilename, AltAttribute, TitleAttribute, IsNew)
SELECT
(
SELECT *
FROM OPENROWSET(BULK ImagePath, SINGLE_BLOB) AS Binary
),
MimeType,
Name,
Alt,
'',
0
FROM nopRMS..Product_Image_Mappings
I receive the following error:
Msg 102, Level 15, State 1, Line 5
Incorrect syntax near 'ImagePath'.
So, I tried adding quotes around the column's name (to no avail):
INSERT INTO nopCommerce..Picture (PictureBinary, MimeType, SeoFilename, AltAttribute, TitleAttribute, IsNew)
SELECT
(
SELECT *
FROM OPENROWSET(BULK 'ImagePath', SINGLE_BLOB) AS Binary
),
MimeType,
Name,
Alt,
'',
0
FROM nopRMS..Product_Image_Mappings
Msg 4860, Level 16, State 1, Line 1
Cannot bulk load. The file "ImagePath" does not exist.
There has to be a way to accomplish this, I just cannot find the proper syntax online anywhere. Does anyone know how to tell SQL Server to get the path (string) from dbo.Product_Image_Mappings.ImagePath?
UPDATE
I forgot to give you an example of a value that dbo.Product_Image_Mappings.ImagePath would return. It's paths like \\DEREK\WebImages\1\ca-82300.jpg...
UPDATE
It appears that Eirikur Eiriksson has provided a solution in this thread, but this looks like an overly-complicated method of achieving the same end...
UPDATE (Attempt Using Eirikur Eiriksson's Method)
DECLARE #SQL_STR NVARCHAR(MAX) = N'';
SELECT #SQL_STR = STUFF(
(
SELECT
N'
UNION ALL
SELECT '
+ N'(SELECT X.BulkColumn FROM OPENROWSET(BULK '
+ NCHAR(39) + im.ImagePath + NCHAR(39)
+ N', SINGLE_BLOB) AS X) AS PictureBinary,'
+ NCHAR(39) + im.MimeType + NCHAR(39)
+ N' AS MimeType,'
+ NCHAR(39) + im.Name + NCHAR(39)
+ N' AS SeoFilename,'
+ NCHAR(39) + REPLACE(im.Alt, '''', '''''') + NCHAR(39)
+ N' AS AltAttribute,'
+ N'NULL AS TitleAttribute,'
+ N'0 AS IsNew'
FROM nopRMS..Product_Image_Mappings im
FOR XML PATH(''), TYPE
).value('.[1]','NVARCHAR(MAX)'),1,12,N''
)
INSERT INTO nopCommerce..Picture (PictureBinary, MimeType, SeoFilename, AltAttribute, TitleAttribute, IsNew)
EXEC (#SQL_STR);
This kinda worked, but it only inserted 42 rows (out of 7200+)... I need this to be 100% accurate :( I admit though, I may need to change something about this query, but I don't know anything about it (aside from the basic INSERT, SELECT, etc.)

Maybe don't use OPENROWSET? What you are wanting can be handled in a much simpler and cleaner manner using SQLCLR. You can create a Scalar UDF to just read the contents of a file and return that as a VARBINARY(MAX). Then it will fit in nicely to your existing query. For example:
INSERT INTO nopCommerce.dbo.Picture (PictureBinary, MimeType, SeoFilename,
AltAttribute, TitleAttribute, IsNew)
SELECT
dbo.GetBinaryFile([ImagePath]) AS [PictureBinary],
MimeType,
Name,
Alt,
'',
0
FROM nopRMS.dbo.Product_Image_Mappings;
And how much code does it take for dbo.GetBinaryFile()? Here it is:
using System;
using System.Data.SqlTypes;
using System.IO;
using Microsoft.SqlServer.Server;
[return:SqlFacet(MaxSize = -1)]
[SqlFunction(IsDeterministic = false, IsPrecise = true)]
public static SqlBytes GetBinaryFile([SqlFacet(MaxSize = 1000)] SqlString FilePath)
{
if (FilePath.Value.Trim().Equals(String.Empty))
{
return SqlBytes.Null;
}
return new SqlBytes(File.ReadAllBytes(FilePath.Value));
}
And the T-SQL wrapper object is the following (please note the WITH RETURNS NULL ON NULL INPUT line as it skips execution if NULL is passed in, hence no need to check for FilePath.IsNull in the C# code :-)
CREATE FUNCTION [dbo].[GetBinaryFile](#FilePath NVARCHAR(1000))
RETURNS VARBINARY(MAX)
WITH RETURNS NULL ON NULL INPUT
AS
EXTERNAL NAME [CSVParser].[CSVUtils].[GetBinaryFile];
The Assembly will need to be marked as WITH PERMISSION_SET = EXTERNAL_ACCESS. Many people go the easy route of setting the database property of TRUSTWORTHY to ON in order to accomplish this, but that is a security risk and isn't even necessary. Just do the following and you can set the Assembly to EXTERNAL_ACCESS while keeping TRUSTWORTHY set to OFF:
Sign the Assembly.
Create an Asymmetric Key in master from that DLL.
Create a Login (also in master) from that Asymmetric Key.
Grant the new Login the EXTERNAL ACCESS ASSEMBLY permission.
You can find detailed instructions on how to do this in Visual Studio / SSDT (SQL Server Data Tools) in the following article that I wrote: Stairway to SQLCLR Level 7: Development and Security (that site does require free registration in order to view the content).
Also, for anyone that does not want to bother with creating / deploying the Assembly, a similar function is available (though not for free) in the Full version of the SQL# library (which I created, and while many functions are free, the File_* file system functions are only in the Full version).

Related

MultiLine Query in Snowflake Stored Procedure

I am trying to build one stored procedure in snowflake which will
i. Delete existing records from dimension table.
ii. Insert the records into the dimension table from corresponding materialized view.
iii. Once the data is inserted, record the entry into a config table.
First two are running fine, third one is not working, code below
Table DIM_ROWCOUNT
CREATE TABLE ANALYTICSLAYER.AN_MEDALLIA_P.DIM_ROWCOUNT
(
"TABLE_NAME" VARCHAR(500),
"VIEW_NAME" VARCHAR(500),
"TABLE_ROWCOUNT" VARCHAR(500) ,
"VIEW_ROWCOUNT" VARCHAR(500),
"LOADDATE" timestamp
)
The SP has parameter as SRC_TABLE_NAME which should be loaded for column : TABLE_NAME,
VIEW_NAME will be derived inside code.
TABLE_ROWCOUNT and VIEW_ROWCOUNT will be calculated within the code.
I need to have a multi line query for insertion.
How to create the insert statement multiline?
var config_details = `INSERT INTO DBNAME.SCHEMANAME.TABLENAME
SELECT '${VAR_TABLE_NAME}','${VAR_VIEW_NAME}','${VAR_TABLE_ROW_COUNT}','${VAR_VIEW_ROW_COUNT}',getdate();`
var exec_config_details = snowflake.createStatement( {sqlText: config_details} );
var result_exec_config_details = exec_config_details.execute();
result_exec_config_details.next();
Any help in appreciated.
I tend to write complex SQL statements in a javascript SP by appending each line/portion of text to the variable, as I find it easier to read/debug
Something like this should work (though I haven't tested it so there may be typos). Note that to get single quotes round each value I am escaping them in the code (\'):
var insert_DIM_ROWCOUNT = 'INSERT INTO DBNAME.TEST_P.DIM_ROWCOUNT ';
insert_DIM_ROWCOUNT += 'SELECT \'' + SRC_TABLE_NAME + '\', \'' + VIEW_NAME + '\', \'';
insert_DIM_ROWCOUNT += TABLE_ROWCOUNT + '\', \'' + VIEW_ROWCOUNT + '\', ';
insert_DIM_ROWCOUNT += GETDATE();
I then test this to make sure the SQL statement being created is what I want by adding a temporary line of code, after this, to just return the SQL string:
return insert_DIM_ROWCOUNT;
Once I'm happy the SQL is correct I then comment out/delete this line and let the SQL execute.

Able to access data through pyodbc and SELECT statements, but no new tables show up in SQL

I have the following code which is working with no errors and returning the expected output when I print the results of the pyodbc cursor I created.
cnxn = pyodbc.connect(MY_URL)
cursor = cnxn.cursor()
cursor.execute(
'''
CREATE TABLE tablename(
filename VARCHAR(100),
synopsis TEXT,
abstract TEXT,
original TEXT,
PRIMARY KEY (filename)
)
'''
)
for file in file_names_1:
try:
query = produce_row_query(file, tablename, find_tag_XML)
cursor.execute(query)
except pyodbc.DatabaseError as p:
print(p)
result = cursor.execute(
'''
SELECT filename,
DATALENGTH(synopsis),
DATALENGTH(abstract),
original
FROM ml_files
'''
)
for row in cursor.fetchall():
print(row)
However, no new tables are showing up in my actual MS SQL server. Am I missing a step to push the changes or something of that nature?
You need to commit changes or else they will not be updated in your actual database.
cnxn.commit()

How do you manage package configurations at scale in an enterprise ETL environment with SSIS 2014?

I'm migrating some packages from SSIS 2008 to 2014. MS is touting moving to project deployment and using SSIS environments for configuration because it's more flexible, but I'm not finding that to be the case at all.
In previous versions, when it came to configurations, I used a range of techniques. Now, if I want to use project deployment, I'm limited to environments.
For those variables that are common to all packages, I can set up an environment, no problem. The problem is those configuration settings that are unique to each package. It seems insane to set up an environment for each package.
Here is the question: I have several dozen packages with hundreds of configuration values that are unique to the package. If I can't store and retrieve these values from a table like in 2008, how do you do it in 2014?
That's not necessarily true about only being able to use environments. While you are limited to the out of the box configuration options, I'm working with a team and we've been able to leverage a straightforward system of passing variable values to the packages from a table. The environment contains some connection information, but any variable value that needs to be set at runtime are stored as row data.
In the variable values table, beside the reference to the package, one field contains the variable name and the other the value. A script task calls a stored proc and returns a set of name/value pairs and the variables within the package gets assigned the passed in value accordingly. It's the same script code for each package. We only need to make sure the variable name in the table matches the variable name in the package.
That coupled with the logging data has proven to be a very effective way to manage packages using the project deployment model.
Example:
Here's a simple package mocked up to show the process. First, create a table with the variable values and a stored procedure to return the relevant set for the package you're running. I chose to put this in the SSISDB, but you can use just about any database to house these objects. I'm also using an OLEDB connection and that's important because I reference the connection string in the Script Task which uses an OLEDB library.
create table dbo.PackageVariableValues
(PackageName NVARCHAR(200)
, VariableName NVARCHAR(200)
, VariableValue NVARCHAR(200)
)
create proc dbo.spGetVariableValues
#packageName NVARCHAR(200)
as
SELECT VariableName, VariableValue
FROM dbo.PackageVariableValues
WHERE PackageName = #packageName
insert into dbo.PackageVariableValues
select 'Package', 'strVariable1', 'NewValue'
union all select 'Package', 'intVariable2', '1000'
The package itself, for this example, will just contain the Script Task and a couple variables we'll set at runtime.
I have two variables, strVariable1 and intVariable2. Those variable names map to the row data I inserted into the table.
Within the Script Task, I pass the PackageName and TaskName as read-only variables and the variables that will be set as read-write.
The code within the script task does the following:
Sets the connection string based on the connection manager specified
Builds the stored procedure call
Executes the stored procedure and collects the response
Iterates over each row, setting the variable name and value
Using a try/catch/finally, the script returns some logging details as well as relevant details if failed
As I mentioned earlier, I'm using the OLEDB library for the connection to SQL and procedure execution.
Here's the script task code:
public void Main()
{
string strPackageName;
strPackageName = Dts.Variables["System::PackageName"].Value.ToString();
string strCommand = "EXEC dbo.spGetVariableValues '" + strPackageName + "'";
bool bFireAgain = false;
OleDbDataReader readerResults;
ConnectionManager cm = Dts.Connections["localhost"];
string cmConnString = cm.ConnectionString.ToString();
OleDbConnection oleDbConn = new OleDbConnection(cmConnString);
OleDbCommand cmd = new OleDbCommand(strCommand);
cmd.Connection = oleDbConn;
Dts.Events.FireInformation(0, Dts.Variables["System::TaskName"].Value.ToString(), "All necessary values set. Package name: " + strPackageName + " Connection String: " + cmConnString, String.Empty, 0, ref bFireAgain);
try
{
oleDbConn.Open();
readerResults = cmd.ExecuteReader();
if (readerResults.HasRows)
{
while (readerResults.Read())
{
var VariableName = readerResults.GetValue(0);
var VariableValue = readerResults.GetValue(1);
Type VariableDataType = Dts.Variables[VariableName].Value.GetType();
Dts.Variables[VariableName].Value = Convert.ChangeType(VariableValue, VariableDataType);
}
Dts.Events.FireInformation(0, Dts.Variables["System::TaskName"].Value.ToString(), "Completed assigning variable values. Closing connection", String.Empty, 0, ref bFireAgain);
}
else
{
Dts.Events.FireError(0, Dts.Variables["System::TaskName"].Value.ToString(), "The query did not return any rows", String.Empty, 0);
}
}
catch (Exception e)
{
Dts.Events.FireError(0, Dts.Variables["System::TaskName"].Value.ToString(), "There was an error in the script. The messsage returned is: " + e.Message, String.Empty, 0);
}
finally
{
oleDbConn.Close();
}
}
The portion that sets the values has two important items to note. First, this is set to look at the first two columns of each row in the result set. You can change this or return additional values as part of the row, but you're working with a 0-based index and don't want to return a bunch of unnecessary columns if you can avoid it.
var VariableName = readerResults.GetValue(0);
var VariableValue = readerResults.GetValue(1);
Second, since the VariableValues column in the table can contain data that needs to be typed differently when it lands in the variable, I take the variable data type and perform a convert on the value to validate that it matches. Since this is done within a try/catch, the resulting failure will return a conversion message that I can see in the output.
Type VariableDataType = Dts.Variables[VariableName].Value.GetType();
Dts.Variables[VariableName].Value = Convert.ChangeType(VariableValue, VariableDataType);
Now, the results (via the Watch window):
Before
After
In the script, I use fireInformation to return feedback from the script task as well as any fireError in the catch blocks. This makes for readable output during debugging as well as when you go look in the SSISDB execution messages table (or execution reports)
To show an example of the error output, here's a bad value passed from the procedure that will fail conversion.
Hopefully that gives you enough to go on. We've found this to be really flexible yet manageable.
When configuring an SSIS package, you have 3 options: use design time values, manually edit values and use an Environment.
Approach 1
I have found success with a mixture of the last two. I create a folder: Configuration and a single Environment, Settings. No projects are deployed to Configuration.
I fill the Settings environment with anything that is likely to be shared across projects. Data base connection strings, ftp users and passwords, common file processing locations, etc.
Per deployed project, the things we find we need to configure are handled through explicit overrides. For example, the file name changes by environment so we'd have set the value via the editor but instead of clicking OK, we click the Script button up on top. That generates a call like
DECLARE #var sql_variant = N'DEV_Transpo*.txt';
EXEC SSISDB.catalog.set_object_parameter_value
#object_type = 20
, #parameter_name = N'FileMask'
, #object_name = N'LoadJobCosting'
, #folder_name = N'Accounting'
, #project_name = N'Costing'
, #value_type = V
, #parameter_value = #var;
We store the scripts and run them as part of the migration. It's lead to some scripts looking like
SELECT #var = CASE ##SERVERNAME
WHEN 'SQLSSISD01' THEN N'DEV_Transpo*.txt'
WHEN 'SQLSSIST01' THEN N'TEST_Transpo*.txt'
WHEN 'SQLSSISP01' THEN N'PROD_Transpo*.txt'
END
But it's a one time task so I don't think it's onerous. The assumption with how our stuff works is that it's pretty static, once we get it figured out, so there's not much churn once it's working. Rarely do the vendors redefine their naming standards.
Approach 2
If you find that approach unreasonable, then perhaps resume using a table for configuration of the dynamic-ish stuff. I could see two implementations working on that.
Option A
The first is set from an external actor. Basically, the configuration step from above but instead of storing the static scripts, a simple cursor will go an apply them.
--------------------------------------------------------------------------------
-- Set up
--------------------------------------------------------------------------------
CREATE TABLE dbo.OptionA
(
FolderName sysname
, ProjectName sysname
, ObjectName sysname
, ParameterName sysname
, ParameterValue sql_variant
);
INSERT INTO
dbo.OptionA
(
FolderName
, ProjectName
, ObjectName
, ParameterName
, ParameterValue
)
VALUES
(
'MyFolder'
, 'MyProject'
, 'MyPackage'
, 'MyParameter'
, 100
);
INSERT INTO
dbo.OptionA
(
FolderName
, ProjectName
, ObjectName
, ParameterName
, ParameterValue
)
VALUES
(
'MyFolder'
, 'MyProject'
, 'MyPackage'
, 'MySecondParameter'
, 'Foo'
);
The above simply creates a table that identifies all the configurations that should be applied and where they should go.
--------------------------------------------------------------------------------
-- You might want to unconfigure anything that matches the following query.
-- Use cursor logic from below substituting this as your source
--SELECT
-- *
--FROM
-- SSISDB.catalog.object_parameters AS OP
--WHERE
-- OP.value_type = 'V'
-- AND OP.value_set = CAST(1 AS bit);
--
-- Use the following method to remove existing configurations
-- in place of adding them
--
--EXECUTE SSISDB.catalog.clear_object_parameter_value
-- #folder_name = #FolderName
-- #project_name = #ProjectName
-- #object_type = 20
-- #object_name = #ObjectName
-- #parameter_name = #ParameterName
--------------------------------------------------------------------------------
Thus begins the application of configurations
--------------------------------------------------------------------------------
-- Apply configurations
--------------------------------------------------------------------------------
DECLARE
#ProjectName sysname
, #FolderName sysname
, #ObjectName sysname
, #ParameterName sysname
, #ParameterValue sql_variant;
DECLARE Csr CURSOR
READ_ONLY FOR
SELECT
OA.FolderName
, OA.ProjectName
, OA.ObjectName
, OA.ParameterName
, OA.ParameterValue
FROM
dbo.OptionA AS OA
OPEN Csr;
FETCH NEXT FROM Csr INTO
#ProjectName
, #FolderName
, #ObjectName
, #ParameterName
, #ParameterValue;
WHILE (##fetch_status <> -1)
BEGIN
IF (##fetch_status <> -2)
BEGIN
EXEC SSISDB.catalog.set_object_parameter_value
-- 20 = project
-- 30 = package
#object_type = 30
, #folder_name = #FolderName
, #project_name = #ProjectName
, #parameter_name = #ParameterName
, #parameter_value = #ParameterValue
, #object_name = #ObjectName
, #value_type = V;
END
FETCH NEXT FROM Csr INTO
#ProjectName
, #FolderName
, #ObjectName
, #ParameterName
, #ParameterValue;
END
CLOSE Csr;
DEALLOCATE Csr;
When do you run this? Whenever it needs to be run. You could set up a trigger on OptionA to keep this tightly in sync or make it as part of the post deploy process. Really, whatever makes sense in your organization.
Option B
This is going be much along the lines of Vinnie's suggestion. I would design a Parent/Orchestrator package that is responsible for finding all the possible configurations for the project and then populate variables. Then, make use of the cleaner variable passing for child packages with the project deployment model.
Personally, I don't care for that approach as it puts more responsibility on the developers that implement the solution to get the coding right. I find it has a higher cost of maintenance and not all BI developers are comfortable with code. And that script needs to be implemented across a host of parent type packages and tends to lead to copy and paste inheritance and nobody likes that.

INSERT variable values into a table

I have several variables in an SSIS package that I would like inserting into a table.
example:-
#financialMonth, #Status, #Comments
The Variables have been populated along the way with values based on lookups, filename, dates, etc, and I want to store them in a results table.
Is using the execute SQL task the way to do this ?
Do I need to call a sproc and pass those variales as parameters ?
I've tried putting the following T-SQL into the SQLStatement property
INSERT INTO FilesProcessed
(ProcessedOn, ProviderCode, FinancialMonth,
FileName, Status, Comments)
SELECT GETDATE(), 'ABC' , 201006,
'ABC_201005_Testology.csv',
'Imported','Success'
I tried hardcoding the values above to get it to work
These are the columns on the table I'm inserting into
Column_name Type Computed Length
fileID int no 4
ProcessedOn datetime no 8
ProviderCode nchar no 6
FinancialMonth int no 4
FileName nvarchar no 510
Status nvarchar no 40
Comments nvarchar no 510
This is the Expression code that feeds the SQLStatementSource property
"INSERT INTO FilesProcessed (ProcessedOn, ProviderCode, FinancialMonth,
FileName, Status, Comments) SELECT GETDATE() AS ProcessedOn, '"
+ #[User::providerCode] + "' , "
+ (DT_STR,6,1252)#[User::financialMonth] + ", '"
+ #[User::fileName] + "', 'Imported' AS Status,'Successfully' AS Comments "
Unfortunately I'm missing something, and can't quite get it to work.
The Error message I'm getting is ...
Error: 0xC002F210 at Log entry in
FilesProcessed, Execute SQL Task:
Executing the query "INSERT INTO
FilesProcessed (ProcessedOn,
ProviderCode, FinancialMonth,
FileName, Status, Comments) SELECT
GETDATE(), 'ABC' , 201006,
'DAG_201005_Testology.csv',
'Imported','Successfully'" failed with
the following error: "An error
occurred while extracting the result
into a variable of type (DBTYPE_I2)".
Possible failure reasons: Problems
with the query, "ResultSet" property
not set correctly, parameters not set
correctly, or connection not
established correctly.
Please
a). Advise whether the Execute SQL Task is the way to do what I want to do.
b). Give me any pointers or pitfalls to look out for and check.
Thanks in advance.
OK, here is what I did.
I created an Execute SQL task and configured, thus :-
General Tab
ConnectionType = OLE DB
SQLSourceType = Direct Input
SQLStatement = (left blank)
BypassPrepare = True
ResultSet = None
Parameter Mapping
(none - leave blank)
Result Set
(none - leave blank)
Expressions
SQLStatementSource = "INSERT INTO FilesProcessed (ProcessedOn, ProviderCode, FinancialMonth, FileName, Status, Comments) SELECT GETDATE(), '" + #[User::providerCode] + "' , " + (DT_STR,6,1252)#[User::financialMonth] + ", '" + #[User::fileName] + "', 'Import - Success', '" + #[User::fileComments] + "'"
Then as long as I set up the variables and populate them in the variables window (the Expression editor will not let you save an expression that references a variable that does not exist. Keep notepad handy to store the contents while you go back and edit the variables window, and add new variables in ;)
Build the expression slowly, using the Parse expression button regularly to check.
make sure that the data types of the VALUES match the destination column data types.
see: http://social.msdn.microsoft.com/forums/en-US/sqlintegrationservices/thread/e8f82288-b980-40a7-83a6-914e217f247d/
A couple of speculative suggestions
The Error message says An error occurred while extracting the result into a variable of type (DBTYPE_I2). But this is a straight insert statement. There shouldn't be a result except for rows affected. Do you have any parameter mappings erroneously set to Output?
What if you try and run the SQL Query from the error message directly in management studio? Does that give you an error?
In the above table definition FinancialMonth as int datatype as
FinancialMonth int no 4
while inseting casting as :
(DT_STR,6,1252)#[User::financialMonth]
I think it's purely a datatype mismatch with the target table definition.

Find SQLite Column Names in Empty Table

For kicks I'm writing a "schema documentation" tool that generates a description of the tables and relationships in a database. I'm currently shimming it to work with SQLite.
I've managed to extract the names of all the tables in a SQLite database via a query on the sqlite_master table. For each table name, I then fire off a simple
select * from <table name>
query, then use the sqlite3_column_count() and sqlite3_column_name() APIs to collect the column names, which I further feed to sqlite3_table_column_metadata() to get additional info. Simple enough, right?
The problem is that it only works for tables that are not empty. That is, the sqlite_column_*() APIs are only valid if sqlite_step() has returned SQLITE_ROW, which is not the case for empty tables.
So the question is, how can I discover column names for empty tables? Or, more generally, is there a better way to get this type of schema info in SQLite?
I feel like there must be another hidden sqlite_xxx table lurking somewhere containing this info, but so far have not been able to find it.
sqlite> .header on
sqlite> .mode column
sqlite> create table ABC(A TEXT, B VARCHAR);
sqlite> pragma table_info(ABC);
cid name type notnull dflt_value pk
---------- ---------- ---------- ---------- ---------- ----------
0 A TEXT 0 0
1 B VARCHAR 0 0
Execute the query:
PRAGMA table_info( your_table_name );
Documentation
PRAGMA table_info( your_table_name ); doesn't work in HTML5 SQLite.
Here is a small HTML5 SQLite JavaScript Snippet which gets the column names from your_table_name even if its empty. Hope its helpful.
tx.executeSql('SELECT name, sql FROM sqlite_master WHERE type="table" AND name = "your_table_name";', [], function (tx, results) {
var columnParts = results.rows.item(0).sql.replace(/^[^\(]+\(([^\)]+)\)/g, '$1').split(',');
var columnNames = [];
for(i in columnParts) {
if(typeof columnParts[i] === 'string')
columnNames.push(columnParts[i].split(" ")[0]);
}
console.log(columnNames);
///// Your code which uses the columnNames;
});
Execute this query
select * from (select "") left join my_table_to_test b on -1 = b.rowid;
You can try it at online sqlite engine
The PRAGMA statement suggested by #pragmanatu works fine through any programmatic interface, too. Alternatively, the sql column of sqlite_master has the SQL statement CREATE TABLE &c &c that describes the table (but, you'd have to parse that, so I think PRAGMA table_info is more... pragmatic;-).
If you are suing SQLite 3.8.3 or later (supports the WITH clause), this recursive query should work for basic tables. On CTAS, YMMV.
WITH
Recordify(tbl_name, Ordinal, Clause, Sql)
AS
(
SELECT
tbl_name,
0,
'',
Sql
FROM
(
SELECT
tbl_name,
substr
(
Sql,
instr(Sql, '(') + 1,
length(Sql) - instr(Sql, '(') - 1
) || ',' Sql
FROM
sqlite_master
WHERE
type = 'table'
)
UNION ALL
SELECT
tbl_name,
Ordinal + 1,
trim(substr(Sql, 1, instr(Sql, ',') - 1)),
substr(Sql, instr(Sql, ',') + 1)
FROM
Recordify
WHERE
Sql > ''
AND lower(trim(Sql)) NOT LIKE 'check%'
AND lower(trim(Sql)) NOT LIKE 'unique%'
AND lower(trim(Sql)) NOT LIKE 'primary%'
AND lower(trim(Sql)) NOT LIKE 'foreign%'
AND lower(trim(Sql)) NOT LIKE 'constraint%'
),
-- Added to make querying a subset easier.
Listing(tbl_name, Ordinal, Name, Constraints)
AS
(
SELECT
tbl_name,
Ordinal,
substr(Clause, 1, instr(Clause, ' ') - 1),
trim(substr(Clause, instr(Clause, ' ') + 1))
FROM
Recordify
WHERE
Ordinal > 0
)
SELECT
tbl_name,
Ordinal,
Name,
Constraints
FROM
Listing
ORDER BY
tbl_name,
lower(Name);

Resources