I am reading the database schema for VistaDB 4.0 database using the standard ADO.NET 'DbConnection.GetSchema' API. I haven't found a way to obtain the 'Identity' setting for a column? The 'Columns' schema collection doesn't seem to have a column for this and I am not aware of any other collection that I should look into.
If it is not possible by querying any of the available collections, do I have to query some system table or view?
Any help would be appreciated.
There are no "sys" tables in VistaDB. There is a [database schema] table that contains most of what you need though.
[database schema]
You can get the identity columns for a database using the database schema table like this:
select * from [database schema] where typeid = 6
Look in the help file for the typeid list and what they mean.
Then once you have the list, you can match it up to the typeid for tables to see what table the identity column came from.
The only catch with the database schema table is that you cannot self reference or join it to itself (design limitation). So if you need to pull and reference from itself you have to do it in two commands, or through a temp table. The help file has an example of how to do this as well.
Alternate Way
You can also find all the identity columns using a VistaDB stored proc:
select * from VistaDBColumnSchema() where is_identity = true
DDA
If you need to find the next value, seed, etc you can also get those through DDA (Direct Data Access) methods.
The Identities property on an IVistaDBTableSchema object is a collection of the identities for that table. That collection can then be walked to pull the individual values.
The identity information included is the Seed, Step, Tablename, and Columnname.
ADO.NET GetSchemaTable Way
And yes, there is still another way. You can call GetSchemaTable on a reader to get some more information about the underlying structure.
using (VistaDBConnection cn = new VistaDBConnection("Data Source=" + dbName))
{
cn.Open();
using (VistaDBCommand cmd = new VistaDBCommand("Select * from simpletable", cn))
{
using (VistaDBDataReader myReader = cmd.ExecuteReader(CommandBehavior.KeyInfo))
{
//Retrieve column schema into a DataTable.
DataTable schemaTable = myReader.GetSchemaTable();
foreach (DataRow myField in schemaTable.Rows)
{
foreach (DataColumn myProperty in schemaTable.Columns)
{
System.Diagnostics.Debug.WriteLine(myProperty.ColumnName + " = " + myField[myProperty].ToString());
}
}
}
}
}
Related
I am not sure if this is possible but I have not been able to come across clear documentation for this use case. I am using F# 4 and the FSharp.Data.SqlClient library to connect to SQL Server 2016. I am wanting to call a stored procedure that returns multiple tables and turn those tables into the corresponding records. In this case the first table is made up of items and the second table is made up of customers.
My instinct is that it should look something like this:
let items, customers = cmd.Execute()
My gut is that items would be an IEnumerable<item> and customers would be an IEnumerable<customer> where item and customer are both Record types. What it appears is happening though is that FSharp.Data.SqlClient is only seeing the first returned table from the stored procedure. I am working on a SQL Server 2016 Developer instance. Here is the T-SQL to setup the example:
create table Item (
ItemID int identity(1, 1) primary key,
ItemName nvarchar(50)
)
go
create table Customer (
CustomerID int identity(1, 1) primary key,
CustomerName nvarchar(50)
)
go
insert into Item (ItemName) values ('A');
insert into Item (ItemName) values ('B');
insert into Item (ItemName) values ('C');
insert into Customer (CustomerName) values ('Gary');
insert into Customer (CustomerName) values ('Sergei');
insert into Customer (CustomerName) values ('Elise');
go
create procedure dbo.ExampleProcedure
as
begin
set nocount on;
select
ItemID,
ItemName
from Item
select
CustomerID,
CustomerName
from Customer
end;
And here is the F# script that I am testing with. It shows what I would like to be able to do but I get a compile error on the last line:
#r "../packages/FSharp.Data.SqlClient.1.8.2/lib/net40/FSharp.Data.SqlClient.dll"
#r "../packages/FSharp.Data.2.3.2/lib/net40/FSharp.Data.dll"
#r "System.Xml.Linq.dll"
open FSharp.Data
[<Literal>]
let connStr =
"Data Source=**connection string**;"
type queryExample = SqlProgrammabilityProvider<connStr>
do
use cmd = new queryExample.dbo.ExampleProcedure(connStr)
let items, customers = cmd.Execute()
I am wanting items to correspond to the first returned table and customers to correspond to the second returned table. The intellisense suggests that FSharp.Data.SqlClient is only seeing the first table. When I hover over cmd.Execute() the popup says "This expression was expected to have type 'a*'b but here has type System.Collections.Generic.IEnumerable<SqlProgrammabilityProvider<...>.dbo.ExampleProcedure.Record>". If I do the following I get access to the Items query in the stored procedure:
// Learn more about F# at http://fsharp.org. See the 'F# Tutorial' project
// for more guidance on F# programming.
#r "../packages/FSharp.Data.SqlClient.1.8.2/lib/net40/FSharp.Data.SqlClient.dll"
#r "../packages/FSharp.Data.2.3.2/lib/net40/FSharp.Data.dll"
#r "System.Xml.Linq.dll"
open FSharp.Data
[<Literal>]
let connStr =
"Data Source=**connection string**;"
type queryExample = SqlProgrammabilityProvider<connStr>
do
use cmd = new queryExample.dbo.ExampleProcedure(connStr)
for item in cmd.Execute() do
printfn "%A" item.ItemID
Is this even possible? Is my approach wrong? I could not find clear documentation on this use case but I thought it would be common enough it would be covered.
Update
Just to clarify what I am trying to achieve I am showing how I solve this in C#. In C# I create a DataSet object and populate it with the results of the Stored Procedure. From there I pick out the individual tables to work with. After extracting the tables I then use LINQ to transform the rows into the corresponding objects. It often looks something like the following:
using System.Data;
using System.Data.SqlClient;
var connStr = "**connection string**"
var sqlConnection = new SqlConnection(connStr );
var sqlCommand = new SqlCommand("ExampleProcedure", sqlConnection);
sqlCommand.CommandType = CommandType.StoredProcedure;
var dataSet = new DataSet();
var adapter = new SqlDataAdapter(sqlCommand);
adapter.Fill(dataSet);
var itemsTable = dataSet.Tables[0];
// Turn the itemsTable into a List<Item> using LINQ here
var customersTable = dataSet.Tables[1];
// Turn the customersTable into List<Customer> using LINQ here
I find this to be overly verbose for such a simple thing as extracting the individual tables but perhaps I am too sensitive to code clutter. I know that F# must have a more elegant and terse way to express this.
I don't know F#, however this is a data access problem.
When a stored procedure returns multiple resultsets, you need to access they in sequence, one by one.
cmd.ExecuteReader() returns an instance of a datareader pointing to the first resultset. You need to process this resultset, may be filling a list with instances of a custom class, than you call the method "NextResult" and you will have access to the next resultset and so on.
A reference for the method "NextResult": https://msdn.microsoft.com/pt-br/library/system.data.sqlclient.sqldatareader.nextresult(v=vs.110).aspx
I am using SQL Server 2016.
I have a stored procedure GET_RECORDS that takes input parameters for filter and outputs a CURSOR parameter
I want to get this cursor in my SSIS package
I had created data flow task, OleDb source and variables for parameter values. Then mapped parameters
Params mapping screen
but when I wanted to save the component - I got an error
error screen
I tried to add clause WITH RESULT SETS with some dummy columns, but my procedure doesn't return any result set
What am I doing wrong?
Any advices will be helpful.
Thank you.
With regards, Yuriy.
The source component is trying to determine what columns and types will be returned. Because you are using dynamic SQL the metadata can change each time you run it.
With result sets allows you to define the data being returned but should only be used if you are guaranteed to have those results every time you execute.
EDIT:
I create a connection and run the command so that it populates a data table. Then I put the column headers into a string array. There are plenty of examples out there.
Then I use the following function to create a destination table. Finally I create a datareader and pass that to the .Net SqlBulkCopy. Hope this helps.
private void CreateTable(string TableName, string[] Fields)
{
if (TableExists(TableName) && Overwrite)
{
SqlCommand = new SqlCommand($"Drop Table [{TableName}]", SqlConnection);
SqlCommand.ExecuteNonQuery();
}
string Sql = $"Create Table [{TableName}] (";
int ColumnNumber = 1;
foreach (string Field in Fields)
{
string FieldValue = Field;
if (! HasHeaders)
{
FieldValue = "Column" + ColumnNumber;
ColumnNumber++;
}
Sql += $"[{FieldValue}] Varchar(8000),";
}
Sql = Sql + "ImportFileID Int, ID Int Identity(1,1) Not Null, Constraint [PK_" + TableName + "] Primary Key Clustered ([ID] Asc))";
SqlCommand = new SqlCommand(Sql, SqlConnection);
SqlCommand.ExecuteNonQuery();
}
Use ado.net source instead of oledb source, define a simple select and get the columns you wish to return. Now you can define expresión in the dataflow properties.
Search ado.net source dynamic sql
:)
try to return the records and use foreach in ETL instead of cursor
https://www.simple-talk.com/sql/ssis/implementing-foreach-looping-logic-in-ssis/
I think you can do it from a simple way, but I don't know what you are you doing, exactly...
I am trying to use Dapper support my data access for my server app.
My server app has another application that drops records into my database at a rate of 400 per minute.
My app pulls them out in batches, processes them, and then deletes them from the database.
Since data continues to flow into the database while I am processing, I don't have a good way to say delete from myTable where allProcessed = true.
However, I do know the PK value of the rows to delete. So I want to do a delete from myTable where Id in #listToDelete
Problem is that if my server goes down for even 6 mintues, then I have over 2100 rows to delete.
Since Dapper takes my #listToDelete and turns each one into a parameter, my call to delete fails. (Causing my data purging to get even further behind.)
What is the best way to deal with this in Dapper?
NOTES:
I have looked at Tabled Valued Parameters but from what I can see, they are not very performant. This piece of my architecture is the bottle neck of my system and I need to be very very fast.
One option is to create a temp table on the server and then use the bulk load facility to upload all the IDs into that table at once. Then use a join, EXISTS or IN clause to delete only the records that you uploaded into your temp table.
Bulk loads are a well-optimized path in SQL Server and it should be very fast.
For example:
Execute the statement CREATE TABLE #RowsToDelete(ID INT PRIMARY KEY)
Use a bulk load to insert keys into #RowsToDelete
Execute DELETE FROM myTable where Id IN (SELECT ID FROM #RowsToDelete)
Execute DROP TABLE #RowsToDelte (the table will also be automatically dropped if you close the session)
(Assuming Dapper) code example:
conn.Open();
var columnName = "ID";
conn.Execute(string.Format("CREATE TABLE #{0}s({0} INT PRIMARY KEY)", columnName));
using (var bulkCopy = new SqlBulkCopy(conn))
{
bulkCopy.BatchSize = ids.Count;
bulkCopy.DestinationTableName = string.Format("#{0}s", columnName);
var table = new DataTable();
table.Columns.Add(columnName, typeof (int));
bulkCopy.ColumnMappings.Add(columnName, columnName);
foreach (var id in ids)
{
table.Rows.Add(id);
}
bulkCopy.WriteToServer(table);
}
//or do other things with your table instead of deleting here
conn.Execute(string.Format(#"DELETE FROM myTable where Id IN
(SELECT {0} FROM #{0}s", columnName));
conn.Execute(string.Format("DROP TABLE #{0}s", columnName));
To get this code working, I went dark side.
Since Dapper makes my list into parameters. And SQL Server can't handle a lot of parameters. (I have never needed even double digit parameters before). I had to go with Dynamic SQL.
So here was my solution:
string listOfIdsJoined = "("+String.Join(",", listOfIds.ToArray())+")";
connection.Execute("delete from myTable where Id in " + listOfIdsJoined);
Before everyone grabs the their torches and pitchforks, let me explain.
This code runs on a server whose only input is a data feed from a Mainframe system.
The list I am dynamically creating is a list of longs/bigints.
The longs/bigints are from an Identity column.
I know constructing dynamic SQL is bad juju, but in this case, I just can't see how it leads to a security risk.
Dapper request the List of object having parameter as a property so in above case a list of object having Id as property will work.
connection.Execute("delete from myTable where Id in (#Id)", listOfIds.AsEnumerable().Select(i=> new { Id = i }).ToList());
This will work.
I have a SQLite database in one system, I need to extract the data stored in SQLite to Oracle database. How do I do this?
Oracle provides product called the Oracle Database Mobile Server (previously called Oracle Database Lite) which allows you to synchronize between a SQLite and an Oracle database. It provides scalable bi-directional sync, schema mapping, security, etc. The Mobile Server supports both synchronous and asynchronous data sync. If this is more than a one-time export and you need to keep your SQLite and Oracle Databases in sync, this is a great tool!
Disclaimer: I'm one of the Product Managers for Oracle Database Mobile Server, so I'm a bit biased. However, the Mobile Server really is a great tool to use for keeping your SQLite (or Berkeley DB) and Oracle Databases in sync.
You'll have to convert the SQLite to a text file (not certain of the format) and then use Oracle to load the database from text (source is http://www.orafaq.com/wiki/SQLite). You can use the .dump command from the SQLite interactive shell to dump to a text file (see the docs for syntax).
SQL Loader is a utility that will read a delimited text file and import it into an oracle database. You will need to map out how each column from your flat file out of sqlite matches to the corresponding one in the Oracle database. Here is a good FAQ that should help you get started.
If you are a developer, you could develop an application to perform the sync. You would do
SELECT name FROM sqlite_master WHERE type='table'
to get the table names, then you could re-create them in Oracle (you can do DROP TABLE tablename in Oracle first, to avoid a conflict, assuming SQLite will be authoritative) with CREATE TABLE commands. Getting the columns for each one takes
SELECT sql FROM sqlite_master WHERE type='table' and name='MyTable'
And then you have to parse the result:
string columnNames = sql.replace(/^[^\(]+\(([^\)]+)\)/g, '$1').replace(/ [^,]+/g, '').split(',');
string[] columnArray = columnNames.Split(',');
foreach (string s in columnArray)
{
// Add column to table using:
// ALTER TABLE MyTable ADD COLUMN s NVARCHAR(250)
}
A StringBuilder can be used to collect the table name with its columns to create your INSERT command. To add the values, it would just be a matter of doing SELECT * FROM MyTable for each of the tables during your loop through the table names you got back from the initial query. You would iterate the columns of the rows of the datatable you were returned and add the values to the StringBuilder:
INSERT INTO MyTable ( + columnA, columnB, etc. + ) VALUES ( datarow[0], datarow[1], etc. + ).
Not exactly like that, though - you fill in the data by appending the column name and its data as you run through the loops. You can get the column names by appending s in that foreach loop, above. Each column value is then set using a foreach loop that gives you each object obj in drData.ItemArray. If all you have are string fields, it's easy, you just add obj.ToString() to your StringBuilder for each column value in your query like I have below. Then you run the query after collecting all of the column values for each row. You use a new StringBuilder for each row - it needs to get reset to INSERT INTO MyTable ( + columnA, columnB, etc. + ) VALUES ( prior to each new row, so the new column values can be appended.
If you have mixed datatypes (i.e. DATE, BLOB, etc.), you'll need to determine the column types along the way, store it in a list or array, then use a counter to determine the index of that list/array slot and get the type, so you know how to translate your object into something Oracle can use - whether that means simply adding to_date() to the result, with formatting, for a date (since SQLite stores these as date strings with the format yyyy-MM-dd HH:mm:ss), or adding it to an OracleParameter for a BLOB and sending that along to a RunOracleCommand function. (I did not go into this, below.)
Putting all of this together yields this:
string[] columnArray = null;
DataTable dtTableNames = GetSQLiteTable("SELECT name FROM sqlite_master WHERE type='table'");
if (dtTableNames != null && dtTableNames.Rows != null)
{
if (dtTableNames.Rows.Count > 0)
{
// We have tables
foreach (DataRow dr in dtTableNames.Rows)
{
// Do everything about this table here
StringBuilder sb = new StringBuilder();
sb.Append("INSERT INTO " + tableName + " ("); // we will collect column names here
string tableName = dr["NAME"] != null ? dr["NAME"].ToString() : String.Empty;
if (!String.IsNullOrEmpty(tableName))
{
RunOracleCommand("DROP TABLE " + tableName);
RunOracleCommand("CREATE TABLE " + tableName);
}
DataTable dtColumnNames = GetSQLiteTable("SELECT sql FROM sqlite_master WHERE type='table' AND name='"+tableName+"'");
if (dtColumnNames != null && dtColumnNames.Rows != null)
{
if (dtColumnNames.Rows.Count > 0)
{
// We have columns
foreach (DataRow drCol in dtTableNames.Rows)
{
string sql = drCol["SQL"] != null ? drCol["SQL"].ToString() : String.Empty;
if (!String.IsNullOrEmpty(sql))
{
string columnNames = sql.replace(/^[^\(]+\(([^\)]+)\)/g, '$1').replace(/ [^,]+/g, '').split(',');
columnArray = columnNames.Split(',');
foreach (string s in columnArray)
{
// Add column to table using:
RunOracleCommand("ALTER TABLE " + tableName + " ADD COLUMN " + s + " NVARCHAR(250)"); // can hard-code like this or use logic to determine the datatype/column width
sb.Append("'" + s + "',");
}
sb.TrimEnd(",");
sb.Append(") VALUES (");
}
}
}
}
// Get SQLite Table data for insertion to Oracle
DataTable dtTableData = GetSQLiteTable("SELECT * FROM " + tableName);
if (dtTableData != null && dtTableData.Rows != null)
{
if (dtTableData.Rows.Count > 0)
{
// We have data
foreach (DataRow drData in dtTableData.Rows)
{
StringBuilder sbRow = sb; // resets to baseline for each row
foreach (object obj in drData.ItemArray)
{
// This is simplistic and assumes you have string data for an NVARCHAR field
sbRow.Append("'" + obj.ToString() + "',");
}
sbRow.TrimEnd(",");
sbRow.Append(")");
RunOracleCommand(sbRow.ToString());
}
}
}
}
}
}
All of this assumes you have a RunOracleCommand() void function that can take a SQL command and run it against an Oracle DB, and a GetSQLiteTable() function that can return a DataTable from your SQLite DB by passing it a SQL command.
Note that this code is untested, as I wrote it directly in this post, but it is based heavily on code I wrote to sync Oracle into SQLite, which has been tested and works.
I have formed few temp tables in my query (TSQL) like - #temphold1, #temphold2, #temphold3..... #temphold10. Each temp table has different schema (different columns) each derived by grouping data from different tables with specific conditions. I need to determine a way to carry all these temp tables to the User Interface and display each table seperately. Is there a way I can add all temp tables with specific indexer that i can retrieve at the User Interface.
thanks for any reply.
No, there is no such indexer.
However, SQL Server and ADO.NET support returning multiple result sets by selecting each table in turn.
See this howto on MSDN (How To Handle Multiple Results by Using the DataReader in Visual C# .NET).
So, in your stored procedure:
-- after populating your temp tables:
SELECT * FROM #table1
SELECT * FROM #table2
SELECT * FROM #table3
In essence, after reading the first recordset, you call NextResult() on the DataReader in order to get the results of the next select:
while(dr.Read())
{
// process data from #table1
}
dr.NextResult();
while(dr.Read())
{
// process data from #table2
}
dr.NextResult();
while(dr.Read())
{
// process data from #table3
}
If you're returning results to C#, you can do it with a DataAdapter like this:
using (SqlConnection conn = new SqlConnection("your connection string")) {
SqlParameter[] sqlParams = new SqlParameter[] {
new SqlParameter("#param1",10),
new SqlParameter("#param2","test")
};
conn.Open();
SqlDataAdapter sa = new SqlDataAdapter("spStoredProcName", conn);
sa.SelectCommand.CommandType = CommandType.StoredProcedure;
sa.SelectCommand.Parameters.AddRange(sqlParams);
DataSet ds = new DataSet();
sa.Fill(ds);
//ds.Tables[0] == first table
//ds.Tables[1] == second table
//... etc.
//Do whatever you need to here
}