I am using SSIS (VS 2008) and I want to take a populated resultset variable (populated from an EXECUTE SQL task) and convert within an SSIS Script Component (c#) it into an array.
The resultset contains a single column.
What would be the code to do this?
Edit: This is for a backfilling exercise; in normal circumstances, a web app calls a proc on the db that passes in a text string and gets a list of matching words based on this kind of query:
SELECT [Term]
FROM [dbo].[ATableOFTerms] WITH ( NOLOCK )
WHERE CHARINDEX(' ' + Term + ' ', #StringBeingSearched) > 0
Once the web app gets this list, it has a lot of business rules around what it converts it to but essentially it takes the list of matching terms and generates a single string that is then used for passing into a fulltext search.
The backfilling process in SSIS needs to mimic that so the script component has the same web app code that deals with the business logic but I needed c# code that does the above. Which is fine, I am returning all the [ATableTerms] into a resultset variable but now I need populate an array from the variable.
When I have this I can use string.Contains to achieve the same result.
IndexOf(term, StringComparison.OrdinalIgnoreCase) >= 0This is how it was solved, although I'd love to get an alternative to this.
OleDbDataAdapter adapter = new OleDbDataAdapter();
string Title = " " + Row.Title + " ";
DataTable dt = new DataTable();
adapter.Fill(dt , Variables.ATableOfTerms);
ArrayList terms = new ArrayList();
ArrayList returnedTerms = new ArrayList();
foreach (DataRow dr in dt.Rows)
{
terms.Add(" " + dr["Term"] + " ");
}
foreach (string term in terms)
{
if (Title.IndexOf(term, StringComparison.OrdinalIgnoreCase) >= 0)
{
returnedTerms.Add(term.Trim());
}
}
Related
I have two excel files and I want to import these files to SQL temporary table.
First excel file:
T1 T2 T3 T4 Total
1,472 1,364 1,422 – 4,258
-152.6 -152.6 -152.6 –
1,958 1,939 1,942 –
-122.6 -123.7 -122.2 –
Second excel file:
T1 T2 T3 T4 T5 Total
1,472 1,364 1,422 – 12.2 4,258
-152.6 -152.6 -152.6 – 1000.12
1,958 1,939 1,942 – 50.23
-122.6 -123.7 -122.2 – 185.25
Is there any way in SSIS to identify the files on the basis of number of columns? I need to identify report on the basis of the column number.
Objects from the Microsoft.Office.Interop.Excel namespace can be used in a C# Script Task to do this as follows. This example outputs the file name and number of columns into an SSIS object variable (User::SSISObjectVariable") that can be used to apply further logic and processing in the package, such as storing in a database table or otherwise. The full file path is the first column in the object variable and the count of columns is the second. Also be sure to add a reference to the Microsoft.CSharp namespace in the script. The object variable will need to be included in the ReadWriteVariables field of the Script Task and if the source folder is stored in a variable (as done below) then add this variable in the ReadOnlyVariables field.
using Microsoft.Office.Interop.Excel;
using System.Data;
using System.IO;
using System.Collections.Generic;
List<string> excelFileList = new List<string>();
//get source directory
string filePath = Dts.Variables["User::FilePathVariable"].Value.ToString();
DirectoryInfo di = new DirectoryInfo(filePath);
System.Data.DataTable dt = new System.Data.DataTable();
dt.Columns.Add("FilePath", typeof(System.String));
dt.Columns.Add("ColumnCount", typeof(System.Int32));
foreach (FileInfo fi in di.EnumerateFiles())
{
//optional- check file extension and prefix
if (fi.Extension == ".xls" && fi.Name.StartsWith("Prefix"))
{
//get full file path
excelFileList.Add(fi.FullName);
}
}
foreach (string excelFile in excelFileList)
{
Microsoft.Office.Interop.Excel.Application xlApp = new Microsoft.Office.Interop.Excel.Application(); ;
Microsoft.Office.Interop.Excel.Workbook xlWorkbook = xlApp.Workbooks.Open(excelFile);
Microsoft.Office.Interop.Excel.Worksheet xlWorksheet = xlWorkbook.Sheets[1];
int columnCount;
//get number of columns
columnCount = xlWorksheet.Cells.Find("*", System.Reflection.Missing.Value,
System.Reflection.Missing.Value, System.Reflection.Missing.Value,
Microsoft.Office.Interop.Excel.XlSearchOrder.xlByColumns, Microsoft.Office.Interop.Excel.XlSearchDirection.xlPrevious,
false, System.Reflection.Missing.Value, System.Reflection.Missing.Value).Column;
//build data row to hold file path and column count
DataRow dr = dt.NewRow();
dr["FilePath"] = excelFile;
dr["ColumnCount"] = columnCount;
dt.Rows.Add(dr);
xlApp.Workbooks.Close();
xlApp.Quit();
xlWorkbook = null;
xlApp = null;
}
GC.Collect();
GC.WaitForPendingFinalizers();
//populate object variable
Dts.Variables["User::SSISObjectVariable"].Value = dt;
If you need to import excels with different schemas, you have two approaches:
(1) SSIS approach: Script Task + 2 Data flow tasks
In case that you only have two structures, then you can follow these steps:
Add a variable of type System.Int32 example: #[User::ColumnsCount]
Add a variable of type System.String to store the file path example: #[User::FilePath]
Add a script Task with and Select #[User::FilePath] as ReadOnly variable and #[User::ColumnsCount] as ReadWrite Variable
Inside the script Task write a similar script:
string FilePath = Dts.Variables["User::FilePath"].Value.toString();
string ExcelConnectionString = "Provider=Microsoft.ACE.OLEDB.12.0;
"Data Source='" + FilePath +
"';Extended Properties=\"Excel 12.0;HDR=YES;\"";
using (OleDbConnection OleDBCon = new OleDbConnection(ExcelConnectionString))
{
if (OleDBCon.State != ConnectionState.Open)
OleDBCon.Open();
using (OleDbCommand cmd = new OleDbCommand(strcommand, OleDBCon))
{
DataTable dtTable = new DataTable("Table1");
cmd.CommandType = CommandType.Text;
//replace Sheet1$ with the sheet name if it is different
cmd.CommandText = "SELECT * FROM Sheet1$"
using (OleDbDataAdapter daGetDataFromSheet = new OleDbDataAdapter(cmd))
{
daGetDataFromSheet.FillSchema(dtTable, SchemaType.Source);
Dts.Variables["User::ColumnsCount"].Value = dt.Columns.Count;
}
}
}
Add two Data Flow task, on for each Excel structure
Link the Script Task to each one of these Data Flow Task
Click on each Precedence constraint (link between tasks) and change the precendence type to Expression and Constraint and add the appropriate expression for each case:
5 columns:
#[User::ColumnsCount] == 5
6 columns:
#[User::ColumnsCount] == 6
Set the Delay Validation property to True for both Data Flow Tasks
TL DR: In case that you only have two structures, you can add two Data Flow Tasks (one for each structure), then you can use a Script Task to identify the columns count and execute the appropriate Data Flow Task based on the Columns Count (using precedence constraints expressions).
(2) C# approach: SchemaMapper class library
Recently i started a new project on Github, which is a class library developed using C#. You can use it to import tabular data from excel, word , powerpoint, text, csv, html, json and xml into SQL server table with a different schema definition using schema mapping approach. check it out at:
SchemaMapper: C# Schema mapping class library
You can follow this Wiki page for a step-by-step guide:
Import data from multiple files into one SQL table step by step guide
I have an access database for which I need to run a query that is available in postgres DBs, I was wondering if there is a possible that this can be accomplished:
Insert into Table (Col1,Col2...) values(Val1,Val2,...) returning * (ore even just an id defining that specific set of data that was just inserted)?
I'm using c# to communicate to the DB.Anything would help, thank you.
The code I basically use is the following :
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Data.OleDb;
namespace Testquery1
{
class Program
{
static void Main(string[] args)
{
string query = "INSERT INTO Table ( Val1, Val2, Val3 ) values(14,2,1)";
Test1 queryselect = new Test1();
queryselect.dataconnection(query);
}
}
class Test1
{
public OleDbConnection connection = new OleDbConnection();
string path = System.Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments);
string fileloc = #"DataBase.accdb";
string provider = #"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=";
public void dataconnection(string query)
{
connection.ConnectionString = provider + path + fileloc;
Console.WriteLine(connection.ConnectionString);
connection.Open();
OleDbCommand command = new OleDbCommand();
command.Connection = connection;
command.CommandText = query;
command.ExecuteNonQuery();
connection.Close();
}
}
}
Unfortunately with .net you cannot do a append or make table query between two different connections.
However, you CAN in Access.
If you have a working PostgreSQL SQL query that returns records, then you can simple save that query in Access (as a pass-through query.
You can now use that query in Access and send it to a NEW local table in Access. (Access supports this concept, .net does not)
You can thus either append or “make table” of that data.
And more interesting is you can even append between different connections in Access. So you can append from a PostgreSQL table to say a MySQL, or Oracle or FoxPro table inside of access.
Again, you can’t do this in .net.
So, assume a working raw SQL (PostgreSQL format) query that works in PostgreSQL? Take that SAME working query and save it in Access as a pass-through query.
Then in Access you can append to a table in Access (assuming same table structure with:
VBA (Access code)
Dim strSQL as string
strSQL = "INSERT INTO LocalTable SELECT * from QryPassR"
Currentdb.Execute strSQL
And if you want to MAKE a new table in Access with the SAME structure, so make table (not append), you can go:
Dim strSQL as string
strSQL = " SELECT * INTO LocalTable FROM qryPassR"
Currentdb.Execute strSQL
You can also in VBA code change the PostgreSQL to include criteria for that select.
(air code - does not take into account SQL injection issue).
Dim strCity as string.
strCity = inputbox("What city to pull from PostGres?")
dim strSQL as string
strSQL = "select * from tbleHotels where City = '" = strCity & "'"
With currentdb.QueryDefs("QryPassR"
.SQL = strSQL
End with
strSQL = "INSERT INTO LocalTable SELECT * from QryPassR"
Currentdb.Execute strSQL
‘ above will copy all the records from PostGreSQL of city = Edmonton into the Access table (called local table in this example).
And as noted, you not limited to “LocalTable” being a access table, it could be a FoxPro table, MySQL, SQL server etc. So you not limited to JUST using Access tables in the above with your given SQL. So any linked table you have in Access can be used – including ones that point to other database systems.
If you must use .net, then you have to:
Connect to first database.
Execute query to pull and fill a datatable.
Connect to second database.
Create (open) a data table based on second database.
Loop (iterate) each row from first connection datatable and copy the row into the second datatable (based on 2nd connection).
You have to do a row by row copy. (but there is ImportRow method of the .net datatable, so you don’t have to loop by a column by column copy.
(but you have to loop row by row).
In Access this looping is not required and in fact you can use SQL commands that operate on both tables, including as per above the make table and append, and you can even do relation joins between such tables - even those based on different connections (data sources).
Edit
Based on comments, it looks like the simple question is:
After I insert a row into Access, how can I get the last ID (identity) of that insert?
The follow vb.net code will do this:
Imports System.Data.OleDb
Dim MyCon As New OleDb.OleDbConnection("Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\Test2\test44.accdb")
MyCon.Open()
Dim strSQL As String = "insert into tblHotels2 (City) VAlues('Edmonton')"
Dim cmd As New OleDb.OleDbCommand(strSQL, MyCon)
Dim r As Integer
cmd.ExecuteNonQuery()
cmd.CommandText = "select ##IDENTITY"
r = cmd.ExecuteScalar
Debug.Print(r)
Output = last PK id (autonumber)
I am trying to create an SQL statement using user-supplied data. I use code similar to this in C#:
var sql = "INSERT INTO myTable (myField1, myField2) " +
"VALUES ('" + someVariable + "', '" + someTextBox.Text + "');";
var cmd = new SqlCommand(sql, myDbConnection);
cmd.ExecuteNonQuery();
and this in VB.NET:
Dim sql = "INSERT INTO myTable (myField1, myField2) " &
"VALUES ('" & someVariable & "', '" & someTextBox.Text & "');"
Dim cmd As New SqlCommand(sql, myDbConnection)
cmd.ExecuteNonQuery()
However,
this fails when the user input contains single quotes (e.g. O'Brien),
I cannot seem to get the format right when inserting DateTime values and
people keep telling me that I should not do this because of "SQL injection".
How do I do it "the right way"?
Use parameterized SQL.
Examples
(These examples are in C#, see below for the VB.NET version.)
Replace your string concatenations with #... placeholders and, afterwards, add the values to your SqlCommand. You can choose the name of the placeholders freely, just make sure that they start with the # sign. Your example would look like this:
var sql = "INSERT INTO myTable (myField1, myField2) " +
"VALUES (#someValue, #someOtherValue);";
using (var cmd = new SqlCommand(sql, myDbConnection))
{
cmd.Parameters.AddWithValue("#someValue", someVariable);
cmd.Parameters.AddWithValue("#someOtherValue", someTextBox.Text);
cmd.ExecuteNonQuery();
}
The same pattern is used for other kinds of SQL statements:
var sql = "UPDATE myTable SET myField1 = #newValue WHERE myField2 = #someValue;";
// see above, same as INSERT
or
var sql = "SELECT myField1, myField2 FROM myTable WHERE myField3 = #someValue;";
using (var cmd = new SqlCommand(sql, myDbConnection))
{
cmd.Parameters.AddWithValue("#someValue", someVariable);
using (var reader = cmd.ExecuteReader())
{
...
}
// Alternatively: object result = cmd.ExecuteScalar();
// if you are only interested in one value of one row.
}
A word of caution: AddWithValue is a good starting point and works fine in most cases. However, the value you pass in needs to exactly match the data type of the corresponding database field. Otherwise, you might end up in a situation where the conversion prevents your query from using an index. Note that some SQL Server data types, such as char/varchar (without preceding "n") or date do not have a corresponding .NET data type. In those cases, Add with the correct data type should be used instead.
Why should I do that?
It's more secure: It stops SQL injection. (Bobby Tables won't delete your student records.)
It's easier: No need to fiddle around with single and double quotes or to look up the correct string representation of date literals.
It's more stable: O'Brien won't crash your application just because he insists on keeping his strange name.
Other database access libraries
If you use an OleDbCommand instead of an SqlCommand (e.g., if you are using an MS Access database), use ? instead of #... as the placeholder in the SQL. In that case, the first parameter of AddWithValue is irrelevant; instead, you need to add the parameters in the correct order. The same is true for OdbcCommand.
Entity Framework also supports parameterized queries.
VB.NET Example Code
This is the example code for the wiki answer in vb.net, assuming Option Strict On and Option Infer On.
INSERT
Dim sql = "INSERT INTO myTable (myField1, myField2) " &
"VALUES (#someValue, #someOtherValue);"
Using cmd As New SqlCommand(sql, myDbConnection)
cmd.Parameters.AddWithValue("#someValue", someVariable)
cmd.Parameters.AddWithValue("#someOtherValue", someTextBox.Text)
cmd.ExecuteNonQuery()
End Using
UPDATE
Dim sql = "UPDATE myTable SET myField1 = #newValue WHERE myField2 = #someValue;"
' see above, same as INSERT
SELECT
Dim sql = "SELECT myField1, myField2 FROM myTable WHERE myField3 = #someValue;"
Using cmd As New SqlCommand(sql, myDbConnection)
cmd.Parameters.AddWithValue("#someValue", someVariable)
Using reader = cmd.ExecuteReader()
' ...
End Using
' Alternatively: Dim result = cmd.ExecuteScalar()
' if you are only interested in one value of one row.
End Using
I am trying to retrieve Two Columns; App_ID & App_Slot from a table su_Appointments from an sql server database using vb, and I have two conditions the Date and Time; App_Date & App_Time for the query, now when I run the query it throws an error saying : Incorrect syntax near '2014'. The query is as follows
I am going to store App_ID into the variable AP_ID
CODE
Using Query As New SqlCommand("Select App_ID From su_Appointments Where (App_Date = ' and App_Time = ' )" & DT & TM, sqlcon)
sqlcon.Open()
Dim dr As SqlDataReader = Query.ExecuteReader()
While dr.Read()
AP_ID = dr(0)
End While
End Using
sqlcon.Close()
Well, your syntax is effectively wrong
A SELECT statement requires
SELECT <fieldA>, <FieldB>, ....
FROM <Table>
WHERE <FieldX> = <Condition1> AND <FieldZ> = <Condition2>
But, a part from this basic error, you need to start using a parameterized query approach
Using sqlcon = new SqlConnection(.....connectionstring...)
Dim cmdText = "Select App_ID From su_Appointments Where App_Date =#dt and App_Time = #tm"
Using Query = New SqlCommand(cmdText, sqlcon)
sqlcon.Open()
Query.Parameters.AddWithValue("#dt", DT)
Query.Parameters.AddWithValue("#tm", TM)
Using dr =Query.ExecuteReader()
While dr.Read()
AP_ID = dr(0)
End While
End Using
End Using
End Using
With a parameterized query, you get many benefits. There is no possibility of Sql Injection Attacks, the text of your command is more clear and understandable, the parameters are treated for correct quoting by the code itself so you don't need to check for single quotes inside your strings, or format correctly dates and decimal numbers.
Eventually, you could encounter a different problem. If your columns App_Date and App_Time are of type datetime then you need to pass parameters of the appropriate type, not simply strings. Instead if, these fields are of type nvarchar (or some other kind of text type) then you pass strings but you will have problems storing and querying correctly in these fields.
Is it possible (in Vb.Net 2005), without manually parsing the dataset table properties, to create the table and add it to the database?
We have old versions of our program on some machines, which obviously has our old database, and we are looking for a way to detect if there is a missing table and then generate the table based on the current status of the table in the dataset. We were re-scripting the table every time we released a new version (if new columns were added) but we would like to avoid this step if possible.
See this MSDN Forum Post: Creating a new Table in SQL Server from ADO.net DataTable.
Here the poster seems to be trying to do the same thing as you, and provides code that generates a Create Table statement using the schema contained in a DataTable.
Assuming this works as it should, you could then take that code, and submit it to the database through SqlCommand.ExecuteNonQuery() in order to create your table.
Here is the code:
SqlConnection con = new SqlConnection("Data Source=.;uid=sa;pwd=sa123;database=Example1");
con.Open();
string sql = "Create Table abcd (";
foreach (DataColumn column in dt.Columns)
{
sql += "[" + column.ColumnName + "] " + "nvarchar(50)" + ",";
}
sql = sql.TrimEnd(new char[] { ',' }) + ")";
SqlCommand cmd = new SqlCommand(sql, con);
SqlDataAdapter da = new SqlDataAdapter(cmd);
cmd.ExecuteNonQuery();
using (var adapter = new SqlDataAdapter("SELECT * FROM abcd", con))
using(var builder = new SqlCommandBuilder(adapter))
{
adapter.InsertCommand = builder.GetInsertCommand();
adapter.Update(dt);
}
con.Close();
I hope you got the problem solved.
Here dt is the name of the DataTable.
Alternatively you can replace:
adapter.update(dt);
with
//if you have a DataSet
adapter.Update(ds.Tables[0]);