Trying to send data to a stored procedure that accepts Table Valued Parameter, neither it does any action nor returns any error.
def InsertResults (self, dataset, name):
try:
sqlConn = self.sqlConnect()
with sqlConn.cursor() as cursor:
cursor.execute(f'CALL [dbo].[ImportAllocations] (?)', (dataset,))
result = cursor.fetchall()
print(result)
except AssertionError as error:
print(error)
finally:
print(f"Inserted data into {name}. Records inserted {len(dataset)}")
My data is well formed in DataFrame.
This is stored procedure header:
CREATE PROC [dbo].[ImportAllocations](#DataTable [dbo].[udt_Allocation] READONLY)
Looking at SQL Profiler, and it seems to be ignoring the call. I tried some other simple example as per in following issue, and that works fine.
The issue was with pandas Dataframe. This works once converted into tuples of single element without index.
listTuple = list(dataset.itertuples(index=False))
cursor.execute(f'EXEC {name} ?', (listTuple,))
Related
I am using SQL Server 2016.
I have a stored procedure GET_RECORDS that takes input parameters for filter and outputs a CURSOR parameter
I want to get this cursor in my SSIS package
I had created data flow task, OleDb source and variables for parameter values. Then mapped parameters
Params mapping screen
but when I wanted to save the component - I got an error
error screen
I tried to add clause WITH RESULT SETS with some dummy columns, but my procedure doesn't return any result set
What am I doing wrong?
Any advices will be helpful.
Thank you.
With regards, Yuriy.
The source component is trying to determine what columns and types will be returned. Because you are using dynamic SQL the metadata can change each time you run it.
With result sets allows you to define the data being returned but should only be used if you are guaranteed to have those results every time you execute.
EDIT:
I create a connection and run the command so that it populates a data table. Then I put the column headers into a string array. There are plenty of examples out there.
Then I use the following function to create a destination table. Finally I create a datareader and pass that to the .Net SqlBulkCopy. Hope this helps.
private void CreateTable(string TableName, string[] Fields)
{
if (TableExists(TableName) && Overwrite)
{
SqlCommand = new SqlCommand($"Drop Table [{TableName}]", SqlConnection);
SqlCommand.ExecuteNonQuery();
}
string Sql = $"Create Table [{TableName}] (";
int ColumnNumber = 1;
foreach (string Field in Fields)
{
string FieldValue = Field;
if (! HasHeaders)
{
FieldValue = "Column" + ColumnNumber;
ColumnNumber++;
}
Sql += $"[{FieldValue}] Varchar(8000),";
}
Sql = Sql + "ImportFileID Int, ID Int Identity(1,1) Not Null, Constraint [PK_" + TableName + "] Primary Key Clustered ([ID] Asc))";
SqlCommand = new SqlCommand(Sql, SqlConnection);
SqlCommand.ExecuteNonQuery();
}
Use ado.net source instead of oledb source, define a simple select and get the columns you wish to return. Now you can define expresión in the dataflow properties.
Search ado.net source dynamic sql
:)
try to return the records and use foreach in ETL instead of cursor
https://www.simple-talk.com/sql/ssis/implementing-foreach-looping-logic-in-ssis/
I think you can do it from a simple way, but I don't know what you are you doing, exactly...
Given:
CREATE PROCEDURE my_procedure
#Param INT
AS
SELECT Col1, Col2
FROM Table
WHERE Col2 = #Param
I would like to be able to use this as:
import pandas as pd
import pyodbc
query = 'EXEC my_procedure #Param = {0}'.format(my_param)
conn = pyodbc.connect(my_connection_string)
df = pd.read_sql(query, conn)
But this throws an error:
ValueError: Reading a table with read_sql is not supported for a DBAPI2 connection. Use an SQLAlchemy engine or specify an sql query
SQLAlchemy does not work either:
import sqlalchemy
engine = sqlalchemy.create_engine(my_connection_string)
df = pd.read_sql(query, engine)
Throws:
ValueError: Could not init table 'my_procedure'
I can in fact execute the statement using pyodbc directly:
cursor = conn.cursor()
cursor.execute(query)
results = cursor.fetchall()
df = pd.DataFrame.from_records(results)
Is there a way to send these procedure results directly to a DataFrame?
Use read_sql_query() instead.
Looks like #joris (+1) already had this in a comment directly under the question but I didn't see it because it wasn't in the answers section.
Use the SQLA engine--apart from SQLAlchemy, Pandas only supports SQLite. Then use read_sql_query() instead of read_sql(). The latter tries to auto-detect whether you're passing a table name or a fully-fledged query but it doesn't appear to do so well with the 'EXEC' keyword. Using read_sql_query() skips the auto-detection and allows you to explicitly indicate that you're using a query (there's also a read_sql_table()).
import pandas as pd
import sqlalchemy
query = 'EXEC my_procedure #Param = {0}'.format(my_param)
engine = sqlalchemy.create_engine(my_connection_string)
df = pd.read_sql_query(query, engine)
https://code.google.com/p/pyodbc/wiki/StoredProcedures
I am not a python expert, but SQL Server sometimes returns counts for statement executions. For instance, a update will tell how many rows are updated.
Just use the 'SET NO COUNT;' at the front of your batch call. This will remove the counts for inserts, updates, and deletes.
Make sure you are using the correct native client module.
Take a look at this stack overflow example.
It has both a adhoc SQL and call stored procedure example.
Calling a stored procedure python
Good luck
This worked for me after added SET NOCOUNT ON thanks #CRAFTY DBA
sql_query = """SET NOCOUNT ON; EXEC db_name.dbo.StoreProc '{0}';""".format(input)
df = pandas.read_sql_query(sql_query , conn)
Using ODBC syntax for calling stored procedures (with parameters instead of string formatting) works for loading dataframes using pandas 0.14.1 and pyodbc 3.0.7. The following examples use the AdventureWorks2008R2 sample database.
First confirm expected results calling the stored procedure using pyodbc:
import pandas as pd
import pyodbc
connection = pyodbc.connect(driver='{SQL Server Native Client 11.0}', server='ServerInstance', database='AdventureWorks2008R2', trusted_connection='yes')
sql = "{call dbo.uspGetEmployeeManagers(?)}"
params = (3,)
cursor = connection.cursor()
rows = cursor.execute(sql, params).fetchall()
print(rows)
Should return:
[(0, 3, 'Roberto', 'Tamburello', '/1/1/', 'Terri', 'Duffy'), (1, 2, 'Terri', 'Duffy',
'/1/', 'Ken', 'Sánchez')]
Now use pandas to load the results into a dataframe:
df = pd.read_sql(sql=sql, con=connection, params=params)
print(df)
Should return:
RecursionLevel BusinessEntityID FirstName LastName OrganizationNode \
0 0 3 Roberto Tamburello /1/1/
1 1 2 Terri Duffy /1/
ManagerFirstName ManagerLastName
0 Terri Duffy
1 Ken Sánchez
EDIT
Since you can't update to pandas 0.14.1, load the results from pyodbc using pandas.DataFrame.from_records:
# get column names from pyodbc results
columns = [column[0] for column in cursor.description]
df = pd.DataFrame.from_records(rows, columns=columns)
Given:
In the original post I was attempting to call db.Database.ExecuteCommand, since then I changed that to what you see here. (Which works). Execute Command was always returning 0xfffff. SQLQuery returns the value of the last select statement or the rows inserted depending on how you code it up.
using (db=new MyEntites())
{
var inserted = db.Database.SqlQuery<decimal>(query,parms}
}
Where last two lines in query are:
SET NOCOUNT OFF
Select SCOPE_IDENTITY() AS newID
From the documentation for ExecuteSqlCommand method, the return value is
The result returned by the database after executing the command.
This means you need to RETURN the value in your procedure, not as a result set. So do this in your stored procedure as the last command:
RETURN SCOPE_IDENTITY()
Thanks to your suggestions, Step 1 put something like this in the query string as last select:
Select SCOPE_IDENTITY() as TESTID;
Then use this EF form of SQLQuery, the type of decimal is what is returned for the select example above.
using (db=new MyEntites())
{
var identity = db.Database.SqlQuery<decimal>(query,parms}
}
Say you have a stored procedure or function returning multiple rows, as discussed in How to return multiple rows from the stored procedure? (Oracle PL/SQL)
What would be a good way, using Scala, to "select * from table (all_emps);" (taken from URL above) and read the multiple rows of data that would be the result?
As far as I can see it is not possible to do this using Squeryl. Is there a scalaified tool like Squeryl that I can use, or do I have to drop to JDBC?
Functions that return tables are an Oracle specific feature, I doubt an ORM (be it Scala or even Java) would have support for such a proprietary extension.
So I think you're more or less on your own :).
Probably the easiest way is to use a plain JDBC java.sql.Statement and execute "select * from table (all_emps)" with the executeQuery method.
To address the second part of your question about a way to select from table in a more scala-esque way, I am using Slick. Quoting from their example documentation:
case class Coffee(name: String, supID: Int, price: Double)
implicit val getCoffeeResult = GetResult(r => Coffee(r.<<, r.<<, r.<<))
Database.forURL("...") withSession {
Seq(
Coffee("Colombian", 101, 7.99),
Coffee("Colombian_Decaf", 101, 8.99),
Coffee("French_Roast_Decaf", 49, 9.99)
).foreach(c => sqlu"""
insert into coffees values (${c.name}, ${c.supID}, ${c.price})
""").execute)
val sup = 101
val q = sql"select * from coffees where sup_id = $sup".as[Coffee]
// A bind variable to prevent SQL injection ^
q.foreach(println)
}
Though I am not sure how it's dealing (if at all) with stored procs/functions.
I want to map a SQL Server stored procedure with MyBatis, using annotations.
#Select(value = "{call sp_cen_obliczcene(" +
"#{wytworId, mode=IN, jdbcType=NUMERIC}, " +
"#{rodzajCenyId, mode=IN, jdbcType=NUMERIC}, " +
"#{walutaId, mode=IN, jdbcType=NUMERIC}, " +
"#{jmId, mode=IN, jdbcType=NUMERIC}, " +
"#{ilosc, mode=IN, jdbcType=DECIMAL}, " +
"#{data, mode=IN, jdbcType=DATE})}")
#Result(property = "kwota", column = "kwota", javaType = BigDecimal.class, jdbcType = JdbcType.DECIMAL)
#Options(statementType = StatementType.CALLABLE)
public DtoCena dajCene(CriteriaCena parametry);
The procedure selects one row - I am interested in one column. Now, I've mapped a procedure before, only I had multiple rows and selected more then one column from them. Everything worked perfectly fine. When I mapped new procedure, in a similar way I got an error:
### The error occurred while setting parameters
### SQL: {call sp_cen_obliczcene(?, ?, ?, ?, ?, ?)}
### Cause: java.lang.NullPointerException
I started the SQL Profiler and saw that the procedure is called properly with the given parameters. I've noticed that the procedure I'm mapping is executing other procedures. They're performing some updates. When I changed my annotation to #Update I got an other error: that Integer cannot be cast to DtoCena type. I changed the return value of the method to Integer and I got no errors but as you can guess it did not return what I was looking for.
The question is, can I map a stored procedure which updates tables AND returns a ResultSet? I can do this using JDBC, but is this possible with MyBatis? Am I doing something wrong when using the #Select annotation?
Looks like the #Update returns the affected row count ...
Anyway, I don't think the issue is related to calling stored procedure, this is merely a mapping issue that would occur with simple select.
You must use #Result annotation inside #Results annotation, otherwise it is ignored.
Here is a simplified, yet functional, code:
#Select("select 'hello' as h, 1 as n from dual")
#Results({
#Result(column="n")
})
Integer test();
Just add a property attribute and change return type to retrieve result into an object.