The Python 3.8.1 code that I am working with makes several calls to stored procedures, gets results, does a simple select statement passing a string -- all successfully, using pypyodbc. My last call has me stumped as it simply fails to produce any results, or fails miserably. The stored procedure updates 1 column in 3 tables based on an OrderID, and inserts 1 row in another table. Using SSMS, the call would look like this:
exec GK_set_order '123456'
where 123456 is the OrderID. OrderID is a varchar(). I have logged into SSMS as the user in my Python connect statement and executed that stored procedure and it works every time.
In Python, the order number is used throughout as ADCOrderID, a string. My suspicion is that the single quotes are what is biting me. My first attempt was to create a separate string and execute that string.
exec_SP = "exec GK_set_order '" + ADCOrderID + "'"
cur.execute(exec_SP)
A print(exec_SP) statement shows that it is exactly what I want. But I get no errors, no exceptions, and nothing is updated in the database. I've tried 20 different variations, including double and triple quotes. Nothing was effective.
Changed to:
exec_SP = "exec GK_set_Order(?)"
cur.execute(exec_SP, ADCOrderID)
Now I get an error:
TypeError: Params must be in a list, tuple, or Row
What am I fundamentally missing?
Related
Invoke-Sqlcmd -ServerInstance '.' -Database 'MyDB'
-Query 'EXEC SprocA #param1= "value";EXEC SprocB #param1= "value";'
Basically I have my Invoke-SqlCmd running a query that invokes two stored procedures. Both the stored procedures output a bunch of rows.
However if sprocA does not output any results (empty select results or no rows), then the invoke command does not seem to print the output of the second sprocB even if it has data.
If I change the order of the stored procedures in my Invoke-SqlCmd commands query parameter, then this works perfectly and returns the output of the first stored procedure.
If I had three stored procedure calls where the first returns data and the second does not and the third does, it prints output of the first result and third result.
Basically it does not print any output only if the first stored procedure has no output. Seems weird.
Anything I can do to get around this SQL wise ? Could be a PowerShell thing?
I was also able to repro this with two Select statements where one returns data and the other does not.
This is the documented behavior of invoke-sqlcmd
When this cmdlet is run, the first result set that the script returns
is displayed as a formatted table. If subsequent result sets contain
different column lists than the first, those result sets are not
displayed. If subsequent result sets after the first set have the same
column list, their rows are appended to the formatted table that
contains the rows that were returned by the first result set.
It looks like both result sets are actually returned, but not output by defaut.
EG
PS C:\> Invoke-SqlCmd "select 1 a; select 2 b, 3 c;" | % { $_ | Out-Default }
outputs
a
-
1
b c
- -
2 3
Actually, it is hard to believe MS made such a mistake or maybe it's not a mistake. Whatever, when you run Invoke-SqlCmd and with a query like the following,
select * from table1 where id = 1111 -- non-exists id, this select returns nothing
select * from table2 where id = 2222 -- exists id, this select returns something
On SSMS you can see 2 result sets, the first one is empty.
However, the Invoke-SqlCmd doesn't return anything when the first result set is empty, even other result sets are not. My face was like ?_?
Another approach is to write you own invoke SQL function like following to return whatever result sets, even the empty ones.
$sql_Conn = New-Object System.Data.SqlClient.SQLConnection
$sql_Conn.ConnectionString = $sqlConnectionString
$sql_Conn.Open()
$sql_cmd = New-Object system.Data.SqlClient.SqlCommand($Query, $sql_Conn)
$sql_ds = New-Object system.Data.DataSet
$sql_da = New-Object system.Data.SqlClient.SqlDataAdapter($sql_cmd)
[void]$sql_da.fill($sql_ds)
$sql_Conn.Close()
return $sql_ds
Everything is fine until a new problem comes about the keyword GO in your script, you must know it if you use SSMS. The thing is, this GO is not a SQL command. it is just a separator used by SSMS. MS developed codes can handle the GO in a good manner, e.g. SSMS, Invoke-SqlCmd and sqlcmd.exe. If you use your own SQL invoke function you will get syntax issue
Incorrect syntax near 'GO'
While people most likely to ask you to update the SQL script to remove all GO lines, however, things are not always under controlled, normally need to work with different people and teams.
At last, I have to trim the GO in my scripts like the following
$Query = $Query -ireplace "(^|\r|\n)[ \t]*\bGO\b[ \t]*(\r|\n|$)", '$1$2'
https://github.com/LarrysGIT/Invoke-Sql
Of course, the story is not over, the more I am trying to automate SQL related tasks. The more issue found. There are multiple ways to automatically execute SQL script. None of them are perfect so far.
Invoke-Sql (My own script)
* Is able to handle the key separator 'GO'
* Is able to handle duplicate columns
* Fully support multiple result sets, even the first result set is empty
* Unable to handle `Create or alter` keywords if there are contents ahead
* Unable to handle special characters like '194 160' (non-breaking space) in SQL script (edited by some document edit tool, MS word for example)
Invoke-SqlCmd
* Is able to handle the key separator 'GO'
* Is able to handle special characters like 'non-breaking space'
* Unable to handle duplicate columns
* Unable to fully handle multiple result sets (when the first table is empty)
sqlcmd.exe
* Is able to handle all things (briefly tested)
* The returned result sets are plain text, hard to parse
Microsoft.SqlServer.SMO snapin
* The API of SQL server management studio
* Theoretically should be able to handle all cases
* Need to dig more
I have a table that contains just under a million rows. I'm building a form using SSIS that asks for user input and uses the values as parameters to build a view from source data. I'm having trouble getting SSIS to create the view from a variable.
The purpose of this 'tool' is to provide a dialogue that programmatically builds a view and later an update statement based upon parameters defined via a form that will execute an SSIS package. A number of the ppl on my team know 0 SQL. Therefore this circumvents any SQL knowledge. Creating an entirely standalone app is not ideal as it would require too much additional overhead on my side and would deviate from a number of our existing processes that currently use SSIS/SQL to achieve similar results.
With that here is what I've tried/trying.
I have an SSIS package that contains 'Execute SQL Task'
This task brings up a form with 5 inputs (variables)
var1,var2,var3,var4,var5.
some vars are strings others are doubles, ints etc... (they all vary)
You populate the fields and hit okay.
These variables are passed to an 'Execute Package Task'.
Inside this package (Package B)
the vars are used in an 'Execute SQL Task'.
This task is attempting to take the users input and create a view with a where clause containing 4 other variables.
example:
Create View ? AS Select col1,col2,col3,col4 WHERE
col1 = ?
AND col2 =?
AND col3 =?.........
First it appears that using ? in the create view is invalid.
The error being:
Error: 0x0 at Build_Query: Incorrect syntax near '#P1'.
Error: 0xC002F210 at Build_Query, Execute SQL Task: Executing the query "CREATE VIEW ? as Select * from S_t_equip_template
..." failed with the following error: "The batch could not be analyzed because of compile errors.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established
correctly.
Task failed: Build_Query
If I use the create view variable as an expression and remove the variables/paramerters for the where statements, I can create the view no problem.
However the where statements throw errors once I add them back in. I've tried evaluating these as an expression in the 'Execute SQL Task' but as these are of various types I get the error:
[Execute SQL Task] Error: Executing the query "
CREATE VIEW testing AS SELECT
P.label,P.uniq..." failed with the following error: "The metadata could not
be determined because statement 'CREATE VIEW testingagain AS SELECT
P.label,P.uniqueid,C.label as Child_Label,C.uniqueid as Child_uni' does not
support metadata discovery.". Possible failure reasons: Problems with the
query, "ResultSet" property not set correctly, parameters not set correctly,
or connection not established correctly.
No idea what is going on. Any help would be appreciated
I've googled the error and found some info but the other use-cases are so different that it's difficult to understand the actual cause, or another work around.
AS requested (simplified example):
I've created a package variable (datatype string) called: View_Name
Execute SQL Task:
CREATE VIEW #[User::View_Name] AS SELECT
* from table1
where col1 = 100;
Specifically it does not like that I use a variable here.
If I set the View name everything works until: I move on to my Where clauses that contain variables.
Create a variable called type (datatype int)
I map the variable/parameter in my sql task
Example:
CREATE VIEW tempTable AS SELECT
* from table1
where col1 = ?;
This won't work, same error.
If i attempt to do the above via an expression or expressions I get the following error
[Execute SQL Task] Error: Executing the query "CREATE VIEW test_45678 AS SELECT P.label,P.uniquei..." failed with the following error: "Must declare the scalar variable "#".". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.
Generally having to due with the variables cannot be evaluated this way. I'm guessing I'd need to evaluate each piece individually and build the expression piece by piece. That's fine but very inefficient and not maintainable.
I had some fundamental misunderstanding in my attempt to evaluate my expressions. Specifically syntax issues and having to cast each variable to a string.
My SQLstatement variable
ON C.pid = P.ID
where
C.width >="+(DT_WSTR, 8)#[User::Width] +"-"+ (DT_WSTR, 8)#[User::Range]+........
The final expression looks like so:
"Create View "+#[User::View_Name] + " AS SELECT " + #[User::SQLStatement]
I'm trying to create a wrapper in T-SQL for a procedure where I'm not sure what the data types are. I can run the wrapper without an INSERT INTO statement and I get the data just fine, but I need to have it in a table.
Whenever I use the INSERT INTO I get an error:
Column name or number of supplied values does not match table definition
I've parsed back through my code and can't see where any column names don't match up, so I'm thinking that it has to be a data type. I've looked through the procedure I'm wrapping to see if I can find what the data types are, but some aren't defined there; I've referenced the tables they pull some data from to find the definitions; I've run SQL_VARIANT_PROPERTY on all of the data to see what data type it is (although some of them come up null).
Is there some better way for me to track down exactly where the error is?
I think you can find out your stored procedure result schema, using sp_describe_first_result_set (available from SQL2012) and FMTONLY. Something like this:
EXEC sp_describe_first_result_set
#tsql = N'SET FMTONLY OFF; EXEC yourProcedure <params are embedded here>'
More details can be found here.
However, if I remember correctly, this works only if your procedure used deterministic schemas (no SELECT INTO #tempTable or similar things).
One trick to find out the schema of your result is to actually materialize the result into ad-hoc created table. However, this is not easy since SELECT INTO does not work with EXEC procedure. One work-around is this:
1) Define a linked-server to the instance itself. E.g. loopback
2) Execute your procedure like this (for SQL 2008R2):
SELECT * INTO tempTableToHoldDataAndStructure
FROM OPENQUERY(' + #LoopBackServerName + ', ''set fmtonly off exec ' + #ProcedureFullName + ' ' + #ParamsStr
where
#LoopBackServerName = 'loopback'
#ProcedureFullName = loopback.database.schema.procedure_name
#ParamsStr = embedded parameters
For SQL2012 I think the execution might fail if RESULT SETS are not provided (i.e. schema definition of the expected result, which is kind of a chicken-egg problem in this case):
' WITH RESULT SETS (( ' + #ResultSetStr + '))'');
Okay, I have a solution to my problem. It's tedious, but tedious I can do. Randomly guessing is what drives me crazy. The procedure I'm wrapping dumps 51 columns. I already know I can get it to work without putting anything into a table. So I decided to comment out part of the select statement in the procedure I'm wrapping so it's only selecting 1 column. (First I made a copy of that procedure so I don't screw up the original; then I referenced the copy from my wrapper). Saved both, ran it, and it worked. So far so good. I could have done it line by line, but I'm more of a binary kind of guy, so I went about halfway down--now I'm including about 25 columns in both the select statement and my table--and it's still working. Repeat procedure until it doesn't work any more, then backtrack until it does again. My error was in identifying one of the data types followed by "IDENTITY". I'm not sure what will happen when I leave that out, but at least my wrapper works.
I encounter some strange behavior with a dynamic SQL Query.
In a stored procedure I construct an insert query string out of multiple Strings. I execute the insert query in the SP like that - due to single nvarchar length restrictions.
EXEC(#QuerySelectPT+#QueryFromPT+#QueryFromPT)
If I print each part of the query, put these parts together and execute them manually in Management Studio the query works fine and inserts the data. But, if i execute the query in the EXEC() Method in the stored procedure, I get a
Column name or number of supplied values does not match table definition.
Error Message.
Did multiple check on the amount, spelling of columns in my query and in my insert table, but I have not found any differences so far.
Any advices?
Your count of columns for insert are different from count of columns for select. Print the statement before exec and find the error.
It as shot in the dark but seen you are telling the queries are valid and if you build the final query manually and it is working, the issue could be caused by string truncation.
Could you try:
EXEC(CAST(#QuerySelectPT AS VARCHAR(MAX))+#QueryFromPT+#QueryFromPT);
Also, as the Management Studio's message tab and selects are limited to 4000 symbols I think, you can test if the whole query is assembled correctly like this:
SELECT CAST(#QuerySelectPT+#QueryFromPT+#QueryFromPT AS XML)
So I have a stored procedure in SQL Server. I've simplified its code (for this question) to just this:
CREATE PROCEDURE dbo.DimensionLookup as
BEGIN
select DimensionID, DimensionField from DimensionTable
inner join Reference on Reference.ID = DimensionTable.ReferenceID
END
In SSIS on SQL Server 2012, I have a Lookup component with the following source command:
EXECUTE dbo.DimensionLookup WITH RESULT SETS (
(DimensionID int, DimensionField nvarchar(700) )
)
When I run this procedure in Preview mode in BIDS, it returns the two columns correctly. When I run the package in BIDS, it runs correctly.
But when I deploy it out to the SSIS catalog (the same server the database is on), point it to the same data sources, etc. - it fails with the message:
EXECUTE statement failed because its WITH RESULT SETS clause specified 2 column(s) for result set number 1, but the statement sent
3 column(s) at run time.
Steps Tried So Far:
Adding a third column to the result set - I get a different error, VS_NEEDSNEWMETADATA - which makes sense, kind of proof there's no third column.
SQL Profiler - I see this:
exec sp_prepare #p1 output,NULL,N'EXECUTE dbo.DimensionLookup WITH RESULT SETS ((
DimensionID int, DimensionField nvarchar(700)))',1
SET FMTONLY ON exec sp_execute 1 SET FMTONLY OFF
So it's trying to use FMTONLY to get the result set data ... needless to say, running SET FMTONLY ON and then running the command in SSMS myself yields .. just the two columns.
SET NOTCOUNT ON - Nothing changed.
So, two other interesting things:
I deployed it out to my local SQL 2012 install and it worked fine, same connections, etc. So it may be a server / database configuration. Not sure what if anything it is, I didn't install the dev server and my own install was pretty much click through vanilla.
Perhaps the most interesting thing. If I remove the join from the procedure's statement so it just becomes
select DimensionID, DimensionField from DimensionTable
It goes back to just sending 2 columns in the result set! So adding a join, without adding any additional output columns, ups the result set to 3 columns. Even if I add 6 more joins, just 3 columns. So one guess is its some sort of metadata column that only gets activated when there's a join.
Anyway, as you can imagine, it's driving me kind of mad. I have a workaround to load the data into a temp table and just return that, but why won't this work? What extra column is being sent back? Why only when I add a join?
Gah!
So all credit to billinkc: The reason is because of a patch.
In Version 11.0.2100.60, SSIS Lookup SQL command metadata is gathered using the old SET FMTONLY method. Unfortunately, this doesn't work in 2012, as the Books Online entry on SET FMTONLY helpfully notes:
Do not use this feature. This feature has been replaced by sp_describe_first_result_set.
Too bad they didn't follow their own advice!
This has been patched as of version 11.0.2218.0. Metadata is correctly gathered using the sp_describe_first_result_set system stored procedure.
This can happen if the specified WITH results set in SSIS identifies that there are more columns than being returned by the stored proc being called. Check your stored proc and ensure that you have the correct number of output columns as the WITH results set.