I am working on a project which consists transferring a few thousands of Excel rows to a SQL Server (it was also called T-SQL if I'm right?) database. I put together some logic in VBA to shape up the data.
Let me give you some context first about this. The data I'm about to transfer are invoice files. On each row, there are code of the stock items, prices, invoice number, invoice date, name of the client etc. These needs to be transferred to the database of a proprietary ERP system.
There are two tables on the database which I'm interested in for now. First one holds the header data for the invoice (client data, date, invoice number, invoice total etc.). Second table holds the information on the stock items (what has been sold, how many and for how much money etc).
After each insert onto the first table, I have to get the inserted row's primary key, in order to insert rows to the second table, which requires the PK of the first table on each row.
Now, my approach was to use the SCOPE_IDENTITY() function of the T-SQL. When I try to do it on the database directly via SQL Server Management Studio, it works without a hitch.
But when I try to use it in the code, it returns an empty recordset.
Code I'm using is as follows:
Public Function Execute(query As String, Optional is_batch As Boolean = False) As ADODB.Recordset
If conn.State = 0 Then
OpenConnection
End If
Set rs = conn.Execute(query) 'this is the actual query to be executed
Dim identity As ADODB.Recordset 'this rs supposed to hold the PK of inserted row, but returns an empty recordset
Set identity = conn.Execute("SELECT SCOPE_IDENTITY();")
If TypeName(identity.Fields(0).Value) = "Null" Then
pInsertedId = -1
Else
pInsertedId = identity.Fields(0).Value 'I'm saving it in an object variable, to access it easily later on
End If
Set Execute = rs 'to be returned to the caller
'closing the connection is handled outside this procedure
End Function
When I run this on VBA, second query SELECT SCOPE_IDENTITY();just returns an empty recordset. Same query works successfully when ran on the db directly.
Actually I'm able to pull this off by other means. There is a UUID column which I'm supposed to insert to the row in the first table. I can just simply query the table with this UUID and get the PK, but I'm just curious why this won't work.
Any ideas?
Your code doesn't insert any data, so no identity values are generated in the current scope, as defined in the official documentation for SCOPE_IDENTITY():
Returns the last identity value inserted into an identity column in the same scope. A scope is a module: a stored procedure, trigger, function, or batch. Therefore, if two statements are in the same stored procedure, function, or batch, they are in the same scope.
Your code effectively is the same as inserting data in one query window in SSMS and querying SCOPE_IDENTITY() in another query window. Well, this isn't how it works. You must query it in the same scope, i.e. a stored procedure, trigger, function, or batch. Otherwise, use ID values generated by you and insert them with the data.
Related
I want to set local variables or pass parameters from Excel to SQL. I've found similar questions, but all referred to old versions of Excel and/or the answers showed how to filter or manipulate output from a generic SQL query in the Power Query Editor, rather than pass a parameter or modify the SQL, so that the SQL Server supplies data in the needed form.
I'm building a large Microsoft Excel spreadsheet that depends on ten different SQL queries, all against a common SQL Server database. Excel and SQL Server are installed on my laptop and are current versions (as of 16 Mar 2022). All ten queries share a common date restriction, imposed in the WHERE clauses of the queries. The tables accessed and the form of output are very different, so there is no easy way to combine the ten queries into a single query. The queries contain multiple levels of aggregation (e.g. SUM(...)) so I need to restrict the records accessed prior to aggregation and passing results from the query back to Excel.
As currently written, each query begins by setting two date values in local variables. For example,
DECLARE #BEGIN_DATE AS smalldatetime;
DECLARE #END_DATE AS smalldatetime;
#BEGIN_DATE = CAST('2021-03-01 00:00' AS smalldatetime);
#END_DATE = CAST('2021-03-02 23:59' AS smalldatetime);
Every one of the ten queries includes a line in the WHERE clause similar to
WHERE
PickUpDate BETWEEN #BEGIN_DATE AND #END_DATE
Every query will use the same pair of dates. However, the column filtered (PickUpDate above) changes from one query to the next.
As it is, I have to manually edit each of the ten queries to change the two dates--twenty edits in all. This is time-consuming and error-prone. Ideally, I'd like to set the date range in the spreadsheet, in a pop-up dialog box, or any other convenient way and pass the dates to the SQL queries. Then by selecting Data > Refresh All in Excel, update all my tables at once.
Is this possible, and if so, how does one do it?
The answer from David Browne is generally on-target. But I found some difficulties reading data from an Excel table directly into the SQL, given security restrictions in the latest version of Excel/Power Query. Also, since this was the first time I worked directly in M-code and the advanced editor, it was challenging to fill-in the gaps.
I finally got a nice solution running; here is what worked for me.
First, I stored the parameter values in a two-column table. My table is named "ParameterTable" with column headers named "Parameter_Name" and "Value". The value(s) to pass to SQL Server are stored in the Value column. My table has two rows with row entries labeled "Begin_DateTime" and "End_DateTime".
Secondly I created a callable function named “ftnGetParameter.” Select Data > Get Data > From Other Sources > Blank Query. Then select “Advanced Editor.” Delete any boilerplate added by Excel, and enter and save this function
let theParameter=(TableName,ParameterLabel) =>
let
Source=Excel.CurrentWorkbook(){[Name=TableName]}[Content],
value = Source{[Parameter_Name=ParameterLabel]}[Value]
in
value
in
theParameter
Thirdly, code-up your SQL statement as usual. I was trying to pass dates to SQL, so I initially coded with string literals. Enter the query in the usual way. I used Data > Get Data > From Database > From SQL Server Database. Then pasted in the SQL. The two relevant lines in my query looked like this:
DECLARE #BEGIN_DATE AS SMALLDATETIME='2021-01-01 00:00';
DECLARE #END_DATE AS SMALLDATETIME='2021-12-31 23:59';
You could skip this step, but it allowed me to get complex SQL code entered, formatted, and running before I invoked the function to pass the parameters.
Finally, simply replace the string literals in the SQL with code to call the function. My first few lines of M-code looks like this:
let
Source = Sql.Database("DESKTOP-04P8E8C", "nfbdata",
[Query=
"
DECLARE #BEGIN_DATE AS SMALLDATETIME= '" & ftnGetParameter("ParameterTable","Begin_DateTime") & "';
DECLARE #END_DATE AS SMALLDATETIME='" & ftnGetParameter("ParameterTable","End_DateTime") & "' (… the query continues )
Excel will issue some warnings about running the query and prompt you to edit permissions. Once permission has been granted, the function reads the text from the parameter table and passes it into the SQL.
I found that the function call was not optional. Apparently, importing the code directly into a native call to SQL Server is considered an unacceptable security risk.
Many thanks to Mr. David Browne. His post definitely points in the right direction.
You can reference a table on a sheet from Power Query and integrate values from that table into your other queries. Eg if ParameterTable is a single-row table on some worksheet with a column called "StartDate", something like
let
theDate = Date.From( Record.Field(Table.First(ParameterTable),"StartDate") ),
Source = Sql.Databases("localhost"),
AdventureWorksDW2017 = Source{[Name="AdventureWorksDW2017"]}[Data],
dbo_DimDate = AdventureWorksDW2017{[Schema="dbo",Item="DimDate"]}[Data],
#"Filtered Rows" = Table.SelectRows(dbo_DimDate, each [FullDateAlternateKey] = theDate )
in
#"Filtered Rows"
for M query folding, or
let
theDate = Date.From( Record.Field(Table.First(ParameterTable),"StartDate") ),
sql = "
select *
from dimDate
where FullDateAlternateKey = '" & Text.From(theDate) & "'
",
Source = Sql.Database("localhost", "adventureworksdw2017", [Query=sql])
in
Source
for dynamic SQL.
I am trying to create a global temp table using the results from one query, which can then be selected as a table and manipulated further several times without having to reprocess the data over and over.
This works perfectly in SQL management studio, but when I try to add the table through an Excel query, the table can be referenced at that time, but it is not created in Temporary Tables in the tempdb database.
I have broken it down into a simple example.
If I run this in SQL management studio, the result of 1 is returned as expected, and the table ##testtable1 is created in Temporary Tables
set nocount on;
select 1 as 'Val1', 2 as 'Val2' into ##testtable1
select Val1 from ##testtable1
I can then run another select on this table, even in a different session, as you'd expect. E.g.
Select Val2 from ##testtable1
If I don't drop ##testtable1, running the below in a query in Excel returns the result of 2 as you'd expect.
Select Val2 from ##testtable1
However, if I run the same Select... into ##testtable1 query directly in Excel, that correctly returns the result of 1, but the temptable is not created.
If I then try to run
Select Val2 from ##testtable1
As a separate query, it errors saying "Invalid object name '##testtable1'
The table is not listed within Temporary Tables in SQL management studio.
It is as if it is performing a drop on the table after the query has finished executing, even though I am not calling a drop.
How can I resolve this?
Read up on global temp tables(GTT). They persist as long as there is a session referencing it. In SSMS, if you close the session that created the GTT prior to using it in another session, the GTT would be discarded. This is what is happening in Excel. Excel creates a connection, executes and disconnects. Since there are no sessions using the GTT when Excel disconnects, the GTT is discarded.
I would highly recommend you create a normal table rather than use a GTT. Because of their temporary nature and dependence on an active session, you may get inconsistent results when using a GTT. If you create a normal table instead, you can be certain it will still exist when you try to use it later.
The code to create/clean the table is pretty simple.
IF OBJECT_ID('db.schema.tablename') IS NOT NULL
TRUNCATE TABLE [tablename]
ELSE
CREATE [tablename]...
GO
You can change the truncate to a delete to clean up a specific set of data and place it at the start of each one of your queries.
is it possible you could use a view? assuming that you are connecting to 5 DBs on the same server can you union the data together in a view:
CREATE VIEW [dbo].[testView]
AS
SELECT *
FROM database1.dbo.myTable
UNION
SELECT *
FROM database2.dbo.myTable
Then in excel:
Data> New Query > From Database > FromSQL Server Database
enter DB server
Select the view from the appropriate DB - done :)
OR call the view however you are doing it (e.g. vba etc.)
equally you could use a stored procedure and call that from VBA .. basically anything that moves more of the complexity to the server side to make your life easier :D
You can absolutely do this. Notice how I'm building a temp table from SQL called 'TmpSql' ...this could be any query you want. Then I set it to recordset 1. Then I create another recordset 2, that goes and gets the temp table data.
Imagine if you were looping on the first cn.Execute where TmpSql is changing.. This allows you to build a Temporary table coming from many sources or changing variables. This is a powerful solution.
cn.open "Provider= ..."
sql = "Select t.* Into #TTable From (" & TmpSql & ") t "
Set rs1 = cn.Execute(sql)
GetTmp = "Select * From #TTable"
rs2.Open GetTmp, cn, adOpenDynamic, adLockBatchOptimistic
If Not rs2.EOF Then Call Sheets("Data").Range("A2").CopyFromRecordset(rs2)
rs2.Close
rs1.Close
cn.Close
I opened a report I started in BIDS in MS SQL Server Report Builder 3.0, as I read an answer here on SO that said that was the easiest way to create a table containing all the values in a Dataset.
So I opened my .rdl file there, selected the Insert tab, then Table > Table Wizard, and the dataset from the "Choose an existing dataset in this report or a shared dataset" list.
When I select the "Next" button of the wizard, though, all lists are empty (Available fields, Column groups, Row groups, Values).
If I select "Next" again, I get, "The values field list must contain at least one field."
Those are auto-populated, though, and, as written above, are as empty as a politican's brain.
Is it because my dataset is a StoredProc, and returns data from a temp table? If so, is there a workaround?
Note: I also tried the Matrix > Matrix Wizard, with the same results.
UPDATE
Also and doubtless relatedly (no pun intended), when I try to run the report from within ReportBuilder, I see:
What a revoltin' development!
UPDATE 2
And when I return to BIDS to work on the project and try to add an Expression in a Matrix, in the Edit Expression dialog, on selecting the Dataset of interest, I get, " dataset has no fields."
Ay, caramba!
UPDATE 3
In response to lrb's answer: I don't know if my SP is really unparseable or not; it does return values from a temp table - Here is the end of it:
SELECT PLATYPUSDESCRIPTION, WEEK1USAGE, WEEK2USAGE, USAGEVARIANCE,
WEEK1PRICE, WEEK2PRICE, PRICEVARIANCE, PRICEVARIANCEPERCENTAGE
FROM #TEMPCOMBINED
ORDER BY PLATYPUSDESCRIPTION;
Could that (using a temp table) be the problem?
UPDATE 4
When adding an Expression to a textbox like so:
=Fields!PLATYPUSDESCRIPTION.Value
...I get the following fingerwag on the Preview tab:
The definition of the report 'bla' is invalid. The Value expression for the textbox 'textbox4' refers to the field 'PLATYPUSDESCRIPTION'. Report item expressions can only refer to fields within the current data set scope or, if inside an aggregate, the specified data set scope.
Surely there's a way to use results from temp tables in an SSRS report, no es cierto?
This will happen when the query or stored procedure can not be parsed with certainty. For example, if your data set is a store procedure that returns something like the following:
IF(#SomVariable=1)
SELECT 1,2,3,4
ELSE
SELECT 'A','B','C'
The above logic in a SP would be horrible, however, the field name and datatypes can not be determined. The same holds true in other edge case scenarios.
What you can do for a work around is to trick the parser by modifying your sp and offering up a clean return statement, then changing the sp back to its original form. Since the metadata is persistent until the next refresh, your values will hold. NOTE : If the problem occurs when returning temporary tables in your dataset see #4 below.
1. Modify your existing stored procedure
ALTER PROCEDURE MyProcedureThatDoesNotParse()
AS
BEGIN
/*COMMENT OUT CURRENT SP LOGIC
...
*/
SELECT
MyField1=1,
MyField2='String',
MyField3=0.01
END
2. IN SSRS Refresh the fields for your dataset.
NOTE : You will see MyField1,MyField2 and MyField3 in the fields list.
3. Revert the changes to your stored procedure.
4. For queries or SP's that return a local #temporary table, global ##temporary table or a table valued #variable, it seems that aliasing the temp structure works. I.E
SELECT * FROM #TABLE --Does not always parse in SSRS
SELECT * FROM #TABLE T --Seems to be able to be parsed by SSRS
Change command type on the report builder, choose " text " and write exec yourprocedurename. It will work
I'm building a small application and I came across an issue that I need to resolve. When I insert a new client into the SQL-SERVER I need to create an ID number to go with the client. I have a value, say - 1000 in a reference table, that gets pulled from the table, incremented by 1, then put back into the ref table, and the value 1001 gets assigned to the client. Before it gets saved to the client, I add 'TOL' to the number - so when save is complete, the ID is TOL1001. The issue I need to resolve is to check the tblClient_TABLE, to make sure that ID TOL1001 doesn't exist already before doing the insert for a new client.
I'm not really sure where I should do it, because on SAVE, I call the function that increments the number, assigns TOL to it and stores the value in an invisible textbox, so when I do my insert, it just pulls the value from the textbox...
strSQL = "INSERT into tblClient_TABLE (ID) values ("txtIDnumber.text")
I obviously have more data to insert, it's just i'm struggling with finding a logic way to check for the already existing ID.
Thanks for any ideas, help!
Your database is able to use identity columns (=autoincrement). So, if you insert a new record, an identity column will get the next value (you can rely on the uniqueness).
How do you get this number? The insert statement has (for mssql) an "output inserted" clause, and if you use ado with executescalar you get your inserted id.
The SQL command (add the vb6 code for ado command you must)
INSERT INTO [TABLENAME] ( [COL1], [COL2] ) OUTPUT INSERTED.MYID VALUES ( #COL1, #COL2 )
... add your Parameter values here ....
result = adoCommand.ExecuteScalar()
(something like that, don´t have VB6 at the office ...)
I'm using the following sql for an ADO recordset against a SQLServer backend in VB6:
select c.name, taxid=
case when exists(select 1 from sometable where fld='abc')
then c.SSN else null end
When I try to update the taxid field in a row within the recordset locally ADO complains with the error "Multiple-step operation generated errors. Check each status value." I assume it's bothered by the fact that the taxid field is coming from a calculated field and not a raw table column. For my purposes I'm never going to be persisting these changes back to the database so I'm looking for a way to tell ADO that have no intent to persist changes so that it will allow me to change the data locally.
I think that #HK1's suggestion is a good one, though I'm not sure what happens to your ability to alter any of the values in the recordset whether the column you're trying to update is computed or not. It's been a long time since I played with classic ADO but if the recordset is disconnected it may become read only at that point.
But if you have no interest in using the recordset to perform updates, and you need to alter values locally, perhaps you should consider storing the results in a local array first? That way you can minimize the locking and cursor options of the recordset, for example, and immediately close the recordset and free up those resources.
rs.Open cmd, conn, adOpenForwardOnly, adLockReadOnly
Dim MyArray
MyArray = rs.GetRows()
rs.Close: set rs = nothing
Now you can manipulate MyArray however you want...