I have a database (MS SQL) with a column that contains a query string. I want to take that query string and move it into a column that stores some settings in Json.
I could manually do this if it was only a few columns in one database, but this is part of a major upgrade that will be pushed out to over 50 sites. I'd like to have a T-SQL script that I could run on every database to perform this task for me.
Here's a sample of what the data will look like in the first column (as a query string):
KEY1=VALUE1&KEY2=VALUE2
I'd like to format that like so:
{"KEY1":"VALUE1","KEY2":"VALUE2"}
I'd appreciate any ideas you can throw my way!
Maybe you can use this
SELECT '{"' + REPLACE(REPLACE(#x, '=', '":"'), '&', '","') + '"}'
of course #x is your column I was testing with #x = N'KEY1=VALUE1&KEY2=VALUE2'
Related
I want to set local variables or pass parameters from Excel to SQL. I've found similar questions, but all referred to old versions of Excel and/or the answers showed how to filter or manipulate output from a generic SQL query in the Power Query Editor, rather than pass a parameter or modify the SQL, so that the SQL Server supplies data in the needed form.
I'm building a large Microsoft Excel spreadsheet that depends on ten different SQL queries, all against a common SQL Server database. Excel and SQL Server are installed on my laptop and are current versions (as of 16 Mar 2022). All ten queries share a common date restriction, imposed in the WHERE clauses of the queries. The tables accessed and the form of output are very different, so there is no easy way to combine the ten queries into a single query. The queries contain multiple levels of aggregation (e.g. SUM(...)) so I need to restrict the records accessed prior to aggregation and passing results from the query back to Excel.
As currently written, each query begins by setting two date values in local variables. For example,
DECLARE #BEGIN_DATE AS smalldatetime;
DECLARE #END_DATE AS smalldatetime;
#BEGIN_DATE = CAST('2021-03-01 00:00' AS smalldatetime);
#END_DATE = CAST('2021-03-02 23:59' AS smalldatetime);
Every one of the ten queries includes a line in the WHERE clause similar to
WHERE
PickUpDate BETWEEN #BEGIN_DATE AND #END_DATE
Every query will use the same pair of dates. However, the column filtered (PickUpDate above) changes from one query to the next.
As it is, I have to manually edit each of the ten queries to change the two dates--twenty edits in all. This is time-consuming and error-prone. Ideally, I'd like to set the date range in the spreadsheet, in a pop-up dialog box, or any other convenient way and pass the dates to the SQL queries. Then by selecting Data > Refresh All in Excel, update all my tables at once.
Is this possible, and if so, how does one do it?
The answer from David Browne is generally on-target. But I found some difficulties reading data from an Excel table directly into the SQL, given security restrictions in the latest version of Excel/Power Query. Also, since this was the first time I worked directly in M-code and the advanced editor, it was challenging to fill-in the gaps.
I finally got a nice solution running; here is what worked for me.
First, I stored the parameter values in a two-column table. My table is named "ParameterTable" with column headers named "Parameter_Name" and "Value". The value(s) to pass to SQL Server are stored in the Value column. My table has two rows with row entries labeled "Begin_DateTime" and "End_DateTime".
Secondly I created a callable function named “ftnGetParameter.” Select Data > Get Data > From Other Sources > Blank Query. Then select “Advanced Editor.” Delete any boilerplate added by Excel, and enter and save this function
let theParameter=(TableName,ParameterLabel) =>
let
Source=Excel.CurrentWorkbook(){[Name=TableName]}[Content],
value = Source{[Parameter_Name=ParameterLabel]}[Value]
in
value
in
theParameter
Thirdly, code-up your SQL statement as usual. I was trying to pass dates to SQL, so I initially coded with string literals. Enter the query in the usual way. I used Data > Get Data > From Database > From SQL Server Database. Then pasted in the SQL. The two relevant lines in my query looked like this:
DECLARE #BEGIN_DATE AS SMALLDATETIME='2021-01-01 00:00';
DECLARE #END_DATE AS SMALLDATETIME='2021-12-31 23:59';
You could skip this step, but it allowed me to get complex SQL code entered, formatted, and running before I invoked the function to pass the parameters.
Finally, simply replace the string literals in the SQL with code to call the function. My first few lines of M-code looks like this:
let
Source = Sql.Database("DESKTOP-04P8E8C", "nfbdata",
[Query=
"
DECLARE #BEGIN_DATE AS SMALLDATETIME= '" & ftnGetParameter("ParameterTable","Begin_DateTime") & "';
DECLARE #END_DATE AS SMALLDATETIME='" & ftnGetParameter("ParameterTable","End_DateTime") & "' (… the query continues )
Excel will issue some warnings about running the query and prompt you to edit permissions. Once permission has been granted, the function reads the text from the parameter table and passes it into the SQL.
I found that the function call was not optional. Apparently, importing the code directly into a native call to SQL Server is considered an unacceptable security risk.
Many thanks to Mr. David Browne. His post definitely points in the right direction.
You can reference a table on a sheet from Power Query and integrate values from that table into your other queries. Eg if ParameterTable is a single-row table on some worksheet with a column called "StartDate", something like
let
theDate = Date.From( Record.Field(Table.First(ParameterTable),"StartDate") ),
Source = Sql.Databases("localhost"),
AdventureWorksDW2017 = Source{[Name="AdventureWorksDW2017"]}[Data],
dbo_DimDate = AdventureWorksDW2017{[Schema="dbo",Item="DimDate"]}[Data],
#"Filtered Rows" = Table.SelectRows(dbo_DimDate, each [FullDateAlternateKey] = theDate )
in
#"Filtered Rows"
for M query folding, or
let
theDate = Date.From( Record.Field(Table.First(ParameterTable),"StartDate") ),
sql = "
select *
from dimDate
where FullDateAlternateKey = '" & Text.From(theDate) & "'
",
Source = Sql.Database("localhost", "adventureworksdw2017", [Query=sql])
in
Source
for dynamic SQL.
I am using Power query & I would like to assign a comma delimited string from an excel cell to a sql server variable. The power query I have so far is below. My Parameter is in a Excel "Table3" in "Column2" :
let
ProdParameterSource =Excel.CurrentWorkbook(){[Name="Table3"]}[Content]{0}[Column2],
Source = Sql.Database("server", "database", [Query="Declare #Product varchar(max) = "&(ProdParameterSource)&" select #Product"])
in
Source
I see the below error :
The parameters I am using as seen in the excel sheet are below :
How can I fix this error and see my input parameters in the sql server variable ABC as '44,216' (with the inverted comma).
I do not do a lot of power query but I do work in Power BI Desktop version.
First, you need to look at the reference to the SQL.Database() on MSDN.
It is looking for a whole query, either a dynamic string you make up or a call to a stored procedure.
My simple example below, pulls data from the Adventure Works DW 2012 Customer table.
What you are missing is a TSQL statement or multiple TSQL statements. Use the semicolon to combine statements into one batch to be called.
The example below create a variable #x as an integer and returns the value. This is almost like you example above.
You will be prompted for security (trusted credentials) and it will tell you that it is an unencrypted connection.
I did some looking around the NET. This article from Reeves Smith is like my last example but slanted towards power query and excel. Just remember, both products use the same engine.
https://reevessmith.wordpress.com/2014/08/19/power-query-and-stored-procedures-with-parameters/
Again
This will fix it: [Query="Declare #Product varchar(max) = '" & ProdParameterSource & "' select #Product"]).
This will fix it in a safer way, since you will escape any extra single-quotes which could break out of the string value: [Query="Declare #Product varchar(max) = '" & Text.Replace(ProdParameterSource, "'", "''") &"' select #Product"]).
What happened is that Power Query treats the text passed to Query as the entire script, and when you built the script you didn't put the value of ProdParameterSource in quotes, so the script appears to set a varchar value to 44,216 (without quotes). This is an invalid statement.
I have managed to link a Sybase server to SSMS and am able to query it using openquery but I get an error every time I try to concatenate columns. ie:
Select * from openquery(SybaseServer, 'select top 1000 first_name + middle_name + Last_name as "full_name", * from dbname..tablename')
I realize this won't put a space between the names which is fine as I'm just using that as an example.
My thought is either it's something weird with the Sybase syntax or it's the structure of the open query. I know that within the open query I need to use " " instead of ' ' so is there a trick to getting concatenate to work? Sorry if this is basic but googled for about 30 mins and nothing directly answered my question. I don't think it's the Sybase syntax from what I have read so gotta be the openquery structure.
realized the columns needed to be converted to same data type... added conversions and good ole fashioned + worked.
I am using Lucid Works to create index of solr.
Source : Database
I have two similar columns in my database, Party1, Party2.
I am using a SQL Statement like this;
SELECT top 1000000 OrderId as id
, Party1 as PartyNameFirst
, Party2 as PartyNameLast
FROM dbo.vw_SolrRPSTRD
I wanted to get both Party as a single fieled, seperated by ",". I know it is deals with multivalued field and splitby function.
But unable to get sample.
Please guide me in this.
Thanks In Advance....
Looks like you are using SQL Server. You can concatenate them in SQL itself and keep a single-valued Solr field:
SELECT top 1000000 OrderId as id,
Party1 + ', ' + Party2 as PartyName
FROM dbo.vw_SolrRPSTRD
In a project using a MSSQL 2005 Database we are required to log all data manipulating actions in a logging table. One field in that table is supposed to contain the row before it was changed. We have a lot of tables so I was trying to write a stored procedure that would gather up all the fields in one row of a table that was given to it, concatenate them somehow and then write a new log entry with that information.
I already tried using FOR XML PATH and it worked, but the client doesn't like the XML notation, they want a csv field.
Here's what I had with FOR XML PATH:
DECLARE #foo varchar(max);
SET #foo = (SELECT * FROM table WHERE id = 5775 FOR XML PATH(''));
The values for "table", "id" and the actual id (here: 5775) would later be passed in via the call to the stored procedure.
Is there any way to do this without getting XML notation and without knowing in advance which fields are going to be returned by the SELECT statement?
We used XML path and as you've discovered, it works very well. Since one of SQL's features is to store XML properly, the CSV makes no sense.
What you could try is a stored proc that reads out the XML in CSV format (fake it). I would. Since you won't likely be reading the data that much compared to saving it, the overhead is negligible.
How about:
Set #Foo = Stuff(
( Select ',' + MyCol1 + ',' + MyCol2 ...
From Table
Where Id = 5775
For Xml Path('')
), 1, 1, '')
This will produce a CSV line (presuming the inner SQL returns a single row). Now, this solves the second part of your question. As to the first part of "without knowing in advance which fields", there is no means to do this without using dynamic SQL. I.e., you have to build the SQL statement as a string on the fly. If you are going to do that you might as well build the entire CSV result on the fly.