I am trying to add a table into my power bi using the SQL Server data source. The thing is that a new table is added daily with the format of the year YYMMDD. Example: MYTABLE191121, tomorrow it will be MYTABLE191122.
How can i write a query in Power BI that always looks at the latest table based on today's date? I want to be able to refresh the content and have latest table's data.
Thank you
If your database name is TEST_DB on localhost your query can be like this:
let
Source = Sql.Databases("localhost"),
TEST_DB = Source{[Name="TEST_DB"]}[Data],
// chain a few functions to create today's date in YYMMDD
TODAYS_DATE = Text.Range(Text.Remove(Date.ToText(DateTime.Date(DateTime.LocalNow())),"-"),2),
// simply use & to concat MYTABLE and today's date
todays_table = TEST_DB{[Schema="dbo",Item="MYTABLE" & TODAYS_DATE]}[Data]
in
todays_table
This will create a query like this: SELECT * FROM [dbo].[MYTABLE191122] into the table todays_table.
Related
I want to set local variables or pass parameters from Excel to SQL. I've found similar questions, but all referred to old versions of Excel and/or the answers showed how to filter or manipulate output from a generic SQL query in the Power Query Editor, rather than pass a parameter or modify the SQL, so that the SQL Server supplies data in the needed form.
I'm building a large Microsoft Excel spreadsheet that depends on ten different SQL queries, all against a common SQL Server database. Excel and SQL Server are installed on my laptop and are current versions (as of 16 Mar 2022). All ten queries share a common date restriction, imposed in the WHERE clauses of the queries. The tables accessed and the form of output are very different, so there is no easy way to combine the ten queries into a single query. The queries contain multiple levels of aggregation (e.g. SUM(...)) so I need to restrict the records accessed prior to aggregation and passing results from the query back to Excel.
As currently written, each query begins by setting two date values in local variables. For example,
DECLARE #BEGIN_DATE AS smalldatetime;
DECLARE #END_DATE AS smalldatetime;
#BEGIN_DATE = CAST('2021-03-01 00:00' AS smalldatetime);
#END_DATE = CAST('2021-03-02 23:59' AS smalldatetime);
Every one of the ten queries includes a line in the WHERE clause similar to
WHERE
PickUpDate BETWEEN #BEGIN_DATE AND #END_DATE
Every query will use the same pair of dates. However, the column filtered (PickUpDate above) changes from one query to the next.
As it is, I have to manually edit each of the ten queries to change the two dates--twenty edits in all. This is time-consuming and error-prone. Ideally, I'd like to set the date range in the spreadsheet, in a pop-up dialog box, or any other convenient way and pass the dates to the SQL queries. Then by selecting Data > Refresh All in Excel, update all my tables at once.
Is this possible, and if so, how does one do it?
The answer from David Browne is generally on-target. But I found some difficulties reading data from an Excel table directly into the SQL, given security restrictions in the latest version of Excel/Power Query. Also, since this was the first time I worked directly in M-code and the advanced editor, it was challenging to fill-in the gaps.
I finally got a nice solution running; here is what worked for me.
First, I stored the parameter values in a two-column table. My table is named "ParameterTable" with column headers named "Parameter_Name" and "Value". The value(s) to pass to SQL Server are stored in the Value column. My table has two rows with row entries labeled "Begin_DateTime" and "End_DateTime".
Secondly I created a callable function named “ftnGetParameter.” Select Data > Get Data > From Other Sources > Blank Query. Then select “Advanced Editor.” Delete any boilerplate added by Excel, and enter and save this function
let theParameter=(TableName,ParameterLabel) =>
let
Source=Excel.CurrentWorkbook(){[Name=TableName]}[Content],
value = Source{[Parameter_Name=ParameterLabel]}[Value]
in
value
in
theParameter
Thirdly, code-up your SQL statement as usual. I was trying to pass dates to SQL, so I initially coded with string literals. Enter the query in the usual way. I used Data > Get Data > From Database > From SQL Server Database. Then pasted in the SQL. The two relevant lines in my query looked like this:
DECLARE #BEGIN_DATE AS SMALLDATETIME='2021-01-01 00:00';
DECLARE #END_DATE AS SMALLDATETIME='2021-12-31 23:59';
You could skip this step, but it allowed me to get complex SQL code entered, formatted, and running before I invoked the function to pass the parameters.
Finally, simply replace the string literals in the SQL with code to call the function. My first few lines of M-code looks like this:
let
Source = Sql.Database("DESKTOP-04P8E8C", "nfbdata",
[Query=
"
DECLARE #BEGIN_DATE AS SMALLDATETIME= '" & ftnGetParameter("ParameterTable","Begin_DateTime") & "';
DECLARE #END_DATE AS SMALLDATETIME='" & ftnGetParameter("ParameterTable","End_DateTime") & "' (… the query continues )
Excel will issue some warnings about running the query and prompt you to edit permissions. Once permission has been granted, the function reads the text from the parameter table and passes it into the SQL.
I found that the function call was not optional. Apparently, importing the code directly into a native call to SQL Server is considered an unacceptable security risk.
Many thanks to Mr. David Browne. His post definitely points in the right direction.
You can reference a table on a sheet from Power Query and integrate values from that table into your other queries. Eg if ParameterTable is a single-row table on some worksheet with a column called "StartDate", something like
let
theDate = Date.From( Record.Field(Table.First(ParameterTable),"StartDate") ),
Source = Sql.Databases("localhost"),
AdventureWorksDW2017 = Source{[Name="AdventureWorksDW2017"]}[Data],
dbo_DimDate = AdventureWorksDW2017{[Schema="dbo",Item="DimDate"]}[Data],
#"Filtered Rows" = Table.SelectRows(dbo_DimDate, each [FullDateAlternateKey] = theDate )
in
#"Filtered Rows"
for M query folding, or
let
theDate = Date.From( Record.Field(Table.First(ParameterTable),"StartDate") ),
sql = "
select *
from dimDate
where FullDateAlternateKey = '" & Text.From(theDate) & "'
",
Source = Sql.Database("localhost", "adventureworksdw2017", [Query=sql])
in
Source
for dynamic SQL.
I have a table with some data like subtotal,tax,Grand,Date
I want to take summary report from this data using Sql query.
when I use query its getting all the selected date's data (example: if start date 31-07-20, end date 01-08-20) then result showing each days details. My goal is to get summary of each days.
What I am getting:
What I am looking for:
SELECT SUM(Sub) as Sub, SUM(VAT) as VAT, SUM(Grand) as Grand, Date
FROM <WHATEVER>
GROUP BY Date
This is SQL 101...
My question is a mess, but what I mean is basically this: let's say I have a table called EXECUTELATER (in my SQL Server database for my app) and I have a column called DATE. I want to execute queries every time a row/rows in that table has DATE <= current system time, do something with the data, put some data in another table and delete this row from the table EXECUTELATER.
One thing in my mind is to create another simple app logged with another user to query every millisecond and check if there is a DATE matching, but that sounds absurd.
Is there a way I can do this within the web app? Can anyone suggest me a smart way I can make this work? I don't mind if I have to work with full SQL date format or store the date in long type, work with the web app, create a new application for that use, use something in SQL Server itself or whatever.
Again sorry for the badly structured question, I'm not sure how to put this together. Thank you for checking!
you can use triggers like :
create TRIGGER [dbo].[TrgX]
ON [dbo].[EXECUTELATER]
AFTER UPDATE,insert
AS
BEGIN
SET NOCOUNT ON;
update [dbo].[tblx]
set [columnx] = x
from inserted i
inner join [dbo].[tblx] p on i.[Relatedcoulumn] = p.Relatedcoulumn
where i.[Date] >= getdate()
END
I have a table in a Microsoft Azure SQLDW that has a date field, as well as other columns. There is a row for each day in the past 10 years of business days. Currently the table has not been stored in such a way like "order by date" etc. Below is the code I have so far:
import pyodbc driver = '{ODBC Driver 13 for SQL Server}'
conn = pyodbc.connect('DRIVER='+driver+';
PORT=1433;SERVER='+server+';
DATABASE='+database+';
UID='+username+';
PWD='+ password)
cursor = conn.cursor() cursor.execute("SELECT * FROM index_att") i = 0 for row in cursor: i += 1 print(i)
If I'm using python to loop through every row in this table and I want to go in chronological order (from oldest date to current date) does the table need to be stored in a way that it is ordered by date? And do I need to add an additional incremental ID that starts at 1 with the oldest date and goes up each day?
Or is there a way to do this loop through dates with the data not sorted?
I would also ask you to review your process because doing row level operations on a Azure SQL Data Warehouse can cause large amounts of data movement.
Currently the table has not been stored in such a way like "order by date" etc.
How the data is stored is irrelevant. If you want ordered data, the client just needs to request it with an ORDER BY clause on a SELECT query.
Add an order by to your select statement:
cursor = conn.cursor()
cursor.execute("SELECT * FROM index_att ORDER BY [MyDate]")
i = 0
for row in cursor:
i += 1 print(i)
I need some help. I am suppose to get XML data from column of a table. The XML data has many tags. One of the tag contains values like
<Code date= 30/03/2013>
<apr id =1> -2 </apr>
<rdv id =2> 1 </rdv>
</code>
I need to run a particular task which checks the date. Like if today's date= (Code date -2) or
today's date = (code date + 1)
run a mailer task.
How should I go about it ? Please forgive me for the XML data format. I am naive in XML.
Create 2 variables to store the XML column Date value and other one to get the current date .To write an expression ,select the variable ,right click on it .In the properties windows Set EvaluateAsExpression =true and write the expression
Name DataType Expression
ExecDate DateTime
CurrentDate DateTime GETDATE()
Use Execute SQL Task .Set Resultset to Single Row and write the following code to read the date value from the XML column in your table
SELECT
convert(datetime,XMLData.value('(/Code/#date)[1]', 'varchar(10)'),103) as PkgDate
from yourTable
Demo in SQL Fiddle
In the resultset tab map the output from the above sql query with the variable ExecDate
Result Name Variable Name
PkgDate ExecDate
Put another Task or component Ex SEND Mail TASK after Execute SQL Task and write an expression in the Precedence Constraint.
Expression : #ExecDate==#CurrentDate
Only when the Date value from the XML column matches with the Current Date which is stored in the variable #CurrentDate then only the other components after Execute SQL Task will execute .
today's date= (Code date -2) or today's date = (code date + 1) run a mailer task.
In order to perform the above expression ,you need to change the sql code in Execute SQL Task
To get CodeDate +1
SELECT
Dateadd(day,1,convert(datetime,XMLData.value('(/Code/#date)[1]', 'varchar(10)'),103)) as PkgDate
from yourTable
I don't have the SSIS environment to test it now but this is how you should be doing it