I am building a lookup tool that will take Column Name and Table Name as inputs from Tableau as Parameters, and plug those in to Tableau Custom SQL as Parameter values, but I cannot find a way to make the Column Name dynamic in the select query.
Here is the idea:
Select ParameterColumnName
FROM TABLE(ParameterTableName)
I have investigated Snowflake's "IDENTIFIER" commands, but those seem to work only in the FROM clause of the query. I have tested with a static column name and dynamic table name and that works fine.
I have also explored setting variables with the parameter column name values, but I cannot seem to use "SET" in Tableau Custom SQL--it does not recognize SET. Appreciate any pointers.
Update: This post appears to address why I cannot set variables from Custom SQL, but if that's not it, let me know!
Related
I have a quite strange / unusual request for an SSRS report. The client wants to be able to paste in large lists of ID numbers from an Excel sheet column (normally < 100 values but can be as many as 20,000 + values) as a search parameter in the report.
Now, I know how to pass multi value parameters from SSRS to a stored proc etc, that's not the issue. The issue is with this requirement to literally paste in a list of IDs into the multi value parameter input box and then limit the dataset based on that list (rather than pre-populate the Multi Value parameter with a list of values based on a query / SP as you normally would)
My question is what would be the best method / approach to this problem as I have never had a similar ask in many years of SSIS development? I need to make the solution as "self-service" as possible too, so as easy as running an SSRS report from report manager ideally. I know I could just import the Excel data into a table in the database and join to that etc but ideally would like something the user can run without the need for any tech input to import data or run SQL through SSMS to get the datset.
When you copy/paste from excel its just a tab delimited string. You can configure a string parameter to allow multiple values, and then in the expression editor, split the string by tab values or by line breaks.
Here's how:
In the parameter properties check the 'Allow multiple values' checkbox on the
general tab
Then in the datasource properties click the 'fx' (expression) button next to the parameter on the parameters tab
In the expression editor that appears, type the following:
=replace(split(Parameters!MyParameter.Value,vbCrLf),"\x0009","")
This will split the string by line breaks, then strip the tab characters out. After that you can treat it like any multi-valued parameter.
If you have a typical multi-value parameter setup then you can just copy/paste the column from excel directly in, it will automatically put each cell row copied from excel on a new line.
So if your query looked something like
SELECT * FROM myTable WHERE EmployeeName IN(#empName)
and empName was you report parameter, if no available values are configured you'll get an empty list when you click in the parameter field, just copy paste direct from Excel and it will work.
I'm not sure if there would be any limits or how good performance would be especially if copying thousand of values but certainly for a reasonably small number of items this will work.
The only other way I can think of that means no real extra work for the user would be to have them drop the workbook into a specified folder (maybe with a subfolder based on their SSRS username, then use openrowset to read the contents either directly into a dataset or better still, into a permanent table with their username and the parameter value on each row.
The openrowset statement could sit at the top of your main dataset query
Then your query could do something simple like
DELETE FROM myParamValueTable WHERE UserName = #UserID
INSERT INTO myParamValueTable
SELECT * FROM (OPENROWSET .....) xl
SELECT * FROM myTable t
JOIN myParamValueTable p
ON t.EmployeeName = p.EmployeeName
WHERE p.UserName = #UserID
I am trying to see if there is an easy answer for this. I have done something similar using multiple pick dropdown parameters in SSRS but this appears to be different.
My scenario is this, so maybe there is an even better answer.
I have a production server that I do not want to make any changes to including temp tables or functions. The production server has a table of clients with about 1600 records. I have set up an SSIS package that will allow transfer of data from production to dev based on a clientid. So my sources would have a query similar to Select Field From Table Where ClientId = ?
This works fine. Now I want to load more than one client, based an data in the clients table. It may be Select ClientId From Clients where Field = A and returns multiple ClientIds.
I am able to populate a comma delimited list from an execute sql task to a SSIS variable, so it maybe 1,4,8.
If I change my source query to use ClientId in (?) I get a conversion error.
I have looked at many posts that advocate a temp table or a function which I want to avoid. Select IN using varchar string with comma delimited values
I have contemplated building the entire sql statement into a variable but this don't seem like the right path as I have many tables to query and transfer where using ClientId = ? works well without having to build each individual SQL statement to a variable.
Is there an easy fix I am missing? I will turn my research now to try to find out how I did this in SSRS but I thought that I should try a post here to see if someone has accomplished this before.
I appreciate any info on this, thank you.
EDIT: Key note is that the column on clients is on the dev server, so I cannot just use a select in the where clause as the column does not exist on the production server.
EDIT: I did not mention that I am specifically looking at OLEDB sources mapping a parameter to ? in the sql statement.
EDIT: Narrowing down on this but having trouble relating SSRS and SSIS functionality. In SSRS its called a multi-value parameter in the following link the key line is
WHERE Production.ProductInventory.ProductID IN (#ProductID)
https://msdn.microsoft.com/en-us/library/dn385719(v=sql.110).aspx
This one looks good as well
https://sqlblogcasts.com/blogs/simons/archive/2007/11/22/RS-HowTo---Pass-a-multivalue-parameter-to-a-query-using-IN.aspx
I will keep researching and thank you for the help so far.
I think this sums it up best
This functionality is limited to strictly using embedded SQL.
What SSRS does is transform your SQL column IN (#value) to column IN
(#selectedvalue1,#selectedvalue2) etc.
You need to forget anything you have about the other ways of passing
lists to SQL i.e. building strings etc. and make sure you declare the
data types are correct for the value of your parameter.
You do not need to use the Join(parameters!,",") trick UNLESS
you are passing the list to a stored procedure.
In which case you then need to use some function to turn the delimited
list into a rowset as you have done.
I hope that helps
The core question is if I can get the same functionality in SSIS as in SSRS. It reminds me of macro substitution..
If you dont want to create a function, you can use the following in your t-sql statement.
Declare #ClientIds nvarchar(50) = '123,456'; --<-- Comma delimited list of Client Ids
Select Field
From Table
Where ClientId IN (
SELECT CAST(RTRIM(LTRIM(Split.a.value('.', 'VARCHAR(100)'))) AS INT) ClientIDs
FROM (
SELECT Cast ('<X>'
+ Replace(#ClientIds, ',', '</X><X>')
+ '</X>' AS XML) AS Data
) AS t CROSS APPLY Data.nodes ('/X') AS Split(a)
)
i work with text table in tableau and i want to be able to click on a cell in the table, so that the value of the cell would be transfered\sent to sql query or procedure, does anyone know if this is possible?
if not, is there a way i could define a column in a text table as a parameter with multiple values?
Thanks
Yes it is possible if you use a custom sql option. once you connect to your DB/DS select custom sql and then pass your variable through tableau parameter.
I'm working with a new REST backend talking to a SQL Server. Our REST api allows for the caller to pass in the columns/fields they want returned (?fields=id,name,phone).
The idea seems very normal. The issue I'm bumping up against is resistance to dynamically generating the SQL statement. Any arguments passed in would be passed to the database using a parameterized query, so I'm not concerned about SQL injection.
The basic idea would be to "inject" the column-names passed in, into a SQL that looks like:
SELECT <column-names>
FROM myTable
ORDER BY <column-name-to-sort-by>
LIMIT 1000
We sanitize all column names and verify their existence in the table, to prevent SQL injection issues. Most of our programmers are used to having all SQL in static files, and loading them from disk and passing them on to the database. The idea of code creating SQL makes them very nervous.
I guess I'm curious if others actually do this? If so, how do you do this? If not, how do you manage "dynamic columns and dynamic sort-by" requests passed in?
I think a lot of people do it especially when it comes to reporting features. There are actually two things one should do to stay on the safe side:
Parameterize all WHERE clause values
Use user input values to pick correct column/table names, don't use the user values in the sql statement at all
To elaborate on item #2, I would have a dictionary where Key is a possible user input and Value is a correponding column/table name. You can store this dictionary wherever you want: config file, database, hard code, etc. So when you process user input you just check a dictionary if the Key exists and if it does you use the Value to add a column name to your query. This way you just use user input to pick required column names but don't use the actual values in your sql statement. Besides, you might not want to expose all columns. With a predefined dictionary you can easily control the list of available columns for a user.
Hope it helps!
I've done similar to what Maksym suggests. In my case, keys were pulled directly from the database system tables (after scrubbing the user request a bit for syntactic hacks and permissions).
The following query takes care of some minor injection issues through the natural way SQL handles the LIKE condition. This doesn't go as far as handling permissions on each field (as some fields are forbidden based on the log-in) but it provides a very basic way to retrieve these fields dynamically.
CREATE PROC get_allowed_column_names
#input VARCHAR(MAX)
AS BEGIN
SELECT
columns.name AS allowed_column_name
FROM
syscolumns AS columns,
sysobjects AS tables
WHERE
columns.id = tables.id AND
tables.name = 'Categories' AND
#input LIKE '%' + columns.name + '%'
END
GO
-- The following only returns "Picture"
EXEC get_allowed_column_names 'Category_,Cat%,Picture'
GO
-- The following returns both "CategoryID and Picture"
EXEC get_allowed_column_names 'CategoryID, Picture'
GO
This might seem like a silly question, but I'm surprised that I didn't find a clear answer to this already:
Is it possible to use SQL Server parameters for writing a query with dynamic column names (and table names), or does the input just need to be sanitized very carefully?
The situation is that tables and their column names (and amount of columns) are generated dynamically and there is no way to know beforehand to manually write a query. Since the tables & columns aren't known I can't use an ORM, so I'm resorting to manual queries. Usually I'd use parameters to fill in values to prevent SQL injection, however I'm pretty sure that this cannot be done the same way when specifying the table name and/or column names. I want to create generic queries for insert, update, upsert, and select, but I obviously don't want to open myself up to potential injection. Is there a best practices on how to accomplish this safely?
Just as an FYI - I did see this answer, but since there's no way for me to know the column / table names beforehand a case statement probably won't work for this situation.
Environment: SQL Server 2014 via ADO.NET (.NET 4.5 / C#)
There is no mechanism for passing table or column references to procedures. You just pass them as strings and then use dynamic SQL to build your queries. You do have to take precautions to ensure that your string parameters are valid.
One way to do this would be to validate that all table and column reference strings have valid names in sys.tables and sys.columns before building your T-SQL queries. Then you can be sure that they can be used safely.
You can also use literal parameters with dynamic sql when using the sp_executesql procedure. You can't use it to validate your table and column names, but it validates and prevents SQL injection with your other parameters.