I need to pass an array to a prepared statement defined on AWS Redshift to filter my query. Since this is not supported in Redshift, I was trying to find a work around using a Python UDF. Something like:
-- a function to split a comma separated list of values (string is already validated)
CREATE FUNCTION split_str_to_ints (string char) RETURNS int[] IMMUTABLE as $$
def split_stoi(string):
ints = [int(item) for item in string.split(',')]
return ints
$$ LANGUAGE plpythonu;
-- a prepared statement to return orders filtered by company id
PREPARE get_orders (char) as
SELECT order.id
order.company_id,
COUNT(order.id) AS order_count
FROM order
INNER JOIN company ON (order.company_id = company.company_id)
WHERE company.company_id IN (split_str_to_ints($1))
GROUP BY order.id, order.company_id
ORDER BY order_count DESC;
EXECUTE get_orders('1,2,3,4')
But when I want to define the function I get an error saying that array of integers is not a supported plpythonu UDF return type.
Is there any way other way to pass list of integers (or chars) to a prepared statement on Redshift?
Related
I'm trying to make a SELECT on multiple joined VARIANT columned tables. The main record is returned as DATA and all supporting information around it are made up of supporting joined tables returned as INCLUDED.
I'm using an ARRAY_CONSTRUCT_COMPACT on the supporting VARIANT records AND ARRAY_AGG(DISTINCT [record]) to aggregate them and de-duplicate.
The issue is that ARRAY_AGG produces multiple records on one of my joined tables (ENTITIES). When they are constructed using ARRAY_CONSTRUCT_COMPACT() the result is an array of arrays which I need flattened into a single array of objects.
I've tried using combinations of ARRAY_COMPACT and ARRAY_CAT with nested ARRAY_AGGS, caveat being that ARRAY_CAT only accepts 2 arguments. The below code has got me the closest to what I need but I can't seem to figure out how to FLATTEN that final array as INCLUDED.
SELECT
a1.appointment data,
ARRAY_CONSTRUCT_COMPACT(
ARRAY_AGG(DISTINCT c1.call),
ARRAY_AGG(DISTINCT e1.entity),
ARRAY_AGG(DISTINCT a2.address)
) included
FROM APPOINTMENTS a1
INNER JOIN CALLS c1 ON c1.call:id = a1.appointment:callId
INNER JOIN ENTITIES e1 ON e1.entity:id IN (
a1.appointment:relationships.agent,
a1.appointment:relationships.consultant,
a1.appointment:relationships.contact
)
INNER JOIN ADDRESSES a2 ON a2.address:id = a1.appointment:relationships:office
WHERE a1.appointment:id = 'some_appointment_id'
GROUP BY a1.appointment;
The output of my INCLUDED column is currently:
[[{}], [{},{}], [{}]]
Which I need flattened to:
[{}, {}, {}, {}]
Any help would be greatly appreciated!
You can use javascript in functions that will make this nice and tidy:
CREATE OR REPLACE FUNCTION ARRAY_CAT_ALL( source Array )
RETURNS ARRAY
LANGUAGE JAVASCRIPT
AS
$$
return [].concat.apply([], SOURCE);
$$;
Then you can wrap your ARRAY_CONSTRUCT_COMPACT result with this function which will flatten the array of arrays into a single array.
A further enhancement to Bradley's answer to handle duplicates in your array, which was my specific use case when I found this!
CREATE OR REPLACE FUNCTION ARRAY_CAT_ALL( source Array )
RETURNS ARRAY
LANGUAGE JAVASCRIPT
AS
$$
const temp_array = [].concat.apply([], SOURCE)
const unique = new Set(temp_array);
return [...unique];
$$;
I have seen a similar post here but my situation is slightly different from anything I've found so far. I am trying to call a postgres function with parameters that I can leverage in the function logic as they pertain to the jsonb query. Here is an example of the query I'm trying to recreate with parameters.
SELECT *
from edit_data
where ( "json_field"#>'{Attributes}' )::jsonb #>
'{"issue_description":"**my description**",
"reporter_email":"**user#generic.com**"}'::jsonb
I can run this query just fine in PGAdmin but all my attempts thus far to run this inside a function with parameters for "my description" and "user#generic.com" values have failed. Here is a simple example of the function I'm trying to create:
CREATE OR REPLACE FUNCTION get_Features(
p1 character varying,
p2 character varying)
RETURNS SETOF edit_metadata AS
$BODY$
SELECT * from edit_metadata where ("geo_json"#>'{Attributes}' )::jsonb #> '{"issue_description":**$p1**, "reporter_email":**$p2**}'::jsonb;
$BODY$
LANGUAGE sql VOLATILE
COST 100
ROWS 1000;
I know that the syntax is incorrect and I've been struggling with this for a day or two. Can anyone help me understand how to best deal with these double quotes around the value and leverage a parameter here?
TIA
You could use function json_build_object:
select json_build_object(
'issue_description', '**my description**',
'reporter_email', '**user#generic.com**');
And you get:
json_build_object
-----------------------------------------------------------------------------------------
{"issue_description" : "**my description**", "reporter_email" : "**user#generic.com**"}
(1 row)
That way there's no way you will input invalid syntax (no hassle with quoting strings) and you can swap the values with parameters.
My last question Passing an array to stored to postgres was a bit unclear. Now, to clarify my objective:
I want to create an Postgres stored procedure which will accept two input parameters. One will be a list of some amounts like for instance (100, 40.5, 76) and the other one will be list of some invoices ('01-2222-05','01-3333-04','01-4444-08'). After that I want to use these two lists of numbers and characters and do something with them. For example I want to take each amount from this array of numbers and assign it to corresponding invoice.
Something like that in Oracle would look like this:
SOME_PACKAGE.SOME_PROCEDURE (
789,
SYSDATE,
SIMPLEARRAYTYPE ('01-2222-05','01-3333-04','01-4444-08'),
NUMBER_TABLE (100,40.5,76),
'EUR',
1,
P_CODE,
P_MESSAGE);
Of course, the two types SIMPLEARRAYTYPE and NUMBER_TABLE are defined earlier in DB.
You will love this new feature of Postgres 9.4:
unnest(anyarray, anyarray [, ...])
unnest() with the much anticipated (at least by me) capability to unnest multiple arrays in parallel cleanly. The manual:
expand multiple arrays (possibly of different types) to a set of rows. This is only allowed in the FROM clause;
It's a special implementation of the new ROWS FROM feature.
Your function can now just be:
CREATE OR REPLACE FUNCTION multi_unnest(_some_id int
, _amounts numeric[]
, _invoices text[])
RETURNS TABLE (some_id int, amount numeric, invoice text) AS
$func$
SELECT _some_id, u.* FROM unnest(_amounts, _invoices) u;
$func$ LANGUAGE sql;
Call:
SELECT * FROM multi_unnest(123, '{100, 40.5, 76}'::numeric[]
, '{01-2222-05,01-3333-04,01-4444-08}'::text[]);
Of course, the simple form can be replaced with plain SQL (no additional function):
SELECT 123 AS some_id, *
FROM unnest('{100, 40.5, 76}'::numeric[]
, '{01-2222-05,01-3333-04,01-4444-08}'::text[]) AS u(amount, invoice);
In earlier versions (Postgres 9.3-), you can use the less elegant and less safe form:
SELECT 123 AS some_id
, unnest('{100, 40.5, 76}'::numeric[]) AS amount
, unnest('{01-2222-05,01-3333-04,01-4444-08}'::text[]) AS invoice;
Caveats of the old shorthand form: besides being non-standard to have set-returning function in the SELECT list, the number of rows returned would be the lowest common multiple of each arrays number of elements (with surprising results for unequal numbers). Details in these related answers:
Parallel unnest() and sort order in PostgreSQL
Is there something like a zip() function in PostgreSQL that combines two arrays?
This behavior has finally been sanitized with Postgres 10. Multiple set-returning functions in the SELECT list produce rows in "lock-step" now. See:
What is the expected behaviour for multiple set-returning functions in SELECT clause?
Arrays are declared by adding [] to the base datatype. You declare them as a parameter the same way you declare regular parameters:
The following function accepts an array of integers and and array of strings and will return some dummy text:
create function array_demo(p_data integer[], p_invoices text[])
returns text
as
$$
select p_data[1] || ' => ' || p_invoices[1];
$$
language sql;
select array_demo(array[1,2,3], array['one', 'two', 'three']);
SQLFiddle demo: http://sqlfiddle.com/#!15/fdb8d/1
I have a database table in SQL Server 2014 with only an ID column (int) and a column xmldata of type XML.
This xmldata column contains for example:
<book>
<title>a nice Novel</title>
<author>Maria</author>
<author>Peter</author>
</book>
As expected, I have multiple books, therefore multiple rows with xmldata.
I now want to execute a query for all books, where Peter is an Author. I tried this in some xPath2.0 testers and got to the conclusion that:
/book/author/concat(text(), if(position() != last())then ',' else '')
works.
If you try to port this success into SQL Server 2014 Express it looks like this, which is correctly escaped syntax etc.:
SELECT id
FROM books
WHERE 'Peter' IN (xmldata.query('/book/author/concat(text(), if(position() != last())then '','' else '''')'))
SQL Server however does not seem to support a construction like /concat(...) because of:
The XQuery syntax '/function()' is not supported.
I am at a loss then however, why /text() would work in:
SELECT id, xmldata.query('/book/author/text()')
FROM books
which it does.
My constraints:
I am bound to use SQL Server
I am bound to xpath or something else that can be "injected" as the statement above (if the structure of the xml or the database changes, the xpath above could be changed isolated and the application logic above that constructs the Where clause will not be touched) SEE EDIT
Is there a way to make this work?
regards,
BillDoor
EDIT:
My second constraint boils down to this:
An Application constructs the Where clause by
expression <operator> value(s)
expression is stored in a database and is mapped by the xmlTag eg.:
| tokenname| querystring
| "author" | "xmldata.query(/book/author/text())"
the values are presented by the Requesting user. so if the user asks for the author "Peter" with operator "EQUALS" the application constructs:
xmaldata.query(/book/author/text()) = "Peter"
as where clause.
If the customer now decides that author needs to be nested in an <authors> element, i can simply change the expression in the construction-database and the whole machine keeps running without any changes to code, simply manageable.
So i need a way to achieve that
<xPath> <operator> "Peter"
or any other combination of this three isolated components (see above: "Peter" IN <xPath>...) gets me all of Peters' books, even if there are multiple unsorted authors.
This would not suffice either (its not sqlserver syntax, but you get the idea):
WHERE xmldata.exist('/dossier/client[text() = "$1"]', "Peter") = 1;
because the operator is still nested in the expression, i could not request <> "Peter".
I know this is strange, please don't question the concept as a whole - it has a history :/
EDIT: further clarification:
The filter-rules come into the app in an XML structure basically:
Operator: "EQ"
field: "name"
value "Peter"
evaluates to:
expression = lookupExpressionForField("name") --> "table2.xmldata.value('book/author/name[1]', 'varchar')"
operator = lookUpOperatorMapping("EQ") --> "="
value = FormatValues("Peter") --> "Peter" (if multiple values are passed FormatValues cosntructs a comma seperated list)
the application then builds:
- constructClause(String expression,String operator,String value)
"table2.xmldata.value('book/author/name[1]', 'varchar')" + "=" + "Peter"
then constructs a Select statement with the result as WHERE clause.
it does not build it like this, unescaped, unfiltered for injection etc, but this is the basic idea.
i can influence how the input is Transalted, meaning I can implement the methods:
lookupExpressionForField(String field)
lookUpOperatorMapping(String operator)
Formatvalues(List<String> values) | Formatvalues(String value)
constructClause(String expression,String operator,String value)
however i choose to do, i can change the parameter types, I can freely implement them. The less the better of course. So simply constructing a comma-seperated list with xPath would be optimal (like if i could somewhere just tick "enable /function()-syntax in xPath" in sqlserver and the /concat(if...) would work)
How about something like this:
SET NOCOUNT ON;
DECLARE #Books TABLE (ID INT NOT NULL IDENTITY(1, 1) PRIMARY KEY, BookInfo XML);
INSERT INTO #Books (BookInfo)
VALUES (N'<book>
<title>a nice Novel</title>
<author>Maria</author>
<author>Peter</author>
</book>');
INSERT INTO #Books (BookInfo)
VALUES (N'<book>
<title>another one</title>
<author>Bob</author>
</book>');
SELECT *
FROM #Books bk
WHERE bk.BookInfo.exist('/book/author[text() = "Peter"]') = 1;
This returns only the first "book" entry. From there you can extract any portion of the XML field using the "value" function.
The "exist" function returns a boolean / BIT. This will scan through all "author" nodes within "book", so there is no need to concat into a comma-separated list only for use in an IN list, which wouldn't work anyway ;-).
For more info on the "value" and "exist" functions, as well as the other functions for use with XML data, please see:
xml Data Type Methods
I understand that in Postgres pure, you can pass an integer array into a function but that this isn't supported in the .NET data provider Npgsql.
I currently have a DbCommand into which I load a call to a stored proc, add in a parameter and execute scalar to get back an Id to populate an object with.
This now needs to take n integers as arguments. These are used to create child records linking the newly created record by it's id to the integer arguments.
Ideally I'd rather not have to make multiple ExecuteNonQuery calls on my DbCommand for each of the integers, so I'm about to build a csv string as a parameter that will be split on the database side.
I normally live in LINQ 2 SQL savouring the Db abstraction, working on this project with manual data access it's all just getting a bit dirty, how do people usually go about passing these kinds of parameters into postgres?
See: http://www.postgresql.org/docs/9.1/static/arrays.html
If your non-native driver still does not allow you to pass arrays, then you can:
pass a string representation of an array (which your stored procedure can then parse into an array -- see string_to_array)
CREATE FUNCTION my_method(TEXT) RETURNS VOID AS $$
DECLARE
ids INT[];
BEGIN
ids = string_to_array($1,',');
...
END $$ LANGUAGE plpgsql;
then
SELECT my_method(:1)
with :1 = '1,2,3,4'
rely on Postgres itself to cast from a string to an array
CREATE FUNCTION my_method(INT[]) RETURNS VOID AS $$
...
END $$ LANGUAGE plpgsql;
then
SELECT my_method('{1,2,3,4}')
choose not to use bind variables and issue an explicit command string with all parameters spelled out instead (make sure to validate or escape all parameters coming from outside to avoid SQL injection attacks.)
CREATE FUNCTION my_method(INT[]) RETURNS VOID AS $$
...
END $$ LANGUAGE plpgsql;
then
SELECT my_method(ARRAY [1,2,3,4])
I realize this is an old question, but it took me several hours to find a good solution and thought I'd pass on what I learned here and save someone else the trouble. Try, for example,
SELECT * FROM some_table WHERE id_column = ANY(#id_list)
where #id_list is bound to an int[] parameter by way of
command.Parameters.Add("#id_list", NpgsqlDbType.Array|NpgsqlDbType.Integer).Value = my_id_list;
where command is a NpgsqlCommand (using C# and Npgsql in Visual Studio).
You can always use a properly formatted string. The trick is the formatting.
command.Parameters.Add("#array_parameter", string.Format("{{{0}}}", string.Join(",", array));
Note that if your array is an array of strings, then you'll need to use array.Select(value => string.Format("\"{0}\", value)) or the equivalent. I use this style for an array of an enumerated type in PostgreSQL, because there's no automatic conversion from the array.
In my case, my enumerated type has some values like 'value1', 'value2', 'value3', and my C# enumeration has matching values. In my case, the final SQL query ends up looking something like (E'{"value1","value2"}'), and this works.
Full Coding Structure
postgresql function
CREATE OR REPLACE FUNCTION admin.usp_itemdisplayid_byitemhead_select(
item_head_list int[])
RETURNS TABLE(item_display_id integer)
LANGUAGE 'sql'
COST 100
VOLATILE
ROWS 1000
AS $BODY$
SELECT vii.item_display_id from admin.view_item_information as vii
where vii.item_head_id = ANY(item_head_list);
$BODY$;
Model
public class CampaignCreator
{
public int item_display_id { get; set; }
public List<int> pitem_head_id { get; set; }
}
.NET CORE function
DynamicParameters _parameter = new DynamicParameters();
_parameter.Add("#item_head_list",obj.pitem_head_id);
string sql = "select * from admin.usp_itemdisplayid_byitemhead_select(#item_head_list)";
response.data = await _connection.QueryAsync<CampaignCreator>(sql, _parameter);