I working in PostgreSQL past years, but i had no idea about array concept and how to handle array in PostgreSQL. i need a dynamic query for selecting columns in multiple table and the result will be in cursor, columns names should be dynamically will change.
for e.g (in a multiple table totally 30 columns is there If user need col1, col5,col6,col25), so select statement query will be dynamically will change like:
select col1, col5,col6,col25 from table ....
another user need col2,col5,col7,col29,col26, select statement will change dynamically as
select col2,col5,col7,col29,col26 from table .... and so on.
stored procedure passing parameters will be array
create or replace function func_my_method(check_in character varying, sel character varying[])...
this sel[] contains like
sel[0]:='col1_name'
sel[1]:='col5_name'
sel[2]:='col6_name'
sel[3]:='col25_name'
so first we have to split the array values in separate variable and these variable will be assumed in select statement will be
'select'||col1, col5,col6,col25||'from......'
Finally want to say briefly I need to pass an array in parameters and have to separate a array values and it will assign to separate variables. these variable will use to select a statement in dynamic manner
A bare refcursor can contain any number of columns. Although you'll need a special statement to read from it: FETCH ...
CREATE OR REPLACE FUNCTION func_my_method(check_in text, sel text[], ref refcursor)
RETURNS refcursor
LANGUAGE plpgsql
AS $func$
BEGIN
OPEN ref FOR EXECUTE 'SELECT ' || (SELECT string_agg(quote_ident(c), ', ')
FROM unnest(sel) c) || ' FROM ...';
RETURN ref;
END;
$func$;
SELECT func_my_method('check_in', ARRAY['col1', 'col2'], 'sample_name');
FETCH ALL IN sample_name;
http://rextester.com/ZCZT84224
Note: You could omit the refcursor parameter & DECLARE one in your function body. This way PostgreSQL will generate a (non-conflicting) name for the refcursor, which will be returned when calling SELECT func_my_method(...). You'll need that name in the FETCH ... statement.
Update: If you want to fully qualify (some) columns (i.e. write table name & column too), you'll need either:
CREATE OR REPLACE FUNCTION func_my_method2(check_in text, sel text[], ref refcursor)
RETURNS refcursor
LANGUAGE plpgsql
AS $func$
BEGIN
OPEN ref FOR EXECUTE 'SELECT ' || (SELECT string_agg((SELECT string_agg(quote_ident(c), '.')
FROM unnest(string_to_array(fq, '.')) c), ', ')
FROM unnest(sel) fq) || ' FROM ...';
RETURN ref;
END;
$func$;
SELECT func_my_method2('check_in', ARRAY['col1', 'check_in.col2'], 'sample_name2');
FETCH ALL IN sample_name2;
(this will split the sel parameter into "parts" of the fully qualified name on . -- but have a disadvantage: the table & column names cannot contain .)
Or:
CREATE OR REPLACE FUNCTION func_my_method3(check_in text, sel text[][], ref refcursor)
RETURNS refcursor
LANGUAGE plpgsql
AS $func$
BEGIN
OPEN ref FOR EXECUTE 'SELECT ' || (SELECT string_agg((SELECT string_agg(quote_ident(sel[i][j]), '.')
FROM generate_subscripts(sel, 2) j), ', ')
FROM generate_subscripts(sel, 1) i) || ' FROM ...';
RETURN ref;
END;
$func$;
SELECT func_my_method3('check_in', ARRAY[ARRAY['check_in', 'col1'], ARRAY['check_in', 'col2']], 'sample_name3');
FETCH ALL IN sample_name3;
(but this has an uncomfortable consequence: since arrays need to be rectangular, all column sub-arrays need to be the same exact dimensions; so you'll need to provide table name for all of the columns or to neither of them.)
http://rextester.com/JNI24740
Related
I have input like below
I want like below
I was trying with
Sales External?HR?Purchase Department
I did LISTAGG because finally i want in separate columns
Query Output would be like below,
meaning it should search for first occurrence of the separator (in this case "?", it can be anything but not common ones like "-" or "/" as the separator needs to be separate than sting value) and then extract the phrase before the first separator and create one column and put the value. Then it should look for second occurrence of the separator and then extract the word and keep creating columns, there can be multiple separators.
I tried with SPLIT_PART but it does not maintain the sequence in real data scenario and data does not come correct as per sequence.
I also tried with REGEXP_INSTR, but unable to use special characters as separators.
Any thought?
Regex Extract should work for you:
SELECT
REGEXP_SUBSTR_ALL("Sales External?HR?Purchase Department", "(.*)\?")
You can use LATERAL FLATTEN to convert your array into rows:
WITH MY_CTE AS (
SELECT
REGEXP_SUBSTR_ALL("Sales External?HR?Purchase Department", "(.*)\?")
)
SELECT
*
FROM
LATERAL FLATTEN(INPUT => MY_CTE, MODE=> 'ARRAY')
Deeper dive into some more cases: https://dwgeek.com/snowflake-convert-array-to-rows-methods-and-examples.html/
Here's a simplified version of the data. It uses a CTE with array_agg to group the rows. It then changes from arrays to columns. To add more columns, you can use max(), min(), or any_value() functions to get them through the aggregation. (Note that use of any_value() will not allow use of cached results from the result set cache since it's flagged as nondeterministic.)
create or replace table T1 (EMPID int, ROLE string, ACCESS string, ACCESS_LVL string, ITERATION string);
insert into T1(EMPID, ROLE, ACCESS, ACCESS_LVL, ITERATION) values
(1234, 'Sales Rep', 'Specific', 'REGION', 'DEV'),
(1234, 'Purchase Rep', 'Specific', 'EVERY', 'PROD'),
(1234, 'HR', NULL, 'Dept', 'PROD'),
(4321, 'HR', 'Foo', 'Foo', 'Foo')
;
with X as
(
select EMPID
,array_agg(nvl(ROLE,'')) within group (order by ROLE) ARR_ROLE
,array_agg(nvl(ACCESS,'')) within group (order by ROLE) ARR_ACCESS
,array_agg(nvl(ACCESS_LVL,'')) within group (order by ROLE) ARR_ACCESS_LVL
,array_agg(nvl(ITERATION,'')) within group (order by ROLE) ARR_ITERATION
from T1
group by EMPID
)
select EMPID
,ARR_ROLE[0]::string as ROLE1
,ARR_ROLE[1]::string as ROLE2
,ARR_ROLE[2]::string as ROLE3
,ARR_ACCESS[0]::string as ACCESS1
,ARR_ACCESS[1]::string as ACCESS2
,ARR_ACCESS[2]::string as ACCESS3
,ARR_ACCESS_LVL[0]::string as ACCESS_LVL1
,ARR_ACCESS_LVL[1]::string as ACCESS_LVL2
,ARR_ACCESS_LVL[2]::string as ACCESS_LVL3
,ARR_ITERATION[0]::string as ITERATION1
,ARR_ITERATION[1]::string as ITERATION2
,ARR_ITERATION[2]::string as ITERATION3
from X
;
There's nothing particular that seems interesting to sort the rows into the array so that ROLE1, ROLE2, ROLE3, etc. are deterministic. I showed simply sorting on the name of the role, but it could be any order by within that group.
Here's a stored proc that will produce a table result with a dynamic set of columns based on the input string and specified delimiter.
If you are looking for a way to generate dynamic column names based on values, I recommend visiting Felipe Hoffa's blog entry here:
https://medium.com/snowflake/dynamic-pivots-in-sql-with-snowflake-c763933987c
create or replace procedure pivot_dyn_results(input string, delimiter string)
returns table ()
language SQL
AS
declare
max_count integer default 0;
lcount integer default 0;
rs resultset;
stmt1 string;
stmt2 string;
begin
-- Get number of delimiter separated values (assumes no leading or trailing delimiter)
select regexp_count(:input, '\\'||:delimiter, 1) into :max_count from dual;
-- Generate the initial row-based result set of parsed values
stmt1 := 'SELECT * from lateral split_to_table(?,?)';
-- Build dynamic query to produce the pivoted column based results
stmt2 := 'select * from (select * from table(result_scan(last_query_id(-1)))) pivot(max(value) for index in (';
-- initialize look counter for resulting columns
lcount := 1;
stmt2 := stmt2 || '\'' || lcount || '\'';
-- append pivot statement for each column to be represented
FOR l in 1 to max_count do
lcount := lcount + 1;
stmt2 := stmt2 || ',\'' || lcount || '\'';
END FOR;
-- close out the pivot statement
stmt2 := stmt2 || '))';
-- execute the
EXECUTE IMMEDIATE :stmt1 using (input, delimiter);
rs := (EXECUTE IMMEDIATE :stmt2);
return table(rs);
end;
Invocation:
call pivot_dyn_results([string],[delimiter]);
call pivot_dyn_results('Sales External?HR?Billing?Purchase Department','?');
Results:
I have a column that contains a JSON array of strings, which I would like to transform into boolean columns. These columns are true if the value was present in the array.
Let's say I have the following columns in Postgres.
|"countries"|
---------------
["NL", "BE"]
["UK"]
I would like to transform this into boolean columns per market. e.g.
|"BE"|"NL"|"UK"|
--------------------
|True|True|False|
|False|False|True|
I know I can manually expand it using case statements for each country code, but there are 200+ countries.
Is there are more elegant solution?
Displaying a various list of columns whose labels are known only at the runtime is not so obvious with postgres. You need some dynamic sql code.
Here is a full dynamic solution whose result is close from your expected result and which relies on the creation of a user-defined composite type and on the standard functions jsonb_populate_record and jsonb_object_agg :
First you create the list of countries as a new composite type :
CREATE TYPE country_list AS () ;
CREATE OR REPLACE PROCEDURE country_list () LANGUAGE plpgsql AS
$$
DECLARE country_list text ;
BEGIN
SELECT string_agg(DISTINCT c.country || ' text', ',')
INTO country_list
FROM your_table
CROSS JOIN LATERAL jsonb_array_elements_text(countries) AS c(country) ;
EXECUTE 'DROP TYPE IF EXISTS country_list' ;
EXECUTE 'CREATE TYPE country_list AS (' || country_list || ')' ;
END ;
$$ ;
Then you can call the procedure country_list () just before executing the final query :
CALL country_list () ;
or even better call the procedure country_list () by trigger when the list of countries is supposed to be modified :
CREATE OR REPLACE FUNCTION your_table_insert_update()
RETURNS trigger LANGUAGE plpgsql VOLATILE AS
$$
BEGIN
IF EXISTS ( SELECT 1
FROM (SELECT jsonb_object_keys(to_jsonb(a.*)) FROM (SELECT(null :: country_list).*) AS a) AS b(key)
RIGHT JOIN jsonb_array_elements_text(NEW.countries) AS c(country)
ON c.country = b.key
WHERE b.key IS NULL
)
THEN CALL country_list () ;
END IF ;
RETURN NEW ;
END ;
$$ ;
CREATE OR REPLACE TRIGGER your_table_insert_update AFTER INSERT OR UPDATE OF countries ON your_table
FOR EACH ROW EXECUTE FUNCTION your_table_insert_update() ;
CREATE OR REPLACE FUNCTION your_table_delete()
RETURNS trigger LANGUAGE plpgsql VOLATILE AS
$$
BEGIN
CALL country_list () ;
RETURN OLD ;
END ;
$$ ;
CREATE OR REPLACE TRIGGER your_table_delete AFTER DELETE ON your_table
FOR EACH ROW EXECUTE FUNCTION your_table_delete() ;
Finally, you should get the expected result with the following query, except that the column label are lower case, and NULL is replacing false in the result :
SELECT (jsonb_populate_record(NULL :: country_list, jsonb_object_agg(lower(c.country), true))).*
FROM your_table AS t
CROSS JOIN LATERAL jsonb_array_elements_text(t.countries) AS c(country)
GROUP BY t
full test result in dbfiddle.
I've created a full-text indexed column on a table.
I have a stored procedure to which I may pass the value of a variable "search this text". I want to search for "search", "this" and "text" within the full-text column. The number of words to search would be variable.
I could use something like
WHERE column LIKE '%search%' OR column LIST '%this%' OR column LIKE '%text%'
But that would require me to use dynamic SQL, which I'm trying to avoid.
How can I use my full-text search to find each of the words, presumably using CONTAINS, and without converting the whole stored procedure to dynamic SQL?
If you say you definitely have SQL Table Full Text Search Enabled, Then you can use query like below.
select * from table where contains(columnname,'"text1" or "text2" or "text3"' )
See link below for details
Full-Text Indexing Workbench
So I think I came up with a solution. I created the following scalar function:
CREATE FUNCTION [dbo].[fn_Util_CONTAINS_SearchString]
(
#searchString NVARCHAR(MAX),
#delimiter NVARCHAR(1) = ' ',
#ANDOR NVARCHAR(3) = 'AND'
)
RETURNS NVARCHAR(MAX)
AS
BEGIN
IF #searchString IS NULL OR LTRIM(RTRIM(#searchString)) = '' RETURN NULL
-- trim leading/trailing spaces
SET #searchString = LTRIM(RTRIM(#searchString))
-- remove double spaces (prevents empty search terms)
WHILE CHARINDEX(' ', #searchString) > 0
BEGIN
SET #searchString = REPLACE(#searchString,' ',' ')
END
-- reformat
SET #searchString = REPLACE(#searchString,' ','" ' + #ANDOR + ' "') -- replace spaces with " AND " (quote) AND (quote)
SET #searchString = ' "' + #searchString + '" ' -- surround string with quotes
RETURN #searchString
END
I can get my results:
DECLARE #ftName NVARCHAR (1024) = dbo.fn_Util_CONTAINS_SearchString('value1 value2',default,default)
SELECT * FROM Table WHERE CONTAINS(name,#ftName)
I would appreciate any comments/suggestions.
For your consideration.
I understand your Senior wants to avoid dynamic SQL, but it is my firm belief that Dynamic SQL is NOT evil.
In the example below, you can see that with a few parameters (or even defaults), and a 3 lines of code, you can:
1) Dynamically search any source
2) Return desired or all elements
3) Rank the Hit rate
The SQL
Declare #SearchFor varchar(max) ='Daily,Production,default' -- any comma delim string
Declare #SearchFrom varchar(150) ='OD' -- table or even a join statment
Declare #SearchExpr varchar(150) ='[OD-Title]+[OD-Class]' -- Any field or even expression
Declare #ReturnCols varchar(150) ='[OD-Nr],[OD-Title]' -- Any field(s) even with alias
Set #SearchFor = 'Sign(CharIndex('''+Replace(Replace(Replace(#SearchFor,' , ',','),', ',''),',',''','+#SearchExpr+'))+Sign(CharIndex(''')+''','+#SearchExpr+'))'
Declare #SQL varchar(Max) = 'Select * from (Select Distinct'+#ReturnCols+',Hits='+#SearchFor+' From '+#SearchFrom + ') A Where Hits>0 Order by Hits Desc'
Exec(#SQL)
Returns
OD-Nr OD-Title Hits
3 Daily Production Summary 2
6 Default Settings 1
I should add that my search string is comma delimited, but you can change to space.
Another note CharIndex can be substanitally faster that LIKE. Take a peek at
http://cc.davelozinski.com/sql/like-vs-substring-vs-leftright-vs-charindex
I have two tables with different columns in PostgreSQL 9.3:
CREATE TABLE person1(
NAME TEXT NOT NULL,
AGE INT NOT NULL
);
CREATE TABLE person2(
NAME TEXT NOT NULL,
AGE INT NOT NULL,
ADDRESS CHAR(50),
SALARY REAL
);
INSERT INTO person2 (Name, Age, ADDRESS, SALARY)
VALUES ('Piotr', 20, 'London', 80);
I would like to copy records from person2 to person1, but column names can change in program, so I would like to select joint column names in program. So I create an array containing the intersection of column names. Next I use a function: insert into .... select, but I get an error, when I pass the array variable to the function by name. Like this:
select column_name into name1 from information_schema.columns where table_name = 'person1';
select column_name into name2 from information_schema.columns where table_name = 'person2';
select * into cols from ( select * from name1 intersect select * from name2) as tmp;
-- Create array with name of columns
select array (select column_name::text from cols) into cols2;
CREATE OR REPLACE FUNCTION f_insert_these_columns(VARIADIC _cols text[])
RETURNS void AS
$func$
BEGIN
EXECUTE (
SELECT 'INSERT INTO person1 SELECT '
|| string_agg(quote_ident(col), ', ')
|| ' FROM person2'
FROM unnest(_cols) col
);
END
$func$ LANGUAGE plpgsql;
select * from cols2;
array
------------
{name,age}
(1 row)
SELECT f_insert_these_columns(VARIADIC cols2);
ERROR: column "cols2" does not exist
What's wrong here?
You seem to assume that SELECT INTO in SQL would assign a variable. But that is not so.
It creates a new table and its use is discouraged in Postgres. Use the superior CREATE TABLE AS instead. Not least, because the meaning of SELECT INTO inside plpgsql is different:
Combine two tables into a new one so that select rows from the other one are ignored
Concerning SQL variables:
User defined variables in PostgreSQL
Hence you cannot call the function like this:
SELECT f_insert_these_columns(VARIADIC cols2);
This would work:
SELECT f_insert_these_columns(VARIADIC (TABLE cols2 LIMIT 1));
Or cleaner:
SELECT f_insert_these_columns(VARIADIC array) -- "array" being the unfortunate column name
FROM cols2
LIMIT 1;
About the short TABLE syntax:
Is there a shortcut for SELECT * FROM?
Better solution
To copy all rows with columns sharing the same name between two tables:
CREATE OR REPLACE FUNCTION f_copy_rows_with_shared_cols(
IN _tbl1 regclass
, IN _tbl2 regclass
, OUT rows int
, OUT columns text)
LANGUAGE plpgsql AS
$func$
BEGIN
SELECT INTO columns -- proper use of SELECT INTO!
string_agg(quote_ident(attname), ', ')
FROM (
SELECT attname
FROM pg_attribute
WHERE attrelid IN (_tbl1, _tbl2)
AND NOT attisdropped -- no dropped (dead) columns
AND attnum > 0 -- no system columns
GROUP BY 1
HAVING count(*) = 2
) sub;
EXECUTE format('INSERT INTO %1$s(%2$s) SELECT %2$s FROM %3$s'
, _tbl1, columns, _tbl2);
GET DIAGNOSTICS rows = ROW_COUNT; -- return number of rows copied
END
$func$;
Call:
SELECT * FROM f_copy_rows_with_shared_cols('public.person2', 'public.person1');
Result:
rows | columns
-----+---------
3 | name, age
Major points
Note the proper use of SELECT INTO for assignment inside plpgsql.
Note the use of the data type regclass. This allows to use schema-qualified table names (optionally) and defends against SQL injection attempts:
Table name as a PostgreSQL function parameter
About GET DIAGNOSTICS:
Count rows affected by DELETE
About OUT parameters:
Returning from a function with OUT parameter
The manual about format().
Information schema vs. system catalogs.
I need to create a function like this (scaled down to a minimum) where I send an array of strings that should be matched. But I cant make the query to work.
create or replace function bar(x text[]) returns table (c bigint) language plpgsql as $$
begin
return query select count(1) as counter from my_table where my_field in (x);
end;$$;
and call it like this
select * from bar(ARRAY ['a','b']);
I could try to let the parameter x be a single text string and then use something like
return query execute 'select ... where myfield in ('||x||')';
So how would I make it work with the parameter as an array?
would that be better or worse compared to let the parameter be a string?
Yes, an array is the cleaner form. String matching would leave corner cases where separators and patterns combined match ...
To find strings that match any of the given patterns, use the ANY construct:
CREATE OR REPLACE FUNCTION bar(x text[])
RETURNS bigint LANGUAGE sql AS
$func$
SELECT count(*) -- alias wouldn't visible outside function
FROM my_table
WHERE my_field = ANY(x);
$func$;
count(*) is slightly faster than count(1). Same result.
Note, I am using a plain SQL function (instead of plpgsql). Either has its pros and cons.
That's fixed with the help of unnest that converts an array to a set (btw, the function doesn't have to be plpgsql):
CREATE OR REPLACE FUNCTION bar(x text[]) RETURNS BIGINT LANGUAGE sql AS $$
SELECT count(1) AS counter FROM my_table
WHERE my_field IN (SELECT * FROM unnest(x));
$$;
The problem with using the array seems to be fixed by using
return query select count(1) as counter from my_table where my_field in (array_to_string(x,','));
The point of effiency still remains unsolved.