when I go to the history page, I see a list of executed queries, but when I check a particular one, I can't see its arguments just a question marks instead of a values. Is it possible to see these values or are they secret for security reasons?
SELECT * FROM table(information_schema.QUERY_HISTORY_BY_WAREHOUSE(WAREHOUSE_NAME => concat('', ?, ''), RESULT_LIMIT => ?)) WHERE QUERY_ID IN (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
Related
I have a Logstash pipeline that fetches data from MS SQL view that joins to tables A and B and put the denormalised data into ES.
Initially, INSERTS or UPDATES could happen only for table A. Therefore, to configure Logstash to pick up only newly inserted or updated records since last iteration of the polling loop, I have defined the tracking_column field which refers updatedDate timestamp column in table A:
jdbc {
#Program Search
jdbc_connection_string => "jdbc:sqlserver://__DB_LISTNER__"
jdbc_user => “admin”
jdbc_password => “admin”
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_driver_library => "/usr/share/logstash/drivers/mssql-jdbc-6.2.2.jre8.jar"
sql_log_level => "info"
tracking_column => "updated_date_timestamp"
use_column_value => true
tracking_column_type => "timestamp"
schedule => "*/1 * * * *"
statement => "select *, updateDate as updated_date_timestamp from dbo.MyView where (updateDate > :sql_last_value and updateDate < getdate()) order by updateDate ASC"
last_run_metadata_path => "/usr/share/logstash/myfolder/.logstash_jdbc_last_run"
}
Now, the UPDATES can also happen in table B. With this new requirement I am confused how can I configure Logstash to track changes on the table B as well in the same pipeline. Can I define multiple tracking_columns for the same pipeline?
Another two options I have in mind but not sure about them are:
Generate a composite value from updateDate fields of table A and B, that will be referenced by the tracking_column. But I am not sure how the SQL query should look like then?
Create another pipeline that will track changes for table B only. Though, the drawback, I see for this approach, is that the existing and new pipelines will do duplicate work on the initial iterations in order to process all the records from the DB view.
Please, advise me how should I go from here?
I found this ES discussion that suggests to use a function to select greatest value of provided dates in the SQL query. For the SQL server there is GREATEST function, but it is not recognised by SQL server I am currently using. Long story short, as a workaround I found iff() function which I use for dates comparing. So my SQL query looks like this:
select *, iif(A.updatedDate>B.updatedDate, A.updatedDate, B.updatedDate) as updated_date_timestamp from dbo.MyView where (iif(A.updatedDate>B.updatedDate, A.updatedDate, B.updatedDate) > :sql_last_value and iif(A.updatedDate>B.updatedDate, A.updatedDate, B.updatedDate) < getdate()) order by iif(A.updatedDate>B.updatedDate, A.updatedDate, B.updatedDate) ASC, id ASC
I have searched around and can not find much on this topic. I have a table, that gets logging information. As a result the column I am interested in contains multiple values that I need to search against. The column is formatted in a php URL style. i.e.
/test/test.aspx?DS_Vendor=55039&DS_ProdVer=7.90.100.0&DS_ProdLang=EN&DS_Product=MTT&DS_OfficeBits=32
This makes all searches end up with really long regexes to get data. Then join statements to combine data.
Is there a way in BigQuery, or SQL Server that I can pull the information from that column and put it into new columns?
Example:
The information I would like extracted begins after the ?, and ends at &, The string can sometimes be longer, and contains additional headers.
Thanks,
Below is for BigQuery Standard SQL and addresses below aspect of your question
Is there a way in BigQuery, ... that I can pull the information from that column and put it into new columns?
#standardSQL
CREATE TEMP FUNCTION parseColumn(kv STRING, column_name STRING) AS (
IF(SPLIT(kv, '=')[OFFSET(0)]= column_name, SPLIT(kv, '=')[OFFSET(1)], NULL)
);
WITH `project.dataset.table` AS (
SELECT '/test/test.aspx?extra=abc&DS_Vendor=55039&DS_ProdVer=7.90.100.0&DS_ProdLang=EN&DS_Product=MTT&DS_OfficeBits=32' AS url UNION ALL
SELECT '/test/test.aspx?DS_Vendor=55192&DS_ProdVer=4.30.100.0&more=123&DS_ProdLang=DE&DS_Product=MTE&DS_OfficeBits=64'
)
SELECT
MIN(parseColumn(kv, 'DS_Vendor')) AS DS_Vendor,
MIN(parseColumn(kv, 'DS_ProdVer')) AS DS_ProdVer,
MIN(parseColumn(kv, 'DS_ProdLang')) AS DS_ProdLang,
MIN(parseColumn(kv, 'DS_Product')) AS DS_Product,
MIN(parseColumn(kv, 'DS_OfficeBits')) AS DS_OfficeBits
FROM `project.dataset.table`,
UNNEST(REGEXP_EXTRACT_ALL(url, r'[?&]([^?&]+)')) AS kv
GROUP BY url
with the result as below
Row DS_Vendor DS_ProdVer DS_ProdLang DS_Product DS_OfficeBits
1 55039 7.90.100.0 EN MTT 32
2 55192 4.30.100.0 DE MTE 64
Below is also addressed
The string can sometimes be longer, and contains additional headers.
One example using BigQuery (with standard SQL):
SELECT REGEXP_EXTRACT_ALL(url, r'[?&]([^?&]+)')
FROM (
SELECT '/test/test.aspx?DS_Vendor=55039&DS_ProdVer=7.90.100.0&DS_ProdLang=EN&DS_Product=MTT&DS_OfficeBits=32' AS url
)
This returns the parts of the URL as an ARRAY<STRING>. To go one step further, you can get back an ARRAY<STRUCT<key STRING, value STRING>> with a query of this form:
SELECT
ARRAY(
SELECT AS STRUCT
SPLIT(part, '=')[OFFSET(0)] AS key,
SPLIT(part, '=')[OFFSET(1)] AS value
FROM UNNEST(REGEXP_EXTRACT_ALL(url, r'[?&]([^?&]+)')) AS part
) AS keys_and_values
FROM (
SELECT '/test/test.aspx?DS_Vendor=55039&DS_ProdVer=7.90.100.0&DS_ProdLang=EN&DS_Product=MTT&DS_OfficeBits=32' AS url
)
...or with the keys and values as top-level columns:
SELECT
SPLIT(part, '=')[OFFSET(0)] AS key,
SPLIT(part, '=')[OFFSET(1)] AS value
FROM (
SELECT '/test/test.aspx?DS_Vendor=55039&DS_ProdVer=7.90.100.0&DS_ProdLang=EN&DS_Product=MTT&DS_OfficeBits=32' AS url
)
CROSS JOIN UNNEST(REGEXP_EXTRACT_ALL(url, r'[?&]([^?&]+)')) AS part
I have a requirement to pull data in a specific format and I'm struggling slightly with the ROWTOCOL function and was hoping a fresh pair of eyes might be able to help.
I'm using 10g Oracle DB (10.2) so LISTAGG which appears to do what I need to achieve is not an option.
I need to aggregate a number of usernames into a string delimited with a '$' but I also need to concatenate another column to to build up email addresses.
select
rowtocol('select username_id from username where user_id = '||s.user_id|| 'order by USERNAME_ID asc','#'||d.domain_name||'$')
from username s, domain d
where s.user_id = d.user_id
(I've simplified the query specific to just this function as the actual query is quite large and all works except for this particular function.)
in the DOMAIN Table I have a number of domains such as 'hotmail.com','gmail.com' etc
I need to concatenate the username, an '#' symbol followed by the domain and all delimited with a '$'
such as ......
joe.bloggs#gmail.com$joeblogs#gmail.com$joe_bloggs#gmail.com
I've battled with this and I've got close but in reverse?!.....
gmail.com$joe.bloggs#gmail.com$joeblogs#gmail.com$joe_bloggs
I've also noticed that if I play around with the delimiter (,'#'||d.domain_name||'$') it has a tendency to drop off the first character as can be seen above the preceding '#' has been dropped from the first email address.
Can anyone offer any suggestions as to how to get this working?
Many Thanks in advance!
Assuming you're using the rowtocol function from OTN, and have tables something like:
create table username (user_id number, username_id varchar2(20));
create table domain (user_id number, domain_name varchar2(20));
insert into username values (1, 'joe.bloggs');
insert into username values (1, 'joebloggs');
insert into username values (1, 'joe_bloggs');
insert into domain values (1, 'gmail.com');
Then your original query gets three rows back:
gmail.com$joe.bloggs
gmail.com$joe_bloggs#gmail.com$joebloggs
gmail.com$joe_bloggs#gmail.com$joebloggs
You're passing the data from each of your user IDs to a separate call to rowtocol, which isn't really what you want. You can get the result I think you're after by reversing it; pass the main query that joins the two tables as the select argument to the function, and have that passed query do the username/domain concatenation - that is a separate step to the string aggregation:
select
rowtocol('select s.username_id || ''#'' || d.domain_name from username s join domain d on d.user_id = s.user_id', '$')
from dual;
which gets a single result:
joe.bloggs#gmail.com$joe_bloggs#gmail.com$joebloggs#gmail.com
Whether that fits into your larger query, which you haven't shown, is a separate question. You might need to correlate it with the rest of your query.
There are other ways to string aggregation in Oracle, but this function is one way, and you already have it installed. I'd look at alternatives though, such as ThomasG's answer, which make it a bit clearer what's going on I think.
As Alex told you in comments, this ROWTOCOL isn't a standard function so if you don't show its code, there's nothing we can do to fix it.
However you can accomplish what you want in Oracle 10 using the XMLAGG built-in function.
try this :
SELECT
rtrim (xmlagg (xmlelement (e, s.user_id || '#' || d.domain_name || '$')).extract ('//text()'), '$') whatever
FROM username s
INNER JOIN domain d ON s.user_id = d.user_id
I have an audit running on multiple tables in my database that is triggered when a user runs a select against said tables. I am attempting to parse the 'statement' returned from the audit and get a username out of the information. Here is my query:
SELECT *
FROM (SELECT LastName,
queryLastName,
statement,
Event_time AS CurrentDateTime,
'AuditFile' AS SourceTable
FROM (SELECT statement,
event_time,
SUBSTRING(server_principal_name, 9, len(server_principal_name) - 8) AS LastName,
SUBSTRING(SUBSTRING(statement, CHARINDEX('LastName like ''%', statement) + 16, 20), 0, CHARINDEX('%''', SUBSTRING(statement, CHARINDEX('LastName like ''', statement) + 16, 20))) AS queryLastName
FROM #MyTempTable
WHERE server_principal_name != 'abc'
AND CHARINDEX('LastName like ''%', statement) > 0) AS A) AS B
WHERE querylastname != ''
Now, the above query will correctly return any SELECT query that has Lastname like '%name%', but i'd also like to be able to return names looked up in other ways such as '%name' or '_name__'. Are there any more elegant solutions to solve this problem?
Edit: The Audit file contains a column called 'Statement'. This holds what the query was that triggered the audit. For example,
SELECT * from tblUserNames where LastName Like '%Smith%'
will trigger the audit and place the above query in statement.
My query takes Smith out of the string however if a user entered '%Smith' instead, the query would not work as it does not pick up variations. Same goes for 'Smith', 'Smith' etc. If possible, I need a more elegant way to handle multiple situations where a user could look up a name using wildcards on either end of the name.
I want to update on a linked server the result of a query as well from a linked server.
The first sql snippet gives me the value to be updated:
SELECT mmdb_vessel.IMONo, mmdb_vessel.DeathDate
From OPENQUERY(MMDB, 'SELECT FunctionalLocation, IMONo, VesselStatus, CONVERT(VARCHAR(10), DeathDate, 102) AS DeathDate
FROM VESSEL
WHERE VESSEL.VesselStatusID <> 42 AND VESSEL.DeathDate is not null') as mmdb_vessel
, eb_all_v
WHERE
eb_all_v.IMO_No = mmdb_vessel.IMONo
AND eb_all_v.status = 'in service'
the second is actually what I'm not able to implement, it should show what I want to achieve:
UPDATE EPI2..EPI.PLANT
SET KIND_OF_LIQUIDATION_NO = 1
, LIQUIDATION_DATE = [result from snippet above].DeathDate
Where EPI2..EPI.PLANT.IMONo = [result from snippet above].IMONo
I'm not so sure if my explanation is sufficient, please feel free to ask for additional information!
Thanks, already in advance,
Werner
I would recommend to select the data from the remote server first and store the required data e.g. in a temptable, because LinkedServer and updates can have some sideeffects (e.g. performing a tablescan on the remote table, altough you would not expect it if an updaet is involved, etc) - but this depends on your exact usage/scenario.
Select data you need to update
SELECT * INTO #tmpTable FROM LINKEDSERVER.EPI.dbo.PLANT WHERE ....
Perform the update on local server
UPDATE EPI2..EPI.PLANT SET KIND_OF_LIQUIDATION_NO = 1, LIQUIDATION_DATE = t.DeathDate FROM #tmpTable t INNER JOIN EPI2..EPI.PLANT p on t.IMONo = p.IMONo