I'm trying to speed up my Log Parser Lizard queries to IIS logs on one of our servers.
This kind of query works but it's very slow:
SELECT TOP 100 * FROM '\\myserver\c$\inetpub\logs\LogFiles\W3SVC1\u_ex*.log'
ORDER BY time DESC
If I specify today's log filename that's a lot quicker:
SELECT TOP 100 * FROM '\\myserver\c$\inetpub\logs\LogFiles\W3SVC1\u_ex190731.log'
ORDER BY time DESC
I'm trying to find a way to achieve this without having to keep changing the filename within the query to match today's date. I can't find any way of using variables or functions like strcat within the FROM section of the query.
So in simpler terms, is there any way to inject today's date into a query like this:
SELECT * FROM 'C:\test\%DATE%.txt'
I found that LogParserLizard supports inline VB.NET code, specified using <% ... %> tags, so it was just a matter of inserting the date using that syntax and it worked fine.
SELECT TOP 100 *
FROM '\\myserver\c$\inetpub\logs\LogFiles\W3SVC1\u_ex<% return DateTime.Now.ToString("yyMMdd") %>.log'
ORDER BY time DESC
or in my simplified version it would be:
SELECT * FROM 'C:\test\<% return DateTime.Now.ToString("yyMMdd") %>.txt'
--converts to `C:\test\190731.txt`
Related
I saw a demo using something like this
SHOW WAREHOUSES;
SELECT * FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()))
WHERE "auto_suspend" IS NULL;
This allows to use the resultset from SHOW ... as part of a normal SQL statement.
VERY useful!
However, when I tried this in DBeaver, my LAST_QUERY_ID() always returned the result of something like "SELECT CURRENT_DATABASE(), CURRENT_SCHEMA()".
Which I discovered is precisely because DBeaver does this before any request.
Is it possible to stop this?
PS:
Of course, I can get the initial code to work by looking for an older query, like
SHOW WAREHOUSES;
SELECT * FROM TABLE(RESULT_SCAN(LAST_QUERY_ID(-2))) -- added -2
WHERE "auto_suspend" IS NULL;
But i would prefer not to have to do this...
A work around I like is to simply write the SHOW to a table. It makes it easier to use the result multiple times.
Also, if you use a transaction you don't need the -2 or Alt-X for Execute script. So
BEGIN
SHOW WAREHOUSES;
CREATE OR REPLACE TEMPORARY TABLE SHOW_WH AS
SELECT * FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()));
END;
Then
SELECT * FROM SHOW_WH WHERE "auto_suspend" IS NULL;
I just discovered that Snowflake themselves have released an extension for Visual Studio code and that does not suffer from this detail...
It does not solve the DBeaver thing... but...
And it does pretty good colour coding, intellisense, etc
Requirement: Need to fetch only the latest file everyday example here its 20200902 file
Example Files in S3:
#stagename/2020/09/reporting_2020_09_20200902000335.gz
#stagename/2020/09/reporting_2020_09_20200901000027.gz
Code:
select distinct metadata$filename from
#stagename/2020/09/
(file_format=>APP_SKIP_HEADER,pattern=>'.*/reporting_*20200902*.gz');
This will work no matter what the naming conventions of the files. Since your files appear to have a naming convention based on date and are one per point in time, you may not need to use the date to do this as you could use the name. You'll still want to use the result_scan approach.
I haven't found a way to get the date for a file in a stage other than using the LIST command. The docs say that FILE_NAME and FILE_ROW_NUMBER are the only available metadata in a select query. In any case, that approach reads the data, and we only want to read the metadata.
Since a LIST command is a metadata query, you'll need to query the result_scan to use a where clause.
One final issue that I ran into while working on a project: the last_modified date in the LIST command is in format that requires a somewhat long conversion expression to convert to timestamp. I made a UDF to do the conversion so that it's more readable. If you'd prefer putting the expression directly in the SQL, that's fine too.
First, create the UDF.
create or replace function LAST_MODIFIED_TO_TIMESTAMP(LAST_MODIFIED string)
returns timestamp_tz
as
$$
to_timestamp_tz(left(LAST_MODIFIED, len(LAST_MODIFIED) - 4) || ' ' || '00:00', 'DY, DD MON YYYY HH:MI:SS TZH:TZM')
$$;
Next, list the files in your stage or subdirectory of the stage.
list #stagename/2020/09/
Before running any other query in the session, run this one on the last query ID. You can of course run it any time in 24 hours if you specify the query ID explicitly.
select "name",
"size",
"md5",
"last_modified",
last_modified_to_timestamp("last_modified") LAST_MOD
from table(result_scan(last_query_id()))
order by LAST_MOD desc
limit 1
I did not get much help from Snowflake documentation about how I can take give column names using snowflake functions.
I have automated report which will do calculation for the given dates. here it is
select
sum(case when logdate = to_date(dateadd('day', - 10, '2019-11-14')) then eng_fees + data_fees end) AS to_date(dateadd('day', - 10, '2019-11-14'))
from myTable
where logdate = '2019-11-04'
I am getting following output for my column name below
to_date(dateadd('day', - 10, '2019-11-14'))
100
My expected output for my column name
2019-11-04
100
how can I print expected date as column name in Snowflake?
Your statement is using an AS to name your column. Snowflake will treat that as a literal, not a calculation. In order to do what you're requesting inside a function, you'll need to use a Javascript Function, I think. This will allow you to dynamically build the SQL Statement with your calculated column name predefined.
https://docs.snowflake.net/manuals/sql-reference/udf-js.html
You can't have dynamic column names without using external functionality (or TASK scheduling).
You can create a JavaScript Stored Procedure that generates a VIEW where the column names can be set by dynamic parameters / expressions.
The normal way of handling this is to use a reporting tool that can display your fixed column result set with dynamic headers or run dynamic SQL altogether.
I see 2 paths of getting close to what you want:
1) you can simply use UNION to have your headers displayed as the first row and change the actual column aliases into 1....N - so that they provide the number of column - this is going to be the fastest and the cheapest
2) you can use dynamic SQL to generate a query that will have your alias names dynamically filled and then simply run it (and you can use to example PIVOT to construct such query)
I have a log parser query that gets the top 200 uris, however I don't want any cs-uri-stem entries that have a dot (.) in them.
This is as close as I've come, but it seems like the wildcards are not acting as I expected:
"SELECT TOP 200 cs-uri-stem, COUNT(*) AS Total INTO \Top200URIs_NoDots.csv
FROM "\2015-01\U*.log"
GROUP BY cs-uri-stem
HAVING cs-uri-stem NOT LIKE '%.%'
ORDER BY Total DESC"
When I run this I get an Error:
... HAVING cs-uri-stem NOT LIKE ''...
Error: Syntax Error: <having-clause>: not a valid <expression>
Why is it ignoring the '%'s and everything between?
HAVING is for filtering group results using aggregate functions on the grouped data. Filtering on the grouped data is more processing-intensive because the grouping must be completed first. In this case, your query will be more optimally performed using a WHERE clause anyway. Also, remember to use %% if this is in a batch file. A single % denotes a batch variable and won't make it to the program's arguments.
Assume that I had the following t-sql:
select * from customer where contains(suburb, '"mount*"').
Now it brings back both: "Mount Hills" and "Blue Mountain". How do I strict it to search only the very beginning of the word which in this case is only "Mount Hills"?
thanks
'contains' is used for exactly that. Use
select * from customer where charindex('mount', suburb)=1
or
select * from customer where suburb like 'mount%'
but that's slower.
Your query works correctly, you asked server "give me all record where ANY word in column 'suburb'" starts with 'mount'.
You need to be more specific what are you trying to accomplish. Match beginning of the entire value stored in column? LIKE is your friend then.