COPY INTO SALES.SF_DEV.CUSTOMER_PARQUET_TGT FROM #EXTERNAL_STAGE_PARQUET_DEMO/partioned/customer/Country=INDIA/ MATCH_BY_COLUMN_NAME = CASE_SENSITIVE FILE_FORMAT = (TYPE = PARQUET);
The above command will load all the files recursively from dir Country=INDIA and all its sub-directories.
Que: Do we have an option to control recursive file loading, what I mean to load only files from Country=INDIA not from its sub-directories?
P.S. This image is with the pattern suggested by #gokhan
Can you try to use the pattern option?
COPY INTO SALES.SF_DEV.CUSTOMER_PARQUET_TGT
FROM #EXTERNAL_STAGE_PARQUET_DEMO/partioned/customer/Country=INDIA/
MATCH_BY_COLUMN_NAME = CASE_SENSITIVE FILE_FORMAT = (TYPE = PARQUET)
PATTERN = '[^\/]*';
The pattern is to prevent any sub-directories (files containing the slash characters).
My test environment:
ls #my_stage/test/Country=INDIA/;
+--------------------------------------------------------------------------+
| name |
+--------------------------------------------------------------------------+
| s3://mysecretbucketname/test/test/Country=INDIA/data_0_0_0.csv.gz |
| s3://mysecretbucketname/test/test/Country=INDIA/hododo/data_0_0_0.csv.gz |
+--------------------------------------------------------------------------+
ls #my_stage/test/Country=INDIA/ pattern = '[^\/]*';
+-------------------------------------------------------------------+
| name |
+-------------------------------------------------------------------+
| s3://mysecretbucketname/test/test/Country=INDIA/data_0_0_0.csv.gz |
+-------------------------------------------------------------------+
Related
Let's assume I have a JSON file that contains an array:
[ {"someField": 335, "someField2": "jkdf}, {"someField": 1, "someField2": "sdf}
Is there any plugin / simple way instead of writing own console program, to convert mentioned file into simple text table:
|someField |someField2|
| 335 | jkdf |
| 1 | sdf |
I am working with the view SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY. It would be extremely helpful to have an exhaustive list of query types that might appear in the column QUERY_TYPE, with the type of commands that generate them. For example, does a PUT command generate a PUT query type? Or is it something like "LOAD"?
If anyone knows where such a list can be found, please post a link. Snowflake's documentation of the view does not provide any list.
Thanks all who have answered so far. Since the consensus is that no such list exists, here is a merge of the entries provided so far with the values found in my own database. Please keep posting additional answers if your DB contains entries not found below. This way, sooner or later, we will have a fairly complete list:
QUERY_TYPE
CREATE_USER
REVOKE
DROP_CONSTRAINT
RENAME_SCHEMA
UPDATE
CREATE_VIEW
CREATE_TASK
RENAME_TABLE
INSERT
ALTER_TABLE_ADD_COLUMN
RENAME_COLUMN
MERGE
BEGIN_TRANSACTION
ALTER_VIEW_MODIFY_SECURITY
GRANT
ALTER_SESSION
DELETE
DROP_ROLE
DESCRIBE
UNKNOWN
TRUNCATE_TABLE
DROP
SHOW
ALTER_WAREHOUSE_SUSPEND
GET_FILES
UNLOAD
CREATE_NETWORK_POLICY
ALTER_TABLE_DROP_COLUMN
CREATE
REMOVE_FILES
ALTER
ALTER_USER
PUT_FILES
COPY
ALTER_ACCOUNT
DROP_TASK
CREATE_CONSTRAINT
DESCRIBE_QUERY
SELECT
RENAME_USER
COMMIT
RENAME_VIEW
USE
CREATE_TABLE
ALTER_NETWORK_POLICY
CREATE_ROLE
ALTER_TABLE_MODIFY_COLUMN
SET
ALTER_USER_ABORT_ALL_JOBS
ROLLBACK
LIST_FILES
UNSET
CREATE_TABLE_AS_SELECT
DROP_USER
ALTER_WAREHOUSE_RESUME
QUERY_TYPE
ALTER_PIPE
ALTER_ROLE
ALTER_TABLE
ALTER_TABLE_DROP_CLUSTERING_KEY
ALTER_USER_RESET_PASSWORD
CREATE_EXTERNAL_TABLE
CREATE_MASKING_POLICY
CREATE_SEQUENCE
CREATE_STREAM
DROP_STREAM
RENAME_DATABASE
RENAME_FILE_FORMAT
RENAME_ROLE
RENAME_WAREHOUSE
RESTORE
By the looks of it there is no complete list of query types that show up in this table. Best I can do is give you a list from my own database, which still doesn't contain things like alter role etc. To answer your other question a PUT command is actually PUT_FILES by the looks of it:
select distinct query_type from SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY;
+-------------------------+
|QUERY_TYPE |
+-------------------------+
|ALTER |
|ALTER_SESSION |
|ALTER_TABLE_ADD_COLUMN |
|ALTER_TABLE_DROP_COLUMN |
|ALTER_TABLE_MODIFY_COLUMN|
|ALTER_USER |
|ALTER_WAREHOUSE_RESUME |
|ALTER_WAREHOUSE_SUSPEND |
|BEGIN_TRANSACTION |
|COMMIT |
|COPY |
|CREATE |
|CREATE_CONSTRAINT |
|CREATE_EXTERNAL_TABLE |
|CREATE_MASKING_POLICY |
|CREATE_ROLE |
|CREATE_SEQUENCE |
|CREATE_STREAM |
|CREATE_TABLE |
|CREATE_TABLE_AS_SELECT |
|CREATE_USER |
|CREATE_VIEW |
|DELETE |
|DESCRIBE |
|DESCRIBE_QUERY |
|DROP |
|DROP_CONSTRAINT |
|DROP_STREAM |
|DROP_USER |
|GET_FILES |
|GRANT |
|INSERT |
|LIST_FILES |
|MERGE |
|PUT_FILES |
|REMOVE_FILES |
|RENAME_COLUMN |
|RENAME_DATABASE |
|RENAME_TABLE |
|RESTORE |
|REVOKE |
|ROLLBACK |
|SELECT |
|SET |
|SHOW |
|TRUNCATE_TABLE |
|UNKNOWN |
|UNLOAD |
|UPDATE |
|USE |
+-------------------------+
Added ours ... 16 extra's ... pass it on :-)
QUERY_TYPE
ALTER
ALTER_ACCOUNT
ALTER_PIPE
ALTER_ROLE
ALTER_SESSION
ALTER_TABLE
ALTER_TABLE_ADD_COLUMN
ALTER_TABLE_DROP_CLUSTERING_KEY
ALTER_TABLE_DROP_COLUMN
ALTER_TABLE_MODIFY_COLUMN
ALTER_USER
ALTER_USER_ABORT_ALL_JOBS
ALTER_USER_RESET_PASSWORD
ALTER_WAREHOUSE_RESUME
ALTER_WAREHOUSE_SUSPEND
BEGIN_TRANSACTION
COMMIT
COPY
CREATE
CREATE_CONSTRAINT
CREATE_EXTERNAL_TABLE
CREATE_MASKING_POLICY
CREATE_NETWORK_POLICY
CREATE_ROLE
CREATE_SEQUENCE
CREATE_STREAM
CREATE_TABLE
CREATE_TABLE_AS_SELECT
CREATE_TASK
CREATE_USER
CREATE_VIEW
DELETE
DESCRIBE
DESCRIBE_QUERY
DROP
DROP_CONSTRAINT
DROP_ROLE
DROP_STREAM
DROP_TASK
DROP_USER
GET_FILES
GRANT
INSERT
LIST_FILES
MERGE
PUT_FILES
REMOVE_FILES
RENAME_COLUMN
RENAME_DATABASE
RENAME_FILE_FORMAT
RENAME_ROLE
RENAME_SCHEMA
RENAME_TABLE
RENAME_USER
RENAME_VIEW
RENAME_WAREHOUSE
RESTORE
REVOKE
ROLLBACK
SELECT
SET
SHOW
TRUNCATE_TABLE
UNKNOWN
UNLOAD
UNSET
UPDATE
USE
Here are some additional ones:
ALTER_AUTO_RECLUSTER
ALTER_SET_TAG
ALTER_TABLE_MODIFY_CONSTRAINT
ALTER_UNSET_TAG
CALL
DROP_SESSION_POLICY
RECLUSTER
I am trying to get user's information from Azure AD directly, like DisplayName and UserPrincipalName, using KQL. Is there a way to do so?
Ended up exporting the needed user's attributes using PowerShell and copying the output into a Blob container, then ran the below KQL query to join the file content with the query:
let UserAtt = externaldata (UserPrincipalName:string, DisplayName:string) [
#"URL to the file location in the blob storage"
h#"?sp="Secret token"
] with (format="csv", ignoreFirstRecord=true);
UserAtt
| join kind=inner (
OfficeActivity
| where TimeGenerated > ago(1h)
| where (Operation =~ "Set-Mailbox" and Parameters contains 'ForwardingSmtpAddress')
or (Operation =~ 'New-InboxRule' and Parameters contains 'ForwardTo')
| extend parsed=parse_json(Parameters)
| extend fwdingDestination_initial = (iif(Operation=~"Set-Mailbox", tostring(parsed[1].Value), tostring(parsed[2].Value)))
| where isnotempty(fwdingDestination_initial)
| extend fwdingDestination = iff(fwdingDestination_initial has "smtp", (split(fwdingDestination_initial,":")[1]), fwdingDestination_initial )
| parse fwdingDestination with * '#' ForwardedtoDomain
| parse UserId with *'#' UserDomain
| extend subDomain = ((split(strcat(tostring(split(UserDomain, '.')[-2]),'.',tostring(split(UserDomain, '.')[-1])), '.') [0]))
| where ForwardedtoDomain !contains subDomain
| extend Result = iff( ForwardedtoDomain != UserDomain ,"Mailbox rule created to forward to External Domain", "Forward rule for Internal domain")
| extend ClientIPAddress = case( ClientIP has ".", tostring(split(ClientIP,":")[0]), ClientIP has "[", tostring(trim_start(#'[[]',tostring(split(ClientIP,"]")[0]))), ClientIP )
| extend Port = case(
ClientIP has ".", (split(ClientIP,":")[1]),
ClientIP has "[", tostring(split(ClientIP,"]:")[1]),
ClientIP
)
| project TimeGenerated, UserId, UserDomain, subDomain, Operation, ForwardedtoDomain, ClientIPAddress, Result, Port, OriginatingServer, OfficeObjectId, fwdingDestination
| extend timestamp = TimeGenerated, AccountCustomEntity = UserId, IPCustomEntity = ClientIPAddress, HostCustomEntity = OriginatingServer)
on $left.UserPrincipalName == $right.AccountCustomEntity
In inotify I see the following opcodes:
const agnosticEvents = unix.IN_MOVED_TO | unix.IN_MOVED_FROM |
unix.IN_CREATE | unix.IN_ATTRIB | unix.IN_MODIFY |
unix.IN_MOVE_SELF | unix.IN_DELETE | unix.IN_DELETE_SELF
Is there a way to get when a file was opened/accessed? For example, if I typed in cat <file> or open <file> or vim <file>. How would I get that event to flow-through, or is it not possible?
This question already has answers here:
Find and Replace Inside a Text File from a Bash Command
(18 answers)
Closed 4 years ago.
This is output of the grep -E "scsi0:" *.vmx | grep -E "fileName" command in one of my directory:
scsi0:0.fileName = "vmname.vmdk"
scsi0:1.fileName = "vmname_1.vmdk"
scsi0:2.fileName = "vmname_2.vmdk"
scsi0:3.fileName = "P120_vmname_2.vmdk"
scsi0:4.fileName = "P120_vmname_3.vmdk"
I need to rewrite above output inside that vmx file so it looks like following
scsi0:0.fileName = "vmname.vmdk"
scsi0:1.fileName = "vmname_1.vmdk"
scsi0:2.fileName = "vmname_2.vmdk"
scsi0:3.fileName = "vmname_3.vmdk"
scsi0:4.fileName = "vmname_4.vmdk"
So in essence the script needs to the following:
look for the line which contains scsi0 and filename and remove everything after double quote before vmname
check what is the number in that line after "scsi0:" and add/replace that number after the underscore, so P120_vmname_2 becomes vmname_3
The thing is that there can by any number of characters before vmname which need to be removed and lines which need to be fixed can be anywhere in the source file.
Do I need to assign individual line output to separate variable or it is possible to manipulate them with just one?
Thanks
Could you pipe your grep output into sed?
If so, you could do a substitution like this:
sed -r 's/^(.*:)([[:digit:]]+)(.*) = \".*vmname_[[:digit:]]+/\1\2\3 = \"vmname_\2/'
# ^ ^ ^ ^ ^ ^ ^
# | | | | | | Replace number.
# | | | | | Re-create line start
# | | | | Match incorrect number
# | | | Match garbage before "vmname"
# | | Match Filename
# | Correct number
# Line start
So your actual command would look like this:
grep -E "scsi0:" *.vmx | grep -E "fileName" | \
sed -r 's/^(.*:)([[:digit:]]+)(.*) = \".*vmname_[[:digit:]]+/\1\2\3 = \"vmname_\2/'
Converts this:
scsi0:0.fileName = "vmname.vmdk"
scsi0:1.fileName = "vmname_1.vmdk"
scsi0:2.fileName = "vmname_2.vmdk"
scsi0:3.fileName = "P120_vmname_2.vmdk"
scsi0:4.fileName = "P120_vmname_3.vmdk"
scsi0:5.fileName = "P12asdasdsada_asdasd_sdsad0_vmname_3.vmdk"
scsi0:6.fileName = "vmname_3.vmdk"
To this:
scsi0:0.fileName = "vmname.vmdk"
scsi0:1.fileName = "vmname_1.vmdk"
scsi0:2.fileName = "vmname_2.vmdk"
scsi0:3.fileName = "vmname_3.vmdk"
scsi0:4.fileName = "vmname_4.vmdk"
scsi0:5.fileName = "vmname_5.vmdk"
scsi0:6.fileName = "vmname_6.vmdk"