S3 credentials in COPY INTO not working with session variables - snowflake-cloud-data-platform

Question

Running this in a worksheet I'm expecting the data to be loaded into my_table:
SET AWS_ACCESS_KEY_ID='XXXX';
SET AWS_SECRET_ACCESS_KEY='XXXX';
COPY INTO my_table (name,dog_name) FROM s3://my-bucket/*.csv
credentials=(aws_key_id='$AWS_ACCESS_KEY_ID' aws_secret_key='$AWS_SECRET_ACCESS_KEY')
file_format=(type=csv field_delimiter=',' empty_field_as_null=true compression=gzip
field_optionally_enclosed_by='"');
But I'm getting Failure using stage area. Cause: [The AWS Access Key Id you provided is not valid.].
Replacing the variables directly with value is working, am I missing something?

Answer 1

The variables in the SET command here is a SQL variable which can't be passed as an option inside a COPY INTO command dynamically, you can create a Stored Proc to use for this scenario, try using the following:
create or replace procedure execute_copy_into(AWS_ACCESS_KEY_ID STRING,AWS_SECRET_ACCESS_KEY STRING)
returns string
language javascript
execute as CALLER
as
$$
var sql_command="COPY INTO my_table (name,dog_name) FROM s3://my-bucket/*.csv \
credentials=(aws_key_id='"+AWS_ACCESS_KEY_ID+"' aws_secret_key='"+AWS_SECRET_ACCESS_KEY+"') \
file_format=(type=csv field_delimiter=',' empty_field_as_null=true compression=gzip \
field_optionally_enclosed_by='"'); "
var stmt = snowflake.createStatement(
{
sqlText: sql_command
}
);
var rs = stmt.execute();
// Looping to return all files processed
while (rs.next()) {
result += rs.getColumnValue(1) + "\n";
}
return result;
$$
;
// Now call the stored proc
SET AWS_ACCESS_KEY_ID='XXXX';
SET AWS_SECRET_ACCESS_KEY='XXXX';
CALL execute_copy_into($AWS_ACCESS_KEY_ID,$AWS_SECRET_ACCESS_KEY);
// Or just eliminate the variables and call it without
CALL execute_copy_into('XXXX','XXXX');