I need to unload huge chunks of data from snowflake into S3. Currently says that I can use the copy into feature, where I use sts assume role temporary creds to do this unload.
The problem I am wondering is: If I specified my assume role duration to be 12 hours, but unload takes longer than that. Would the unload fail then?
https://docs.snowflake.com/en/sql-reference/sql/copy-into-location.html#additional-cloud-provider-parameters
Related
Context
Developing a platform with contents and some files. Users can create a resource, and set it:
private: only accessible for him
protected: accessible via a password that can expire
public: no security rules to access
Problem
Considering we have that 2 tables:
We want to protect our element table if visibility = 'protected'. So creator should give a password for resource access, and set an expiration date.
Users can set many passwords with a different expiration date.
How can we store that password, with security, and thinking about the user can share a password, close password access, and get his password access to share resources when he needs them.
We talk here about a conception of the solution, don't care about languages or orm.
Potential solution
Cron job
Create a table joining passwords with entities, and when the user sets a password, launch a cron job that will set an attribute like active to false when the expiration date is reached.
Problem: making our app stateful, if cloud-based and the pod crash, cron job fell in void...
Store it in base64
To allow users to get back already set passwords, we have to get symmetric encryption algorithm to encrypt and decrypt the password, but that exposes resources if the database is compromised..
Also creating a table joining passwords with entities here.
No more ideas for the moment... How would you do ?
The best solution should be stateless and don't store passwords in clear.
I can think of other two solutions
A PubSub mechanism, which you can choose to trigger when to execute. For example, when you create a password today and want it to expire tomorrow. you can create a pubsub task that will trigger in a day. The task will remove that password.
A JWT token, which is a token that encoded the information in there, which includes the expiration date. When you verify that token, you will verify the signature to make sure it is not tampered and also if it is still valid. If you need to store secret inside it, you can use RS256.
I'm new to Snowflake. I have created snowpipe, stages also configured SQS in AWS.
Data is not getting loaded into table through snowpipe when I placed files in my S3 bucket.
If I'm executing statement: alter pipe snow_pipename refresh then only data getting loaded into table.
Do I need to do any more setup/instructions for auto ingest data load.
The SQS notification might not have been set up properly on the S3 bucket. Check the configuration to set it up in the below link
https://docs.snowflake.com/en/user-guide/data-load-snowpipe-auto-s3.html#step-4-configure-event-notifications
Ensure you have configured your snowflake IAM role correctly with appropriate policies and trust relationship.
Step 5 of the documentation:
https://docs.snowflake.com/en/user-guide/data-load-snowpipe-auto-s3.html#step-4-configure-event-notifications
Also, ensure that these values are reflected in your STAGE_CREDENTIALS within your stage, using DESCRIBE STAGE snowpipe_emp;
I have created an Access DB with a nice front-end, however the db is not split.
The DB is in a network location and it is shared between 5-8 users. Data loss occurs from time to time: some stored queries just disappear when sending their output via e-mail through a macro, for example.
I have read on several sites that to prevent this, I should split the db and make each user work on his/her frontend.
However, in the context I am it would be much better to have the users working on a single shared file.
The question is: If I split the database and make the users share the front-end file, would I be preventing data loss/corruption?
I've been trying to implement authentication and session management in a node.js application using socket.io.
And from almost all the resources I found, I came across the term "session store".
There are open source tools that handles sessions for us, but we have to provide them with a session store.
Some tools has built in storage for sessions in memory, for example the module express-session comes with a default in memory session store, but also this warning:
Warning The default server-side session storage, MemoryStore, is purposely not designed for a production environment. It will leak memory under most conditions, does not scale past a single process, and is meant for debugging and developing.
So I searched for the available stable session stores and it turns out that most of the names are databases that I've heard of.
For example, here's a list of session stores and another one at GitHub that I've came across.
The names include MongoDB, MySQL, SQLite, cassandra, firebase etc, hence the confusion.
So the question is, are session stores and database the same..? (I can think of it like - when we're using the database for storing session details we call it session store but it's in fact a database)
If not, how do they differ..?
Session store is a place where session data is being stored on server.
On web its usually being identified by a cookie stored in clients browser.
So it allows your app to identify user and keep him logged in for example.
Session can either be memory, some database, simple files, or any other place you can come up with to store session data.
If you project uses some database, you can configure your session store to use the same database, to avoid having another database on server just for the purpose of session store.
Differences between different session stores:
Memory session store is going to be reset on every app re-lauch. Also its fastest.
Database session store, is going to be safe with app
re-lauch. And at some point you will have alot of session objects
which you might want to clean up. And same session stored in database can be even accessed from different apps.
Session store is a method of storing information about user as a session with unique identifier. It could be stored in memory or in database. Socket.io can utilize the same session (id) being used in express app by socket-express-session package, if I am not mistaken.
You can then use session information to grant/restrict access, for example.
Pretty basic question but I haven't been able to find an answer. Using Transit I can "move" files from one S3 bucket on one AWS account to another S3 bucket on another AWS account, but what it actually does is download the files from the first then upload them to the second.
Is there a way to move files directly from one S3 account to another without downloading them in between?
Yes, there is a way. And its pretty simple, though it's hard to find it. 8)
For example, suppose your first account username is acc1#gmail.com and second is acc2#gmail.com.
Open AWS Management Console as acc1. Get to the Amazon S3 bucket properties, and in the "Permissions" tab click "Add more permissions". Then add List and View Permissions for "Authenticated Users".
Next, in AWS IAM (it's accessible from among the console tabs) of acc2 create a user with full access to the S3 bucket (to be more secure, you can set up exact permissions, but I prefer to create a temporary user for the transfer and then delete it).
Then you can use s3cmd (using the credentials of the newly created user in acc2) to do something like:
s3cmd cp s3://acc1_bucket/folder/ s3://acc2_bucket/folder --recursive
All transfer will be done on Amazon's side.
Use the aws cli (I used ubuntu 14 ec2 instance) and just run the following command:
aws s3 sync s3://bucket1 s3://bucket2
You will need to specify the account details for one, and have public write access or public read access to the other.
This will sync the two buckets. You can use the same command again later to sync quickly. Best part is that it doesn't seem t require any bandwidth (e.g. files are not passing through local computer).
If you are just looking for a ready made solution there are a few solutions out there that can do this. Bucket Explorer works on Mac and Windows and can copy across accounts as can Cloudberry S3 Explorer and S3 Browser but they are Windows only so may not work for you.
I suspect the AWS console could also do it with the appropriate permissions setup but I haven't tested this.
You can also do it using the AWS API as long as you have given the AWS account you are using write permissions to the destination bucket.
boto works well. See this thread. Using boto, you copy objects straight from one bucket to another, rather than downloading them to the local machine and uploading them to another bucket.
Move S3 files from One account to another account
Let's consider there are two accounts source account and destination account. And two buckets source-bucket and destination bucket. We want to move all files from source-bucket to destination-bucket. We can do it by the following steps:
aws configure
Configure your destination account using the credential or the IAM role.
Create user policy for the destination account user.
Give destination user access to the source-bucket by modifying the source-bucket policy and adding destination account user policy into it. By this way, destination user will have the access to source-bucket.
aws s3 ls s3://source-bucket/
this will check whether the destination account is having access to source-bucket. Just for confirmation do this.
aws s3 cp s3://source-bucket s3://destination-bucket --recursive
this will copy source-bucket all files to destination-bucket. All files are copied using --recursive flag.
aws s3 mv s3://source-bucket s3://destination-bucket --recursive
this will move all the files from source-bucket to destination-bucket.
Alternative you can use the sync command
- aws s3 sync s3://source-bucket s3://detination-bucket
For Better Explanation follow the link
On Mac OS X I used the Transmit app from Panic. I opened one window for each S3 account (using the API Keys and secrets). I could then drag from one bucket in one window to another bucket in the other window. No need to download files locally first.
Andrew is correct, Transmit downloads the files locally then uploads the files.
CrossFTP can copy S3 files straight from one bucket to another without downloading them. It is a GUI S3 client that works on Windows, Mac, and Linux.
You can user Cyberduck (open source)
For newly created files (NOT existing objects), you can take advantage of new functionality from AWS. It is Cross-Region Replication (under "Versioning" for the S3 bucket). You can create a policy that will allow you to replicate new objects to a bucket in a different account.
For existing objects, you will still need to copy your objects using another method - unless AWS introduces native functionality for this in the future.
One can so it with running following :
aws s3 mv (sync for keeping buckets in sync) s3://source-bucket s3://destination-bucket --recursive
Attach a bucket policy to the source bucket in Source Account.
Attach an AWS Identity and Access Management (IAM) policy to a user or role in Destination Account.
Use the IAM user or role in Destination Account to perform the cross-account move.