Is it possible for an IoT device on Google Cloud Platform to upload a file to the cloud somehow? In this case it's not exactly a telemetry data publishing. A direct access to the bucket could be useful. Either that or access to my AppEngine-based application, but if I upload it to my app (that will in the next step put the file into a bucket), how can I possibly authenticate the device to make sure that the device that sends it to me is actually the device it claims to be?
How big of a file? What's the device?
One of the big advantages of using Cloud IoT Core is that it handles the security piece so you don't have to think about that part. You can send any Base64 encoded data, so sending a file should be fine. The problem is the quota on size. Only 256KB per message, so you might need to break down the file into parts before sending and reassemble on the other side if they're big files.
The other way you could go is use IAM and service accounts as security and have the device speak directly to one of the other services like Cloud Storage. As you say, that does make authentication more difficult on device because it's using a bearer token (service_account.json). The service account can be heavily limited in its authority though (e.g. write-only to a specific Cloud Storage bucket) to limit what it could do if it were compromised.
I managed to solve the problem by passing a Signed URL to the device via the configuration. https://cloud.google.com/storage/docs/access-control/signed-urls
Related
I have a social media app deployed on App Engine where users can upload and share photos/videos with a private group of people. For writes, I have a POST endpoint that accepts uploaded files and writes them to one GCS bucket that's not public. For reading, a GET endpoint checks with Cloud SQL if this user is authorized to access the media file - if yes, it returns the file stream. The file is stored only for 48 hours and average retrieval is 20 times per day. Users are authenticated using Firebase email link login.
The issue with the current approach is that my GET endpoint is an expensive middleman for reading the GCS file stream and passing it on to the clients, adding to the cost as may times the API is invoked.
There's no point caching the file on App Engine because cache hit ratio will be extremely low for my use case.
The GET API could return a GCS file URL instead of File Stream, if I make the GCS bucket public. But that would mean anyone can access the file with this public URL, not just my app or limited user. Plus, the entire bucket is vulnerable now.
I could create an ACL for each GCS file object, but ACLs work only for users with Google accounts and my app uses email link authentication. There's also a limit on ACL entries per object in case the file needs to be shared with more than 100 people.
The last option I have is to create a signed link that works for a short duration, enabling limited unauthorized sharing.
Also tagging Google Photos. In case the partner sharing program can help with this problem, then I can migrate from GCS to Google Photos for storage.
This looks like a common use-case for media based apps. Are there any recommended design patterns to achieve the goal in a cost effective way?
This is my first week learning GCP, so I maybe wrong in some of the points shared above.
We currently use the blobstore to handle user uploads (and will likely shift to GCS). Our solution allows users to upload files but I've recently found that users could potentially upload a virus (knowingly or unknowingly).To mitigate this risk I'm considering limiting file types to images and/or pdfs (this would be checked server side). Would this prevent a virus from being uploaded or should I also perform a virus scan on the files once they're uploaded?
If running a virus scan, is there a simple for solution for doing this with GAE or do I need a separate cloud compute instance running it's own virus scan?
Thanks
Rob
Any time you delegate authority to upload an object to an untrusted client, there is risk that the client or malicious code posing as the client can upload malicious content. As far as I am aware, neither Google App Engine's Blobstore service nor Google Cloud Storage provide virus scanning as a service, so you'd have to bring your own. Limiting file types doesn't actually inhibit bad content being uploaded, as some browsers will ignore the stated file type after sniffing file content and render or execute the malicious object.
If you want to do this yourself for a Google Cloud Storage upload, the best practice would be to restrict the upload to have a private ACL, perform whatever sanitization you want, and when determined to be valid, change the ACL to allow broader permissions.
/via Vinny P:
There are online virus-scanning tools you can use programmatically, or you can run an anti-virus engine on Compute Engine or in an App Engine Flexible Environment. Alternatively, if these are supposed to be user-owned files under 25 MB, you could upload the files to Google Drive which will provide virus scanning, and retrieve the files via the Drive API.
I studied and could successfully replicate the quickstart.py example on https://developers.google.com/drive/web/quickstart/quickstart-python to upload a file to my google drive using command line.
However, I wish to write an app that does the same, but through a cron job i.e. uploads a file everyday at 8am say, without the need to authenticate each time. Is there sample code/examples that I can look at to implement the oauth steps without the command line intervention?
Thanks!
You can use your App Engine app's built-in Service Account to authorize requests to the Google Drive API.
https://cloud.google.com/appengine/docs/python/appidentity/
https://developers.google.com/identity/protocols/OAuth2ServiceAccount
Your app will need to have an embedded Refresh Token, or some way of fetching it from a secure server. The Refresh Token acts a bit like a stored username/password, albeit with constrained access. Therefore you need to consider the security implications. For example, since it's uploading, it will only need drive.file scope, so your corpus of Drive files remain inaccessible.
If you're happy with the security implications, then the steps you need are described How do I authorise an app (web or installed) without user intervention? (canonical ?)
I have Amazon S3 as my file storage server and EC2 instances as my application logic server. Now my app needs to upload some files which need necessary processing. I can think of two ways to do this:
Upload the file directly from mobile devices and get the file name and location(url), then send the url to my backend. Backend get the file by URL and do its job.
Send the file to backend using a multipart form, backend accepts the file do its job and finally save the file to Amazon S3.
Which is the standard way? What are the reasons?
Sending the object direct to Amazon S3 will be more scalable, less error-prone and cheaper (since you need less web server capacity to handle the uploads). Send the corresponding information to a Simple Queueing Service (SQS) queue that the back-end service can monitor and process. That way, if your back-end is ever offline, the jobs will simply queue-up and will get processed when the server is running again. A good use of loose coupling.
A third option would be to send the file directly from your mobile to Amazon S3, using Metadata fields to identify originating user, and then configure the S3 bucket to trigger some code in AWS Lambda that can process the file. This could do all the processing, or could simply trigger a process on your web server. Again, this reduces load on the web server and would not require sending a message to trigger the processing.
I build Android app link to Google Cloud Storage. I want to allow access to GCS to my android app ONLY.
Google offers three solutions to securely connect to GCS:
Oauth 2.0 (So with google account)
Cookie-base Account (With google account too)
Service Account Authentication (With private Key, but locally installed on Android App: Very Bad if someone decompile my .apk)
Source: https://developers.google.com/storage/docs/authentication?hl=FR
Is there any other solution to connect securely over GCS ? I would like to connect on GCS to this way (Restrict to Android client ID: SHA1 to your .apk) : https://developers.google.com/appengine/docs/java/endpoints/auth
It is possible with GCS ? Should I use Blobstore to do that ?
Thanks in advance
This is something of a fundamental problem with computing. You can never completely trust that an application running on hardware that is under the total control of an unknown third party has not been somehow tampered with. There are many, many techniques to make tampering much more difficult, but remote systems will never be completely secure. There are several ways to verify that a user has a particular Google account, but you can't easily trust with certainty that a certain app is exactly your app.
That said, there are plenty of ways to design a secure application without trusting the client. What does your app need to be authorized to do? Upload objects? Download secure objects? Is there something bad that a user masquerading as your application could do?
I think you can use 1) to authenticate the information. The app will forward the authentication request to your server (with your own app login token), and when the user is validated by your own services, then the app will receive the oauth token to send to gcloud and receive the desired file.