How may I upload a data file to my Google app engine folder using any standard file transfer protocol e.g. SFTP, WebDav etc.?
This is so I can use a regular desktop file transfer client
So far the only ways I've found are Google proprietary, and web form upload.
GAE's filesystem is readonly, i.e. you can't really make changes to any of the files once the app is deployed.
You can upload your files to GCS by mounting a bucket as a local directory but as you noted that option doesn't seem to be available on Windows.
You could also use gsutil to transfer files to/from the bucket, like that: gsutil cp *.txt gs://my-bucket, even on Windows (see more details here) but if you want a GUI client - just google for gui clients for gcs, many storage browsers i.e. more or less advanced ftp clients seem to support it (see for example Cyberduck, CrossFTP, etc).
Related
We currently use the blobstore to handle user uploads (and will likely shift to GCS). Our solution allows users to upload files but I've recently found that users could potentially upload a virus (knowingly or unknowingly).To mitigate this risk I'm considering limiting file types to images and/or pdfs (this would be checked server side). Would this prevent a virus from being uploaded or should I also perform a virus scan on the files once they're uploaded?
If running a virus scan, is there a simple for solution for doing this with GAE or do I need a separate cloud compute instance running it's own virus scan?
Thanks
Rob
Any time you delegate authority to upload an object to an untrusted client, there is risk that the client or malicious code posing as the client can upload malicious content. As far as I am aware, neither Google App Engine's Blobstore service nor Google Cloud Storage provide virus scanning as a service, so you'd have to bring your own. Limiting file types doesn't actually inhibit bad content being uploaded, as some browsers will ignore the stated file type after sniffing file content and render or execute the malicious object.
If you want to do this yourself for a Google Cloud Storage upload, the best practice would be to restrict the upload to have a private ACL, perform whatever sanitization you want, and when determined to be valid, change the ACL to allow broader permissions.
/via Vinny P:
There are online virus-scanning tools you can use programmatically, or you can run an anti-virus engine on Compute Engine or in an App Engine Flexible Environment. Alternatively, if these are supposed to be user-owned files under 25 MB, you could upload the files to Google Drive which will provide virus scanning, and retrieve the files via the Drive API.
Q1: Where do you think is the right place to put a SQLite database file (database.sqlite) in Azure Web App file system? For example:
D:\home\data\database.sqlite
D:\home\site\database.sqlite
D:\home\site\wwwroot\database.sqlite
other?
Q2: What else should be taken into consideration in order to make sure that the database file won't be accessible to public users as well as not being accidentally overwritten during deployments or when the app is scaled up/down? (The Web App is configured for deployments from a Local Git Repository)
Q3: Where to learn more about the file system used in Azure App Service, the official source URL? E.g. how it's shared between multiple VMs within a single Web App, how does it work when the App is scaled up/down, what's the difference between D:\home (persistent) vs D:\local (non-persistent)...
Note that SQLite does not work in Azure Blob Storage, so that one is not an option. Please, don't suggest alternative storage solutions, this question is specifically about SQLite.
References
Appropriate Uses For SQLite
In a Web App, your app is deployed to d:\home\site\wwwroot. This is the area where you may write files. As an example, the ghost deployment writes its SQLite database to d:\home\site\wwwroot\content\data\ghost.db. (easy to see this, if you open up the kudu console via yourapp.scm.azurewebsites.net):
This file area is shared amongst your web app instances. Similar to an SMB file share, but specific to web apps (and different than Azure's File Service).
The content under wwwroot is durable, unless you delete your app service. Scaling up/down impacts the amount of space available. (I have no idea what happens if you scale down and the smaller size has less disk space than what you're consuming already).
I would say the best location would be app_data folder in the site/wwwroot folder. Create the folder if it doesn't exist.
Web Apps can connect to storage accounts so you can in fact use blob storage and connect that to your web app. So in terms of learning more about it then you need to be looking at the appropriate page of documentation.
In your Web App settings you can then select which storage account to use. You can find this under Settings > Data Connections where you can select Storage from the drop down box.
In my GWT application, a 'root' user upload a specific text file with data and that data should be available to anyone who have access to the app (using GAE).
What's the classic way to store a data that will be available to all users? I don't want to use any database (objectify!?) since this is a relatively small amount of information and it changes from time to time by root.
I was wondering if there was such static MAP on the 'engine level' (not user's session) that this info can be stored (and if the server is down - no bigi, root will upload again)
Thanks
You have three primary options:
Add this file to your /war/ directory and deploy with the app. This is what we typically do with all static files that rarely change (like .css file, images, etc.) This file will be available to all users, whether they are authenticated or not.
Add this file to your /war/WEB-INF/ directory and deploy with the app. This file will be available to your server-side code, so you can read it on the server-side and show to a user. This way you can decide which users can see this file and which users should not have access to it.
Upload this file to Google Cloud Storage. You can do it through an app, or you can simply upload it manually to a bucket using a GCS console or gsutil command-line tool. Then you simply provide a link to your users. The advantage of this option is that you do not have to redeploy your app when a file changes.
The only reason to go with the first two options is to have this file under version control. If you don't need that, I would recommend going with the GCS option.
I'm writing an app where users can write Notes, and each note can have many files attached to it.
I would like users to be able to click 'Browse', select multiple files, which will be uploaded when the user clicks 'Save Note'.
I want these files to be uploaded directly into Amazon S3 (or some other cloud storage solution?) without going through my server, so I don't have to worry about uploads blocking my server.
What is the best way to accomplish this?
I have seen many examples to upload directly into Amazon S3, but none of them support multiple files. Will I somehow have to do this all in Javascript by looping through a collection of files selected with the Browse button?
Thanks!
Technically, your javascript residing in the browser could make HTTP RESTful calls to AWS and store data in S3, but then you would be exposing the security credentials to connect to AWS in the script.. not good.
I guess the only way is to process it thru a web-server which can securely access AWS and store the notes.. or, you could just write those notes to a local disk (where the webserver sits), and schedule tools like s3cmd to automatically synch them with S3 buckets.
I have some big files stored on Google Storage. I would like users to be able to download them only when they are authenticated to my GAE application. The user would use a link of my GAE such as http://myapp.appspot.com/files/hugefile.bin
My first try works for files which sizes are < 32mb. Using the Google Storage experimental API, I could read the file first then serve it to the user. It required my GAE application to be a team member of the project which Google Storage was enabled. Unfortunately this doesn’t work for large files, and it hogs bandwidth by first downloading the file to GAE and then serving it to the player.
Does anyone have an idea on how to carry out that?
You can store files up to 5GB in size using the Blobstore API: http://code.google.com/appengine/docs/python/blobstore/overview.html
Here's the Stackoverflow thread on this: Upload file bigger than 40MB to Google App Engine?
One thing to note, is reading blobstore can only be done in 32MB increments, but the API provides ways to accessing portions of the file for reads: http://code.google.com/appengine/docs/python/blobstore/overview.html#Serving_a_Blob
FYI in the upcoming 1.6.4 release of AppEngine we've added the ability to pass a Google Storage object name to the blobstore.send_blob() to send Google Storage files of any size from you AppEngine application.
Here is the pre-release announcement for 1.6.4.