I'm wondering if it's possible to download a file from Google Cloud Storage with a different name than the one that has in the bucket.
For example, in Google Cloud Storage I have stored a file named 123-file.txt but when I download it I would like choose a different name, let's say file.txt
I've noticed that the link for download it is like:
https://storage.cloud.google.com/bucket_name%2F123-file.txt?response-content-disposition=attachment;%20filename=123-file.txt
So I've tried to change it to:
https://storage.cloud.google.com/bucket_name%2F123-file.txt?response-content-disposition=attachment;%20filename=file.txt
But it still keeps downloading as 123-file.txt instead of file.txt
The response-content-disposition parameter can only be used by authorized requests. Anonymous links don't work with it. You have a few options:
The content-disposition of a particular object is part of its metadata and can be permanently set. If you always want a specific file to be downloaded with a specific name, you can just permanently set the content-disposition metadata for the object.
You can also generate signed URLs that include the response-content-disposition query parameter. Then the users will be making authorized requests to download the resource.
example (first option Brandon Yarbrough) with javascript library:
const storage = new Storage()
const fileBucket = storage.bucket('myBucket')
const file = fileBucket.file('MyOriginalFile.txt')
const newName = "NewName.txt"
await file.save(content, {
metadata: {
contentDisposition: `inline; filename="${newName}"`
}
})
the following is a part of a python script i've used to remove the forward slashes - added by google cloud buckets when to represent directories - from multiple objects, it's based on this blog post, please keep in mind the double quotes around the content position "file name"
def update_blob_download_name(bucket_name):
""" update the download name of blobs and remove
the path.
:returns: None
:rtype: None
"""
# Storage client, not added to the code for brevity
client = initialize_google_storage_client()
bucket = client.bucket(bucket_name)
for blob in bucket.list_blobs():
if "/" in blob.name:
remove_path = blob.name[blob.name.rfind("/") + 1:] # rfind gives that last occurence of the char
ext = pathlib.Path(remove_path).suffix
remove_id = remove_path[:remove_path.rfind("_id_")]
new_name = remove_id + ext
blob.content_disposition = f'attachment; filename="{new_name}"'
blob.patch()
Related
I have a Flask app where a user can upload an image and the image is saved on a static folder on the filesystem.
Currently, I'm using Google App Engine for hosting and found that it's not possible to save to the static folder on the standard environment. Here is the code
def save_picture(form_picture,name):
picture_fn = name + '.jpg'
picture_path = os.path.join(app.instance_path, 'static/image/'+ picture_fn)
output_size = (1000,1000)
i = Image.open(form_picture)
i.thumbnail(output_size)
i.save(picture_path)
return picture_path
#app.route('/image/add', methods=['GET', 'POST'])
def addimage():
form = Form()
if form.validate_on_submit():
name = 'randomname'
try:
picture_file = save_picture(form.image.data,name)
return redirect(url_for('addimage'))
except:
flash("unsuccess")
return redirect(url_for('addimage'))
My question is if I change from standard to flex environment would it be possible to save to a static folder? If not what are the other hosting options that I should consider? Do you have any suggestions?
Thanks in advance.
following your's advice I'm changing to use Cloud Storage. i'm wondering what should i use from upload_from_file(), upload_from_filename() or upload_from_string(). the source_file takes data from form.photo.data from flask-wtform. i'm not successfully saving on the cloud storage yet. this is my code:
def upload_blob(bucket_name, source_file, destination_blob_name):
storage_client = storage.Client()
bucket = storage_client.get_bucket(bucket_name)
blob = bucket.blob(destination_blob_name)
blob.upload_from_filename(source_file)
return destination_blob_name
#app.route('/image/add', methods=['GET', 'POST'])
def addimage():
form = Form()
if form.validate_on_submit():
name = 'randomname'
try:
filename = 'foldername/'+ name + '.jpg'
picture_file = upload_blob('mybucketname', form.photo.data, filename)
return redirect(url_for('addimage'))
except:
flash("unsuccess")
return redirect(url_for('addimage'))
I have successfully able to save file on google cloud storage by changing the save_picture function just in case anyone have trouble with this in the future:
app.config['BUCKET'] = 'yourbucket'
app.config['UPLOAD_FOLDER'] = '/tmp'
def save_picture(form_picture,name):
picture_fn = secure_filename(name + '.jpg')
picture_path = os.path.join(app.config['UPLOAD_FOLDER'], picture_fn)
output_size = (1000,1000)
i = Image.open(form_picture)
i.thumbnail(output_size)
i.save(picture_path)
storage_client = storage.Client()
bucket = storage_client.get_bucket(app.config['BUCKET'])
blob = bucket.blob('static/image/'+ picture_fn)
blob.upload_from_filename(picture_path)
return picture_path
The problem with storing it to some folder is that it would live on that one instance and other instances would not be able to access it. Furthermore, instances in GAE come and go, so you would lose the image eventually.
You should use Google Cloud Storage for this:
from google.cloud import storage
client = storage.Client()
bucket = client.get_bucket('bucket-id-here')
blob = bucket.get_blob('remote/path/to/file.txt')
blob.upload_from_string('New contents!')
https://googleapis.dev/python/storage/latest/index.html
With Flask and Appengine, Python3.7, I save files to a bucket in the following way, because I want to loop it for many files:
for key, upload in request.files.items():
file_storage = upload
content_type = None
identity = str(uuid.uuid4()) # or uuid.uuid4().hex
try:
upload_blob("f00b4r42.appspot.com", request.files[key], identity, content_type=upload.content_type)
The helper function:
from google.cloud import storage
def upload_blob(bucket_name, source_file_name, destination_blob_name, content_type="application/octet-stream"):
"""Uploads a file to the bucket."""
storage_client = storage.Client()
bucket = storage_client.get_bucket(bucket_name)
blob = bucket.blob(destination_blob_name)
blob.upload_from_file(source_file_name, content_type=content_type)
blob.make_public()
print('File {} uploaded to {}.'.format(
source_file_name,
destination_blob_name))
Changing from Google App Engine Standard Environment to Google App Engine Flexible Environment will allow you to write to disk, as well as to choose a Compute Engine machine type with more memory for your specific application [1]. If you are interested on following this path find all the relevant documentation from migrating a Python app here.
Nonetheless, as it was explained by user #Alex on his provided answer as instances are created (the number of instances is scaled up) or deleted (the number of instances is scaled down) according to your load, the better option in your particular case would be to use Cloud Storage. Find an example for uploading objects to Cloud Storage with Python here.
I'm trying to use the code from this question getServingUrl() Method using Google Cloud Storage value
GcsFilename gcsFilename = new GcsFilename("bucketName", "objectName");
ImagesService is = ImagesServiceFactory.getImagesService();
String filename = String.format("/gs/%s/%s", gcsFilename.getBucketName(), gcsFilename.getObjectName());
String servingUrl = is.getServingUrl(ServingUrlOptions.Builder.withGoogleStorageFileName(filename));
I don't understand why there is /gs/ in the beginning of the path - it causes the code to produce a url that doesn't exist. but when I remove it I get 'java.lang.IllegalArgumentException: Google storage filenames must be prefixed with /gs/'
So back to the original question - how to get a serving url from a GCS file?
I can do
"http://storage.googleapis.com/"+ filename.getBucketName()+"/"+ filename.getObjectName()
but it breaks on local server, thus I assume is not the correct way.
It depends on how you want to access the file. If you access the file directly from the client (browser), then you can use a public URL:
"http://storage.googleapis.com/"+ filename.getBucketName()+"/"+ filename.getObjectName()
If you access this file internally - in your code, or call getServingUrl(), you use "/gs/" prefix. For example:
Image picture = ImagesServiceFactory.makeImageFromFilename("/gs/" + filename.getBucketName() + "/" + filename.getObjectName());
GcsFilename fileName = new GcsFilename(filename.getBucketName(), filename.getObjectName());
I think there is just a problem with the file name in your example.
I have a public bucket here:
http://storage.googleapis.com/tripket1/
And all the files in this bucket have the ACL set to 'public-read'. Yet when I try to view any of the files, such as:
http://storage.googleapis.com/tripket1/2013-05-25%2019.17.32_150.jpg
it returns a 'NoSuchKey' error.
<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
</Error>
What could be causing this problem? These files were uploaded using the GCS client library for Java. Here's a code snipped from the uploader:
GcsFilename thumbGcsFilename = new GcsFilename(bucketName, thumb_filename);
GcsFileOptions options = new GcsFileOptions.Builder().mimeType("image/" + photo_extension).acl("public-read").build();
GcsOutputChannel outputChannel = gcsService.createOrReplace(thumbGcsFilename, options);
outputChannel.write(ByteBuffer.wrap(newImageData));
outputChannel.close();
LOGGER.info("Wrote file");
String thumb_url_str = String.format("http://storage.googleapis.com/%s/%s", bucketName, thumb_filename);
return thumb_url_str;
You need to escape the % character in your object names.
For example, you have the following object:
gs://tripket1/2013-05-25%2019.17.32_150.jpg
Since you have a literal percent sign in your object's name, it must be escaped as %25 when URL encoded, so you can access the object with this URL:
http://storage.googleapis.com/tripket1/2013-05-25%252019.17.32_150.jpg
If you don't escape it, the %20 in your object name gets turned into a space () when being decoded at the server side, and it doesn't find an object name with a space in it.
Since it's a publicly readable bucket, I used gsutil to look at its contents, and I see the object you're trying to read is called
2013-05-25%2019.17.32_150.jpg
rather than
06b78005-4ad8-43d6-8fc5-bab867b653af/2013-05-25%2019.17.32_150.jpg
How do I write a Google Apps Script that deletes files?
This finds files:
var ExistingFiles = DocsList.find(fileName);
But DocsList.deleteFile does not exist to delete a file.
Is there a way to move those files to another Folder or to Trash?
The other workaround I would consider is to be able to override an existing file with the same name.
Currently when I want to create a file with a name already used in MyDrive then it creates a second file with the same name. I would like to keep 1 file (the new one is kept and the old one is lost).
There are 3 services available to delete a file.
DriveApp - Built-in to Apps Script
Advanced Drive Service - Built-in to Apps Script but must be enabled. Has more capability than DriveApp
Google Drive API - Not built-in to Apps Script, but can be used from Apps Script using the Drive REST API together with UrlFetchApp.fetch(url,options)
The DocsList service is now deprecated.
The Advanced Drive Service can be used to delete a file without sending it to the trash. Seriously consider the risk of not being able to retrieve the deleted file. The Advanced Drive Service has a remove method which removes a file without sending it to the trash folder. Advanced services have many of the same capabilities as the API's, without needing to make an HTTPS GET or POST request, and not needing an OAuth library.
function delteFile(myFileName) {
var allFiles, idToDLET, myFolder, rtrnFromDLET, thisFile;
myFolder = DriveApp.getFolderById('Put_The_Folder_ID_Here');
allFiles = myFolder.getFilesByName(myFileName);
while (allFiles.hasNext()) {//If there is another element in the iterator
thisFile = allFiles.next();
idToDLET = thisFile.getId();
//Logger.log('idToDLET: ' + idToDLET);
rtrnFromDLET = Drive.Files.remove(idToDLET);
};
};
This combines the DriveApp service and the Drive API to delete the file without sending it to the trash. The Drive API method .remove(id) needs the file ID. If the file ID is not available, but the file name is, then the file can first be looked up by name, and then get the file ID.
In order to use DriveAPI, you need to add it through the Resources, Advanced Google Services menu. Set the Drive API to ON. AND make sure that the Drive API is turned on in your Google Cloud Platform. If it's not turned on in BOTH places, it won't be available.
Now you may use the following if the file is as a spreadsheet, doc etc.:
DriveApp.getFileById(spreadsheet.getId()).setTrashed(true);
or if you already have the file instead of a spreadsheet, doc etc. you may use:
file.setTrashed(true);
This code uses the DocsList Class which is now deprecated.
try this :
function test(){
deleteDocByName('Name-of-the-file-to-delete')
}
function deleteDocByName(fileName){
var docs=DocsList.find(fileName)
for(n=0;n<docs.length;++n){
if(docs[n].getName() == fileName){
var ID = docs[n].getId()
DocsList.getFileById(ID).setTrashed(true)
}
}
}
since you can have many docs with the same name I used a for loop to get all the docs in the array of documents and delete them one by one if necessary.
I used a function with the filename as parameter to simplify its use in a script, use test function to try it.
Note : be aware that all files with this name will be trashed (and recoverable ;-)
About the last part of your question about keeping the most recent and deleting the old one, it would be doable (by reading the last accessed date & time) but I think it is a better idea to delete the old file before creating a new one with the same name... far more logical and safe !
Though the The service DocsList is now deprecated, as from the Class Folder references, the settrashed method is still valid:
https://developers.google.com/apps-script/reference/drive/folder#settrashedtrashed
So should work simply this:
ExistingFiles.settrashed(true);
Here is another way to do it without the need of Drive API. (based on Allan response).
function deleteFile(fileName, folderName) {
var myFolder, allFiles, file;
myFolder = DriveApp.getFoldersByName(folderName).next();
allFiles = myFolder.getFilesByName(fileName);
while (allFiles.hasNext()) {
file = allFiles.next();
file.getParents().next().removeFile(file);
}
}
Here is a slightly modified version using the above. This will backup said file to specified folder, also remove any old previous backups with the same name so there are no duplicates.
The idea is here to backup once per day, and will retain 1 month of backups in your backup folder of choice. Remember to set your trigger to daily in your Apps Script.
https://gist.github.com/fmarais/a962a8b54ce3f53f0ed57100112b453c
function archiveCopy() {
var file = DriveApp.getFileById("original_file_id_to_backup");
var destination = DriveApp.getFolderById("backup_folder_name");
var timeZone = Session.getScriptTimeZone();
var formattedDate = Utilities.formatDate(new Date(),timeZone,"dd"); // 1 month backup, one per day
var name = SpreadsheetApp.getActiveSpreadsheet().getName()+"_"+formattedDate;
// remove old backup
var allFiles = destination.getFilesByName(name);
while (allFiles.hasNext()) {
var thisFile = allFiles.next();
thisFile.setTrashed(true);
};
// make new backup
file.makeCopy(name,destination);
}
I'd like to use GAE to allow a few users to upload files and later retrieve them. Files will be relatively small (a few hundred KB), so just storing stuff as a blob should work. I haven't been able to find any examples of something like this. There are a few image uploading examples out there but I'd like to be able to store word documents, pdfs, tiffs, etc. Any ideas/pointers/links? Thanks!
The same logic used for image uploads apply for other archive types. To make the file downloadable, you add a Content-Disposition header so the user is prompted to download it. A webapp simple example:
class DownloadHandler(webapp.RequestHandler):
def get(self, file_id):
# Files is a model.
f = Files.get_by_id(file_id)
if not f:
return self.error(404)
# Set headers to prompt for download.
headers = self.response.headers
headers['Content-Type'] = f.content_type or 'application/octet-stream'
headers['Content-Disposition'] = 'attachment; filename="%s"' % f.filename
# Add the file contents to the response.
self.response.out.write(f.contents)
(untested code, but you get the idea :)
It sounds like you want to use the Blobstore API.
You don't mention if you are using Python or Java so here are links to both.
I use blobstore API that admits any file upload/download up to 50 MB.