I am aiming to take a file a user attaches through an Lightning Component and create a document object containing the data.
So far I have overcome the request size limits by chunking the data being uploaded into 1MB chunks. When the Apex Aura method receives these chunks of data it will either create a new document (if it is the first chunk), or will retrieve the existing document and add the new chunk to the end.
Data is received Base64 encoded, and then decoded server-side.
As the document data is stored as a Blob, the original file contents will be read as a String, and then appended with the chunk received. The new contents are then converted back into a Blob to be stored within the ContentVersion object.
The problem I'm having is that strings in Apex have a maximum length of 6,000,000 or so. Whenever the file size exceeds 6MB, this limit is hit during the concatenation, and will cause the file upload to halt.
I have attempted to avoid this limit by converting the Blob to a String only when necessary for the concatenation (as suggested here https://developer.salesforce.com/forums/?id=906F00000008w9hIAA) but this hasn't worked. I'm guessing it was patched because it's still technically allocating a string larger then the limit.
Code's really simple when appending so far:
ContentVersion originalDocument = [SELECT Id, VersionData FROM ContentVersion WHERE Id =: <existing_file_id> LIMIT 1];
Blob originalData = originalDocument.VersionData;
Blob appendedData = EncodingUtil.base64Decode(<base_64_data_input>);
Blob newData = Blob.valueOf(originalData.toString() + appendedData.toString());
originalDocument.VersionData = newData;
You will have hard time with it.
You could try offloading the concatenation to asynchronous process (#future/Queueable/Schedulable/Batchable), they'll have 12MB RAM instead of 6. Could buy you some time.
You could try cheating by embedding an iframe (Visualforce or lightning:container tag? Or maybe a "canvas app") that would grab your file and do some manual JavaScript magic calling normal REST API for document upload: https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/dome_sobject_insert_update_blob.htm (last code snippet is about multiple documents). Maybe jsforce?
Can you upload it somewhere else (SharePoint? Heroku?) and have that system call into SF to push them (no Apex = no heap size limit). Or even look "Files Connect" up.
Can you send an email with attachments? Crude but if you write custom Email-to-Case handler class you'll have 36 MB of RAM.
You wrote "we needed multiple files to be uploaded and the multi-file-upload component provided doesn't support all extensions". That may be caused by these:
In Experience Builder sites, the file size limits and types allowed follow the settings determined by site file moderation.
lightning-file-upload doesn't support uploading multiple files at once on Android devices.
if the Don't allow HTML uploads as attachments or document records security setting is enabled for your organization, the file uploader cannot be used to upload files with the following file extensions: .htm, .html, .htt, .htx, .mhtm, .mhtml, .shtm, .shtml, .acgi, .svg.
Related
I was thinking if I store a video or a movie and open that box will that video will be stored in my RAM or else it just load from ROM. I am a bit confused: Can anyone explain this to me?
I think you have misunderstood the concept of Database.
Any Database solution is to only store pure informational organized data. Not to store large files such as media, documents, or images.
On the contrary, storage need not be organized, all files can exist in one folder.
So, any database solution you use, always store Data Types.
In this case you can have a Data Model, which is also an essential thing in using a Database.
#HiveType(typeId: 0)
class Movie extends HiveObject {
#HiveField(0)
String name;
#HiveField(1)
int path;
}
Since Hive supports Dart objects, you don't have to convert toJson or any such for string the Data.
So when you have the file fetched from Storag, you can get the path using path_provider or from the File itself, and then Create a Object
File file = await // get the movie file using any means
final path = file.path
var box = await Hive.openBox('Movies');
var m = Movie()
..name = 'Batman Begins'
..path = path ;
box.add(m);
m.save();
Hope this clears your doubt.
Copy/save your video/media files in the Local File Storage and save file path in Hive Box.
Whenever you need get path from hive then get the file from local storage using that path.
I don't see a solution to this using the available api documentation.
It is also not available on the web console.
Is it possible to get the file url using the Watson Discovery Service?
If you need to store the original source/file URL, you can include it as a field within your documents in the Discovery service, then you will be able to query that field back out when needed.
I also struggled with this request but ultimately got it working using Python bindings into Watson Discovery. The online documentation and API reference is very poor; here's what I used to get it working:
(Assume you have a Watson Discovery service and have a created collection):
# Programmatic upload and retrieval of documents and metadata with Watson Discovery
from watson_developer_cloud import DiscoveryV1
import os
import json
discovery = DiscoveryV1(
version='2017-11-07',
iam_apikey='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
url='https://gateway-syd.watsonplatform.net/discovery/api'
)
environments = discovery.list_environments().get_result()
print(json.dumps(environments, indent=2))
This gives you your environment ID. Now append to your code:
collections = discovery.list_collections('{environment-id}').get_result()
print(json.dumps(collections, indent=2))
This will show you the collection ID for uploading documents into programmatically. You should have a document to upload (in my case, an MS Word document), and its accompanying URL from your own source document system. I'll use a trivial fictitious example.
NOTE: the documentation DOES NOT tell you to append , 'rb' to the end of the open statement, but it is required when uploading a Word document, as in my example below. Raw text / HTML documents can be uploaded without the 'rb' parameter.
url = {"source_url":"http://mysite/dis030.docx"}
with open(os.path.join(os.getcwd(), '{path to your document folder with trailing / }', 'dis030.docx'), 'rb') as fileinfo:
add_doc = discovery.add_document('{environment-id}', '{collections-id}', metadata=json.dumps(url), file=fileinfo).get_result()
print(json.dumps(add_doc, indent=2))
print(add_doc["document_id"])
Note the setting up of the metadata as a JSON dictionary, and then encoding it using json.dumps within the parameters. So far I've only wanted to store the original source URL but you could extend this with other parameters as your own use case requires.
This call to Discovery gives you the document ID.
You can now query the collection and extract the metadata using something like a Discovery query:
my_query = discovery.query('{environment-id}', '{collection-id}', natural_language_query="chlorine safety")
print(json.dumps(my_query.result["results"][0]["metadata"], indent=2))
Note - I'm extracting just the stored metadata here from within the overall returned results - if you instead just had:
print(my_query) you'll get the full response from Discovery ... but ... there's a lot to go through to identify just your own custom metadata.
I'm looking to store images for an application in an MSSQL database. (I understand that there is some debate about whether this or file system storage is better; that's another thread though.) I'm looking at doing something similar to http://forum.codecall.net/topic/40286-tutorial-storing-images-in-mysql-with-php/ but in CodeIgniter, something along the lines of:
foreach ($_FILES as $upload_name => $info) {
if ($info['name']) {
// Temporary file name stored on the server
$tmpName = $info['tmp_name'];
// Read the file
$fp = fopen($tmpName, 'r');
$data = fread($fp, filesize($tmpName));
fclose($fp);
//model code consolidated here for ease of question-asking
$db = $this->load->database();
$stmt = $db->insert('my_table', array('image' => $data));
}
}
My question is mostly along the lines of security. Basically is there any particular concerns I should have for sanitizing image binary data inserts versus other sorts of string data? I took out the addslashes() in the code from the site linked above because I know CI's active records do some sanitization on their own but I don't know if it is better to have it (or do some other prep work altogether).
If I understand your question correctly, you should not have to worry about it as long as you store the file_type (The file's Mime type) with it and fore the Mime type with the binary data. Then whenever you handle the data you make sure and use it with the proper Mime type so even if they upload a script of virus you can make sure it is only rendered as an image instead of letting your server or the browser handle it.
Other than this I do not think you will need to pull the upload into memory and try and scrub it.
I recently started using bottle and GAE blobstore and while I can upload the files to the blobstore I cannot seem to find a way to download them from the store.
I followed the examples from the documentation but was only successful on the uploading part. I cannot integrate the example in my app since I'm using a different framework from webapp/2.
How would I go about creating an upload handler and download handler so that I can get the key of the uploaded blob and store it in my data model and use it later in the download handler?
I tried using the BlobInfo.all() to create a query the blobstore but I'm not able to get the key name field value of the entity.
This is my first interaction with the blobstore so I wouldn't mind advice on a better approach to the problem.
For serving a blob I would recommend you to look at the source code of the BlobstoreDownloadHandler. It should be easy to port it to bottle, since there's nothing very specific about the framework.
Here is an example on how to use BlobInfo.all():
for info in blobstore.BlobInfo.all():
self.response.out.write('Name:%s Key: %s Size:%s Creation:%s ContentType:%s<br>' % (info.filename, info.key(), info.size, info.creation, info.content_type))
for downloads you only really need to generate a response that includes the header "X-AppEngine-BlobKey:[your blob_key]" along with everything else you need like a Content-Disposition header if desired. or if it's an image you should probably just use the high performance image serving api, generate a url and redirect to it.... done
for uploads, besides writing a handler for appengine to call once the upload is safely in blobstore (that's in the docs)
You need a way to find the blob info in the incoming request. I have no idea what the request looks like in bottle. The Blobstoreuploadhandler has a get_uploads method and there's really no reason it needs to be an instance method as far as I can tell. So here's an example generic implementation of it that expects a webob request. For bottle you would need to write something similar that is compatible with bottles request object.
def get_uploads(request, field_name=None):
"""Get uploads for this request.
Args:
field_name: Only select uploads that were sent as a specific field.
populate_post: Add the non blob fields to request.POST
Returns:
A list of BlobInfo records corresponding to each upload.
Empty list if there are no blob-info records for field_name.
stolen from the SDK since they only provide a way to get to this
crap through their crappy webapp framework
"""
if not getattr(request, "__uploads", None):
request.__uploads = {}
for key, value in request.params.items():
if isinstance(value, cgi.FieldStorage):
if 'blob-key' in value.type_options:
request.__uploads.setdefault(key, []).append(
blobstore.parse_blob_info(value))
if field_name:
try:
return list(request.__uploads[field_name])
except KeyError:
return []
else:
results = []
for uploads in request.__uploads.itervalues():
results += uploads
return results
For anyone looking for this answer in future, to do this you need bottle (d'oh!) and defnull's multipart module.
Since creating upload URLs is generally simple enough and as per GAE docs, I'll just cover the upload handler.
from bottle import request
from multipart import parse_options_header
from google.appengine.ext.blobstore import BlobInfo
def get_blob_info(field_name):
try:
field = request.files[field_name]
except KeyError:
# Maybe form isn't multipart or file wasn't uploaded, or some such error
return None
blob_data = parse_options_header(field.content_type)[1]
try:
return BlobInfo.get(blob_data['blob-key'])
except KeyError:
# Malformed request? Wrong field name?
return None
Sorry if there are any errors in the code, it's off the top of my head.
I have an image in blob store which is uploaded by users(their profile pic). I want to make a copy of the same and and re-size the copy so that it can be displayed as a thumbnail. I want to make a copy of the same instead of using the ImageService because this would be used more often compared to the profile image.
What I am doing here is this:
reader = profile_image.open() #get binary data from blob
data = reader.read()
file_name = files.blobstore.create(mime_type=profile_image.content_type)#file to write to
with files.open(file_name, 'a') as f:
f.write(data)
files.finalize(file_name)
blob_key = files.blobstore.get_blob_key(file_name)
image = images.Image(blob_key = blob_key)
image.resize(width=32, height=32)
entity.small_profile_pic = <MyImageModel>(caption=<caption given by user>,
picture=str(blob_key))
This is giving me error:
BadValueError: Image instance must have a complete key before it can be stored as a reference.
I think this is because the blob is not saved(put()) into the datastore, but how do i do it. Doed files.blobstore.get_blob_key(file_name) not do it ?
I would also like to ask: does the blobstore also cache the dynamically transformed images images served using get_serving_url() ...
I would use the get_serving_url method. In the doc is stated that:
The get_serving_url() method allows you to generate a stable, dedicated URL for serving web-suitable image thumbnails. You simply store a single copy of your original image in Blobstore, and then request a high-performance per-image URL. This special URL can serve that image resized and/or cropped automatically, and serving from this URL does not incur any CPU or dynamic serving load on your application (though bandwidth is still charged as usual). Images are served with low latency from a highly optimized, cookieless infrastructure.
Also the code you posted doesn't seem to follow the exampled posted in the docs. I would use something like this
img = images.Image(blob_key=original_image_key)
img.resize(width=32, height=32)
thumbnail = img.execute_transforms(output_encoding=images.JPEG)
file_name = files.blobstore.create(mime_type='image/jpeg')#file to write to
with files.open(file_name, 'a') as f:
f.write(thumbnail)
files.finalize(file_name)
blob_key = files.blobstore.get_blob_key(file_name)