I'm trying to upload a file with an em/en dash '–' character in its name (encoded in utf8). The character is rendered correctly in the browser request (looking at chromes network monitor). But no encoding information is sent with the request to the server, the character is then interpreted as 'â' (code page 28591, iso-8859-1) by the server. I've come to understand that using
filename*=UTF-8''myFileWith–In.pdf
may work but can't seem to manipulate the filename attribute in a suitable way, for example not using double quotes. Source
My angular code looks something like this, is it possible from here to add encoding information to the content disposition?
function uploadFile(file, url)
{
$upload.upload({
url: url,
method: 'POST',
file: file,
});
}
On the server I can use the following code to correct the encoding but it assumes that all incoming traffic was encoded in utf8, there needs to be a way of identifying the encoding in the request.
Encoding iso = Encoding.GetEncoding("ISO-8859-1");
Encoding utf8 = Encoding.UTF8;
byte[] isoBytes = iso.GetBytes(file.FileName);
string msg = utf8.GetString(isoBytes);
You can use fileName option of upload to set the content-disposition's filename. See the readme reference for explanation.
function uploadFile(file, url)
{
$upload.upload({
url: url,
method: 'POST',
file: file,
fileName: 'UTF-8\'\'' + file.name
});
}
Related
I am using react with fetch for sending an image to the server.
I have tried using Content-type = application/x-www-form-urlencoded to pass my data to the server but it replaces special characters with spaces (i.e. + becomes a space).
I have switched the header to be Content-type: multipart/form-data but that throws the error
Failed to load resource: the server responded with a status of 500
(Internal Server Error).
I have added a boundary to the Content-type as boundary=abcdefg.
That did not change anything and I am not sure what my boundary would be.
Finding a clear answer with straight forward examples about boundaries have been impossible to get.
The data that I am sending is a large string.
If needed I can post that as well.
Here is a sample of the code that is causing the problem:
SaveTest4(data: string) {
const options = {
method: 'post',
headers: {
"Content-type": "multipart/form-data; boundary=abcdefg"
},
body: 'data=' + data
}
fetch('api/DataPoint/AddTest4', options);
}
Based on part of your analysis, it sounds like you're trying to send base64-encoded data. A content type of application/x-www-form-urlencoded will result in the server performing URL decoding, which will replace each instance of + in the content body with a space character.
When you used a content type of multipart/form-data, the server fails with status 500 because the data you provided wasn't a properly constructed MIME document.
My psychic debugging powers tell me that you're trying to post a base64-encoded file to a ASP.NET MVC WebAPI endpoint that's expecting a JSON document. You might have a controller method that looks like this:
[HttpPost("api/DataPoint/AddTest4")]
public void AddTest4([FromBody] string data) { ... }
If you send with a content type of application/json, this endpoint will expect a document that looks like this:
"{base64-encoded-data}"
Note that there are quotes around the data, because a quoted string is a proper JSON document. You'd just use JSON.stringify() to create the quoted string in this case, which would escape any quotes within the string correctly.
If you send with application/x-www-form-urlencoded, you'd need to send a document that looks like this:
data={base64-encoded-data}
But, as you note, you'd have to make sure you escape all of the special characters in the payload; you can do this using window.encodeURIComponent(), which would translate each "+" to "%2B", among other things.
If the files that you're uploading to this endpoint are large, it would be significantly better to use an instance of FormData. This would allow the browser to stream the file to the server in chunks instead of reading it into memory and base64-encoding it in JavaScript.
I have a openstack object storage container to which I'm trying to upload files directly from browser.
As per the documentation here, I can upload the file using a PUT request and I'm doing this using Angularjs provided $http.put method as shown below.
$http.put(temporaryUploadUrl,
formData,
{
headers: {
'Content-Type': undefined
}
}
).then(function (res) {
console.log("Success");
});
The file uploads successfully and it has no problems in authentication and gives me a 201 Created response. However the file is now containing junk lines on the top and bottom of it because its a multipart request sent using FormData().
Sample file content before upload:
Some sample text
here is more text
here is some other text
File content after downloadiong back from openstack container :
------WebKitFormBoundaryTEeQZVW5hNSWtIqS
Content-Disposition: form-data; name="file"; filename="c.txt"
Content-Type: text/plain
Some sample text
here is more text
here is some other text
------WebKitFormBoundaryTEeQZVW5hNSWtIqS--
I tried the FileReader to read the selected file as a binary string and wrote the content to the request body instead of FormData and the request which works fine for text files but not the binary files like XLSX or PDF The data is entirely corrupted this way.
EDIT:
The following answer is now considered a less performing workaround As
it will encode the entire file to base64 multipart form data. I would
suggest go ahead with #georgeawg's Answer if you are not Looking for a
formData + POST solution
Openstack also provides a different approach using FormData for uploading one or more files in a single go as mentioned in this documentation. Funny this was never visible in google search.
Here is a brief of it.
First you need to generate a signature similar to tempUrl signature using the following python procedure.
import hmac
from hashlib import sha1
from time import time
path = '/v1/my_account/container/object_prefix'
redirect = 'https://myserver.com/some-page'
max_file_size = 104857600
max_file_count = 1
expires = 1503124957
key = 'mySecretKey'
hmac_body = '%s\n%s\n%s\n%s\n%s' % (path, redirect,
max_file_size, max_file_count, expires)
signature = hmac.new(key, hmac_body, sha1).hexdigest()
Then in your javascript call post to the container like this.
var formData = new FormData();
formData.append("max_file_size", '104857600');
formData.append("max_file_count", '1');
formData.append("expires", '1503124957');
formData.append("signature", signature);
formData.append("redirect", redirect);
formData.append("file",fileObject);
$http.post(
"https://www.example.com/v1/my_account/container/object_prefix",
formData,
{
headers: {'Content-Type': undefined},
transformRequest: angular.identity
}
).then(function (res) {
console.log(response);
});
Points to note.
The formData in POST request should contain only these
parameters.
The file entry in the formData should be the last one.(Not sure why
it doesnt work the other way around).
The formData content like path with prefix, epoch time, max file
size, max file count and the redirection urls should be the same as
the one which were used to generate the signature. Otherwise you will
get a 401 Unauthorized.
I tried the FileReader to read the selected file as a binary string and wrote the content to the request body instead of FormData and the request which works fine for text files but not the binary files like XLSX or PDF The data is entirely corrupted this way.
The default operation for the $http service is to use Content-Type: application/json and to transform objects to JSON strings. For files from a FileList, the defaults need to be overridden:
var config = { headers: {'Content-Type': undefined} };
$http.put(url, fileList[0], config)
.then(function(response) {
console.log("Success");
}).catch(function(response) {
console.log("Error: ", response.status);
throw response;
});
By setting Content-Type: undefined, the XHR send method will automatically set the content type header appropriately.
Be aware that the base64 encoding of 'Content-Type': multipart/form-data adds 33% extra overhead. It is more efficient to send Blobs and File objects directly.
Sending binary data as binary strings, will corrupt the data because the XHR API converts strings from DOMSTRING (UTF-16) to UTF-8. Avoid binary strings as they are non-standard and obsolete.
I have a PDF file that I serve from a WebApi 2 application to an AngularJS client. I use file-saver to then save the file on the client as follows (in TypeScript):
this.$http.get(`${webUrl}api/pdf?id=${fileDto.id}`)
.then((response: ng.IHttpPromiseCallbackArg<any>) => {
var file = new Blob([response.data], { type: 'application/pdf' });
saveAs(file, 'my.pdf');
});
The reason I do this is so that I can use a bearer token to authorize access to the PDF (this is added via an interceptor). This works except for when the PDF file contains non UTF8 characters. In this latter case the file still downloads, but when I open it, it appears blank. Opening the file I can see that the non UTF8 characters are replaced with a □ character. In the JavaScript when I inspect the string value of response.data in the debugger I see that those characters are represented by �. Am I right in assuming that, since the file has been written from a string in JavaScript, no matter what I do I will not be able to correctly save a file with non UTF8 characters from JavaScript?
The � character is the Unicode Replacement Character \uFFFD which is inserted by the UTF-8 parser when it tries to parse illegal UTF-8.
PDF files are not UTF-8 strings; they are binary files.
To avoid the conversion from UTF-8 to DOMstring (UTF-16), set the config to responseType: 'blob':
var config = {responseType: 'blob'};
this.$http.get(`${webUrl}api/pdf?id=${fileDto.id}`, config)
.then((response: ng.IHttpPromiseCallbackArg<any>) => {
̶v̶a̶r̶ ̶f̶i̶l̶e̶ ̶=̶ ̶n̶e̶w̶ ̶B̶l̶o̶b̶(̶[̶r̶e̶s̶p̶o̶n̶s̶e̶.̶d̶a̶t̶a̶]̶,̶ ̶{̶ ̶t̶y̶p̶e̶:̶ ̶'̶a̶p̶p̶l̶i̶c̶a̶t̶i̶o̶n̶/̶p̶d̶f̶'̶ ̶}̶)̶;
var file = response.data;
saveAs(file, 'my.pdf');
});
For more information, see
MDN Web API Reference - XHR ResponseType
AngularJS $http Service API Reference
I'm having some trouble getting my angular application to parse my json-data correctly.
When my json-feed contains e.g. { "title": "Halldórsson Pourié" }
my application shows Halld�rsson Pouri�
I figure the problem is the charset, but i can't find where to change it.
Currently i am using ng-bind-html and using $sce.trustAsHtml(), and I'm very sure the problem occurs when $http.get(url) parses my json.
So how do i tell $http.get(url) to parse the data with a specific charset?
I had a similar issue and resolved it by using:
encodeURIComponent(JSON.stringify(p_Query))
where p_Query is a JSON containing the details of the request (i.e. your { "title": "Halldórsson Pourié" }).
EDIT:
You may also need to add to the header of your GET request the following:
'Content-Type': 'application/x-www-form-urlencoded ; charset=UTF-8'
Had same problem with accented characters and some scientific notation in text/JSON fields and found that AngularJS (or whatever native JavaScript XHR/fetch functions it is using) was flattening everything to UTF-8 no matter what else we tried.
The other respondent here claimed that UTF-8 should somehow still be accommodating your extended character set, but we found otherwise: taking samples of the source data directly, we loaded it into text editor with different encodings and UTF-8 would still squash extended characters to that � placeholder.
In our case, the encoding was ISO-8859-15 but you may be sufficient with ISO-8859-1.
Try adding this $http configuration to your $http.get() call, accordingly:
$http.get(url, {
responseType: 'arraybuffer',
transformResponse: function(data) {
let textDecoder = new TextDecoder('ISO-8859-15'); // your encoding may vary!
return JSON.parse(textDecoder.decode(data));
}
});
This should intercept AngularJS default response transformation and instead apply the function here. You will, of course, want to consider adding some error handling, etc.
If there are better ways to do this without a custom transformResponse function, I have yet to find anything in the AngularJS documentation.
$http.get(url, {responseType: 'arraybuffer', }).then(function (response) { var textDecoder = new TextDecoder('ISO-8859-1'); console.log(textDecoder.decode(response.data));});
This code will replace all your � placeholders into normal characters.
What is a correct way to uploading files via custom input forms in Yesod?
I developed a frontend in AngularJS, and I pass data to and from the server via JSON objects. However, this does not work when uploading files. For this reason I resorted to forms.
Gathering different resources I ended up with the following code. On the client side I use the following function:
this.submitNewMeas = function() {
var selected_file = document.getElementById('measurements').files[0];
var fd = new FormData();
fd.append("measurementsFile", selected_file);
var response = $http({
method: 'POST',
url: '/measurements/'+this.currMedia.id,
headers: {
'Content-Type': 'multipart/form-data'
},
data: {
fd
},
transformRequest: formDataObject
});
...
};
And on the server side I receive data as follows:
postMeasurementsR :: MediaId -> Handler Value
postMeasurementsR mediaId = do
result <- runInputPost $ iopt fileField "measurementsFile"
case result of
Just fileInfo -> do
saveMeas fileInfo "measDir"
saveMeas :: FileInfo -> FilePath -> HandlerT App IO (FilePath)
saveMeas file dest = do
let filename = unpack $ fileName file
dest = dest </> filename
liftIO $ fileMove file dest
return filename
However this causes a server error
20/Nov/2014:13:40:15 +0100 [Error#yesod-core] <<loop>> #(yesod-core-1.4.3:Yesod.Core.Class.Yesod ./Yesod/Core/Class/Yesod.hs:502:5)
Status: 500 Internal Server Error 54.70131s
If I don't try to save the data, then the file information is received, and I am able to see the file name on the server.
I've successfully managed to upload files using Yesod forms, but when it comes to custom forms all my attempts so far have failed.
Any help is greatly appreciated.
Thanks.
dest = dest </> filename
That's the source of your infinite loop: you're saying "dest is defined as itself plus filename." What's happening is that you're shadowing the original declaration of dest. A simple solution is to call the second declaration something like dest' (or better yet, something even more informative).