I have a pdf uploaded in a Azure Blobstorage and I'm facing some problems during the download routine.
My application runs with springboot and I use an OutputStream provided by a HttpServletResponse in a #RestController method to stream the bytes from the blobstorage to the request.
In the frontend I have a Reac application receiving the information and executing the download through the browser.
Every time I stream the bytes to the frontend I got a corrupted file. It works just fine when I execute a request through Insomnia or Postman.
I tried to compare the files as texts and I could see some differences between them.
Differences
The size of the corrupted file is almost double the size of the original
When I opened files on Notepad++ they seems to be in a different encoding
corrupted
consistent
It looks like there are some characters bad interpreted
corrupted
consistent
My frontend uses FileSaver #2.0.2 to persist the file on disk
const blobParts = [];
const blobOptions = {
type: axiosResponse.headers['content-type'],
};
blobParts.push(axiosResponse.data);
const file = new File(
blobParts,
axiosResponse.headers['content-disposition'].split('=')[1],
blobOptions,
);
return FileSaver.saveAs(file);
I'm wondering if there's a way of keep the ANSI encoding through the persistence process or if there's is a way
Related
My robot has a web-based Lua IDE, and now I send the script contents with a GET request, Base64 Encoded. But for some reason, my server sometimes only decodes a part of the file, and sometimes it works just fine with a way longer file. I've written the server myself in C++ (it's on GitHub
), and it uses libmicrohttpd
Is there a more reliable way of sending files through a HTTP request than the method I'm using right now?
I am using Silverlight multi file uploader and Uploading the document in Azure Blob as Byte Array.
//Append the memory stream into ByteArray
using (MemoryStream ms = new MemoryStream())
{
stream.CopyTo(ms);
return ms.ToArray();
}
// Upload the file
blob.UploadByteArray(bytes);
Upload document appears to be corrupt Intermittently.
Any Suggestions?
The Windows Azure Storage Client Library protects the integrity of the blob being uploaded by verifying an MD5 hash of the data when it is sent to the Windows Azure Storage Service (in most cases). If you are using an HTTPS connection to the service, this would also verify the data was sent without errors.
The details of how MD5 hashes are used: http://blogs.msdn.com/b/windowsazurestorage/archive/2011/02/18/windows-azure-blob-md5-overview.aspx
I believe the corruption you are seeing occurred between the client's web browser and your application. You will need to have the user try their upload again.
By the way, your code creates two additional copies of the data unnecessarily (MemoryStream and byte array). Instead, try this:
blob.UploadFromStream(stream);
I'm trying to upload files to Google Drive with ProgressListener and ChunkSize enabled (thus with DirectUploadEnabled disabled). This way i have a more reliable upload and the possibility for a progress indication to the user.
I transfer the files from the GWT website to the GAE with a FormPanel and a FileUploadField which POSTS the file to GAE on submit(). On the GAE i receive the file with an UploadServlet which uses org.apache.commons.fileupload to receive the documents as a stream. I don't want to receive the complete documents on the GAE because the documents are to big. Therefore i start the upload (insert) to Google Drive with the received stream from the incoming request.
Now there's a problem; for the insert i need to know the size of the stream;
int lContentLength = getRequest().getContentLength();
FileItemStream lFileItemStream = getFileItemStream();
InputStream lInputStream = lFileItemStream.openStream();
BufferedInputStream lBufferedInputStream = new BufferedInputStream(lInputStream);
InputStreamContent lInputStreamContent = new InputStreamContent(pContentType, lBufferedInputStream);
lInputStreamContent.setLength(lContentLength);
My first guess was the ContentLengt from the incoming Servlet request. But this is not correct because this concerns the complete request (which also contains other fields which are used as parameters). Without the Drive option DirectUploadEnabled i need the exact stream size from the uploaded document, otherwise the upload stall's at the end...
How do i grap this document size? The Google example is stupid because it uses a local file;
https://code.google.com/p/google-api-java-client/wiki/MediaUpload
Yes from a local file it is easy to get the file size (mediaFile.length()). But from a website ... Several sites specify it is not possible to grab the file size before submit() from the website, and it seems also impossible to determine the stream-size on GAE without loading the complete file...
How do i determine this streamsize? Is there another solution for this problem?
Objective: Suppose the client submits a string or text file to the server (Google App Engine) using a web form. I want the server to modify the original file and serve it back to the client.
I think the only way to serve files from GAE is using the Blobstore, right? Then, as we cannot modify blobs, I believe a solution would be:
Client uploads a file using HttpRequest
Server reads the uploaded file and copies it to a temp buffer (not sure if is there a method to do this)
Server deletes original blob
Server modifies data in the temp buffer
Server writes the modified buffer to the Blobstore
Server serves the new blob to the client
Would this work? Could you think about any other solution?
Thanks
I think the only way to serve files from GAE is using the Blobstore, right?
Wrong. A 'file' is just a way of storing data on disk; there's nothing about serving them from a webserver that requires the data come from an actual, writable disk file. You can simply accept the user's data via a form upload, modify it, and serve it back to them, without it having to ever touch disk, the blobstore, or any other permanent storage medium.
This only becomes a problem if the user's data is too large to fit in memory, in which case you will have to store the data somewhere while you work on it, such as in the blobstore.
http://code.google.com/appengine/kb/java.html#fileforms
shows you how to do it for file upload, which has to be performed thro multipart form-data.
Similarly for non-file data, where you read straight from the request stream.
You don't even have to store the file/input stream. Just spit out the processed data into the output response stream, while reading the input FileItemStream or request inputstream.
If your file/input processing requires look-forward, determine the maximum distance of look-forward and use that distance as your buffer size.
Further Edits
To respond to the client with a file type, set the response content-type or mime-type.
e.g., I've had apps which dynamically generated gifs, jpgs, xls, cvs, etc.
There isn't any difference whether source of response stream is a file you read or a stream that you generate dynamically. Because, even if you had a stored file that needs to be sent as response to client, you could still have to convert it into a response stream and flag the content-type appropriately.
For dynamically generated content, unless you need to cache the output, you need not generate the file into a web URL-visible location and then generate a new html page with the link, and send that html page to the browser. You don't need the user's browser to have to refresh itself just to get that link.
You would simply send the "file" directly with the response stream. You could design your GWT client to accept the "file", perhaps in a named frame, where the named frame src url is the app that performs the dynamic generation of the file.
Read http://en.wikipedia.org/wiki/Mime-type to find the content-type you need.
If the target client's browser does not have the content handler set-up for the response's content-type, it would as for a treatment or be treated as a file download.
I had frequently used jsp or jspx to generate dynamically generated charts or spreadsheets. No stored files involved. The response is written to while the request is being read. Let's look at the jsp page directive to set the content-type to invoke MS Excel on a CSV.
<%# page language="java" contentType="application/vnd-ms-excel; charset=UTF-8"
pageEncoding="UTF-8"%>
For a servlet, ServletResponse.setContentType(String)
is method to set the content-type.
I'm building a gwt app that stores the text of random webpages in a datastore text field. Often the text is formatted UTF-8. All the files of my app are stored as UTF-8 and when I run the application on my local machine the entire process works fine. UTF-8 text is stored as such and retrievable ftom the local version of the app engine as UTF-8. However when I deploy the app to the google app engine somewhere between when I store the text and when I retrieve it it is no longer UTF-8 which causes non-ascii characters to be displayed as ?.
When I view the datastore in the appengine control panel all the special characters appear as ? which leads me to believe that it is a problem when writing to the database.
Does anyone know how to fix this?
The app itself is a little big.
Here's some pseudocode:
Text webPageText = new Text(<STRING THAT CONTAINS UNICODE CHARACTERS>);
/*Some Code to store Text object on datastore
Specifically I'm using javax.jdo.PersistenceManager to do this.
Some Code to retrieve text from datastore. */
String retrievedText = webPageText.getValue();
The problem is that retrievedText comes back with ? instead of unicode characters.
Here's a similar problem in python that I found: Trying to store Utf-8 data in datastore getting UnicodeEncodeError. Though my app is not getting any errors.
Unfortunately I think Java strings are default utf-8 and I can't find any code that will let me declare them explicitly as utf-8.
Edit: I've now built a small webapp that takes in unicode text and stores it in the datastore and then retrieves it with no problems. I still have no idea where the problem is in my original source code but I'm going to change the way my code handles webpage retrieval to match the smaller app that I just built. Thank you everyone for your help.
Fixed same issue by setting both request and response encoding to utf-8.
Request encoding results in valid string stored in datastore, without it values will be stored as "????..."
Requests: if you use Apache HTTP Client, this is done in the following way:
Get request:
NameValuePair... params;
...
String url = urlBase + URLEncodedUtils.format(Arrays.asList(params), "UTF-8");
HttpGet httpGet = new HttpGet(url);
Post request:
NameValuePair... params;
...
HttpPost httpPost = new HttpPost(url);
httpPost.setEntity(new UrlEncodedFormEntity(Arrays.asList(params), "UTF-8"));
Response: if you build your response in HttpServlet, this is done in a following way:
HttpServletResponse resp;
...
resp.setContentType("text/html; charset=utf-8");
I tried to convert String to ByteArray and then store it as datastore blob.
//Save String as Blob
Blob webPageText = new Blob(<STRING THAT CONTAINS UNICODE CHARACTERS>.getBytes());
//Retrieve Blob as String
String retrievedText = new String(webPageText.getBytes());
I originally thought this had solved the problem but I had by mistake only tested it on my local server. This code still returns ? instead of unicode characters which leads me to believe that the problem isn't in the datastore but in the transfer from the app engine to the client.
Encoding Solution: Cause Browser use "8859_1" charset
=> Before
Save Datastore, I convert charset.
new String(req.getParameter("title").getBytes("8859_1"),"utf-8")
When I ran this application on my local machine, it was fine. But when I deployed, I faced the same issue you saw. I solved this problem by:
After
=> Save Datastore Code.
new String(req.getParameter("title").getBytes("utf-8"),"utf-8")
These links may prove useful, afterall:
How to set Google App Engine java Content-Type to UTF-8
http://code.google.com/appengine/docs/python/tools/webapp/buildingtheresponse.html