how to remove request header from HttpInputStream - file

I need help with a servlet.
I need to read a inputStream in one request and write a tiff file.
The inputStream come with request header and i dont know how remove that bytes and write only the file.
See initial bytes from the writen file.
-qF3PFkB8oQ-OnPe9HVzkqFtLeOnz7S5Be
Content-Disposition: form-data; name=""; filename=""
Content-Type: application/octet-stream; charset=ISO-8859-1
Content-Transfer-Encoding: binary
I want to remove that and write only bytes from the tiff file.
PS: sender of file its not me.

I'm not sure why you're not using HttpServletRequest's getInputStream() method to get the content without its headers, either way you have the option to start reading the input stream and ignoring the content until you find two consecutive CRLF's, which defines the end of the headers.
One way of doing that is like this:
String headers = new java.util.Scanner(inputStream).next("\\r\\n\\r\\n");
// Read rset of input stream

Apache commons solve 90% of your problems... only need know what keywords use in search :)
"parse multipart request"
and google say:
http://www.oreillynet.com/onjava/blog/2006/06/parsing_formdata_multiparts.html
int boundaryIndex = contentType.indexOf("boundary=");
byte[] boundary = (contentType.substring(boundaryIndex + 9)).getBytes();
ByteArrayInputStream input = new ByteArrayInputStream(buffer.getBytes());
MultipartStream multipartStream = new MultipartStream(input, boundary);
boolean nextPart = multipartStream.skipPreamble();
while(nextPart) {
String headers = multipartStream.readHeaders();
System.out.println("Headers: " + headers);
ByteArrayOutputStream data = new ByteArrayOutputStream();
multipartStream.readBodyData(data);
System.out.println(new String(data.toByteArray());
nextPart = multipartStream.readBoundary();
}

For me I use annotation and parameter like this:
#Consumes(MediaType.APPLICATION_OCTET_STREAM)
public Response testUpload(File uploadedInputStream)
And then I can read the file content with:
byte[] totalBytes = Files.readAllBytes(Paths.get(uploadedInputStream.toURI()));
Then I have to ignore the first 4 lines, also the end-of-content part, like this:
int headerLen = 0;
int index = 0;
while(totalBytes[index] != '\n' && index < totalBytes.length) {
headerLen++;
index++;
}
//ignore next three line
for (int i = 0; i < 3; i++) {
index++;
while (totalBytes[index] != '\n' && index < totalBytes.length) {
index++;
}
}
index++;
out.write(totalBytes, index, totalBytes.length - index - (headerLen+3));
out.flush();
out.close();

Related

End of Json string identification

Json string : "{\n\t"tag": "Value",\n\t"attributes": {\n\t\t"rfid": "2"\n\t},\n\t"dat": "1"\n}
I am receiving the Json string from web server part by part i.e 8bytes at a time
When i am trying to collect the data in one buffer with the below logic in C
static char *jsonString ;
bool found_json = false;
jsonString = (char*)malloc (1024, sizeof(char));
while(data[i] != "}")
{
found_json = true;
for( i = 0; i< len; i++)
{
memcpy(jsonString, data, len);
}
}
can someday throw some light that how to detect end of the string of Json as there will be two closing Json object
I think you have two proper ways: either fully parse the JSON (you can use some libraries for that) or somehow receive the length of the string (if this is a HTTP request then there should be Content-Length header which indicates the length). Things like messing with curly braces is not reliable because even a simple number like 1233 or boolean values like true are valid JSON strings.
Here is some pseudo code for finding the end of your string:
open = 0;
close = 0;
while ( visit each character in your buffer )
{
if (new character == '{')
open++;
else if (new character == '}')
close++;
if (open == 0) // Did we even start the JSON string yet?
discard character
else if (open == close) // Matching number of { and } found. We are done.
{
process JSON string
open = close = 0; // Prepare for next incoming JSON string.
}
}

How to send a gzip file as byte array in HTTP request in JMeter

I have a post request where I have to send a gzip file as byte array.
I have tried the following:
Created a BeanShell preprocessor to convert the file to byte array with the following function(got this function from here) :
import java.io.BufferedInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
FileInputStream in = new FileInputStream("C:\Users\New\Desktop\Load_Testing");
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
for (int i; (i = in.read(buffer)) != -1; ) {
bos.write(buffer, 0, i);
}
in.close();
byte[] postData = bos.toByteArray();
bos.close();
vars.put("postData", new String(postData));
Added the parameter postData in HTTP Request Sampler under Parameter's tab as ${postData} . Also tried adding the parameter ${postData} in HTTP Request Sampler under BodyData tab.
But the file is not sent.
I also tried sending the file by adding it under the tab "Send Files With the Request" section in HTTP request sampler with appropriate encoding.
But for all the cases I am getting the error :
:{"Result":"Error uploading data stream
The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.
at System.IO.Compression.GZipDecoder.ReadHeader(InputBuffer input)
at System.IO.Compression.Inflater.Decode()
at System.IO.Compression.Inflater.Inflate(Byte[] bytes, Int32 offset, Int32 length)
at System.IO.Compression.DeflateStream.Read(Byte[] array, Int32 offset, Int32 count)
at System.IO.Compression.GZipStream.Read(Byte[] array, Int32 offset, Int32 count)
at System.IO.Stream.ReadByte()
at ServiceData.CompressionHelper.UnGZip(Byte[] input)
First of all, I don't like your file path C:\Users\New\Desktop\Load_Testing. In Java and Beanshell you need to escape back slashes like C:\\Users\\New\\Desktop\\Load_Testing
Second: make sure that you're sending a correct gzip file. If Your C:\Users\New\Desktop\Load_Testing file is not in gzip format, you'll need to convert compress it in your Beanshell Pre-Processor as follows:
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.util.zip.GZIPOutputStream;
byte[] buffer = new byte[1024];
GZIPOutputStream gzos = new GZIPOutputStream(new FileOutputStream("C:/Users/New/Desktop/Load_Testing.gz"));
FileInputStream in = new FileInputStream("C:/Users/New/Desktop/Load_Testing");
int len;
while ((len = in.read(buffer)) > 0) {
gzos.write(buffer, 0, len);
}
in.close();
gzos.finish();
gzos.close();
And finally, make sure that you're sending correct Accept-Encoding header. To do so add a HTTP Header Manager and include gzip value in Accept-Encoding stanza.
Hope this helps.

read cloud storage content with "gzip" encoding for "application/octet-stream" type content

We're using "Google Cloud Storage Client Library" for app engine, with simply "GcsFileOptions.Builder.contentEncoding("gzip")" at file creation time, we got the following problem when reading the file:
com.google.appengine.tools.cloudstorage.NonRetriableException: java.lang.RuntimeException: com.google.appengine.tools.cloudstorage.SimpleGcsInputChannelImpl$1#1c07d21: Unexpected cause of ExecutionException
at com.google.appengine.tools.cloudstorage.RetryHelper.doRetry(RetryHelper.java:87)
at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:129)
at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:123)
at com.google.appengine.tools.cloudstorage.SimpleGcsInputChannelImpl.read(SimpleGcsInputChannelImpl.java:81)
...
Caused by: java.lang.RuntimeException: com.google.appengine.tools.cloudstorage.SimpleGcsInputChannelImpl$1#1c07d21: Unexpected cause of ExecutionException
at com.google.appengine.tools.cloudstorage.SimpleGcsInputChannelImpl$1.call(SimpleGcsInputChannelImpl.java:101)
at com.google.appengine.tools.cloudstorage.SimpleGcsInputChannelImpl$1.call(SimpleGcsInputChannelImpl.java:81)
at com.google.appengine.tools.cloudstorage.RetryHelper.doRetry(RetryHelper.java:75)
... 56 more
Caused by: java.lang.IllegalStateException: com.google.appengine.tools.cloudstorage.oauth.OauthRawGcsService$2#1d8c25d: got 46483 > wanted 19823
at com.google.common.base.Preconditions.checkState(Preconditions.java:177)
at com.google.appengine.tools.cloudstorage.oauth.OauthRawGcsService$2.wrap(OauthRawGcsService.java:418)
at com.google.appengine.tools.cloudstorage.oauth.OauthRawGcsService$2.wrap(OauthRawGcsService.java:398)
at com.google.appengine.api.utils.FutureWrapper.wrapAndCache(FutureWrapper.java:53)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
at com.google.appengine.tools.cloudstorage.SimpleGcsInputChannelImpl$1.call(SimpleGcsInputChannelImpl.java:86)
... 58 more
What else should be added to read files with "gzip" compression to be able to read the content in app engine? ( curl cloud storage URL from client side works fine for both compressed and uncompressed file )
This is the code that works for uncompressed object:
byte[] blobContent = new byte[0];
try
{
GcsFileMetadata metaData = gcsService.getMetadata(fileName);
int fileSize = (int) metaData.getLength();
final int chunkSize = BlobstoreService.MAX_BLOB_FETCH_SIZE;
LOG.info("content encoding: " + metaData.getOptions().getContentEncoding()); // "gzip" here
LOG.info("input size " + fileSize); // the size is obviously the compressed size!
for (long offset = 0; offset < fileSize;)
{
if (offset != 0)
{
LOG.info("Handling extra size for " + filePath + " at " + offset);
}
final int size = Math.min(chunkSize, fileSize);
ByteBuffer result = ByteBuffer.allocate(size);
GcsInputChannel readChannel = gcsService.openReadChannel(fileName, offset);
try
{
readChannel.read(result); <<<< here the exception was thrown
}
finally
{
......
It is now compressed by:
GcsFilename filename = new GcsFilename(bucketName, filePath);
GcsFileOptions.Builder builder = new GcsFileOptions.Builder().mimeType(image_type);
builder = builder.contentEncoding("gzip");
GcsOutputChannel writeChannel = gcsService.createOrReplace(filename, builder.build());
ByteArrayOutputStream byteStream = new ByteArrayOutputStream(blob_content.length);
try
{
GZIPOutputStream zipStream = new GZIPOutputStream(byteStream);
try
{
zipStream.write(blob_content);
}
finally
{
zipStream.close();
}
}
finally
{
byteStream.close();
}
byte[] compressedData = byteStream.toByteArray();
writeChannel.write(ByteBuffer.wrap(compressedData));
the blob_content is compressed from 46483 bytes to 19823 bytes.
I think it is the google code's bug
https://code.google.com/p/appengine-gcs-client/source/browse/trunk/java/src/main/java/com/google/appengine/tools/cloudstorage/oauth/OauthRawGcsService.java, L418:
Preconditions.checkState(content.length <= want, "%s: got %s > wanted %s", this, content.length, want);
the HTTPResponse has decoded the blob, so the Precondition is wrong here.
If I good understand you have to set mineType:
GcsFileOptions options = new GcsFileOptions.Builder().mimeType("text/html")
Google Cloud Storage does not compress or decompress objects:
https://developers.google.com/storage/docs/reference-headers?csw=1#contentencoding
I hope that's what you want to do .
Looking at your code it seems like there is a mismatch between what is stored and what is read. The documentation specifies that compression is not done for you (https://developers.google.com/storage/docs/reference-headers?csw=1#contentencoding). You will need to do the actual compression manually.
Also if you look at the implementation of the class that throws the exception (https://code.google.com/p/appengine-gcs-client/source/browse/trunk/java/src/main/java/com/google/appengine/tools/cloudstorage/oauth/OauthRawGcsService.java?r=81&spec=svn134) you will notice that you get the original contents back but you're actually expecting compressed content. Check the method readObjectAsync in the above mentioned class.
It looks like the content persisted might not be gzipped or the content-length is not set properly. What you should do is verify length of the compressed stream just before writing it into the channel. You should also verify that the content length is set correctly when doing the http request. It would be useful to see the actual http request headers and make sure that content length header matches the actual content length in the http response.
Also it looks like contentEncoding could be set incorrectly. Try using:.contentEncoding("Content-Encoding: gzip") as used in this TCK test. Although still the best thing to do is inspect the HTTP request and response. You can use wireshark to do that easily.
Also you need to make sure that GCSOutputChannel is closed as that's when the file is finalized.
Hope this puts you on the right track. To gzip your contents you can use java GZIPInputStream.
I'm seeing the same issue, easily reproducable by uploading a file with "gsutil cp -Z", then trying to open it with the following
ByteArrayOutputStream output = new ByteArrayOutputStream();
try (GcsInputChannel readChannel = svc.openReadChannel(filename, 0)) {
try (InputStream input = Channels.newInputStream(readChannel))
{
IOUtils.copy(input, output);
}
}
This causes an exception like this:
java.lang.IllegalStateException:
....oauth.OauthRawGcsService$2#1883798: got 64303 > wanted 4096
at ....Preconditions.checkState(Preconditions.java:199)
at ....oauth.OauthRawGcsService$2.wrap(OauthRawGcsService.java:519)
at ....oauth.OauthRawGcsService$2.wrap(OauthRawGcsService.java:499)
The only work around I've found is to read the entire file into memory using readChannel.read:
int fileSize = 64303;
ByteBuffer result = ByteBuffer.allocate(fileSize);
try (GcsInputChannel readChannel = gcs.openReadChannel(new GcsFilename("mybucket", "mygzippedfile.xml"), 0)) {
readChannel.read(result);
}
Unfortunately, this only works if the size of the bytebuffer is greater or equal to the uncompressed size of the file, which is not possible to get via the api.
I've also posted my comment to an issue registered with google: https://code.google.com/p/googleappengine/issues/detail?id=10445
This is my function for reading compressed gzip files
public byte[] getUpdate(String fileName) throws IOException
{
GcsFilename fileNameObj = new GcsFilename(defaultBucketName, fileName);
try (GcsInputChannel readChannel = gcsService.openReadChannel(fileNameObj, 0))
{
maxSizeBuffer.clear();
readChannel.read(maxSizeBuffer);
}
byte[] result = maxSizeBuffer.array();
return result;
}
The core is that you cannot use the size of the saved file cause Google Storage will give it to you with the original size, so it checks the sizes you expected and the real size and these are differents:
Preconditions.checkState(content.length <= want, "%s: got %s > wanted
%s", this, content.length, want);
So i solved it allocating the biggest amount possible for these files using BlobstoreService.MAX_BLOB_FETCH_SIZE. Actually maxSizeBuffer is only allocated once outsize of the function
ByteBuffer maxSizeBuffer = ByteBuffer.allocate(BlobstoreService.MAX_BLOB_FETCH_SIZE);
And with maxSizeBuffer.clear(); all data is flushed again.

JavaMail and non-ASCII character in filenames

I can send attachments that have non-ascii filenames in JavaMail but I am not able to download them. I am getting java.io.FileNotFoundException specifically for those attachments whose file names contain non-ascii characters.
FYI: I am using something like messageBodyPart.setFileName(MimeUtility.encodeText(filename[i])) to encode the text and MimeUtility.decodeText(bodyPart.getFileName()) to decode the non-ascii file names
Is there a workaround for this?
EDIT
#Bill, here is part of my code that reads attachments. I have also added the properties.setProperty("mail.mime.decodeparameters", "true") and properties.setProperty("mail.mime.decodefilename", "true") properties in my code.
if (message[a].getContent() instanceof MimeMultipart) {
Multipart multipart = (Multipart) message[a].getContent();
for (int i = 0; i < multipart.getCount(); i++) {
bodyPart = multipart.getBodyPart(i);
disposition = bodyPart.getDisposition();
if (disposition != null && (disposition.equals(BodyPart.ATTACHMENT) || (disposition.equals(BodyPart.INLINE)))) {
DataHandler handler = bodyPart.getDataHandler();
String path = bodyPart.getFileName();
String[] str = path.split("/");
String fileName = str[str.length - 1];
String filePath = ReadConfigPropertiesFile.getPropertyValue("server.buildpath");
System.out.println(fileName);
File tempDir = new File(filePath + user);
if (!tempDir.exists()) {
tempDir.mkdir();
}
File saveFile = new File(tempDir + "/" + fileName);
int count = 0;
while (saveFile.exists()) {
count++;
saveFile = new File(tempDir + "/" + count + "_" + fileName);
}
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(saveFile));
byte[] buff = new byte[2048];
InputStream is = bodyPart.getInputStream();
int ret = 0;
while ((ret = is.read(buff)) > 0) {
bos.write(buff, 0, ret);
}
bos.close();
is.close();
//System.out.println(bodyPart.getContentType());
}else {
//display body (message) of the attachment;
//System.out.println(bodyPart.getContent().toString());
}
}
}
The above code raises the FileNotFoundException exception at BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(saveFile)) line and this is getting raised for the attachments whose file names are non-ascii characters (something like ሰላም.pdf). Every thing else works fine.
This answer taken from comment of #semytech (OP). It was hard to find it there, so I will add it as answer for more visibility. It helped me with hebrew filenames.
MimeBodyPart attachment = new MimeBodyPart();
attachment.setFileName(MimeUtility.encodeText(filename, "UTF-8", null));
You should never need to do the encoding or decoding yourself.
There are two sets of properties you can set to tell JavaMail to do the encoding/decoding for you:
mail.mime.encodefilename/mail.mime.decodefilename
mail.mime.encodeparameters/mail.mime.decodeparameters
See the javadocs for the javax.mail.internet package for details.
The first set uses a non-standard encoding technique, similar to what you're doing yourself. This works fine with some older mailers that use this technique.
The second set uses a MIME standard encoding technique. This version works with most modern mailers.
None of this explains why you're getting FileNotFoundException, but then you didn't provide enough detail to know what you're doing when you get the exception.

javamail: why is text/plain attachment matter returned by getContent, and not from InputStream?

In trying to download mail with attachments from gmail, my test mail includes a text file as attachment.
The attachment part returns Text attachment content-type, and even the filename correctly. But the loop condition over attachment InputStream is never non-zero.
After a bit of trial and error it turned out that the content for text/plain is available using the getContent method for the part (in the case below introducing the call
att_mbp.getContent()
returned the content in the attached text file)
if (BodyPart.ATTACHMENT.equalsIgnoreCase(att_mbp.getDisposition())) {
att_mbp.getContentType();
// process each attachment
// read the filename
file = att_mbp.getFileName();
InputStream stream = att_mbp.getInputStream();
BufferedInputStream br = new BufferedInputStream(stream);
BufferedOutputStream bout = new BufferedOutputStream(new FileOutputStream(file));
while (br.available() > 0) {
// this loop is never executed for text/plain
bout.write(br.read());
}
bout.flush();
bout.close();
}
My question is - Why is the text/plain attachment body only available from getContent(), and not from the attached InputStream instance too?
Ok. I finally figured it out.
The call to available () always returns 0.
The code worked when I modified it as follows
int dataByte;
while( ( dataByte = br.read() ) > 0 ){
bout.write( dataByte );
}
According to javadoc, descendants of InputStream should override available. It looks like this is not the case here.

Resources