javamail throws java.io.UnsupportedEncodingException: unknown-8bit - jakarta-mail

There were some emails that I try to read using javamail lib. When the email contains the MIME header (Content-Type: text/plain; charset="unknown-8bit"), I get this error: java.io.UnsupportedEncodingException: unknown-8bit
Any ideas why is this happening?

Because "unknown-8bit" is not a known charset name. This is explained in the JavaMail FAQ, along with alternatives for handling this problem. I've copied the answer here but note that this may become out of date. Please be sure to search the JavaMail FAQ for any other JavaMail problems you might have.
Q: Why do I get the UnsupportedEncodingException when I invoke getContent() on a bodypart that contains text data?
A: Textual bodyparts (i.e., bodyparts whose type is "text/plain", "text/html", or "text/xml") return Unicode String objects when getContent() is used. Typically, such bodyparts internally hold their textual data in some non Unicode charset. JavaMail (through the corresponding DataContentHandler) attempts to convert that data into a Unicode string. The underlying JDK's charset converters are used to do this. If the JDK does not support a particular charset, then the UnsupportedEncodingException is thrown. In this case, you can use the getInputStream() method to retrieve the content as a stream of bytes. For example:
String s;
if (part.isMimeType("text/plain")) {
try {
s = part.getContent();
} catch (UnsupportedEncodingException uex) {
InputStream is = part.getInputStream();
/*
* Read the input stream into a byte array.
* Choose a charset in some heuristic manner, use
* that charset in the java.lang.String constructor
* to convert the byte array into a String.
*/
s = convert_to_string(is);
} catch (Exception ex) {
// Handle other exceptions appropriately
}
}
There are some commonly used charsets that the JDK does not yet support. You can find support for some of these additional charsets in the JCharset package at http://www.freeutils.net/source/jcharset/.
You can also add an alias for an existing charset already supported by the JDK so that it will be known by an additional name. You can create a charset provider for the "bad" charset name that simply redirects to an existing charset provider; see the following code. Create an appropriate CharsetProvider subclass and include it along with the META-INF/services file and the JDK will find it. Obviously you could get significantly more clever and redirect all unknown charsets to "us-ascii", for instance.
==> UnknownCharsetProvider.java <==
import java.nio.charset.*;
import java.nio.charset.spi.*;
import java.util.*;
public class UnknownCharsetProvider extends CharsetProvider {
private static final String badCharset = "x-unknown";
private static final String goodCharset = "iso-8859-1";
public Charset charsetForName(String charset) {
if (charset.equalsIgnoreCase(badCharset))
return Charset.forName(goodCharset);
return null;
}
public Iterator<Charset> charsets() {
return Collections.emptyIterator();
}
}
==> META-INF/services/java.nio.charset.spi.CharsetProvider <==
UnknownCharsetProvider

Related

Cant use InternetAddress with umlauts anymore after switching to com.sun.mail:javax.mail

we recently switched from javax.mail:mail to com.sun.mail:javax.mail.
Since then the following code fails:
new InternetAddress("chr#möllers.de", false).validate();
Caught: javax.mail.internet.AddressException: Domain contains control or whitespace in string ``chr#möllers.de''
javax.mail.internet.AddressException: Domain contains control or whitespace in string ``chr#möllers.de''
The implementation of InternetAddress#validate() has obviously changed. A few additional lines concerning CRLF checks are followed by this snippet:
else if (c <= 040 || c >= 0177) {
throw new AddressException(
"Domain contains control or whitespace", addr);
Every char >= 177 is treated as control or whitespace - which is wrong, e.g. for umlauts (ö = 246).
So the exception message is misleading.
Did the change of validate() introduce a bug?
By now, Internet email addresses may contain umlauts encoded in punycode. Thats why i expected to be safe passing a string with umlauts.
Is InternetAddress intended to be used with an encoded String in this case?
Thanks in advance
Update to Bill Shannons answer
The nicely formatted Groovy script mentioned in my comment:
#GrabResolver(name='snapshots', root='https://maven.java.net/content/repositories/snapshots/', m2Compatible='true')
#Grab("com.sun.mail:javax.mail:1.6.0-SNAPSHOT")
import javax.mail.internet.InternetAddress
new InternetAddress("chr#möllers.de", false)
Update: test with latest snapshot
import org.junit.Test;
import javax.mail.internet.AddressException;
import javax.mail.internet.InternetAddress;
public class ValidateEmailTest {
#Test
public void test() throws AddressException {
new InternetAddress("chr#möllers.de", true).validate();
new InternetAddress("chr#möllers.de", false).validate();
}
}
The test runs successfully (not throwing an AddressException) with the latest snapshot 1.6.0-SNAPSHOT that is currently from Tue Feb 21.
Use of non-ASCII characters in domain names requires support for RFC 6530, RFC 6531, and RFC 6532. Both the client and the server need to support these new standards. I've added such support to JavaMail 1.6; you can download a development SNAPSHOT release as described on the JavaMail web page. You need to ensure that the server supports the SMTPUTF8 extension, and you need to set the Session property mail.mime.allowutf8 to true.
If you're able to test this new support for internationalized email addresses using a real mail server, please let me know your experiences, good or bad, at javamail_ww#oracle.com. Thanks.

Camel doesn't retrieve SQS messages attributes

Here is the route:
from("aws-sqs://myQueue?accessKey=RAW(xxx)&secretKey=RAW(yyy)&deleteAfterRead=false")
.log("Attributes: ${header.CamelAwsSqsAttributes}")
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
Map<String, String> messageAttributes = (Map<String, String>) exchange.getIn().getHeader("CamelAwsSqsAttributes");
...
}
});
The .log() shows an empty map as well as if I print messageAttributes from the processor.
I also tried with the header "CamelAwsSqsMessageAttributes" instead of "CamelAwsSqsAttributes" but still nothing.
I see the attributes from the AWS console though.
By the way I get the message body, and I use Camel 2.15
I figured it out, here is an example to get queue attributes and message attributes:
main.bind("sqsAttributeNames", Collections.singletonList("All"));
main.bind("sqsMessageAttributeNames", Collections.singletonList("All"));
Or add those objects to the registry if you don't use org.apache.camel.main.Main
Then:
from("aws-sqs://myQueue?accessKey=RAW(xxx)&secretKey=RAW(yyy)&deleteAfterRead=false&attributeNames=#sqsAttributeNames&messageAttributeNames=#sqsMessageAttributeNames")
Of course you can replace Collections.singletonList("All") with the list of attributes you need if you don't want all of them.
I faced the same issue. When I am using camel-aws 2.16.x and I have my endpoint configured as follow
from("aws-sqs://myQueue?...&messageAttributeNames=#sqsMsgAttributeNames")
.to(...)
Then I have defined a Collection of String in my spring configuration file
#Bean
public Collection<String> sqsMsgAttributeNames() {
return Arrays.asList("Attr1", "Attr2");
}
Above settings work fine but ever since I upgraded to camel-aws 2.17.3. It no longer works. As mentioned in Camel SQS Component, collection of string no longer will be supported for messageAttributeNames and it should be a String with attributes separated by comma.
Note: The string containing attributes should not contain any white
spaces otherwise camel-aws component will only read the first
attribute. I went through the pain to debug on this. Besides, setting the
attribute value to be "All" does not work for me, none of the message
attributes will be read.
Below is the changes I made that allowed camel-aws's SqsConsumer to work again:
#Bean
public String sqsMsgAttributeNames() {
return String.format("%s,%s", "Attr1", "Attr2");
}
It is not an issue of Camel. It can be the default behavior of SQS or aws-java-sdk-core library.
As a quick solution this aws-sqs URL can be used
aws-sqs://myQueue?<other attributes here>&attributeNames=All
Keep in mind that localstack can work well without attributeNames parameter, unlike SQS.

Wildcards in GCS bucket Java client api

Using Wildcards in file name i am trying to read files from GCS bucket.
in gsutil command line wildcards is working in specifying file names.
but in java client api
GcsFilename filename = new GcsFilename(BUCKETNAME, "big*");
it is searching for file named "big*" instead of file starting with big .
please help me how i can use Wildcards in GCSFilename.
Thanks in advance.
Wildcard characters are a feature of gsutil, but they're not an inherent part of the Google Cloud Storage API. You can, however, handle this the same way that gsutil does.
If you want to find the name of every object that begins with a certain prefix, Google Cloud Storage's APIs provide a list method with a "prefix" argument. Only objects matching the prefix will be returned. This doesn't work for arbitrary regular expressions, but it will work for your example.
The documentation for the list method goes into more detail.
As Brandon Yarbrough mentioned, GcsFilename represent a name of a single GCS Object, which could include any valid UTF-8 character [excluding a few such as \r \n but including '*' though
not recommended). see https://developers.google.com/storage/docs/bucketnaming#objectnames for more info.
GAE GCS client does not support listing yet (though that is planned to be added), so for now you can use the GCS XML or JSON API directly (using urlfetch) or use the Java GCS api client, https://developers.google.com/api-client-library/java/apis/storage/v1
See example for the latter option:
public class ListServlet extends HttpServlet {
public static final List<String> OAUTH_SCOPES =
ImmutableList.of("https://www.googleapis.com/auth/devstorage.read_write");
#Override
protected void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
try {
String bucket = req.getParameter("bucket");
AppIdentityCredential cred = new AppIdentityCredential(OAUTH_SCOPES);
Storage storage = new Storage.Builder(new UrlFetchTransport(), new JacksonFactory(), cred)
.setApplicationName(SystemProperty.applicationId.get()).build();
Objects.List list = storage.objects().list(bucket);
for (StorageObject o : list.execute().getItems()) {
resp.getWriter().println(o.getName() + " -> " + o);
}
} catch (Exception ex) {
throw new ServletException(ex);
}
}
}

Handling multipart attachments in CXF APIs

I am trying to develop an API call using Apache CXF that takes in an attachment along with the request. I followed this tutorial and this is what I have got so far.
#POST
#Path("/upload")
#RequireAuthentication(false)
public Response uploadWadl(MultipartBody multipartBody){
List<Attachment> attachments = multipartBody.getAllAttachments();
DataHandler dataHandler = attachments.get(0).getDataHandler();
try {
InputStream is = dataHandler.getInputStream();
} catch (IOException e) {
e.printStackTrace();
}
return Response("OK");
}
I am getting an InputStream object to the attachment and everything is working fine. However I need to pass the attachment as a java.io.File object to another function. I know I can create a file here, read from the inputstream and write to it. But is there a better solution? Has the CXF already stored it as a File? If so I could just go ahead and use that. Any suggestions?
I'm also interested on this matter. While discussing with Sergey on the CXF mailing list, I learned that CXF is using a temporary file if the attachment is over a certain threshold.
In the process I discovered this blogpost that explains how to use CXF attachment safely.
You can be interested by the exemple on this page as well.
That's all I can say at the moment as I'm investigating right now, I hope that helps.
EDIT : At the moment here's how we handle attachment with CXF 2.6.x. About uploading a file using multipart content type.
In our REST resource we have defined the following method :
#POST
#Produces(MediaType.APPLICATION_JSON)
#Consumes(MediaType.MULTIPART_FORM_DATA)
#Path("/")
public Response archive(
#Multipart(value = "title", required = false) String title,
#Multipart(value = "hash", required = false) #Hash(optional = true) String hash,
#Multipart(value = "file") #NotNull Attachment attachment) {
...
IncomingFile incomingFile = attachment.getObject(IncomingFile.class);
...
}
A few notes on that snippet :
#Multipart is not standard to JAXRS, it's not even in JAXRS 2, it's part of CXF.
In our code we have implemented bean validation (you have to do it yourself in JAXRS 1)
You don't have to use a MultipartBody, the key here is to use an argument of type Attachment
So yes as far as we know there is not yet a possibility to get directly the type we want in the method signature. So for example if you just want the InputStream of the attachment you cannot put it in the signature of the method. You have to use the org.apache.cxf.jaxrs.ext.multipart.Attachment type and write the following statement :
InputStream inputStream = attachment.getObject(InputStream.class);
Also we discovered with the help of Sergey Beryozkin that we could transform or wrap this InputStream, that's why in the above snippet we wrote :
IncomingFile incomingFile = attachment.getObject(IncomingFile.class);
IncomingFile is our custom wrapper around the InputStream, for that you have to register a MessageBodyReader, ParamHandler won't help as they don't work with streams but with String.
#Component
#Provider
#Consumes
public class IncomingFileAttachmentProvider implements MessageBodyReader<IncomingFile> {
#Override
public boolean isReadable(Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) {
return type != null && type.isAssignableFrom(IncomingFile.class);
}
#Override
public IncomingFile readFrom(Class<IncomingFile> type,
Type genericType,
Annotation[] annotations,
MediaType mediaType,
MultivaluedMap<String, String> httpHeaders,
InputStream entityStream
) throws IOException, WebApplicationException {
return createIncomingFile(entityStream, fixedContentHeaders(httpHeaders)); // the code that will return an IncomingFile
}
}
Note however that there have been a few trials to understand what was passed, how, and the way to hot-fix bugs (For example the first letter of the first header of the attachment part was eat so you had ontent-Type instead of Content-Type).
Of course the entityStream represents the actual InputStream of the attachment. This stream will read data either from memory or from disk, depending on where CXF put the data ; there is a size threshold property (attachment-memory-threshold) for that matter. You can also say where the temporary attachments will go (attachment-directory).
Just don't forget to close the stream when you are done (some tool do it for you).
Once everything was configured we tested it with Rest-Assured from Johan Haleby. (Some code are part of our test utils though) :
given().log().all()
.multiPart("title", "the.title")
.multiPart("file", file.getName(), file.getBytes(), file.getMimeType())
.expect().log().all()
.statusCode(200)
.body("store_event_id", equalTo("1111111111"))
.when()
.post(host().base().endWith("/store").toStringUrl());
Or if you need to upload the file via curl in such a way :
curl --trace -v -k -f
--header "Authorization: Bearer b46704ff-fd1d-4225-9dd4-e29065532b73"
--header "Content-Type: multipart/form-data"
--form "hash={SHA256}3e954efb149aeaa99e321ffe6fd581f84d5a497b6fab5c86e0d5ab20201f7eb5"
--form "title=fantastic-video.mp4"
--form "archive=#/the/path/to/the/file/fantastic-video.mp4;type=video/mp4"
-X POST http://localhost:8080/api/video/event/store
To finish this answer, I'd like to mention it is possible to have JSON payload in multipart, for that you can use an Attachment type in the signature and then write
Book book = attachment.getObject(Book.class)
Or you can write an argument like :
#Multipart(value="book", type="application/json") Book book
Just don't forget to add the Content-Type header to the relevant part when performing the request.
It might be worth to say that it is possible to have all the parts in a list, just write a method with a single argument of type List<Attachment>. However I prefer to have the actual arguments in the method signature as it's cleaner and less boilerplate.
#POST
void takeAllParts(List<Attachment> attachments)

Display dynamic image from database or remote source with p:graphicImage and StreamedContent

I'm trying to display image bytes which is saved in database as a StreamedContent in the <p:graphicImage> as follows:
<p:graphicImage value="#{item.imageF}" width="50" id="grpImage" height="80"/>
private StreamedContent content; // getter and setter
public StreamedContent getImageF() {
if (student.getImage() != null) {
InputStream is = new ByteArrayInputStream(student.getImage());
System.out.println("Byte :"+student.getImage());
content = new DefaultStreamedContent(is, "", student.getStuID());
System.out.println("ddd ------------------------------- " + content);
return content;
}
return content;
}
This returns a blank image. How is this caused and how can I solve it?
The stdout prints the following:
INFO: Byte :[B#a2fb48
INFO: ddd ------------------------------- org.primefaces.model.DefaultStreamedContent#b0887b
INFO: Byte :[B#a2fb48
INFO: ddd ------------------------------- org.primefaces.model.DefaultStreamedContent#1d06a92
INFO: Byte :[B#d52f0b
INFO: ddd ------------------------------- org.primefaces.model.DefaultStreamedContent#39a60
INFO: Byte :[B#d52f0b
INFO: ddd ------------------------------- org.primefaces.model.DefaultStreamedContent#8c3daa
INFO: Byte :[B#124728a
INFO: ddd ------------------------------- org.primefaces.model.DefaultStreamedContent#1dbe05b
INFO: Byte :[B#124728a
INFO: ddd ------------------------------- org.primefaces.model.DefaultStreamedContent#66a266
INFO: Byte :[B#a2fb48
INFO: ddd ------------------------------- org.primefaces.model.DefaultStreamedContent#1293976
INFO: Byte :[B#a2fb48
INFO: ddd ------------------------------- org.primefaces.model.DefaultStreamedContent#17b7399
INFO: Byte :[B#d52f0b
INFO: ddd ------------------------------- org.primefaces.model.DefaultStreamedContent#1e245a5
INFO: Byte :[B#d52f0b
INFO: ddd ------------------------------- org.primefaces.model.DefaultStreamedContent#4a7153
INFO: Byte :[B#124728a
INFO: ddd ------------------------------- org.primefaces.model.DefaultStreamedContent#1561bfd
INFO: Byte :[B#124728a
INFO: ddd ------------------------------- org.primefaces.model.DefaultStreamedContent#47a8c2
The <p:graphicImage> requires a special getter method. It will namely be invoked twice per generated image, each in a completely different HTTP request.
The first HTTP request, which has requested the HTML result of a JSF page, will invoke the getter for the first time in order to generate the HTML <img> element with the right unique and auto-generated URL in the src attribute which contains information about which bean and getter exactly should be invoked whenever the webbrowser is about to request the image. Note that the getter does at this moment not need to return the image's contents. It would not be used in any way as that's not how HTML works (images are not "inlined" in HTML output, but they are instead requested separately).
Once the webbrowser retrieves the HTML result as HTTP response, it will parse the HTML source in order to present the result visually to the enduser. Once the webbrowser encounters an <img> element during parsing the HTML source, then it will send a brand new HTTP request on the URL as specified in its src attribute in order to download the content of that image and embed it in the visual presentation. This will invoke the getter method for the second time which in turn should return the actual image content.
In your particular case PrimeFaces was apparently either unable to identify and invoke the getter in order to retrieve the actual image content, or the getter didn't return the expected image content. The usage of #{item} variable name and the lot of calls in the log suggests that you were using it in an <ui:repeat> or a <h:dataTable>. Most likely the backing bean is request scoped and the datamodel isn't properly preserved during the request for the image and JSF won't be able to invoke the getter during the right iteration round. A view scoped bean would also not work as the JSF view state is nowhere available when the browser actually requests the image.
To solve this problem, your best bet is to rewrite the getter method as such so that it can be invoked on a per-request basis wherein you pass the unique image identifier as a <f:param> instead of relying on some backing bean properties which may go "out of sync" during subsequent HTTP requests. It would make completely sense to use a separate application scoped managed bean for this which doesn't have any state. Moreover, an InputStream can be read only once, not multiple times.
In other words: never declare StreamedContent nor any InputStream or even UploadedFile as a bean property; only create it brand-new in the getter of a stateless #ApplicationScoped bean when the webbrowser actually requests the image content.
E.g.
<p:dataTable value="#{bean.students}" var="student">
<p:column>
<p:graphicImage value="#{studentImages.image}">
<f:param name="studentId" value="#{student.id}" />
</p:graphicImage>
</p:column>
</p:dataTable>
Where the StudentImages backing bean can look like this:
#Named // Or #ManagedBean
#ApplicationScoped
public class StudentImages {
#EJB
private StudentService service;
public StreamedContent getImage() throws IOException {
FacesContext context = FacesContext.getCurrentInstance();
if (context.getCurrentPhaseId() == PhaseId.RENDER_RESPONSE) {
// So, we're rendering the HTML. Return a stub StreamedContent so that it will generate right URL.
return new DefaultStreamedContent();
}
else {
// So, browser is requesting the image. Return a real StreamedContent with the image bytes.
String studentId = context.getExternalContext().getRequestParameterMap().get("studentId");
Student student = studentService.find(Long.valueOf(studentId));
return new DefaultStreamedContent(new ByteArrayInputStream(student.getImage()));
}
}
}
Please note that this is a very special case wherein performing business logic in a getter method is completely legit, considering how the <p:graphicImage> works under the covers. Invoking business logic in getters is namely usually frowned upon, see also Why JSF calls getters multiple times. Don't use this special case as excuse for other standard (non-special) cases. Please also note that you can't make use of EL 2.2 feature of passing method arguments like so #{studentImages.image(student.id)} because this argument won't end up in the image URL. Thus you really need to pass them as <f:param>.
If you happen to use OmniFaces 2.0 or newer, then consider using its <o:graphicImage> instead which can be used more intuitively, with an application scoped getter method directly delegating to the service method and supporting EL 2.2 method arguments.
Thus so:
<p:dataTable value="#{bean.students}" var="student">
<p:column>
<o:graphicImage value="#{studentImages.getImage(student.id)}" />
</p:column>
</p:dataTable>
With
#Named // Or #ManagedBean
#ApplicationScoped
public class StudentImages {
#EJB
private StudentService service;
public byte[] getImage(Long studentId) {
return studentService.find(studentId).getImage();
}
}
See also the blog on the subject.
Try including a mime type. In your posted example, you have it as "". The blank image may be because it doesn't recognize the stream as a image file since you made that field an empty string. So add a mime type of image/png or image/jpg and see if that works:
String mimeType = "image/jpg";
StreamedContent file = new DefaultStreamedContent(bytes, mimeType, filename);
There's a couple possibilities here (and please post the entire class if this isn't it).
1) You're not initializing the image properly
2) Your stream is empty so you're getting nothing
I'm assuming student.getImage() has a signature of byte[] so first make sure that that data is actually intact and represents an image. Secondly--you're not specifying a content-type which should be "image/jpg" or whatever you're using.
Here's some boilerplate code to check it with, I'm using Primefaces 2 for this.
/** 'test' package with 'test/test.png' on the path */
#RequestScoped
#ManagedBean(name="imageBean")
public class ImageBean
{
private DefaultStreamedContent content;
public StreamedContent getContent()
{
if(content == null)
{
/* use your database call here */
BufferedInputStream in = new BufferedInputStream(ImageBean.class.getClassLoader().getResourceAsStream("test/test.png"));
ByteArrayOutputStream out = new ByteArrayOutputStream();
int val = -1;
/* this is a simple test method to double check values from the stream */
try
{
while((val = in.read()) != -1)
out.write(val);
}
catch(IOException e)
{
e.printStackTrace();
}
byte[] bytes = out.toByteArray();
System.out.println("Bytes -> " + bytes.length);
content = new DefaultStreamedContent(new ByteArrayInputStream(bytes), "image/png", "test.png");
}
return content;
}
}
and some markup...
<html
xmlns="http://www.w3.org/1999/xhtml"
xmlns:h="http://java.sun.com/jsf/html"
xmlns:p="http://primefaces.prime.com.tr/ui"
>
<h:head>
</h:head>
<h:body>
<p:graphicImage value="#{imageBean.content}" />
</h:body>
</html>
If that code works then you're set up properly. Despite the fact it is garbage code for the streams (don't use it in production) it should give you a point to troubleshoot from. My guess is that you might have something happening in your JPA or other Database framework where you're byte[] is empty or it is formatted wrong. Alternatively you could just have a content-type problem.
Lastly, I would clone the data from the bean so that student.getImage() would only be copied into a new array and then used. This way if you have something unknown going on (something else moving the object or changing the byte[] you're not messing with your streams.
Do something like:
byte[] data = new byte[student.getImage().length]
for(int i = 0; i < data.length; i++)
data[i] = student.getImage()[i];
so that your bean has a copy (or Arrays.copy()--whatever floats your boat). I can't stress enough how something simple like this/content type is usually what's wrong. Good luck with it.
The answer from BalusC is (as usual) the correct one.
But keep one thing (as already stated by him) in mind. The final request is done from the browser to get the URL from the constructed <img> tag. This is not done in a 'jsf context'.
So if you try to e.g. access the phaseId (logging or whatever reason)
context.getCurrentPhaseId().getName()
This will result in a NullPointerException and the somehow misleading error message you will get is:
org.primefaces.application.resource.StreamedContentHandler () - Error in streaming dynamic resource. Error reading 'image' on type a.b.SomeBean
It took me quite some time to figure out what was the problem.

Resources