app engine incoming mail handling - google-app-engine

Question

I am developing a Google App Engine application.
I wish to receive mails under '%username%#appid.appspotmail.com', where %username% belongs to a user of the application.
I just can't figure out what to define in web.xml file.
Any similar solution such as mails to:
'%username%.usermailbox#appid.appspotmail.com'
'usermailbox.%username%#appid.appspotmail.com'
is acceptable (if it makes it easier with the wildcards).
I've tried (as suggested by Gopi)
mapping the relevant servlet to <url-pattern>/_ah/mail/user.*</url-pattern> within the web.xml file. It's not working.
The client gets a bounce message, whereas the server logs, do show a relevant request received by the app, but rejected with a 404. No "No handlers matched this URL." INFO is added to the log entry. In addition, when GETing the generated URL, I don't get a 'This page does not support GET', but rather a plain 404.
If I however send mail to say 'info#appid.appspotmail.com', the logs show a 404 (which they should, as it's not mapped in the web.xml). In addition, for such a request, a "No handlers matched this URL." INFO is added to the relevant log entry.
Needless to say that, Incoming mail IS found under Configured Services.

Answer 1

This change happened when App Engine started using a true Java web server (and so Toby's explanation is spot on... sadly I can't seem to recover my login to vote it up!). My recommendation is to use a Filter. I played around with the filter below when writing a toy app for GAE. You once you've defined the base class at the end of this post, you can can create a series of mail handlers (like the following). All you have to do is register each filter in your web.xml to handle /_ah/mail/*.
public class HandleDiscussionEmail extends MailHandlerBase {
public HandleDiscussionEmail() { super("discuss-(.*)#(.*)"); }
#Override
protected boolean processMessage(HttpServletRequest req, HttpServletResponse res)
throws ServletException
{
MimeMessage msg = getMessageFromRequest(req);
Matcher match = getMatcherFromRequest(req);
...
}
}
public abstract class MailHandlerBase implements Filter {
private Pattern pattern = null;
protected MailHandlerBase(String pattern) {
if (pattern == null || pattern.trim().length() == 0)
{
throw new IllegalArgumentException("Expected non-empty regular expression");
}
this.pattern = Pattern.compile("/_ah/mail/"+pattern);
}
#Override public void init(FilterConfig config) throws ServletException { }
#Override public void destroy() { }
/**
* Process the message. A message will only be passed to this method
* if the servletPath of the message (typically the recipient for
* appengine) satisfies the pattern passed to the constructor. If
* the implementation returns <code>false</code>, control is passed
* o the next filter in the chain. If the implementation returns
* <code>true</code>, the filter chain is terminated.
*
* The Matcher for the pattern can be retrieved via
* getMatcherFromRequest (e.g. if groups are used in the pattern).
*/
protected abstract boolean processMessage(HttpServletRequest req, HttpServletResponse res) throws ServletException;
#Override
public void doFilter(ServletRequest sreq, ServletResponse sres, FilterChain chain)
throws IOException, ServletException {
HttpServletRequest req = (HttpServletRequest) sreq;
HttpServletResponse res = (HttpServletResponse) sres;
MimeMessage message = getMessageFromRequest(req);
Matcher m = applyPattern(req);
if (m != null && processMessage(req, res)) {
return;
}
chain.doFilter(req, res); // Try the next one
}
private Matcher applyPattern(HttpServletRequest req) {
Matcher m = pattern.matcher(req.getServletPath());
if (!m.matches()) m = null;
req.setAttribute("matcher", m);
return m;
}
protected Matcher getMatcherFromRequest(ServletRequest req) {
return (Matcher) req.getAttribute("matcher");
}
protected MimeMessage getMessageFromRequest(ServletRequest req) throws ServletException {
MimeMessage message = (MimeMessage) req.getAttribute("mimeMessage");
if (message == null) {
try {
Properties props = new Properties();
Session session = Session.getDefaultInstance(props, null);
message = new MimeMessage(session, req.getInputStream());
req.setAttribute("mimeMessage", message);
} catch (MessagingException e) {
throw new ServletException("Error processing inbound message", e);
} catch (IOException e) {
throw new ServletException("Error processing inbound message", e);
}
}
return message;
}
}

Answer 2

the following provides a plausable explanation, thanks to
url-pattern and wildcards
which refers to
http://jcp.org/aboutJava/communityprocess/mrel/jsr154/index2.html (scroll to section 11.2)
In the url-pattern the * wildcard behaves differently to how one would assume,
it is treated as a normal character, except
-when the string ends with /* for "path mapping"
-or it begins with *. for "extension mapping"
Too bad, would have been nice to wildcard-match email recipient addresses to different servlets, as depicted in Google's API doc samples. I'm using absolute matches now which isn't as clean as the appid needs to be included.

Answer 3

I think putting an entry similar to below into your web.xml should work to match your second case 'usermailbox.%username%#appid.appspotmail.com
<servlet>
<servlet-name>handlemail</servlet-name>
<servlet-class>HandleMyMail</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>handlemail</servlet-name>
<url-pattern>/_ah/mail/usermailbox.*</url-pattern>
</servlet-mapping>

Answer 4

Well... After trying every possible solution/url-mapping, I went with fast and ugly one.
The gist is to have a single "catch all" mail servlet, to work as a dispatcher to other, specific, servlets. It's like a giant switch, where the parameter is the request URL.
This is NOT what I wished for, but it works, and seems to be the only one that does.
I have a single servlet IncomingMail that handles ALL incoming mail. period.
So now, the only mapping of URLs under /_ah/mail/ is the following:
<servlet>
<servlet-name>IncomingMail</servlet-name>
<servlet-class>IncomingMail</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>IncomingMail</servlet-name>
<url-pattern>/_ah/mail/*</url-pattern>
</servlet-mapping>
In addition, I have the following servlet, mapped as a "plain-old-servlet":
(notice the <url-pattern>, not a "mail mapped" servlet)
<servlet>
<servlet-name>GetUserMail</servlet-name>
<servlet-class>GetUserMail</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>GetUserMail</servlet-name>
<url-pattern>/serv/userMail</url-pattern>
</servlet-mapping>
The catch-all servlet (would eventually) look like a giant switch:
public class IncomingMail extends HttpServlet {
private final String USER_MAIL_PREFIX="http://appid.appspot.com/_ah/mail/user.";
private final String USER_MAIL_SERVLET="/serv/userMail";
...
public void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
String url = req.getRequestURL().toString();
System.out.println("IncomingMail called, with URL: "+url);
String email;
String servlet;
if (url.startsWith(USER_MAIL_PREFIX)) {
email=url.replace(USER_MAIL_PREFIX, "");
servlet=USER_MAIL_SERVLET;
}//userMail
if (url.startsWith(OTHER_PREFIX)) {
//Redirect to OTHER servlet
}
...
System.out.println("forward to '"+servlet+"', with email '"+email+"'");
RequestDispatcher dispatcher=req.getRequestDispatcher(servlet);
try {
req.setAttribute("email", email);
dispatcher.forward(req, resp);
} catch (ServletException e) {
System.err.println(e);
}
}
}
The destination servlet (GetUserMail in this case), does a getRequestParameter("email"), to see the specific destined mailbox.
It will receive all mails sent to 'user.%un%#appid.appspotmail.com', where %un% is a username in the application space.
The email parameter received by the servlet would be of the form '%un%#appid.appspotmail.com', without the discerning prefix.
Each such "specific" servlet, would get "its cut" from the mail dispatcher servlet, with the email parameter already without the discerning prefix.
One note I will add under security:
If you're worried of bogus requests to the "specific servlets", just define them all under a common virtual namespace say /servmail/ in your site, and define a new <security-constraint> to allow requests to originate only within the application itself.
Like so (inside web.xml):
<security-constraint>
<web-resource-collection>
<web-resource-name>MailServlets</web-resource-name>
<description>policy for specific mail servlets</description>
<url-pattern>/servmail/*</url-pattern>
</web-resource-collection>
<auth-constraint>
<role-name>admin</role-name>
</auth-constraint>
</security-constraint>
Would still love to hear from someone that tried and succeeded in doing a wildcard <url-pattern> mail mapping, other than a catch-all one.

Answer 5

I had a similar problem (using Python, so yaml config files rather than XML) and the cause turned out to be because I put the:
- url: /_ah/mail/.+
script: handle_incoming_email.py
login: admin
before an existing catch-all entry:
- url: /.*
script: main.py
This gave 404s on the server and "Message send failure" when sending test messages.
Moving it after the catch-all entry solved the problem.

Answer 6

I'm fairly sure the problem is just that you're trying to use .*. URL expressions in web.xml are globs, not regular expressions, so you should use just * instead - .* will only match strings starting with a dot.