We have an AppEngine that receives automatic data via email from remote sites and stores it into the datastore. We're using a 3rd Party SMTP host now, and /_ah/mail/ is working properly.
A lot of this data is coming from legacy microcontrollers, PLCs, smart meters and the like. They all have a configuration for email address, SMTP server, SMTP user/pass, From address, and interval.
We'd like to setup postfix on a g1-small Compute Engine instance to handle authenticated direct-SMTP connections for the incoming data, but there are no examples of anyone else doing this. Is it as simple as writing a postfix filter to take the data and POST it over to /_ah/mail on AppEngine?
Alternately, is there an easier way that we're missing? We are converting some of the devices to use POST/PUT where possible, but we have a lot of different devices, and that will take time.
Google App Engine provides an SMTP service for inbound email - messages sent to <anything>#<app_id>.appspotmail.com will be sent to /_ah/mail/<anything>. If your devices only need to send email into your system you could point them directly to GAE's mail servers.
Your suggestion of running a inbound mailserver on GCE and using it to forward to HTTP on your app is also a viable solution, and doesn't require abusing email servers. There are even companies that will do this for you!
Related
I'm running an app on the Google App Engine standard environment. We need to send out a lot of emails and it looks like App Engine now lets us use JavaMail to send emails using SMTP - connecting to smtp.gmail.com or smtp.sendgrid.net.
I was wondering if this counts towards the Mail API limits/quotas in App Engine. I'm guessing it shouldn't, since the email is actually sent by the respective SMTP server and it's only outbound traffic from App Engine. Is this correct?
Note: I'm not a Java user, the answer is based on documentation only.
Indeed, using 3rd party partner mail services is the Google recommendation for sending lots of emails. From Quotas and limits:
Each Mail service request counts toward the Mail API Calls quota.
There are also quotas for the number of messages you can send, the
number of Admin messages you can send, and the amount of data and
attachments you can send. For more details, see Mail quotas.
[...]
If you need to send more mail then the quota allows, you can use a
third-party mail provider, such as SendGrid or Mailgun.
And JavaMail can, indeed, use these 3rd party SMTP servers. From Sending email messages:
To send email messages, use the JavaMail classes included with the App
Engine SDK.
When you create a JavaMail Session, if you do not provide any SMTP
server configuration, App Engine uses the Mail service for sending
messages. Alternatively, add SMTP configuration for supported
third-party mail providers such as Mailgun, Mailjet, or
SendGrid.
This last quote appears (depending on how you interpret it, I guess) to support your assumption that when SMTP servers are specified the Mail service would not be used, thus those calls wouldn't be counted against the Mail API Calls quota.
Also each of the 3rd-party mail integration documentation page referenced in these quotes doesn't (or at least doesn't directly) use JavaMail, instead they require specific libraries/modules. Which might have some relationship with bypassing the GAE Mail Api quota limits. For example, from Sending Email with SendGrid:
Sending an email
It's easy to get started with the Java library for SendGrid to
send emails from your App Engine apps.
With the prerequisites complete, make sure you are set up for
Java on your local machine. The last thing you'll need before
writing code is to copy Sendgrid.java to the src directory of
your app. You'll import this class so that you can create a SendGrid
instance and send mail with simple commands.
So, to gain certainty, my suggestion is to actually put your assumption to the test: send some emails using JavaMail while specifying your SMTP server info, then check your actual usage in the developer console Quotas page for matching increases. Notes:
give some ample room for propagation delays - the usage counts might not be immediately updated
if you don't see the Mail section in the Usage table check also, towards the bottom of the page, a Show resources not in use button/link (but, if it's in there, it means the usage is zero).
You could also directly follow the recommended specific instructions for your 3rd party provider of choice and practically avoid the question altogether :)
I understand that in order to link my domain to my GAE application I need to setup Google Apps according to this.
But Google Apps seems to take control of SMTP and email addresses. I need to have my own SMTP server in GCS, which is working perfectly right now using the static ip like someuser#xx.xx.xx.xx
What I need is to be able to enter my GAE web application thru mydomain.com but at the same time being able to receive anyuser#mydomain.com at my SMTP instance at GCS. Is it possible?
Your email settings are determined by your registrar, not by Google Apps. If you don't want to use Gmail as your mail server, you simply keep your DNS records unchanged, as Xavier correctly suggested.
Google App Engine currently limits you to 2,000 emails per day (for free) via their API.
I am trying to find a definitive answer if it is possible to use a third-party system if you need to send more. I know that they disallow raw sockets, so I would assume that there might be trouble with this approach... but surely I'm not the first to see it.
Worst case, I can build a simple offsite web service that my GAE can call... but I'd much rather just be able to send directly through an SMTP server.
Thanks!
Nope.
You're correct: you cannot make raw socket requests, nor any other direct outbound requests except through the urlfetch API. To talk to an external SMTP server, you would need to use a webservice as a proxy.
We use the Postmark mail outsourcing service via the hutools.postmark API. Since the communication is HTTP based, it works like a charm on Google AppEngine. This might be an option for you, although it is also a for-pay service. We use it to get arround GAEs sender restrictions.
I've successfully used third party providers for email services with Google App Engine. I've used both SendGrid and MailGun using their HTTP-API.
When I send emails through google app engine using the mail's send_mail function, they often get placed in the receiver's SPAM folder. Although this problem does not happen for GMail accounts, it does for Yahoo Mail accounts (and presumably several others)
Is there a way to get the emails GAE sends around the SPAM filter?
This is going to depend a great deal on what about the messages you're sending gets them marked as spam. Google's documentation suggests that it's the app engine that actually does the SMTP, so there might be some other google app engine app that someone's marked as spam and your sending IP (which happens to be the same) is getting tarred with the same brush, or it might be something about the content of your messages. Many big mail providers give you some way to determine how 'spammy' they'll think a particular message will be; perhaps you could get information that way?
Another option is the standard 'be careful to add [some address] to your safe senders list if you want to get email from us!' warning in your app.
See this reasoning, provided by Gmail on how it filters incoming email based on the header information. The key thing to note here is that all emails sent via GAE's SMTP infrastructure are sent via email IDs of the format xyz.apphosting.bounces.google.com, which by Gmail's definition itself brings the emails under the category of falsely / apparently bounced email messages, hence they get categorized as spam. The same would hold true for other email service providers that follow similar logic for spam filtering.
GAE still needs to come up with a good solution to this problem. In the meanwhile, the suggestion I would give you is to use an external email service. Since GAE does not allow you to open arbitrary TCP sockets, you can use the URLFetch service to submit requests to external services that can open SMTP connections and send email on behalf of your email ID.
While this approach shifts your email sending costs to the external service, it consumes additional GAE resources, namely- UrlFetch service API calls and UrlFetch Data sent.
Update: Google App Engine plans to launch support for outbound sockets soon as part of their production feature roadmap, so making external SMTP connections directly would become a possibility.
App Engine only allows you to use these formats for XMPP addresses:
app-id#appspot.com
anything#app-id.appspotchat.com
Is there be a way to configure a custom domain so that it forwards XMPP messages to one of these address formats.
For example, if my domain is called myxmpp.com, I could give that out to users as my application's JID and then forward any user chat messages received there directly to myxmpp#appspot.com (assuming myxmpp is the app-id)?
I'm fairly sure that, to do this, you would need a custom XMPP server serving for your domain. I'm also fairly sure that you'd have to write said server yourself.
This site http://www.prosody.im/doc/dns explains what you need to configure in your DNS to do this.