Now I use my own Java FTP program to ftp objects from my PC to my ISP's website server.
I want to use Google App Engine's servlet to get Paypal IPN messages, then store the messages into my own objects and ftp the objects to my ISP's website server, is this doable ? I heard Google App Engine doesn't support FTP.
I don't expect Google to do it for me, but can I use my own Java FTP program in the web app that I upload onto the App Engine to do it ?
Frank
No, you can't open any socket connection except by using URL Fetch
service on HTTP/HTTPS to these port ranges:
80-90, 440-450, 1024-65535.
As of April 9 this year (SDK 1.7.7) this isn't a problem any longer. Outbound sockets (e.g. FTP) are generally available to all billing-enabled apps.
Socket API Overview (Java): https://developers.google.com/appengine/docs/java/sockets/
UPDATE: Our code below may no longer work. This FTP code worked for us before but we see a comment now below that says FTP is no longer supported on App Engine. See the link below. If you try this code and it works or doesn't work for you for straight FTP (TLS is NOT supported BTW) - please comment.
Yes. FTP now works on Google App Engine. (The accepted answer is outdated and no longer true.)
Here is tested and working code on GAE.
#!/usr/bin/env python
from google.appengine.ext import webapp
from ftplib import FTP
class HwHandler(webapp.RequestHandler):
def get(self):
self.response.out.write('FTP Starting...<br>')
ftp = FTP('ftp_site.com')
ftp.login('login', 'password')
ftp.retrlines('LIST') # list directory contents
self.response.out.write('FTP opened')
ftp.quit()
app = webapp.WSGIApplication([
('/', HwHandler)
], debug=True)
Of note, FTP TLS does not appear to work currently. (Trying to do "from ftplib import FTP_TLS" fails.)
You can use the Apache Commons FTP client (org.apache.commons.net.ftp.FTPClient) if you put it into passive mode. Just do the following:
FTPClient client = new FTPClient();
client.connect(FTP_HOST);
client.enterLocalPassiveMode();
Then it won't call ServerSocketFactory, and life should be good!
Related
Some files get uploaded on a daily basis to an FTP server and I need those files under Google Cloud Storage. I don't want to bug the users that upload the files to install any additional software and just let them keep using their FTP client.
Is there a way to use GCS as an FTP server? If not, how can I create a job that periodically picks up the files from an FTP location and puts them in GCS?
In other words: what's the best and simplest way to do it?
You could write yourself an FTP server which uploads to GCS, for example based on pyftpdlib
Define a custom handler which stores to GCS when a file is received
import os
from pyftpdlib.handlers import FTPHandler
from pyftpdlib.servers import FTPServer
from pyftpdlib.authorizers import DummyAuthorizer
from google.cloud import storage
class MyHandler:
def on_file_received(self, file):
storage_client = storage.Client()
bucket = storage_client.get_bucket('your_gcs_bucket')
blob = bucket.blob(file[5:]) # strip leading /tmp/
blob.upload_from_filename(file)
os.remove(file)
def on_... # implement other events
def main():
authorizer = DummyAuthorizer()
authorizer.add_user('user', 'password', homedir='/tmp', perm='elradfmw')
handler = MyHandler
handler.authorizer = authorizer
handler.masquerade_address = add.your.public.ip
handler.passive_ports = range(60000, 60999)
server = FTPServer(("127.0.0.1", 21), handler)
server.serve_forever()
if __name__ == "__main__":
main()
I've successfully run this on Google Container Engine (it requires some effort getting passive FTP working properly) but it should be pretty simple to do on Compute Engine. According to the above configuration, open port 21 and ports 60000 - 60999 on the firewall.
To run it, python my_ftp_server.py - if you want to listen on port 21 you'll need root privileges.
You could setup a cron and rsync between the FTP server and Google Cloud Storage using gsutil rsync or open source rclone tool.
If you can't run those commands on the FTP server periodically, you could mount the FTP server as a local filesystem or drive (Linux, Windows)
I have successfully set up an FTP proxy to GCS using gcsfs in a VM in Google Compute (mentioned by jkff in the comment to my question), with these instructions:
http://ilyapimenov.com/blog/2015/01/19/ftp-proxy-to-gcs.html
Some changes are needed though:
In /etc/vsftpd.conf change #write_enable=YES to
write_enable=YES
Add firewall rules in your GC project to allow
access to ports 21 and passive ports 15393 to 15592 (https://console.cloud.google.com/networking/firewalls/list)
Some possible problems:
If you can access the FTP server using the local ip, but not the remote ip, it's probably because you haven't set up the firewall rules
If you can access the ftp server, but are unable to write, it's probably because you need the write_enable=YES
If you are tying to read on the folder you created on /mnt, but get a I/O error, it's probably because the bucket in gcsfs_config is not right.
Also, your ftp client needs to use the transfer mode set to "passive".
Set up a VM in the google cloud, using some *nix flavor. Set up ftp on it, and point it to a folder abc. Use google fuse to mount abc as a GCS bucket. Voila - back and forth between gcs / ftp without writing any software.
(Small print: fuse rolls up and dies if you push too much data, so bounce it periodically, once a week or once a day; also you might need to set the mount or fuse to allow permissions for all users)
I'm getting this error on app engine using flask to make a Slack bot. It happens whenever I send a POST request from Slackbot.
Unfortunately, the url provided in the error is a dead link. How do I go about using sockets instead of URLFetch?
/base/data/home/apps/[REDACTED]/lib/requests/packages/urllib3/contrib/appengine.py:115:
AppEnginePlatformWarning: urllib3 is using URLFetch on Google App
Engine sandbox instead of sockets. To use sockets directly instead of
URLFetch see https://urllib3.readthedocs.io/en/latest/contrib.html.
As detailed on Google's Sockets documentation, sockets can be used by setting the GAE_USE_SOCKETS_HTTPLIB environment variable. This feature seems to be available only on paid apps, and impacts billing.
Though the error you posted gets logged as an Error in App Engine, this thread suggests (see reply #8) that the error is actually meant as a warning, which the text "AppEnginePlatformWarning" seems to suggest anyway.
The comment block on the source page for appengine.py is also instructive.
You didn't post any information about your implementation, but on Google App Engine Standard edition, using URLFetch via the AppEngineManager should be just fine, though you will get the error.
You can use the following to silence this:
import warnings
import urllib3.contrib.appengine
warnings.filterwarnings('ignore', r'urllib3 is using URLFetch', urllib3.contrib.appengine.AppEnginePlatformWarning)
For me, turns out the presence of requests_toolbelt dependency in my project was the problem: it somehow forced the requests library to use urllib3, which requires URLFetch to be present, otherwise it raises an AppEnginePlatformError. As suggested in the app engine docs, monkey-patching Requests with requests_toolbelt forces the former to use URLFetch, which is no longer supported by GAE in a Python 3 runtime.
The solution was to remove requests_toolbelt from my requirements.txt file
I'm a designer still trying to code up my first app in Google App Engine as an experiment.
I have got to the point where I want to set up memcache to cache my entire site following the instructions at: http://docs.djangoproject.com/en/dev/topics/cache/#memcached
I am clear that I need to add in my settings.py the following:
CACHE_BACKEND = 'memcached://[IP ADDRESS]:[PORT]/'
And then:
MIDDLEWARE_CLASSES = (
'django.middleware.cache.UpdateCacheMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.cache.FetchFromCacheMiddleware',
)
CACHE_MIDDLEWARE_SECONDS=60*2
This might be a really dumb question but what would my IP address and port be for my google app engine site? Is there any considerations I need to make because of the fact that it is being hosted at Google?
How would I find it out
djangoappengine has standard settings. At the top of your settings.py you should have "from djangoappengine.settings_base import *". You can take a look at the settings_base module to see all backends and default settings.
Try: CACHE_BACKEND = 'memcached://?timeout=0'.
Source: http://bitbucket.org/wkornewald/djangoappengine/src/tip/settings_base.py
I'm trying to make http requests from my Google App Engine webapp, and discovered I have to use URLConnection since it's the only whitelisted class. The corresponding Clojure library is clojure.contrib.http.agent, and my code is as follows:
(defroutes example
(GET "/" [] (http/string (http/http-agent "http://www.example.com")))
(route/not-found "Page not found"))
This works fine in my development environment- the browser displays the text for example.com. But when I test it out with Google's development app server:
phrygian:example wei$ dev_appserver.sh war
2010-09-28 14:53:36.120 java[43845:903] [Java CocoaComponent compatibility mode]: Enabled
...
INFO: The server is running at http://localhost:8080/
It just hangs when I load the page. No error, or anything. Any idea what might be going on?
http-agent creates threads so that might be why it does not work.
From the API documentation:
Creates (and immediately returns) an Agent representing an HTTP
request running in a new thread.
You could try http-connection, which is a wrapper around HttpURLConnection, so this should work.
Another alternative is to try clj-http. The API seems to be a bit more high-level, but it uses Apache HttpComponents which might be blacklisted.
I am guessing http.async.client is a definite no-go due to its strong asynchronous approach.
You might want to try appengine.urlfetch/fetch from appengine-clj (http://github.com/r0man/appengine-clj, also in clojars)
The CherryPy web server can supposedly be deployed in the Google App Engine.
Who has done it, and what was the experience like?
What special effort was required (configuration, etc.)?
Would you recommend it to others?
The article is a good example but its slightly out of date now as the patch is no longer required, the latest version of Cherrypy should run without it, I've gotten the sample below running in the development environment.
I've included cherrypy inside a zip file as the google app engine has a limit of one thousand files per application, it also makes it easier to deploy.
I'm also using the cherrypy dispatch handler to route the request.
import sys
sys.path.insert(0, 'cherrypy.zip')
import cherrypy
import wsgiref.handlers
class Root:
exposed = True
def GET(self):
return "give a basic description of the service"
d = cherrypy.dispatch.MethodDispatcher()
conf = {'/':
{
'request.dispatch': d
}
}
app = cherrypy.tree.mount(Root(), "/",conf)
wsgiref.handlers.CGIHandler().run(app)
So far I've not come across any particular issues but I have read some people have had issues with sessions.
See boodebr.org article (missing, but here on the Wayback machine) It works for me.
If you are looking for an example, look for the condition that accepts ServerMode.GAE in ServerInterface.auto in this example.
There is a good article on how to do this over here now here. I haven't actually tried this yet, I stuck with django on App Engine, but it seems to be a solid example.