I would like to centralize my app's performance monitoring, so I should extract somehow GAE dashboard or quota usage details data. Is there any way to do it (for ex. with Google API)?
I do not think there is an API provided by Google to get your quota statistics for App Engine application.
This is an Open Issue : https://code.google.com/p/googleappengine/issues/detail?id=655 and you can get some hints from the thread about what other developers have been trying.
Still waiting on official version.
Khan academy uses curl:
https://github.com/Khan/analytics/tree/master/src/gae_dashboard
Probably against terms of service, but I use mechanize to access the GAE dashboard (verified works from deployed app).
import gaemechanize2._mechanize as mechanize
self.br = mechanize.Browser()
#self.br.set_all_readonly(False) # allow everything to be written to
self.br.set_handle_robots(False) # ignore robots
self.br.set_handle_refresh(False) # can sometimes hang without this
self.br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
self.br.open(given_source)
loginForm = self.br.forms().next()
loginForm["Email"] = self.the_email
loginForm["Passwd"] = self.the_password
response = self.br.open(loginForm.click())
self.the_response=response.read()
if re.search('Verify that it.*you',self.the_response):
logging.info("BREAK: NOT YOU...trying again")
loginForm=self.br.forms().next()
loginForm.set_value(['PhoneVerificationChallenge'],name='challengetype')
loginForm['phoneNumber']=self.the_phone
response1=loginForm.click()
response=self.br.open(response1)
self.the_response=response.read()
Related
I want to put a scraping service using Apache HttpClient to the Cloud. I read problems are possible with Google App Engine, as it's direct network access and threads creation are prohibited. What's about other cloud hosting providers? Have anyone experince with Apache HttpClient + cloud?
AppEngine has threads and direct network access (HTTP only). There is a workaround to make it work with HttpClient.
Also, if you plan to use many parse tasks in parallel, you might check out Task Queue or even mapreduce.
Btw, there is a "misfeature" in GAE that you can not fully set custom User-agent header on your requests - GAE always adds "AppEngine" to the end of it (this breaks requests to certain sites - most notably iTunes).
It's certainly possible to create threads and access other websites from CloudFoundry, you're just time limited for each process. For example, if you take a look at http://rack-scrape.cloudfoundry.com/, it's a simple rack application that inspects the 'a' tags from Google.com;
require 'rubygems'
require 'open-uri'
require 'hpricot'
run Proc.new { |env|
doc = Hpricot(open("http://www.google.com"))
anchors = (doc/"a")
[200, {"Content-Type" => "text/html"}, [anchors.inspect]]
}
As for Apache HttpClient, I have no experience of this but I understand it isn't maintained any more.
I'm trying to make http requests from my Google App Engine webapp, and discovered I have to use URLConnection since it's the only whitelisted class. The corresponding Clojure library is clojure.contrib.http.agent, and my code is as follows:
(defroutes example
(GET "/" [] (http/string (http/http-agent "http://www.example.com")))
(route/not-found "Page not found"))
This works fine in my development environment- the browser displays the text for example.com. But when I test it out with Google's development app server:
phrygian:example wei$ dev_appserver.sh war
2010-09-28 14:53:36.120 java[43845:903] [Java CocoaComponent compatibility mode]: Enabled
...
INFO: The server is running at http://localhost:8080/
It just hangs when I load the page. No error, or anything. Any idea what might be going on?
http-agent creates threads so that might be why it does not work.
From the API documentation:
Creates (and immediately returns) an Agent representing an HTTP
request running in a new thread.
You could try http-connection, which is a wrapper around HttpURLConnection, so this should work.
Another alternative is to try clj-http. The API seems to be a bit more high-level, but it uses Apache HttpComponents which might be blacklisted.
I am guessing http.async.client is a definite no-go due to its strong asynchronous approach.
You might want to try appengine.urlfetch/fetch from appengine-clj (http://github.com/r0man/appengine-clj, also in clojars)
I have a small test App running on GAE under the default free quota. It runs fine locally. When deployed on GAE (appspot), some parts of it do not work. Appspot dashboard does not show any error in the logs. Added code to trap quota limits is not triggered. Why is there a variation between the App running locally, versus failing when uploaded? There are no server error screens when deployed, only non-responsive buttons. (One non-responsive html form submit button, which works fine locally.) I am using Eclipse to run the App locally and also to deploy the same identical code.
The Appspot log is below. The *.jsp script getmoreinputs.jsp is supposed to collect data from a form, upon submit, it is supposed to trigger a servlet named /Calculate. The servlet works perfectly when tested locally, but is not triggered in the live deployment.
Any help would be appreciated.
#
1.
08-22 07:57PM 12.475 /getmoreinputs.jsp?cp=true&iv=true 200 23ms 16cpu_ms 0kb Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 (.NET CLR 3.5.30729),gzip(gfe)
See details
98.236.17.99 - - [22/Aug/2010:19:57:12 -0700] "GET /getmoreinputs.jsp?cp=true&iv=true HTTP/1.1" 200 923 "http://black-scholes.appspot.com/" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 (.NET CLR 3.5.30729),gzip(gfe)" "black-scholes.appspot.com" ms=23 cpu_ms=17 api_cpu_ms=0 cpm_usd=0.000615
My test of your insight suggests: GAE is indifferent to / or // in pathname for submit=
When all else fails, try occult methods. Without any change to code, I complied the GAE App with a new version number. Then I went to Appspot dashboard and deleted the old version (which was not working). Amazing! The deployment of the new version worked perfecly, exactly as the local run. There was no change to the code.
GAE has problems. When deployment != Local, my new rule is: clean out all old versions at deployment site. In my case, I had only 1 old version.
Joe
As I mentioned in my answer to your previous question:
Check your urls are not having a double slash ('//') e.g. /user//listall. This works on dev server but not when you deploy it on app engine.
What I meant by this was, even if your url may not directly show a // , after appending the url suffix to the hostname this is pretty much possible. So I suggest you to try getmoreinputs.jsp instead of /getmoreinputs.jsp (note the '/' removed).
I suspect when this is being appended internally to http://black-scholes.appspot.com/ it is creating a url which looks like http://black-scholes.appspot.com//getmoreinputs.jsp which will not work on app engine when deployed. However this works on the dev server locally. Please give a try.
Now I use my own Java FTP program to ftp objects from my PC to my ISP's website server.
I want to use Google App Engine's servlet to get Paypal IPN messages, then store the messages into my own objects and ftp the objects to my ISP's website server, is this doable ? I heard Google App Engine doesn't support FTP.
I don't expect Google to do it for me, but can I use my own Java FTP program in the web app that I upload onto the App Engine to do it ?
Frank
No, you can't open any socket connection except by using URL Fetch
service on HTTP/HTTPS to these port ranges:
80-90, 440-450, 1024-65535.
As of April 9 this year (SDK 1.7.7) this isn't a problem any longer. Outbound sockets (e.g. FTP) are generally available to all billing-enabled apps.
Socket API Overview (Java): https://developers.google.com/appengine/docs/java/sockets/
UPDATE: Our code below may no longer work. This FTP code worked for us before but we see a comment now below that says FTP is no longer supported on App Engine. See the link below. If you try this code and it works or doesn't work for you for straight FTP (TLS is NOT supported BTW) - please comment.
Yes. FTP now works on Google App Engine. (The accepted answer is outdated and no longer true.)
Here is tested and working code on GAE.
#!/usr/bin/env python
from google.appengine.ext import webapp
from ftplib import FTP
class HwHandler(webapp.RequestHandler):
def get(self):
self.response.out.write('FTP Starting...<br>')
ftp = FTP('ftp_site.com')
ftp.login('login', 'password')
ftp.retrlines('LIST') # list directory contents
self.response.out.write('FTP opened')
ftp.quit()
app = webapp.WSGIApplication([
('/', HwHandler)
], debug=True)
Of note, FTP TLS does not appear to work currently. (Trying to do "from ftplib import FTP_TLS" fails.)
You can use the Apache Commons FTP client (org.apache.commons.net.ftp.FTPClient) if you put it into passive mode. Just do the following:
FTPClient client = new FTPClient();
client.connect(FTP_HOST);
client.enterLocalPassiveMode();
Then it won't call ServerSocketFactory, and life should be good!
The CherryPy web server can supposedly be deployed in the Google App Engine.
Who has done it, and what was the experience like?
What special effort was required (configuration, etc.)?
Would you recommend it to others?
The article is a good example but its slightly out of date now as the patch is no longer required, the latest version of Cherrypy should run without it, I've gotten the sample below running in the development environment.
I've included cherrypy inside a zip file as the google app engine has a limit of one thousand files per application, it also makes it easier to deploy.
I'm also using the cherrypy dispatch handler to route the request.
import sys
sys.path.insert(0, 'cherrypy.zip')
import cherrypy
import wsgiref.handlers
class Root:
exposed = True
def GET(self):
return "give a basic description of the service"
d = cherrypy.dispatch.MethodDispatcher()
conf = {'/':
{
'request.dispatch': d
}
}
app = cherrypy.tree.mount(Root(), "/",conf)
wsgiref.handlers.CGIHandler().run(app)
So far I've not come across any particular issues but I have read some people have had issues with sessions.
See boodebr.org article (missing, but here on the Wayback machine) It works for me.
If you are looking for an example, look for the condition that accepts ServerMode.GAE in ServerInterface.auto in this example.
There is a good article on how to do this over here now here. I haven't actually tried this yet, I stuck with django on App Engine, but it seems to be a solid example.