Since Google AppEngine 1.8.5 there is a new warning in the development environment:
WARNING 2013-09-27 10:10:53,035 api_config.py:1768] Method specifies path
parameters but you are not using a ResourceContainer. This will fail in future
releases; please switch to using ResourceContainer as soon as possible.
What are ResourceContainers and how to use them?
They recently updated the docs to explain this change here: Google App Engine Docs
Basically what you want to do is to separate the request body and the query/path parameters.
The request body will be a normal messages.Message class and you define any additional parameters in the ResourceContainer.
YOUR_RESOURCE_CONTAINER = endpoints.ResourceContainer(
MyRequestBodyMessagesClass,
parameter1=messages.IntegerField(2, required=True)
parameter2=messages.StringField(3))
This change should help to minimize the amount of necessary Message classes because you can mostly reuse the RequestBody-Message for Response-Messages as well.
Note: if you are using the endpoints-proto-datastore there's an open issue about this.
Related
I'm getting this error on app engine using flask to make a Slack bot. It happens whenever I send a POST request from Slackbot.
Unfortunately, the url provided in the error is a dead link. How do I go about using sockets instead of URLFetch?
/base/data/home/apps/[REDACTED]/lib/requests/packages/urllib3/contrib/appengine.py:115:
AppEnginePlatformWarning: urllib3 is using URLFetch on Google App
Engine sandbox instead of sockets. To use sockets directly instead of
URLFetch see https://urllib3.readthedocs.io/en/latest/contrib.html.
As detailed on Google's Sockets documentation, sockets can be used by setting the GAE_USE_SOCKETS_HTTPLIB environment variable. This feature seems to be available only on paid apps, and impacts billing.
Though the error you posted gets logged as an Error in App Engine, this thread suggests (see reply #8) that the error is actually meant as a warning, which the text "AppEnginePlatformWarning" seems to suggest anyway.
The comment block on the source page for appengine.py is also instructive.
You didn't post any information about your implementation, but on Google App Engine Standard edition, using URLFetch via the AppEngineManager should be just fine, though you will get the error.
You can use the following to silence this:
import warnings
import urllib3.contrib.appengine
warnings.filterwarnings('ignore', r'urllib3 is using URLFetch', urllib3.contrib.appengine.AppEnginePlatformWarning)
For me, turns out the presence of requests_toolbelt dependency in my project was the problem: it somehow forced the requests library to use urllib3, which requires URLFetch to be present, otherwise it raises an AppEnginePlatformError. As suggested in the app engine docs, monkey-patching Requests with requests_toolbelt forces the former to use URLFetch, which is no longer supported by GAE in a Python 3 runtime.
The solution was to remove requests_toolbelt from my requirements.txt file
Is there anyway that I can manage the appengine versions and instances through API calls?
What I mean by managing is to start/stop/delete versions deployed to the appengine through API calls.
Is that possible by using gcloud sdk commands from command line ?
Another question , does google provide APIs (or commands) to check the status of running instances ? check if the instance is idle or not and how long its being idle
There is a beta API for managing versions and services here:
https://cloud.google.com/appengine/docs/admin-api/
The API is still beta because it's under active development; there are still a few methods and fields which aren't implemented. Shortly after those are complete, the API will be marked "v1", though v1beta4 and v1beta5 will continue to be supported for several months in transition.
For example, the API doesn't yet include operations on instances, but I expect that List/Get/Delete will be available fairly soon. Since App Engine automatically creates instances for you, there is no create instance API.
I just noticed that the most recent documentation re-skin seems to have hidden the documentation for the REST interface, so I'll drop that link there so you that you can find the currently implemented methods. (Version.Update is also implemented for a few fields, so that documentation update should be coming out very soon.)
2020 UPDATE: You can do it using the apps.services.versions api. You can stop/start a version with the PATCH method, setting the mask to "servingStatus" and in the body set the "servingStatus" field to "STOPPED"/"SERVING".
Similarly, you can use the delete/create methods to launch and remove new versions
Reference:
https://cloud.google.com/appengine/docs/admin-api/reference/rest/v1/apps.services.versions/patch
I'm building some reporting tool. Ideally I want to avoid going through web server logs myself and use (some of) the power of Piwik.
The stuff I get from the visitor log would be a good start, this is at http://example.com/piwik/index.php?module=CoreHome&action=index&idSite=1&period=day&date=yesterday#/module=Live&action=getVisitorLog&idSite=1&period=day&date=yesterday
Unfortunately I can't find a getVisitorLog action in the HTTP API docs at
http://developer.piwik.org/api-reference/reporting-api#Actions (and it's also not an undocumented feature, method=Actions.getVisitorLog gives me
The method 'getVisitorLog' does not exist or is not available in the module '\Piwik\Plugins\Actions\API'.
Is there another way to get to this? Or should I write a plugin for Piwik?
Apparently it is possible through the Live plugin API:
http://developer.piwik.org/api-reference/reporting-api#Live
This works as desired:
http://example.com/piwik/index.php?module=API&method=Live.getLastVisitsDetails&format=JSON&idSite=1&period=day&date=2015-07-21&expanded=1&token_auth=XXXXX&filter_limit=100
Ok, i found this link https://code.google.com/p/gwt-platform/wiki/CrawlerSupport#Using_gwtp-crawler-service that explain how you can make your GWTP app crawlable.
I got some GWTP experience, but i know nothing about AppEngine.
Google said its "crawlservice.appspot.com" can parse any Ajax page. Now I have a page "http://mydomain.com#!article" that has an artice that was pulled from Database. Say that page has the text "this is my article". Now I open this link:
crawlservice.appspot.com/?key=123456&url=http://mydomain.com#!article, then i can see all javascript but I couldn't find the text "this is my article".
Why?
Now let check with a real life example
open this link https://groups.google.com/forum/#!topic/google-web-toolkit/Syi04ArKl4k & you will see the text "If i open that url in IE"
Now you open http://crawlservice.appspot.com/?key=123456&url=https://groups.google.com/forum/#!topic/google-web-toolkit/Syi04ArKl4k you can see all javascript but there is no text "If i open that url in IE",
Why is it?
SO if i use http://crawlservice.appspot.com/?key=123456&url=mydomain#!article then Can google crawler be able to see the text in mydomain#!article?
also why the key=123456, it means everyone can use this service? do we have our own key? does google limit the number of calls to their service?
Could you explain all these things?
Extra Info:
Christopher suggested me to use this example
https://github.com/ArcBees/GWTP-Samples/tree/master/gwtp-samples/gwtp-sample-crawler-service
However, I ran into other problem. My app is a pure GWTP, it doesn't have appengine-web.xml in WEB-INF. I have no idea what is appengine or GAE mean or what is Maven.
DO i need to register AppEngine?
My Appp may have a lot of traffic. Also I am using Godaddy VPS. I don't want to register App Engine since I have to pay for Google for extra traffic.
Everything in my GWTP App is ok right now except Crawler Function.
So if I don't use Google App Engine, then how can i build Crawler Function for GWTP?
I tried to use HTMLUnit for my app, but HTMLUnit doesn't work for GWTP (See details in here Why HTMLUnit always shows the HostPage no matter what url I type in (Crawlable GWT APP)? )
I believe you are not allowed to crawl Google Groups. Probably they are actively trying to prevent this, so you do not see the expected content.
There's a couple points I wish to elaborate on:
The Google Code documentation is no longer maintained. You should look on Github instead: https://github.com/ArcBees/GWTP/wiki/Crawler-Support
You shouldn't use http://crawlservice.appspot.com. This isn't a Google service, it's out of date and we may decide to delete it down the road. This only serves as a public example. You should create your own application on App Engine (https://appengine.google.com/)
There is a sample here (https://github.com/ArcBees/GWTP-Samples/tree/master/gwtp-samples/gwtp-sample-crawler-service) using GWTP's Crawler Service. You can basically copy-paste it. Just make sure you update the <application> tag in appengine-web.xml to the name of your application and use your own service key in CrawlerModule.
Finally, if your client uses GWTP and you followed the documentation, it will work. If you want to try it manually, you must encode the Query Parameters.
For example http://crawlservice.appspot.com/?key=123456&url=http://www.arcbees.com#!service will not work because the hash (everything including and after #) is not sent to the server.
On the other hand http://crawlservice.appspot.com/?key=123456&url=http%3A%2F%2Fwww.arcbees.com%2F%23!service will work.
I use httplib.HTTPConnection within my app. Do I really need to provide host parameter in httplib.HTTPConnection constructor? If so, why? (I mean, I know that it's a mandatory parameter, but I wonder if I could specify None or empty string) And is there any global constant in Google AppEngine and in development server which I can use within my app in order to omit explicitly defined host.
If you leave it out of the constructor, how will the other methods know where to send their messages?
The address/name of the server you're connecting to is the parameter for the HTTPConnection, the URI on that server is what goes into request.
From the python documentation (which is the basis for AppEngine)
h1 = httplib.HTTPConnection(host[, port[, strict[, timeout[, source_address]]]])
h1.request(method, url[, body[, headers]])
[edit]
Remember, it may not always be you who is responsible for this code. Also, why complicate things by including so much more information in the URI when you're (for example) making calls to numerous URI's on the same website?
[/edit]