I'm trying to download data from my Google App Engine app, using the official instructions
Remote API is set up & I call:
appcfg.py download_data --application=appname --url=http://app.address.com/_ah/remote_api --filename=alldata.csv
The connection is established, I'm being asked for my e-mail & password, and then a long trace appears ending up with this:
google.appengine.api.datastore_errors.BadRequestError: app s~appname cannot access app appname's data
Any ideas?
If using the high-replication datastore, change the value passed to --application from appname to s~appname.
Found the problem - as an URL I was giving my own app's domain name instead of the 1.appname.appspot.com.
I ran into this same issue, but when attempting to use the remote API from a custom script as opposed to appcfg.py's bulk downloader.
This issue may have been addressed in more recent GAE SDK releases, but for various reasons I'm still using the older 1.7 release. I found the answer in this bug report: https://code.google.com/p/googleappengine/issues/detail?id=4374
Basically, when calling ConfigureRemoteApi, don't specify the APP_ID argument (just pass None) and specify your app ID via the host argument, e.g. 'myapp-hrd.appspot.com'. ConfigureRemoteApi will figure out your app ID correctly and won't add the 's~' that causes this problem.
Related
Ok, i found this link https://code.google.com/p/gwt-platform/wiki/CrawlerSupport#Using_gwtp-crawler-service that explain how you can make your GWTP app crawlable.
I got some GWTP experience, but i know nothing about AppEngine.
Google said its "crawlservice.appspot.com" can parse any Ajax page. Now I have a page "http://mydomain.com#!article" that has an artice that was pulled from Database. Say that page has the text "this is my article". Now I open this link:
crawlservice.appspot.com/?key=123456&url=http://mydomain.com#!article, then i can see all javascript but I couldn't find the text "this is my article".
Why?
Now let check with a real life example
open this link https://groups.google.com/forum/#!topic/google-web-toolkit/Syi04ArKl4k & you will see the text "If i open that url in IE"
Now you open http://crawlservice.appspot.com/?key=123456&url=https://groups.google.com/forum/#!topic/google-web-toolkit/Syi04ArKl4k you can see all javascript but there is no text "If i open that url in IE",
Why is it?
SO if i use http://crawlservice.appspot.com/?key=123456&url=mydomain#!article then Can google crawler be able to see the text in mydomain#!article?
also why the key=123456, it means everyone can use this service? do we have our own key? does google limit the number of calls to their service?
Could you explain all these things?
Extra Info:
Christopher suggested me to use this example
https://github.com/ArcBees/GWTP-Samples/tree/master/gwtp-samples/gwtp-sample-crawler-service
However, I ran into other problem. My app is a pure GWTP, it doesn't have appengine-web.xml in WEB-INF. I have no idea what is appengine or GAE mean or what is Maven.
DO i need to register AppEngine?
My Appp may have a lot of traffic. Also I am using Godaddy VPS. I don't want to register App Engine since I have to pay for Google for extra traffic.
Everything in my GWTP App is ok right now except Crawler Function.
So if I don't use Google App Engine, then how can i build Crawler Function for GWTP?
I tried to use HTMLUnit for my app, but HTMLUnit doesn't work for GWTP (See details in here Why HTMLUnit always shows the HostPage no matter what url I type in (Crawlable GWT APP)? )
I believe you are not allowed to crawl Google Groups. Probably they are actively trying to prevent this, so you do not see the expected content.
There's a couple points I wish to elaborate on:
The Google Code documentation is no longer maintained. You should look on Github instead: https://github.com/ArcBees/GWTP/wiki/Crawler-Support
You shouldn't use http://crawlservice.appspot.com. This isn't a Google service, it's out of date and we may decide to delete it down the road. This only serves as a public example. You should create your own application on App Engine (https://appengine.google.com/)
There is a sample here (https://github.com/ArcBees/GWTP-Samples/tree/master/gwtp-samples/gwtp-sample-crawler-service) using GWTP's Crawler Service. You can basically copy-paste it. Just make sure you update the <application> tag in appengine-web.xml to the name of your application and use your own service key in CrawlerModule.
Finally, if your client uses GWTP and you followed the documentation, it will work. If you want to try it manually, you must encode the Query Parameters.
For example http://crawlservice.appspot.com/?key=123456&url=http://www.arcbees.com#!service will not work because the hash (everything including and after #) is not sent to the server.
On the other hand http://crawlservice.appspot.com/?key=123456&url=http%3A%2F%2Fwww.arcbees.com%2F%23!service will work.
I'm building a webapp in Go that requires authentication. I'd like to run local tests using appengine/aetest that validate the authentication behavior. However, I do not see any way to create an aetest.Context with a dummy user. Am I missing something?
I had a similar issue with Python sdk. The gist of the solution is to bypass authentication when tests run locally.
You should have access to the [web] app object at the the test setup time - create a user object and save it into the app (or wherever your get_current_user() method will check).
This will let you unit test all application functions except authentication itself. For the later part you can deploy your latest changes as unpublished google app version, then test authentication and if all works - publish the version.
I've discovered some header values that seem to do the trick. appengine/user/user_dev.go has the following:
X-AppEngine-Internal-User-Email
X-AppEngine-Internal-User-Federated-Identity
X-AppEngine-Internal-User-Federated-Provider
X-AppEngine-Internal-User-Id
X-AppEngine-Internal-User-Is-Admin
If I set those headers on the Context's Request when doing in-process tests, things seem to work as expected. If I set the headers on a request that I create separately, things are less successful, since the 'user.Current()' call consults the Context's Request.
These headers might work in a Python environment as well.
I am using the warmup service to carry out precaching/etc. The request gets called with self.request.host being prefixed with a version of the app.
All other handler requests are coming with the expected host name for the app.
So if the app name is myapp - then all requests are called with self.request.host set to myapp.appspot.com, whereas for "_ah/warmup" call it is getting set to nnn.myapp.appspot.com.
My code is expecting the self.request.host to be always 'myapp.appspot.com'. Is this by design or am I missing something.
Thanks.
I think this is by design because the warmup service is for a specific version. All other requests are going straight to your main app URL, which is just "aliased" to whatever version happens to be the default version at the time.
By the way, it is documented that you can access all deployed versions of your app by prefixing the version number to the domain name, so you should be aware that any users could access any version if they know about this, and if you haven't taken countermeasures! So you should definitely support this - it's an official feature of App Engine.
I'm trying to use the "Copy to another app" feature of AppEngine and keep getting an error:
Fetch to http://datastore-admin.moo.appspot.com/_ah/remote_api failed with status 302
This is for a Java app but I followed the instructions on setting up a default Python runtime.
I'm 95% sure it's an authentication issue and the call to remote_api is redirecting to the Google login page. Both apps use Google Apps as the authentication mechanism. I've also tried copying to and from a third app we have which uses Google Accounts for authentication.
Notes:
The user account I log in with is an Owner on all three apps. It's a Google Apps account (if that wasn't obvious).
I have a gmail account this is an Owner on all three apps as well. When I log in to the admin console with it, I don't see the datastore admin console at all when I click it.
I'm able to use the remote_api just fine from the command-line after I enter my details
Tried with both the Python remote_api built-in and the Java one.
I've found similar questions/blog posts about this, one of which required logging in from a browser, then manually submitting the ACSID cookie you get after that's done. Can't do that here, obviously.
OK, I think I got this working.
I'll refer to the two appIDs as "source" and "dest".
To enable datastore admin (as you know) you need to upload a Python project with the app.yaml and appengine_config.py files as described in the docs.
Either I misread the docs or there is an error. The "appID" inthe .yaml should be the app ID you are uploading to to enable DS admin.
The other appID in the appengine_config file, specifically this line:
remoteapi_CUSTOM_ENVIRONMENT_AUTHENTICATION = (
'HTTP_X_APPENGINE_INBOUND_APPID', ['appID'])
Should be the appID of the "source", ID the app id of where the data is coming from in the DS copy operation.
I think this line is what allows the source appID to be authenticated as having permissions to write to the "dest" app ID.
So, I changed that .py, uploaded again to my "dest" app ID. To be sure I made this dummy python app as default and left it as that.
Then on the source app ID I tried the DS copy again, and all the copy jobs were kicked off OK - so it seems to have fixed it.
i'm trying to upload my app engine project for the very first time and i have no clue why it is not working. the error from my terminal is:
[me][~/Desktop]$ appcfg.py update ProjectDir/
Application: tacticalagentz; version: 1
Host: appengine.google.com
Starting update of app: tacticalagentz, version: 1
Scanning files on local disk.
Error 404: --- begin server output ---
This application does not exist (app_id=u'tacticalagentz').
--- end server output ---
i'm using python 2.6.5 and ubuntu 10.04.
not sure if this is relevant, but i just created a google app engine account today. and i also just created the application today (like a couple of hours ago). this is really frustrating because i just want to upload what i have so far (as a demo). in my app.yaml this is my first line:
application: tacticalagentz
Furthermore, i checked on my admin console, and i CLEARLY see the app id right there, and it matches letter for letter with the app id in my app.yaml
could someone please enlighten me and tell me what i am doing wrong? or is it something beyond my comprehension (like indexing issue with Google that they need time to index my app id) ?
thank you very much in advance
apparently adding the "--no_cookies" parameter will work
appcfg.py update --no_cookies ProjectDir/
the way i was able to find my answer was by uploading my app from my Mac OS X (thank god i have linux mac and windows). AppEngine on Mac OS X comes with a GUI interface, and it worked for uploading. so then i found the command they used in the console, which included "--no_cookies". perhaps if you run into similar issues in the future, this is one approach to getting the answer
App Engine for Java have the same problem. The problem is about account login.
If you are using Eclipse, use Sign In button.
If u are using command-line, use "-e" option, like this:
appcfg.sh -e your#email.com update yoursite/
I had the same problem. When I changed the name of the app I used in the launcher to match the one in the app engine, It worked without any problem. The way I figured out, it was the name mismatch which caused the problem. You can see the name of your registered app in the admin console of app engine.(https://appengine.google.com/)
Here's what fixed it for me:
i had an instance of dev_appserver.py myProjDirectory/ on a different terminal.
i guess the scripts are somehow linked and aren't thread safe
An alternate option that worked for me is to just "Clear Deployment Credential" from the Control option of the GUI. When the app was deployed after this, it opened a google page to allow GAE to access the user profile and then deployment was successful.
The key bit is
This application does not exist (app_id=u'tacticalagentz').
which is telling you that appspot.com doesn't know of an application by that name. The admin console (https://appengine.google.com/) shows your applications. Check there. You might have made an inadvertent typo when you registered the app.