I'm trying to upload my data in a local server.
I used the command line:
--url=http://localhost:8080/_ah/remote_api/ --filename=C:\Users\Brayan\Desktop\sns.csv upload_data
My base is huge and the upload runs fine for a while, then it breaks:
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\devappserver2\wsgi_server.py", line 143, in _loop_forever
self._select()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\devappserver2\wsgi_server.py", line 165, in _select
fd_to_callback[fd]()
File "C:\Program Files (x86)\Google\google_appengine\lib\cherrypy\cherrypy\wsgiserver\wsgiserver2.py", line 1944, in tick
self.requests.put(conn)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\devappserver2\wsgi_server.py", line 92, in put
_THREAD_POOL.submit(self._handle, obj)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\devappserver2\thread_executor.py", line 60, in submit
t.start()
File "C:\Python27\lib\threading.py", line 745, in start
_start_new_thread(self.__bootstrap, ())
error: can't start new thread
Somebody knows what happened?
Related
When I'm trying to deploy the latest version of my app to GAE, it's giving me an error. What I have read is that somewhere I might be using Tabs instead of spaces, but I have no clue where? How can I find the location of the error through the log?
2014-10-26 21:58:46 Running command: "['C:\\Python27\\python.exe', '-u', 'C:\\Program Files (x86)\\Google\\google_appengine\\appcfg.py', '--no_cookies', u'--email=email#gmail.com', '-- passin', '--server=clever-app.appspot.com', 'update', 'D:\\Documents\\app-spot Project\\wp39 - Copy']"
09:58 PM Application: clever-app; version: wpfromstarterproject-10262014
09:58 PM Host: clever-app.appspot.com
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\appcfg.py", line 126, in <module>
run_file(__file__, globals())
File "C:\Program Files (x86)\Google\google_appengine\appcfg.py", line 122, in run_file
execfile(_PATHS.script_file(script_name), globals_)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 5400, in <module>
main(sys.argv)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 5391, in main
result = AppCfgApp(argv).Run()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 2981, in Run
self.action(self)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 5047, in __call__
return method()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 3794, in Update
self._UpdateWithParsedAppYaml(appyaml, self.basepath)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 3815, in _UpdateWithParsedAppYaml
updatecheck.CheckForUpdates()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\sdk_update_checker.py", line 243, in CheckForUpdates
runtime=runtime))
File "C:\Program Files (x86)\Google\google_appengine\lib\yaml\lib\yaml\__init__.py", line 82, in safe_load
return load(stream, SafeLoader)
File "C:\Program Files (x86)\Google\google_appengine\lib\yaml\lib\yaml\__init__.py", line 66, in load
return loader.get_data()
File "C:\Program Files (x86)\Google\google_appengine\lib\yaml\lib\yaml\constructor.py", line 38, in get_data
return self.construct_document(self.get_node())
File "C:\Program Files (x86)\Google\google_appengine\lib\yaml\lib\yaml\composer.py", line 27, in get_node
return self.compose_document()
File "C:\Program Files (x86)\Google\google_appengine\lib\yaml\lib\yaml\composer.py", line 37, in compose_document
self.get_event()
File "C:\Program Files (x86)\Google\google_appengine\lib\yaml\lib\yaml\parser.py", line 115, in get_event
self.current_event = self.state()
File "C:\Program Files (x86)\Google\google_appengine\lib\yaml\lib\yaml\parser.py", line 190, in parse_document_end
token = self.peek_token()
File "C:\Program Files (x86)\Google\google_appengine\lib\yaml\lib\yaml\scanner.py", line 128, in peek_token
self.fetch_more_tokens()
File "C:\Program Files (x86)\Google\google_appengine\lib\yaml\lib\yaml\scanner.py", line 257, in fetch_more_tokens
% ch.encode('utf-8'), self.get_mark())
yaml.scanner.ScannerError: while scanning for the next token
found character '\t' that cannot start any token
in "<string>", line 4, column 1:
<meta charset="UTF-8" />
^
2014-10-26 21:58:52 (Process exited with code 1)
You can close this window now.
I've pasted my app.yaml file here as requested: http://pastebin.com/geLGNKCY
Wait a minute. This is all sorts of weirdness then. I apologize that I didn't read through your error stack trace more closely.
Your error trace indicates that things are going wrong around here: https://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/tools/sdk_update_checker.py#235
But this is checking for an SDK update. This should not fail. Very strange!
Can you try with and without the --skip_sdk_update_check flag to your call to appcfg.py, and confirm that what you're seeing is an error during the SDK check?
Is the error repeatable?
Also, from your machine, can you show us what you see if you visit http://appengine.google.com/api/updatecheck? Do you happen to have some web proxy or firewall of some sort? I'm admittedly speculating on what could possibly do this, as I don't understand the error yet. (The error message suggests that somehow you're getting HTML content in a very unexpected place, so something is odd.)
When I do goapp test ./.. in my app directory, I get spurious failures from the dev_appserver.py script. Example:
INFO 2014-06-14 15:09:43,457 devappserver2.py:706] Skipping SDK update check.
WARNING 2014-06-14 15:09:43,457 devappserver2.py:722] DEFAULT_VERSION_HOSTNAME will not be set correctly with --port=0
Traceback (most recent call last):
File "/home/travis/go_appengine/dev_appserver.py", line 82, in <module>
_run_file(__file__, globals())
File "/home/travis/go_appengine/dev_appserver.py", line 78, in _run_file
execfile(_PATHS.script_file(script_name), globals_)
File "/home/travis/go_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 943, in <module>
main()
File "/home/travis/go_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 936, in main
dev_server.start(options)
File "/home/travis/go_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 749, in start
request_data, storage_path, options, configuration)
File "/home/travis/go_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 848, in _create_api_server
default_gcs_bucket_name=options.default_gcs_bucket_name)
File "/home/travis/go_appengine/google/appengine/tools/devappserver2/api_server.py", line 359, in setup_stubs
auto_id_policy=datastore_auto_id_policy)
File "/home/travis/go_appengine/google/appengine/datastore/datastore_sqlite_stub.py", line 629, in __init__
(self.__datastore_file, e))
google.appengine.runtime.apiproxy_errors.ApplicationError: ApplicationError: 3 Data in /tmp/appengine.testapp.travis/datastore.db is corrupt or a different version. Try running with the --clear_datastore flag.
OperationalError('database schema has changed',)
Note that even though this particular example is runing on Travis CI; I do get the same failures when running locally.
If I run individual tests one at a time, they all pass just fine.
I'm trying to use bulkloader to load entities to my GAE app, which is already hosted on GAE. I'm not using DEV server now. It was all fine couple of weeks ago, when I left the app, now after like 2 or 3 updates of GoogleAppEngineLauncher to 1.9 I can't use same commands as before to load entities, because it ends with tons of errors:
[INFO ] Connecting to my-app.appspot.com/remote_api
Traceback (most recent call last):
File "/usr/local/bin/appcfg.py", line 126, in <module>
run_file(__file__, globals())
File "/usr/local/bin/appcfg.py", line 122, in run_file
execfile(_PATHS.script_file(script_name), globals_)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4934, in <module>
main(sys.argv)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4925, in main
result = AppCfgApp(argv).Run()
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 2648, in Run
self.action(self)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4605, in __call__
return method()
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4417, in PerformUpload
run_fn(args)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4298, in RunBulkloader
sys.exit(bulkloader.Run(arg_dict))
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 4408, in Run
return _PerformBulkload(arg_dict)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 4273, in _PerformBulkload
loader.finalize()
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 382, in finalize
self.reserve_keys(self.keys_to_reserve)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1217, in ReserveKeys
datastore._GetConnection()._reserve_keys(ConvertKeys(keys))
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 2108, in _reserve_keys
self._async_reserve_keys(None, keys).get_result()
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 890, in get_result
results = self.__rpcs[0].get_result()
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 612, in get_result
return self.__get_result_hook(self)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 2151, in __reserve_keys_hook
self.check_rpc_success(rpc)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1331, in check_rpc_success
rpc.check_success()
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 578, in check_success
self.__rpc.CheckSuccess()
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 156, in _WaitImpl
self.request, self.response)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 200, in MakeSyncCall
self._MakeRealSyncCall(service, call, request, response)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 236, in _MakeRealSyncCall
raise UnknownJavaServerError("An unknown error has occured in the "
google.appengine.ext.remote_api.remote_api_stub.UnknownJavaServerError: An unknown error has occured in the Java remote_api handler for this call.
AppEngine is written in JAVA, however I use python sdk to use bulkloader.
Same day I used Datastore Admin to wipe app database. This uses much quotas, and this is why upload was failing (just guessing). Still it's weird since my project has billing enabled.
It seems gae assigns very high IDs to the models.
When I download my entities, I get for some entries very big numbers. These were autogenerated in first place. Downloading them as csv is no problem. But deleting the existing data and re-uploading the same data throws an exception.
Exceeded maximum allocated IDs
Trace:
Traceback (most recent call last):
File "/opt/eclipse/plugins/org.python.pydev_2.7.5.2013052819/pysrc/pydevd.py", line 1397, in <module>
debugger.run(setup['file'], None, None)
File "/opt/eclipse/plugins/org.python.pydev_2.7.5.2013052819/pysrc/pydevd.py", line 1090, in run
pydev_imports.execfile(file, globals, locals) #execute the script
File "/home/kave/workspace/google_appengine/appcfg.py", line 171, in <module>
run_file(__file__, globals())
File "/home/kave/workspace/google_appengine/appcfg.py", line 167, in run_file
execfile(script_path, globals_)
File "/home/kave/workspace/google_appengine/google/appengine/tools/appcfg.py", line 4247, in <module>
main(sys.argv)
File "/home/kave/workspace/google_appengine/google/appengine/tools/appcfg.py", line 4238, in main
result = AppCfgApp(argv).Run()
File "/home/kave/workspace/google_appengine/google/appengine/tools/appcfg.py", line 2396, in Run
self.action(self)
File "/home/kave/workspace/google_appengine/google/appengine/tools/appcfg.py", line 3973, in __call__
return method()
File "/home/kave/workspace/google_appengine/google/appengine/tools/appcfg.py", line 3785, in PerformUpload
run_fn(args)
File "/home/kave/workspace/google_appengine/google/appengine/tools/appcfg.py", line 3676, in RunBulkloader
sys.exit(bulkloader.Run(arg_dict))
File "/home/kave/workspace/google_appengine/google/appengine/tools/bulkloader.py", line 4379, in Run
return _PerformBulkload(arg_dict)
File "/home/kave/workspace/google_appengine/google/appengine/tools/bulkloader.py", line 4244, in _PerformBulkload
loader.finalize()
File "/home/kave/workspace/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 384, in finalize
self.increment_id(high_id_key)
File "/home/kave/workspace/google_appengine/google/appengine/tools/bulkloader.py", line 1206, in IncrementId
unused_start, end = datastore.AllocateIds(high_id_key, max=high_id_key.id())
File "/home/kave/workspace/google_appengine/google/appengine/api/datastore.py", line 1965, in AllocateIds
return AllocateIdsAsync(model_key, size, **kwargs).get_result()
File "/home/kave/workspace/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 612, in get_result
return self.__get_result_hook(self)
File "/home/kave/workspace/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1863, in __allocate_ids_hook
self.check_rpc_success(rpc)
File "/home/kave/workspace/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1236, in check_rpc_success
raise _ToDatastoreError(err)
google.appengine.api.datastore_errors.BadRequestError: Exceeded maximum allocated IDs
Usually my Id's are around 26002 but the new id's since a few days ago are as big as 4948283361329150. These are causing problems now. (If I change them to lower values, its all fine, but i didn't generate these ids in first place) Why does GAE have such problems with its own generated ids?
Many Thanks
This is a known issue, fixed in the 1.8.2 or later SDKs.
Note, if you use bulkloader against the dev appserver those SDKs (1.8.2, 1.8.3) unfortunately have a separate bulkloader issue with that use case (see appcfg-py-upload-data-fails-in-google-app-engine-sdk-1-8-2) but not in production.
A popup on my desktop recently offered to update GoogleAppEngineLauncher, and I agreed to it.
My Python apps in PyCharm then showed "unrecognized reference" for "google" in from google.appengine.api import users or any other reference to "google.appengine.api".
So I launched GoogleAppEngineLauncher and clicked something, and it looked like it did an unzip or something. That made the PyCharm reference errors go away.
When I then tried to launch my app with my old configuration, it had a problem with "Additional options". Apparently it no longer recognizes "-p 8081" or "-c". So I changed them to "--port 8081" and "--clear_datastore". Now it terminates with the following stacktrace:
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/dev_appserver.py", line 194, in <module>
_run_file(__file__, globals())
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/dev_appserver.py", line 190, in _run_file
execfile(script_path, globals_)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 545, in <module>
main()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 535, in main
options = PARSER.parse_args()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1688, in parse_args
args, argv = self.parse_known_args(args, namespace)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1720, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1926, in _parse_known_args
start_index = consume_optional(start_index)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1866, in consume_optional
take_action(action, args, option_string)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1794, in take_action
action(self, namespace, argument_values, option_string)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/boolean_action.py", line 67, in __call__
raise ValueError('must be "yes" or "no", not %r' % values)
ValueError: must be "yes" or "no", not '.'
I tried running "helloworld" in GoogleAppEngineLauncher and got the following in the LogConsole:
*** Running dev_appserver with the following flags:
--skip_sdk_update_check=yes --port=8081 --admin_port=8000 --clear_datastore
Python command: /usr/bin/python2.7
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/dev_appserver.py", line 194, in <module>
_run_file(__file__, globals())
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/dev_appserver.py", line 190, in _run_file
execfile(script_path, globals_)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 545, in <module>
main()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 535, in main
options = PARSER.parse_args()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1678, in parse_args
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1710, in parse_known_args
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1916, in _parse_known_args
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1856, in consume_optional
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1784, in take_action
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/boolean_action.py", line 67, in __call__
raise ValueError('must be "yes" or "no", not %r' % values)
ValueError: must be "yes" or "no", not '/Users/lindsay/Projects/PyCharm/Zephyr/gae-tutorial/helloworld'
This seems to be more or less the same error as from running my app in PyCharm.
I am under tremendous time pressure on this project, and this is completely blocking me. Any help will be most appreciated.
The solution to this problem was that configuration option -c has to be changed to --clear_datastore=yes.
Now I'm getting to another error, "InvalidCertificateException". I'll investigate that separately, and post a separate question if I can't solve it.