I'm trying merge 2 files and to create one new in the BlobStore of the AppEngine. This files are 12.000 rows and have 12 MB. The problem occurs when i writte the new file, because in middle of write appear the next error:
ApplicationError: 10
Traceback (most recent call last):
File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 716, in __call__
handler.post(*groups)
File "/base/data/home/apps/s~ono-hat-vv2/1.369681903434492066/com/__init__.py", line 244, in post
self.principal(cont+1, numFicheritos,"csvTotal","uniendoCSV")
File "/base/data/home/apps/s~ono-hat-vv2/1.369681903434492066/com/__init__.py", line 277, in principal
self.escribirtxt(f, cadenaSalida)
File "/base/data/home/apps/s~ono-hat-vv2/1.369681903434492066/com/__init__.py", line 308, in escribirtxt
f.write(linea)
File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/api/files/file.py", line 326, in write
self._make_rpc_call_with_retry('Append', request, response)
File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/api/files/file.py", line 427, in _make_rpc_call_with_retry
_make_call(method, request, response)
File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/api/files/file.py", line 252, in _make_call
_raise_app_error(e)
File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/api/files/file.py", line 195, in _raise_app_error
raise FileNotOpenedError(e)
FileNotOpenedError: ApplicationError: 10
I have read that the problem is in the time that I open the file, because this can't be greater that 30 s. How can i resolve this problem and merge this file?
Close the file and re-open every so often.
Have you looked at Google Cloud Storage? It may not have this limitation and also the app engine files api has been deprecated and we need to move in that direction anyway. Something to consider.
Related
I am really new on gae, but I have a task that have to deploy on google.
because I don't know how to stop after sudo /usr/local/bin/dev_appserver.py, I just press control+c. after a while, when I tried to deploy again, I receive following message:
INFO 2016-04-03 00:42:40,222 sdk_update_checker.py:229] Checking for updates to the SDK.
INFO 2016-04-03 00:42:41,272 api_server.py:205] Starting API server at: http://localhost:51277
INFO 2016-04-03 00:42:41,275 api_server.py:648] Applying all pending transactions and saving the datastore
INFO 2016-04-03 00:42:41,275 api_server.py:651] Saving search indexes
Traceback (most recent call last):
File "/usr/local/bin/dev_appserver.py", line 83, in <module>
_run_file(__file__, globals())
File "/usr/local/bin/dev_appserver.py", line 79, in _run_file
execfile(_PATHS.script_file(script_name), globals_)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 1040, in <module>
main()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 1033, in main
dev_server.start(options)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 824, in start
self._dispatcher.start(options.api_host, apis.port, request_data)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/dispatcher.py", line 194, in start
_module.start()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/module.py", line 1176, in start
self._balanced_module.start()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 315, in start
self._start_all_fixed_port(host_ports)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 352, in _start_all_fixed_port
raise BindError('Unable to bind %s:%s' % self.bind_addr)
google.appengine.tools.devappserver2.wsgi_server.BindError: Unable to bind localhost:8080
so far, I tried reinstall gaelauncher, but it doesn't work.
please, any help would be good.
After a few days, it worked again. I think it is Google App Engine's thing and I don't know its reason so far. But if you just want to use the deploy-on-localhost function, please step to other tools, it will be easier. Suggestion: MAMP.
I'm tyring to use remote_api_shell.py to access the datastore of my Google App Engine application in the following way:
remote_api_shell.py -s url/of/server --secure
But I get the following error:
Traceback (most recent call last):
File "./remote_api_shell.py", line 133, in <module>
run_file(__file__, globals())
File "./remote_api_shell.py", line 129, in run_file
execfile(_PATHS.script_file(script_name), globals_)
File "/home/vagrant/google-cloud-sdk/platform/google_appengine/google/appengine/tools/remote_api_shell.py", line 160, in <module>
main(sys.argv)
File "/home/vagrant/google-cloud-sdk/platform/google_appengine/google/appengine/tools/remote_api_shell.py", line 156, in main
oauth2=True)
File "/home/vagrant/google-cloud-sdk/platform/google_appengine/google/appengine/tools/remote_api_shell.py", line 74, in remote_api_shell
secure=secure)
File "/home/vagrant/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 768, in ConfigureRemoteApiForOAuth
rpc_server_factory=rpc_server_factory)
File "/home/vagrant/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 835, in ConfigureRemoteApi
app_id = GetRemoteAppIdFromServer(server, path, rtok)
File "/home/vagrant/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 569, in GetRemoteAppIdFromServer
response = server.Send(path, payload=None, **urlargs)
File "/home/vagrant/google-cloud-sdk/platform/google_appengine/google/appengine/tools/appengine_rpc_httplib2.py", line 245, in Send
url, method=method, body=payload, headers=headers)
File "/home/vagrant/google-cloud-sdk/platform/google_appengine/lib/oauth2client/oauth2client/client.py", line 562, in new_request
redirections, connection_type)
File "/home/vagrant/google-cloud-sdk/platform/google_appengine/lib/httplib2/httplib2/__init__.py", line 1584, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/home/vagrant/google-cloud-sdk/platform/google_appengine/lib/httplib2/httplib2/__init__.py", line 1332, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/home/vagrant/google-cloud-sdk/platform/google_appengine/lib/httplib2/httplib2/__init__.py", line 1268, in _conn_request
conn.connect()
File "/home/vagrant/google-cloud-sdk/platform/google_appengine/lib/httplib2/httplib2/__init__.py", line 1014, in connect
self.disable_ssl_certificate_validation, self.ca_certs)
File "/home/vagrant/google-cloud-sdk/platform/google_appengine/lib/httplib2/httplib2/__init__.py", line 80, in _ssl_wrap_socket
cert_reqs=cert_reqs, ca_certs=ca_certs)
File "/usr/lib/python2.7/ssl.py", line 487, in wrap_socket
ciphers=ciphers)
File "/usr/lib/python2.7/ssl.py", line 243, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 405, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [Errno 8] _ssl.c:510: EOF occurred in violation of protocol
Any help would be appreciated.
The problem went away when I switched from using my custom domain name to my-app-id.appspot.com.
Some other considerations:
If you are using https, then your-app-id.appspot.com will not work out of the box. You have to enable it on the Google Apps page corresponding to the domain name. Yes, that means that you will actually have to have Google Apps setup for that domain.
If you have a multi-module setup, make sure you add remote_api: on to the builtins section of the app.yaml of the default module. Or if you have remote-api: on for another module, then use the url that skips the dispatch.yaml file and targets the module directly.
I'm trying to use bulkloader to load entities to my GAE app, which is already hosted on GAE. I'm not using DEV server now. It was all fine couple of weeks ago, when I left the app, now after like 2 or 3 updates of GoogleAppEngineLauncher to 1.9 I can't use same commands as before to load entities, because it ends with tons of errors:
[INFO ] Connecting to my-app.appspot.com/remote_api
Traceback (most recent call last):
File "/usr/local/bin/appcfg.py", line 126, in <module>
run_file(__file__, globals())
File "/usr/local/bin/appcfg.py", line 122, in run_file
execfile(_PATHS.script_file(script_name), globals_)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4934, in <module>
main(sys.argv)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4925, in main
result = AppCfgApp(argv).Run()
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 2648, in Run
self.action(self)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4605, in __call__
return method()
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4417, in PerformUpload
run_fn(args)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4298, in RunBulkloader
sys.exit(bulkloader.Run(arg_dict))
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 4408, in Run
return _PerformBulkload(arg_dict)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 4273, in _PerformBulkload
loader.finalize()
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 382, in finalize
self.reserve_keys(self.keys_to_reserve)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1217, in ReserveKeys
datastore._GetConnection()._reserve_keys(ConvertKeys(keys))
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 2108, in _reserve_keys
self._async_reserve_keys(None, keys).get_result()
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 890, in get_result
results = self.__rpcs[0].get_result()
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 612, in get_result
return self.__get_result_hook(self)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 2151, in __reserve_keys_hook
self.check_rpc_success(rpc)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1331, in check_rpc_success
rpc.check_success()
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 578, in check_success
self.__rpc.CheckSuccess()
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 156, in _WaitImpl
self.request, self.response)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 200, in MakeSyncCall
self._MakeRealSyncCall(service, call, request, response)
File "/Applications/ADT/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 236, in _MakeRealSyncCall
raise UnknownJavaServerError("An unknown error has occured in the "
google.appengine.ext.remote_api.remote_api_stub.UnknownJavaServerError: An unknown error has occured in the Java remote_api handler for this call.
AppEngine is written in JAVA, however I use python sdk to use bulkloader.
Same day I used Datastore Admin to wipe app database. This uses much quotas, and this is why upload was failing (just guessing). Still it's weird since my project has billing enabled.
It seems gae assigns very high IDs to the models.
When I download my entities, I get for some entries very big numbers. These were autogenerated in first place. Downloading them as csv is no problem. But deleting the existing data and re-uploading the same data throws an exception.
Exceeded maximum allocated IDs
Trace:
Traceback (most recent call last):
File "/opt/eclipse/plugins/org.python.pydev_2.7.5.2013052819/pysrc/pydevd.py", line 1397, in <module>
debugger.run(setup['file'], None, None)
File "/opt/eclipse/plugins/org.python.pydev_2.7.5.2013052819/pysrc/pydevd.py", line 1090, in run
pydev_imports.execfile(file, globals, locals) #execute the script
File "/home/kave/workspace/google_appengine/appcfg.py", line 171, in <module>
run_file(__file__, globals())
File "/home/kave/workspace/google_appengine/appcfg.py", line 167, in run_file
execfile(script_path, globals_)
File "/home/kave/workspace/google_appengine/google/appengine/tools/appcfg.py", line 4247, in <module>
main(sys.argv)
File "/home/kave/workspace/google_appengine/google/appengine/tools/appcfg.py", line 4238, in main
result = AppCfgApp(argv).Run()
File "/home/kave/workspace/google_appengine/google/appengine/tools/appcfg.py", line 2396, in Run
self.action(self)
File "/home/kave/workspace/google_appengine/google/appengine/tools/appcfg.py", line 3973, in __call__
return method()
File "/home/kave/workspace/google_appengine/google/appengine/tools/appcfg.py", line 3785, in PerformUpload
run_fn(args)
File "/home/kave/workspace/google_appengine/google/appengine/tools/appcfg.py", line 3676, in RunBulkloader
sys.exit(bulkloader.Run(arg_dict))
File "/home/kave/workspace/google_appengine/google/appengine/tools/bulkloader.py", line 4379, in Run
return _PerformBulkload(arg_dict)
File "/home/kave/workspace/google_appengine/google/appengine/tools/bulkloader.py", line 4244, in _PerformBulkload
loader.finalize()
File "/home/kave/workspace/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 384, in finalize
self.increment_id(high_id_key)
File "/home/kave/workspace/google_appengine/google/appengine/tools/bulkloader.py", line 1206, in IncrementId
unused_start, end = datastore.AllocateIds(high_id_key, max=high_id_key.id())
File "/home/kave/workspace/google_appengine/google/appengine/api/datastore.py", line 1965, in AllocateIds
return AllocateIdsAsync(model_key, size, **kwargs).get_result()
File "/home/kave/workspace/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 612, in get_result
return self.__get_result_hook(self)
File "/home/kave/workspace/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1863, in __allocate_ids_hook
self.check_rpc_success(rpc)
File "/home/kave/workspace/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1236, in check_rpc_success
raise _ToDatastoreError(err)
google.appengine.api.datastore_errors.BadRequestError: Exceeded maximum allocated IDs
Usually my Id's are around 26002 but the new id's since a few days ago are as big as 4948283361329150. These are causing problems now. (If I change them to lower values, its all fine, but i didn't generate these ids in first place) Why does GAE have such problems with its own generated ids?
Many Thanks
This is a known issue, fixed in the 1.8.2 or later SDKs.
Note, if you use bulkloader against the dev appserver those SDKs (1.8.2, 1.8.3) unfortunately have a separate bulkloader issue with that use case (see appcfg-py-upload-data-fails-in-google-app-engine-sdk-1-8-2) but not in production.
A popup on my desktop recently offered to update GoogleAppEngineLauncher, and I agreed to it.
My Python apps in PyCharm then showed "unrecognized reference" for "google" in from google.appengine.api import users or any other reference to "google.appengine.api".
So I launched GoogleAppEngineLauncher and clicked something, and it looked like it did an unzip or something. That made the PyCharm reference errors go away.
When I then tried to launch my app with my old configuration, it had a problem with "Additional options". Apparently it no longer recognizes "-p 8081" or "-c". So I changed them to "--port 8081" and "--clear_datastore". Now it terminates with the following stacktrace:
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/dev_appserver.py", line 194, in <module>
_run_file(__file__, globals())
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/dev_appserver.py", line 190, in _run_file
execfile(script_path, globals_)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 545, in <module>
main()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 535, in main
options = PARSER.parse_args()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1688, in parse_args
args, argv = self.parse_known_args(args, namespace)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1720, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1926, in _parse_known_args
start_index = consume_optional(start_index)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1866, in consume_optional
take_action(action, args, option_string)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1794, in take_action
action(self, namespace, argument_values, option_string)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/boolean_action.py", line 67, in __call__
raise ValueError('must be "yes" or "no", not %r' % values)
ValueError: must be "yes" or "no", not '.'
I tried running "helloworld" in GoogleAppEngineLauncher and got the following in the LogConsole:
*** Running dev_appserver with the following flags:
--skip_sdk_update_check=yes --port=8081 --admin_port=8000 --clear_datastore
Python command: /usr/bin/python2.7
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/dev_appserver.py", line 194, in <module>
_run_file(__file__, globals())
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/dev_appserver.py", line 190, in _run_file
execfile(script_path, globals_)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 545, in <module>
main()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 535, in main
options = PARSER.parse_args()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1678, in parse_args
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1710, in parse_known_args
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1916, in _parse_known_args
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1856, in consume_optional
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1784, in take_action
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/boolean_action.py", line 67, in __call__
raise ValueError('must be "yes" or "no", not %r' % values)
ValueError: must be "yes" or "no", not '/Users/lindsay/Projects/PyCharm/Zephyr/gae-tutorial/helloworld'
This seems to be more or less the same error as from running my app in PyCharm.
I am under tremendous time pressure on this project, and this is completely blocking me. Any help will be most appreciated.
The solution to this problem was that configuration option -c has to be changed to --clear_datastore=yes.
Now I'm getting to another error, "InvalidCertificateException". I'll investigate that separately, and post a separate question if I can't solve it.