Migrated column type from HSTORE to JSONB and am using this snippet of code...
from sqlalchemy.dialects.postgresql import ARRAY, JSONB
if employment_type:
base = base.filter(Candidate.bio["employment_type"].cast(ARRAY).contains(employment_type))
and am getting this error...
127.0.0.1 - - [28/Mar/2016 12:25:13] "GET /candidate_filter/?employment_type_3=true HTTP/1.1" 500 -
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/flask/app.py", line 1836, in __call__
return self.wsgi_app(environ, start_response)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/Library/Python/2.7/site-packages/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Users/surajkapoor/Desktop/lhv-talenttracker/app/views.py", line 660, in investor_filter
base = base.filter(Candidate.bio["employment_type"].cast(ARRAY).contains(employment_type))
File "/Library/Python/2.7/site-packages/sqlalchemy/dialects/postgresql/json.py", line 93, in cast
return self.astext.cast(type_)
File "/Library/Python/2.7/site-packages/sqlalchemy/dialects/postgresql/json.py", line 95, in cast
return sql.cast(self, type_)
File "<string>", line 2, in cast
File "/Library/Python/2.7/site-packages/sqlalchemy/sql/elements.py", line 2314, in __init__
self.type = type_api.to_instance(type_)
File "/Library/Python/2.7/site-packages/sqlalchemy/sql/type_api.py", line 1142, in to_instance
return typeobj(*arg, **kw)
TypeError: __init__() takes at least 2 arguments (1 given)
Candidate.bio["employment_type"] is an array of integers and I'm simply trying to query all the rows that contain a specific integer in them.
Also, .cast() works perfectly on the same column when calling Integer...
if internship:
base = base.filter(Candidate.bio["internship"].cast(Integer) == 1)
SqlAlchemy is probably having difficulty constructing the where clause because it can't figure out what type bio->'employment_type' is.
If the contains method is called from a String object, it would generate a LIKE clause, but for JSONB or ARRAY it would need to generate the #> operator.
To give SqlAlchemy the necessary hints, use explicit casting everywhere, i.e. write your query like
from sqlalchemy import cast
if employment_type:
casted_field = Candidate.bio['employment_type'].cast(JSONB)
casted_values = cast(employment_type, JSONB)
stmt = base.filter(casted_field.contains(casted_values))
In my example, I have a JSONB column named bio with the following data:
{"employment_type": [1, 2, 3]}
Edit: Casting to JSONB works:
>>> from sqlalchemy.dialects.postgresql import JSONB
>>> employment_type = 2
>>> query = (
... session.query(Candidate)
... .filter(Candidate.bio['employment_type'].cast(JSONB).contains(employment_type)))
>>> query.one().bio
{"employment_type": [1, 2, 3]}
Original answer:
I couldn't get .contains to work on Candidate.bio['employment_type'], but we can do the equivalent of the following SQL:
SELECT * FROM candidate WHERE candidate.bio #> '{"employment_type": [2]}';
like this:
>>> employment_type = 2
>>> test = {'employment_type': [employment_type]}
>>> query = (
... session.query(Candidate)
... .filter(Candidate.bio.contains(test)))
>>> query.one().bio
{"employment_type": [1, 2, 3]}
Related
When I added new string in models.py his tables did't create in sqlite3. What I do wrong, again :)
python manage.py makemigrations
python manage.py migrate - I did!
It's happend, when I don't input templates information after messege:
Please enter the default value now as valid Python.
The datetime and django.utils.timezone module are available so you can do e.g. timezone.now.
I imputed just 1 and enter. After thet it broke.
If you need different information give me know, please.
Environment:
Request Method: POST
Request URL: http://127.0.0.1:8000/admin/meetups/meetup/add/
Django Version: 3.2.8
Python Version: 3.10.0
Installed Applications:
['django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'meetups']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware']
Traceback (most recent call last):
File "C:\install\Projects_1\env\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\install\Projects_1\env\lib\site-packages\django\db\backends\sqlite3\base.py", line 423, in execute
return Database.Cursor.execute(self, query, params)
The above exception (table meetups_meetup has no column named enormus) was the direct cause of the following exception:
File "C:\install\Projects_1\env\lib\site-packages\django\core\handlers\exception.py", line 47, in inner
response = get_response(request)
File "C:\install\Projects_1\env\lib\site-packages\django\core\handlers\base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "C:\install\Projects_1\env\lib\site-packages\django\contrib\admin\options.py", line 616, in wrapper
return self.admin_site.admin_view(view)(*args, **kwargs)
File "C:\install\Projects_1\env\lib\site-packages\django\utils\decorators.py", line 130, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "C:\install\Projects_1\env\lib\site-packages\django\views\decorators\cache.py", line 44, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "C:\install\Projects_1\env\lib\site-packages\django\contrib\admin\sites.py", line 232, in inner
return view(request, *args, **kwargs)
File "C:\install\Projects_1\env\lib\site-packages\django\contrib\admin\options.py", line 1657, in add_view
return self.changeform_view(request, None, form_url, extra_context)
File "C:\install\Projects_1\env\lib\site-packages\django\utils\decorators.py", line 43, in _wrapper
return bound_method(*args, **kwargs)
File "C:\install\Projects_1\env\lib\site-packages\django\utils\decorators.py", line 130, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "C:\install\Projects_1\env\lib\site-packages\django\contrib\admin\options.py", line 1540, in changeform_view
return self._changeform_view(request, object_id, form_url, extra_context)
File "C:\install\Projects_1\env\lib\site-packages\django\contrib\admin\options.py", line 1586, in _changeform_view
self.save_model(request, new_object, form, not add)
File "C:\install\Projects_1\env\lib\site-packages\django\contrib\admin\options.py", line 1099, in save_model
obj.save()
File "C:\install\Projects_1\env\lib\site-packages\django\db\models\base.py", line 726, in save
self.save_base(using=using, force_insert=force_insert,
File "C:\install\Projects_1\env\lib\site-packages\django\db\models\base.py", line 763, in save_base
updated = self._save_table(
File "C:\install\Projects_1\env\lib\site-packages\django\db\models\base.py", line 868, in _save_table
results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
File "C:\install\Projects_1\env\lib\site-packages\django\db\models\base.py", line 906, in _do_insert
return manager._insert(
File "C:\install\Projects_1\env\lib\site-packages\django\db\models\manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "C:\install\Projects_1\env\lib\site-packages\django\db\models\query.py", line 1270, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "C:\install\Projects_1\env\lib\site-packages\django\db\models\sql\compiler.py", line 1416, in execute_sql
cursor.execute(sql, params)
File "C:\install\Projects_1\env\lib\site-packages\django\db\backends\utils.py", line 98, in execute
return super().execute(sql, params)
File "C:\install\Projects_1\env\lib\site-packages\django\db\backends\utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "C:\install\Projects_1\env\lib\site-packages\django\db\backends\utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "C:\install\Projects_1\env\lib\site-packages\django\db\backends\utils.py", line 79, in _execute
with self.db.wrap_database_errors:
File "C:\install\Projects_1\env\lib\site-packages\django\db\utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "C:\install\Projects_1\env\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\install\Projects_1\env\lib\site-packages\django\db\backends\sqlite3\base.py", line 423, in execute
return Database.Cursor.execute(self, query, params)
Exception Type: OperationalError at /admin/meetups/meetup/add/
Exception Value: table meetups_meetup has no column named enormus
Errors, when I input: makemigration and migrate
(env) PS C:\install\Projects_1> python manage.py makemigrations
System check identified some issues:
WARNINGS:
meetups.Meetup.participant: (fields.W340) null has no effect on ManyToManyField.
You are trying to add a non-nullable field 'dates' to meetup without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
2) Quit, and let me add a default in models.py
Select an option: 1
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now
Type 'exit' to exit this prompt
>>> '2021-10-10'
You are trying to add a non-nullable field 'organizer_emails' to meetup without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
2) Quit, and let me add a default in models.py
Select an option: 1
The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now
Type 'exit' to exit this prompt
>>> test#.test.com
Invalid input: invalid syntax (<string>, line 1)
>>> 'test#.test.com'
Migrations for 'meetups':
meetups\migrations\0011_auto_20211026_2116.py
- Remove field enormus from meetup
- Add field dates to meetup
- Add field organizer_emails to meetup
(env) PS C:\install\Projects_1> python manage.py migrate
System check identified some issues:
WARNINGS:
meetups.Meetup.participant: (fields.W340) null has no effect on ManyToManyField.
Operations to perform:
Apply all migrations: admin, auth, contenttypes, meetups, sessions
Running migrations:
Applying meetups.0005_auto_20211026_1234...Traceback (most recent call last):
File "C:\install\Projects_1\manage.py", line 22, in <module>
main()
File "C:\install\Projects_1\manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "C:\install\Projects_1\env\lib\site-packages\django\core\management\__init__.py", line 419, in execute_from_command_line
utility.execute()
File "C:\install\Projects_1\env\lib\site-packages\django\core\management\__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\install\Projects_1\env\lib\site-packages\django\core\management\base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "C:\install\Projects_1\env\lib\site-packages\django\core\management\base.py", line 398, in execute
output = self.handle(*args, **options)
File "C:\install\Projects_1\env\lib\site-packages\django\core\management\base.py", line 89, in wrapped
res = handle_func(*args, **kwargs)
File "C:\install\Projects_1\env\lib\site-packages\django\core\management\commands\migrate.py", line 244, in handle
post_migrate_state = executor.migrate(
File "C:\install\Projects_1\env\lib\site-packages\django\db\migrations\executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "C:\install\Projects_1\env\lib\site-packages\django\db\migrations\executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "C:\install\Projects_1\env\lib\site-packages\django\db\migrations\executor.py", line 227, in apply_migration
state = migration.apply(state, schema_editor)
File "C:\install\Projects_1\env\lib\site-packages\django\db\migrations\migration.py", line 126, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "C:\install\Projects_1\env\lib\site-packages\django\db\migrations\operations\fields.py", line 104, in database_forwards
schema_editor.add_field(
File "C:\install\Projects_1\env\lib\site-packages\django\db\backends\sqlite3\schema.py", line 330, in add_field
self._remake_table(model, create_field=field)
File "C:\install\Projects_1\env\lib\site-packages\django\db\backends\sqlite3\schema.py", line 191, in _remake_table
self.effective_default(create_field)
File "C:\install\Projects_1\env\lib\site-packages\django\db\backends\base\schema.py", line 324, in effective_default
return field.get_db_prep_save(self._effective_default(field), self.connection)
File "C:\install\Projects_1\env\lib\site-packages\django\db\models\fields\__init__.py", line 842, in get_db_prep_save
return self.get_db_prep_value(value, connection=connection, prepared=False)
File "C:\install\Projects_1\env\lib\site-packages\django\db\models\fields\__init__.py", line 1271, in get_db_prep_value
value = self.get_prep_value(value)
File "C:\install\Projects_1\env\lib\site-packages\django\db\models\fields\__init__.py", line 1266, in get_prep_value
return self.to_python(value)
File "C:\install\Projects_1\env\lib\site-packages\django\db\models\fields\__init__.py", line 1228, in to_python
parsed = parse_date(value)
File "C:\install\Projects_1\env\lib\site-packages\django\utils\dateparse.py", line 75, in parse_date
match = date_re.match(value)
TypeError: expected string or bytes-like object
Using Tensorflow 2.3, I'm trying to create a tf.data.Dataset without labels.
I have my .png files in a folder './Folder/'. For creating the minimal working sample, I think the only relevant line is the one where I am calling tf.keras.preprocessing.image_dataset_from_directory. The class definition is here.
dataset = tf.keras.preprocessing.image_dataset_from_directory('./Folder/',label_mode=None,batch_size=100)
When the Python interpreter reaches the line above, it returns this error message:
Traceback (most recent call last):
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/framework/op_def_library.py", line 465, in _apply_op_helper
values = ops.convert_to_tensor(
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 1473, in convert_to_tensor
raise ValueError(
ValueError: Tensor conversion requested dtype string for Tensor with dtype float32: <tf.Tensor 'args_0:0' shape=() dtype=float32>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "04-vaeAnomalyScores.py", line 135, in <module>
historicKLD, encoder, decoder, vae = artVAE_Instance.run_autoencoder() # Train
File "/media/roi/9b168630-3b62-4215-bb7d-fed9ba179dc7/images/largePatches/artvae.py", line 386, in run_autoencoder
trainingDataSet = self.loadImages(self.trainingDir)
File "/media/roi/9b168630-3b62-4215-bb7d-fed9ba179dc7/images/largePatches/artvae.py", line 231, in loadImages
dataset = tf.keras.preprocessing.image_dataset_from_directory(dir[:-1]+'Downscaled/',label_mode=None,batch_size=self.BATCH_SIZE)
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/keras/preprocessing/image_dataset.py", line 192, in image_dataset_from_directory
dataset = paths_and_labels_to_dataset(
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/keras/preprocessing/image_dataset.py", line 219, in paths_and_labels_to_dataset
img_ds = path_ds.map(
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1695, in map
return MapDataset(self, map_func, preserve_cardinality=True)
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 4041, in __init__
self._map_func = StructuredFunctionWrapper(
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 3371, in __init__
self._function = wrapper_fn.get_concrete_function()
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2938, in get_concrete_function
graph_function = self._get_concrete_function_garbage_collected(
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2906, in _get_concrete_function_garbage_collected
graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 3213, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 3065, in _create_graph_function
func_graph_module.func_graph_from_py_func(
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py", line 986, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 3364, in wrapper_fn
ret = _wrapper_helper(*args)
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 3299, in _wrapper_helper
ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py", line 255, in wrapper
return converted_call(f, args, kwargs, options=options)
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py", line 532, in converted_call
return _call_unconverted(f, args, kwargs, options)
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py", line 339, in _call_unconverted
return f(*args, **kwargs)
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/keras/preprocessing/image_dataset.py", line 220, in <lambda>
lambda x: path_to_image(x, image_size, num_channels, interpolation))
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/keras/preprocessing/image_dataset.py", line 228, in path_to_image
img = io_ops.read_file(path)
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/ops/gen_io_ops.py", line 574, in read_file
_, _, _op, _outputs = _op_def_library._apply_op_helper(
File "/home/roi/.local/lib/python3.8/site-packages/tensorflow/python/framework/op_def_library.py", line 492, in _apply_op_helper
raise TypeError("%s expected type of %s." %
TypeError: Input 'filename' of 'ReadFile' Op has type float32 that does not match expected type of string.
Thank you so much for your help.
One way to fix this I found is to put all your images in another sub-directory inside the directory whose path you are feeding to the image_dataset_from_directory.
Taking your example, you would create a new folder, let's call it new_folder, inside of ./Folder/ where you would put all your images, such that now the path to all your images is ./Folder/new_folder/. Then you can call the image_dataset_from_directory method with the exact same arguments as you have done in your question:
tf.keras.preprocessing.image_dataset_from_directory(
'./Folder/',
label_mode=None,
batch_size=100
)
I found this to work for me so hopefully someone else will also find it helpful!
Others have reported a similar error, but the solutions given do not solve my problem.
For example there is a good answer here. The answer in the link mentions how ndb changes from a first use to a later use and suggests there is a problem because a first run produces a None in the Datastore. I cannot reproduce or see that happening in the Datastore for my sdk, but that may be because I am running it here from the interactive console.
I am pretty sure I got an initial good run with the GAE interactive console, but every run since then has failed with the error in my Title to this question.
I have left the print statements in the following code because they show good results and assure me that the error is occuring in the put() at the very end.
from google.appengine.ext import ndb
class Account(ndb.Model):
week = ndb.IntegerProperty(repeated=True)
weeksNS = ndb.IntegerProperty(repeated=True)
weeksEW = ndb.IntegerProperty(repeated=True)
terry=Account(week=[],weeksNS=[],weeksEW=[])
terry_key=terry.put()
terry = terry_key.get()
print terry
for t in list(range(4)): #just dummy input, but like real input
terry.week.append(t)
print terry.week
region = 1 #same error message for region = 0
if region :
terry.weeksEW.append(terry.week)
else:
terry.weeksNS.append(terry.week)
print 'EW'+str(terry.weeksEW)
print 'NS'+str(terry.weeksNS)
terry.week = []
print 'week'+str(terry.week)
terry.put()
The idea of my code is to first build up the terry.week list values incrementally and then later store the whole list to the appropriate region, either NS or EW. So I'm looking for a workaround for this scheme.
The error message is likely of no value but I am reproducing it here.
Traceback (most recent call last):
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/request_handler.py", line 237, in handle_interactive_request
exec(compiled_code, self._command_globals)
File "<string>", line 55, in <module>
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/model.py", line 3458, in _put
return self._put_async(**ctx_options).get_result()
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/tasklets.py", line 383, in get_result
self.check_success()
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along
value = gen.throw(exc.__class__, exc, tb)
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/context.py", line 824, in put
key = yield self._put_batcher.add(entity, options)
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/tasklets.py", line 430, in _help_tasklet_along
value = gen.send(val)
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/context.py", line 358, in _put_tasklet
keys = yield self._conn.async_put(options, datastore_entities)
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1858, in async_put
pbs = [entity_to_pb(entity) for entity in entities]
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/model.py", line 697, in entity_to_pb
pb = ent._to_pb()
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/model.py", line 3167, in _to_pb
prop._serialize(self, pb, projection=self._projection)
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/model.py", line 1422, in _serialize
values = self._get_base_value_unwrapped_as_list(entity)
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/model.py", line 1192, in _get_base_value_unwrapped_as_list
wrapped = self._get_base_value(entity)
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/model.py", line 1180, in _get_base_value
return self._apply_to_values(entity, self._opt_call_to_base_type)
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/model.py", line 1352, in _apply_to_values
value[:] = map(function, value)
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/model.py", line 1234, in _opt_call_to_base_type
value = _BaseValue(self._call_to_base_type(value))
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/model.py", line 1255, in _call_to_base_type
return call(value)
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/model.py", line 1331, in call
newvalue = method(self, value)
File "/Users/brian/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/model.py", line 1781, in _validate
(value,))
BadValueError: Expected integer, got [0, 1, 2, 3]
I believe the error comes from these lines:
terry.weeksEW.append(terry.week)
terry.weeksNS.append(terry.week)
You are not appending another integer; You are appending a list, when an integer is expected.
>>> aaa = [1,2,3]
>>> bbb = [4,5,6]
>>> aaa.append(bbb)
>>> aaa
[1, 2, 3, [4, 5, 6]]
>>>
This fails the ndb.IntegerProperty test.
Try:
terry.weeksEW += terry.week
terry.weeksNS += terry.week
EDIT: To save a list of lists, do not use the IntegerProperty(), but instead the JsonProperty(). Better still, the ndb datastore is deprecated, so... I recommend Firestore, which uses JSON objects by default. At least use Cloud Datastore, or Cloud NDB.
I have the following POST method in my api. After sending a big JSON collection around 350 entries to the api, I get an DeadlineExceedError exception.
I already am utilising the ndb.multi_put_async. I don't know what else I could do to speed this up?
def post(self):
arguments = self.reqparser.parse_args()
json_records = arguments.get('records')
user = User.query(User.email == request.authorization.username).get()
if user:
records_put_list = []
events_put_list = []
for json_record in json_records:
record_id = json_record['record_id']
rec = Record.get_or_insert(record_id,
user=user.key,
record_date=record_date,
timestamp=record_timestamp)
for json_event in json_record['events']:
event = Event.get_or_insert(json_event['event_id'],
parent=rec.key,
user=user.key,
is_deleted=json_event['is_deleted'])
if event.timestamp < json_event['timestamp']:
event.user = user.key
event.record = rec.key
event.date_time = event_dt
event.timestamp = json_event['timestamp']
event.parent = rec.key
events_put_list.append(event)
ndb.put_multi_async(records_put_list)
ndb.put_multi_async(events_put_list)
return '', 201
else:
return '', 401
This is the exception. Any advice what I could do?
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle
result = handler(dict(self._environ), self._StartResponse)
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask/app.py", line 1836, in __call__
return self.wsgi_app(environ, start_response)
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask_restful/__init__.py", line 397, in wrapper
resp = resource(*args, **kwargs)
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask_restful/__init__.py", line 487, in dispatch_request
resp = meth(*args, **kwargs)
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/application/http_basic_auth.py", line 36, in decorated
return f(*args, **kwargs)
File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/application/rest_api_view.py", line 362, in post
is_deleted=json_event['is_deleted'])
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3401, in _get_or_insert
return cls._get_or_insert_async(*args, **kwds).get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 325, in get_result
self.check_success()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 320, in check_success
self.wait()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 304, in wait
if not ev.run1():
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/eventloop.py", line 235, in run1
delay = self.run0()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/eventloop.py", line 197, in run0
callback(*args, **kwds)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 474, in _on_future_completion
self._help_tasklet_along(ns, ds_conn, gen, val)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 371, in _help_tasklet_along
value = gen.send(val)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/context.py", line 1015, in transaction
ok = yield tconn.async_commit(options)
DeadlineExceededError
You are not using the right logic. Do not put the get_or_insert in the loop - that is never going to work for anything except the smallest number of records (and is generally an anti-pattern to avoid). Instead create the entities and append them to the list inside the loop. Once this is done, the put_multi will then take this list and handle all the puts in parallel. There is a question about how many items in a put_multi list is tolerable, but 350 is far from what I remember as a limit.
You can use a Task API to split this processing job into smaller batches (e.g. 100 entries per task - you can find the optimum number).
I'm trying to get an object from my neo4j database using neo4django
>>> # There is a single Person object in the database, so I get a value
>>> slug=Person.objects.get().name_slug
>>> print(slug)
doe-john
>>> # ok, it's there
>>> p=Person.objects.get(name_slug=slug)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/[...]/src/neo4django/neo4django/db/models/manager.py", line 37, in get
return self.get_query_set().get(*args, **kwargs)
File "/[...]/lib/python2.7/site-packages/django/db/models/query.py", line 366, in get
% self.model._meta.object_name)
DoesNotExist: Person matching query does not exist.
>>> p=Person.objects.get(name_slug__exact=slug)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/[...]/src/neo4django/neo4django/db/models/manager.py", line 37, in get
return self.get_query_set().get(*args, **kwargs)
File "/[...]/lib/python2.7/site-packages/django/db/models/query.py", line 366, in get
% self.model._meta.object_name)
DoesNotExist: Person matching query does not exist.
The error message is not sensible. I just received the queried string from the very field, so there must be a match. Any ideas? Or did I stumble upon a bug?
This is really strange, as it works with the other properties, but not with name_slug:
>>> Person.objects.get(surname='Doe')
<Person: Person object>
>>> Person.objects.get(given_name='John')
<Person: Person object>
>>> Person.objects.get(name_slug='john-doe')
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/[...]/src/neo4django/neo4django/db/models/manager.py", line 37, in get
return self.get_query_set().get(*args, **kwargs)
File "/[...]/lib/python2.7/site-packages/django/db/models/query.py", line 366, in get
% self.model._meta.object_name)
DoesNotExist: Person matching query does not exist.
>>> print( p.surname, p.given_name, p.name_slug )
(u'Doe', u'John', u'john-doe')
My model is defined as such:
class Person(models.NodeModel):
surname = models.StringProperty(required=True, indexed=True)
given_name = models.StringProperty(required=True, indexed=True)
name_slug = models.StringProperty(indexed=True)
So the only difference is that it's not required, but that should make no difference, in my understanding of the documentation.
I can't replicate this using neo4django master, on Neo4j 1.9.
I created a test_models.py
from neo4django.db import models
class Person(models.NodeModel):
class Meta:
# since test_models isn't in an app
app_label='test'
surname = models.StringProperty(required=True, indexed=True)
given_name = models.StringProperty(required=True, indexed=True)
name_slug = models.StringProperty(indexed=True)
and then ran
>>> from test_models import Person
>>> john = Person.objects.create(surname=u'Doe', given_name=u'John', name_slug=u'john-doe')
>>> Person.objects.get(name_slug='john-doe')
<Person: Person object>
>>> john == Person.objects.get(name_slug='john-doe')
True
>>> jane = Person.objects.create(surname=u'Doe', given_name=u'Jane', name_slug=u'jane-doe')
>>> jane == Person.objects.get(name_slug='jane-doe')
True
>>> jane == Person.objects.get(given_name='Jane', surname='Doe')
True
Thoughts?