Running migration in a docker kiwitcms/kiwi:latest (v7.2 digest d757b56bc10c) image based docker container.
During migration testcases.0010_remove_bug fails with some database constraint problem.
Is this a bug in the migration script or an issue with data not consistent?
Operations to perform:
Apply all migrations: admin, attachments, auth, bugs, contenttypes, core, django_comments, kiwi_auth, linkreference, management, sessions, sites, testcases, testplans, testruns
Running migrations:
Applying testcases.0010_remove_bug...Traceback (most recent call last):
File "/venv/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute
return self.cursor.execute(query, args)
File "/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute
res = self._query(query)
File "/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query
db.query(q)
File "/venv/lib/python3.6/site-packages/MySQLdb/connections.py", line 239, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1048, "Column 'execution_id' cannot be null")
This is a bug in the migration script. The problem lies in the fact that prior to these changes you could attach bugs directly to test cases, without an actual test execution. This resulted in the case_run_id field being None.
With the new changes bugs can only be attached to test executions and their execution_id field should always have a value.
Can you try this patch and let me know if it works for you:
https://github.com/kiwitcms/Kiwi/pull/1266
Related
For my volttron agent that I used the agent creation wizard to develop, can I get a tip on an error related to this , 'Timezone offset does not match system offset: -18000 != 0. Please, check your config files.'
When testing my script with the from volttron.platform.scheduling import cron feature I noticed the timezone/computer time was way off on my edge device so I reset the time zone with this tutorial which I am thinking definitely screwed things up.
ERROR: volttron.platform.jsonrpc.RemoteError: builtins.ValueError('Timezone offset does not match system offset: -18000 != 0. Please, check your config files.')
Whether or not this makes a difference this edge device does use the fowarding agent to push the data to a central VOLTTRON instance.
2021-05-14 12:45:00,007 (actuatoragent-1.0 313466) volttron.platform.vip.agent.subsystems.rpc ERROR: unhandled exception in JSON-RPC method 'request_new_schedule':
Traceback (most recent call last):
File "/var/lib/volttron/volttron/platform/vip/agent/subsystems/rpc.py", line 158, in method
return method(*args, **kwargs)
File "/home/volttron/.volttron/agents/8f4ee1c0-74cb-4070-8a8c-57bf9bea8a71/actuatoragent-1.0/actuator/agent.py", line 1343, in request_new_schedule
return self._request_new_schedule(rpc_peer, task_id, priority, requests, publish_result=False)
File "/home/volttron/.volttron/agents/8f4ee1c0-74cb-4070-8a8c-57bf9bea8a71/actuatoragent-1.0/actuator/agent.py", line 1351, in _request_new_schedule
local_tz = get_localzone()
File "/var/lib/volttron/env/lib/python3.8/site-packages/tzlocal/unix.py", line 165, in get_localzone
_cache_tz = _get_localzone()
File "/var/lib/volttron/env/lib/python3.8/site-packages/tzlocal/unix.py", line 90, in _get_localzone
utils.assert_tz_offset(tz)
File "/var/lib/volttron/env/lib/python3.8/site-packages/tzlocal/utils.py", line 46, in assert_tz_offset
raise ValueError(msg)
ValueError: Timezone offset does not match system offset: -18000 != 0. Please, check your config files.
This is my raise_setpoints_up function below that is alot like the CSV driver agent code.
def raise_setpoints_up(self):
_log.info(f'*** [Setter Agent INFO] *** - STARTING raise_setpoints_up function!')
schedule_request = []
# create start and end timestamps
_now = get_aware_utc_now()
str_start = format_timestamp(_now)
_end = _now + td(seconds=10)
str_end = format_timestamp(_end)
# wrap the topic and timestamps up in a list and add it to the schedules list
for device in self.jci_device_map.values():
topic = '/'.join([self.building_topic, device])
schedule_request.append([topic, str_start, str_end])
# send the request to the actuator
result = self.vip.rpc.call('platform.actuator', 'request_new_schedule', self.core.identity, 'my_schedule', 'HIGH', schedule_request).get(timeout=4)
_log.info(f'*** [Setter Agent INFO] *** - actuator agent scheduled sucess!')
Thanks for any tips
I suspect the time configured by tzdata is different than the timezone configured by the system since you changed this manually. Give this a try:
sudo dpkg-reconfigure tzdata
use sudo dpkg-reconfigure tzdata to change back to UTC time
(Submitting on behalf of a Snowflake user)
At the time of query execution on Snowflake, I need its query id. So I am using following code snippet:
cursor.execute(query, _no_results=True)
query_id = cursor.sfqid
cursor.query_result(query_id)
This code snippet working fine for small running queries. But for query which takes more than 40-45 seconds to execute, query_result function fails with KeyError u'rowtype'.
Stack trace:
File "snowflake/connector/cursor.py", line 631, in query_result
self._init_result_and_meta(data, _use_ijson)
File "snowflake/connector/cursor.py", line 591, in _init_result_and_meta
for column in data[u'rowtype']:
KeyError: u'rowtype'
Why would this error occur? How to solve this problem?
Any recommendations? Thanks!
The Snowflake Python Connector allows for async SQL execution by using cur.execute(sql, _no_results=True)
This "fire and forget" style of SQL execution allows for the parent process to continue without waiting for the SQL command to complete (think long-running SQL that may time-out).
If this is used, many developers will write code that captures the unique Snowflake Query ID (like you have in your code) and then use that Query ID to "check back on the query status later", in some sort of looping process. When you check back to get the status, you can then get the results from that query_id using the result_scan( ) function.
https://docs.snowflake.net/manuals/sql-reference/functions/result_scan.html
I hope this helps...Rich
Stream Analytics job ( iot hub to CosmosDB output) "Start" command is failing with the following error.
[12:49:30 PM] Source 'cosmosiot' had 1 occurrences of kind
'OutputDataConversionError.RequiredColumnMissing' between processing
times '2019-04-17T02:49:30.2736530Z' and
'2019-04-17T02:49:30.2736530Z'.
I followed the instructions and not sure what is causing this error.
Any suggestions please? Here is the CosmosDB Query:
SELECT
[bearings temperature],
[windings temperature],
[tower sway],
[position sensor],
[blade strain gauge],
[main shaft strain gauge],
[shroud accelerometer],
[gearbox fluid levels],
[power generation],
[EventProcessedUtcTime],
[EventEnqueuedUtcTime],
[IoTHub].[CorrelationId],
[IoTHub].[ConnectionDeviceId]
INTO
cosmosiot
FROM
TurbineData
If you're specifying fields in your query (ie Select Name, ModelNumber ...) rather than just using Select * ... the field names are converted to lowercase by default when using Compatibility Level 1.0, which throws off Cosmos DB. In the portal if you open your Stream Analytics job and go to 'Compatibility level' under the 'Configure' section and select v1.1 or higher that should fix the issue. You can read more about the compatibility levels in the Stream Analytics documentation here: https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-compatibility-level
I'm trying to execute the code below. Some times it works fine. But some times it does not work.
#db.transactional
def _add_data_to_site(self, key):
site = models.Site.get_by_key_name('s:%s' % self.site_id)
if not site:
site = models.Site()
if key not in site.data:
site.data.append(key)
site.put()
memcache.delete_multi(['', ':0', ':1'], key_prefix='s%s' %
self.site_id)
I'm getting the error:
File "/base/data/home/apps/xxxxxxx/1-7-1.366398694339889874xxxxxxx.py", line 91, in _add_data_to_site
site.put()
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1070, in put
return datastore.Put(self._entity, **kwargs)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 579, in Put
return PutAsync(entities, **kwargs).get_result()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 604, in get_result
return self.__get_result_hook(self)
File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1569, in __put_hook
self.check_rpc_success(rpc)
File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1224, in check_rpc_success
raise _ToDatastoreError(err)
BadRequestError: cross-group transaction need to be explicitly specified, see TransactionOptions.Builder.withXG
So, my question is:
If I'm changing only one entity (models.Site) why am I getting a cross-group transaction error?
As mentioned in the logs: "Cross-group transaction need to be explicitly specified".
Try specifying it by using
#db.transactional(xg=True)
Instead of:
#db.transactional
Does this work if you specify parent=None in your get_by_key_name() query?
Essentially, in order to use a transaction, all entities in the transaction must share the same parent (ie you query using one parent, and create a new entity withe the same parent), or you must use a XG transaction. You're seeing a problem because you didn't specify a parent.
You may need to create artificial entities to behave as parents in order to do what you're trying to do.
I had the same issue. By stepping through the client code, I made the following two observations:
1) Setting a parent of (None) seems to still indicate a parent of that kind, even if there's no specific record elected as that parent.
2) Your transaction will include all ReferenceProperty properties as well.
Therefore, you should, theoretically, get the cross-group transaction exception if you haven't declared a parent (by either omitting or setting to (None)) on any of the kinds that you're affecting if there's at least two (because if you're using kind A and kind B, it looks like you're using two different entity groups, for A records and for B records), -as well as- any of the kinds referred-to by any ReferenceProperty properties.
To fix this, you must create, at least, a kind without any properties, that can be set as the parent of all of your previously no-parent records, as well as the parent of all ReferenceProperty properties that they declare.
If that's not sufficient, then set the flag for the cross-group transaction.
Also, the text of the exception, for me, was: "cross-groups transaction need to be explicitly specified" (plural "groups"). I have version 1.7.6 of the Python AppEngine client.
Please upvote this answer if it fits your scenario.
A cross group transaction error refers to the entity groups being used, not the kind used (here Site).
When it occurs, it's because you are attempting a transaction on entities with different parents, hence putting them in different entity groups.
SHAMELESS PLUG:
You should stop using db and move your code to ndb, especially since it seems you're in the development phase.
This is a common exception I'm getting on my application's log daily, usually 5/6 times a day with a traffic of 1K visits/day:
db error trying to store stats
Traceback (most recent call last):
File "/base/data/home/apps/stackprinter/1b.347728306076327132/app/utility/worker.py", line 36, in deferred_store_print_statistics
dbcounter.increment()
File "/base/data/home/apps/stackprinter/1b.347728306076327132/app/db/counter.py", line 28, in increment
db.run_in_transaction(txn)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 1981, in RunInTransaction
DEFAULT_TRANSACTION_RETRIES, function, *args, **kwargs)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 2067, in RunInTransactionCustomRetries
ok, result = _DoOneTry(new_connection, function, args, kwargs)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 2105, in _DoOneTry
if new_connection.commit():
File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1585, in commit
return rpc.get_result()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 530, in get_result
return self.__get_result_hook(self)
File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1613, in __commit_hook
raise _ToDatastoreError(err)
Timeout: The datastore operation timed out, or the data was temporarily unavailable.
The function that is raising the exception above is the following one:
def store_printed_question(question_id, service, title):
def _store_TX():
entity = Question.get_by_key_name(key_names = '%s_%s' % \
(question_id, service ) )
if entity:
entity.counter = entity.counter + 1
entity.put()
else:
Question(key_name = '%s_%s' % (question_id, service ),\
question_id ,\
service,\
title,\
counter = 1).put()
db.run_in_transaction(_store_TX)
Basically, the store_printed_question function check if a given question was previously printed, incrementing in that case the related counter in a single transaction.
This function is added by a WebHandler to a deferred worker using the predefined default queue that, as you might know, has a throughput rate of five task invocations per second.
On a entity with six attributes (two indexes) I thought that using transactions regulated by a deferred task rate limit would allow me to avoid datastore timeouts but, looking at the log, this error is still showing up daily.
This counter I'm storing is not so much important, so I'm not worried about getting these timeouts; anyway I'm curious why Google App Engine can't handle this task properly even at a low rate like 5 tasks per second and if lowering the rate could be a possible solution.
A sharded counter on each question to avoid timeouts would be an overkill to me.
EDIT:
I have set the rate limit to 1 task per second on the default queue; I'm still getting the same error.
A query can only live for 30 seconds. See my answer to this question for some sample code to break a query up using cursors.
Generally speaking, a timeout like this is usually because of write contention. If you've got a transaction going and you're writing a bunch of stuff to the same entity group concurrently, you run into write contention issues (a side effect of optimistic concurrency). In most cases, if you make your entity group smaller, that will usually minimize this problem.
In your specific case, based on the code above, it's most probably because you should be using a sharded counter to avoid stacking of serialized writes.
Another far less likely possibility (mentioned here only for completeness) is that the tablet your data is on is being moved.