KiwiTCMS restoring a json backup file fails with "Problem installing fixture '-' error" - kiwi-tcms

I have been using KiwiTCMS docker image installation. There were a crash lately and machine broke down. I do have Kiwi docker image 6.11 and backup for same image available.
Backup is made with command:
docker exec -it kiwi_web /Kiwi/manage.py dumpdata --all --indent 2 > database.json
KiwiTCMS is run up with
docker-compose up -d
Now when trying to restore the backup with sequence (docker-compose.yml is pointing to correct kiwi-web image, and SQL is pointing to kiwi-db 5.5 mariadb image), I get an error
# docker-compose up -d
# docker exec -it kiwi_web /Kiwi/manage.py migrate
# cat database.json | docker exec -i kiwi_web /Kiwi/manage.py loaddata --format json -
Error:
Traceback (most recent call last):
File "/venv/lib/python3.6/site-packages/django/db/models/options.py", line 565, in get_field
return self.fields_map[field_name]
KeyError: 'execution'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/venv/lib/python3.6/site-packages/django/core/serializers/json.py", line 69, in Deserializer
yield from PythonDeserializer(objects, **options)
File "/venv/lib/python3.6/site-packages/django/core/serializers/python.py", line 118, in Deserializer
field = Model._meta.get_field(field_name)
File "/venv/lib/python3.6/site-packages/django/db/models/options.py", line 567, in get_field
raise FieldDoesNotExist("%s has no field named '%s'" % (self.object_name, field_name))
django.core.exceptions.FieldDoesNotExist: LinkReference has no field named 'execution'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Kiwi/manage.py", line 12, in <module>
execute_from_command_line(sys.argv)
File "/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/venv/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/venv/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File "/venv/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
self.loaddata(fixture_labels)
File "/venv/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 114, in loaddata
self.load_label(fixture_label)
File "/venv/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 172, in load_label
for obj in objects:
File "/venv/lib/python3.6/site-packages/django/core/serializers/json.py", line 73, in Deserializer
raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '-':
How could I proceed to save the backupped data somehow?
Thank you very much,
-Mika

FieldDoesNotExist: LinkReference has no field named 'execution'
well this tells you what your error is! You are trying to restore a DB dump from version 6.11 onto a later version and that will not work because there have been [possibly multiple] changes in the database schema meanwhile.
docker-compose.yml is pointing to correct kiwi-web image
This is obviously not the case because LinkReference.execution field appeared in version 7.0, see https://kiwitcms.readthedocs.io/en/latest/changelog.html#id108
So to recap:
you are trying to restore a database dump from an older (or newer) version of Kiwi TCMS while the app server is running an incompatible version. This will not work.
Make sure your docker-compose.yml is using Kiwi TCMS v6.11 and not latest
Your options:
Figure out what fields have changed (from which version is your json dump) and modify the JSON structure to match - time consuming and prone to errors
Get yourself exactly the same old version of Kiwi TCMS and then restore the data onto it.

Related

Can't run Apache Beam on separate Flink cluster. Official instructions don't work

LE: I want to do this on macOS.
I am trying to run Apache Beam on a separate Flink cluster, as described here: https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
(1) Start a Flink cluster which exposes the Rest interface
(e.g. localhost:8081 by default).
Flink 1.13.1 starts successfully on localhost:8081.
(2) Start JobService with Flink Rest endpoint: docker run --net=host apache/beam_flink1.10_job_server:latest --flink-master=localhost:8081.
I am running this instead, which again starts successfully:
docker run --net=host apache/beam_flink1.13_job_server:latest --flink-master=localhost:8081
(3) Submit the pipeline as above.
I'm running the WordCount pipeline as:
python -m apache_beam.examples.wordcount --input /Users/stefan/datastore/input.txt
 --output /Users/stefan/datastore/output.txt
 --runner=PortableRunner --job_endpoint=localhost:8099
After 1 minute the pipeline crashes with a grpc.FutureTimeoutError. This is the full output:
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.8 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.8_sdk:2.31.0
INFO:root:No image given, using default Python SDK image
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.8 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.8_sdk:2.31.0
INFO:root:Python SDK container image set to "apache/beam_python3.8_sdk:2.31.0" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x137d441f0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function lift_combiners at 0x137d44280> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x137d449d0> ====================
Traceback (most recent call last):
File "/usr/local/Cellar/python#3.8/3.8.11/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/Cellar/python#3.8/3.8.11/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/examples/wordcount.py", line 94, in <module>
run()
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/examples/wordcount.py", line 89, in run
output | 'Write' >> WriteToText(known_args.output)
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/pipeline.py", line 585, in __exit__
self.result = self.run()
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/pipeline.py", line 564, in run
return self.runner.run_pipeline(self, self._options)
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/portable_runner.py", line 438, in run_pipeline
job_service_handle = self.create_job_service(options)
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/portable_runner.py", line 317, in create_job_service
return self.create_job_service_handle(server.start(), options)
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/job_server.py", line 54, in start
grpc.channel_ready_future(channel).result(timeout=self._timeout)
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/grpc/_utilities.py", line 140, in result
self._block(timeout)
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/grpc/_utilities.py", line 86, in _block
raise grpc.FutureTimeoutError()
grpc.FutureTimeoutError
Any idea what's going on? Thank you for help. I also opened a bug regarding this issue here: https://issues.apache.org/jira/browse/BEAM-12657
The cluster is started from a docker container. You have to publish the ports so that the Python code executed on the host can access the port.
See https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/docker/ for the --publish option used in the docker run cmd.

django.db.migrations.exceptions.InconsistentMigrationHistory error as part of 6.0.1 to 6.1 upgrade

Environment: running Kiwi in docker containers from Docker repository on Windows Server 2016. I have not made changes to my clone of the Kiwi repo or the docker images.
Repro:
1. Follow upgrade steps on blog to upgrade/migrate from Kiwi 6.0 to 6.0.1
2. Repeat steps to upgrade/migrate from Kiwi 6.0.1 to 6.1
Expect:
No errors
Result:
Exception during migration, I clicked around my Kiwi instance after running the migrate command and didn't see anything obviously wrong with the data or associations between cases, plans and runs. But early days...
Full call stack:
C:\Kiwi>docker exec -it kiwi_web /Kiwi/manage.py migrate
Traceback (most recent call last):
File "/Kiwi/manage.py", line 12, in <module>
execute_from_command_line(sys.argv)
File "/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/venv/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv
self.execute(*args, **cmd_options)
File "/venv/lib/python3.6/site-packages/django/core/management/base.py", line 353, in execute
output = self.handle(*args, **options)
File "/venv/lib/python3.6/site-packages/django/core/management/base.py", line 83, in wrapped
res = handle_func(*args, **kwargs)
File "/venv/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 85, in handle
executor.loader.check_consistent_history(connection)
File "/venv/lib/python3.6/site-packages/django/db/migrations/loader.py", line 297, in check_consistent_history
connection.alias,
django.db.migrations.exceptions.InconsistentMigrationHistory: Migration management.0013_remove_env is applied before its dependency testruns.0019_remove_testruntag_user on database 'default'.
Argh, this shouldn't happen.
Repro:
1. Follow upgrade steps on blog to upgrade/migrate from Kiwi 6.0 to 6.0.1
Can you describe the steps exactly ? If you did a git pull then your docker-compose.yml points to kiwitcms/kiwi:latest and you have to manually modify that for the intermediate upgrade step.
Repeat steps to upgrade/migrate from Kiwi 6.0.1 to 6.1
When did you get the traceback, before step 2 or after step 2 ?
Note: This is similar to https://github.com/kiwitcms/Kiwi/issues/561 and there isn't much we can do about it really.
To debug (I hope you have a backup) you have to revert back to the previous version you had (6.0 as you say), then upgrade to 6.0.1 and post ./manage.py showmigrations. Then from that try to upgrade to 6.1 and again inspect showmigrations to see if everything is applied.
Note: in the case of traceback during migrate migrations have not been applied so it should be safe to attemp migrate again once you know which is the starting version and target version you are upgrading to.

gcloud preview app run: "OSError: [Errno 2] No such file or directory"

When running gcloud preview app run from Google Compute Engine, running Ubuntu Trust 14.04 image, I get the following error:
(Note: I fetched the complete gcloud SDK from https://cloud.google.com/sdk/)
$ gcloud preview app run appengine-modules-guestbook/src/main/webapp
Java module found in [/home/yaraju_gmail_com/appengine-modules-sample-java/appengine-modules-guestbook/src/main/webapp]
INFO: Skipping SDK update check.
INFO: Starting API server at: http://localhost:46920
INFO: Starting module "default" running at: http://localhost:8080
INFO: Starting admin server at: http://localhost:8000
Exception in thread Instance Adjustment:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/yaraju_gmail_com/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/module.py", line 1492, in _loop_adjusting_instances
self._adjust_instances()
File "/home/yaraju_gmail_com/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/module.py", line 1469, in _adjust_instances
self._add_instance(permit_warmup=True)
File "/home/yaraju_gmail_com/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/module.py", line 1354, in _add_instance
if not inst.start():
File "/home/yaraju_gmail_com/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/instance.py", line 274, in start
self._runtime_proxy.start()
File "/home/yaraju_gmail_com/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/http_runtime.py", line 270, in start
stderr=subprocess.PIPE)
File "/home/yaraju_gmail_com/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/safe_subprocess.py", line 132, in start_process_file
stderr=stderr)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
What can I do to fix this?
You might be missing one of the dependencies.
Ensure that you have run the following:
sudo apt-get install openjdk-7-jdk
gcloud components update gae-java

SSL certificates on Google App Engine

I'm having issues getting started using GAE (1.7.6 on OS X 10.6.8 with Python 2.7) and compared to amazon services it is abysmal. However, I would like to at least get the sample app deployed. After working through the other issues I now have a small sample application ready to deploy. When I try to deploy it, I receive an SSL error as follows:
*** Running appcfg.py with the following flags:
--no_cookies --email=mathsboy#gmail.com --passin update
02:15 PM Host: appengine.google.com
02:15 PM Application: tew-helloworld; version: 1
Traceback (most recent call last):
File "google_appengine/appcfg.py", line 171, in <module>
run_file(__file__, globals())
File "google_appengine/appcfg.py", line 167, in run_file
execfile(script_path, globals_)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4155, in <module>
main(sys.argv)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4146, in main
result = AppCfgApp(argv).Run()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 2334, in Run
self.action(self)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 3881, in __call__
return method()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 2892, in Update
updatecheck.CheckForUpdates()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/sdk_update_checker.py", line 258, in CheckForUpdates
runtime=runtime))
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 393, in Send
f = self.opener.open(req)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 394, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 412, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1207, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/fancy_urllib/fancy_urllib/__init__.py", line 383, in do_open
url_error.reason.args[1])
fancy_urllib.InvalidCertificateException: Host appengine.google.com returned an invalid certificate (_ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed):
To learn more, see http://code.google.com/appengine/kb/general.html#rpcssl
If deploy fails you might need to 'rollback' manually.
The "Make Symlinks..." menu option can help with command-line work.
*** appcfg.py has finished with exit code 1 ***
There was some advice on this in another thread stating that you would need to remove cacerts.txt, though I am unable to locate this file on OS X. Also, it seems retarded that in order to have the code upload I'd have to delete a cacert text file. But maybe I'm an idiot.
Another problem I'm facing here, is that OS X doesn't work well with GAE due to the difficulty in managing python versions. I've got a total mess on my hands now after installing multiple versions of python, EPD, ActiveState, MacPorts. None of these worked in solving me earlier problems despite assurances they would. It's really pretty appalling that this is the state of google's application management. It just seems like GAE was built by a bunch of guys from mid-90s microsoft....anyone advice on the above issue would be greatly appreciated. Cheers
Fourth line from end in your error. Relevant instructions under the link.
http://code.google.com/appengine/kb/general.html#rpcssl
Make sure (from documentation):
To do this you must have the ssl Python module installed on your system.
Workaround: Use the command line argument --skip_sdk_update_check=yes when you run dev_appserver.py.
I had the same problem and noticed the GUI tool (GoogleAppEngineLauncher) uses --skip_sdk_update_check=yes every time you start an app. This is as of SDK 1.8.7.
The root cause seems to be a known bug in the GAE development server's module whiltelisting system. Even if you have the ssl module installed (as the documentation says is required), ssl relies on _ssl, which is not whitelisted. This is why you can execute import ssl in a regular python shell session with no problem, but get an error when you execute the same line in a GAE app.

Fatal devappserver2 error: PYTHON_EGG_CACHE in Appengine 1.7.6 and 1.7.5 python 2.7, win7

I attempted to use Appengine SDK version 1.7.6 but was receiving fatal errors (same as outlined below) so reverted to 1.7.5, where my app works fine. However, the errors I saw in 1.7.6 recur when I attempt to run the experimental devappserver in 1.7.5. I am running Python 2.7 on a win 7 machine. Standard install selections for AppEngine.
These error messages refer to permissions relating to the PYTHON_EGG_CACHE, but...
- 1) They do not occur for the 'old' dev_appserver
- 2) The application can successfully extract files to the location noted. In fact, I changed the path of the PYTHON_EGG_CACHE to another location, gave it wide open permissions, and still received the error.
- 3) I have pyyaml in my site-packages, so am not sure why it would need to do a temporary extract anyway.
Any help looking for a resolution to this would be appreciated, since I'll be forced to use the new devappserver in a couple of releases time.
Command line error dump below:
python "C:\Program Files (x
86)\Google\google_appengine\devappserver2.py" .
WARNING 2013-03-25 19:10:41,029 devappserver2.py:497] devappserver2.py is curre
ntly experimental but will eventually replace dev_appserver.py in the App Engine
Python SDK. For more information and to report bugs, please see: http://code.go
ogle.com/p/appengine-devappserver2-experiment/
INFO 2013-03-25 19:10:41,171 appcfg.py:618] Checking for updates to the SDK.
WARNING 2013-03-25 19:10:44,525 simple_search_stub.py:975] Could not read searc
h indexes from c:\users\ben\appdata\local\temp\appengine.dev~upskillme-main.Ben\
search_indexes
INFO 2013-03-25 19:10:44,539 api_server.py:148] Starting API server at: http
://localhost:49302
INFO 2013-03-25 19:10:44,555 dispatcher.py:96] Starting server "default" run
ning at: http://localhost:8080
INFO 2013-03-25 19:10:44,571 admin_server.py:112] Starting admin server at:
http://localhost:8000
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\_python_runtime.py", line
150, in <module>
run_file(__file__, globals())
File "C:\Program Files (x86)\Google\google_appengine\_python_runtime.py", line
146, in run_file
execfile(script_path, globals_)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
vappserver2\python\runtime.py", line 39, in <module>
from google.appengine.ext.remote_api import remote_api_stub
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\remo
te_api\remote_api_stub.py", line 75, in <module>
import yaml
File "C:\Program Files (x86)\Google\google_appengine\lib\yaml-3.10\yaml\__init
__.py", line 14, in <module>
from cyaml import *
File "C:\Program Files (x86)\Google\google_appengine\lib\yaml-3.10\yaml\cyaml.
py", line 5, in <module>
from _yaml import CParser, CEmitter
File "C:\Python27\lib\site-packages\pyyaml-3.10-py2.7-win32.egg\_yaml.py", lin
e 7, in <module>
File "C:\Python27\lib\site-packages\pyyaml-3.10-py2.7-win32.egg\_yaml.py", lin
e 4, in __bootstrap__
File "C:\Python27\lib\site-packages\pkg_resources.py", line 882, in resource_f
ilename
self, resource_name
File "C:\Python27\lib\site-packages\pkg_resources.py", line 1351, in get_resou
rce_filename
self._extract_resource(manager, self._eager_to_zip(name))
File "C:\Python27\lib\site-packages\pkg_resources.py", line 1406, in _extract_
resource
manager.extraction_error() # report a user-friendly error
File "C:\Python27\lib\site-packages\pkg_resources.py", line 928, in extraction
_error
raise err
pkg_resources.ExtractionError: Can't extract file(s) to egg cache
The following error occurred while trying to extract file(s) to the Python egg
cache:
[Error 5] Access is denied: 'C:\\Users\\Ben\\AppData\\Roaming\\Python-Eggs\\py
yaml-3.10-py2.7-win32.egg-tmp\\_yaml.pyd'
The Python egg cache directory is currently set to:
C:\Users\Ben\AppData\Roaming\Python-Eggs
Perhaps your account does not have write access to this directory? You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.
Exception in thread Thread-4:
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 551, in __bootstrap_inner
self.run()
File "C:\Python27\lib\threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
vappserver2\server.py", line 1045, in _loop_adjusting_instances
self._adjust_instances()
If anyone comes along with the same issue, the resolution that worked for me was as follows:
Unzip the PyYAML-3.10-py2.7-win32.egg in your python27 site-packages
directory into a new directory within site-packages. You'll need to
unzip it to into a directory name like NEWPyYAML-3.10-py2.7-win32.egg
to avoid name conflicts.
Rename your original egg. to something else (e.g., OLDPyYAML-3.10-py2.7-win32.egg).
Rename your new directory to PyYAML-3.10-py2.7-win32.egg
All references to the egg will now actually point to a directory in your site-packages containing the unzipped python files. The new devappserver should be able to work with this.
Unsure how to unzip an egg in windows? See here (presuming you have Winzip or similar installed):
http://mail.python.org/pipermail/chicago/2007-July/002301.html

Resources