is there any documentation about kubernetes data structure on etcd? - database

Kubernetes stores all its data on etcd but I can not find any documentation regarding the hierarchy in which those data are stored.
I have been looking around with no success at all so any help will be appreciated.
Thanks in advance.

Kubernetes store the data in the following format:
/registry/<object-type>/<name-space>/<object-name>
Eg:
/registry/pods/default/mypod
To get the details of a particular object:
Syntax:
export ETCDCTL_API=3
sudo etcdctl --endpoints <etcd-server-endpoint-ip> --cert=<client-certificate> --key=<client-key> --cacert=<ca-certificate> get /registry/<object-type>/<name-space>/<object-name>
Example:
sudo etcdctl --cert=/etc/ssl/etcd/ssl/node-kube-master.pem --key=/etc/ssl/etcd/ssl/node-kube-master-key.pem --cacert=/etc/ssl/etcd/ssl/ca.pem get /registry/pods/default/foo
Output:
/registry/pods/default/foo
k8s
v1Pod▒
▒
foodefault"*$2b95568a-d1e8-4c50-9770-fe0a9ba6cbf02▒ˮ▒Z
runfooz▒▒
kubectl-runUpdatev▒ˮ▒FieldsV1:▒
▒{"f:metadata":{"f:labels":{".":{},"f:run":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"foo\"}":{".":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}B▒▒
Go-http-clientUpdatev▒ˮ▒FieldsV1:▒
▒{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.233.120.2\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}Bstatus▒
▒.....redacting more info for clean output
Getting the structure:
Now, if you want to check the structure of the data stored in the etcd use --prefix --keys-only.
sudo etcdctl --cert=<client-certificate> --key=<client-key> --cacert=<ca-certificate> get /registry --prefix --keys-only
Example:
sudo etcdctl --cert=/etc/ssl/etcd/ssl/node-kube-master.pem --key=/etc/ssl/etcd/ssl/node-kube-master-key.pem --cacert=/etc/ssl/etcd/ssl/ca.pem get /registry --prefix --keys-only
/registry/apiregistration.k8s.io/apiservices/v1.
/registry/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.apps
/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.autoscaling
/registry/apiregistration.k8s.io/apiservices/v1.batch
/registry/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io
.....

Related

Robot Framework Parameterizing using yaml file

Hi can anybody help me parameterize the string word so it will fetch from my yaml. I tried to run however I'm getting an error it shows failed: Using YAML variable files requires PyYAML module to be installed. Typically you can install it by running pip install pyyaml. but I already install pyyaml on my local machine. your response is highly appreciated. Thank you so much
Expected Result: ${String} parameter should get the value from my robot.yaml (Ralph) value
VS Terminal Screenshot:
.robot screenshot
robot.yaml file screenshot:
CMD Screenshot:
In robot.yaml define PYTHONPATH like this:
PYTHONPATH:
- .
- string: "RALPH"
Make sure you have installed PyYAML, then include robot.yaml and collections library in the robot file:
Variables path_to_file/robot.yaml
Library Collections
After this you can extract string value inside the test like this:
${value} = pop from dictionary ${PYTHONPATH[1]} string
log to console ${value}
This will print:
RALPH
Second item in PYTHONPATH list is a dictionary, so you first need to access ${PYTHONPATH[1]} and then pop the needed key (in your case string) in order to return its value.

Status 500 error while committing changes to Solr index

I have added files to our Solr index with a curl command:
curl http://<servername>:8983/solr/<collection>/update -H \"Content-Type: text/xml\" --data-binary 1.xml
This ran fine and added the files.
On the Solr UI, in the statistics section for the collection, the "Current" symbol has changed from a green tick to a red circle (indicating there are uncommitted changes).
I'm now trying to commit the changes using the curl:
curl http://<servername>:8983/solr/<collection>/update?commit=true
However this produces the following error response:
{
"responseHeader":{
"status":500,
"QTime":6},
"error":{
"msg":"/opt/solr/solr_dir/solr-8.8.0/server/solr/inc_tickets/data/index/_qz.fdt",
"trace":"java.nio.file.AccessDeniedException: /opt/solr/solr_dir/solr-8.8.0/server/solr/inc_tickets/data/index/_qz.fdt\n\tat sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)\n\tat sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)\n\tat sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)\n\tat sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)\n\tat java.nio.channels.FileChannel.open(FileChannel.java:287)\n\tat java.nio.channels.FileChannel.open(FileChannel.java:335)\n\tat org.apache.lucene.util.IOUtils.fsync(IOUtils.java:469)\n\tat org.apache.lucene.store.FSDirectory.fsync(FSDirectory.java:331)\n\tat org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:286)\n\tat org.apache.lucene.store.NRTCachingDirectory.sync(NRTCachingDirectory.java:158)\n\tat org.apache.lucene.store.LockValidatingDirectoryWrapper.sync(LockValidatingDirectoryWrapper.java:68)\n\tat org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:5103)\n\tat org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3461)\n\tat org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3771)\n\tat org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3729)\n\tat org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:675)\n\tat org.apache.solr.update.processor.RunUpdateProcessorFactory$RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:97)\n\tat org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)\n\tat org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)\n\tat org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1082)\n\tat org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1069)\n\tat org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:169)\n\tat org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)\n\tat org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:76)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:216)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2646)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:794)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:567)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:427)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:357)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)\n\tat org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\n\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\n\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)\n\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)\n\tat java.lang.Thread.run(Thread.java:748)\n",
"code":500}}
The user that Solr is running under doesn't have access to the /opt/solr/solr_dir/... directory.
Give it access or change the owning user for the directory where the index is stored (and the configuration if you want to update it through the API).
You can use chmod to change access rights to a directory or chown <user>.<group> to change who the owner of the directory is. If you include -R it will change the property for the whole hierarchy.

How to deploy snowflake DB with sqitch?

I'm new to sqitch.
I'm trying to create a Jenkins pipeline to deploy snowflake using sqitch.
So, I got sqitch and snowsql installed to my Ubuntu 18.04 LTS in Azure.
I just:
clone existing sqitch repository
.git
.gitignore
sqitch
snowflake
deploy
revert
sqitch
sqitch.conf
sqitch.plan
modify the following config: $WORKSPACE/.snowsql/config
[connections]
accountname = acc_name
region = east-us-2.azure
username = user_name
private_key_path = "/path/rsa_key.p8"
authenticator = SNOWFLAKE_JWT
create sqitch config: $WORKSPACE/sqitch/snowflake/sqitch.conf
[core]
engine = snowflake
[engine "snowflake"]
target = dev
client = snowsql
[target "dev"]
uri = "db:snowflake://client.east-us-2.azure/DEV_DB?Driver=SnowflakeDSIIDriver;warehouse=DEV_WH;authenticator=SNOWFLAKE_JWT;UID=DEV;PRIV_KEY_FILE=/path/rsa_key.p8;PRIV_KEY_FILE_PWD=password;"
Then I try to run "sqitch verify" and get the following output:
Trace begun at /usr/share/perl5/App/Sqitch/Engine.pm line 116
App::Sqitch::Engine::load('App::Sqitch::Engine', 'HASH(0x55ba14ad8ce0)') called at /usr/share/perl5/App/Sqitch/Target.pm line 55
App::Sqitch::Target::__ANON__('App::Sqitch::Target=HASH(0x55ba14ad8470)') called at (eval 278) line 22
App::Sqitch::Target::engine('App::Sqitch::Target=HASH(0x55ba14ad8470)') called at /usr/share/perl5/App/Sqitch/Command/status.pm line 113
App::Sqitch::Command::status::execute(undef) called at /usr/share/perl5/App/Sqitch.pm line 205
App::Sqitch::try {...} at /usr/share/perl5/Try/Tiny.pm line 100
eval {...} at /usr/share/perl5/Try/Tiny.pm line 93
Try::Tiny::try('CODE(0x55ba14acfb70)', 'Try::Tiny::Catch=REF(0x55ba15e7f2b0)') called at /usr/share/perl5/App/Sqitch.pm line 225
App::Sqitch::go('App::Sqitch') called at /usr/bin/sqitch line 14
What am I doing wrong?
For me it looks like the system missing some packages or so.
Sqitch was installed as follows:
sudo apt-get install sqitch libdbd-pg-perl libdbd-odbc-perl
Looks like I needed to do the following:
sudo apt install cpanminus && cpanm App::Sqitch

Unable to install devstack with designate

I am new to the OpenStack environment and started to get into it with a small DevStack setup. I worked the following instructions on a Ubuntu 18.04 machine through and everything worked fine. In order to play with some dns zones I started to research about designate. After adapting the following instructions to my setup I got some errors.
Executing stack.sh produces the following error:
++/opt/stack/designate/devstack/plugin.sh:source:5 set +o xtrace
2021-01-12 21:44:39.009 | Initializing Designate
DROP DATABASE
Could not load 'database': type object 'deprecated' has no attribute 'WALLABY'
Could not load 'pool': type object 'deprecated' has no attribute 'WALLABY'
Could not load 'tlds': type object 'deprecated' has no attribute 'WALLABY'
usage: designate [-h] [--config-dir DIR] [--config-file PATH] [--debug]
[--log-config-append PATH] [--log-date-format DATE_FORMAT]
[--log-dir LOG_DIR] [--log-file PATH] [--nodebug]
[--nouse-journal] [--nouse-json] [--nouse-syslog]
[--nowatch-log-file]
[--syslog-log-facility SYSLOG_LOG_FACILITY] [--use-journal]
[--use-json] [--use-syslog] [--watch-log-file]
{} ...
designate: error: argument category: invalid choice: 'database' (choose from )
Error on exit
World dumping... see /opt/stack/logs/worlddump-2021-01-12-214442.txt for details
nova-compute: no process found
neutron-dhcp-agent: no process found
neutron-l3-agent: no process found
neutron-metadata-agent: no process found
neutron-openvswitch-agent: no process found
I was not sure if my setup was legit. So I tried to use the example config from the designate tutorial. But the same problem occurred.
My actual local.conf:
[[local|localrc]]
USE_PYTHON3=True
ADMIN_PASSWORD=***
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=$ADMIN_PASSWORD
DEST=/opt/stack
SERVICE_HOST=192.168.1.***
HOST_IP=$SERVICE_HOST
disable_service mysql
enable_service postgresql
enable_plugin designate https://opendev.org/openstack/designate
enable_service tempest
Checking the plugin.sh. It looks like the error occurred from this function:
function init_designate {
# (Re)create designate database
recreate_database designate utf8
# Init and migrate designate database
$DESIGNATE_BIN_DIR/designate-manage database sync
init_designate_backend
}
Hope somebody can give me a hint to run DevStack with designate.
Thanks in advance.
The issue you are having is a version mismatch with the cloud install and the designate plugin. Designate is expecting a newer verison of the oslo_log package.
Check that the "devstack" version you have checked out is on the master branch.
The line:
enable_plugin designate https://opendev.org/openstack/designate
Is pulling the master branch of designate for the devstack plugin.
If you are trying to install on a stable branch version OpenStack, you will need to specify a reference for the devstack plugin as well (example, stable/victoria):
enable_plugin designate https://opendev.org/openstack/designate stable/victoria
As mentioned above, you will also need to enable the designate services:
enable_service designate,designate-central,designate-api,designate-worker,designate-producer,designate-mdns

Error in Using YCSB with Gemfire

Hii Am Using YCSB for Benchmarking Pivotal Gemfire My Gemfire server is running properly and by using following command am running the benchmarking test.
bin/ycsb load gemfire -P workloads/workloada -p gemfire.serverhost=x.x.x.x -P gemfire-binding/conf/cache.xml -p gemfire.serverport=40404 -s > load.txt
Loading workload...
Starting test.
0 sec: 0 operations;
Exception in thread "Thread-1" java.lang.IllegalStateException: You must use client-cache in the cache.xml when ClientCacheFactory is used.
at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:316)
at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:274)
at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:3495)
at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:926)
at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.init(GemFireCacheImpl.java:708)
at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:533)
at com.gemstone.gemfire.cache.client.ClientCacheFactory.basicCreate(ClientCacheFactory.java:207)
at com.gemstone.gemfire.cache.client.ClientCacheFactory.create(ClientCacheFactory.java:161)
at com.yahoo.ycsb.db.GemFireClient.init(GemFireClient.java:125)
at com.yahoo.ycsb.DBWrapper.init(DBWrapper.java:63)
at com.yahoo.ycsb.ClientThread.run(Client.java:189)
0 sec: 0 operations;
Please anybody tell can tell me where am going wrong
Thanks in advance
In your GemFireClient.java you are setting up a client cache with the ClientCacheFactory class but the cache.xml that you are giving it for configuration is specifying a cache element instead of a client-cache element. Try changing your cache.xml to use a client-cache similar to the below example. Note that when configuring a client-cache in your cache.xml you will need to use a different DTD than the one used above.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE client-cache PUBLIC
"-//GemStone Systems, Inc.//GemFire Declarative Caching 7.0//EN"
"http://www.gemstone.com/dtd/cache7_0.dtd">
<client-cache copy-on-read="false" >
<region name="usertable" refid="PARTITION"/>
</client-cache>
you do not need to specify a cache.xml when running the YCSB client. The cache.xml in gemfire-binding/conf folder is meant to be supplied to the GemFire server.

Resources