How to deploy snowflake DB with sqitch? - snowflake-cloud-data-platform

I'm new to sqitch.
I'm trying to create a Jenkins pipeline to deploy snowflake using sqitch.
So, I got sqitch and snowsql installed to my Ubuntu 18.04 LTS in Azure.
I just:
clone existing sqitch repository
.git
.gitignore
sqitch
snowflake
deploy
revert
sqitch
sqitch.conf
sqitch.plan
modify the following config: $WORKSPACE/.snowsql/config
[connections]
accountname = acc_name
region = east-us-2.azure
username = user_name
private_key_path = "/path/rsa_key.p8"
authenticator = SNOWFLAKE_JWT
create sqitch config: $WORKSPACE/sqitch/snowflake/sqitch.conf
[core]
engine = snowflake
[engine "snowflake"]
target = dev
client = snowsql
[target "dev"]
uri = "db:snowflake://client.east-us-2.azure/DEV_DB?Driver=SnowflakeDSIIDriver;warehouse=DEV_WH;authenticator=SNOWFLAKE_JWT;UID=DEV;PRIV_KEY_FILE=/path/rsa_key.p8;PRIV_KEY_FILE_PWD=password;"
Then I try to run "sqitch verify" and get the following output:
Trace begun at /usr/share/perl5/App/Sqitch/Engine.pm line 116
App::Sqitch::Engine::load('App::Sqitch::Engine', 'HASH(0x55ba14ad8ce0)') called at /usr/share/perl5/App/Sqitch/Target.pm line 55
App::Sqitch::Target::__ANON__('App::Sqitch::Target=HASH(0x55ba14ad8470)') called at (eval 278) line 22
App::Sqitch::Target::engine('App::Sqitch::Target=HASH(0x55ba14ad8470)') called at /usr/share/perl5/App/Sqitch/Command/status.pm line 113
App::Sqitch::Command::status::execute(undef) called at /usr/share/perl5/App/Sqitch.pm line 205
App::Sqitch::try {...} at /usr/share/perl5/Try/Tiny.pm line 100
eval {...} at /usr/share/perl5/Try/Tiny.pm line 93
Try::Tiny::try('CODE(0x55ba14acfb70)', 'Try::Tiny::Catch=REF(0x55ba15e7f2b0)') called at /usr/share/perl5/App/Sqitch.pm line 225
App::Sqitch::go('App::Sqitch') called at /usr/bin/sqitch line 14
What am I doing wrong?
For me it looks like the system missing some packages or so.
Sqitch was installed as follows:
sudo apt-get install sqitch libdbd-pg-perl libdbd-odbc-perl

Looks like I needed to do the following:
sudo apt install cpanminus && cpanm App::Sqitch

Related

Setup Xdebug for Shopware docker failed

I try to setup Xdebug for shopware-docker without success.
VHOST_[FOLDER_NAME_UPPER_CASE]_IMAGE=ghcr.io/shyim/shopware-docker/6/nginx:php74-xdebug
After replacing your Folder Name and running swdc up Xdebug should be activated.
Which folder name should I place?
Using myname, the same name as in /var/www/html/myname, return error on swdc up myname:
swdc up myname
[+] Running 2/0
⠿ Network shopware-docker_default Created 0.0s
⠿ Container shopware-docker-mysql-1 Created 0.0s
[+] Running 1/1
⠿ Container shopware-docker-mysql-1 Started 0.3s
.database ready!
[+] Running 0/1
⠿ app_myname Error 1.7s
Error response from daemon: manifest unknown
EDIT #1
With this setup VHOST_MYNAME_IMAGE=ghcr.io/shyim/shopware-docker/6/nginx:php81-xdebug (versioned Xdebug) the app started:
// $HOME/.config/swdc/env
...
VHOST_MYNAME_IMAGE=ghcr.io/shyim/shopware-docker/6/nginx:php81-xdebug
But set a debug breakpoint (e.g. in index.php), nothing happens
EDIT #2
As #Alex recommend, i place xdebug_break() inside my code and it works.
Stopping on the breakpoint the debugger log aswers with hints/warnings like described in the manual:
...
Cannot find a local copy of the file on server /var/www/html/%my_path%
Local path is //var/www/html/%my_path%
...
click on Click to set up path mapping to open the modal
click inside modal select input Use path mapping (...)
input field File path in project response with undefined
But i have already set up the mapping like described in the manual, go to File | Settings | PHP | Servers:
Why does not work my mapping? Where failed my set up?
The path mapping needs to be between your local project path on your workstation and the path inside the docker containers. Without xDebug has a hard time mapping the breakpoints from PHPStorm to the actual code inside the container.
If mapping the path correctly does not work and if its a possibility for you, i can highly recommend switching to http://devenv.sh for your development enviroment. Shopware itself promotes this new enviroment in their documentation: https://developer.shopware.com/docs/guides/installation/devenv and provides an example on how to enable xdebug:
# devenv.local.nix File
{ pkgs, config, lib, ... }:
{
languages.php.package = pkgs.php.buildEnv {
extensions = { all, enabled }: with all; enabled ++ [ amqp redis blackfire grpc xdebug ];
extraConfig = ''
# Copy the config from devenv.nix and append the XDebug config
# [...]
xdebug.mode=debug
xdebug.discover_client_host=1
xdebug.client_host=127.0.0.1
'';
};
}
A correct path mapping should not be needed here, as your local file location is the same for XDebug and your PHPStorm.

is there any documentation about kubernetes data structure on etcd?

Kubernetes stores all its data on etcd but I can not find any documentation regarding the hierarchy in which those data are stored.
I have been looking around with no success at all so any help will be appreciated.
Thanks in advance.
Kubernetes store the data in the following format:
/registry/<object-type>/<name-space>/<object-name>
Eg:
/registry/pods/default/mypod
To get the details of a particular object:
Syntax:
export ETCDCTL_API=3
sudo etcdctl --endpoints <etcd-server-endpoint-ip> --cert=<client-certificate> --key=<client-key> --cacert=<ca-certificate> get /registry/<object-type>/<name-space>/<object-name>
Example:
sudo etcdctl --cert=/etc/ssl/etcd/ssl/node-kube-master.pem --key=/etc/ssl/etcd/ssl/node-kube-master-key.pem --cacert=/etc/ssl/etcd/ssl/ca.pem get /registry/pods/default/foo
Output:
/registry/pods/default/foo
k8s
v1Pod▒
▒
foodefault"*$2b95568a-d1e8-4c50-9770-fe0a9ba6cbf02▒ˮ▒Z
runfooz▒▒
kubectl-runUpdatev▒ˮ▒FieldsV1:▒
▒{"f:metadata":{"f:labels":{".":{},"f:run":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"foo\"}":{".":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}B▒▒
Go-http-clientUpdatev▒ˮ▒FieldsV1:▒
▒{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.233.120.2\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}Bstatus▒
▒.....redacting more info for clean output
Getting the structure:
Now, if you want to check the structure of the data stored in the etcd use --prefix --keys-only.
sudo etcdctl --cert=<client-certificate> --key=<client-key> --cacert=<ca-certificate> get /registry --prefix --keys-only
Example:
sudo etcdctl --cert=/etc/ssl/etcd/ssl/node-kube-master.pem --key=/etc/ssl/etcd/ssl/node-kube-master-key.pem --cacert=/etc/ssl/etcd/ssl/ca.pem get /registry --prefix --keys-only
/registry/apiregistration.k8s.io/apiservices/v1.
/registry/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.apps
/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.autoscaling
/registry/apiregistration.k8s.io/apiservices/v1.batch
/registry/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io
.....

How to use better-sqlite3 with electron-react-boilerplate

Consider this example created by using the following steps:
git clone --depth=1 https://github.com/electron-react-boilerplate/electron-react-boilerplate.git better-sqlite3-test
cd better-sqlite3-test
yarn
cd ./release/app
yarn add better-sqlite3
cd ../..
yarn add -D #types/better-sqlite3
Now, as soon as I start using the database.js like this:
class B3SqliteDB {
constructor() {
let db = null;
}
startDB = () => {
const Database = require('better-sqlite3');
this.db = new Database('upStore.db', { verbose: console.log });
const createTable =
"CREATE TABLE IF NOT EXISTS newTable ('id' VARCHAR(10) NOT NULL, 'name' VARCHAR(50) NOT NULL);";
this.db.exec(createTable);
};
}
export default new B3SqliteDB;
I am getting the error:
Uncaught TypeError: Database is not a constructor
at B3SqliteDB.startDB (renderer.dev.js:65346)
at renderer.dev.js:65285
Can someone please help to figure this out! TYA
This worked for me
Downgrade electron to 13.6
Install better-sqlite3 in root
Rebuild the package
Test
yarn remove electron
yarn add electron#13.6.1
yarn add better-sqlite3;
cd node_modules/better-sqlite3;
../.bin/electron-rebuild
cd ../..
npm run start
enter image description here
Note
You can try other desktop options (Electron Alternatives) that support integration ReactJS | VueJS just fine
Wails (Golang)
https://github.com/wailsapp/wails
Lorka (Golang)
https://github.com/zserge/lorca
Tauri (Rust)
https://github.com/tauri-apps/tauri
Flutter Desktop (Dart)
https://flutter.dev/desktop
What I love is that the app size is just 10 - 20 times smaller than a sample Hello World Electron App.

Zeppelin authentication with Jdbc realm

I have been trying to set up zeppelin with authentication with Shiro JDBC realm. After all my attempts, I have not been able to get it working. The basic authentication works but with JDBC realm it fails.
The zeppelin server was created following the doc: http://zeppelin.apache.org/docs/0.9.0/quickstart/kubernetes.html
The POD is working.
I enabled the Shiro by extending the docker image. My Dockerfile:
ARG ZEPPELIN_IMAGE=apache/zeppelin:0.9.0
FROM ${ZEPPELIN_IMAGE}
#https://hub.docker.com/r/apache/zeppelin/dockerfile
WORKDIR ${Z_HOME}
ADD /zeppelin/shiro.ini ${Z_HOME}/conf/
ADD https://repo1.maven.org/maven2/mysql/mysql-connector-java/6.0.4/mysql-connector-java-6.0.4.jar ${Z_HOME}/lib/
ENV CLASSPATH=${Z_HOME}/lib/mysql-connector-java-6.0.4.jar:${CLASSPATH}
ENTRYPOINT [ "/usr/bin/tini", "--" ]
WORKDIR ${Z_HOME}
CMD ["bin/zeppelin.sh"]
My shiro.ini taken from https://gist.github.com/adamjshook/6c42b03fdb09b60cd519174d0aec1af5
[main]
ds = com.mysql.jdbc.jdbc2.optional.MysqlDataSource
ds.serverName = localhost
ds.databaseName = zeppelin
ds.user = zeppelin
ds.password = zeppelin
jdbcRealm = org.apache.shiro.realm.jdbc.JdbcRealm
jdbcRealmCredentialsMatcher = org.apache.shiro.authc.credential.Sha256CredentialsMatcher
jdbcRealm.credentialsMatcher = $jdbcRealmCredentialsMatcher
ps = org.apache.shiro.authc.credential.DefaultPasswordService
pm = org.apache.shiro.authc.credential.PasswordMatcher
pm.passwordService = $ps
jdbcRealm.dataSource = $ds
jdbcRealm.credentialsMatcher = $pm
shiro.loginUrl = /api/login
[urls]/** = authc
Now, when I deploy the zeppelin server, I get:
rg.apache.shiro.config.ConfigurationException: Unable to instantiate class [com.mysql.jdbc.jdbc2.optional.MysqlDataSource] for object named 'ds'. Please ensure you've specified the fully qualified class name correctly.
at org.apache.shiro.config.ReflectionBuilder.createNewInstance(ReflectionBuilder.java:327)
at org.apache.shiro.config.ReflectionBuilder$InstantiationStatement.doExecute(ReflectionBuilder.java:961)
at org.apache.shiro.config.ReflectionBuilder$Statement.execute(ReflectionBuilder.java:921)
at org.apache.shiro.config.ReflectionBuilder$BeanConfigurationProcessor.execute(ReflectionBuilder.java:799)
at org.apache.shiro.config.ReflectionBuilder.buildObjects(ReflectionBuilder.java:278)
at org.apache.shiro.config.IniSecurityManagerFactory.buildInstances(IniSecurityManagerFactory.java:181)
at org.apache.shiro.config.IniSecurityManagerFactory.createSecurityManager(IniSecurityManagerFactory.java:139)
at org.apache.shiro.config.IniSecurityManagerFactory.createSecurityManager(IniSecurityManagerFactory.java:107)
at org.apache.shiro.config.IniSecurityManagerFactory.createInstance(IniSecurityManagerFactory.java:98)
at org.apache.shiro.config.IniSecurityManagerFactory.createInstance(IniSecurityManagerFactory.java:47)
at org.apache.shiro.config.IniFactorySupport.createInstance(IniFactorySupport.java:150)
at org.apache.shiro.util.AbstractFactory.getInstance(AbstractFactory.java:47)
Caused by: org.apache.shiro.util.UnknownClassException: Unable to load class named [com.mysql.jdbc.jdbc2.optional.MysqlDataSource] from the thread context, current, or system/application ClassLoaders. All heuristics have been exhausted. Class could not be found.
at org.apache.shiro.util.ClassUtils.forName(ClassUtils.java:152)
at org.apache.shiro.util.ClassUtils.newInstance(ClassUtils.java:168)
at org.apache.shiro.config.ReflectionBuilder.createNewInstance(ReflectionBuilder.java:320)
... 40 more
Not sure why it is failing even I have defined the jar file on classpath.
Issue with jar was not having the right permissions. Got it fixed with below Dockerfile
ARG ZEPPELIN_IMAGE=apache/zeppelin:0.9.0
FROM ${ZEPPELIN_IMAGE}
#https://hub.docker.com/r/apache/zeppelin/dockerfile
WORKDIR ${Z_HOME}
USER root
ADD /zeppelin/shiro.ini ${Z_HOME}/conf/
ADD https://repo1.maven.org/maven2/mysql/mysql-connector-java/6.0.4/mysql-connector-java-6.0.4.jar ${Z_HOME}/lib/
ENV CLASSPATH=${Z_HOME}/lib/mysql-connector-java-6.0.4.jar:${CLASSPATH}
RUN chmod 777 ${Z_HOME}/lib/mysql-connector-java-6.0.4.jar
USER 1000
ENTRYPOINT [ "/usr/bin/tini", "--" ]
WORKDIR ${Z_HOME}
CMD ["bin/zeppelin.sh"]

Logstash Configuration Error - jdbc_driver_library is not set

I am using Logstash to move data from my Microsoft SQL Server database to ElasticSearch. I receive the following error in the log files when I try to run logstash.
I run:
sudo -Hu logstash /usr/share/logstash/bin/logstash --path.settings=/etc/logstash -t
Error:
Exception: LogStash::ConfigurationError
Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:163:in `open_jdbc_connection'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:221:in `execute_statement'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:277:in `execute_query'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:263:in `run'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:309:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:302:in `block in start_input'
[2019-09-18T00:16:36,250][ERROR][logstash.javapipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Jdbc jdbc_user=>[user_name], jdbc_password=><password>, statement=>"SELECT * FROM [table] WHERE delete_flag = 'N'", jdbc_connection_string=>"jdbc:sqlserver://[IP]:1433;databaseName=[database];", id=>"04af4c4719615e3a3a03c3df3e5262aa40c40e85fb55a5886a1797d42eb3a729", jdbc_driver_class=>"com.microsoft.sqlserver.jdbc.SQLServerDriver", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_76e4bd7a-70dd-46cf-acd3-c3726a8e003f", enable_metric=>true, charset=>"UTF-8">, jdbc_paging_enabled=>false, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, sql_log_level=>"info", connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, parameters=>{"sql_last_value"=>1970-01-01 00:00:00 UTC}, last_run_metadata_path=>"/usr/share/logstash/.logstash_jdbc_last_run", use_column_value=>false, tracking_column_type=>"numeric", clean_run=>false, record_last_run=>true, lowercase_column_names=>true>
Error: com.microsoft.sqlserver.jdbc.SQLServerDriver not loaded. :jdbc_driver_library is not set, are you sure you included
the proper driver client libraries in your classpath?
I am running ElasticSearch/Logstash on an Ubuntu 18 server. I installed ElasticSearch, Logstash, Java 11, and downloaded the jdbc and placed it in the same folder as my logstash config.
My Logstash Config is:
input {
jdbc {
jdbc_driver_library => "/etc/logstash/conf.d/sqljdbc42.jar"
jdbc_connection_string => "jdbc:sqlserver://[my_ip]:1433;databaseName=[db_name];"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_user => "[username]"
jdbc_password => "[password]"
schedule => "0 4 * * * America/New_York"
statement => "SELECT * FROM [table] WHERE delete_flag = 'N'"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "products"
}
}
I specified the jdbc driver library, not sure why I keep getting an error why logstash can't find it.
I also tried to run (below)
sudo bash -c "export CLASSPATH=.:/etc/logstash/conf.d/sqljdbc42.jar"
Note: I took out some data, ip, dbname, username, etc.
This has been an issue with logstash-filter-jdbc_static for a while now. You need to move your .jar to /logstash-core/lib/jars/ directory. And specify the .jar location in your config as "".
See below for more details:
https://github.com/logstash-plugins/logstash-filter-jdbc_static/issues/47

Resources