Zeppelin authentication with Jdbc realm - apache-zeppelin

I have been trying to set up zeppelin with authentication with Shiro JDBC realm. After all my attempts, I have not been able to get it working. The basic authentication works but with JDBC realm it fails.
The zeppelin server was created following the doc: http://zeppelin.apache.org/docs/0.9.0/quickstart/kubernetes.html
The POD is working.
I enabled the Shiro by extending the docker image. My Dockerfile:
ARG ZEPPELIN_IMAGE=apache/zeppelin:0.9.0
FROM ${ZEPPELIN_IMAGE}
#https://hub.docker.com/r/apache/zeppelin/dockerfile
WORKDIR ${Z_HOME}
ADD /zeppelin/shiro.ini ${Z_HOME}/conf/
ADD https://repo1.maven.org/maven2/mysql/mysql-connector-java/6.0.4/mysql-connector-java-6.0.4.jar ${Z_HOME}/lib/
ENV CLASSPATH=${Z_HOME}/lib/mysql-connector-java-6.0.4.jar:${CLASSPATH}
ENTRYPOINT [ "/usr/bin/tini", "--" ]
WORKDIR ${Z_HOME}
CMD ["bin/zeppelin.sh"]
My shiro.ini taken from https://gist.github.com/adamjshook/6c42b03fdb09b60cd519174d0aec1af5
[main]
ds = com.mysql.jdbc.jdbc2.optional.MysqlDataSource
ds.serverName = localhost
ds.databaseName = zeppelin
ds.user = zeppelin
ds.password = zeppelin
jdbcRealm = org.apache.shiro.realm.jdbc.JdbcRealm
jdbcRealmCredentialsMatcher = org.apache.shiro.authc.credential.Sha256CredentialsMatcher
jdbcRealm.credentialsMatcher = $jdbcRealmCredentialsMatcher
ps = org.apache.shiro.authc.credential.DefaultPasswordService
pm = org.apache.shiro.authc.credential.PasswordMatcher
pm.passwordService = $ps
jdbcRealm.dataSource = $ds
jdbcRealm.credentialsMatcher = $pm
shiro.loginUrl = /api/login
[urls]/** = authc
Now, when I deploy the zeppelin server, I get:
rg.apache.shiro.config.ConfigurationException: Unable to instantiate class [com.mysql.jdbc.jdbc2.optional.MysqlDataSource] for object named 'ds'. Please ensure you've specified the fully qualified class name correctly.
at org.apache.shiro.config.ReflectionBuilder.createNewInstance(ReflectionBuilder.java:327)
at org.apache.shiro.config.ReflectionBuilder$InstantiationStatement.doExecute(ReflectionBuilder.java:961)
at org.apache.shiro.config.ReflectionBuilder$Statement.execute(ReflectionBuilder.java:921)
at org.apache.shiro.config.ReflectionBuilder$BeanConfigurationProcessor.execute(ReflectionBuilder.java:799)
at org.apache.shiro.config.ReflectionBuilder.buildObjects(ReflectionBuilder.java:278)
at org.apache.shiro.config.IniSecurityManagerFactory.buildInstances(IniSecurityManagerFactory.java:181)
at org.apache.shiro.config.IniSecurityManagerFactory.createSecurityManager(IniSecurityManagerFactory.java:139)
at org.apache.shiro.config.IniSecurityManagerFactory.createSecurityManager(IniSecurityManagerFactory.java:107)
at org.apache.shiro.config.IniSecurityManagerFactory.createInstance(IniSecurityManagerFactory.java:98)
at org.apache.shiro.config.IniSecurityManagerFactory.createInstance(IniSecurityManagerFactory.java:47)
at org.apache.shiro.config.IniFactorySupport.createInstance(IniFactorySupport.java:150)
at org.apache.shiro.util.AbstractFactory.getInstance(AbstractFactory.java:47)
Caused by: org.apache.shiro.util.UnknownClassException: Unable to load class named [com.mysql.jdbc.jdbc2.optional.MysqlDataSource] from the thread context, current, or system/application ClassLoaders. All heuristics have been exhausted. Class could not be found.
at org.apache.shiro.util.ClassUtils.forName(ClassUtils.java:152)
at org.apache.shiro.util.ClassUtils.newInstance(ClassUtils.java:168)
at org.apache.shiro.config.ReflectionBuilder.createNewInstance(ReflectionBuilder.java:320)
... 40 more
Not sure why it is failing even I have defined the jar file on classpath.

Issue with jar was not having the right permissions. Got it fixed with below Dockerfile
ARG ZEPPELIN_IMAGE=apache/zeppelin:0.9.0
FROM ${ZEPPELIN_IMAGE}
#https://hub.docker.com/r/apache/zeppelin/dockerfile
WORKDIR ${Z_HOME}
USER root
ADD /zeppelin/shiro.ini ${Z_HOME}/conf/
ADD https://repo1.maven.org/maven2/mysql/mysql-connector-java/6.0.4/mysql-connector-java-6.0.4.jar ${Z_HOME}/lib/
ENV CLASSPATH=${Z_HOME}/lib/mysql-connector-java-6.0.4.jar:${CLASSPATH}
RUN chmod 777 ${Z_HOME}/lib/mysql-connector-java-6.0.4.jar
USER 1000
ENTRYPOINT [ "/usr/bin/tini", "--" ]
WORKDIR ${Z_HOME}
CMD ["bin/zeppelin.sh"]

Related

Setup Xdebug for Shopware docker failed

I try to setup Xdebug for shopware-docker without success.
VHOST_[FOLDER_NAME_UPPER_CASE]_IMAGE=ghcr.io/shyim/shopware-docker/6/nginx:php74-xdebug
After replacing your Folder Name and running swdc up Xdebug should be activated.
Which folder name should I place?
Using myname, the same name as in /var/www/html/myname, return error on swdc up myname:
swdc up myname
[+] Running 2/0
⠿ Network shopware-docker_default Created 0.0s
⠿ Container shopware-docker-mysql-1 Created 0.0s
[+] Running 1/1
⠿ Container shopware-docker-mysql-1 Started 0.3s
.database ready!
[+] Running 0/1
⠿ app_myname Error 1.7s
Error response from daemon: manifest unknown
EDIT #1
With this setup VHOST_MYNAME_IMAGE=ghcr.io/shyim/shopware-docker/6/nginx:php81-xdebug (versioned Xdebug) the app started:
// $HOME/.config/swdc/env
...
VHOST_MYNAME_IMAGE=ghcr.io/shyim/shopware-docker/6/nginx:php81-xdebug
But set a debug breakpoint (e.g. in index.php), nothing happens
EDIT #2
As #Alex recommend, i place xdebug_break() inside my code and it works.
Stopping on the breakpoint the debugger log aswers with hints/warnings like described in the manual:
...
Cannot find a local copy of the file on server /var/www/html/%my_path%
Local path is //var/www/html/%my_path%
...
click on Click to set up path mapping to open the modal
click inside modal select input Use path mapping (...)
input field File path in project response with undefined
But i have already set up the mapping like described in the manual, go to File | Settings | PHP | Servers:
Why does not work my mapping? Where failed my set up?
The path mapping needs to be between your local project path on your workstation and the path inside the docker containers. Without xDebug has a hard time mapping the breakpoints from PHPStorm to the actual code inside the container.
If mapping the path correctly does not work and if its a possibility for you, i can highly recommend switching to http://devenv.sh for your development enviroment. Shopware itself promotes this new enviroment in their documentation: https://developer.shopware.com/docs/guides/installation/devenv and provides an example on how to enable xdebug:
# devenv.local.nix File
{ pkgs, config, lib, ... }:
{
languages.php.package = pkgs.php.buildEnv {
extensions = { all, enabled }: with all; enabled ++ [ amqp redis blackfire grpc xdebug ];
extraConfig = ''
# Copy the config from devenv.nix and append the XDebug config
# [...]
xdebug.mode=debug
xdebug.discover_client_host=1
xdebug.client_host=127.0.0.1
'';
};
}
A correct path mapping should not be needed here, as your local file location is the same for XDebug and your PHPStorm.

Phpstan doctrine database connection error

I'm using doctrine in a project (not symfony). In this project I also use phpstan, i installed both phpstan/phpstan-doctrine and phpstan/extension-installer.
My phpstan.neon is like this:
parameters:
level: 8
paths:
- src/
doctrine:
objectManagerLoader: tests/object-manager.php
Inside tests/object-manager.php it return the result of a call to a function that return the entity manager.
Here is the code that create the entity manager
$database_url = $_ENV['DATABASE_URL'];
$isDevMode = $this->isDevMode();
$proxyDir = null;
$cache = null;
$useSimpleAnnotationReader = false;
$config = Setup::createAnnotationMetadataConfiguration(
[$this->getProjectDirectory() . '/src'],
$isDevMode,
$proxyDir,
$cache,
$useSimpleAnnotationReader
);
// database configuration parameters
$conn = [
'url' => $database_url,
];
// obtaining the entity manager
$entityManager = EntityManager::create($conn, $config);
When i run vendor/bin/phpstan analyze i get this error:
Internal error: An exception occurred in the driver: SQLSTATE[08006] [7] could not translate host name "postgres_db" to address: nodename nor servname provided, or not known
This appear because i'm using docker and my database url is postgres://user:password#postgres_db/database postgres_db is the name of my database container so the hostname is known inside the docker container.
When i run phpstan inside the container i do not have the error.
So is there a way to run phpstan outside docker ? Because i'm pretty sure that when i'll push my code the github workflow will fail because of this
Do phpstan need to try to reach the database ?
I opened an issue on the github of phpstan-doctrine and i had an answer by #jlherren that explained :
The problem is that Doctrine needs to know what version of the DB server it is working with in order to instantiate the correct AbstractPlatform implementation, of which there are several available for the same DB vendor (e.g. PostgreSQL94Platform or PostgreSQL100Platform for postgres, and similarly for other DB drivers). To auto-detect this information, it will simply connect to the DB and query the version.
I just changed my database url from:
DATABASE_URL=postgres://user:password#database_ip/database
To:
DATABASE_URL=postgres://user:password#database_ip/database?serverVersion=14.2

How to deploy snowflake DB with sqitch?

I'm new to sqitch.
I'm trying to create a Jenkins pipeline to deploy snowflake using sqitch.
So, I got sqitch and snowsql installed to my Ubuntu 18.04 LTS in Azure.
I just:
clone existing sqitch repository
.git
.gitignore
sqitch
snowflake
deploy
revert
sqitch
sqitch.conf
sqitch.plan
modify the following config: $WORKSPACE/.snowsql/config
[connections]
accountname = acc_name
region = east-us-2.azure
username = user_name
private_key_path = "/path/rsa_key.p8"
authenticator = SNOWFLAKE_JWT
create sqitch config: $WORKSPACE/sqitch/snowflake/sqitch.conf
[core]
engine = snowflake
[engine "snowflake"]
target = dev
client = snowsql
[target "dev"]
uri = "db:snowflake://client.east-us-2.azure/DEV_DB?Driver=SnowflakeDSIIDriver;warehouse=DEV_WH;authenticator=SNOWFLAKE_JWT;UID=DEV;PRIV_KEY_FILE=/path/rsa_key.p8;PRIV_KEY_FILE_PWD=password;"
Then I try to run "sqitch verify" and get the following output:
Trace begun at /usr/share/perl5/App/Sqitch/Engine.pm line 116
App::Sqitch::Engine::load('App::Sqitch::Engine', 'HASH(0x55ba14ad8ce0)') called at /usr/share/perl5/App/Sqitch/Target.pm line 55
App::Sqitch::Target::__ANON__('App::Sqitch::Target=HASH(0x55ba14ad8470)') called at (eval 278) line 22
App::Sqitch::Target::engine('App::Sqitch::Target=HASH(0x55ba14ad8470)') called at /usr/share/perl5/App/Sqitch/Command/status.pm line 113
App::Sqitch::Command::status::execute(undef) called at /usr/share/perl5/App/Sqitch.pm line 205
App::Sqitch::try {...} at /usr/share/perl5/Try/Tiny.pm line 100
eval {...} at /usr/share/perl5/Try/Tiny.pm line 93
Try::Tiny::try('CODE(0x55ba14acfb70)', 'Try::Tiny::Catch=REF(0x55ba15e7f2b0)') called at /usr/share/perl5/App/Sqitch.pm line 225
App::Sqitch::go('App::Sqitch') called at /usr/bin/sqitch line 14
What am I doing wrong?
For me it looks like the system missing some packages or so.
Sqitch was installed as follows:
sudo apt-get install sqitch libdbd-pg-perl libdbd-odbc-perl
Looks like I needed to do the following:
sudo apt install cpanminus && cpanm App::Sqitch

drone ci publish generated latex pdf

actually I am using travis but I want to change to drone.
For all tex documents I'm using a small Makefile with a Container to generate my pdf file and deploy it on my repository.
But since I'm using gitea I want to set up my integration pipeline with drone, but I don't know how I can configure the .drone.yml to deploy my pdf file on every tag als release.
Actually I'm using the following .drone.yml and I am happy the say, that's build process works fine at the moment.
clone:
git:
image: plugins/git
tags: true
pipeline:
pdf:
image: volkerraschek/docker-latex:latest
pull: true
commands:
- make
and this is my Makefile
# Docker Image
IMAGE := volkerraschek/docker-latex:latest
# Input tex-file and output pdf-file
FILE := index
TEX_NAME := ${FILE}.tex
PDF_NAME := ${FILE}.pdf
latexmk:
latexmk \
-shell-escape \
-synctex=1 \
-interaction=nonstopmode \
-file-line-error \
-pdf ${TEX_NAME}
docker-latexmk:
docker run \
--rm \
--user="$(shell id -u):$(shell id -g)" \
--net="none" \
--volume="${PWD}:/data" ${IMAGE} \
make latexmk
Which tags and conditions are missing in my drone.yml to deploy my index.pdf as release in gitea when I push a new git tag?
Volker
I have this setup on my gitea / drone pair. This is a MWE of my .drone.yml:
pipeline:
build:
image: tianon/latex
commands:
- pdflatex <filename.tex>
gitea_release:
image: plugins/gitea-release
base_url: <gitea domain>
secrets: [gitea_token]
files: <filename.pdf>
when:
event: tag
So rather than setting up the docker build in the Makefile, we add a step using docker image with latex, compile the pdf, and use a pipeline step to release.
You'll also have to set your drone repo to trigger builds on a tags and set a gitea API token to use. To set the API token, you can use the command line interface:
$ drone secret add <org/repo> --name gitea_token --value <token value> --image plugins/gitea-release
You can set up the drone repo to trigger builds in the repository settings in the web UI.
Note that you'll also likely have to allow *.pdf attachments in your gitea settings, as they are disallowed by default. In your gitea app.ini add this to the attachment section:
[attachment]
ENABLED = true
PATH = /data/gitea/attachments
MAX_SIZE = 10
ALLOWED_TYPES = */*
In addition to Gabe's answer, if you are using an NGINX reverse proxy, you might also have to allow larger file uploads in your nginx.conf. (This applies to all file types, not just .pdf)
server {
[ ... ]
location / {
client_max_body_size 10M; # add this line
proxy_pass http://gitea:3000;
}
}
This fixed the problem for me.

RM + DSC to node in untrusted domain

So I mention the untrusted domain aspect because I went through all the hoops around credential delegation and trusted hosts lists etc to allow me to successfully push a DSC configuration from my RM server to a target node (not using RM, just native DSC). I get that bit and it works, great.
Now when I use those same scripts in RM (with some minor edits for the format expected by RM), RM reports a successful deploy but all that has happened is the components bits have been copied to the target node to the default location for $applicationPathRoot (C:\Windows\DtlDownloads), there is no real evidence of an attempt to apply a mof file.
My RM server and target nodes are in different domains with no trust. Both servers are W2k8R2 (+ WMF4 of course). I'm running with Update 4 of RM server and client.
Here are the DSC scripts I'm running in RM:
CopyDSCResources.ps1
Configuration CopyDSCResource
{
param (
[Parameter(Mandatory=$false)]
[ValidateNotNullOrEmpty()]
[String] $ModulePath = "$env:ProgramFiles\WindowsPowershell\Modules")
#[PSCredential] $credential = get-credential
Node VCTSCFDSMWEB01
{
File DeployWebDeployResource
{
Ensure = "Present"
SourcePath = "C:\test.txt"
DestinationPath = "D:\temp"
Force = $true
Type = "File"
}
}
}
CopyDSCResource -ConfigurationData $configData -Verbose
# test outside of RM
#CopyDSCResource -ConfigurationData CopyDSCResource.ConfigData.psd1
#Start-DscConfiguration -Path .\CopyDSCResource -Credential $credential -Verbose -Wait
CopyDSCResource.ConfigData.psd1
##{
$configData = #{
AllNodes = #(
#{
NodeName = "*"
PSDscAllowPlainTextPassword = $true
},
#{
NodeName = "VCTSCFDSWEB01.rlg.test"
Role = "WebServer"
}
)
}
I'm afraid I cant seem to upload screenshots from my current location but in terms of RM, I have a vNext environment with a single server linked, a vNext release path with a single 'Dev' stage and a vNext release template with a single 'Deploy PS/DSC' action. The configuration of the action is:
ServerName - VCTSCFDSMWEB01
ComponentName - COpyDSCResource vNext
PSScriptPath - copydscresources.ps1
PSConfigurationPath - copydscresource.configdata.psd1
UseCredSSP - true
When I run a new release, the deploy stage reports success and when I view the Deployment log files I get the following:
Upload components - Successfully uploaded to the normalized store.
Deploy Using PS/DSC - Copying recursively from \vcxxxxtfs03\Drops\CorrespondenceCI\CorrespondenceCI20150114.1\Scripts to C:\Windows\DtlDownloads\CopyDSCResource vNext succeeded.
Finally the DSC event log has the following:
Job {CD3BE350-4072-4C8B-835F-4B4D1C46D65D} :
Configuration is sent from computer NULL by user sid S-1-5-18.
This compares markedly to the same event log entry when run outside of RM:
Job {34F78498-CF18-4F2A-9874-EB54FDA2D990} :
Configuration is sent from computer VCXXXXTFS01 by user sid S-1-5-21-1034805355-1149422947-1317505720-10867.
Any pointers appreciated
It would be good if I could see evidence of a mof file being created on the RM server for example, anybody know where I can find this??
Turns out the crucial element was that my DSC script had to use an environment variable for naming the node. So:
Node $env:COMPUTERNAME
No idea why but it works!

Resources