How to set up user interpretation for interpreters when running zeppelin in a docker image? - apache-zeppelin

Question

I'm setting up a docker image on the basis of the zeppelin docker image. Now I'm packing my own configuration into my own docker image. I've got it connected to LDAP for logging in and for user impersonation. Like this, overriding the ZEPPELINIMPERSONATECMD in zeppelin-env.sh.
Running whoami with the sh interpreter works fine now. And running id also shows all the correct user information from ldap.
However when I switch the python interpreter to the isolated per user setting and turn on user interpretation it will fail with a ImportError[1] response in the note itself. The server logging looks like this[2], I replaced my user name with user_name.
I've tried not overwriting the the ZEPPELINIMPERSONATECMD in zeppelin-env.sh. That will just result in a runtime exception[3] when running anything.
I've tried copying the /zeppelin/interpreter/python/py4j-0.9.2/src/py4j folder to /tmp but that will just result in python not responding after 10 seconds.
Does anyone have any ideas on how to run the python interpreter as the logged in user?
[1]:
Traceback (most recent call last):
File "/tmp/zeppelin_python-702917387527627656.py", line 20, in <module>
from py4j.java_gateway import java_import, JavaGateway, GatewayClient
ImportError: No module named py4j.java_gateway
Traceback (most recent call last):
File "/tmp/zeppelin_python-702917387527627656.py", line 20, in <module>
from py4j.java_gateway import java_import, JavaGateway, GatewayClient
ImportError: No module named py4j.java_gateway
python is not responding
[2]:
INFO [2019-08-14 15:13:14,716] ({pool-2-thread-2} ShellScriptLauncher.java[launch]:48) - Launching Interpreter: python
INFO [2019-08-14 15:13:14,727] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:115) - Thrift server for callback will start. Port: 40221
INFO [2019-08-14 15:13:14,738] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:190) - Run interpreter process [/zeppelin/bin/interpreter.sh, -d, /zeppelin/interpreter/python, -c, 172.17.0.3, -p, 40221, -r, :, -u, user_name, -l, /zeppelin/local-repo/python, -g, python]
INFO [2019-08-14 15:13:16,400] ({pool-7-thread-1} RemoteInterpreterManagedProcess.java[callback]:123) - RemoteInterpreterServer Registered: CallbackInfo(host:172.17.0.3, port:40445)
INFO [2019-08-14 15:13:16,440] ({pool-2-thread-2} RemoteInterpreter.java[call]:168) - Create RemoteInterpreter org.apache.zeppelin.python.PythonInterpreter
INFO [2019-08-14 15:13:16,540] ({pool-2-thread-2} RemoteInterpreter.java[call]:168) - Create RemoteInterpreter org.apache.zeppelin.python.IPythonInterpreter
INFO [2019-08-14 15:13:16,544] ({pool-2-thread-2} RemoteInterpreter.java[call]:168) - Create RemoteInterpreter org.apache.zeppelin.python.PythonInterpreterPandasSql
INFO [2019-08-14 15:13:16,545] ({pool-2-thread-2} RemoteInterpreter.java[call]:168) - Create RemoteInterpreter org.apache.zeppelin.python.PythonCondaInterpreter
INFO [2019-08-14 15:13:16,547] ({pool-2-thread-2} RemoteInterpreter.java[call]:168) - Create RemoteInterpreter org.apache.zeppelin.python.PythonDockerInterpreter
INFO [2019-08-14 15:13:16,549] ({pool-2-thread-2} RemoteInterpreter.java[call]:142) - Open RemoteInterpreter org.apache.zeppelin.python.PythonInterpreter
INFO [2019-08-14 15:13:16,549] ({pool-2-thread-2} RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:436) - Push local angular object registry from ZeppelinServer to remote interpreter group python:user_name:
WARN [2019-08-14 15:13:27,703] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2316) - Job 20190814-151311_1784127416 is finished, status: ERROR, exception: null, result: %text Traceback (most recent call last):
File "/tmp/zeppelin_python-4627212054430132450.py", line 20, in <module>
from py4j.java_gateway import java_import, JavaGateway, GatewayClient
ImportError: No module named py4j.java_gateway
%text Traceback (most recent call last):
File "/tmp/zeppelin_python-4627212054430132450.py", line 20, in <module>
from py4j.java_gateway import java_import, JavaGateway, GatewayClient
ImportError: No module named py4j.java_gateway
%text python is not responding
INFO [2019-08-14 15:13:27,713] ({pool-2-thread-2} VFSNotebookRepo.java[save]:196) - Saving note:2EJWQC1Y4
INFO [2019-08-14 15:13:27,715] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:120) - Job 20190814-151311_1784127416 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-python:user_name:-shared_session
[3]:
java.lang.RuntimeException: ssh: connect to host localhost port 22: Cannot assign requested address
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:205)
at org.apache.zeppelin.interpreter.ManagedInterpreterGroup.getOrCreateInterpreterProcess(ManagedInterpreterGroup.java:64)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:111)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:164)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:132)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:299)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:407)
at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Answer 1

I managed to fix this by adding:
export PYTHONPATH=/zeppelin/interpreter/python/py4j-0.9.2/src
in zeppelin-env.sh.
After that the interpreter ran fine.