I developed a Flink job and submitted my job using Apache Flink dashboard. Per my understanding, when I submit my job, my jar should be available on Flink server. I tried to figure out path of my jar but couldn't able to. Does Flink keep these jar file on server? If yes, where I can find? Any documentation? Please help. Thanks!
JAR files are renamed when they are uploaded and stored in a directory that can be configured with the web.upload.dir configuration key.
If the web.upload.dir parameter is not set, the JAR files are stored in a dynamically generated directory under the jobmanager.web.tmpdir (default is System.getProperty("java.io.tmpdir")).
Related
I want to create a Apache Flink standalone cluster with serveral taskmanagers. I would like to use HDFS and Hive. Therefore i have to add some Hadoop dependencies.
After reading the documentation, the recommended way is to set the HADOOP_CLASSPATH env variable. But how do i have to add the hadoop files? Should i download the source files in some directory like /opt/hadoop ont the taskmanagers and set the variable to this path?
I only know the old but deprecated way downloading a Uber-Jar with the dependencies and place it under the /lib folder.
Normally you'd do the standard Hadoop installation, since you (for HDFS) need Node Managers running on every server (with appropriate configuration), plus the NameNode running on your master server.
So then you can do something like this on the master server where you're submitting your Flink workflow:
export HADOOP_CLASSPATH=`hadoop classpath`
export HADOOP_CONF_DIR=/etc/hadoop/conf
I'm new to flink and trying to submit my flink program to my flink cluster.
I have a flink cluster running on remote kubernetes and a blob storage on Azure.
I know how to submit a flink job when I have the jar file on my local machine but no idea how to submit the job with the remote jar file(the jar can be access by https)
checked the documents and it seems doesn't provide something like what we do in spark
Thanks in advance
I think you can use an init container to download the job jar into a shared volume, then submit the local jar to Flink.
Ads: Google's Flink Operator supports remote job jar, see this example.
I have flink server running on Kubernetes cluster. I have a job jar which is bulky due to product and third party dependencies.
I run it via
ExecutionEnvironment env = ExecutionEnvironment.createRemoteEnvironment(host, port, jar);
The jar size is around 130 MB after optimization.
I want to invoke the remoteExecution without jar upload so that the upload does not happen everytime when the job needs to be executed. Is there a way to upload the jar once and call it remotely without mentioning the jar (in java)?
You could deploy a per job cluster on Kubernetes. This will submit your user code jar along with the Flink binaries to your Kubernetes cluster. The downside is that you cannot change the job afterwards without restarting the Flink cluster.
Following is my use-case:
I have started using Camunda Platform and am accessing the Camunda process engine using a Custom Application written in java. So basically there are 2 applications or war files in Tomcat server. Initially I was using the inbuilt h2 database for the Camunda application and SQL server database for my Custom Application and was using integrated authentication mechanism which worked perfectly fine. Now I have replicated the h2 database to SQL server database and am again using integrated authentication for the process engine to connect to SQL Server.
Problem:
After deploying the 2 wars, when I restart the Tomcat server, the sqljdbc_auth.dll present in the tomcat bin folder gets loaded successfully by the Camunda application and process engine successfully accepts requests, accesses the database and gives correct responses on the Camunda Web applications(Cockpit,Tasklist,Admin) but when I try to login on my Custom application then I get the following error:
"
null.null Failed to load the sqljdbc_auth.dll cause : Native Library C:\Users\Aakanksha\Desktop\BACKUP\$CAMUNDA_HOME\server\apache-tomcat-8.0.24\bin\sqljdbc_auth.dll already loaded in another classloader
"
I do understand why this is happening and have followed the following solutions already:
Sol.1 -
Added the sqljdbc4.jar file to $TOMCAT_HOME/lib folder
Added the sqljdbc_auth.dll file to $TOMCAT_HOME/bin folder
Sol.2 - Added sqljdbc4.jar and sqljdbc_auth.dll file to separate war files i.e.
WEBINF/lib folders.
Sol.3 -
Removed the dll file from $TOMCAT_HOME/bin folder and added the same to Windows/System32
Added this path to the PATH Environment variable
Sol.4 -
Added the dll file to Java/JDK/bin folder.
Sol.1, Sol.3 and Sol.4 - The dll files was loaded and used successfully by one war but not the other with the same error.
Sol.2 - lead to error "
com.microsoft.sqlserver.jdbc.AuthenticationJNI. Failed to load the sqljdbc_auth.dll cause : no sqljdbc_auth in java.library.path
"
Both of my applications are running at the same time and have different SQL Server databases to make connections to. Is it even possible for 2 wars running at the same instant to use a shared dll for making connection to different databases?
Kindly share your suggestions and ideas.
I was having the same issue. We changed our war files to use jndi lookups from tomcat configuration. That works great we now can deploy multiple war files.
For more information see
http://www.baeldung.com/spring-persistence-jpa-jndi-datasource
Johan
How do I update an existing configuration file of SolrCloud in the Zoo Keeper?
I am using Solr4 Beta version with ZooKeeper 3.3.6. I have updated a configuration file, and restarted the Solr Instance which uploads the configuration file to the ZooKeeper. But when I check the configuration file from the SolrCloud Admin console, I don't see the updates. I am not able to understand if this is an issue with SolrCloud admin console or if I am not successful in uploading the config file to ZooKeeper.
Can someone who is familiar with ZooKeeper tell me on how to update an existing configuration file in the ZooKeeper, and how to verify the change in the ZooKeeper?
Solr 4 comes with some helpful scripts
cloud-scripts/zkcli.sh -cmd upconfig -zkhost 127.0.0.1:2181 -d solr/your_default_collection_with_the_config/conf/ -n config_name_userd_by_all_collections
After that you have to reload cores.
SolrCloud provides two option for uploading configuration files to ZK. If you have multiple cores while starting give option -Dbootstrap_conf=true. This will upload the index configuration files for all the cores. If you only want to upload configuration file of one core give two startup parameters -Dbootstrap_confdir and -Dcollection.configName.
I had multiple cores defined in the instance. You would have to upload each configuration by changing -Dcollection.configName argument and restart the Solr instance every time