When I start Logstash with my config file, the following error appears. I've added multiple classpaths to the current jdbc jar. Had the driver class been renamed or something? Thanks
[2018-09-11T11:44:57,241][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Jdbc jdbc_connection_string=>"jdbc:sqlserver://localhost:1433/test?useSSL=false&serverTimezone=GMT&DatabaseName=test", jdbc_driver_class=>"com.microsoft.sqlserver.jdbc.SQLServerDriver", jdbc_user=>"test", jdbc_password=><password>, statement=>"SELECT * FROM phones", id=>"d4c77c6d8f243cf0b447d00db3567bad38756544341a7e9fc53c5e611a6cec8d", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_6b256972-6c6a-481a-87f0-8b587adf7e53", enable_metric=>true, charset=>"UTF-8">, jdbc_paging_enabled=>false, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, sql_log_level=>"info", connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, parameters=>{"sql_last_value"=>1970-01-01 01:00:00 +0100}, last_run_metadata_path=>"C:\\Users\\ross.massie/.logstash_jdbc_last_run", use_column_value=>false, tracking_column_type=>"numeric", clean_run=>false, record_last_run=>true, lowercase_column_names=>true>
Error: com.microsoft.sqlserver.jdbc.SQLServerDriver not loaded. :jdbc_driver_library is not set, are you sure you included
the proper driver client libraries in your classpath?
Exception: LogStash::ConfigurationError
Stack: C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/plugin_mixins/jdbc.rb:162:in `open_jdbc_connection'
C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/plugin_mixins/jdbc.rb:220:in `execute_statement'
C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/inputs/jdbc.rb:269:in `execute_query'
C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/inputs/jdbc.rb:255:in `run'
C:/Program Files/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:512:in `inputworker'
C:/Program Files/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:505:in `block in start_input'
My config snippet:
input {
jdbc {
jdbc_connection_string => "jdbc:sqlserver://localhost:1433/test?useSSL=false&serverTimezone=GMT&DatabaseName=test"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_driver_library => "C:\Program Files\logstash-6.3.2\lib\mssql-jdbc-7.0.0.jre8.jar"
jdbc_user => "test"
jdbc_password => "test123"
statement => "SELECT * FROM phones"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "phones"
}
}
I simply made a custom bit of .NET coding to do the job rather than hooking them together like the official documentation suggests (badly).
I highly recommend everyone else do the same to save yourself the time I lost.
Related
I am using Logstash to move data from my Microsoft SQL Server database to ElasticSearch. I receive the following error in the log files when I try to run logstash.
I run:
sudo -Hu logstash /usr/share/logstash/bin/logstash --path.settings=/etc/logstash -t
Error:
Exception: LogStash::ConfigurationError
Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:163:in `open_jdbc_connection'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:221:in `execute_statement'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:277:in `execute_query'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:263:in `run'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:309:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:302:in `block in start_input'
[2019-09-18T00:16:36,250][ERROR][logstash.javapipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Jdbc jdbc_user=>[user_name], jdbc_password=><password>, statement=>"SELECT * FROM [table] WHERE delete_flag = 'N'", jdbc_connection_string=>"jdbc:sqlserver://[IP]:1433;databaseName=[database];", id=>"04af4c4719615e3a3a03c3df3e5262aa40c40e85fb55a5886a1797d42eb3a729", jdbc_driver_class=>"com.microsoft.sqlserver.jdbc.SQLServerDriver", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_76e4bd7a-70dd-46cf-acd3-c3726a8e003f", enable_metric=>true, charset=>"UTF-8">, jdbc_paging_enabled=>false, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, sql_log_level=>"info", connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, parameters=>{"sql_last_value"=>1970-01-01 00:00:00 UTC}, last_run_metadata_path=>"/usr/share/logstash/.logstash_jdbc_last_run", use_column_value=>false, tracking_column_type=>"numeric", clean_run=>false, record_last_run=>true, lowercase_column_names=>true>
Error: com.microsoft.sqlserver.jdbc.SQLServerDriver not loaded. :jdbc_driver_library is not set, are you sure you included
the proper driver client libraries in your classpath?
I am running ElasticSearch/Logstash on an Ubuntu 18 server. I installed ElasticSearch, Logstash, Java 11, and downloaded the jdbc and placed it in the same folder as my logstash config.
My Logstash Config is:
input {
jdbc {
jdbc_driver_library => "/etc/logstash/conf.d/sqljdbc42.jar"
jdbc_connection_string => "jdbc:sqlserver://[my_ip]:1433;databaseName=[db_name];"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_user => "[username]"
jdbc_password => "[password]"
schedule => "0 4 * * * America/New_York"
statement => "SELECT * FROM [table] WHERE delete_flag = 'N'"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "products"
}
}
I specified the jdbc driver library, not sure why I keep getting an error why logstash can't find it.
I also tried to run (below)
sudo bash -c "export CLASSPATH=.:/etc/logstash/conf.d/sqljdbc42.jar"
Note: I took out some data, ip, dbname, username, etc.
This has been an issue with logstash-filter-jdbc_static for a while now. You need to move your .jar to /logstash-core/lib/jars/ directory. And specify the .jar location in your config as "".
See below for more details:
https://github.com/logstash-plugins/logstash-filter-jdbc_static/issues/47
I have been trying to import data from MS SQL Server to Elastic Search using Logstash. However, I am getting logstash pipeline and undefined method `close_jdbc_connection' error. I have not found the exact solution for this issue. The code used and error messages are as follows -
logstash config -
input {
jdbc {
jdbc_driver_library => "C:\elasticsearch-5.4.3\elasticsearch-5.4.3\lib\sqljdbc42.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://<servername>,<portname>;databaseName=<db_name>"
jdbc_user => "user"
jdbc_password => "password"
statement => "select * from dbo.jobstatus"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
}
}
filter {
mutate {
remove_field => [ "message", "path", "score", "#version", "host" ]
}
#mutate { convert => ["REGCONNTYPEID","integer"]}
}
output {
elasticsearch {
hosts => "localhost"
index => "sql_elk_dc_stats"
document_type => "devices"
}
stdout { codec => rubydebug { metadata=> true } }
}
Output -
[2017-10-15T18:20:11,768][INFO ][logstash.agent ] Successfully started
Logstash API endpoint {:port=>9600}
[2017-10-15T18:20:40,971][WARN ][logstash.inputs.jdbc ] Failed test_connecti
on.
[2017-10-15T18:20:40,978][ERROR][logstash.pipeline ] A plugin had an unre
coverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::Jdbc jdbc_driver_library=>"C:\\elasticsearch-5.4.3\
\elasticsearch-5.4.3\\lib\\sqljdbc42.jar", jdbc_driver_class=>"com.microsoft.sql
server.jdbc.SQLServerDriver", jdbc_connection_string=>"jdbc:sqlserver://<servername>,<port>;databaseName=db_name", jdbc
_user=>"user", jdbc_password=><password>, statement=>"select * from dbo.jo
bstatus", jdbc_paging_enabled=>true, jdbc_page_size=>50000, id=>"36bb27ae9af8f6a
086048a0a0f6a22d4a32b1be6-1", enable_metric=>true, codec=><LogStash::Codecs::Pla
in id=>"plain_8d7b9383-b58f-4dfe-82c8-20066ced2652", enable_metric=>true, charse
t=>"UTF-8">, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdb
c_pool_timeout=>5, sql_log_level=>"info", connection_retry_attempts=>1, connecti
on_retry_attempts_wait_time=>0.5, parameters=>{"sql_last_value"=>false}, last_ru
n_metadata_path=>"C:\\Users\\ghosmrin/.logstash_jdbc_last_run", use_column_value
=>false, tracking_column_type=>"numeric", clean_run=>false, record_last_run=>tru
e, lowercase_column_names=>true>
Error: undefined method `close_jdbc_connection' for #<Sequel::JDBC::Database:0
x764acb8b>
Note - I am using windows 7, ELK 5.4.3 version. The sql login is able to connect to the SQL server.
It looks like there was a bug https://github.com/logstash-plugins/logstash-input-jdbc/issues/227 - that was reported in July - It looks like it made it into the 4.2.4 version of the JDBC plugin https://github.com/logstash-plugins/logstash-input-jdbc/blob/v4.2.4/CHANGELOG.md - are you using an earlier version of the plugin?
With that said, I suspect the could still be an issue with your connection, the plugin may just be poorly handling a connection issue... As you've sensibly replaced you db/server details it's impossible to categorically say if there's an issue with the connection string... The one thing I'll say is that you need a colon between the server and port (you have a comma), you might also need a semi-colon at the end of the string, but I don't think it's strictly necessary:
jdbc_connection_string => "jdbc:sqlserver://<servername>:<portname>;databaseName=<db_name>;"
https://learn.microsoft.com/en-us/sql/connect/jdbc/building-the-connection-url
adapter.js file:
var procedure1Statement = WL.Server.createSQLStatement("select [sname] from [TestDb].[dbo].[studentinfo]");
function procedure1() {
return WL.Server.invokeSQLStatement({
preparedStatement : procedure1Statement,
parameters : []
});
}
adapter.xml file:
<dataSourceDefinition>
<driverClass>com.microsoft.sqlserver.jdbc.SQLServerDriver</driverClass>
<url>jdbc:sqlserver://localhost;databaseName=TestDb</url>
<user>user</user>
<password>pass</password>
</dataSourceDefinition>
getting error like this:
{
"errors": [
"Runtime: java.lang.ClassNotFoundException: Class com.microsoft.jdbc.sqlserver.SQLServerDriver not found in Worklight platform or project \/HelloWorld"
],
Read the error. Your error is that the driver is missing.
Make sure you have placed the jdbc driver in your-project\server\lib
Then make sure you are correctly pointing to the database in the XML file: IBM Worklight 6.1 - Failed connecting to MS SQL using SQL adapter
so my idea was to install tomcat7 using puppet, and then deploy the war file of Solr as a web app. Here is what i found.
There are many tomcat7 modules on puppet forge but none of them work out of box, and i am not sure if any of them actually work and a lot of them pertains to have code as documentation.
Take puppet module install fhuertas-tomcat7 as first example:
installs fhuertas-tomcat7 (v0.0.1)
i get an error, when i run:
sudo puppet apply --modulepath=/home/qa/puppet_qa/modules/ -e "include tomcat7" --debug
Error: Could not find data item service_path in any Hiera data file and no default supplied
and similar scenario follows for :
puppet module install llehmijo-tomcat7_rhel ( no longer maintained )
https://github.com/Spredzy/puppet-tomcat7 ( claims to be for CentOs ) but has an Apt (apt for centos ??) pre-requisite, plus it did not install either.
All i want to do is to install tomcat7 via puppet, and then install Solr. seems to be a simple request. Meanwhile i am working on my own to resolve the exact issue, and was able to install and run tomcat7, but not sure how i can install tomcat-users.xml
here is a portion of my init.pp
exec {'start service':
command => 'sh "startup.sh"',
cwd => "/usr/share/apache-tomcat-7.0.42/bin",
path => '/usr/share/apache-tomcat-7.0.42/bin/:/usr/bin:/bin',
#require => File['/usr/share/apache-tomcat-7.0.42/conf/tomcat-users.xml']
}
so if i un-comment the require => File[]
Error: Could not find dependency File[/usr/share/apache-tomcat-7.0.42/conf/tomcat-users.xml] for Exec[start service]
file { "/etc/tomcat7/tomcat-users.xml":
owner => 'root',
require => Package['tomcat'],
notify => Service['tomcat'],
content => template('tomcat/tomcat-users.xml.erb')
}
This works, as for the modules not working on puppet forge , and github, i think there is no resolution as if not working then these modules can be taken as guidelines, or hints.
I used the supported one https://forge.puppetlabs.com/puppetlabs/tomcat and following the examples everything was properly installed using packages (at least on Ubuntu). It is also possible to install it from source.
https://github.com/puppetlabs/puppetlabs-tomcat/tree/master/examples. See the example below:
class { 'java': }
class { 'tomcat':
install_from_source => false,
user => 'tomcat7',
require => Class['java']
}
tomcat::instance { 'tomcat7':
package_name => 'tomcat7',
require => Class['tomcat']
}->
tomcat::instance { 'tomcat7-admin':
package_name => 'tomcat7-admin',
}->
tomcat::config::server::tomcat_users {
'tomcat-admin':
catalina_base => '/var/lib/tomcat7',
element => 'user',
password => 'test',
roles => ['manager-gui','admin'];
'deployer':
catalina_base => '/var/lib/tomcat7',
element => 'user',
password => 'deployer',
roles => ['manager-script'];
}->
tomcat::service { 'tomcat7':
service_ensure => running,
catalina_base => '/var/lib/tomcat7',
require => Tomcat::Instance['tomcat7']
}
I always used PHP 5.2.3 version, but now I updated PHP to version 5.3.2.
I have problem with MSSQL, I can`t connect to MSSQL server.
I have downloaded a SQL server driver for PHP 1.1
There are a lot of files, I used php_sqlsrv_53_ts_vc9. Put it on php/ext directory php_sqlsrv_53_ts_vc9.
I have added extension=php_sqlsrv_53_ts_vc9.dll (in php.ini), but it gives an error:
Call to undefined function mssql_connect() in C:\webserver\www\MSSQl\db_mssql.class.php on line 26
I have used:
$serverName = "$sql_server";
$connectionInfo = array( "Database"=>$sql_db_name,"UID"=>$sql_user,"PWD"=>$sql_pass);
$conn = sqlsrv_connect($serverName,$connectionInfo);
if( $conn === false )
{
echo "Could not connect.\n";
die( sqlsrv_errors());
}
Then all work!
Open php.ini ,just add this line
extension=php_sqlsrv_53_ts_vc9.dll
you need to know what compiler do you use
phpinfo();
Compiler MSVC9 (Visual C++ 2008)
Than Add it.