undefined method `close_jdbc_connection' for #<Sequel::JDBC::Database:0 x764acb8b> - sql-server

I have been trying to import data from MS SQL Server to Elastic Search using Logstash. However, I am getting logstash pipeline and undefined method `close_jdbc_connection' error. I have not found the exact solution for this issue. The code used and error messages are as follows -
logstash config -
input {
jdbc {
jdbc_driver_library => "C:\elasticsearch-5.4.3\elasticsearch-5.4.3\lib\sqljdbc42.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://<servername>,<portname>;databaseName=<db_name>"
jdbc_user => "user"
jdbc_password => "password"
statement => "select * from dbo.jobstatus"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
}
}
filter {
mutate {
remove_field => [ "message", "path", "score", "#version", "host" ]
}
#mutate { convert => ["REGCONNTYPEID","integer"]}
}
output {
elasticsearch {
hosts => "localhost"
index => "sql_elk_dc_stats"
document_type => "devices"
}
stdout { codec => rubydebug { metadata=> true } }
}
Output -
[2017-10-15T18:20:11,768][INFO ][logstash.agent ] Successfully started
Logstash API endpoint {:port=>9600}
[2017-10-15T18:20:40,971][WARN ][logstash.inputs.jdbc ] Failed test_connecti
on.
[2017-10-15T18:20:40,978][ERROR][logstash.pipeline ] A plugin had an unre
coverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::Jdbc jdbc_driver_library=>"C:\\elasticsearch-5.4.3\
\elasticsearch-5.4.3\\lib\\sqljdbc42.jar", jdbc_driver_class=>"com.microsoft.sql
server.jdbc.SQLServerDriver", jdbc_connection_string=>"jdbc:sqlserver://<servername>,<port>;databaseName=db_name", jdbc
_user=>"user", jdbc_password=><password>, statement=>"select * from dbo.jo
bstatus", jdbc_paging_enabled=>true, jdbc_page_size=>50000, id=>"36bb27ae9af8f6a
086048a0a0f6a22d4a32b1be6-1", enable_metric=>true, codec=><LogStash::Codecs::Pla
in id=>"plain_8d7b9383-b58f-4dfe-82c8-20066ced2652", enable_metric=>true, charse
t=>"UTF-8">, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdb
c_pool_timeout=>5, sql_log_level=>"info", connection_retry_attempts=>1, connecti
on_retry_attempts_wait_time=>0.5, parameters=>{"sql_last_value"=>false}, last_ru
n_metadata_path=>"C:\\Users\\ghosmrin/.logstash_jdbc_last_run", use_column_value
=>false, tracking_column_type=>"numeric", clean_run=>false, record_last_run=>tru
e, lowercase_column_names=>true>
Error: undefined method `close_jdbc_connection' for #<Sequel::JDBC::Database:0
x764acb8b>
Note - I am using windows 7, ELK 5.4.3 version. The sql login is able to connect to the SQL server.

It looks like there was a bug https://github.com/logstash-plugins/logstash-input-jdbc/issues/227 - that was reported in July - It looks like it made it into the 4.2.4 version of the JDBC plugin https://github.com/logstash-plugins/logstash-input-jdbc/blob/v4.2.4/CHANGELOG.md - are you using an earlier version of the plugin?
With that said, I suspect the could still be an issue with your connection, the plugin may just be poorly handling a connection issue... As you've sensibly replaced you db/server details it's impossible to categorically say if there's an issue with the connection string... The one thing I'll say is that you need a colon between the server and port (you have a comma), you might also need a semi-colon at the end of the string, but I don't think it's strictly necessary:
jdbc_connection_string => "jdbc:sqlserver://<servername>:<portname>;databaseName=<db_name>;"
https://learn.microsoft.com/en-us/sql/connect/jdbc/building-the-connection-url

Related

Logstash Configuration Error - jdbc_driver_library is not set

I am using Logstash to move data from my Microsoft SQL Server database to ElasticSearch. I receive the following error in the log files when I try to run logstash.
I run:
sudo -Hu logstash /usr/share/logstash/bin/logstash --path.settings=/etc/logstash -t
Error:
Exception: LogStash::ConfigurationError
Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:163:in `open_jdbc_connection'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:221:in `execute_statement'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:277:in `execute_query'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:263:in `run'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:309:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:302:in `block in start_input'
[2019-09-18T00:16:36,250][ERROR][logstash.javapipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Jdbc jdbc_user=>[user_name], jdbc_password=><password>, statement=>"SELECT * FROM [table] WHERE delete_flag = 'N'", jdbc_connection_string=>"jdbc:sqlserver://[IP]:1433;databaseName=[database];", id=>"04af4c4719615e3a3a03c3df3e5262aa40c40e85fb55a5886a1797d42eb3a729", jdbc_driver_class=>"com.microsoft.sqlserver.jdbc.SQLServerDriver", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_76e4bd7a-70dd-46cf-acd3-c3726a8e003f", enable_metric=>true, charset=>"UTF-8">, jdbc_paging_enabled=>false, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, sql_log_level=>"info", connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, parameters=>{"sql_last_value"=>1970-01-01 00:00:00 UTC}, last_run_metadata_path=>"/usr/share/logstash/.logstash_jdbc_last_run", use_column_value=>false, tracking_column_type=>"numeric", clean_run=>false, record_last_run=>true, lowercase_column_names=>true>
Error: com.microsoft.sqlserver.jdbc.SQLServerDriver not loaded. :jdbc_driver_library is not set, are you sure you included
the proper driver client libraries in your classpath?
I am running ElasticSearch/Logstash on an Ubuntu 18 server. I installed ElasticSearch, Logstash, Java 11, and downloaded the jdbc and placed it in the same folder as my logstash config.
My Logstash Config is:
input {
jdbc {
jdbc_driver_library => "/etc/logstash/conf.d/sqljdbc42.jar"
jdbc_connection_string => "jdbc:sqlserver://[my_ip]:1433;databaseName=[db_name];"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_user => "[username]"
jdbc_password => "[password]"
schedule => "0 4 * * * America/New_York"
statement => "SELECT * FROM [table] WHERE delete_flag = 'N'"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "products"
}
}
I specified the jdbc driver library, not sure why I keep getting an error why logstash can't find it.
I also tried to run (below)
sudo bash -c "export CLASSPATH=.:/etc/logstash/conf.d/sqljdbc42.jar"
Note: I took out some data, ip, dbname, username, etc.
This has been an issue with logstash-filter-jdbc_static for a while now. You need to move your .jar to /logstash-core/lib/jars/ directory. And specify the .jar location in your config as "".
See below for more details:
https://github.com/logstash-plugins/logstash-filter-jdbc_static/issues/47

Logstash - Solr Integration

Below is my actual log,
--2019-05-09 06:49:05.590 -TRACE 6293 --- [ntainer#0-0-C-1] c.s.s.service.MessageLogServiceImpl : [41a6811cbc1c66eda0e942712a12a003d6bf4654b3edb6d24bf159b592afc64f1557384545548] Event => Message Failure Identified : INVALID_STRUCTURE
My given grok filter pattern,
match => {
"message" => "--%{TIMESTAMP_ISO8601:logtime} -%{LOGLEVEL:level} (?<pid>\d+) --- \[(?<thread>[^\]]+)] (?<classname>[\w.]+)\s+: \[(?<token>[^\]]+)] Event \=> Message Failure Identified : (?<code>[\w]+)"
}
After doing some adding/removing desired filed below is my tokenized form,
{
"code" => "INVALID_STRUCTURE",
"event" => "message_failure",
"token" => "41a6811cbc1c66eda0e942712a12a003d6bf4654b3edb6d24bf159b592afc64f1557384545548",
"logtime" => "2019-05-09 06:49:05.590"
}
Now I want to send it to solr, but while sending this is giving me the warning,
[WARN ][logstash.outputs.solrhttp] An error occurred while indexing: undefined method `iso8601' for nil:NilClass
I think it related to "logtime" field since that is the only portion which deals with ISO8601. Nothing extra information found in the logs. What is the problem here?
Finally got the answer. Thanks
this blog
Logstash will use a ruby library called Rsolr for connecting to Solr. Logstash will install the latest version of Rsolr, but latest Rsolr has a bub with timestamp datatype which will prevent indexing to Solr. To prevent this we will have to manually install the working version of Rsolr.
Changed the default rsolr plugin as mentioned in the post and everything started working

Access Sql Server through Ruby Active Record JDBC

I've been attempting to connect to a remote JDBC/SqlServer database using Active Record in Ruby (specifically JRuby). Here is my code:
require 'activerecord'
require 'activerecord-jdbc-adapter'
ActiveRecord::Base.establish_connection(
:adapter => 'sqlserver',
:username => '<username>',
:password => '<password>',
:database => '<database_name>',
:url => '<database_url>',
)
#connection = ActiveRecord::Base.connection
puts #connection
I've also put 'jdbc' as the adapter, and that also did not work.
And here is an abbreviated version of the error I am getting:
NameError: cannot load Java class com.microsoft.sqlserver.jdbc.SQLServerDriver
for_name at org/jruby/javasupport/JavaClass.java:286
get_proxy_class at org/jruby/javasupport/JavaUtilities.java:34
block in java_import at uri:classloader:/jruby/java/core_ext/object.rb:49
map at org/jruby/RubyArray.java:2486
java_import at uri:classloader:/jruby/java/core_ext/object.rb:36
block in driver_class at C:/jruby-9.1.12.0/lib/ruby/gems/shared/gems/activerecord-jdbc-adapter-1.3.23/lib/arjdbc/jdbc/driver.rb:24
module_eval at org/jruby/RubyModule.java:2833
block in driver_class at C:/jruby-9.1.12.0/lib/ruby/gems/shared/gems/activerecord-jdbc-adapter-1.3.23/lib/arjdbc/jdbc/driver.rb:23
synchronized at org/jruby/javasupport/JavaObject.java:257
It's not so much a specific problem but more that I'm just unsure where to go from here. Using the 'activerecord-sqlserver-adapter' in pure Ruby doesn't work because of the JDBC on the server. (I might be misusing the terms here, I'm rather unfamiliar with how databases work.)
However, the jdbc/sqlserver gems seem to be not well-supported or out of date. There's probably no silver bullet for this, but any kind of direction would be of enormous help.
You still need to require 'active_record', no?
And I am not sure you need the require 'arel'. That is the low level sql builder/formatter implemented by active_record.
require 'active_record'
require 'activerecord-jdbc-adapter'
ActiveRecord::Base.pluralize_table_names = false
ActiveRecord::Base.establish_connection(
:adapter => "sqlserver",
:host => "SqlServerHostName",
:database => "HostDbName"
)

ZF2 PDO_IBM Driver with Relational Database Name

Have a Zend Framework 2.4 connection factory that needs to establish a connection with an AS400 iSeries database. The connection has to be made this way because there are multiple test environment and the factory needs to accommodate each.
The method is using Zend\Db\Adapter\Adapter and I pass that class an array of database connection parameters.
At issue: Zend\Db\Adapter doesn't accept the relational database (directory) name. I'm assuming that since the driver is PDO_IBM, there would be some expectation of a field for explicitly defining the name for the directory.
Here is the method:
public function conn($dbs) {
$this->adapter = new Adapter(array(
'driver' => $dbs['db']['driver'],
'dbname' => $dbs['db']['dbname'],
'username' => $dbs['db']['username'],
'password' => $dbs['db']['password'],
'hostname' => $dbs['db']['hostname'],
'port' => $dbs['db']['port'],
));
var_dump($this->adapter);
return $this->adapter;
}
Adapter is an alias for \Zend\Db\Adapter\Adapter
And here is the object that gets created.
["driver":protected]=> object(Zend\Db\Adapter\Driver\Pdo\Pdo)#224 (4){
["connection":protected]=>object(Zend\Db\Adapter\Driver\Pdo\Connection)#225 (6) {
["driver":protected]=> *RECURSION*
["profiler":protected]=> NULL
["driverName":protected]=> string(3)"ibm"
["connectionParameters":protected]=> array(6) {
["driver"]=> string(7) "PDO_IBM"
["dbname"]=> string(7) “<relational_database_name>”
["username"]=> string(3) “<user_name"
["password"]=> string(3) “<password>"
["hostname"]=> string(9) "127.0.0.1"
["port"]=> string(3) "446"
}
I can instantiate the connection object using:
$conn = new \Zend\Db\Adapter\Adapter( );
Pdo=ibm:<relational_database_name>
But this isn't a workable solution for this situation. Finally, here is the error:
Connect Error: SQLSTATE=42705, SQLConnect: -950 Relational database dbname=;hos not in relational database directory.
For the sake of completeness, here is the configuration that worked for ZF2 Zend\Db\Adapter\Adapter running on Zend Server 6 and connecting to an AS400 iSeries database.
//concat the driver and rel. db directory name into one string
//$dsn name is required for Zend\Db to correctly read it into memory
$dsn = "ibm:" . $db_dir_name;
$this->adapter = new Adapter(array(
'driver' => $driver, // Pdo
'dsn' => $dsn,
'username' => <user_name>,
'password' => <user_pwd>
));
This wasn't documented anywhere and figured it out through trial and error.

Suppressing connection error with DBI and DBD:ODBC SQL Server Native Client 10.0

Writing a script to get SQL Server instance names from a table, then attempting to connect to each of these instances to pull back database configuration information. All database instances involved are some version of SQL Server. If the connection fails (due to a bad password, instance is down, etc.) the intention is to print a user-defined error message ("Unable to connect to $inst, skipping.") and continue through the list. I'm having trouble suppressing the default error message from ODBC (SQL Server Native Client 10.0).
Connection is attempted like this:
eval {
my $dbh = DBI->connect(
"dbi:ODBC:Driver={SQL Server Native Client 10.0};Server=<instance_name>;Uid=<user_name>;Pwd=<password>;",
{ PrintError => 0, RaiseError => 1, AutoCommit => 1 }
);
};
It is my (probably incorrect) understanding that PrintError => 0 should suppress the error message and RaiseError => 1 will cause DBI to die if the connect method fails, at which point I can check $# for the error and print a user-defined message. I have also looked at the HandleError attribute but have not had any success.
Is this a completely unrealistic scenario, or is this a result of the ODBC driver I'm working with?
Per bohica's suggestions, working code looks like:
eval {
my $dbh = DBI->connect(
"dbi:ODBC:Driver={SQL Server Native Client 10.0};Server=<instance_name>;",
"Username",
"Password",
{ PrintError => 0, RaiseError => 1, AutoCommit => 1 }
);
};
Username and password were moved out of connection string and passed as separate parameters to DBI connect method.
Assuming you fix the problem Pedro mentions then, PrintError=>0 suppresses errors and you might want to look at PrintWarn as well. The RaiseError=>1 will cause the connect to die if the connect fails and in your example the error will be in $#.
connect is a class method; you call it with DBI->connect, which returns a db handle ($dbh in your case).

Resources