I am using Logstash to move data from my Microsoft SQL Server database to ElasticSearch. I receive the following error in the log files when I try to run logstash.
I run:
sudo -Hu logstash /usr/share/logstash/bin/logstash --path.settings=/etc/logstash -t
Error:
Exception: LogStash::ConfigurationError
Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:163:in `open_jdbc_connection'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:221:in `execute_statement'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:277:in `execute_query'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:263:in `run'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:309:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:302:in `block in start_input'
[2019-09-18T00:16:36,250][ERROR][logstash.javapipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Jdbc jdbc_user=>[user_name], jdbc_password=><password>, statement=>"SELECT * FROM [table] WHERE delete_flag = 'N'", jdbc_connection_string=>"jdbc:sqlserver://[IP]:1433;databaseName=[database];", id=>"04af4c4719615e3a3a03c3df3e5262aa40c40e85fb55a5886a1797d42eb3a729", jdbc_driver_class=>"com.microsoft.sqlserver.jdbc.SQLServerDriver", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_76e4bd7a-70dd-46cf-acd3-c3726a8e003f", enable_metric=>true, charset=>"UTF-8">, jdbc_paging_enabled=>false, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, sql_log_level=>"info", connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, parameters=>{"sql_last_value"=>1970-01-01 00:00:00 UTC}, last_run_metadata_path=>"/usr/share/logstash/.logstash_jdbc_last_run", use_column_value=>false, tracking_column_type=>"numeric", clean_run=>false, record_last_run=>true, lowercase_column_names=>true>
Error: com.microsoft.sqlserver.jdbc.SQLServerDriver not loaded. :jdbc_driver_library is not set, are you sure you included
the proper driver client libraries in your classpath?
I am running ElasticSearch/Logstash on an Ubuntu 18 server. I installed ElasticSearch, Logstash, Java 11, and downloaded the jdbc and placed it in the same folder as my logstash config.
My Logstash Config is:
input {
jdbc {
jdbc_driver_library => "/etc/logstash/conf.d/sqljdbc42.jar"
jdbc_connection_string => "jdbc:sqlserver://[my_ip]:1433;databaseName=[db_name];"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_user => "[username]"
jdbc_password => "[password]"
schedule => "0 4 * * * America/New_York"
statement => "SELECT * FROM [table] WHERE delete_flag = 'N'"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "products"
}
}
I specified the jdbc driver library, not sure why I keep getting an error why logstash can't find it.
I also tried to run (below)
sudo bash -c "export CLASSPATH=.:/etc/logstash/conf.d/sqljdbc42.jar"
Note: I took out some data, ip, dbname, username, etc.
This has been an issue with logstash-filter-jdbc_static for a while now. You need to move your .jar to /logstash-core/lib/jars/ directory. And specify the .jar location in your config as "".
See below for more details:
https://github.com/logstash-plugins/logstash-filter-jdbc_static/issues/47
Related
I'm using doctrine in a project (not symfony). In this project I also use phpstan, i installed both phpstan/phpstan-doctrine and phpstan/extension-installer.
My phpstan.neon is like this:
parameters:
level: 8
paths:
- src/
doctrine:
objectManagerLoader: tests/object-manager.php
Inside tests/object-manager.php it return the result of a call to a function that return the entity manager.
Here is the code that create the entity manager
$database_url = $_ENV['DATABASE_URL'];
$isDevMode = $this->isDevMode();
$proxyDir = null;
$cache = null;
$useSimpleAnnotationReader = false;
$config = Setup::createAnnotationMetadataConfiguration(
[$this->getProjectDirectory() . '/src'],
$isDevMode,
$proxyDir,
$cache,
$useSimpleAnnotationReader
);
// database configuration parameters
$conn = [
'url' => $database_url,
];
// obtaining the entity manager
$entityManager = EntityManager::create($conn, $config);
When i run vendor/bin/phpstan analyze i get this error:
Internal error: An exception occurred in the driver: SQLSTATE[08006] [7] could not translate host name "postgres_db" to address: nodename nor servname provided, or not known
This appear because i'm using docker and my database url is postgres://user:password#postgres_db/database postgres_db is the name of my database container so the hostname is known inside the docker container.
When i run phpstan inside the container i do not have the error.
So is there a way to run phpstan outside docker ? Because i'm pretty sure that when i'll push my code the github workflow will fail because of this
Do phpstan need to try to reach the database ?
I opened an issue on the github of phpstan-doctrine and i had an answer by #jlherren that explained :
The problem is that Doctrine needs to know what version of the DB server it is working with in order to instantiate the correct AbstractPlatform implementation, of which there are several available for the same DB vendor (e.g. PostgreSQL94Platform or PostgreSQL100Platform for postgres, and similarly for other DB drivers). To auto-detect this information, it will simply connect to the DB and query the version.
I just changed my database url from:
DATABASE_URL=postgres://user:password#database_ip/database
To:
DATABASE_URL=postgres://user:password#database_ip/database?serverVersion=14.2
When I start Logstash with my config file, the following error appears. I've added multiple classpaths to the current jdbc jar. Had the driver class been renamed or something? Thanks
[2018-09-11T11:44:57,241][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Jdbc jdbc_connection_string=>"jdbc:sqlserver://localhost:1433/test?useSSL=false&serverTimezone=GMT&DatabaseName=test", jdbc_driver_class=>"com.microsoft.sqlserver.jdbc.SQLServerDriver", jdbc_user=>"test", jdbc_password=><password>, statement=>"SELECT * FROM phones", id=>"d4c77c6d8f243cf0b447d00db3567bad38756544341a7e9fc53c5e611a6cec8d", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_6b256972-6c6a-481a-87f0-8b587adf7e53", enable_metric=>true, charset=>"UTF-8">, jdbc_paging_enabled=>false, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, sql_log_level=>"info", connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, parameters=>{"sql_last_value"=>1970-01-01 01:00:00 +0100}, last_run_metadata_path=>"C:\\Users\\ross.massie/.logstash_jdbc_last_run", use_column_value=>false, tracking_column_type=>"numeric", clean_run=>false, record_last_run=>true, lowercase_column_names=>true>
Error: com.microsoft.sqlserver.jdbc.SQLServerDriver not loaded. :jdbc_driver_library is not set, are you sure you included
the proper driver client libraries in your classpath?
Exception: LogStash::ConfigurationError
Stack: C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/plugin_mixins/jdbc.rb:162:in `open_jdbc_connection'
C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/plugin_mixins/jdbc.rb:220:in `execute_statement'
C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/inputs/jdbc.rb:269:in `execute_query'
C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/inputs/jdbc.rb:255:in `run'
C:/Program Files/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:512:in `inputworker'
C:/Program Files/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:505:in `block in start_input'
My config snippet:
input {
jdbc {
jdbc_connection_string => "jdbc:sqlserver://localhost:1433/test?useSSL=false&serverTimezone=GMT&DatabaseName=test"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_driver_library => "C:\Program Files\logstash-6.3.2\lib\mssql-jdbc-7.0.0.jre8.jar"
jdbc_user => "test"
jdbc_password => "test123"
statement => "SELECT * FROM phones"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "phones"
}
}
I simply made a custom bit of .NET coding to do the job rather than hooking them together like the official documentation suggests (badly).
I highly recommend everyone else do the same to save yourself the time I lost.
I have been trying to import data from MS SQL Server to Elastic Search using Logstash. However, I am getting logstash pipeline and undefined method `close_jdbc_connection' error. I have not found the exact solution for this issue. The code used and error messages are as follows -
logstash config -
input {
jdbc {
jdbc_driver_library => "C:\elasticsearch-5.4.3\elasticsearch-5.4.3\lib\sqljdbc42.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://<servername>,<portname>;databaseName=<db_name>"
jdbc_user => "user"
jdbc_password => "password"
statement => "select * from dbo.jobstatus"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
}
}
filter {
mutate {
remove_field => [ "message", "path", "score", "#version", "host" ]
}
#mutate { convert => ["REGCONNTYPEID","integer"]}
}
output {
elasticsearch {
hosts => "localhost"
index => "sql_elk_dc_stats"
document_type => "devices"
}
stdout { codec => rubydebug { metadata=> true } }
}
Output -
[2017-10-15T18:20:11,768][INFO ][logstash.agent ] Successfully started
Logstash API endpoint {:port=>9600}
[2017-10-15T18:20:40,971][WARN ][logstash.inputs.jdbc ] Failed test_connecti
on.
[2017-10-15T18:20:40,978][ERROR][logstash.pipeline ] A plugin had an unre
coverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::Jdbc jdbc_driver_library=>"C:\\elasticsearch-5.4.3\
\elasticsearch-5.4.3\\lib\\sqljdbc42.jar", jdbc_driver_class=>"com.microsoft.sql
server.jdbc.SQLServerDriver", jdbc_connection_string=>"jdbc:sqlserver://<servername>,<port>;databaseName=db_name", jdbc
_user=>"user", jdbc_password=><password>, statement=>"select * from dbo.jo
bstatus", jdbc_paging_enabled=>true, jdbc_page_size=>50000, id=>"36bb27ae9af8f6a
086048a0a0f6a22d4a32b1be6-1", enable_metric=>true, codec=><LogStash::Codecs::Pla
in id=>"plain_8d7b9383-b58f-4dfe-82c8-20066ced2652", enable_metric=>true, charse
t=>"UTF-8">, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdb
c_pool_timeout=>5, sql_log_level=>"info", connection_retry_attempts=>1, connecti
on_retry_attempts_wait_time=>0.5, parameters=>{"sql_last_value"=>false}, last_ru
n_metadata_path=>"C:\\Users\\ghosmrin/.logstash_jdbc_last_run", use_column_value
=>false, tracking_column_type=>"numeric", clean_run=>false, record_last_run=>tru
e, lowercase_column_names=>true>
Error: undefined method `close_jdbc_connection' for #<Sequel::JDBC::Database:0
x764acb8b>
Note - I am using windows 7, ELK 5.4.3 version. The sql login is able to connect to the SQL server.
It looks like there was a bug https://github.com/logstash-plugins/logstash-input-jdbc/issues/227 - that was reported in July - It looks like it made it into the 4.2.4 version of the JDBC plugin https://github.com/logstash-plugins/logstash-input-jdbc/blob/v4.2.4/CHANGELOG.md - are you using an earlier version of the plugin?
With that said, I suspect the could still be an issue with your connection, the plugin may just be poorly handling a connection issue... As you've sensibly replaced you db/server details it's impossible to categorically say if there's an issue with the connection string... The one thing I'll say is that you need a colon between the server and port (you have a comma), you might also need a semi-colon at the end of the string, but I don't think it's strictly necessary:
jdbc_connection_string => "jdbc:sqlserver://<servername>:<portname>;databaseName=<db_name>;"
https://learn.microsoft.com/en-us/sql/connect/jdbc/building-the-connection-url
so my idea was to install tomcat7 using puppet, and then deploy the war file of Solr as a web app. Here is what i found.
There are many tomcat7 modules on puppet forge but none of them work out of box, and i am not sure if any of them actually work and a lot of them pertains to have code as documentation.
Take puppet module install fhuertas-tomcat7 as first example:
installs fhuertas-tomcat7 (v0.0.1)
i get an error, when i run:
sudo puppet apply --modulepath=/home/qa/puppet_qa/modules/ -e "include tomcat7" --debug
Error: Could not find data item service_path in any Hiera data file and no default supplied
and similar scenario follows for :
puppet module install llehmijo-tomcat7_rhel ( no longer maintained )
https://github.com/Spredzy/puppet-tomcat7 ( claims to be for CentOs ) but has an Apt (apt for centos ??) pre-requisite, plus it did not install either.
All i want to do is to install tomcat7 via puppet, and then install Solr. seems to be a simple request. Meanwhile i am working on my own to resolve the exact issue, and was able to install and run tomcat7, but not sure how i can install tomcat-users.xml
here is a portion of my init.pp
exec {'start service':
command => 'sh "startup.sh"',
cwd => "/usr/share/apache-tomcat-7.0.42/bin",
path => '/usr/share/apache-tomcat-7.0.42/bin/:/usr/bin:/bin',
#require => File['/usr/share/apache-tomcat-7.0.42/conf/tomcat-users.xml']
}
so if i un-comment the require => File[]
Error: Could not find dependency File[/usr/share/apache-tomcat-7.0.42/conf/tomcat-users.xml] for Exec[start service]
file { "/etc/tomcat7/tomcat-users.xml":
owner => 'root',
require => Package['tomcat'],
notify => Service['tomcat'],
content => template('tomcat/tomcat-users.xml.erb')
}
This works, as for the modules not working on puppet forge , and github, i think there is no resolution as if not working then these modules can be taken as guidelines, or hints.
I used the supported one https://forge.puppetlabs.com/puppetlabs/tomcat and following the examples everything was properly installed using packages (at least on Ubuntu). It is also possible to install it from source.
https://github.com/puppetlabs/puppetlabs-tomcat/tree/master/examples. See the example below:
class { 'java': }
class { 'tomcat':
install_from_source => false,
user => 'tomcat7',
require => Class['java']
}
tomcat::instance { 'tomcat7':
package_name => 'tomcat7',
require => Class['tomcat']
}->
tomcat::instance { 'tomcat7-admin':
package_name => 'tomcat7-admin',
}->
tomcat::config::server::tomcat_users {
'tomcat-admin':
catalina_base => '/var/lib/tomcat7',
element => 'user',
password => 'test',
roles => ['manager-gui','admin'];
'deployer':
catalina_base => '/var/lib/tomcat7',
element => 'user',
password => 'deployer',
roles => ['manager-script'];
}->
tomcat::service { 'tomcat7':
service_ensure => running,
catalina_base => '/var/lib/tomcat7',
require => Tomcat::Instance['tomcat7']
}
I have created my database, and used cake bake my_project to create my project, and now when I try cake bake all form within my project directory, I get an error relating to Mysql. When I view my project from the browser, it all lights up green, including connecting to the database.
Warning Error: PDO::__construct(): [2002] No such file or directory (trying to connect via unix:///var/mysql/mysql.sock) in [/Users/billy/Documents/projects/cakephp/lib/Cake/Model/Datasource/Database/Mysql.php, line 160]
Any ideas as to what this error is about?
EDIT: Added relevant configuration details (php --info |grep mysql)
I don't really know what to look for in here, but maybe someone does...
$ php --info |grep mysql
Configure Command => '/var/tmp/apache_mod_php/apache_mod_php-53.3.1~2/php/configure' '--prefix=/usr' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--disable-dependency-tracking' '--sysconfdir=/private/etc' '--with-apxs2=/usr/sbin/apxs' '--enable-cli' '--with-config-file-path=/etc' '--with-libxml-dir=/usr' '--with-openssl=/usr' '--with-kerberos=/usr' '--with-zlib=/usr' '--enable-bcmath' '--with-bz2=/usr' '--enable-calendar' '--with-curl=/usr' '--enable-exif' '--enable-ftp' '--with-gd' '--with-jpeg-dir=/BinaryCache/apache_mod_php/apache_mod_php-53.3.1~2/Root/usr/local' '--with-png-dir=/BinaryCache/apache_mod_php/apache_mod_php-53.3.1~2/Root/usr/local' '--enable-gd-native-ttf' '--with-ldap=/usr' '--with-ldap-sasl=/usr' '--enable-mbstring' '--enable-mbregex' '--with-mysql=mysqlnd' '--with-mysqli=mysqlnd' '--with-pdo-mysql=mysqlnd' '--with-mysql-sock=/var/mysql/mysql.sock' '--with-iodbc=/usr' '--enable-shmop' '--with-snmp=/usr' '--enable-soap' '--enable-sockets' '--enable-sysvmsg' '--enable-sysvsem' '--enable-sysvshm' '--with-xmlrpc' '--with-iconv-dir=/usr' '--with-xsl=/usr' '--enable-zend-multibyte' '--enable-zip' '--with-pcre-regex=/usr'
mysql
Client API version => mysqlnd 5.0.7-dev - 091210 - $Revision: 300533 $
mysql.allow_local_infile => On => On
mysql.allow_persistent => On => On
mysql.connect_timeout => 60 => 60
mysql.default_host => no value => no value
mysql.default_password => no value => no value
mysql.default_port => no value => no value
mysql.default_socket => /var/mysql/mysql.sock => /var/mysql/mysql.sock
mysql.default_user => no value => no value
mysql.max_links => Unlimited => Unlimited
mysql.max_persistent => Unlimited => Unlimited
mysql.trace_mode => Off => Off
mysqli
Client API library version => mysqlnd 5.0.7-dev - 091210 - $Revision: 300533 $
mysqli.allow_local_infile => On => On
mysqli.allow_persistent => On => On
mysqli.default_host => no value => no value
mysqli.default_port => 3306 => 3306
mysqli.default_pw => no value => no value
mysqli.default_socket => /var/mysql/mysql.sock => /var/mysql/mysql.sock
mysqli.default_user => no value => no value
mysqli.max_links => Unlimited => Unlimited
mysqli.max_persistent => Unlimited => Unlimited
mysqli.reconnect => Off => Off
mysqlnd
mysqlnd => enabled
Version => mysqlnd 5.0.7-dev - 091210 - $Revision: 300533 $
PDO drivers => mysql, sqlite, sqlite2
pdo_mysql
Client API version => mysqlnd 5.0.7-dev - 091210 - $Revision: 300533 $
pdo_mysql.default_socket => /var/mysql/mysql.sock => /var/mysql/mysql.sock
Edit: More info
I tried the suggestion from #0k32, but unfortunately it did not work for me. I soft linked php from my XAMPP folder to /usr/bin/php, and checked it was the right one with which php. I then did a php --info | grep sock and got:
MYSQL_SOCKET => /Applications/XAMPP/xamppfiles/var/mysql/mysql.sock
mysql.default_socket => /Applications/XAMPP/xamppfiles/var/mysql/mysql.sock => /Applications/XAMPP/xamppfiles/var/mysql/mysql.sock
MYSQLI_SOCKET => /Applications/XAMPP/xamppfiles/var/mysql/mysql.sock
mysqli.default_socket => /Applications/XAMPP/xamppfiles/var/mysql/mysql.sock => /Applications/XAMPP/xamppfiles/var/mysql/mysql.sock
Which seems to suggest that the correct version of MySQL is being activated. Still I am getting exactly the same error.
I think this issue could appear because of different configuration files (php.ini) used by apache's php module and command line php. Default settings for mysql connections may differ in those configs.
Compare phpinfo() output from apache and php --info from command line.
Update:
In your case it seems you are on MacOS and are using MAMP/MAMP PRO. So am I. And I had the same problem.
My solution was very simple. I replaced default php cli binary with a symlink to MAMP's php binary:
$ sudo mv /usr/bin/php /usr/bin/php_default
$ sudo ln -s /Applications/MAMP/bin/php/php5.3.6/bin/php /usr/bin/php
MAMP's php is connecting to MySQL via /Applications/MAMP/tmp/mysql/mysql.sock by default and MacOS's default command line php is trying to use /var/mysql/mysql.sock.
Anyway I think it's a good practice to use the same php with the same config for both web and cli.
update:
A smarter solution would be not to replace default php but just add path to MAMP's php binaries to the PATH. So in your ~/.bash_profile:
export PATH=/Applications/MAMP/bin/php/php5.3.6/bin:$PATH
Correct it if needed.
Then reopen terminal or execute this: source ~/.bash_profile. Then you can check which php you are using by executing which php. It should point to MAMP's php.