I have a strange problem with Logstash. I am providing a log file as input to logstash. The configuration is as follows:
input {
file {
type => "apache-access"
path => ["C:\Users\spanguluri\Downloads\logstash\bin\test.log"]
}
}
output {
elasticsearch {
protocol => "http"
host => "10.35.143.93"
port => "9200"
index => "latestindex"
}
}
I am running elasticsearch server already and verifying if the data is being received with
curl queries. The problem is, no data is being received when the input is a file. However, if I change input to stdin { } as follows, it sends all input data smoothly:
input {
stdin{ }
}
output {
elasticsearch {
protocol => "http"
host => "10.35.143.93"
port => "9200"
index => "latestindex"
}
}
I don't get where I am going wrong. Can someone please take a look at this?
You should set start_position under your file section:
start_position => "beginning"
It defaults to end and so won't read any existing lines in your file, only newly added ones:
start_position
Value can be any of: "beginning", "end"
Default value is "end"
Choose where Logstash starts initially reading files: at the beginning
or at the end. The default behavior treats files like live streams and
thus starts at the end. If you have old data you want to import, set
this to ‘beginning’
This option only modifies “first contact” situations where a file is
new and not seen before. If a file has already been seen before, this
option has no effect.
In addition to the provided answer, I had to change the path from c:\my\path to c:/my/path in order for it to read the files.
Related
I have the following problem: I am new to logstash and I am trying to simply transfer information from one .log file to another. My config file looks like this and logstash is currently running on Windows 10.
input {
file {
path => "C:/Downloads/Test/input.log"
sincedb_path => "NUL"
start_position => "beginning"
ignore_older => 0
}
}
output {
file {
path => "C:/Downloads/Test/output.log"
}
}
The input file looks like this.
INFO 2018-11-12 13:47:22,378 [AGENT] - [KEY] [METHOD] MESSAGE
The next step would be applying a filter to only transfer the ERROR lines to the output file using a grok filter.
Can anyone help me please?
I would like to know how to update the permissions for a file that is not getting deployed by puppet. The file is actually getting deployed by an RPM. I have tried the following with no luck:
file { '/some/directory/myfile.conf' :
ensure => 'file',
replace => 'no',
owner => 'someuser',
group => 'somegroup',
mode => '0644'
}
This actually removes the content of the file and leaves an empty file. However it sets the right permissions and mode. I would like to keep the content. I am using puppet 2.7.3.
I am pretty new to Logstash and Elasticsearch.
I have a Problem that I am not able to configure out.
My configuration is for testing is a 2 EL nodes running on OS X + Kibana and Logstash. All with the actual stable releases.
I am reading with logstash a log with following informations:
64.12.89.186 {"register":"07-015", "tag":["Server1", "Proxy", "Web", "picture"], "comment":"texttext"}
149.174.107.97 {"register":"07-015", "tag":["Server1", "Proxy", "Web", "picture"], "comment":"texttext"}
149.174.110.102 {"register":"07-015", "tag":["Server1", "Proxy", "Web", "picture"], "comment":"texttext"}
and write them in EL.
Configurationfile from logstash is this:
input {
file {
path => ["/scenario_02/data/ipinfo4_log"]
stat_interval => 1
discover_interval => 5
}
}
filter {
grok {
match => { "message" => "%{IP:ip} %{GREEDYDATA:data}" }
}
json {
source => "data"
}
}
filter {
geoip {
source => "ip"
target => "geoip"
}
}
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}
So far so good. The new informations will be added at the end of the log file.
Now to my Problem: The file are read only when I start logstash.
When in the log file get new informations during logstash is running, no new documents will be written to EL.
When I stop logstash and start it again. The new informations from the log file will be added.
Did I understand something wrong that the informations from the log file will not added or checked automatically in intervals ? Or I have to restart always logstash to read the file again?
Thanks a lot for your help.
Paris
Try:
start_position => "end" after 'path => ...' . It can help, but end is a default for start_position. BTW, if you are using lumberjack? then files always will be checked for a new info by default.
I'm trying to configure the rewrite rules for lighttpd so that a specific cakephp controller-action is executed if a file is not found. The controller-action will generate the data (png file), save it for future use, and then serve it to the client. The next time someone tries to access the file, it will be served by lighttpd directly without executing any php. In other words, the png files are cached so there is no need to recreate them.
From what I can tell, url.rewrite-if-not-file is ignored if rewrite-once has a match. Thus, the following can serve up my cached files but not my uncached files.
url.rewrite-if-not-file = (
"^/scan/(.+)\.png" => "/mycontroller/scan/$1"
)
url.rewrite-once = (
"^/(css|files|img|js)/(.*)" => "/$1/$2",
"^/favicon.ico" => "/favicon.ico",
"^/scan/(.+\.png)" => "/scan/$1",
"^([^\?]*)(\?(.+))?$" => "/index.php?url=$1&$3",
)
The only solution I can think of now is to delete the 3rd rule and modify ^([^\?]*)(\?(.+))?$ so it ignores urls starting with \scan\.
Any other suggestions?
i'd do something like this in a controller:
if (file_exists($file_name)) {
$this->redirect($file_name);
} else {
$this->redirect(array('action' => 'some_action', 'controller' => 'some_controller'));
}
I just wanted to know if there is a way to limit the file input dialog to show only certain kinds of files. My web page only can accept .bin or .gz file types, but the user can select other file types and try to upload them.
What would be the best way to prevent the wrong kind of file from being uploaded?
Here is my controller for file uploading:
public function uploadFile()
{
$this->Session->write('isFileUpload', false);
$this->Session->write('isFileLarge', false);
if($this->request->is('post'))
{
$uploadedFile = array();
// Check if the Document object is set
// If it is set, process the file for uploading,
if(isset($this->request->data['Document']))
{
$filename = $this->request->data['Document']['MyFile']['tmp_name'];
$uploadedFile['MyFile']['name'] = $this->request->data['Document']['MyFile']['name'];
$uploadedFile['MyFile']['type'] = $this->request->data['Document']['MyFile']['type'];
$uploadedFile['MyFile']['size'] = $this->request->data['Document']['MyFile']['size'];
// Move the file to the /home/spectracom folder
$filePath = DS . 'home' . DS . $uploadedFile['MyFile']['name'];
if (move_uploaded_file($filename, $filePath))
{
$this->Session->write('isFileUpload', true);
$this->Session->write('isFileLarge', false);
$this->redirect('/tools/upgradebackup');
}
else
{
$this->Session->write('isFileUpload', false);
$this->Session->write('isFileLarge', true);
$this->redirect('/tools/upgradebackup');
}
}
else
{
$this->Session->write('isFileUpload', false);
$this->Session->write('isFileLarge', true);
$this->redirect('/tools/upgradebackup');
}
}
}
I basically check that the file exists, or else it is too large, and when it returns to the main upgrade page it sets the session variables.
Thanks
You limit what the browser allows the user to select in the file selection dialog using the accept attribute, although not all browsers support it.
I think this should work for creating the input (you'll need to play around with the MIME types to see what works):
echo $this->Form->input('MyFile', array('type' => 'file', 'options' => array('accept' => 'application/gzip,application/gzipped,application/octet-stream')));
You should also validate the files once they arrive on the server by setting up validation on your model (look at extension and mimeType validation rules).
You can also use JavaScript to validate the file extension once it has been selected by the user, and clear the file input field if it has the wrong extension.
Tested with Cakephp 3.4
$this->Form->control('my_file', ['label' => 'Upload File','type' => 'file', 'accept' => 'application/msword']);