My Kibana5.6.8 logstash configuration seems only reading one log file
My logstash.conf on /home/elastichsearch/confLogs is
input {
file {
type => "static"
path => "/home/elasticsearch/static_logs/**/*Web.log*" exclude => "*.zip"
start_position => beginning
sincedb_path => "/dev/null"
}
}
filter {
if [type] == "static" {
if [message] !~ /(.+)/ {
drop { }
}
grok{
patterns_dir => "./patterns"
overwrite => [ "message" ]
# 2017-08-07 11:47:35,466 INFO [http-bio-10.60.2.19-10267-exec-60] jsch.DeployManagerFileUSImpl (DeployManagerFileUSImpl.java:155) - Deconnexion de l'hote qvizzza3
# 2017-08-07 11:47:51,775 ERROR [http-bio-10.60.2.19-10267-exec-54] service.BindingsRSImpl (BindingsRSImpl.java:143) - Can't find bindings file deployed on server
# 2017-08-03 16:01:11,352 WARN [Thread-552] pcf2.AbstractObjetMQDAO (AbstractObjetMQDAO.java:137) - Descripteur de
match => [ "message", "%{TIMESTAMP_ISO8601:logdate},%{INT} %{LOGLEVEL:logLevel} \[(?<threadname>[^\]]+)\] %{JAVACLASS:package} \(%{JAVAFILE:className}:%{INT:line}\) - %{GREEDYDATA:message}" ]
}
# 2017-08-03 16:01:11,352
date{
match => [ "logdate", "YYYY-MM-dd hh:mm:ss" ]
target => "logdate"
}
}
}
output {
elasticsearch { hosts => ["192.168.99.100:9200"]}
My logs directory, with load balanced logrotate files
static_logs
--prd1
----mlog Web.log
----mlog Web.log.1
----mlog Web.log.2
--prd2
----mlog Web.log
----mlog Web.log.2
Where is my mistake ?
My patterns are on /home/elasticsearch/confLogs/patterns/grok-patterns qui with TIMESTAMP_ISO8601
Regards
If my log files are more 140M, logdate filter is not viewing as an date field, but as an string field !!!
Related
i making map using geoip, elastic stack
but i have problem
docker-elk, filebeat
input {
beats {
port => 5000
}
}
filter {
grok {
match => { "message" => "%{COMMONAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
target => "geoip"
database => "/home/semicolon/DDYZD/ELK/GeoLite2-City-CSV_20210202/GeoLite2-City-Blocks-IPv4.csv"
}
}
## Add your filters / logstash plugins configuration here
output {
elasticsearch {
hosts => ":9200"
user => ""
password => ""
index => 'logstash-ddyzd-%{+YYYY.MM.dd}'
}
}
this is my logstash pipeline code, and User and password are filled in
docker-compose up than make error
[2021-02-06T13:20:10,419][ERROR][logstash.filters.geoip ] Invalid setting for geoip filter plugin:
logstash_1 |
logstash_1 | filter {
logstash_1 | geoip {
logstash_1 | # This setting must be a path
logstash_1 | # File does not exist or cannot be opened /home/semicolon/DDYZD/ELK/GeoLite2-City-CSV_20210202/GeoLite2-City-Blocks-IPv4.csv
logstash_1 | database => "/home/semicolon/DDYZD/ELK/GeoLite2-City-CSV_20210202/GeoLite2-City-Blocks-IPv4.csv"
logstash_1 | ...
logstash_1 | }
logstash_1 | }
like this
Database is exist at /home/semicolon/DDYZD/ELK/GeoLite2-City-CSV_20210202/GeoLite2-City-Blocks-IPv4.csv
and i also try mmdb file, but same result
please help me
I am trying to upload file in cakephp version 4.
I am following this docs
I have tried in controller
if ($this->request->is('post')) {
$image = $this->request->getData('image');
$fileName = $image->getClientFilename();
$targetPath = WWW_ROOT.'img'.DS.$fileName;
$image->moveTo($targetPath);
$user = $this->Users->patchEntity($user, $this->request->getData()); //line 58
$user->image = $fileName;
$this->Users->save($user);
}
Image upload is working fine, name also saving in database. But when a validation error occur I am getting
Warning (4096): Object of class Laminas\Diactoros\UploadedFile could not be converted to string [CORE\src\Database\Type\StringType.php, line 97]
logs
Cake\Database\Type\StringType::marshal() - CORE\src\Database\Type\StringType.php, line 97
Cake\ORM\Marshaller::Cake\ORM\{closure}() - CORE\src\ORM\Marshaller.php, line 78
Cake\ORM\Marshaller::merge() - CORE\src\ORM\Marshaller.php, line 558
Cake\ORM\Table::patchEntity() - CORE\src\ORM\Table.php, line 2761
App\Controller\UsersController::add() - APP/Controller\UsersController.php, line 58
Cake\Controller\Controller::invokeAction() - CORE\src\Controller\Controller.php, line 524
Cake\Controller\ControllerFactory::invoke() - CORE\src\Controller\ControllerFactory.php, line 79
Cake\Http\BaseApplication::handle() - CORE\src\Http\BaseApplication.php, line 229
Cake\Http\Runner::handle() - CORE\src\Http\Runner.php, line 77
Cake\Http\Runner::handle() - CORE\src\Http\Runner.php, line 77
Cake\Http\Middleware\CsrfProtectionMiddleware::process() - CORE\src\Http\Middleware\CsrfProtectionMiddleware.php, line 132
Cake\Http\Runner::handle() - CORE\src\Http\Runner.php, line 73
Cake\Http\Runner::run() - CORE\src\Http\Runner.php, line 58
Cake\Routing\Middleware\RoutingMiddleware::process() - CORE\src\Routing\Middleware\RoutingMiddleware.php, line 162
Cake\Http\Runner::handle() - CORE\src\Http\Runner.php, line 73
Cake\Routing\Middleware\AssetMiddleware::process() - CORE\src\Routing\Middleware\AssetMiddleware.php, line 68
First of all, your code example blindly trusts the client, it accepts any file, and allows to choose arbitrary target locations by passing a path as the filename! Never use client data without validating/sanitizing it, trusting the file date provided by the client can lead to all sorts of vulnerabilities!!!
That being said, usually you either use a separate field for the file upload, or a custom database type that doesn't transform the data when it is being marshalled, so that you can transform it manually afterwards. Currently you seem to use the field for the upload that is supposed to hold a string in the database, hence the marshaller will try to convert the input to a string, and that will of course fail.
So for example, rename the form control from image to image_file (no such column should exist in the database), add proper validation rules for image_file, something along the lines of this, to ensure that the upload is valid, ie check the type, the size, the name, etc:
$validator
->notEmptyFile('image_file')
->uploadedFile('image_file', [
'types' => ['image/png'], // only PNG image files
'minSize' => 1024, // Min 1 KB
'maxSize' => 1024 * 1024 // Max 1 MB
])
->add('image_file', 'minSize', [
'rule' => ['imageSize', [
// Min 10x10 pixel
'width' => [Validation::COMPARE_GREATER_OR_EQUAL, 10],
'height' => [Validation::COMPARE_GREATER_OR_EQUAL, 10],
]]
])
->add('image_file', 'maxSize', [
'rule' => ['imageSize', [
// Max 100x100 pixel
'width' => [Validation::COMPARE_LESS_OR_EQUAL, 100],
'height' => [Validation::COMPARE_LESS_OR_EQUAL, 100],
]]
])
->add('image_file', 'filename', [
'rule' => function (UploadedFileInterface $file) {
// filename must not be a path
$filename = $file->getClientFilename();
if (strcmp(basename($filename), $filename) === 0) {
return true;
}
return false;
}
])
->add('image_file', 'extension', [
'rule' => ['extension', ['png']] // .png file extension only
]);
And then handle the upload after patching, ie after validation has taken place, and only move the file if validation was successful!
$user = $this->Users->patchEntity($user, $this->request->getData());
if (!$user->getErrors()) {
// never trust anything in `$image` if you haven't properly validated it!!!
$image = $this->request->getData('image_file');
$fileName = $image->getClientFilename();
// if you want to take it a step further, drop the user supplied filename
// $fileName = \Cake\Utility\Security::hash($image->getClientFilename(), 'sha1');
// and re-encode the image to remove possible dangerous payloads, meta data, etc
// reEncodeImage($image->getStream());
$image->moveTo(WWW_ROOT . 'img' . DS . $fileName);
$user->image = $fileName;
}
if ($this->Users->save($user)) {
// success
} else {
// failure
}
Once you've got a grip on how things work, you should probably consider moving the upload handling logic in your table class or a behavior (the beforeSave() event/callback is very popular for executing upload logic), or some kind of service.
You may also want to have a look at how existing plugins are handling uploads, see for example https://github.com/FriendsOfCake/awesome-cakephp#user-content-files
// add
public function add()
{
$info = $this->Infos->newEmptyEntity(); //debug($info);
$this->Authorization->authorize($info);
if ($this->request->is('post')) {
$data = $this->request->getData(); //debug($data);
$dimage = $this->request->getData('image');
$name = $dimage->getClientFilename(); //debug($name);
$type = $dimage->getClientMediaType(); //debug($type);
$size = $dimage->getSize(); //debug($size);
$tmpName = $dimage->getStream()->getMetadata('uri'); //debug($tmpName);
$error = $dimage->getError(); //debug($error);
$targetPath = WWW_ROOT . 'img' . DS . 'infos' . DS . $name; //debug($targetPath);
if ($type == 'image/jpeg' || $type == 'image/jpg' || $type == 'image/png') {
if (!empty($name)) {
if ( ($size > 0) && ($size < 1048576) && ($error == 0)) { // less 1MB
$dimage->moveTo($targetPath);
}
}
}
$data['image'] = $name; //debug($info); //debug($data);
$info = $this->Infos->patchEntity($info, $data);
$info->user_id = $this->request->getAttribute('identity')->getIdentifier();
if ($this->Infos->save($info)) {
$this->Flash->success(__('The info has been saved.'));
return $this->redirect(['controller' => 'Infos', 'action' => 'view', $info->id]);
} else {
$this->Flash->error(__('The info could not be saved. Please, try again.'));
return $this->redirect(['controller'=>'Infos', 'action' => 'add']);
}
}
$infoCategories = $this->Infos->InfoCategories->find('list', ['limit' => 200])->all();
$this->set(compact('info','infoCategories'));
}
Good afternoon !
I am creating a private chat , and I get the first objective , get the message and users with the following code.
public function getEnviarMensajes($id,$identificador){
$user = User::find($identificador);
$idReceptor = $user->id;
$idEmisor = Auth::user()->id;
$mesageuser = MessageUser::with('user')->with('message')->get();
$name = [];
$content = [];
foreach($mesageuser as $users){
foreach($users->message as $mensajes){
if($users->user->id==$idEmisor){
$name[] = $users->user->name;
echo "<p>";
$content[] = $mensajes->contenido;
echo "<p>";
echo $mensajes->created_at;
echo "<p>";
}
if($users->user->id==$idReceptor){
$name[] = $users->user->name;
echo "<p>";
$content[] = $mensajes->contenido;
echo "<p>";
echo $mensajes->created_at;
echo "<p>";
}
}
}
However , I have a problem , this results i need to "join" , in many ocasions I used an array , but in this case ? How I can save differents rows in Laravel ? , also I need to sort this content for datetime row.
Could anyone helps to me ?
I think you can cut your code at this way:
$mesageuser = MessageUser::with('user')->with('message')->get();
$messages = [];
foreach($mesageuser as $users){
foreach($users->message as $mensajes){
$messages[] = [
'nombre' => $users->user->name,
'contenido' => $mensajes->contenido,
'creado' => $mensajes->created_at,
'emisor' => ($users->user->id==$idEmisor) // true or false
];
}
}
return view('chat', ['messages' => $messages]);
The two if statements are useless because inside of them the code is the same. And I had used an multidimensional array in order to store all the messages with the following structure:
[
[
'nombre' => 'Marta',
'contenido' => 'Hola',
'creado' => '24/03/2015',
'emisor' => false
],
[
'nombre' => 'Salomé',
'contenido' => 'adiós',
'creado' => '24/03/2015',
'emisor' => true
]
]
If you want, for example, order by date, then edit your query:
$mesageuser = MessageUser::with('user')->with(['message' => function($query){
$query->orderBy('created_at', 'desc');
}])->get();
I am using mime_content_type() in PHP 5.5 to get a MIME type, but it throws fatal: error function not found.
How can I achieve this on PHP 5.5?
Make use of the finfo() functions.
A simple illustration:
<?php
$finfo = finfo_open(FILEINFO_MIME_TYPE);
echo finfo_file($finfo, "path/to/image_dir/image.gif");
finfo_close($finfo);
OUTPUT :
image/gif
Note : Windows users must include the bundled php_fileinfo.dll DLL file in php.ini to enable this extension.
I've spent too much time trying to get the finfo functions to work, properly. I finally just ended up creating my own function to match the file extension to any array of mime types. It's not a full-proof way of assuring that the files are truly what the extension denotes them to be, but that problem can be mitigated by how you process I/O of said files on your server(s).
function mime_type($file) {
// there's a bug that doesn't properly detect
// the mime type of css files
// https://bugs.php.net/bug.php?id=53035
// so the following is used, instead
// src: http://www.freeformatter.com/mime-types-list.html#mime-types-list
$mime_type = array(
"3dml" => "text/vnd.in3d.3dml",
"3g2" => "video/3gpp2",
"3gp" => "video/3gpp",
"7z" => "application/x-7z-compressed",
"aab" => "application/x-authorware-bin",
"aac" => "audio/x-aac",
"aam" => "application/x-authorware-map",
"aas" => "application/x-authorware-seg",
"abw" => "application/x-abiword",
"ac" => "application/pkix-attr-cert",
"acc" => "application/vnd.americandynamics.acc",
"ace" => "application/x-ace-compressed",
"acu" => "application/vnd.acucobol",
"adp" => "audio/adpcm",
"aep" => "application/vnd.audiograph",
"afp" => "application/vnd.ibm.modcap",
"ahead" => "application/vnd.ahead.space",
"ai" => "application/postscript",
"aif" => "audio/x-aiff",
"air" => "application/vnd.adobe.air-application-installer-package+zip",
"ait" => "application/vnd.dvb.ait",
"ami" => "application/vnd.amiga.ami",
"apk" => "application/vnd.android.package-archive",
"application" => "application/x-ms-application",
// etc...
// truncated due to Stack Overflow's character limit in posts
);
$extension = \strtolower(\pathinfo($file, \PATHINFO_EXTENSION));
if (isset($mime_type[$extension])) {
return $mime_type[$extension];
} else {
throw new \Exception("Unknown file type");
}
}
Edit:
I'd like to address Davuz's comment (since it keeps getting up-voted) and remind everyone that I put in the pseudo disclaimer at the top that this isn't "full-proof." So, please keep that in mind when considering the approach I've offered in my answer.
mime_content_type() is not deprecated and works fine.
Why is mime_content_type() deprecated in PHP?
http://php.net/manual/en/function.mime-content-type.php
As of PHP 5.3, it's even built-in.
$finfo = finfo_open(FILEINFO_MIME_TYPE); should do it.
Taken from the php.net docs. Your function is deprecated and probably already removed.
http://www.php.net/manual/en/function.finfo-file.php
You should understand that file_get_contents will upload whole file to the memory, it is not good way to get only mime type. You don't need to use buffer method and file_get_contents function in this case.
To prevent any errors and warnings, better do like this.
$filename = 'path to your file';
if (class_exists('finfo')) {
$finfo = new finfo(FILEINFO_MIME_TYPE);
if (is_object($finfo)) {
echo $finfo->file($filename);
}
} else {
echo 'fileinfo did not installed';
}
Also you should know $finfo->file will throw PHP Warning if it fail.
If fileinfo is not installed properly, and you have a fresh version of PHP, you can get mime type from headers.
You can use cURL to get mime type from headers.
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_HEADER => true,
CURLOPT_NOBODY => true,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_SSL_VERIFYHOST => false,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_MAXREDIRS => 1,
CURLOPT_URL => $link)
);
$headers = curl_exec($ch);
curl_close($ch);
if (preg_match('/Content-Type:\s(.*)/i', $headers, $matches)) {
echo trim($matches[1], "\t\n\r");
}else {
echo 'There is no content type in the headers!';
}
Also you can use get_headers function, but it more slow than cURL request.
$url = 'http://www.example.com';
$headers = get_headers($url, 1);
echo $headers['Content-Type'];
Get the image size using:
$infFil=getimagesize($the_file_name);
and
echo $infFil["mime"]
The getimagesize returns an associative array which have a MIME key and obviously the image size too
I used it and it works
I use the MimeTypeTool from Bat (https://github.com/lingtalfi/Bat)
It uses fileinfo if available, and defaults back to an "extension => mime type" mapping otherwise.
This is the best solution I found by combining two very good posts
// Thanks to http://php.net/manual/en/function.mime-content-type.php#87856
function getMimeContentType($filename, $ext)
{
if(!function_exists('mime_content_type'))
{
if($mime_types = getMimeTypes())
{
if (array_key_exists($ext, $mime_types))
{
return $mime_types[$ext];
}
elseif (function_exists('finfo_open'))
{
$finfo = finfo_open(FILEINFO_MIME);
$mimetype = finfo_file($finfo, $filename);
finfo_close($finfo);
return $mimetype;
}
}
return 'application/octet-stream';
}
return mime_content_type($filename);
}
// Thanks to http://php.net/manual/en/function.mime-content-type.php#107798
function getMimeTypes()
{
$url = 'http://svn.apache.org/repos/asf/httpd/httpd/trunk/docs/conf/mime.types';
$mimes = array();
foreach(#explode("\n",#file_get_contents($url)) as $x)
{
if(isset($x[0]) && $x[0]!=='#' && preg_match_all('#([^\s]+)#', $x, $out) && isset($out[1]) && ($c = count($out[1])) > 1)
{
for($i=1; $i < $c; $i++)
{
$mimes[$out[1][$i]] = $out[1][0];
}
}
}
return (#sort($mimes)) ? $mimes : false;
}
Use it link this:
$filename = '/path/to/the/file.pdf';
$ext = strtolower(array_pop(explode('.',$filename)));
$content_type = getMimeContentType($filename, $ext);
Will continue to work even if the mime_content_type function is no longer supported in php.
I am trying to store the FBIDs of a Facebook user's friends in the column of a mysql database. I've tried looking up other answer on this issue, and I have tried to implement it (in Laravel 4). Here is what I have done:
In the Facebook.php file, one of the providers:
'friends' => 'https://graph.facebook.com/me/friends?access_token='.$token->access_token
In my Oauth2 Controller:
$friends_list = $user['friends'];
$friends_list_array = json_decode($friends_list,true);
$arr= $friends_list_array['data'];
$friend_ids_arr = array();
foreach($arr as $friend) {
$friend_ids_arr[] = $friend['id'];
}
$friend_ids = implode("," , $friend_ids_arr);
And then I want to store the $friend_ids object in a "text" column in my database. However, when running this, I keep getting the error: Invalid argument supplied for foreach()
But it is very clearly being supplied an array as it should. Is there something I'm not seeing? Thank you for your help.
Actually the returned result is a json, the returned object should look something like this
{
"id": "xxxxxxx",
"name": "Sheikh Heera",
"friends": {
"data": [
{ "name": "RaseL KhaN", "id": "xxx" },
{ "name": "Yizel Herrera", "id": "xxx" }
],
"paging": {
"next": "https://graph.facebook.com/xxx/friends?limit=..."
}
}
}
After you json_decode
$user = json_decode($user, true);
It should look something like
Array
(
[id] => xxxxxxx
[name] => Sheikh Heera
[friends] => Array
(
[data] => Array
(
[0] => Array
(
[name] => RaseL KhaN
[id] => xxx
)
[1] => Array
(
[name] => Yizel Herrera
[id] => xxx
)
)
[paging] => Array
(
[next] => https://graph.facebook.com/xxx/friends?limit=...
)
)
)
So, now you can
$friends_list = $user['friends'];
$data = $friends_list['data'];
Make sure your $data array is not empty and then loop
if(count($data)) {
$friend_ids_arr = array();
foreach($data as $friend) {
$friend_ids_arr[] = $friend['id'];
}
}
So, the foreach will run only when $data has items in it.
Update: It may help you
$url = "https://graph.facebook.com/me?fields=id,name,friends&access_token=YOUR_ACCESS_TOKEN";
$contents = json_decode(file_get_contents($url), true);
$friends = $contents['friends'];
$friend_ids_arr[]
foreach($friends['data'] as $friend)
{
$friend_ids_arr[] = $friend['id'];
}