I'm running a conf file in logstash which gives the output as required but when i open ls to list file it's not getting created.need help in resolving issue.
conf file:
input {
elasticsearch{
hosts =>"localhost:9200"
index=>"index1"
}
}
output{
file{
path => "/opt/elk/logstash-6.5.0/example.json"
}
stdout{}
}
Result of execution:
[2019-10-13T14:07:47,503][INFO ][logstash.outputs.file ]
Opening file {:path=>"/opt/elk/logstash-6.5.0/example.json"}
{
"#timestamp" => 2019-10-13T08:37:46.715Z,
"example" => [
//data within example//
}
[2019-10-13T16:44:19,982][INFO ][logstash.pipeline] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x4f8848b run>"}
but when I run ls to see example.json, it doesn't show up
ls command:
elastic#es-VirtualBox:~/opt/elk/logstash-6.5.0$ ls
bin CONTRIBUTORS fetch.conf Gemfile.lock lib logs logstash-core-plugin-api NOTICE.TXT throw.conf vendor config data Gemfile grokexample.conf LICENSE.txt logstash-core modules output.json tools x-pack
so was wondering if conf only creates a temporary file?
Related
the error
the code
Please help me fix this, I'm desperately trying to.
You are simply getting this error because in your main.js file, you are expecting the commands to be in the ./commands folder while all the commands are actually in the ./command folder without an s. So all you have to do is either change your command folder's name to commands or you can edit your main.js file and change the command folder's name to ./command like this:
const commandFiles = fs.readdirSync('./command/').filter(file => file.endsWith('.js'))
I will recommend you using glob for command handler npm i glob
With this code you can have file tree like this:
Folder
File in folder
File not in folder
And it will still work well
const { glob } = require('glob')
const { promisify } = require('util');
const globPromise = promisify(glob);
const dir = await globPromise(`${process.cwd()}/commands/**/*.js`);
dir.map((x) => {
const file = require(x)
if(file.name) {
client.commands.set(file.name, file)
}
})
mb u can print "command", not "commandS"
Is there any way to list out files from Hadoop hdfs and store only the file names to the local?
example:
I have a file india_20210517_20210523.csv. I m currently copying the files from hdfs to local using copytolocal command but copying files to local is time-consuming as files are huge. All I need is the name of the files to be stored in a .txt file to perform cut operations using bash script.
Kindly help me
The easiest way to do is to use the below command.
hdfs dfs -ls /path/fileNames | awk '{print $8}' | xargs -n 1 basename > Output.txt
How it works:
hdfs dfs -ls : This will list all the information about the path
awk '{print $8}' : To print the 8th column of the output
xargs -n 1 basename : To get the file names alone excluding the path
> Output.txt : To store the file names to a text file
Hope this answers your question.
If you want to do this programmatically, you can use FileSystem and FileStatus objects from Hadoop to:
list the contents of your (current or another) target directory,
check if each of the records of this directory is either a file or another directory, and
write the name of each file as a new line to a file stored locally.
The code for this type of application can look like this:
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import java.io.File;
import java.io.PrintWriter;
public class Dir_ls
{
public static void main(String[] args) throws Exception
{
// get input directory as a command-line argument
Path inputDir = new Path(args[0]);
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
if(fs.exists(inputDir))
{
// list directory's contents
FileStatus[] fileList = fs.listStatus(inputDir);
// create file and its writer
PrintWriter pw = new PrintWriter(new File("output.txt"));
// scan each record of the contents of the input directory
for(FileStatus file : fileList)
{
if(!file.isDirectory()) // only take into account files
{
System.out.println(file.getPath().getName());
pw.write(file.getPath().getName() + "\n");
}
}
pw.close();
}
else
System.out.println("Directory named \"" + args[0] + "\" doesn't exist.");
}
}
So if we want to list the files from the root (.) directory of HDFS, and we have these as the contents under it (notice how we both have directories and text files):
This will be the command line output of the application:
And this will be what's written inside the output.txt text file stored locally:
My Jenkins pipeline runs on the Slave using agent { node { label 'slave_node1' } }.
I use Jenkins file parameter named uploaded_file and upload a file called hello.pdf
My pipeline contains the following code
stage('Precheck')
{
steps {
sh "echo ${WORKSPACE}"
sh "echo ${uploaded_file}
sh "ls -ltr ${WORKSPACE}/*"
Output:
/web/jenkins/workspace/MYCOPY
hello.pdf
ls: cannot access /web/jenkins/workspace/MYCOPY/* No such file or directory
As you can see no files were found on the slave WORKSPACE.
Can you let me understand if I'm checking for the uploaded file in the correct location i.e under WORKSPACE directory?
How can I get the file uploaded to the slave's WORKSPACE?
I'm on jenkins version 2.249.1
Can I get this to work at least on the latest version of Jenkins ?
So do you have a fixed file that that is copied in every build? I.e its the same file?
In that case you can save it as a secret file in jenkins and do the following:
environment {
FILE = credentials('my_file')
}
stages {
stage('Preperation'){
steps {
// Copy your fie to the workspace
sh "cp ${FILE} ${WORKSPACE}"
// Verify the file is copied.
sh "ls -la"
}
}
}
This question already has an answer here:
Upload file in Jenkins input step to workspace
(1 answer)
Closed 3 years ago.
Upload (per progress bottom bar status) goes through in Jenkins. But don't see option to locate or fetch uploaded file from Jenkins.
Here is simple script with File parameters,
properties(
[
parameters(
[ file(name: "file1.zip", description: 'Choose path to upload file1.zip from local system.'),
file(name: "file2.zip", description: 'Choose path to upload file2.zip from local system.') ]
)
]
)
node {
stage("Fetch Uploaded File") {
sh '''
ls -l file1.zip file2.zip
ls -l ${WORKSPACE}/file1.zip ${WORKSPACE}/file2.zip
'''
}
}
Tried with def input file option per other post, but no luck of reaching uploaded file. Any inputs?
def inputFile = input message: 'Upload file', parameters: [file(name: 'data.ear')]
new hudson.FilePath(new File("$workspace/data.ear")).copyFrom(inputFile)
inputFile.delete()
With scripted full pipeline pasted above, getting below error..
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Fetch Uploaded File)
[Pipeline] sh
[testSh] Running shell script
+ ls -l file1.zip file2.zip
ls: cannot access file1.zip: No such file or directory
ls: cannot access file2.zip: No such file or directory
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE
This is a known bug JENKINS-27413.
There is however a library with a workaround that can help you https://github.com/janvrany/jenkinsci-unstashParam-library. As described in the readme you can add this library to your Jenkins (Extending with Shared Libraries) and use it the following way:
library "jenkinsci-unstashParam-library"
node {
def file_in_workspace = unstashParam "file"
sh "cat ${file_in_workspace}"
}
Is there a way to not use the tilde character in the create-react-app build js file names?
After I run npm run build on my create-react-app, I get a couple runtime~main.[hash].js files in the /static/js/ folder . My destination file system does not allow tilde characters in file or folder names.
Can I change config somewhere to output runtime-main.[hash].js with a - instead of tilde??
https://facebook.github.io/create-react-app/docs/production-build
Solved without ejecting:
-Install rescripts: npm install #rescripts/cli --save-dev in your project
-Create a file .rescriptsrc.js at the root directory of your project and put the code below inside:
module.exports = config => {
config.optimization.splitChunks.automaticNameDelimiter = '_';
if (config.optimization.runtimeChunk) {
config.optimization.runtimeChunk = {
name: entrypoint => `runtime_${entrypoint.name}`
};
}
return config;
};
In package.json in sripts section add a new entry "build-re": "rescripts build"
Now when running npm run build-re it will generate _ instead of ~ in file names.
https://github.com/harrysolovay/rescripts