Fetching uploaded files in Jenkins [duplicate] - file

This question already has an answer here:
Upload file in Jenkins input step to workspace
(1 answer)
Closed 3 years ago.
Upload (per progress bottom bar status) goes through in Jenkins. But don't see option to locate or fetch uploaded file from Jenkins.
Here is simple script with File parameters,
properties(
[
parameters(
[ file(name: "file1.zip", description: 'Choose path to upload file1.zip from local system.'),
file(name: "file2.zip", description: 'Choose path to upload file2.zip from local system.') ]
)
]
)
node {
stage("Fetch Uploaded File") {
sh '''
ls -l file1.zip file2.zip
ls -l ${WORKSPACE}/file1.zip ${WORKSPACE}/file2.zip
'''
}
}
Tried with def input file option per other post, but no luck of reaching uploaded file. Any inputs?
def inputFile = input message: 'Upload file', parameters: [file(name: 'data.ear')]
new hudson.FilePath(new File("$workspace/data.ear")).copyFrom(inputFile)
inputFile.delete()
With scripted full pipeline pasted above, getting below error..
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Fetch Uploaded File)
[Pipeline] sh
[testSh] Running shell script
+ ls -l file1.zip file2.zip
ls: cannot access file1.zip: No such file or directory
ls: cannot access file2.zip: No such file or directory
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

This is a known bug JENKINS-27413.
There is however a library with a workaround that can help you https://github.com/janvrany/jenkinsci-unstashParam-library. As described in the readme you can add this library to your Jenkins (Extending with Shared Libraries) and use it the following way:
library "jenkinsci-unstashParam-library"
node {
def file_in_workspace = unstashParam "file"
sh "cat ${file_in_workspace}"
}

Related

How to use Jenkins file parameter in Pipelines

My Jenkins pipeline runs on the Slave using agent { node { label 'slave_node1' } }.
I use Jenkins file parameter named uploaded_file and upload a file called hello.pdf
My pipeline contains the following code
stage('Precheck')
{
steps {
sh "echo ${WORKSPACE}"
sh "echo ${uploaded_file}
sh "ls -ltr ${WORKSPACE}/*"
Output:
/web/jenkins/workspace/MYCOPY
hello.pdf
ls: cannot access /web/jenkins/workspace/MYCOPY/* No such file or directory
As you can see no files were found on the slave WORKSPACE.
Can you let me understand if I'm checking for the uploaded file in the correct location i.e under WORKSPACE directory?
How can I get the file uploaded to the slave's WORKSPACE?
I'm on jenkins version 2.249.1
Can I get this to work at least on the latest version of Jenkins ?
So do you have a fixed file that that is copied in every build? I.e its the same file?
In that case you can save it as a secret file in jenkins and do the following:
environment {
FILE = credentials('my_file')
}
stages {
stage('Preperation'){
steps {
// Copy your fie to the workspace
sh "cp ${FILE} ${WORKSPACE}"
// Verify the file is copied.
sh "ls -la"
}
}
}

Executing bteq using SSH

I try to run a script using SSH. The files are these:
test.sh
#!/bin/sh
export logfile=/home/OprBCIDe/logout.log
export bteqfile=/home/OprBCIDe/test_bteq.btq
echo "Test BTEQ"
bteq < $bteqfile > $logfile 2>&1
test_bteq.btq
.SET ERROROUT STDOUT;
.RUN FILE = /home/OprBCIDe/logon.ini
.EXPORT data file=/home/OprBCIDe/sample.csv
.SET SEPARATOR '|'
SELECT * FROM ds_edw_temp.TEST SAMPLE 100;
.EXPORT reset
.LOGOFF;
.EXIT;
The logon.ini file has the credentials for Teradata.
When I run test.sh file in the local machine, works without problems. But, when I use SSH, the command return this in the logout.log
CLI:Message catalog open failed!: No such file or directory
The file "errmsg.cat" cannot be opened.
There may be problems with your installation.
*** CLI error: -1 Message Not Found!
*** Return code from CLI is: -1
*** Error: Fatal error from CLI.
*** Program exiting!
*** Exiting BTEQ...
*** RC (return code) = 8
I search the problem with errmsg.cat, but I not found nothing specified with Teradata.
EDIT:
I found the solution. Only export COPERR in test.sh with the errmsg.cat location on the remote file system.

What should go in the Dockerfile when using Rasa with Google Cloud Platform

I’m using Google Cloud Platform and Google App Engine (flexible) for my Rasa Opensource chatbot.
I've downloaded the Google Cloud SDK to my local machine and deployed the chatbot using gcloud app deploy to App Engine (I think I'm right that this creates a docker image).
App Engine requires a Dockerfile and an app.yaml file. I just wasn't sure what should go in the Dockerfile. And having some trouble point to the model endpoint where it doesn't think its a zip file.
The Dockerfile looks like:
FROM rasa/rasa
ENV BOT_ENV=production
COPY . /var/www
WORKDIR /var/www
RUN pip install rasa
ENTRYPOINT [ "rasa", "run", "-m", "./models", "-vv", "--enable-api", "--endpoints", "endpoints.yml", "--credentials", "credentials.yml", "-p", "8080"]
C:\Users\xxxxxxxx\Desktop\xxxxxxxxx>gcloud app deploy --version chatbot15
Services to deploy:
descriptor: [C:\Users\xxxxxx\Desktop\xxxxxxxxx\app.yaml]
source: [C:\Users\xxxxxx\Desktop\xxxxxxxx]
target project: [xxxxx-xxxxxxx]
target service: [default]
target version: [chatbot15]
target url: [https://xxxxxxxx.appspot.com]
Do you want to continue (Y/n)? y
Beginning deployment of service [default]...
Building and pushing image for service [default]
Started cloud build [.
To see logs in the Cloud Console: https://console.cloud.google.com/gcr/builds/xxxxxx41f1f-c289-xxxxxxx-9ba1-xxxxxxxxxxx]
------------------------------------------------- REMOTE BUILD OUTPUT --------------------------------------------------
starting build "xxxxxx41f1f-c289-xxxxxxx-9ba1-xxxxxxxxxxx]"
FETCHSOURCE
Fetching storage object: gs://staging.xxxxxxxxxx.appspot.com/us.gcr.io/xxxxxxxxxx/appengine/default.chatbot15:latest#1xxxxxxxxxxx0
Copying gs://staging.xxxxxxxxxxx.appspot.com/us.gcr.io/xxxxxxxxxx/appengine/default.chatbot15:latest#158xxxxxxxxxx2560...
\ [1 files][ 24.3 MiB/ 24.3 MiB]
Operation completed over 1 objects/24.3 MiB.
BUILD
Already have image (with digest): gcr.io/cloud-builders/docker
Sending build context to Docker daemon 38.15MB
Step 1/6 : FROM rasa/rasa
latest: Pulling from rasa/rasa
Digest: sha256:cc6a48c7cxxxc8f1c8c3bxxxxxxxxxxxxxxxxef266a58121f41e22b418c6a
Status: Downloaded newer image for rasa/rasa:latest
---> fcf204982b0f
Step 2/6 : ENV BOT_ENV=production
---> Running in 708a75c57f16
Removing intermediate container 708a75c57f16
---> 780c38217741
Step 3/6 : COPY . /var/www
---> a32cfeb03a84
Step 4/6 : WORKDIR /var/www
---> Running in 4f4f0446c8a4
Removing intermediate container 4f4f0446c8a4
---> 1751f3ffc062
Step 5/6 : RUN pip install rasa
---> Running in b70c61ca4670
[91mWARNING: The directory '/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
[0mRemoving intermediate container b70cxxxx670
---> 3e8xxx0f91c
Step 6/6 : ENTRYPOINT [ "rasa", "run", "-m", "./models", "-vv", "--enable-api", "--endpoints", "endpoints.yml", "--credentials", "credentials.yml", "-p", "8080"]
---> Running in 4xxxxxx6
Removing intermediate container 4xxxxxxxxxx6
---> 6fxxxxxxxxx30
Successfully built 6xxxxxxxxxxx30
Successfully tagged us.gcr.io/xxxxxxxxxx/appengine/default.chatbot15:latest
PUSH
Pushing us.gcr.io/xxxxxxxxxxx/appengine/default.chatbot15:latest
The push refers to repository [us.gcr.io/xxxxxxxxxxx/appengine/default.chatbot15]
0da5206fa56e: Pushed
latest: digest: sha256:d1cxxxxxxxxxxxxxxxxxxxxxxxx724xxxxxx size: 2837
DONE
------------------------------------------------------------------------------------------------------------------------
Updating service [default] (this may take several minutes)...failed.
ERROR: (gcloud.app.deploy) Error Response: [9]
Application startup error:
2020-01-27 19:54:37 DEBUG rasa.core.utils - Available web server routes:
/conversations/<conversation_id>/messages POST add_message
/conversations/<conversation_id>/tracker/events POST append_events
/webhooks/rasa GET custom_webhook_RasaChatInput.health
/webhooks/rasa/webhook POST custom_webhook_RasaChatInput.receive
/webhooks/rest GET custom_webhook_RestInput.health
/webhooks/rest/webhook POST custom_webhook_RestInput.receive
/model/test/intents POST evaluate_intents
/model/test/stories POST evaluate_stories
/conversations/<conversation_id>/execute POST execute_action
/domain GET get_domain
/socket.io GET handle_request
/ GET hello
/model PUT load_model
/model/parse POST parse
/conversations/<conversation_id>/predict POST predict
/conversations/<conversation_id>/tracker/events PUT replace_events
/conversations/<conversation_id>/story GET retrieve_story
/conversations/<conversation_id>/tracker GET retrieve_tracker
/webhooks/socketio GET socketio_webhook.health
/status GET status
/model/predict POST tracker_predict
/model/train POST train
/conversations/<conversation_id>/trigger_intent POST trigger_intent
/model DELETE unload_model
/version GET version
2020-01-27 19:54:37 INFO root - Starting Rasa server on http://localhost:8080
2020-01-27 19:54:37 DEBUG rasa.core.utils - Using the default number of Sanic workers (1).
2020-01-27 19:54:37 INFO root - Enabling coroutine debugging. Loop id 94796527495608.
2020-01-27 19:54:37 DEBUG rasa.model - Extracted model to '/tmp/tmpq4hewp3_'.
2020-01-27 19:54:41.930821: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303)
2020-01-27 19:54:42 DEBUG rasa.core.tracker_store - Connected to InMemoryTrackerStore.
2020-01-27 19:54:42 DEBUG rasa.core.lock_store - Connected to lock store 'InMemoryLockStore'.
2020-01-27 19:54:42 DEBUG rasa.core.nlg.generator - Instantiated NLG to 'TemplatedNaturalLanguageGenerator'.
2020-01-27 19:54:42 DEBUG rasa.core.agent - Requesting model from server http://xxxxxxxxxxx.appspot.com/?hl=en-GB/models/20200123-205743.tar.gz...
2020-01-27 19:54:42 ERROR rasa.core.agent - Could not load model due to File is not a zip file.
[2020-01-27 19:54:42 +0000] [1] [ERROR] Experienced exception while trying to serve
Traceback (most recent call last):
File "/build/lib/python3.6/site-packages/rasa/utils/io.py", line 195, in unarchive
tar = tarfile.open(fileobj=IOReader(byte_array))
File "/usr/local/lib/python3.6/tarfile.py", line 1576, in open
raise ReadError("file could not be opened successfully")
tarfile.ReadError: file could not be opened successfully
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/build/lib/python3.6/site-packages/sanic/app.py", line 1133, in run
serve(**server_settings)
File "/build/lib/python3.6/site-packages/sanic/server.py", line 857, in serve
trigger_events(before_start, loop)
File "/build/lib/python3.6/site-packages/sanic/server.py", line 634, in trigger_events
loop.run_until_complete(result)
File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
File "/build/lib/python3.6/site-packages/rasa/core/run.py", line 247, in load_agent_on_start
action_endpoint=endpoints.action,
File "/build/lib/python3.6/site-packages/rasa/core/agent.py", line 248, in load_agent
model_server,
File "/build/lib/python3.6/site-packages/rasa/core/agent.py", line 64, in load_from_server
await _update_model_from_server(model_server, agent)
File "/build/lib/python3.6/site-packages/rasa/core/agent.py", line 122, in _update_model_from_server
model_server, agent.fingerprint
File "/build/lib/python3.6/site-packages/rasa/core/agent.py", line 180, in _pull_model_and_fingerprint
rasa.utils.io.unarchive(await resp.read(), model_directory)
File "/build/lib/python3.6/site-packages/rasa/utils/io.py", line 200, in unarchive
zip_ref = zipfile.ZipFile(IOReader(byte_array))
File "/usr/local/lib/python3.6/zipfile.py", line 1131, in __init__
self._RealGetContents()
File "/usr/local/lib/python3.6/zipfile.py", line 1198, in _RealGetContents
raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
Traceback (most recent call last):
File "/build/lib/python3.6/site-packages/rasa/utils/io.py", line 195, in unarchive
tar = tarfile.open(fileobj=IOReader(byte_array))
File "/usr/local/lib/python3.6/tarfile.py", line 1576, in open
raise ReadError("file could not be opened successfully")
tarfile.ReadError: file could not be opened successfully
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/build/bin/rasa", line 8, in <module>
sys.exit(main())
File "/build/lib/python3.6/site-packages/rasa/__main__.py", line 76, in main
cmdline_arguments.func(cmdline_arguments)
File "/build/lib/python3.6/site-packages/rasa/cli/run.py", line 83, in run
rasa.run(**vars(args))
File "/build/lib/python3.6/site-packages/rasa/run.py", line 56, in run
**kwargs,
File "/build/lib/python3.6/site-packages/rasa/core/run.py", line 206, in serve_application
endpoints.lock_store if endpoints else None
File "/build/lib/python3.6/site-packages/sanic/app.py", line 1133, in run
serve(**server_settings)
File "/build/lib/python3.6/site-packages/sanic/server.py", line 857, in serve
trigger_events(before_start, loop)
File "/build/lib/python3.6/site-packages/sanic/server.py", line 634, in trigger_events
loop.run_until_complete(result)
File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
File "/build/lib/python3.6/site-packages/rasa/core/run.py", line 247, in load_agent_on_start
action_endpoint=endpoints.action,
File "/build/lib/python3.6/site-packages/rasa/core/agent.py", line 248, in load_agent
model_server,
File "/build/lib/python3.6/site-packages/rasa/core/agent.py", line 64, in load_from_server
await _update_model_from_server(model_server, agent)
File "/build/lib/python3.6/site-packages/rasa/core/agent.py", line 122, in _update_model_from_server
model_server, agent.fingerprint
File "/build/lib/python3.6/site-packages/rasa/core/agent.py", line 180, in _pull_model_and_fingerprint
rasa.utils.io.unarchive(await resp.read(), model_directory)
File "/build/lib/python3.6/site-packages/rasa/utils/io.py", line 200, in unarchive
zip_ref = zipfile.ZipFile(IOReader(byte_array))
File "/usr/local/lib/python3.6/zipfile.py", line 1131, in __init__
self._RealGetContents()
File "/usr/local/lib/python3.6/zipfile.py", line 1198, in _RealGetContents
raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
Thanks.
My dockerfile looks like this:
FROM rasa/rasa
USER root
ENV BOT_ENV=development
COPY . /var/www
WORKDIR /var/www
RUN pip install --no-cache-dir phonenumbers pgeocode
#ENTRYPOINT ["rasa", "run", "-vv", "--enable-api", "actions"]
#ENTRYPOINT ["rasa", "run", "-vv", "--enable-api", "-m", "models", "--cors", "*", "--endpoints", "endpoints.yml", "--credentials", "credentials.yml", "-p", "8080"]
ENTRYPOINT ["rasa", "run", "-p", "8080", "--debug", "--enable-api", "--cors", "*"]
#ENTRYPOINT["rasa", "run", "-p", "8080"]
I have many options because I've been trying different ones as I don't manage to get my bot to take actions on the received messages. With the last one that is commented out, I get to see some HTTP 204 responses from twilio, which I think is good as at least I'm not receiving error responses anymore.
I'm still trying to figure out why my domain and policy are not being taken into account, as I'm getting these messages in the console:
2020-02-25 03:45:47 default[botgister-0] /build/lib/python3.6/site-packages/rasa/utils/common.py:351: UserWarning: No policy ensemble or domain set. Skipping action prediction and execution.
2020-02-25 03:45:47 default[botgister-0] More info at https://rasa.com/docs/rasa/core/policies/
Well, hope this helps.

no permanent file created

I'm running a conf file in logstash which gives the output as required but when i open ls to list file it's not getting created.need help in resolving issue.
conf file:
input {
elasticsearch{
hosts =>"localhost:9200"
index=>"index1"
}
}
output{
file{
path => "/opt/elk/logstash-6.5.0/example.json"
}
stdout{}
}
Result of execution:
[2019-10-13T14:07:47,503][INFO ][logstash.outputs.file ]
Opening file {:path=>"/opt/elk/logstash-6.5.0/example.json"}
{
"#timestamp" => 2019-10-13T08:37:46.715Z,
"example" => [
//data within example//
}
[2019-10-13T16:44:19,982][INFO ][logstash.pipeline] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x4f8848b run>"}
but when I run ls to see example.json, it doesn't show up
ls command:
elastic#es-VirtualBox:~/opt/elk/logstash-6.5.0$ ls
bin CONTRIBUTORS fetch.conf Gemfile.lock lib logs logstash-core-plugin-api NOTICE.TXT throw.conf vendor config data Gemfile grokexample.conf LICENSE.txt logstash-core modules output.json tools x-pack
so was wondering if conf only creates a temporary file?

No URLs matched- while copying file to google cloud storage

I am copying two files into Google Cloud Storage (GCS). When I run the script from informatica, I get error but when I run the same script from Unix it works fine.
Below is a link from GitHub where a similar issue is discussed. I don't understand what's the issue.
------------------------PART OF SCRIPT------------------------
echo "$LFS_File_Path/$File_Name";
gsutil cp "$LFS_File_Path/$File_Name" $GCS_Path;
if [[ $? -eq 0 ]]; then
echo "copy to GCS success for LFS Data File";
else
echo "copy to GCS Failed for LFS" >> $Log_File_Path/$Workflow_Name.txt ;
exit 1
fi
echo "$LFS_File_Path/$Del_File_Name";
gsutil cp "$LFS_File_Path/$Del_File_Name" $GCS_Path;
if [[ $? -eq 0 ]]; then
echo "copy to GCS success for LFS Delete Data File";
else
echo "copy to GCS Failed for LFS" >> $Log_File_Path/$Workflow_Name.txt ;
exit 1
fi
------------------------PART OF SCRIPT------------------------
error:-
CommandException: No URLs matched: /opt/u01/app/informatica/server/infa_shared/TgtFiles/BQ_RT/DW_ORDER_HEADER_DEL.csv
Similar topic:-
https://github.com/GoogleCloudPlatform/gsutil/issues/501
It might be related to the permissions in that file or maybe the command is being run as a user but only root has access to read inside that folder.
resolved it after creating another directory
its because of the hiddenfiles in that directory,the hidden file used the gstuil commands and found some escape characters init
try to create one more directory on the same filesystem and move the files from the directory
its mainly because of the escape characters,try to remove/move the hidden files
it will avoid the the above error
errors:
1)"No URLS matched"
2)CommandException: No URLs matched

Resources