I am using python3.8 for a project, and I want to install all dependencies from pipfile. I got:
[pipenv.exceptions.InstallError]: ModuleNotFoundError: No module named 'zlib'
I tried:
sudo apt install zlib1g
sudo apt install zlib1g-dev
pipenv install zlib
python3.8 -m pip install zlib
But still get the same error. Though I have zlib on python3.6 and python3.9:
> > > $ python3
> > > Python 3.9.6 (default, Jul 25 2021, 17:47:28)
>
> > > import zlib
> > >
> > > exit()
> > > $ python3.6
> > > Python 3.6.9 (default, Jan 26 2021, 15:33:00)
>
> > > import zlib
> > >
> > > exit()
> > > $ python3.8
> > > Python 3.8.3 (default, Jun 22 2020, 19:41:04)
>
> > > import zlib
> > > Traceback (most recent call last):
> > > File "\<stdin\>", line 1, in \<module\>
> > > ModuleNotFoundError: No module named 'zlib'
How can I install zlib on python3.8 ?
Related
This is based on the answer in JMeter & Gitlab: How to Install & SQLServer JDBC Driver?.
In short, the debug output below shows that the appropriate JAR and DLL are in the Java-path on the Docker image, yet I am seeing the following error in the JMeter output:
"DBCP DataSource configured without a 'password'" mssql
"DBCP DataSource configured without a 'user'" mssql
The image uses JMeter 5.4.2 ("latest").
The JDBC driver files I am using are:
mssql-jdbc_auth-9.4.1.x86.dll
mssql-jdbc-9.4.1.jre8.jar
So is it a compatibility issue?
I have JMeter 5.4.1 installed on my computer, and the test plan works fine using the local application.
Here is the output from the Gitlab job:
[0KRunning with gitlab-runner 13.5.0 (ece86343)
[0;m[0K on qaildk03 VFwBJtXZ
[0;msection_start:1641760700:prepare_executor
[0K[0K[36;1mPreparing the "docker" executor[0;m
[0;m[0KUsing Docker executor with image guythedocker/jmeter_mssql:latest ...
[0;m[0KPulling docker image guythedocker/jmeter_mssql:latest ...
[0;m[0KUsing docker image sha256:7ebf2689ca4b15d4053c8148055524bfaf8173b76d2d66f45d0a13ce6992b901 for guythedocker/jmeter_mssql:latest with digest guythedocker/jmeter_mssql#sha256:316e3ce3f37c174147b10ce541e66121a4d5017f597f862b64cb081fae4035b8 ...
[0;msection_end:1641760705:prepare_executor
[0Ksection_start:1641760705:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on runner-vfwbjtxz-project-516-concurrent-0 via qaildk03...
section_end:1641760707:prepare_script
[0Ksection_start:1641760707:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Reinitialized existing Git repository in /builds/guyl/api-performance-test/.git/
[32;1mChecking out 89b301b5 as develop...[0;m
Removing public/
Removing testresults/
[32;1mSkipping Git submodules setup[0;m
section_end:1641760709:get_sources
[0Ksection_start:1641760709:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m
[0;m[32;1m$ ls /opt/apache-jmeter-*/lib/[0;m
etc
mssql-jdbc-9.4.1.jre8.jar <------------------
mssql-jdbc_auth-9.4.1.x86.dll <------------------
etc
[32;1m$ env[0;m
CI_PROJECT_NAMESPACE=guyl
GITLAB_USER_ID=128
CI_RUNNER_VERSION=13.5.0
FF_SKIP_NOOP_BUILD_STAGES=true
LYNX_OP_USERS=50
CI_SERVER_NAME=GitLab
CI_RUNNER_DESCRIPTION=qaildk03
GITLAB_USER_EMAIL=guyl#xxxx.com
CI_SERVER_REVISION=4469fca16f1
CI_RUNNER_EXECUTABLE_ARCH=linux/amd64
CI_REGISTRY_USER=gitlab-ci-token
CI_API_V4_URL=https://gitlab.xxxx.net/api/v4
CI_REGISTRY_PASSWORD=[MASKED]
CI_RUNNER_SHORT_TOKEN=VFwBJtXZ
CI_JOB_NAME=api test
HOSTNAME=runner-vfwbjtxz-project-516-concurrent-0
GITLAB_USER_LOGIN=guyl
OCTO_KEY=[MASKED]
CI_PROJECT_NAME=api-performance-test
CI_PIPELINE_SOURCE=schedule
CI_BUILD_REF_NAME=develop
CI_JOB_STATUS=running
MONYX_LOOPS=100
FF_SHELL_EXECUTOR_USE_LEGACY_PROCESS_KILL=false
CI_PIPELINE_ID=74075
CI_BUILD_REF_SLUG=develop
CI_COMMIT_REF_SLUG=develop
CI_SERVER=yes
CI_COMMIT_SHORT_SHA=89b301b5
FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION=false
CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX=gitlab.xxxx.net:443/guyl/dependency_proxy/containers
PWD=/builds/guyl/api-performance-test
CI_RUNNER_TAGS=
CI_PROJECT_PATH=guyl/api-performance-test
RAMPUP=
CI_SERVER_TLS_CA_FILE=/builds/guyl/api-performance-test.tmp/CI_SERVER_TLS_CA_FILE
MOMA2_LOOPS=0
TZ=Europe/Amsterdam
FF_RESET_HELPER_IMAGE_ENTRYPOINT=true
CI_COMMIT_REF_PROTECTED=true
CI_SERVER_VERSION_MINOR=8
CI_COMMIT_SHA=89b301b5c6370b45bbd7770e1d827fa334a5300d
HOME=/root
FF_NETWORK_PER_BUILD=false
CI_DEPENDENCY_PROXY_PASSWORD=[MASKED]
CI_PROJECT_VISIBILITY=internal
CI_CONCURRENT_PROJECT_ID=0
JMETER_HOME=/opt/apache-jmeter-5.4.2 <------------------------
CI_COMMIT_MESSAGE=Update .gitlab-ci.yml file
USERS=
CI_BUILD_NAME=api test
FF_USE_DIRECT_DOWNLOAD=true
CI_PAGES_DOMAIN=gitlab-pages.xxxx.net
CI_SERVER_VERSION=13.8.6
CI_SERVER_PORT=443
CURRENT_VERSION=10.2
MOMA2_USERS=2
JMETER_DOWNLOAD_URL=https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-5.4.2.tgz
CI_PAGES_URL=https://guyl.gitlab-pages.xxxx.net/api-performance-test
CI_PIPELINE_IID=238
CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]#gitlab.xxxx.net/guyl/api-performance-test.git
CI_SERVER_URL=https://gitlab.xxxx.net
GITLAB_FEATURES=
ENGINE_LOOPS=100
CI_COMMIT_DESCRIPTION=
MOMA2_RAMPUP=6
CI_JOB_STAGE=test
CI_PIPELINE_URL=https://gitlab.xxxx.net/guyl/api-performance-test/-/pipelines/74075
ENGINE_RAMPUP=9
CI_DEFAULT_BRANCH=develop
ENGINE_USERS=0
CI_BUILD_REF=89b301b5c6370b45bbd7770e1d827fa334a5300d
CI_SERVER_VERSION_PATCH=6
CI_COMMIT_TITLE=Update .gitlab-ci.yml file
CI_PROJECT_ROOT_NAMESPACE=guyl
MONYX_USERS=100
GITLAB_USER_NAME=Guy Levin
CI_PROJECT_DIR=/builds/guyl/api-performance-test
LYNX_OP_LOOPS=0
SHLVL=2
CI_RUNNER_ID=24
CI_COMMIT_TIMESTAMP=2022-01-09T20:36:44+00:00
CI_DISPOSABLE_ENVIRONMENT=true
CI_JOB_JWT=[MASKED]
CI_BUILD_ID=169407
CI_SERVER_PROTOCOL=https
LYNX_OP_RAMPUP=25
JMETER_BIN=/opt/apache-jmeter-5.4.2/bin <---------------------------------
CI_COMMIT_REF_NAME=develop
CI_SERVER_HOST=gitlab.xxxx.net
CI_JOB_URL=https://gitlab.xxxx.net/guyl/api-performance-test/-/jobs/169407
CI_JOB_TOKEN=[MASKED]
CI_CONCURRENT_ID=0
CI_BUILD_TOKEN=[MASKED]
CI_COMMIT_BRANCH=develop
FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=true
CI_RUNNER_REVISION=ece86343
CI_BUILD_BEFORE_SHA=0000000000000000000000000000000000000000
CI_DEPENDENCY_PROXY_USER=gitlab-ci-token
MONYX_RAMPUP=0
CI_PROJECT_PATH_SLUG=guyl-api-performance-test
CI_PROJECT_CONFIG_PATH=.gitlab-ci.yml
CI_NODE_TOTAL=1
CI_BUILDS_DIR=/builds
CI_JOB_ID=169407
CI_PROJECT_REPOSITORY_LANGUAGES=javascript,batchfile
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/apache-jmeter-5.4.2/bin <-------------
CI_PROJECT_ID=516
CI=true
GITLAB_CI=true
CI_JOB_IMAGE=guythedocker/jmeter_mssql:latest
CI_COMMIT_BEFORE_SHA=0000000000000000000000000000000000000000
CI_PROJECT_TITLE=api-performance-test
CI_SERVER_VERSION_MAJOR=13
CI_CONFIG_PATH=.gitlab-ci.yml
CI_BUILD_STAGE=test
CI_DEPENDENCY_PROXY_SERVER=gitlab.xxxx.net:443
CI_PROJECT_URL=https://gitlab.xxxx.net/guyl/api-performance-test
OLDPWD=/opt/apache-jmeter-5.4.2
_=/usr/bin/env
[32;1m$ echo $PATH[0;m
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/apache-jmeter-5.4.2/bin <-----------
[32;1m$ /entrypoint.sh --version[0;m
START Running Jmeter on Sun Jan 9 21:38:30 CET 2022
JVM_ARGS=-Xmn502m -Xms2008m -Xmx2008m
jmeter args=--version
jmeter ALL ARGS=-Dlog4j2.formatMsgNoLookups=true --version
Jan 09, 2022 9:38:32 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
_ ____ _ ____ _ _ _____ _ __ __ _____ _____ _____ ____
/ \ | _ \ / \ / ___| | | | ____| | | \/ | ____|_ _| ____| _ \
/ _ \ | |_) / _ \| | | |_| | _| _ | | |\/| | _| | | | _| | |_) |
/ ___ \| __/ ___ \ |___| _ | |___ | |_| | | | | |___ | | | |___| _ <
/_/ \_\_| /_/ \_\____|_| |_|_____| \___/|_| |_|_____| |_| |_____|_| \_\ 5.4.2
Copyright (c) 1999-2021 The Apache Software Foundation
END Running Jmeter on Sun Jan 9 21:38:33 CET 2022
[32;1m$ mkdir testresults[0;m
[32;1m$ mkdir testresults/monyx[0;m
[32;1m$ mkdir testresults/moma2[0;m
[32;1m$ mkdir testresults/lynx_operational[0;m
[32;1m$ mkdir testresults/engine[0;m
[32;1m$ /entrypoint.sh -n -t ./JMeter/Monyx_API/Monyx_API.jmx -l ./monyx.log -e -o ./testresults/monyx -Jthreads=$MONYX_USERS -Jrampup=$MONYX_RAMPUP -JtestCases=$MONYX_LOOPS -Jhost=qa2-payapi.xxxx.com -f # collapsed multi-line command[0;m
START Running Jmeter on Sun Jan 9 21:38:33 CET 2022
JVM_ARGS=-Xmn502m -Xms2008m -Xmx2008m
jmeter args=-n -t ./JMeter/Monyx_API/Monyx_API.jmx -l ./monyx.log -e -o ./testresults/monyx -Jthreads=100 -Jrampup=0 -JtestCases=100 -Jhost=qa2-payapi.xxxx.com -f
jmeter ALL ARGS=-Dlog4j2.formatMsgNoLookups=true -n -t ./JMeter/Monyx_API/Monyx_API.jmx -l ./monyx.log -e -o ./testresults/monyx -Jthreads=100 -Jrampup=0 -JtestCases=100 -Jhost=qa2-payapi.xxxx.com -f
Creating summariser <summary>
Created the tree successfully using ./JMeter/Monyx_API/Monyx_API.jmx
Starting standalone test # Sun Jan 09 21:38:35 CET 2022 (1641760715608)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
summary + 13617 in 00:01:23 = 163.8/s Avg: 577 Min: 5 Max: 49132 Err: 10000 (73.44%) Active: 100 Started: 300 Finished: 200
summary + 5932 in 00:00:30 = 197.2/s Avg: 503 Min: 256 Max: 1501 Err: 0 (0.00%) Active: 98 Started: 300 Finished: 202
summary = 19549 in 00:01:53 = 172.7/s Avg: 555 Min: 5 Max: 49132 Err: 10000 (51.15%)
summary + 489 in 00:00:31 = 15.7/s Avg: 2068 Min: 251 Max: 28079 Err: 0 (0.00%) Active: 100 Started: 400 Finished: 300
summary = 20038 in 00:02:24 = 138.7/s Avg: 591 Min: 5 Max: 49132 Err: 10000 (49.91%)
summary + 93 in 00:00:30 = 3.1/s Avg: 30500 Min: 19297 Max: 47158 Err: 0 (0.00%) Active: 100 Started: 400 Finished: 300
summary = 20131 in 00:02:54 = 115.7/s Avg: 730 Min: 5 Max: 49132 Err: 10000 (49.67%)
summary + 96 in 00:00:34 = 2.8/s Avg: 34369 Min: 22688 Max: 45334 Err: 0 (0.00%) Active: 100 Started: 400 Finished: 300
summary = 20227 in 00:03:28 = 97.3/s Avg: 889 Min: 5 Max: 49132 Err: 10000 (49.44%)
summary + 79 in 00:00:25 = 3.1/s Avg: 35523 Min: 23972 Max: 45774 Err: 0 (0.00%) Active: 100 Started: 400 Finished: 300
summary = 20306 in 00:03:53 = 87.1/s Avg: 1024 Min: 5 Max: 49132 Err: 10000 (49.25%)
summary + 81 in 00:00:33 = 2.4/s Avg: 36941 Min: 24322 Max: 46364 Err: 0 (0.00%) Active: 100 Started: 400 Finished: 300
etc
summary = 63674 in 01:35:23 = 11.1/s Avg: 8779 Min: 0 Max: 114698 Err: 18594 (29.20%)
summary + 613 in 00:00:30 = 20.4/s Avg: 4952 Min: 0 Max: 15629 Err: 204 (33.28%) Active: 100 Started: 600 Finished: 500
summary = 64287 in 01:35:53 = 11.2/s Avg: 8742 Min: 0 Max: 114698 Err: 18798 (29.24%)
DBCP DataSource configured without a 'username'
DBCP DataSource configured without a 'password'
etc
DBCP DataSource configured without a 'password'
DBCP DataSource configured without a 'username'
DBCP DataSource configured without a 'password'
DBCP DataSource configured withosummary + 377 in 00:00:30 = 12.6/s Avg: 7965 Min: 810 Max: 15971 Err: 188 (49.87%) Active: 100 Started: 1700 Finished: 1600
summary = 257193 in 02:40:23 = 26.7/s Avg: 3659 Min: 0 Max: 114698 Err: 61102 (23.76%)
summary + 388 in 00:00:30 = 12.9/s Avg: 7759 Min: 786 Max: 15878 Err: 191 (49.23%) Active: 100 Started: 1700 Finished: 1600
etc
summary + 3850 in 00:00:30 = 128.3/s Avg: 715 Min: 24 Max: 1653 Err: 3850 (100.00%) Active: 100 Started: 2700 Finished: 2600
summary = 383734 in 03:09:53 = 33.7/s Avg: 2835 Min: 0 Max: 114698 Err: 164923 (42.98%)
summary + 3864 in 00:00:30 = 128.7/s Avg: 765 Min: 44 Max: 3479 Err: 3864 (100.00%) Active: 100 Started: 2700 Finished: 2600
summary = 387598 in 03:10:23 = 33.9/s Avg: 2814 Min: 0 Max: 114698 Err: 168787 (43.55%)
summary + 2405 in 00:00:19 = 128.8/s Avg: 707 Min: 24 Max: 1436 Err: 2405 (100.00%) Active: 0 Started: 1 Finished: 1
summary = 390003 in 03:10:42 = 34.1/s Avg: 2801 Min: 0 Max: 114698 Err: 171192 (43.90%)
Tidying up ... # Mon Jan 10 00:49:18 CET 2022 (1641772158705)
... end of run
END Running Jmeter on Mon Jan 10 00:49:44 CET 2022
And here is the Dockerfile:
# inspired by https://github.com/hauptmedia/docker-jmeter and
# https://github.com/hhcordero/docker-jmeter-server/blob/master/Dockerfile
FROM alpine:3.12
ARG JMETER_VERSION="5.4.2"
ENV JMETER_HOME /opt/apache-jmeter-${JMETER_VERSION}
ENV JMETER_BIN ${JMETER_HOME}/bin
ENV JMETER_DOWNLOAD_URL https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-${JMETER_VERSION}.tgz
# Install extra packages
# Set TimeZone, See: https://github.com/gliderlabs/docker-alpine/issues/136#issuecomment-612751142
ARG TZ="Europe/Amsterdam"
ENV TZ ${TZ}
RUN apk update \
&& apk upgrade \
&& apk add ca-certificates \
&& update-ca-certificates \
&& apk add --update openjdk8-jre tzdata curl unzip bash \
&& apk add --no-cache nss \
&& rm -rf /var/cache/apk/* \
&& mkdir -p /tmp/dependencies \
&& curl -L --silent ${JMETER_DOWNLOAD_URL} > /tmp/dependencies/apache-jmeter-${JMETER_VERSION}.tgz \
&& mkdir -p /opt
RUN tar -xzf /tmp/dependencies/apache-jmeter-${JMETER_VERSION}.tgz -C /opt
RUN mkdir -p ${JMETER_HOME}/lib
RUN curl --ssl-no-revoke -L https://repo1.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/9.4.1.jre8/mssql-jdbc-9.4.1.jre8.jar -o mssql-jdbc-9.4.1.jre8.jar
RUN curl --ssl-no-revoke -L https://repo1.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc_auth/9.4.1.x86/mssql-jdbc_auth-9.4.1.x86.dll -o mssql-jdbc_auth-9.4.1.x86.dll
RUN rm -rf /tmp/dependencies
COPY ./mssql-jdbc-9.4.1.jre8.jar /opt/apache-jmeter-${JMETER_VERSION}/lib/
COPY ./mssql-jdbc_auth-9.4.1.x86.dll /opt/apache-jmeter-${JMETER_VERSION}/lib/
# TODO: plugins (later)
# && unzip -oq "/tmp/dependencies/JMeterPlugins-*.zip" -d $JMETER_HOME
# Set global PATH such that "jmeter" command is found
ENV PATH $PATH:$JMETER_BIN
# Entrypoint has same signature as "jmeter" command
COPY entrypoint.sh /
WORKDIR ${JMETER_HOME}
ENTRYPOINT ["/entrypoint.sh"]
JMeter Report, showing errors:
I don't think you can use SQL Server integrated authentication from Linux machine.
In order to be able to use the approach with the Microsoft JDBC Driver and integrated authentication you need to run your builds on a Windows Runner
In case you have to use Linu you can work it around using JTDS JDBC Driver and connection string like:
jdbc:jtds:sqlserver://server_name:1433;databaseName=your-database;domain=your-domain;useNTLMv2=true;
and specify your username and password in the appropriate sections of the JDBC Connection Configuration.
Also be aware that according to 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure article you should always be using the latest version of JMeter so consider upgrading to JMeter 5.4.3 (or whatever is the latest version which is available at JMeter Downloads page) on next available opportunity.
I can file size in MBs but when I am trying to open the file I can't ... saying can't find file.
while read line; do echo $line # or whaterver you want to do with the $line variable
cat $line | grep "PROCEDURE" > result3.txt
chmod 777 result3.txt
done < xreflist.txt;
To be able to find the file size, you need permission to read the directory.
To be able to open the file, you need permission to read the file.
It is perfectly possible to be able to do the one without the other.
$ mkdir junk
$ cd junk
$ echo "Hello World" > no-permission
$ chmod 0 no-permission
$ ls -la
total 8
drwxr-xr-x 3 jonathanleffler staff 96 Dec 29 11:34 .
drwxr-xr-x 18 jonathanleffler staff 576 Dec 29 11:34 ..
---------- 1 jonathanleffler staff 12 Dec 29 11:34 no-permission
$ cat ./no-permission
cat: ./no-permission: Permission denied
$ rm -f ./no-permission
$ cd ..
$ rmdir junk
$
We have services that generate files named with modules names. For ex., acqDou module generates
acqDou_0001.out
acqDou_08981.out
acqDou_23423.out
The acq_cheat module generates files with almost same name, but last numbering is a bit diffrent.
My Requirment is to tar these files on the basis of module names, one tar file per module, such as acqDou.tar and acq_cheat.tar
Below is the programm I attmepted:
cd /home/swap/output/outfiles
for i in *;
do
j=`echo $i | grep -v 'out$'`
if [ -z $j ];
then continue;
fi
count1=$(echo $i | tr -d -c '_' | wc -m)
if [ $count1 -eq 2 ]
then
two=`ls -1 $i | cut -d'_' -f2 | cut -d'.' -f1 | uniq`
else
two=`ls -1 $i | cut -d'_' -f1 | cut -d'.' -f1 | uniq`
fi
_FILE="${_FILE} $two"
done
_FILE2=`echo "${_FILE[#]}" | tr ' ' '\n' | sort -u | tr '\n' ' '`
echo "${_FILE2}"
for m in "${_FILE2}"
do
ls -lrt *${m}*.x
tar -cpf $m.tar /home/swap/output/outfiles/*${m}*
done
}
In case you can distinguish the files by their name, the solution can be so simple straightforward:
tar cf acqDou.tar acqDou*.out*
tar cf acq_cheat.tar acq_cheat*.out*
If there are several such modules, you can even apply a simple for loop to process them:
for m in acqDou acq_cheat
do
tar cf $m.tar ${m}*.out*
done
Make tars for all possible module types present in the directory - in one shot.
lets say you have modules named as.. acqDou, acqDumb, acqcheat, acqfunny - In module_XXXX.out format.
something which may looks like this, as shown below. So i got 4 modules and 500 files of each of them in the directory. They all are empty, but it doesn't matter.
>Wed Oct 05|01:15:13|gaurav#[STATION]:/root/ga/scripts/temp/tmp % ls -lrtha *.out|head -7 ; echo ; ls -lrtha |tail -8
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqDou_0.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqDumb_0.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqcheat_0.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqfunny_0.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqDou_1.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqDumb_1.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqcheat_1.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqDumb_498.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqcheat_498.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqfunny_498.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqDou_499.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqDumb_499.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqcheat_499.out
-rw-rw-r--. 1 gaurav gaurav 0 Oct 5 00:58 acqfunny_499.out
drwxrwxr-x. 2 gaurav gaurav 64K Oct 5 01:14 .
>Wed Oct 05|01:15:30|gaurav#[STATION]:/root/ga/scripts/temp/tmp %
In order to achieve the objective:
we can list the files, strip off the numbers (using sed), sort and unique the list (which gives the possible present module names in the directory) and then feed it to a while loop, to read module names one by one .. and then generate our output tarfile, module name wise and having files of that module only.
Here is the command:
ls *.out |sed 's/[0-9]//g'|sort|uniq|sed 's/_.out//g'|while read module
do
tar -cvf ${module}.tar ${module}*
done
We get something like this, once we finish.
>Wed Oct 05|01:16:43|gaurav#[STATION]:/root/ga/scripts/temp/tmp % ls -lrth *.tar
-rw-rw-r--. 1 gaurav gaurav 260K Oct 5 01:16 acqcheat.tar
-rw-rw-r--. 1 gaurav gaurav 260K Oct 5 01:16 acqDou.tar
-rw-rw-r--. 1 gaurav gaurav 260K Oct 5 01:16 acqDumb.tar
-rw-rw-r--. 1 gaurav gaurav 260K Oct 5 01:16 acqfunny.tar
>Wed Oct 05|01:16:48|gaurav#[STATION]:/root/ga/scripts/temp/tmp %
4 tar files have been created, with their module names. And it contains its respective module files. Let us see by using tar -tvf command. I will run it on all tar files and just take 2 lines as output, per file.
>Wed Oct 05|01:18:29|gaurav#[STATION]:/root/ga/scripts/temp/tmp % ls *.tar|while read file
do
echo ;echo "Looking inside file: $file"
tar -tvf $file|head -2
done
Looking inside file: acqcheat.tar
-rw-rw-r-- gaurav/gaurav 0 2016-10-05 00:58 acqcheat_0.out
-rw-rw-r-- gaurav/gaurav 0 2016-10-05 00:58 acqcheat_100.out
Looking inside file: acqDou.tar
-rw-rw-r-- gaurav/gaurav 0 2016-10-05 00:58 acqDou_0.out
-rw-rw-r-- gaurav/gaurav 0 2016-10-05 00:58 acqDou_100.out
Looking inside file: acqDumb.tar
-rw-rw-r-- gaurav/gaurav 0 2016-10-05 00:58 acqDumb_0.out
-rw-rw-r-- gaurav/gaurav 0 2016-10-05 00:58 acqDumb_100.out
Looking inside file: acqfunny.tar
-rw-rw-r-- gaurav/gaurav 0 2016-10-05 00:58 acqfunny_0.out
-rw-rw-r-- gaurav/gaurav 0 2016-10-05 00:58 acqfunny_100.out
>Wed Oct 05|01:18:42|gaurav#[STATION]:/root/ga/scripts/temp/tmp %
We do really have 500 files in each tar file. Let us just confirm that too.
>Wed Oct 05|01:18:42|gaurav#[STATION]:/root/ga/scripts/temp/tmp % ls *.tar|while read file
do
echo ; echo "File count in tar file: $file"
tar -tvf $file|wc -l
done
File count in tar file: acqcheat.tar
500
File count in tar file: acqDou.tar
500
File count in tar file: acqDumb.tar
500
File count in tar file: acqfunny.tar
500
>Wed Oct 05|01:20:03|gaurav#[STATION]:/root/ga/scripts/temp/tmp %
Cheers,
Gaurav
If I run:
$ ls -l /tmp
total 16
drwxr-xr-x 2 noaccess noaccess 177 Nov 18 09:53 hsperfdata_noaccess
drwxr-xr-x 2 root root 117 Nov 18 09:53 hsperfdata_root
I get the expected result running:
]$ ls -l /tmp | tr -s '[:space:]'
total 16
drwxr-xr-x 2 noaccess noaccess 177 Nov 18 09:53 hsperfdata_noaccess
drwxr-xr-x 2 root root 117 Nov 18 09:53 hsperfdata_root
Yet tr will not impact file output:
$ file /tmp/dummy
/tmp/dummy: empty file
$ file /tmp/dummy | tr -s '[:space:]'
/tmp/dummy: empty file
(the same if I use [:blank:])
I was expecting:
$ file /tmp/dummy | tr -s '[:space:]'
/tmp/dummy: empty file
Am I misusing file? tr? Must I awk? I'm using bash 3.2.51 on SunOS 5.10 and would like to tell XML and gz files apart.
The Solaris 5.10 version of file is producing output that uses tabs instead of spaces:
$ touch /tmp/dummy
$ file /tmp/dummy | cat -vet
/tmp/dummy:^Iempty file$
$
In this case, tr -s squashes a single tab character down to a single tab character...
I have a dir structure like
$ ls /comp/drive/
2009 2010 2011 2012 2013 2014
$ ls 2009
01 02 03 04 05 06 07 09 10 11 12
$ ls 2013
01 02 04 05 06 08 09 10 12
$ ls 2013/04/*.nc
file4.nc file44.nc file45.nc file49.nc
There are dirs like years and each year there are few months dirs and inside are .nc files.
What I want to do is get the array of filenames provided start and end years/months.
e.g. sYear=2011; eYear=2013; sMonth=03; eMonth=08
So, I want to get the array of all filenames from year 2011/03 to 2013/08 only without going inside the dirs.
Any bash trick?
sYear=2011; eYear=2013; sMonth=03; eMonth=08
# prevent bugs from interpreting numbers as hex
sMonth=$(( 10#$sMonth ))
eMonth=$(( 10#$eMonth ))
files=( )
for (( curYear=sYear; curYear <= eYear; curYear++ )); do
# include only months after sMonth
for monthDir in "$curYear"/*/; do
[[ -e $monthDir ]] || continue # ignore years that don't exist
curMonth=${monthDir##*/}
(( curMonth )) || continue # ignore non-numeric directory names
(( curYear == sYear )) && (( 10#$curMonth < sMonth )) && continue
(( curYear == eYear )) && (( 10#$curMonth > eMonth )) && continue
files+=( "$monthDir"/*.nc )
done
done
printf '%q\n' "${files[#]}"
Try this:
sYear=2011
sMonth=03
eYear=2013
eMonth=08
shopt -s nullglob
declare -a files
for year in *; do
(( ${year} < ${sYear} || ${year} > ${eYear} )) && continue
for year_month in ${year}/*; do
month=${year_month##*/}
(( ${year} == ${sYear} && ${month##0} < ${sMonth##0} )) && continue;
(( ${year} == ${eYear} && ${month##0} > ${eMonth##0} )) && continue;
files+=(${year_month}/*.nc)
done
done
echo "${files[#]}"
# printf "$(pwd)/%q\n" "${files[#]}" # for full path