Github action check if a file already exists - file

I have the following git action which allows me to download an image.
I have to make sure if the file already exists to skip the "Commit file" and the "Push changes"
How can I check if the file already exists if it already exists nothing is done.
on:
workflow_dispatch:
name: Scrape File
jobs:
build:
name: Build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
name: Check out current commit
- name: Url
run: |
URL=$(node ./action.js)
echo $URL
echo "URL=$URL" >> $GITHUB_ENV
- uses: suisei-cn/actions-download-file#v1
id: downloadfile
name: Download the file
with:
url: ${{ env.URL }}
target: assets/
- run: ls -l 'assets/'
- name: Commit files
run: |
git config --local user.email "41898282+github-actions[bot]#users.noreply.github.com"
git config --local user.name "github-actions[bot]"
git add .
git commit -m "Add changes" -a
- name: Push changes
uses: ad-m/github-push-action#master
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
branch: ${{ github.ref }}

There are a few options here - you can go directly with bash and do something like this:
if test -f "$FILE"; then
#file exists
fi
or use one of the existing actions like this:
- name: Check file existence
id: check_files
uses: andstor/file-existence-action#v1
with:
files: "assets/${{ env.URL }}"
- name: File exists
if: steps.check_files.outputs.files_exists == 'true'
run: echo "It exists !"

WARNING: Linux (and maybe MacOS) only solution ahead!
I was dealing with a very similar situation some time earlier and developed a method to not just check for added files, but also will be useful if you wanted to check for modified or deleted files or directories as well.
Warning:
This solution works only if the file is added/modified/deleted in git repository.
Introduction:
The command git status --short will return list of untracked, , deleted and modified files. For example:-
D deleted_foo
M modified_foo
?? untracked_dir_foo/
?? untracked_file_foo
A tracked_n_added_foo
Note that we run the same command as git status -s.
Understanding `git status -s` output:
When you read the output, you will see some lines in this form:
** filename
** dirname/
Note that here ** represent the first word of the line (ones like D, ?? etc.).
Here is a summary of all ** in the lines:
**
Meaning
D
File/dir has been deleted.
M
File/dir has been modified.
??
File/dir has been added but not tracked using git add [FILENAME].
A
File/dir has been added and also tracked using git add [FILENAME].
NOTE: Take care of the spaces! Using, for example, M instead of M in the following solution will not work as expected!
Solution:
Shell part of solution:
We can grep the output of git status -s to check whether a file/dir was added/modified/deleted.
The shell part of the solution goes like this:
if git status -s | grep -x "** [FILENAME]"; then
# Do whatever you wanna on match
else
# Do whatever you wanna on no-match
fi
Note: Get desired ** from the table above and replace [FILENAME] with filename.
For example, to check whether a file named foo was modified, use:
git status -s | grep -x " M foo"
Explanation: We use git status -s to get the output and pipe the output to grep. We also use command line option -x with grep so as to match whole line.
Workflow part of solution:
A very simple solution will go like this:
...
- name: Check for file
id: file_check
run: |
if git status -s | grep -x "** [FILENAME]"; then
echo "check_result=true" >> $GITHUB_OUTPUT
else
echo "check_result=false" >> $GITHUB_OUTPUT
fi
...
- name: Run dependent step
if: steps.file_check.outputs.check_result == 'true'
run: |
# Do whatever you wanna do on file found to be
# added/modified/deleted, based on what you set '**' to
...

Related

How to write bash function to print and run command when the command has arguments with spaces or things to be expanded

In Bash scripts, I frequently find this pattern useful, where I first print the command I'm about to execute, then I execute the command:
echo 'Running this cmd: ls -1 "$HOME/temp/some folder with spaces'
ls -1 "$HOME/temp/some folder with spaces"
echo 'Running this cmd: df -h'
df -h
# etc.
Notice the single quotes in the echo command to prevent variable expansion there! The idea is that I want to print the cmd I'm running, exactly as I will type and run the command, then run it!
How do I wrap this up into a function?
Wrapping the command up into a standard bash array, and then printing and calling it, like this, sort-of works:
# Print and run the passed-in command
# USAGE:
# cmd_array=(ls -a -l -F /)
# print_and_run_cmd cmd_array
# See:
# 1. My answer on how to pass regular "indexed" and associative arrays by reference:
# https://stackoverflow.com/a/71060036/4561887 and
# 1. My answer on how to pass associative arrays: https://stackoverflow.com/a/71060913/4561887
print_and_run_cmd() {
local -n array_reference="$1"
echo "Running cmd: ${cmd_array[#]}"
# run the command by calling all elements of the command array at once
${cmd_array[#]}
}
For simple commands like this it works fine:
Usage:
cmd_array=(ls -a -l -F /)
print_and_run_cmd cmd_array
Output:
Running cmd: ls -a -l -F /
(all output of that cmd is here)
But for more-complicated commands it is broken!:
Usage:
cmd_array=(ls -1 "$HOME/temp/some folder with spaces")
print_and_run_cmd cmd_array
Desired output:
Running cmd: ls -1 "$HOME/temp/some folder with spaces"
(all output of that command should be here)
Actual Output:
Running cmd: ls -1 /home/gabriel/temp/some folder with spaces
ls: cannot access '/home/gabriel/temp/some': No such file or directory
ls: cannot access 'folder': No such file or directory
ls: cannot access 'with': No such file or directory
ls: cannot access 'spaces': No such file or directory
The first problem, as you can see, is that $HOME got expanded in the Running cmd: line, when it shouldn't have, and the double quotes around that path argument were removed, and the 2nd problem is that the command doesn't actually run.
How do I fix these 2 problems?
References:
my bash demo program where I have this print_and_run_cmd function: https://github.com/ElectricRCAircraftGuy/eRCaGuy_hello_world/blob/master/bash/argument_parsing__3_advanced__gen_prog_template.sh
where I first documented how to pass bash arrays by reference, as I do in that function:
Passing arrays as parameters in bash
How to pass an associative array as argument to a function in Bash?
Follow-up question:
Bash: how to print and run a cmd array which has the pipe operator, |, in it
If you've got Bash version 4.4 or later, this function may do what you want:
function print_and_run_cmd
{
local PS4='Running cmd: '
local -
set -o xtrace
"$#"
}
For example, running
print_and_run_cmd echo 'Hello World!'
outputs
Running cmd: echo 'Hello World!'
Hello World!
local PS4='Running cmd: ' sets a prefix for commands printed by the shell when the xtrace option is on. The default is + . Localizing it means that the previous value of PS4 is automatically restored when the function returns.
local - causes any changes to shell options to be reverted automatically when the function returns. In particular, it causes the set -o xtrace on the next line to be automatically undone when the function returns. Support for local - was added in Bash 4.4.
From man bash, under the local [option] [name[=value] ... | - ] section (emphasis added):
If name is -, the set of shell options is made local to the function in which local is invoked: shell options changed using the set builtin inside the function are restored to their original values when the function returns.
set -o xtrace (which is equivalent to set -x) causes the shell to print commands, preceded by the expanded value of PS4, before running them.
See help set.
Check your scripts with shellcheck:
Line 2:
local -n array_reference="$1"
^-- SC2034 (warning): array_reference appears unused. Verify use (or export if used externally).
Line 3:
echo "Running cmd: ${cmd_array[#]}"
^-- SC2145 (error): Argument mixes string and array. Use * or separate argument.
^-- SC2154 (warning): cmd_array is referenced but not assigned.
Line 5:
${cmd_array[#]}
^-- SC2068 (error): Double quote array expansions to avoid re-splitting elements.
You might want to research https://github.com/koalaman/shellcheck/wiki/SC2068 . We fix all errors and we get:
print_and_run_cmd() {
local -n array_reference="$1"
echo "Running cmd: ${array_reference[*]}"
# run the command by calling all elements of the command array at once
"${array_reference[#]}"
}
For me it's odd to pass an array by reference in this case. I would pass the actual values. I often do:
prun() {
# in the style of set -x
# print to stderr, so output can be captured
echo "+ $*" >&2
# or echo "+ ${*#Q}" >&2
# or echo "+$(printf " %q" "$#")" >&2
# or echo "+$(/bin/printf " %q" "$#")" >&2
"$#"
}
prun "${cmd_array[#]}"
How do I fix these 2 problems?
Incorporate into your workflow linters, formatters and static analysis tools, like shellcheck, and check the problems they point out.
And quote variable expansion. It's "${array[#]}".
You can achieve what you want with DEBUG trap :
#!/bin/bash
set -T
trap 'test "$FUNCNAME" = print_and_run_cmd || trap_saved_command="${BASH_COMMAND}"' DEBUG
print_and_run_cmd(){
echo "Running this cmd: ${trap_saved_command#* }"
"$#"
}
outer(){
print_and_run_cmd ls -1 "$HOME/temp/some folder with spaces"
}
outer
# output ->
# Running this cmd: ls -1 "$HOME/temp/some folder with spaces"
# ...
I really like #pjh's answer, so I've marked it as correct. It doesn't fully answer my original question though, so if another answer comes along that does, I may have to change that. Anyway, see #pjh's answer or a full explanation of how the below code works, and what all those lines mean. I've helped edit that answer with some of the sources from man bash and help set.
I'd like to change the formatting and provide some more examples, however, to show that variable expansion does take place within the command. I'd also like to provide one version which passes by reference, and one which does not, so you can choose the call style which you like best.
Here are my examples, showing both call styles (print_and_run1 cmd_array and print_and_run2 "${cmd_array[#]}"):
#!/usr/bin/env bash
# Print and run the passed-in command, which is passed in as an
# array **by reference**.
# See here for a full explanation: https://stackoverflow.com/a/71151669/4561887
# USAGE:
# cmd_array=(ls -a -l -F /)
# print_and_run1 cmd_array
print_and_run1() {
local -n array_reference="$1"
local PS4='Running cmd: '
local -
set -o xtrace
# Call the cmd
"${array_reference[#]}"
}
# Print and run the passed-in command, which is passed in as members
# of an array **by value**.
# See here for a full explanation: https://stackoverflow.com/a/71151669/4561887
# USAGE:
# cmd_array=(ls -a -l -F /)
# print_and_run2 "${cmd_array[#]}"
print_and_run2() {
local PS4='Running cmd: '
local -
set -o xtrace
# Call the cmd
"$#"
}
cmd_array=(ls -1 "$HOME/temp/some folder with spaces")
print_and_run1 cmd_array
echo ""
print_and_run2 "${cmd_array[#]}"
echo ""
Sample run and output:
eRCaGuy_hello_world/bash$ ./print_and_run.sh
Running cmd: ls -1 '/home/gabriel/temp/some folder with spaces'
file1.txt
file2.txt
Running cmd: ls -1 '/home/gabriel/temp/some folder with spaces'
file1.txt
file2.txt
This seems to work too:
print_and_run_cmd() {
echo "Running cmd: $1"
eval "$cmd"
}
cmd='ls -1 "$HOME/temp/some folder with spaces"'
print_and_run_cmd "$cmd"
Output:
Running cmd: ls -1 "$HOME/temp/some folder with spaces"
(result of running the cmd is here)
But now the problem is, if I want to print an expanded version of the cmd too, to verify that part worked properly, I can't, or at least, don't know how.

Optimal usage of codecov in a monorepo context with separate flags for each package

I was just wondering what’s the best way to configure codecov for a monorepo setting. For example, let’s say I have packages A and B under my monorepo. The way I’m currently using codecov is by using a github action codecov/codecov-action#v1, by using multiple uses statement in my GitHub workflow YAML file like the following:-
- uses: codecov/codecov-action#v1
with:
files: ./packages/A/coverage/lcov.info
flags: flag_a
name: A
- uses: codecov/codecov-action#v1
with:
files: ./packages/B/coverage/lcov.info
flags: flag_b
name: B
I know it's possible to use a comma-separated value to upload multiple files, but I have to set a separate flag for each package, and doing it that way doesn't seem to work.
Thank you.
If anyone wants to know my solution, heres what I came up with.
I ended up replacing the github action with my own bash script.
final code
#!/usr/bin/env bash
codecov_file="${GITHUB_WORKSPACE}/scripts/codecov.sh"
curl -s https://codecov.io/bash > $codecov_file
chmod +x $codecov_file
cd "${GITHUB_WORKSPACE}/packages";
for dir in */
do
package="${dir/\//}"
if [ -d "$package/coverage" ]
then
file="$PWD/$package/coverage/lcov.info"
flag="${package/-/_}"
$codecov_file -f $file -F $flag -v -t $CODECOV_TOKEN
fi
done
this is what the above bash script does
Downloading the bash uploader script from codecov
Moving to the packages directory where are the packages are located, and going through all the 1st level directories
Change the package name by removing extra slash
If the directory contains coverage directory only then enter into it, since only those packages have been tested.
Create a file and flag variable (removing hypen with underscore as codecov doesn't support hypen in flag name)
Executed the downloaded codecov script by passing the file and flag variable as argument

How to merge duplicate entries that produced by for loop

Following my previous question which got closed— basically I have a script that check availability of packages on target server, the target server and the packages have been stored to an array.
declare -a prog=("gdebi" "firefox" "chromium-browser" "thunar")
declare -a snap=("beer2" "beer3")
# checkvar=$(
for f in "${prog[#]}"; do
for connect in "${snap[#]}"; do
ssh lg#"$connect" /bin/bash <<- EOF
if dpkg --get-selections | grep -qE "(^|\s)"$f"(\$|\s)"; then
status="[INSTALLED] [$connect]"
else
status=""
fi
printf '%s %s\n' "$f" "\$status"
EOF
done
done
With the help of fellow member here, I've made several fix to original script, script ran pretty well— except there's one problem, the output contain duplicate entries.
gdebi [INSTALLED] [beer2]
gdebi
firefox [INSTALLED] [beer2]
firefox [INSTALLED] [beer3]
chromium-browser [INSTALLED] [beer2]
chromium-browser [INSTALLED] [beer3]
thunar
thunar
I know it this is normal behavior, as for pass multiple server from snap array, making ssh travel to all the two server.
Considering that the script checks same package for two server, I want the output to be merged.
If beer2 have firefox packages, but beer3 doesn't.
firefox [INSTALLED] [beer2]
If beer3 have firefox packages, but beer2 doesn't.
firefox [INSTALLED] [beer3]
If both beer2 and beer3 have the packages.
firefox [INSTALLED] [beer2, beer3]
or
firefox [INSTALLED] [beer2] [beer3]
If both beer2 and beer3 doesn't have the package, it will return without extra parameter.
firefox
Sound like an easy task, but for the love of god I can't find how to achieve this, here's list of things I have tried.
Try to manipulate the for loops.
Try putting return value after one successful loops (exit code).
Try nested if.
All of the above doesn't seem to work, I haven't tried changing/manipulate the return string as I'm not really experienced with some text processing such as: awk, sed, tr and many others.
Can anyone shows how It's done ? Would really mean the world to me.
Pure Bash 4+ solution using associative array to store hosts the program is installed on:
#!/usr/bin/env bash
declare -A hosts_with_package=(["gdebi"]="" ["firefox"]="" ["chromium-browser"]="" ["thunar"]="")
declare -a hosts=("beer2" "beer3")
# Collect installed status
# Iterate all hosts
for host in "${hosts[#]}"; do
# Read the output of dpkg --get-selections with searched packages
while IFS=$' \t' read -r package status; do
# Test weather package is installed on host
if [ "$status" = "install" ]; then
# If no host listed for package, create first entry
if [ -z "${hosts_with_package[$package]}" ]; then
# Record the first host having the package installed
hosts_with_package["$package"]="$host"
else
# Additional hosts are concatenated as CSV
hosts_with_package["$package"]="${hosts_with_package[$package]}, $host"
fi
fi
# Feed the whole loop with the output of the dpkg --get-selections for packages names
# Packages names are the index of the hosts_with_package array
done < <(ssh "lg#$host" dpkg --get-selections "${!hosts_with_package[#]}")
done
# Output results
# Iterate the package name keys
for package in "${!hosts_with_package[#]}"; do
# Print package name without newline
printf '%s' "$package"
# If package is installed on some hosts
if [ -n "${hosts_with_package[$package]}" ]; then
# Continue the line with installed hosts
printf ' [INSTALLED] [%s]' "${hosts_with_package[$package]}"
fi
# End with a newline
echo
done
Instead of making several ssh connections in nested loops consider this change
prog=( mysql-server apache2 php ufw )
snap=( localhost )
for connect in ${snap[#]}; do
ssh $connect "
progs=( ${prog[#]} )
for prog in \${progs[#]}; do
dpkg -l | grep -q \$prog && echo \"\$prog [INSTALLED]\" || echo \"\$prog\"
done
"
done
Based on #Ivan answer
#!/bin/bash
prog=( "gdebi" "firefox" "chromium-browser" "thunar" )
snap=( "beer2" "beer3" )
# First, retrieve the list on installed program for each host
for connect in ${snap[#]}; do
ssh lg#"$connect" /bin/bash >/tmp/installed.${connect} <<- EOF
progs=( "${prog[#]}" )
for prog in \${progs[#]}; do
dpkg --get-selections | awk -v pkg=\$prog '\$1 == pkg && \$NF ~ /install/ {print \$1}'
done
EOF
done
# Filter the previous results to format the output as you need
awk '{
f = FILENAME;
gsub(/.*\./,"",f);
a[$1] = a[$1] "," f
}
END {
for (i in a)
print i ":[" substr(a[i],2) "]"
}' /tmp/installed.*
rm /tmp/installed.*
Example of output :
# With prog=( bash cat sed tail something firefox-esr )
firefox-esr:[localhost]
bash:[localhost,localhost2]
sed:[localhost,localhost2]

Strange behaviour of sejda-console in batch for loop

I try to write a wrapper script for songbook generation using lilypond, latex and sejda-console (for the pdf part). Everything works so far, but I have a problem with sejda that is giving me nuts.
Here is the relevant part of my code:
for %%i in (%f%) do (
sejda-console.bat extractbybookmarks -f ".\%%~ni.pdf" -o "export\%%~ni\%title%.pdf" -l 2 -p [BOOKMARK_NAME] -e "%title%" --overwrite
)
where f is a ";"-separated list of files. This command works for the first file, but fails for all others. I can't find any difference between the commands that sejda receives. Here is my console output:
make_sheet.bat -t "Live it up" --supress *.lytex
Configuring Sejda 3.2.30
Starting execution with arguments: 'extractbybookmarks -f .\book_drums.pdf -o export\book_drums\Live it up.pdf -l 2 -p [BOOKMARK_NAME] -e Live it up --overwrite'
Java version: '1.8.0_151'
Validating parameters.
Starting task (org.sejda.impl.sambox.ExtractByOutlineTask#28701274) execution.
Opening C:\Users\skr1_\Desktop\Tools\Songbook\Sample\out\.\book_drums.pdf
Retrieving outline information for level 2 and match regex Live it up
Starting extraction by outline, level 2 and match regex Live it up
Found 0 inherited images and 0 inherited fonts potentially unused
Starting extracting Live it up pages 9 9
Created output temporary buffer C:\Users\skr1_\Desktop\Tools\Songbook\Sample\out\export\book_drums\.sejdaTmp2789047920522272436.tmp
Appended relevant outline items
Filtering annotations
Skipped acroform merge, nothing to merge
Ending extracting Live it up
Task progress: 0% done
Moving C:\Users\skr1_\Desktop\Tools\Songbook\Sample\out\export\book_drums\.sejdaTmp2789047920522272436.tmp to C:\Users\skr1_\Desktop\Tools\Songbook\Sample\out\export\book_drums\Live it up.pdf.
Extraction completed and outputs written to org.sejda.model.output.FileOrDirectoryTaskOutput#478190fc[C:\Users\skr1_\Desktop\Tools\Songbook\Sample\out\export\book_drums\Live it up.pdf]
Task (org.sejda.impl.sambox.ExtractByOutlineTask#28701274) executed in 0 seconds
Completed execution
C:\Users\skr1_\Desktop\Tools\Songbook\Sample>(sejda-console.bat extractbybookmarks -f ".\book_general.pdf" -o "export\book_general\Live it up.pdf" -l 2 -p [BOOKMARK_NAME] -e "Live it up" --overwrite )
Configuring Sejda 3.2.30
Starting execution with arguments: 'extractbybookmarks -f .\book_general.pdf -o export\book_general\Live it up.pdf -l 2 -p [BOOKMARK_NAME] -e Live it up --overwrite'
Java version: '1.8.0_151'
Invalid value (File '.\book_general.pdf' does not exist): --files -f value... : pdf files to operate on. A list of existing pdf files (EX. -f /tmp/file1.pdf or -f /tmp/password_protected_file2.pdf:secret123) (required)
Invalid value (File '.\book_general.pdf' does not exist): --files -f value... : pdf files to operate on. A list of existing pdf files (EX. -f /tmp/file1.pdf or -f /tmp/password_protected_file2.pdf:secret123) (required)
C:\Users\skr1_\Desktop\Tools\Songbook\Sample>(sejda-console.bat extractbybookmarks -f ".\book_guitar.pdf" -o "export\book_guitar\Live it up.pdf" -l 2 -p [BOOKMARK_NAME] -e "Live it up" --overwrite )
Configuring Sejda 3.2.30
Starting execution with arguments: 'extractbybookmarks -f .\book_guitar.pdf -o export\book_guitar\Live it up.pdf -l 2 -p [BOOKMARK_NAME] -e Live it up --overwrite'
Java version: '1.8.0_151'
Invalid value (File '.\book_guitar.pdf' does not exist): --files -f value... : pdf files to operate on. A list of existing pdf files (EX. -f /tmp/file1.pdf or -f /tmp/password_protected_file2.pdf:secret123) (required)
Invalid value (File '.\book_guitar.pdf' does not exist): --files -f value... : pdf files to operate on. A list of existing pdf files (EX. -f /tmp/file1.pdf or -f /tmp/password_protected_file2.pdf:secret123) (required)
Even worse, if I copy the commands that sejda receives and paste them as arguments for a new command, everything works fine.
I suspect that something is happening with the working directory in between, but I don't get it.
Also, note that the output includes the command for subsequent passes of the for-loop (starting with "(sejda-console.bat ...") though echo is off. It is not included for the first run, however.
I'm not an expert with programming, especially not with batch, and any help would be very appreciated.
I'd suggest that sejda.bat is changing the current directory.
Try
pushd
call sejda.bat ...
popd

Mass rename objects on Google Cloud Storage

Is it possible to mass rename objects on Google Cloud Storage using gsutil (or some other tool)? I am trying to figure out a way to rename a bunch of images from *.JPG to *.jpg.
Here is a native way to do this in bash with an explanation below, line by line of the code:
gsutil ls gs://bucket_name/*.JPG > src-rename-list.txt
sed 's/\.JPG/\.jpg/g' src-rename-list.txt > dest-rename-list.txt
paste -d ' ' src-rename-list.txt dest-rename-list.txt | sed -e 's/^/gsutil\ mv\ /' | while read line; do bash -c "$line"; done
rm src-rename-list.txt; rm dest-rename-list.txt
The solution pushes 2 lists, one for the source and one for the destination file (to be used in the "gsutil mv" command):
gsutil ls gs://bucket_name/*.JPG > src-rename-list.txt
sed 's/\.JPG/\.jpg/g' src-rename-list.txt > dest-rename-list.txt
The line "gsutil mv " and the two files are concatenated line by line using the below code:
paste -d ' ' src-rename-list.txt dest-rename-list.txt | sed -e 's/^/gsutil\ mv\ /'
This then runs each line in a while loop:
while read line; do bash -c "$line"; done
Lastly, clean up and delete the files created:
rm src-rename-list.txt; rm dest-rename-list.txt
The above has been tested against a working Google Storage bucket.
https://cloud.google.com/storage/docs/gsutil/addlhelp/WildcardNames
gsutil supports URI wildcards
EDIT
gsutil 3.0 release note
As part of the bucket sub-directory support we changed the * wildcard to match only up to directory boundaries, and introduced the new ** wildcard...
Do you have directories under bucket? if so, maybe you need to go down to each directories or use **.
gsutil -m mv gs://my_bucket/**.JPG gs://my_bucket/**.jpg
or
gsutil -m mv gs://my_bucket/mydir/*.JPG gs://my_bucket/mydir/*.jpg
EDIT
gsutil doesn't support wildcard for destination so far (as of 4/12/'14)
nether API.
so at this moment you need to retrieve list of all JPG files,
and rename each files.
python example:
import subprocess
files = subprocess.check_output("gsutil ls gs://my_bucket/*.JPG",shell=True)
files = files.split("\n")[:-1]
for f in files:
subprocess.call("gsutil mv %s %s"%(f,f[:-3]+"jpg"),shell=True)
please note that this would take hours.
gsutil does not support parallelized and mass-copy/rename.
You have two options:
use a dataflow process to do the operation
or
use GNU parallel to launch it using several processes
If you use GNU Parallel, it is better to deploy a new instance to do the mass copy/rename operation:
First: - Make a list of files you want to copy/rename (a file with source and destination separated by a space or tab), like this:
gs://origin_bucket/path/file gs://dest_bucket/new_path/new_filename
Second: Launch a new compute instance
Third: Login in that instance and install Gnu parallel
sudo apt install parallel
Third: authorize yourself with google (gcloud auth login) because the service account for compute might not have permissions to move/rename the files.
gcloud auth login
Make the copy (gsutil cp) or move (gsutil mv) operation with parallel:
parallel -j 20 --colsep ' ' gsutil mv {1} {2} :::: file_with_source_destination_uris.txt
This will make 20 parallel runs of the gsutil cp operation.
Yes, it is possible:
Move/rename objects and/or subdirectories

Resources