I can successfully run a gsutil command with a windows domain account from the command line in Windows (setting up service account key etc.). When I try to run the same command from a SQL Agent Job using a CmdExec task the job hangs and doesn't complete. I can't see any logging so have no clue what it's waiting for. I've setup the job to run with the same Proxy User that i use to run the gsutil command manually.
Any ideas how I can get this to work or how to see more logging?
Are you using standalone gsutil? Or did you get it as part of installing the Cloud SDK (gcloud)?
If the job hangs for a long time, it could be stuck retrying multiple times. To test if this is the case, you can set the num_retries option to be very small, but above 0 (e.g. 1) either in your .boto file or the the command arguments via this option:
gsutil -o 'Boto:num_retries=1' <rest of command here...>
A second thing to note (at least for the version of gsutil that doesn't come with gcloud) is that gsutil looks for your boto config file (which specifies the credentials it should use) in your home directory by default. If you're running gsutil as a different user (maybe your SQL Agent Job runs as its own dedicated user?), it will look for a .boto file in that user's home directory. The same should apply for the gcloud version -- gcloud uses credentials based on the user executing it. You can avoid this by copying your .boto file to somewhere that the job has permission to read from, along with setting the BOTO_CONFIG environment variable to that path before running gsutil. From the cmd shell, this would look something like:
set BOTO_CONFIG=C:\some\path\.boto && gsutil <rest of command here...>
Note: If you're not sure which boto config file you're normally using, you can find out by running gsutil version -l and looking at the line that displays your config path(s).
Related
so I have this
.bat
file:
#echo off
cd C:\Users\user\Downloads
gcloud auth activate-service-account --key-file=keyFileName.json
gcloud auth print-access-token
pause
During the first
gcloud
command, it will suddenly crash the command prompt halfway, but when I copy and paste each line manually into command prompt in the same location as the location I am trying to
cd
to in the
.bat
file, it works... Any idea why? I am on Windows 10 by the way.
Searching on Google, I found two related issues, on Github and Stackoverflow.
Github's solution was using python which was not what I needed and Stackoverflow's one did not have anyone helping him/her...
Thanks
When running some gcloud commands, they have the expectation that they may be interacting with a user. In addition, they can adversely interact with the current command line processor (CMD on Windows). This can result in the command line processor ending prematurely. One solution is to run your scripted gcloud commands in their own local / nested instance of a command line processor.
If today, your script contains:
gcloud ... some command parameters ...
replace this with:
cmd /c gcloud ... some command parameters ...
This will cause the gcloud command to run in its own nested environment which won't interfere with the top level (scripted) environment.
You can also just add CALL before each command, for example:
CALL gcloud app deploy
CALL gcloud app describe
Trying to run gcloud init to initialize the Google App Engine Engine SDK by typing ./google-cloud-sdk/bin/gcloud init but it showed: no such file or directory or command not found. Is something wrong with my PATH? My path is:
/Users/AnneLutz/Documents/google-cloud-sdk\
If you typing ./google-cloud-sdk/bin/gcloud init and you installed Cloud SDK in /Users/AnneLutz/Documents/google-cloud-sdk, then your current directory should be /Users/AnneLutz/Documents in order for what you type to work.
That said you should add /Users/AnneLutz/Documents/google-cloud-sdk/bin to you path. To do this, assuming you are using bash you can
source /Users/AnneLutz/Documents/google-cloud-sdk/path.bash.inc
To make it so that every-time you start your shell you can add it to shell profile. For example you can add above source command at the end of ~/.bash_profile file.
It looks like you used the option to download the SDK zip file and are then trying to configure your environment with that download option. If you aren't comfortable with setting environment variables, you might want to instead try installing using the "interactive" installer, which will automate the steps for making the commands always available on your system.
The directions are here, but for Mac OS users are basically:
Enter the following at a command prompt:
curl https://sdk.cloud.google.com | bash
Restart your shell:
exec -l $SHELL
Run gcloud init to initialize the gcloud environment:
gcloud init
For many, this procedure is easier than getting everything configured manually.
I am trying to create a scheduled SQL Agent Job that executes a .bat file with a gsutil rsync command to synchronize a local folder with a Google Cloud Bucket. The task runs without an error, but does not finish or actually perform the rsync command.
I can however manually run the .bat file with success as the root user, as I have a key handshake with the GCP service account. I can also successfully execute other non-gsutil .bat files from the SQL Agent Job, as I have created credentials and proxy to run as the root user
The .bat file looks like this :
gsutil rsync -r G:\MyBackupFolder gs://my-coldline-storage-bucket
and I am running the SQL Agent step as type Operating system (CmdExec)
HI a newbie to server management... We need to automate hourly script to run drush to re-index solr, run cron and clear cache on multiple servers. I'm sure there has to be .bat file or something?
At first: You have no '.bat' files in any unix system (but ofcourse you can write scripts named something.bat, but nothing special happens ;-)).
You need to have drush installed somewhere in your sytem. I use to install it in /usr/local/share/drush and put a link from /usr/local/share/drush/drushto /usr/local/bin/drush. Then run crontab -e to edit the schedule. An editor is launched inside your console window. If it shows an empty window or the file contains only lines starting with "#", then put
MAILTO=your.mail#example.com
PATH=/usr/local/bin:/bin:/usr/bin:$HOME/bin
#daily drush #live -q -y cron > /dev/null
In this case drush cron is executed daily for a drupal installation with a site-alias #live. The output is sent to /dev/null, so that I do not get any error messages.
PS: Get familiar with the cron systems and the crontab command as well as shell scripting. They are unix' standard tools and are needed for those kind of tasks.
PS2: You also want to know the concept of drush site-aliases. Run drush topic docs-aliases to learn more about it.
I have a windows 8 jenkins slave (really more than one) and I want to have a task which clones a mirror of a git repository if it doesn't exist and always fetches latest.
I've got this 'execute windows batch' step:
if not exist "server-reference" (
git clone --mirror git#ssh.com:something/somewhere.git server-reference
)
pushd server-reference
git fetch --all
popd
exit /B %ERRORLEVEL%
The console output for the job gets to this point
c:\jenkins\workspace\thing\server-reference>git fetch --all
Fetching origin
and never finishes
If I switch an exit above the fetch then I get the expected exit.
And if I run the batch file commands as the jenkins user on the windows box everything works as expected. So I (think I) know that the git fetch --all runs on the slave without prompting for input
How do I amend this script so that it completes when Jenkins runs it?
A number of things can cause this problem. Here are things that have worked for me in the past.
I know you stated this above, but just to double-check: Confirm that when you run git fetch --all manually on the slave machine, it doesn't prompt you for a password. If it does, Jenkins would get stuck on that step, so you'll need to set up an SSH keypair.
Make sure that you are calling C:\Program Files (x86)\Git\cmd\git.exe and not C:\Program Files (x86)\Git\bin\git.exe in Jenkins. The second one can cause this problem in certain versions of git.
On the slave machine, set the environment variable HOME to C:\users\[username], filling in your user's username. (For older OS's it was C:\Documents and Settings\[username].) Then recycle the Jenkins service and try again.
Double check that in Administrative Tools > Services, the Jenkins slave service "Log On As" is set to your user and the password is correct. If not, set those values, recycle the Jenkins service and try again.
If none of the above worked, try uninstalling and reinstalling git. And if you aren't using it, try installing the newest version of git.