Teamcity not loading environment variables - Fortify automation - batch-file

I have Teamcity currently setup to run a batch file, this batch file executes a fortify scan. It seems the environment variable 'PATH' had loaded correctly on one attempt and started to perform the scan. On the next build attempt the batch file couldn't locate one of the executable(sourceanalyzer.exe). When looking into the user defined parameters it seems different environment variables were loaded than the previous build attempt. It looks like the system environment variables load on the successful attempt and then a users environment variable loaded on the failed attempt. Is there a way to permanently set the environment variable PATH in the agent to load only the system environment variables?
UPDATE:
I have tried several things, to include passing in a Fortify environment variable, which does allow TeamCity to start running the scan. However, it looks like I hit another snag with Fortify's plugin for vs2015. The new error states it cannot find the plugin. I assume this is because pathways are hard coded? Seeing as TeamCity doesn't use the System environment variables and I have to pass them in for TeamCity to find these directories. Is there an easier way to use the batch file to load the system environment variables to avoid hard coding pathways? Would setlocal in the batch file help load these system environment variables, so I can just call sourceanalyzer with out creating environment variables or hard coding pathways?

IIRC, Teamcity will ask you if you want to install the build agent on the System account or the user account. By default it selects the System account and as long as you aren't running any GUI apps, you won't notice the difference... until something like this happens. If Fortify is GUI based, then reinstall your build agent on the user account and ignore the following. Otherwise...
When you set your PATH variables using the System(Control Panel) advanced settings, there are two panes, one for user and one for system. Here you can inspect the System variables to make sure they are correct.
What I will generally do is create a new key, say FORTIFY_PATHand prepend %FORTIFY_PATH% to the System PATH variable. THEN RESTART YOUR MACHINE. The path won't get updated correctly until you do.
Next, login to the system account using PSExec: https://superuser.com/a/596395 and try to run your tools from that command prompt to verify that they are working in the build agent's environment. I once had trouble getting an SVN script to upload until I logged into the System account and provided my SVN password. Some settings are stored in %APPDATA% which is different from the user account.
If you can't get Fortify to run from the System command prompt, then you should probably reinstall your build agent to your user account. Or install Fortify to the System account (if possible).
When configuring TeamCity build agents, check the agent system and environment variables By going to Agents->Agent->Agent Parameters, or the /agentDetails.html?id=1&tab=agentParameters&kind=envpath on your server.
After changing the parameters, restart the agent or restart the agent's machine.

Related

Batch Scripting .. what means: <name>.exe //IS//%SERVICE_NAME

So i have a Batch Script which i got trough a task at work, there is an error inside the script which i lokated at the following position of the code
REM Install service
echo register Service...
%INSTALL_PATH%<name_of_exe>.exe //IS//%SERVICE_NAME% #error here
echo successfully installed %SERVICE_NAME%.
So as squashman told after starting of an .exe there follow parameters. So to this point it is clear,
//IS//%SERVICE_NAME% #is parameter
So i want to know if //IS// is some kind of "batch-magic". So it is not some kind of crazy Windows Path since there are "/" not "\". So my it is some kind of character-escaping?
So i found it out.
There is a Programm called Procrun from ApacheCommons with the following description
Procrun is a set of applications that allow Windows users to wrap (mostly) Java applications (e.g. Tomcat) as a Windows service.
The service can be set to automatically start when the machine boots and will continue to run with no user logged onto the machine.
So now the Problem line was easy to understand since the .exe belongs to procrun.
<name>.exe //IS//%SERVICE_NAME% # //IS - install service
# //%SERVICE_NAME% - the name

Why is my Scheduled Task updating its 'Last Run Time' correctly, and giving a 'Last Run Result' of '(0x0)', but still not actually working?

I have a batch file which is set to run as a Scheduled Task on Windows Server 2012.
When I run the batch file by hand from the command line, it works. When I right-click on my task in Task Scheduler and manually run it, it still works fine.
But, if I let the task run according to the schedule set for it... then it seems to work sometimes, but not others.
I have already set the task to run as a given user; I have set its 'Start in' directory correctly; I have tried giving it highest privileges. None of that helps.
The basic answer is that the batch file for the task is running, and that the last step of the task is returning 0x0. If the task is apparently 'not doing anything', it is because some earlier step of the task is failing silently
Why? In my case, and I think this could easily effect other people, the answer is that the batch file for a scheduled task sees different environment variables depending on whether the user it runs as is currently logged in or not.
More Detail:
In particular, if the task is set to run as Administrator, then while Administrator is logged in the task sees one set of variables (whether it is run manually or on the schedule), but when Administrator is not logged in, it sees a different set of variables.
This can be very hard to debug - basically, you need to put in a lot of logging!
When you are running a batch file as a Scheduled Task on Windows Server 2012, it only sees shared environment variables. It does not see the user specific environment variables for the user you have set it to run as, unless the user in question is currently logged in.
You can see the problem in action by putting SET > test.txt into a batch file on its own, and running it as a task in different circumstances (manually; on a schedule when logged in; on a schedule when not logged in).
UPDATE:
From more detailed testing, it seems that when the user which the task is set to run as is not logged in, the variables USERDNSDOMAIN,
USERDOMAIN and USERNAME do get set correctly for that user anyway. The variable USERPROFILE gets set incorrectly to the value for the Windows default user (i.e. C:\Users\Default). Everything else gets set incorrectly to the set of shared environment variables only (note that this is obviously not the correct set for the specified user, and is also not even the correct set for the Windows default user, which should get its user specific environment variables from HKEY_USERS\.Default\Environment).
Note:
This is not the same issue as windows 7 task scheduler doesn't use updated path , and in fact changes to any shared environment variables, including PATH, do get seen straight away (on the tests I did, on Windows 2012 R2), with no restart of any process.

Batch File not starting automatically as a service

i am having trouble while starting a batch file as a service. the batch file runs fine when started manually but it doesnt starts a service and no ouput is observed. i have used nssm service manager to start the service.
below are the commands which i have used :
D:\nssm-2.24\win32>nssm install call
D:\nssm-2.24\win32>nssm start call
while installing i have provided the path of batch file.
the batch file contains the windows script to start few programs automatically.
you cannot just install any old executable as a service, and certainly not a batch file. a service is a program with a specific API which makes it react to service manager calls. ignore that, i just read up on nssm. still, there are probably better ways.
your use case sound rather like you might want to put your batch file in the autostart startmenu folder, to run it at login/startup.
or a scheduled task, if you want to restart it regularly.
one thing to consider, too, is the user under which the script is executed.

Jenkins: Unable to execute batch file on a virtual machine

I created a new free-style software project and under "Configure", I added a "Execute Windows batch command" under the section "Build".
My command in the text-box is as follows:
call \\ukvc-miu29-12\c$\Testing1\testing.bat
My console output when I try to run my build is as follows:
Started by user anonymous
Building in workspace C:\Program Files (x86)\Jenkins\jobs\Run comparison script\workspace
[workspace] $ cmd /c call C:\Windows\TEMP\hudson1649447975658279345.bat
C:\Program Files (x86)\Jenkins\jobs\Run comparison script\workspace>call \\ukvc-miu29-12\c$\Testing1\testing.bat
Access is denied.
C:\Program Files (x86)\Jenkins\jobs\Run comparison script\workspace>exit 1
Build step 'Execute Windows batch command' marked build as failure
Finished: FAILURE
Have spent close to two hours looking for a solution online, but I am unable to find one.
New to Jenkins, so any suggestion/s is/are appreciated, thanks!
I guess Jenkins runs as another user than the one you're logged in with. So that user probably isn't allowed to access that network share (or doesn't have the login credentials cached).
The easiest way of making sure it will work, would probably be to add the same user with the same password that Jenkins runs under to \\ukvc-miu29-12.
(As for the call, I'm not sure you really need it and I have a vague suspicion that you end up failing the build every time if you use call here. The temporary batch Jenkins executes there looks like it relies on the fact that control won't return if the batch file was executed successfully. But that's just a guess on my part.)

Work on a remote project with Eclipse via SSH

I have the following boxes:
a) A Windows box with Eclipse CDT,
b) A Linux box, accessible for me only via SSH.
Both the compiler and the hardware required to build and run my project is only on machine B.
I'd like to work "transparently" from a Windows box on that project using Eclipse CDT and be able to build, run and debug the project remotely from within the IDE.
How do I set up that:
The building will work? Any simpler solutions than writing a local makefile which would rsync the project and then call a remote makefile to initiate the actual build? Does Eclipse managed build have a feature for that?
The debugging will work?
Preferably - the Eclipse CDT code indexing will work? Do I have to copy all required header files from machine B to machine A and add them to include path manually?
Try the Remote System Explorer (RSE). It's a set of plug-ins to do exactly what you want.
RSE may already be included in your current Eclipse installation. To check in Eclipse Indigo go to Window > Open Perspective > Other... and choose Remote System Explorer from the Open Perspective dialog to open the RSE perspective.
To create an SSH remote project from the RSE perspective in Eclipse:
Define a new connection and choose SSH Only from the Select Remote System Type screen in the New Connection dialog.
Enter the connection information then choose Finish.
Connect to the new host. (Assumes SSH keys are already setup.)
Once connected, drill down into the host's Sftp Files, choose a folder and select Create Remote Project from the item's context menu. (Wait as the remote project is created.)
If done correctly, there should now be a new remote project accessible from the Project Explorer and other perspectives within eclipse. With the SSH connection set-up correctly passwords can be made an optional part of the normal SSH authentication process. A remote project with Eclipse via SSH is now created.
The very simplest way would be to run Eclipse CDT on the Linux Box and use either X11-Forwarding or remote desktop software such as VNC.
This, of course, is only possible when you Eclipse is present on the Linux box and your network connection to the box is sufficiently fast.
The advantage is that, due to everything being local, you won't have synchronization issues, and you don't get any awkward cross-platform issues.
If you have no eclipse on the box, you could thinking of sharing your linux working directory via SMB (or SSHFS) and access it from your windows machine, but that would require quite some setup.
Both would be better than having two copies, especially when it's cross-platform.
I'm in the same spot myself (or was), FWIW I ended up checking out to a samba share on the Linux host and editing that share locally on the Windows machine with notepad++, then I compiled on the Linux box via PuTTY. (We weren't allowed to update the ten y/o versions of the editors on the Linux host and it didn't have Java, so I gave up on X11 forwarding)
Now... I run modern Linux in a VM on my Windows host, add all the tools I want (e.g. CDT) to the VM and then I checkout and build in a chroot jail that closely resembles the RTE.
It's a clunky solution but I thought I'd throw it in to the mix.
My solution is similar to the SAMBA one except using sshfs. Mount my remote server with sshfs, open my makefile project on the remote machine. Go from there.
It seems I can run a GUI frontend to mercurial this way as well.
Building my remote code is as simple as: ssh address remote_make_command
I am looking for a decent way to debug though. Possibly via gdbserver?
I tried ssh -X but it was unbearably slow.
I also tried RSE, but it didn't even support building the project with a Makefile (I'm being told that this has changed since I posted my answer, but I haven't tried that out)
I read that NX is faster than X11 forwarding, but I couldn't get it to work.
Finally, I found out that my server supports X2Go (the link has install instructions if yours does not). Now I only had to:
download and unpack Eclipse on the server,
install X2Go on my local machine (sudo apt-get install x2goclient on Ubuntu),
configure the connection (host, auto-login with ssh key, choose to run Eclipse).
Everything is just as if I was working on a local machine, including building, debugging, and code indexing. And there are no noticeable lags.
I had the same problem 2 years ago and I solved it in the following way:
1) I build my projects with makefiles, not managed by eclipse
2) I use a SAMBA connection to edit the files inside Eclipse
3) Building the project:
Eclipse calles a "local" make with a makefile which opens a SSH connection
to the Linux Host. On the SSH command line you can give parameters which
are executed on the Linux host. I use for that parameter a makeit.sh shell script
which call the "real" make on the linux host.
The different targets for building you can give also by parameters from
the local makefile --> makeit.sh --> makefile on linux host.
The way I solved that one was:
For windows:
Export the 'workspace' directory from the Linux machine using samba.
Mount it locally in windows.
Run Eclipse, using the mounted 'workspace' directory as the eclipse workspace.
Import the project you want and work on it.
For Linux:
Mount the 'workspace' directory using sshfs
Run Eclipse.
Run Eclipse, using the mounted 'workspace' directory as the eclipse workspace.
Import the project you want and work on it.
In both cases you can either build and run through Eclipse, or build on the remote machine via ssh.
For this case you can use ptp eclipse https://eclipse.org/ptp/ for source browsing and building.
You can use this pluging to debug your application
http://marketplace.eclipse.org/content/direct-remote-c-debugging
How to edit in Eclipse locally, but use a git-based script I wrote (sync_git_repo_from_pc1_to_pc2.sh) to synchronize and build remotely
The script I wrote to do this is sync_git_repo_from_pc1_to_pc2.sh.
Readme: README_git-sync_repo_from_pc1_to_pc2.md
Update: see also this alternative/competitor: GitSync:
How to use Sublime over SSH
https://github.com/jachin/GitSync
This answer currently only applies to using two Linux computers [or maybe works on Mac too?--untested on Mac] (syncing from one to the other) because I wrote this synchronization script in bash. It is simply a wrapper around git, however, so feel free to take it and convert it into a cross-platform Python solution or something if you wish
This doesn't directly answer the OP's question, but it is so close I guarantee it will answer many other peoples' question who land on this page (mine included, actually, as I came here first before writing my own solution), so I'm posting it here anyway.
I want to:
develop code using a powerful IDE like Eclipse on a light-weight Linux computer, then
build that code via ssh on a different, more powerful Linux computer (from the command-line, NOT from inside Eclipse)
Let's call the first computer where I write the code "PC1" (Personal Computer 1), and the 2nd computer where I build the code "PC2". I need a tool to easily synchronize from PC1 to PC2. I tried rsync, but it was insanely slow for large repos and took tons of bandwidth and data.
So, how do I do it? What workflow should I use? If you have this question too, here's the workflow that I decided upon. I wrote a bash script to automate the process by using git to automatically push changes from PC1 to PC2 via a remote repository, such as github. So far it works very well and I'm very pleased with it. It is far far far faster than rsync, more trustworthy in my opinion because each PC maintains a functional git repo, and uses far less bandwidth to do the whole sync, so it's easily doable over a cell phone hot spot without using tons of your data.
Setup:
Install the script on PC1 (this solution assumes ~/bin is in your $PATH):
git clone https://github.com/ElectricRCAircraftGuy/eRCaGuy_dotfiles.git
cd eRCaGuy_dotfiles/useful_scripts
mkdir -p ~/bin
ln -s "${PWD}/sync_git_repo_from_pc1_to_pc2.sh" ~/bin/sync_git_repo_from_pc1_to_pc2
cd ..
cp -i .sync_git_repo ~/.sync_git_repo
Now edit the "~/.sync_git_repo" file you just copied above, and update its parameters to fit your case. Here are the parameters it contains:
# The git repo root directory on PC2 where you are syncing your files TO; this dir must *already exist*
# and you must have *already `git clone`d* a copy of your git repo into it!
# - Do NOT use variables such as `$HOME`. Be explicit instead. This is because the variable expansion will
# happen on the local machine when what we need is the variable expansion from the remote machine. Being
# explicit instead just avoids this problem.
PC2_GIT_REPO_TARGET_DIR="/home/gabriel/dev/eRCaGuy_dotfiles" # explicitly type this out; don't use variables
PC2_SSH_USERNAME="my_username" # explicitly type this out; don't use variables
PC2_SSH_HOST="my_hostname" # explicitly type this out; don't use variables
Git clone your repo you want to sync on both PC1 and PC2.
Ensure your ssh keys are all set up to be able to push and pull to the remote repo from both PC1 and PC2. Here's some helpful links:
https://help.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh
https://help.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent
Ensure your ssh keys are all set up to ssh from PC1 to PC2.
Now cd into any directory within the git repo on PC1, and run:
sync_git_repo_from_pc1_to_pc2
That's it! About 30 seconds later everything will be magically synced from PC1 to PC2, and it will be printing output the whole time to tell you what it's doing and where it's doing it on your disk and on which computer. It's safe too, because it doesn't overwrite or delete anything that is uncommitted. It backs it up first instead! Read more below for how that works.
Here's the process this script uses (ie: what it's actually doing)
From PC1: It checks to see if any uncommitted changes are on PC1. If so, it commits them to a temporary commit on the current branch. It then force pushes them to a remote SYNC branch. Then it uncommits its temporary commit it just did on the local branch, then it puts the local git repo back to exactly how it was by staging any files that were previously staged at the time you called the script. Next, it rsyncs a copy of the script over to PC2, and does an ssh call to tell PC2 to run the script with a special option to just do PC2 stuff.
Here's what PC2 does: it cds into the repo, and checks to see if any local uncommitted changes exist. If so, it creates a new backup branch forked off of the current branch (sample name: my_branch_SYNC_BAK_20200220-0028hrs-15sec <-- notice that's YYYYMMDD-HHMMhrs--SSsec), and commits any uncommitted changes to that branch with a commit message such as DO BACKUP OF ALL UNCOMMITTED CHANGES ON PC2 (TARGET PC/BUILD MACHINE). Now, it checks out the SYNC branch, pulling it from the remote repository if it is not already on the local machine. Then, it fetches the latest changes on the remote repository, and does a hard reset to force the local SYNC repository to match the remote SYNC repository. You might call this a "hard pull". It is safe, however, because we already backed up any uncommitted changes we had locally on PC2, so nothing is lost!
That's it! You now have produced a perfect copy from PC1 to PC2 without even having to ensure clean working directories, as the script handled all of the automatic committing and stuff for you! It is fast and works very well on huge repositories. Now you have an easy mechanism to use any IDE of your choice on one machine while building or testing on another machine, easily, over a wifi hot spot from your cell phone if needed, even if the repository is dozens of gigabytes and you are time and resource-constrained.
Resources:
The whole project: https://github.com/ElectricRCAircraftGuy/eRCaGuy_dotfiles
See tons more links and references in the source code itself within this project.
How to do a "hard pull", as I call it: How do I force "git pull" to overwrite local files?
Related:
git repository sync between computers, when moving around?

Resources