I would like to extend our shake-based build system to include running integration tests, however running those tests require firing up some docker containers on which the tests depend and which may or may not be rebuilt as part of the build process.
In short I would need to track "live" processes, and kill/restart them if their underlying image has changed. How could I achieve that with shake?
I'm going to assume we want to run docker image and change when image changes (generalising it for any name isn't hard, but distracts from the important bits).
Imagine the service is always running, never falls over, and is entirely in the control of Shake. We could write:
"runner" %> \out -> do
need ["image"]
cmd_ "docker kill"
cmd_ "docker start image"
writeFile' out ""
Here we have a rule that produces runner, and as a side effect depends on image and starts/stops docker. Any test using the runner should do need ["runner"] before they start. If they also depend on the contents of the image, they should do need ["runner","image"].
If the images aren't under the control of Shake, and other people are prodding them, you probably want to addOracle to define an oracle that detects the currently running docker images, and depend on that in runner.
Related
Following https://docs.hiro.so/smart-contracts/devnet I can't get the command clarinet integrate to work. I have installed Docker on my mac and am running version 0.28.0 of clarinet. Running command within 'my-react-app/clarinet' where all clarity related files live (contracts, settings, tests, and Clarinet.toml).
My guess is it could be an issue with Docker?
The issue was that I downloaded my Devnet.toml file from a repo that was configured incorrectly. The configuration I needed was:
[network]
name = "devnet"
I increased the CPU and Memory in Docker as well.
There is an issue when the command attempts to spin up the stacks explorer, but I was informed that there are several existing issues with the stacks explorer from clarinet integrate at the moment.
Depending on how the last devnet was terminated, you could have some containers running. This issue should be fixed in the next incoming release, meanwhile, you'd need to terminate this stale containers manually.
Apart from Ludo's suggestions, I'd also look into your Docker resources. The default CPU/memory allocation should allow you to get started with Clarinet, but just in case, you could alter it to see if that makes a difference. Here's my settings for your reference:
Alternatively, to tease things out, you could reuse one of the samples (eg: hirosystems/stacks-billboard) instead of running your project. See if the sample comes up as expected; if it does, there could be something missing in your project.
My Goal
I've been using devcontainers in combination with WSL2 for a little while now. But I keep running into issues and besides that I like off-loading resources of my laptop to a server. Moving the containers to a native Linux server would solve my issues.
My ideal situation would be to have a solution that works just like working locally on my Windows laptop (later probably moving to Macbook) but using the facilities of a Linux server (which has systemd and netns) and moving the workload there as well so my laptop doesn't sound like a vacuum cleaner.
My Journey
I'm trying to setup remote containers as described here: https://code.visualstudio.com/remote/advancedcontainers/develop-remote-host
Actually the containers are running fine, I'm using the second storage solution what means I add the following to my .devcontainer.json:
"workspaceMount": "source=/home/marvink/code,target=/workspaces,type=bind,consistency=cached"
And my workflow currently looks something like this:
Clone project locally (with .devcontainer already in the project)
Add workspaceMount above to devcontainer.json
Clone project on remote (e.g. to /home/marvink/code/new-project)
Open project locally
Build and reopen in container
Work on the files on the remote
My issue
This works but now I have files on my local drive that never get touched which isn't ideal but not a disaster, the bigger issue is when I want to update the devcontainer. I need to do that locally (in a seperate window), manually need to copy paste that to the remote if I want to commit it to git and off-course I sometimes forget this and try to edit it remotely which is causing a lot of frustration (and sometimes it seems like it does use the remote config, but that might have been a mistake?).
This is why I want to setup rsync both ways to sync changes to files and as a bonus I can edit files locally when I'm offline. In the link it's described how to do it manually but I want it automated so that I can't forget or make mistakes.
From Powershell I'm able to run an rsync command that syncs one-way and I can extend that to sync 2-way:
wsl rsync -rlptzv --progress --exclude=.git '$PWD' 'marvink#s-dev01:~/code/new-project'
This needs to be ran locally but I can't find a way to do that. I'd need to run a task locally for example, but that isn't possibly when working on a remote (https://github.com/microsoft/vscode-remote-release/issues/168).
The other way around doesn't seem like an option to me as I don't want to expose any ports on my laptop and firewalls would get in the way depending on where I am.
My question
My workflow still seems a bit convoluted so I'm open to suggestions on that end but any ideas on how to sync my workspace files?
You don't need a local version of your code (containing the .devcontainer folder) if you're storing that code on the remote server. You should be able to open an ssh target in VScode using the Remote - SSH extension, which is the recommended approach in the link you added. The extension Remote - Containers 'stacks' on top of the SSH extension, so once connected over SSH you then connect to the container using the .devcontainer.json configuration located on your remote server.
If you don't want to use the extension and use a bind mount + specify docker.host in your settings.json file, you can sync code using the approaches in that same article, through SSHFS, docker machine, or rsync.
The task: Run a tensorflow train.py script I wrote in the cloud with at least 32GB of memory.
Requirements: The script has some dependencies like numpy, scipy, and mkt. I need to be able to install these. I just want a no-nonsense ssh shell like experience. I want to put all my files including the training data in a directory, pip install the packages if necessary, then just hit python train.py and let it run. I'm not looking to run a web app or have Google's machine learning platform do it for me.
All the tutorials around seem needlessly complicated, like they're meant for scaled deployments with http requests and all that. I'm looking for a simple way to run code on a server since my computer is too weak for machine learning.
Don't use AppEngine -- use Compute Engine instead. Almost the same thing, but very simple and you are completely in control of what you run, what you install etc.
Simple steps that should work for you:
-Create a Compute Engine instance
-Chose operating system (Ubuntu xx, but you can choose others instead)
-Chose how many CPUs and how much memory you want (select Customize in order to set yourself the CPU/memory ratio rather than getting default options)
-Enable HTTP/HTTPs in order to be able to use Tensorboard later
-Once created, SSH into the machine. Python is already pre-installed (2.7 default, but 3.x also available as Python3 alias)
-Install Tensorflow, Numpy, Pandas, and whatever you want with simple PIP
-You can also install Bazel if you want to build Tensorflow from source and to speed up the CPU operations
-Install gcsfuse if you want to copy/paste stuff quickly from cloud storage buckets
-Use tmux if you want to run several Tensorflow sessions in parallel (i.e.to try different hyperparameters/etc.)
This is all very clean and simple and works really well. Don't forget to shut it down after finished. You can also create a Preemptable instance to make it super-cheap (but it can be shut down at any time without warning, but happens rarely).
I am currently working on an automated build/CI system for various embedded firmware projects which have been developed in Rowley Associates CrossStudio. The projects can each be built at the command line using CrossBuild.
Now, on to the Docker part:
We need a way of guaranteeing consistent build environments. A build must run identically on any engineer workstation or the build server. As all of the build steps, including running CrossBuild can be executed in a command line Linux environment, I opted to use Docker containers to guarantee environmental consistency.
My intention is to use Docker containers as disposable 'build bots' in the following way. When a build is initiated (either manually by the engineer or by an automated build process), a container is created from the appropriate image, the process runs to completion, outputs are copied to persistent storage and then the container is thrown away.
At the moment, I'm in the process of walking through the build steps manually to prove that everything works as I expected. It doesn't!
I have a Docker container with the appropriate tools installed and can manually invoke CrossBuild and successfully build my project. Unfortunately, the build takes about 30 minutes to complete. This compares to a build time of ~1.5 minutes if I use the same tool directly on my Windows workstation.
I have a Windows 7 (x64) workstation and so to run the Docker container, I'm using Boot2Docker on VirtualBox.
If I observe the CPU and memory usage of the Docker container (either by running ps -aux inside the Boot2Docker VM or observing the resource usage of the Boot2Docker VM in Windows Task Manager), barely any resources are being used (<5% CPU usage, tens of megabytes of RAM). If I build the project using CrossBuild directly on Windows, the CPU usage fluctuates but peaks at 25% (i.e. maxing out one of my 4 threads).
I have proved that, in principle, processes inside the Docker container can occupy all available CPU resources by writing a simple infinite loop in Python, running it and observing CPU usage in Task Manager on the host PC. As expected, a single core was fully utilised.
Further information
Behind the scenes, CrossBuild is driving GCC-ARM
In order to get data in to and out of the Docker container, I'm using VirtualBox shared folders and then creating the container using the -v argument for each share.
Current lines of enquiry
I just had a moment of inspiration and started to wonder whether there might be a read/write bandwidth constraint caused by the way that I'm getting data in and out of the container (i.e. the CPU is never being fully utilised as most of the time is spent waiting for reads and writes). I will investigate this possibility.
Sharing drives from Windows to VirtualBox is notoriously slow. If you want to build from your local machine, use Docker for Windows instead. If you want to replicate a cloud CI environment, you create a volume
docker volume create --name mydata
upload data to it
docker run -v mydata:/data --name temp alpine top
docker cp /my/local/dir temp:/data
docker rm -f temp
Then mount that docker volume as needed in your other CI container (that step can be included in the above).
Note that for a real CI, your data could come from other sources like github. In that case, you can create a container just to download the data into the docker volume.
I am trying to launch the two finesse suite on different browser using Jenkins but not able to do so. Even I have parameterized my build but same browser is launching each time.
Do your execution scripts have a hard coded reference to which browser to use, or do you leave that to the system default? If system default, then it makes sense that you would have both running in the same browser since each suite is just choosing the system default (which is the same).
If you could script the registry (windows) or path (other OS) change to modify which browser is the system default, then you can override the system default browser at runtime and choose which one you want run for that thread. There may be problems if the two jobs kick off at the same time and both are trying to change the system default browser.