Is there a way to store required libraries persistently? I'm running into error code H10 - c

Im deploying a sinatra webapp that makes use of a c library, through Ruby FFI. I need this library to be compiled on the target machine. I do this through a Rakefile normally, but because of Heroku's ephemeral hard drive, any time I run rake the compiled files dissappear. How can I make these libraries persist? Is there a way to push the files to git from heroku?
Possibly related: Whenever I try to access the page, the application crashes with a code=H10 error. The app runs fine locally.
My attempt at customizing a buildpack:
#!/usr/bin/env ruby
# This script compiles an application so it can run on Heroku.
# It will install the application's specified version of Ruby, it's dependencies
# and certain framework specific requirements (such as calling `rake assets:precompile`
# for rails apps). You can see all features described in the devcenter
# https://devcenter.heroku.com/articles/ruby-support
$stdout.sync = true
$:.unshift File.expand_path("../../../lib", __FILE__)
require "language_pack"
require "language_pack/shell_helpers"
begin
# my addition begins here
`mkdir /app/src`
`cd /app && curl -s https://www.astro.com/ftp/swisseph/swe_unix_src_2.10.02.tar.gz | tar xzvf -`
`cd '/app/src' && make libswe.so`
# my addition ends here
LanguagePack::ShellHelpers.initialize_env(ARGV[2])
if pack = LanguagePack.detect(ARGV[0], ARGV[1])
pack.topic("Compiling #{pack.name}")
pack.log("compile") do
pack.compile
end
end
rescue Exception => e
LanguagePack::ShellHelpers.display_error_and_exit(e)
end
Alternatively, I also tried:
#!/usr/bin/env bash
# The actual compilation code lives in `bin/support/ruby_compile`. This file instead
# bootstraps the ruby needed and then executes `bin/support/ruby_compile`
BUILD_DIR=$1
CACHE_DIR=$2
ENV_DIR=$3
BIN_DIR=$(cd $(dirname $0); pwd)
BUILDPACK_DIR=$(dirname $BIN_DIR)
# my addition begins here
mkdir /app/src
cd /app && curl -s https://www.astro.com/ftp/swisseph/swe_unix_src_2.10.02.tar.gz | tar xzvf -
cd '/app/src' && make libswe.so
# my addition ends here
source "$BIN_DIR/support/bash_functions.sh"
heroku_buildpack_ruby_install_ruby "$BIN_DIR" "$BUILDPACK_DIR"
if detect_needs_java "$BUILD_DIR"; then
cat <<EOM
## Warning: Your app needs java
The Ruby buildpack determined your app needs java installed
we recommend you add the jvm buildpack to your application:
$ heroku buildpacks:add heroku/jvm --index=1
-----> Installing Java
EOM
compile_buildpack_v2 "$BUILD_DIR" "$CACHE_DIR" "$ENV_DIR" "https://buildpack-registry.s3.us-east-1.amazonaws.com/buildpacks/heroku/jvm.tgz" "heroku/jvm"
fi
$heroku_buildpack_ruby_dir/bin/ruby $BIN_DIR/support/ruby_compile $#
Both seem to work at first (i.e. compile the C library, output the files I need), but when I run heroku run bash or the web application Im not able to find the files. This is the specific error, btw:
/app/vendor/bundle/ruby/3.0.0/gems/ffi-1.15.5/lib/ffi/library.rb:145:in `block in ffi_lib': Could not open library '/app/src/libswe.so': /app/src/libswe.so: cannot open shared object file: No such file or directory (LoadError)
I've also tried heroku release phase, but even then the files did not persist. Procfile:
release: bundle exec rake
web: bundle exec thin start -R config.ru -e $RACK_ENV -p ${PORT:-5000}
Rakefile:
#require "bundler/gem_tasks"
task default: [:clean, :c_build, :get_ephe]
task :clean do
`rm ./src/libswe.so`
`rm -rf ephe`
end
task :c_build do
`wget https://www.astro.com/ftp/swisseph/swe_unix_src_2.10.02.tar.gz`
`tar xvf swe_unix_src_2.10.02.tar.gz`
`rm swe_unix_src_2.10.02.tar.gz`
`cd src && make libswe.so && echo "Compiled Library"`
end
task :get_ephe do
`mkdir ephe`
`wget -P ephe https://www.astro.com/ftp/swisseph/ephe/seas_12.se1`
`wget -P ephe https://www.astro.com/ftp/swisseph/ephe/seas_18.se1`
`wget -P ephe https://www.astro.com/ftp/swisseph/ephe/sefstars.txt`
`wget -P ephe https://www.astro.com/ftp/swisseph/ephe/semo_12.se1`
`wget -P ephe https://www.astro.com/ftp/swisseph/ephe/semo_18.se1`
`wget -P ephe https://www.astro.com/ftp/swisseph/ephe/sepl_12.se1`
`wget -P ephe https://www.astro.com/ftp/swisseph/ephe/sepl_18.se1`
end

Related

React client and Golang server in same Dockerfile

I've build a React client application supported with a API written in Golang. I would like to use Docker to run these both apps using docker run.
I have the following project structure:
zid
|
|-web/ (my react folder)
main.go
Dockerfile
|
My goal is to run the main.go file in the zid folder and start the webapplication in the zid/web folder. The main.go file starts a API using Gin Gonic that will listen and serve on port 10000.
So I've tried the following:
# Build the Go API
FROM golang:latest as go_builder
RUN mkdir /zid
WORKDIR /zid
COPY . /zid
RUN GOOS=linux GOARCH=amd64 go build -a -ldflags "-linkmode external -extldflags '-static' -s -w" -o /go/bin/zid
# Build the React application
FROM node:alpine as node_builder
COPY --from=go_builder /zid/web ./
RUN npm install
RUN npm run build
# Final stage build, this will be the container with Go and React
FROM alpine:latest
RUN apk --no-cache add ca-certificates
COPY --from=go_builder /go/bin/zid /go/zid
COPY --from=go_builder /zid/ca /go/ca
COPY --from=node_builder /build ./web
EXPOSE 3000
WORKDIR /go
CMD ./zid
Next I did the following:
Build it with docker build -t zid . (no errors)
Run it with docker run -p 3000:3000 --rm zid
When I run this, it will startup the API, but when I go to http://localhost:3000/ then I get a Page does not work ERR: ERR_EMPTY_RESPONSE.
So the API starts up, but the npm build doens't. I am not sure what I am doing wrong, because the Docker container both contains the correct folders (go and web).
As you can see in the image it's all there I believe. What am I missing?
EDIT:
I am using the (*gin.Engine).Run() function to set the listen and serve on port 10000. In my local build my React application is sending request to localhost:10000. I always simply used npm start on the side of my React app (localhost:3000). My goal is to do the same but then all in one Dockerfile.
I am still a little unsure if I should EXPOSE ports 10000 & 3000 in my Dockerfile.
My HandleRequest function:
//Start the router and listen/serve.
func HandleRequests() {
router := SetupRouter()
router.Run(":10000")
}
My SetupRouter function:
//Setup the gin router
func SetupRouter() *gin.Engine {
router := gin.Default()
router.Use(CORSMiddleware())
router.POST("/auth/login", login)
router.POST("/component/deploy", deployComponent)
router.POST("/project/create", createProject)
router.POST("/diagram/create", createDiagram)
router.PATCH("/diagram/update", updateDiagram)
router.DELETE("/diagram/delete/:id", deleteDiagram)
router.GET("/diagram/:id", getDiagram)
router.GET("/project/list", getProjectsByUsername)
router.GET("/project/:id", getProject)
router.GET("/project/diagrams/:id", getDiagramsOfProject)
router.DELETE("/project/delete/:id", deleteProject)
router.GET("/application/list", applicationList)
router.GET("/instance/status/:id", getInstanceStatus)
router.GET("/user", getUser)
return router
}
Btw I just want to use the Docker container for Development and learning purpose only.
I've used the following multi-stage Docker build to create:
static VueJS UI HTML assets
compiled Go API http server (serving the above HTML assets)
Note: both Go and VueJS source is download from one git repo - but you could just as easily modify this to copy the two code-bases from local development directories.
#
# go build
#
FROM golang:1.16.5 AS go-build
#
# here we pull pkg source directly from git (and all it's dependencies)
#
RUN go get github.com/me/vue-go/rest
WORKDIR /go/src/github.com/me/vue-go/rest
RUN CGO_ENABLED=0 go build
#
# node build
#
FROM node:13.12.0 AS node-build
WORKDIR /app/vue-go
COPY --from=go-build go/src/github.com/me/vue-go/vue-go ./
# produces static html 'dist' here:
#
# /app/vue-go/dist
#
RUN npm i && npm run build
#
# final layer: include just go-binary and static html 'dist'
#
FROM scratch
COPY --from=go-build \
/go/src/github.com/me/vue-go/rest/rest \
/app/vue-go
COPY --from=node-build \
app/vue-go/dist \
/app/dist/
CMD ["/app/vue-go"]
I don't use Gin - but to use native net/http fileserver serving APIs and static HTML assets, use something like:
h := http.NewServeMux()
// serve static HTML directory:
if conf.StaticDir != "" {
log.Printf("serving on '/' static files from %q", conf.StaticDir)
h.Handle(
"/",
http.StripPrefix(
"/",
http.FileServer(
http.Dir(conf.StaticDir), // e.g. "../vue-go/dist" vue.js's html/css/js build directory
),
),
)
}
// handle API route(s)
h.Handle("/users",
authHandler(
http.HandlerFunc(handleUsers),
),
)
and start the service:
s := &http.Server{
Addr: ":3000", // external-facing IP/port
Handler: h,
}
log.Fatal(s.ListenAndServe())
then to build & run:
docker build -t zid .
docker run -p 3000:3000 --rm zid
I've found a solution! I've created a script on basis of multi-service container and then a run this script in my Dockerfile.
my script (start.sh):
#!/bin/sh
# Start the first process
./zid &
ZID_PID=$!
# Start the second process
cd /web
npm start &
WEB_PID=$!
# Naive check runs checks once a minute to see if either of the processes exited.
# This illustrates part of the heavy lifting you need to do if you want to run
# more than one service in a container. The container exits with an error
# if it detects that either of the processes has exited.
# Otherwise it loops forever, waking up every 60 seconds
while sleep 60; do
ps -fp $ZID_PID
ZID_PROCESS_STATUS=$?
if [ $ZID_PROCESS_STATUS -ne 0 ]; then
echo "ZID process has already exited."
exit 1
fi
ps -fp $WEB_PID
WEB_PROCESS_STATUS=$?
if [ $WEB_PROCESS_STATUS -ne 0 ]; then
echo "WEB process has already exited."
exit 1
fi
done
Here I first start my go executable and then I do a npm start
In my Dockerfile I do the following:
# Build the Go API
FROM golang:latest as go_builder
RUN mkdir /zid
WORKDIR /zid
COPY . /zid
RUN GOOS=linux GOARCH=amd64 go build -a -ldflags "-linkmode external -extldflags '-static' -s -w" -o /go/bin/zid
# Build the React application
FROM node:alpine as node_builder
COPY --from=go_builder /zid/web ./web
WORKDIR /web
RUN npm install
# Final stage build, this will be the container with Go and React
FROM node:alpine
RUN apk --no-cache add ca-certificates procps
COPY --from=go_builder /go/bin/zid /go/zid
COPY --from=go_builder /zid/static /go/static
COPY --from=go_builder /zid/ca /go/ca
COPY --from=node_builder /web /web
COPY --from=go_builder /zid/start.sh /go/start.sh
RUN chmod +x /go/start.sh
EXPOSE 3000 10000
WORKDIR /go
CMD ./start.sh
Here I am creating a Go executable, copy and npm install my /web folder and in de final stage build I start my ./start.sh script.
This will start my Golang application and the React development server. I hope it helps for others.

Which ChromeDriver & Headless Chrome versions exist that are compatible with ruby 2.7?

The issue
I have a web scraper running in AWS lambda but in a few weeks AWS lambda will stop supporting Ruby 2.7. I built my scraper last year using this tutorial.
I need to find a version of chrome driver & headless chrome that is compatible with Ruby 2.7, But I don't know exactly where to start.
I have looked at the ChromeDriver's downloads portal But I don't see any indication there that Chrome driver will work for ruby 2.7 or any other specific version of ruby for that matter.
The code I have works by accessing the ChromeDriver binary and starting it inside a specific folder
I downloaded the specific binaries I am using by running these commands:
# serverless chrome
wget https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-37/stable-headless-chromium-amazonlinux-2017-03.zip
unzip stable-headless-chromium-amazonlinux-2017-03.zip -d bin/
rm stable-headless-chromium-amazonlinux-2017-03.zip
# chromedriver
wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
unzip chromedriver_linux64.zip -d bin/
rm chromedriver_linux64.zip
Solution
I found the solution to this problem. Ruby 2.7 that Lambda offers by default runs on top of Amazon Linux 2 (which lacks many important libraries & dependencies), unfortunately, there's nothing you can do to change that.
However, Amazon offers you the ability to run your code in a custom docker image that can be up to 10GB in size.
I fixed this problem by creating my own image using the following Dockerfile
FROM public.ecr.aws/lambda/ruby:2.7
# Install dependencies needed to run MySQL & Chrome
RUN yum -y install libX11
RUN yum -y install dejavu-sans-fonts
RUN yum -y install procps
RUN yum -y install mysql-devel
RUN yum -y install tree
RUN mkdir /var/task/lib
RUN cp /usr/lib64/mysql/libmysqlclient.so.18 /var/task/lib
RUN gem install bundler
RUN yum -y install wget
RUN yum -y groupinstall 'Development Tools'
# Ruby Gems
ADD Gemfile ${LAMBDA_TASK_ROOT}/
ADD Gemfile.lock ${LAMBDA_TASK_ROOT}/
RUN bundle config set path 'vendor/bundle' && \
bundle install
# Install chromedriver & chromium
RUN mkdir ${LAMBDA_TASK_ROOT}/bin
# Chromium
RUN wget https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-37/stable-headless-chromium-amazonlinux-2017-03.zip
RUN unzip stable-headless-chromium-amazonlinux-2017-03.zip -d ${LAMBDA_TASK_ROOT}/bin/
RUN rm stable-headless-chromium-amazonlinux-2017-03.zip
# Chromedriver
RUN wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
RUN unzip chromedriver_linux64.zip -d ${LAMBDA_TASK_ROOT}/bin/
RUN rm chromedriver_linux64.zip
# Copy function code
COPY app.rb ${LAMBDA_TASK_ROOT}
WORKDIR ${LAMBDA_TASK_ROOT}
RUN tree
RUN ls ${LAMBDA_TASK_ROOT}/bin
# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
CMD [ "app.handle" ]
Notes
If your code was previously deployed using a zip file you will have to either destroy the previous function or create a second function with the code update, it all comes down to how you want to handle deployment.
It is possible to automate the deployment process using the serverless framework

Reusable docker image for AngularJS

We have an AngularJS application. We wrote a dockerfile for it so it's reusable on every system. The dockerfile isn't a best practice and it's maybe some weird build up (build and hosting in same file) for some but it's just created to run our angularjs app locally on each PC of every developer.
Dockerfile:
FROM nginx:1.10
... Steps to install nodejs-legacy + npm
RUN npm install -g gulp
RUN npm install
RUN gulp build
.. steps to move dist folder
We build our image with docker build -t myapp:latest .
Every developer is able to run our app with docker run -d -p 80:80 myapp:latest
But now we're developing other backends. So we have a backend in DEV, a backend in UAT, ...
So there are different URLS which we need to use in /config/xx.json
{
...
"service_base": "https://backend.test.xxx/",
...
}
We don't want to change that URL every time, rebuild the image and start it. We also don't want to declare some URLS (dev, uat, prod, ..) which can be used there. We want to perform our gulp build process with an environment variable instead of a hardcoded URL.
So we we can start our container like this:
docker run -d -p 80:80 --env URL=https://mybackendurl.com app:latest
Is there someone who has experience with this kind of issues? So we'll need an env variable in our json and building it and add the URL later on if that's possible.
EDIT : Better option is to use build args
Instead of passing URL at docker run command, you can use docker build args. It is better to have build related commands to be executed during docker build than docker run.
In your Dockerfile,
ARG URL
And then run
docker build --build-arg URL=<my-url> .
See this stackoverflow question for details
This was my 'solution'. I know it isn't the best docker approach but just for our developers it was a big help.
My dockerfile looks like this:
FROM nginx:1.10
RUN apt-get update && \
apt-get install -y curl
RUN sed -i "s/httpredir.debian.org/`curl -s -D - http://httpredir.debian.org/demo/debian/ | awk '/^Link:/ { print $2 }' | sed -e 's#<http://\(.*\)/debian/>;#\1#g'`/" /etc/apt/sources.list
RUN \
apt-get clean && \
apt-get update && \
apt-get install -y nodejs-legacy && \
apt-get install -y npm
WORKDIR /home/app
COPY . /home/app
RUN npm install -g gulp
RUN npm install
COPY start.sh /
CMD ["./start.sh"]
So after the whole include of the app + npm installation inside my nginx I start my container with the start.sh script.
The content of start.sh:
#!/bin/bash
sed -i 's#my-url#'"$DATA_ACCESS_URL"'#' configs/config.json
gulp build
rm -r /usr/share/nginx/html/
//cp right folders which are created by gulp build to /usr/share/nginx/html
...
//start nginx container
/usr/sbin/nginx -g "daemon off;"
So the build will happen when my container starts. Not the best way of course but it's all for the needs of the developers. Have an easy local frontend.
The sed command will perform a replace on the config file which contains something like:
{
"service_base": "my-url",
}
So my-url will be replaced by my the content of my environment variable which I willd define in my docker run command.
Than I'm able to perform.
docker run -d -p 80:80 -e DATA_ACCESS_URL=https://mybackendurl.com app:latest
And every developer can use the frontend locally and connect with their own backend URL.

Docker: permission on file created by npm inside the container

I have a Dockerfile to create a dev enviroment to develop a sailsJS app.
I just mount my source code into the container. I make my Git commit on my host machine but i would like to execute all my npm command in the container.
I have the following Dockerfile and i am running Docker (1.4.1) in ubuntu 14.10:
FROM ubuntu:14.04
### Utils ###
RUN apt-get update
RUN apt-get -y install build-essential git wget tar vim supervisor
### MongoDB ###
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
RUN echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | tee /etc/apt/sources.list.d/mongodb.list
RUN apt-get update
RUN apt-get install -y mongodb-org
RUN mkdir -p /data/db
### NodeJS ###
WORKDIR /tmp
RUN wget -O node http://nodejs.org/dist/v0.10.33/node-v0.10.33-linux-x64.tar.gz
RUN tar xf node
RUN mv node-v0.10.33-linux-x64 /usr/local/node
RUN ln -s /usr/local/node/bin/* /usr/local/bin
### Supervisord ###
RUN mkdir -p /var/log/supervisor
COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf
### Project ###
RUN npm install -g sails bower
WORKDIR /opt/sails
CMD ["/usr/bin/supervisord"]
EXPOSE 27017 1337
I run my container with the following command :
docker run -d -ti -p 1337:1337 -p 27017:27017 -v ~/dev/pinne:/opt/sails --name test-app loikg/sailsjs-mongo
The problem is that when I use command with npm inside the container that create files like sails genearet api I don't have the writing permission on them in the host machine.
How can i solve that ?
Users and Groups do not sync from host->container.
Your services in the container are running as root (UID:0 GID:0). Any files created by root in the container will need root access on the host.
One solution is to create a UID/GID inside the container that matches the UID/GID on the host. Then all your processes inside the container need to use that UID/GID so the files have the correct ownership/permissions.
Remember, it's UserID not user name. And GroupID not group name. The names need not match, only the numeric ID's.
It's kind of a pita. You will have to change your dockerfile to add the user, make sure your processes that create files are run with the correct uid, etc.
One of the workarounds is to use overlapping volumes, e.g.
... -v ~/dev/pinne:/opt/sails:ro -v /opt/sails/node_modules ...
would allow writing to /opt/sails/node_modules. The downside is that the changes will be lost upon the container termination (unless you copy the volumes data via --volumes-from). Another caveat AFAIR is that the path (~dev/pinne/node_modules -> /opt/sails/node_modules) should exist for this technique to work.

Heroku app buildpack fails after compiling successfully

I'm trying to build a C application using a custom heroku buildpack specifically for it. The problem is that after compiling it gets to the very last stage -----> Installing and fails. Would someone be able to explain what I need to do in order to successfully deploy it?
Commands:
$ git clone https://github.com/znc/znc.git && cd znc
$ heroku create --stack cedar --buildpack http://github.com/lonnen/heroku-buildpack-znc.git
Creating dark-clouds-666...
done, stack is cedar
BUILDPACK_URL=http://github.com/lonnen/heroku-buildpack-znc.git
http://dark-clouds-666.herokuapp.com/ | git#heroku.com:dark-clouds-666.git
Git remote heroku added
$ git push heroku master
... then it goes on to build everything seemingly normal ...
-----> Fetching custom git buildpack... done
-----> C app detected
-----> Running autogen.sh
-----> Configuring
-----> Compiling with Make
Linking znc...
ZNC was successfully compiled.
Use 'make install' to install ZNC to '/usr/local'.
-----> Installing
test -d /usr/local/bin || /usr/bin/install -c -d /usr/local/bin
test -d /usr/local/include/znc || /usr/bin/install -c -d /usr/local/include/znc
/usr/bin/install: cannot change permissions of `/usr/local/include/znc': No such file or directory
make: *** [install] Error 1
! Push rejected, failed to compile C app
To git#heroku.com:dark-clouds-666.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'git#heroku.com:dark-clouds-666.git'
Your user does not have permission to write in /usr/local. Usually, make install is done by root user, where make could be done by normal users.
The fastest way is to execute all by root user, which is unsafe, as you can modify everything, leading to a broken system.
You can set an install location different from /usr/local, or you can force heroku to ask for root permission (sudo make install).

Resources