I'm using custom runtime on GAE Flexible environment.
Imagemagick is installed and it's working as expected.
RUN apt-get update && \
apt-get install imagemagick -y
I now need additional fonts installed (as some pdfs, when being converted to images, aren't rendering correctly).
Are the steps to install a new font:
apt-get install the font(s)
Somehow get imagick_type_gen ( https://martin-thoma.com/add-a-new-font-to-imagemagick/)
Call perl imagick_type_gen > types.xml
Copy over imagemagick types with types generated in step 3.
Are the steps outlined correct?
How does one download a script that isn't available via apt-get (step 2)?
Does GAE flexible environment allow you to overwrite files (steps 3 and 4 above).
Thanks
Related
The issue
I have a web scraper running in AWS lambda but in a few weeks AWS lambda will stop supporting Ruby 2.7. I built my scraper last year using this tutorial.
I need to find a version of chrome driver & headless chrome that is compatible with Ruby 2.7, But I don't know exactly where to start.
I have looked at the ChromeDriver's downloads portal But I don't see any indication there that Chrome driver will work for ruby 2.7 or any other specific version of ruby for that matter.
The code I have works by accessing the ChromeDriver binary and starting it inside a specific folder
I downloaded the specific binaries I am using by running these commands:
# serverless chrome
wget https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-37/stable-headless-chromium-amazonlinux-2017-03.zip
unzip stable-headless-chromium-amazonlinux-2017-03.zip -d bin/
rm stable-headless-chromium-amazonlinux-2017-03.zip
# chromedriver
wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
unzip chromedriver_linux64.zip -d bin/
rm chromedriver_linux64.zip
Solution
I found the solution to this problem. Ruby 2.7 that Lambda offers by default runs on top of Amazon Linux 2 (which lacks many important libraries & dependencies), unfortunately, there's nothing you can do to change that.
However, Amazon offers you the ability to run your code in a custom docker image that can be up to 10GB in size.
I fixed this problem by creating my own image using the following Dockerfile
FROM public.ecr.aws/lambda/ruby:2.7
# Install dependencies needed to run MySQL & Chrome
RUN yum -y install libX11
RUN yum -y install dejavu-sans-fonts
RUN yum -y install procps
RUN yum -y install mysql-devel
RUN yum -y install tree
RUN mkdir /var/task/lib
RUN cp /usr/lib64/mysql/libmysqlclient.so.18 /var/task/lib
RUN gem install bundler
RUN yum -y install wget
RUN yum -y groupinstall 'Development Tools'
# Ruby Gems
ADD Gemfile ${LAMBDA_TASK_ROOT}/
ADD Gemfile.lock ${LAMBDA_TASK_ROOT}/
RUN bundle config set path 'vendor/bundle' && \
bundle install
# Install chromedriver & chromium
RUN mkdir ${LAMBDA_TASK_ROOT}/bin
# Chromium
RUN wget https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-37/stable-headless-chromium-amazonlinux-2017-03.zip
RUN unzip stable-headless-chromium-amazonlinux-2017-03.zip -d ${LAMBDA_TASK_ROOT}/bin/
RUN rm stable-headless-chromium-amazonlinux-2017-03.zip
# Chromedriver
RUN wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
RUN unzip chromedriver_linux64.zip -d ${LAMBDA_TASK_ROOT}/bin/
RUN rm chromedriver_linux64.zip
# Copy function code
COPY app.rb ${LAMBDA_TASK_ROOT}
WORKDIR ${LAMBDA_TASK_ROOT}
RUN tree
RUN ls ${LAMBDA_TASK_ROOT}/bin
# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
CMD [ "app.handle" ]
Notes
If your code was previously deployed using a zip file you will have to either destroy the previous function or create a second function with the code update, it all comes down to how you want to handle deployment.
It is possible to automate the deployment process using the serverless framework
Google official documentation is available here:
https://cloud.google.com/appengine/downloads#Google_App_Engine_SDK_for_PHP
But it doesn't provide sufficient information about the following step:
"4 - Build and install the PHP interpreter and App Engine PHP extension. Specify the path to php-cgi and gae_runtime_module.so when running the development server."
I'm using a new Virtualbox machine with Ubuntu 15.10 and PhpStorm to test GAE.
Could someone please provide clear instructions about step 4? What do I need to do to install the php interpreter and the App Engine php extension?
P.s. I've already searched with google but I only found old/confusing tutorials
That GAE PHP extension seems like a quite new thing. Don't remember using it on the SDK in Ubuntu 14.04.
You need to build PHP and that extension from source. You should grab the latest PHP5.5 branch from their source repo (http://php.net/git.php) and build it. That linked page contains instructions on building PHP but the procedure is similar to the following:
$ git clone <php-src>
$ cd ./php-src/
$ git checkout PHP-5.5
$ ./buildconf
$ ./configure --prefix="/opt/php55"
$ sudo make && sudo make install
And remember to pick the modules and packages you want to compile with PHP5.5 to be used in the SDK. I think Google had an official list of modules and extensions they use inside GAE PHP and inside the SDK PHP. The prefix argument tells the compiler where to install the resulting application.
Then you need to get that source for the PHP extension and build it
$ git clone https://github.com/GoogleCloudPlatform/appengine-php-extension
$ cd appengine-php-extension
$ phpize # remember to use the phpize from the just built PHP5.5 binaries
$ ./configure
$ sudo make && sudo make install
(That Git repository contains detailed building instructions so you should probably refer to them when building.)
Enable the resulting .so for the PHP5.5 you just built using the PHP configuration files.
After that you need to install the PHP SDK and configure it to use the newly built PHP binary
$ dev_appserver.py <...> --php_executable_path=/opt/php55/bin/php-cgi
The SDK will let you know if the built PHP binaries are incompatible with the SDK version you use. I remember compiling the PHP from source around 5 times before it worked without any warnings.
But essentially they are telling you to compile PHP from source, then compile their extension from source and then use the built PHP+extension with the downloaded SDK. These instructions are from the top of my head so you may need to adjust the commands and procedures.
The process can be simplified by using Docker, here is an image you can use: https://hub.docker.com/r/mhariri/docker-google-appengine-php/
To run your app, you just need docker installed, and then run the following command in your app directory:
docker run -it -v $(pwd):/app --rm --net=host mhariri/docker-google-appengine-php
I try to create a .bat file to automate the installation of several apps on a virtual machine. I use Chocolatey, but I have some difficulties for the installation of 3 apps: WinMerge, AdobeReader, and Paint.net
I want install these apps on another drive: F:\Program Files. I tried many commands, but none work.
> choco upgrade adobereader -y -ia "INSTALLDIR=""F:\Program Files\Adobe"""
> choco upgrade paint.net -y -ia "INSTALLDIR=""F:\Program Files\paint.net"""
> choco upgrade winmerge -y -ia "INSTALLDIR=""F:\Program Files\WinMerge"""
Paint.net are installed but in the C:\ drive. Winmerge extracts all the files on the F: drive, but isn't installed. And Adobe Reader fails.
It may help to understand first that there are different types of installers for Windows applications (in Chocolatey terminology we call them native installers) and they all accept that custom directory location with a different argument (and sometimes not at all if they are a homegrown custom version). We have this documented at https://github.com/chocolatey/choco/wiki/GettingStarted#overriding-default-install-directory-or-other-advanced-install-concepts (which you've likely read based on your use of install arguments).
MSI- it is either TARGETDIR or INSTALLDIR (as you have).
NSIS - /D location with out quotes even if spaces and must be passed as the very last argument.
InnoSetup - /DIR="location"
There is a reason we have https://github.com/chocolatey/choco/issues/258 (with an original https://github.com/chocolatey/chocolatey/issues/32 from 2011) and a better reason we haven't implemented it yet - it's not exactly easy to do without knowing what the installer type is ahead of time. It is something that is planned, because it would be a huge win for everyone if we could support one switch to override them all.
I finally run these 3 commands and it's working.
choco upgrade adobereader -y -ia "INSTALLDIR=""F:\Program Files\Adobe"""
choco upgrade paint.net -y -ia "TARGETDIR=""F:\Program Files\paint.net"""
choco upgrade winmerge -y -ia "/DIR=""F:\Program Files\WinMerge"""
I would like to customise a (Python) Standard Runtime Managed VM.
In theory, this should be possible by adding some extra commands to the VM Dockerfile.
Google's documentation states that a VM Dockerfile is automatically generated when the App is first deployed;
If you are using a standard runtime, the SDK will create a Dockerfile for you the first time you run the gcloud preview app deploy commands. The file will exist in a predetermined location:
If you are developing in Java, the Dockerfile appears in the root of the compiled Web Application Archive directory (WAR)
If you are developing in Python or Go, the Dockerfile appears in the root of your application directory.
And that extra commands can indeed be added;
You can add more docker commands to this file, while continuing to run and deploy your app with the standard runtime declaration.
However in practice the Dockerfile is automatically deleted immediately after deployment competes, preventing any customisation.
Has anyone managed to add Dockerfile commands to a Managed VM with a Standard Runtime? Any help would be gratefully appreciated.
I tried the same thing and did not succeed. There is however an equivalent way of doing this that I fell back to.
You can create a custom runtime that mimics the standard runtime.
You can do this because Google provides the Docker base images for all the standard runtimes. Mimicking a standard runtime is therefore simply a matter of selecting the right base image in the Dockerfile of the custom runtime. For the standard Python App Engine VM the Dockerfile is:
FROM gcr.io/google_appengine/python-compat
ADD . /app
Now that you have recreated the standard runtime as a custom runtime, you can modify the Dockerfile to make any customizations you need.
Important Note
The development server does not support custom Dockerfiles (you will get an error about --custom-entrypoint), so you have to move your test environment to App Engine servers if you are doing this. I think this is true regardless of whether you are using a standard runtime and customizing the Dockerfile or using a custom runtime. See this answer.
A note about the development server not working with custom runtimes - dev_appserver.py doesn't deal with Docker or Dockerfiles, which is why it complains about needing you to specify --custom_entrypoint. However as a workaround you can manually set up the dependencies locally. Here's an example using 'appengine-vm-fortunespeak' which uses a custom runtime based on python-compat:
$ git clone https://github.com/GoogleCloudPlatform/appengine-vm-fortunespeak-python.git
$ cd appengine-vm-fortunespeak-python
# Local dependencies from Dockerfile must be installed manually
$ sudo pip install -r requirements.txt
$ sudo apt-get update && install -y fortunes libespeak-dev
# We also need gunicorn since its used by python-compat to serve the app
$ sudo apt-get install gunicorn
# This is straight from dev_appserver.py --help
$ dev_appserver.py app.yaml --custom_entrypoint="gunicorn -b localhost:{port} main:app"
Note that if you are using any of the non -compat images, you can run your app directly using Docker since they don't need to emulate the legacy App Engine API, for example using 'getting-started-python' which uses the python runtime:
$ git clone https://github.com/GoogleCloudPlatform/getting-started-python.git
$ cd 6-pubsub
# (Configure the app according to the tutorial ...)
$ docker build .
$ docker images # (note the IMAGE_ID)
$ docker run -p 127.0.0.1:8080:8080 -t IMAGE_ID
Try the above with any -compat images and you will have problems - for example on python-compat you'll see initialization errors in runtime/google/appengine/tools/vmboot.py. It needs to be run on a real Managed VM instance.
My project use Django, and I need run the Django's ORM in Compute Engine, for this I want sync in one repository both Compute Engine and App Engine with my development server.
When I try:
gcloud init <proyect>
From the Compute Engine, the log is:
ERROR: Unable to initialize project [tcontur2], cleaning up [***].
ERROR: (gcloud.init) Your git version 1.7.10 is older than the minimum version 1.8.1.
Please install a newer version of git.
I try
apt-get update
apt-get upgrade
And nothing.
I too try:
git clone https://source.developers.google.com/myproyect
But the authenication fails. What is the username and password? It is not my gmail user account.
Is possible sync the Compute Engine with the code in App Engine? or only is possible with another repository? This option suppose other payment to Github, I dont want spend more.
I reproduced your issue from my fresh Debian instance and I was able to solve it. I downloaded the new git version from: http://code.google.com/p/git-core/downloads/detail?name=git-1.9.0.tar.gz
Also I needed to install some packages that you may have already installed but just in case I'll let you know which are them.
These are the steps I've taken:
sudo apt-get install libssl-dev libcurl4-openssl-dev libexpat-dev gettext
wget http://git-core.googlecode.com/files/git-1.9.0.tar.gz
tar -xvzf git-1.9.0.tar.gz
cd git-1.9.0
sudo make prefix=/usr all
sudo make prefix=/usr install
After that you can check your git version with:
git --version
Now, if the output of the last command displays "git version 1.9.0", you should be able to create the repository with the command:
gclod init <project>