I'm new to google cloud and I'm trying my best to implement CI / CD on a simple MERN stack application.
I was able to set up google cloud build and get the deploy with automatic triggers (from github branch) to work.
I have only one question: in the cloudbuild.yaml file I have inserted several steps, to do first npm install, then npm build and finally deploy. The process is quite slow and it also copies all the code to the server, not just the build folder.
I would like to understand how to configure the file to be able to use the build folder already present in the github repository and do the deploy directly, without further steps.
On the frontend I use react, the only thing I have to do is serve the static index.html file.
Here is my current configuration (only the frontend, to understand)
steps:
# Step 0: FRONTEND npm install
- name: "gcr.io/cloud-builders/npm"
dir: 'frontend'
args: ['install']
timeout: "5m"
# Step 1: FRONTEND optimized build
- name: "gcr.io/cloud-builders/npm"
dir: 'frontend'
args: ["run", "build"]
timeout: "5m"
# Step 2: FRONTEND Deploy cloud app frontend
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy", "-v", "0-1-2", "app.yaml"]
dir: 'frontend'
timeout: "20m"
timeout: 60m
In my remote repository I have the build folder ready to use, but I can't figure out how to do the job.
I've also tried something similar to this:
Upload build folder in Google Cloud Build
without success.
One of the problems is that at the end of the process it appears that cloud build is trying to start npm start.
Finally, I am currently using express to simply serve static files.
Using google cloud can I avoid using it and simply configure the handlers from the app.yaml file?
What web server does it use under the hood to serve files if I don't implement express?
If you only need to deploy the build folder, then the link you've provided should do the job. You don't need to run npm install and npm run build unless you want to to copy all your code and create the build folder using Cloud Build.
Credits to #Guillaume (updated, assuming the .yaml file is located at the root directory of the project and build located at a directory called frontend):
steps:
# Step 0: FRONTEND copy only build and deploy cloud app frontend
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: bash
dir: 'frontend'
args:
- '-c'
- |
cd ..
cp app-production-build.yaml frontend/build
cd frontend/build
gcloud app deploy
timeout: '1600s'
By default, Cloud Build uses a directory named /workspace as a working directory. If you set the dir field in the build step, the working directory is set to /workspace/<dir>. Basically for this build step, Cloud Build's currently on a directory called frontend in the root directory of your project/repo.
Also as Guillaume mentioned, you need to update your app.yaml URL handlers for this to work. I tested this sample inside the build folder of a new React app and it works for me:
runtime: nodejs12
service: frontend-react
handlers:
# Serve all static files with url ending with a file extension
- url: /(.*\..+)$
static_files: \1
upload: (.*\..+)$
# Catch all handler to index.html
- url: /.*
static_files: index.html
upload: index.html
To answer your last question, you can serve static files directly from your app and what you need to do is to configure URL handlers such as static_dir or static_files.
Take note that each file inside build/static are cached to speed up performance so there could be a delay whenever you make several deployments.
Node.js on App Engine Standard doesn't use any webserver under the hood and you have to set it up by specifying it on your package.json. Here's some web frameworks recommended by Google docs.
Related
I am working on a project built by React/IONIC. Now I'm trying to add social sign in to the application, and to do so I need to put a file in the following directory:
/.well-known/apple-app-site-association
But the issue is, whenever I try to deploy the app on Amplify (it's connected to github repo), it doesn't deploy /public folder.
My Amplify.yml file is as follows:
version: 1.2.9 env: variables: REACT_APP_API_ENDPOINT: https://******.execute-api.ca-central-1.amazonaws.com/staging frontend: phases: preBuild: commands: ['npm ci'] build: commands: ['npm run build'] artifacts: baseDirectory: build files: - '**/*' cache: paths: - 'node_modules/**/*'
I would really appreciate your help here.
contents of public folder are deployed on root folder. So you /.well-known/apple-app-site-association will be available from https://<yourdomian.com>/.well-known/apple-app-site-association
You will have to add custom Rewrites and Redirect in AWS Amplify in AWS Console as the default Rewrite will redirect all request to Index.html for React App to handle
So far I just deployed a React app on Amplify.
And I do not ask in amplify.yml the deployment of the public/ folder. Rather I create a build of my project with a yarn build, which embeds the content of the public folder inside the new build/ folder.
So I hope Ionic does not ask to deploy the public folder on top of build/ folder.
I figured out as well that I needed to configure the URL routes inside rewrite and redirect menu of Amplify whenever my react app makes GET requests like my-domain.com/privacy.pdf.
As long as I did not specify the ".pdf" in the previously existing...
</^[^.]+$|\.(?!(css|gif|ico|jpg|js|png|txt|svg|woff|ttf|map|json)$)([^.]+$)/> /index.html 200 (Rewrite)`
...rule, the routing was not able to fetch the privacy.pdf file.
I would suggest to look at the file extension of your logo too if the file extension of the sign you try to load is not included in the rewrite rule.
TL;DR
Remove all of your redirects, so it will host the public folder and its contents.
There's no need to redeploy the server. The changes are made on the fly.
I've spent a fair amount of time reading through Stack Overflow questions and can't find what I'm looking for.
I'm creating a Flask API and React frontend to be deployed on GAE. My directory structure looks like this:
application_folder
-> api
-> -> app.yaml
-> -> main.py
-> react_frontend
-> -> app.yaml
-> -> {directories after using create-react-app}
For development
1.) The package.json of the React app, I've set a proxy:
"proxy": "http://localhost:5000/"
My React frontend is running on port 3000.
2.) I fetch within the React frontend's App.js file as such:
fetch('api/endpoint')
Works like a dream.
For Production
However, when deploying to GAE, I have to make the following changes:
1.) Remove proxy from package.json. I couldn't find a way for this to work with a proxy as I received a 404 error on the React frontend on production.
2.) Add Access-Control-Allow-Origin to Flask API in main.py.
#app.route("/api/endpoint", methods=["GET"])
def endpoint():
resp = make_response({"cat": 15})
resp.headers["Access-Control-Allow-Origin"] = "*"
return resp
3.) Now, I have to fetch with absolute path from App.js.
fetch('flask-production-domain.com/api/endpoint')
My question is, would you recommend a way of deploying to production/setting up local development such that I don't have to rewrite the code base when I deploy to production?
By the way my app.yaml files read as such:
# api app.yaml
runtime: python38
env_variables:
none_of: "your_business"
# frontend app.yaml
runtime: nodejs12
service: banana
handlers:
- url: /static
static_dir: build/static
- url: /(.*\.(json|ico|js))$
static_files: build/\1
upload: build/.*\.(json|ico|js)$
- url: .*
static_files: build/index.html
upload: build/index.html
Thanks in advance.
EDIT
I've accepted dishant's answer, but I've done it slightly differently than they've suggested.
Firstly, these two resources are incredible. This guy wrote "the book" on Flask:
https://www.youtube.com/watch?v=Q2eafQYgglM
https://www.youtube.com/watch?v=qEfduVAQ8FQ&t=1289s
In the second video, he describes two methods of deploying. The first one is similar to what dishant suggested. In the video Miguel describes this option as his lesser favorite of the two he describes, as serving the files from python is slow. For me, it works well for now, as it's the "easier" of the two, and it seems like you can easily switch down the road.
So, I've made the following changes:
I changed the directory structure to:
Top Level
-> directories after using create-react-app
application_folder
-> api
-> -> app.yaml
-> -> main.py
Instead of creating the routing in the app.yaml file, I added the following to the main.py fie:
app = Flask(__name__, static_folder="build", static_url_path="/")
#app.route("/")
def index():
return app.send_static_file("index.html")
I added the following line to the package.json to create a script to build and move the static files to the api directory.
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"create-app": "yarn build && rm -r api/build && cp -r build api/build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
So now I run yarn create-app to build.
I'm going to look into Travis for CI/CD, since I've used that before.
From https://create-react-app.dev/docs/proxying-api-requests-in-development/:
Keep in mind that proxy only has effect in development (with npm
start), and it is up to you to ensure that URLs like /api/todos point
to the right thing in production. You don’t have to use the /api
prefix. Any unrecognized request without a text/html accept header
will be redirected to the specified proxy.
You would not serve your frontend from memory in a deployed environment(app engine). Instead, you should build your app and package it along with the Flask app and set routes in app.yaml to point to frontend and backend properly.
You only need 1 app.yaml file.
Example:
Build your react app using npm run build
Copy the build files to your flask app in a new folder called build.
app.yaml:
runtime: python38
env_variables:
none_of: "your_business"
handlers:
# frontend
- url: /static
static_dir: build/static
- url: /(.*\.(json|ico|js|html))$
static_files: build/\1
upload: build/.*\.(json|ico|js|html)$
# backend
- url: /.*
script: auto
You can leverage ci/cd services like cloud build or Jenkins to package the apps and deploy them on app engine.
If you want to use proxy within your deployment, you don't have many options indeed. For example, most of the situations where you need to use proxy with App Engine, it's needed to use an instance on Compute Engine, so you can set static IP and configure better it - as explained in this case here.
Another option would be to deploy your API's backend code with Extensible Service Proxy (ESP to an App Engine Flex environment, using Cloud Endpoints. As explained here, you can configure like this, so ESP can obtain your API's Endpoints configuration, which allows ESP to proxy requests and responses so that Endpoints can manage your API.
These are some different approaches that might give you an idea of how to do it, working more as a starting point than as a solution. However, you are kind of limited on your options here, as your use case is specific and GAE has some specific settings available as well. I would recommend you to keep doing what is working right now and then, give it a try in one of this other options.
I am experiencing a strange issue with dependencies when deploying my application to Google App Engine (Python 2.7).
I have specified my dependencies in the requirements.txt file:
Flask==1.0.2
werkzeug>=0.14
flask-cors
twilio
httplib2
gunicorn
Jinja2
google-cloud-ndb
exponent_server_sdk
These work fine with the local development server when installed locally with pip from the requirements.txt folder.
However, when I deploy to app engine, it appears that the module exponent_server_sdk has not been installed (ImportError: No module named exponent_server_sdk).
I normally deploy by pushing to a git repository which is then deployed with Google Cloud Build ("gcr.io/cloud-builders/gcloud"). There are no errors when I deploy and other dependencies in the requirements.txt file work fine such as twilio or Jinja2.
Whilst investigating this I tried pushing directly from my development machine with gcloud app deploy. When I do this, for some strange reason, exponent_server_sdk is available and works correctly.
This behaviour seems very strange to me and I can't find any documentation of people experiencing similar errors. I wonder if anyone can give me any guidance about what might be causing this issue or places I can look to find errors (for example logs of the process of installing the requirements.txt file during deployment/instance startup).
In response to the comments:
I am running in the standard environment.
My cloudbuild.yaml looks like this:
steps:
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy", "app.yaml", "dispatch.yaml", "service_1.yaml", "service_2.yaml", "service_3.yaml", "index.yaml"]
timeout: "1600s"
The .yaml file for the service in which I am experiencing the error looks like this:
runtime: python27
api_version: 1
threadsafe: true
service: notifications
libraries:
- name: ssl
version: latest
handlers:
- url: /.*
script: notifications.app
For the (1st gen) standard environment the deployment process does not use a requirements.txt file, the libraries should be installed inside your app. From Copying a third-party library:
Use pip (version 6 or later) with the -t <directory> flag to
copy the libraries into the folder you created in the previous step.
For example:
pip install -t lib/ <library_name>
Yes, if you want you can use a requirements.txt file to do the installation inside your app, but that file itself is not used by the deployment process.
Since you're not deploying directly from the environment in which you had the libraries installed you'd have to add the installation directory to the git repo as well.
I am trying to implement CI/CD functionality using cloud build I have repository available on github. I have integrated my github repository with cloud source repository.Now after creating build triggers when I am running it is showing error:
ERROR: (gcloud.app.deploy) There is a cloudbuild.yaml in the current directory, and the runtime field in /workspace/app.yaml is currently set to [runtime: nodejs]. To use your cloudbuild.yaml to build a custom runtime, set the runtime field to [runtime: custom]. To continue using the [nodejs] runtime, please remove the cloudbuild.yaml from this directory.
Here is my cloudbuild.yaml file:
steps:
#Install
- name: 'gcr.io/cloud-builders/git'
args: ['clone', 'https://github.com/GoogleCloudPlatform/cloud-builders']
#Deploy
- name: 'gcr.io/cloud-builders/gcloud'
args: ['app', 'deploy']
app.yaml
env: flex
runtime: nodejs
Someone please let me know what I am missing or doing wrong.Any help would be appreciated.
THANKS
Try moving all project files including the app.yaml into a subdirectory except couldbuild.yaml and specify the subdirectory containing them in the couldbuild.yaml :
#Deploy
- name: 'gcr.io/cloud-builders/gcloud'
args: ['app', 'deploy', '[SUBDIRECTORY/app.yaml]']
Cloud build builds a Docker file that already specifies the runtime, so you won't have to re-specify it again in app.yaml. It's perfectly fine to change the runtime in app.yaml to custom.
I created a simple Vue.js application. Then I created a build for production using
npm run build command which creates dist folder in the project structure.
Then I use gcloud app deploy command to deploy it to Google App Engine, but then the deployment stops and gives error as:
ERROR: (gcloud.app.deploy) INVALID_ARGUMENT: This deployment has too many files. New versions are limited to 10000 files for this app.
Can someone please tell me what is the proper way to deploy Vue.js application to Google App Engine?
Have you tried deploying your Vue.js app to Google App Engine using Cloud Build? I have had no problem deploying any Vue.js apps in this way. Try following this tutorial for complete instructions.
Basically, you would have to include the following two files in your project root directory when deploying your Vue.js app to Google App Engine via Cloud Build:
App.yaml
runtime: nodejs10
handlers:
# Serve all static files with urls ending with a file extension
- url: /(.*\..+)$
static_files: dist/\1
upload: dist/(.*\..+)$
# catch all handler to index.html
- url: /.*
static_files: dist/index.html
upload: dist/index.html
and
cloudbuild.yaml
steps:
- name: node:10.15.1
entrypoint: npm
args: ["install"]
- name: node:10.15.1
entrypoint: npm
args: ["run", "build"]
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy"]
timeout: "1600s"
In the case you're not using cloud build you may just use the app.yaml above and reference the steps implied in the cloudbuild.yaml, which means:
run npm install
run npm run build
run gcloud app deploy
You have too many files in the project.
In your app.yaml file, add the skip_files tag to it so the deployment does not include unnecessary files or folder in the upload. You can also mix with regex so for example:
skip_files:
- node_modules/
- .gitignore
- src/
- public/
- babel.config.js
- ^(.*/)?\..*$
I found some Google Cloud Platform documentation that may be helpful to your issue, in this link under the deployment section it's indicated that for every deployment you only can upload 10,000 files per version and each file is limited to a maximum size of 32 megabytes. You're running into one of this two problems, I suggest to check this out on your app and try to deploy it again.
if you don't want to waste $10 on namecheap for an SSL cert and hours of your time (like me) you probably want to add the secure: always option to each handler.
- url: /.*
static_files: dist/index.html
upload: dist/index.html
secure: always
like so! That's if you're using a custom domain though.