GAE deployment (standard F1) outperforming (standard F2) with identical source? - google-app-engine

I have 2 GCP projects "dev", and "prod" for a simple NextJS app. I am deploying to App Engine with the following configurations:
app.dev.yaml
runtime: nodejs16
instance_class: F1
handlers:
- url: /.*
secure: always
script: auto
- url: /icons
static_dir: /public/icons
app.prod.yaml
runtime: nodejs16
instance_class: F2
handlers:
- url: /.*
secure: always
script: auto
- url: /icons
static_dir: /public/icons
next.config.js
const nextConfig = {
experimental: {
externalDir: true
},
reactStrictMode: true,
images: {
domains: ["ik.imagekit.io"]
},
distDir: "build"
};
module.exports = nextConfig;
package.json (scripts)
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start -p 8080",
"deploy:dev": "npm run build && gcloud app deploy --quiet --project=<dev-project> --appyaml=app.dev.yaml",
"deploy:prod": "npm run build && gcloud app deploy --quiet --project=<prod-project> --appyaml=app.prod.yaml",
"lint": "next lint --fix",
"format": "prettier --write './**/*.{js,jsx,ts,tsx,css,md,json}' --config ./.prettierrc"
},
My dev environment (F1 instance) performs substantially better than my prod environment (F2). Way faster cold boots and image loading. The dev instance was able to quickly render a 2MB hero jpg that consistently caused a memory fault on the prod instance. I've played around with different instance classes for prod (and eventually tinified the jpg), but nothing seems to perform as well as my dev standard environments F1. I have double and triple checked the instance classes in the GCP console for each project. At this point, I do not want to go to flex. I need to understand what is driving the inconsistency between the projects to be able to accurately gauge the appropriate GAE deployment for production.

Related

GCP App Engine and Cloud Build: Cannot find module '/workspace/server.js'

TLDR: Sorry for the longest question in history, but I hope this is comprehensive for any users having a similar problem. My app deploys successfully from cloud shell to my domain; however, when I try cloud build, I get
Cannot find module '/workspace/server.js'
The error likely has to due with my build handlers in the app.yaml, or something to do with my cloudbuild.yaml.
Solution: use the right handlers in app.yaml standard and properly set up your cloudbuild.yaml
I am having trouble using App Engine and Cloud Build together. I am using Cloud Build to set up CICD with my Github repo. I think the issue is due to the fact that I have been not been deploying the production build to app engine. I was able to successfully deploy manually (dev version) with:
gcloud app deploy
Now, I am having trouble with my Cloud Build. In particular, I am trying to run flex environment, but I keep getting "Neither "start" in the "scripts" section of "package.json" nor the "server.js" file were found." But my package.json has a startup script?
I also tried standard environment instead of flex, but I couldn't get the handlers figured out. I tried dozens of examples. I have included the standard environment app.yaml so you can see it.
Here's my package.json:
{
"name": "flier-mapper",
"version": "0.1.0",
"private": true,
"dependencies": {
"#firebase/storage": "^0.8.4",
"#parcel/transformer-sass": "^2.0.0",
"#react-google-maps/api": "^2.7.0",
"#testing-library/jest-dom": "^5.11.4",
"#testing-library/react": "^11.1.0",
"#testing-library/user-event": "^12.1.10",
"bootstrap": "^5.1.3",
"cors": "^2.8.5",
"emailjs-com": "^3.2.0",
"firebase": "^9.2.0",
"firebase-admin": "^10.0.0",
"firebase-functions": "^3.16.0",
"react": "^17.0.2",
"react-bootstrap": "^2.0.2",
"react-dom": "^17.0.2",
"react-ga": "^3.3.0",
"react-helmet": "^6.1.0",
"react-icons": "^4.3.1",
"react-pricing-table": "^0.3.0",
"react-router-dom": "^5.3.0",
"react-scripts": "4.0.3",
"react-tabs": "^3.2.3",
"stripe": "^8.188.0",
"web-vitals": "^1.0.1"
},
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
"eslintConfig": {
"extends": [
"react-app",
"react-app/jest"
]
},
"browserslist": {
"production": [
">0.2%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
}
}
Here's my cloudbuild.yaml. I also tried deleting the dir line:
steps:
- name: "gcr.io/cloud-builders/npm"
dir: 'frontend'
args: ['install']
timeout: "5m"
- name: "gcr.io/cloud-builders/npm"
dir: 'frontend'
args: ["run", "build"]
timeout: "5m"
- name: "gcr.io/cloud-builders/gcloud"
entrypoint: bash
args:
- "-c"
- |
cp app.yaml ./build
cd build
gcloud app deploy
timeout: "1600s"
timeout: 60m
Here's my flex app.yaml:
runtime: nodejs
env: flex
# This sample incurs costs to run on the App Engine flexible environment.
# The settings below are to reduce costs during testing and are not appropriate
# for production use. For more information, see:
# https://cloud.google.com/appengine/docs/flexible/nodejs/configuring-your-app-with-app-yaml
manual_scaling:
instances: 1
resources:
cpu: 1
memory_gb: 0.5
disk_size_gb: 10
And here's one of the many handlers I tried with a standard environemnt:
runtime: nodejs14
# This sample incurs costs to run on the App Engine flexible environment.
# The settings below are to reduce costs during testing and are not appropriate
# for production use. For more information, see:
# https://cloud.google.com/appengine/docs/flexible/nodejs/configuring-your-app-with-app-yaml
manual_scaling:
instances: 1
handlers:
# Serve all static files with url ending with a file extension
- url: /(.*\..+)$
static_files: build/\1
upload: build/(.*\..+)$
# Catch all handler to index.html
- url: /.*
static_files: build/index.html
upload: build/index.html
- url: /.*
secure: always
redirect_http_response_code: 301
script: auto
EDIT
So, I got it to run on standard environment, which is useful for https requests. Here's my app.yaml that I got working:
runtime: nodejs14
# This sample incurs costs to run on the App Engine flexible environment.
# The settings below are to reduce costs during testing and are not appropriate
# for production use. For more information, see:
# https://cloud.google.com/appengine/docs/flexible/nodejs/configuring-your-app-with-app-yaml
manual_scaling:
instances: 2
handlers:
- url: /.*
secure: always
redirect_http_response_code: 301
script: auto
However, now my problem is this:
Error: Cannot find module '/workspace/server.js'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:889:15)
at Function.Module._load (internal/modules/cjs/loader.js:745:27)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)
at internal/main/run_main_module.js:17:47 {
Which is essentially the same issue as above.
EDIT 2
I'm wondering if it is because now that I am running a production build I need to add the correct handlers. But as mentioned above, I have tried dozens of combinations of various answers on the internet to no avail.
Here's another standard app.yaml I tried:
runtime: nodejs16
# This sample incurs costs to run on the App Engine flexible environment.
# The settings below are to reduce costs during testing and are not appropriate
# for production use. For more information, see:
# https://cloud.google.com/appengine/docs/flexible/nodejs/configuring-your-app-with-app-yaml
manual_scaling:
instances: 2
handlers:
- url: /.*
secure: always
redirect_http_response_code: 301
script: auto
- url: /
static_files: build/index.html
upload: build/index.html
- url: /
static_dir: build
Is it something to do with one of my packages? If so, which one? There aren't any specials ones that I know of...
I also added this to my package.json, but it didn't do anything either:
"start": "PORT=8080 react-scripts start",
EDIT 3
I have tried these posts: Error: Cannot find module '/workspace/server.js'
could not find module workspace/server.js
Error: Cannot find module '/workspace/server.js' upon node js deploy on google app engine
https://medium.com/#calebmackdaven/so-you-want-to-start-using-google-cloud-ce9054e84fa8
This app.yaml built on cloud build; however, I now get a new error "The requested URL / was not found on this server.":
runtime: nodejs16
# This sample incurs costs to run on the App Engine flexible environment.
# The settings below are to reduce costs during testing and are not appropriate
# for production use. For more information, see:
# https://cloud.google.com/appengine/docs/flexible/nodejs/configuring-your-app-with-app-yaml
manual_scaling:
instances: 2
handlers:
- url: /static/js/(.*)
static_files: build/static/js/\1
upload: build/static/js/(.*)
- url: /static/css/(.*)
static_files: build/static/css/\1
upload: build/static/css/(.*)
- url: /static/media/(.*)
static_files: build/static/media/\1
upload: build/static/media/(.*)
- url: /(.*\.(json|ico))$
static_files: build/\1
upload: build/.*\.(json|ico)$
- url: /
static_files: build/index.html
upload: build/index.html
- url: /.*
static_files: build/index.html
upload: build/index.html
- url: /.*
secure: always
redirect_http_response_code: 301
script: auto
My logs say:
Static file referenced by handler not found: build/index.html
EDIT 4
I tried a different cloudbuild.yaml:
steps:
- name: node
entrypoint: npm
args: ['install']
- name: node
entrypoint: npm
args: ['run', 'build']
- name: 'bash'
args: ['gcloud app deploy']
However, now the error I get in my npm run build stage is :
Error message "error:0308010C:digital envelope routines::unsupported"
I checked out this answer, so I tried a different node version 14 and still got the same issue. I tried the following, but got the same error:
"start": "react-scripts --openssl-legacy-provider start"
Here's my error:
Step #1: Error: error:0308010C:digital envelope routines::unsupported
Step #1: at new Hash (node:internal/crypto/hash:67:19)
Step #1: at Object.createHash (node:crypto:130:10)
Step #1: at module.exports (/workspace/node_modules/webpack/lib/util/createHash.js:135:53)
Step #1: at NormalModule._initBuildHash (/workspace/node_modules/webpack/lib/NormalModule.js:417:16)
Step #1: at /workspace/node_modules/webpack/lib/NormalModule.js:452:10
Step #1: at /workspace/node_modules/webpack/lib/NormalModule.js:323:13
Step #1: at /workspace/node_modules/loader-runner/lib/LoaderRunner.js:367:11
Step #1: at /workspace/node_modules/loader-runner/lib/LoaderRunner.js:233:18
Step #1: at context.callback (/workspace/node_modules/loader-runner/lib/LoaderRunner.js:111:13)
Step #1: at /workspace/node_modules/babel-loader/lib/index.js:59:103 {
Step #1: opensslErrorStack: [ 'error:03000086:digital envelope routines::initialization error' ],
Step #1: library: 'digital envelope routines',
Step #1: reason: 'unsupported',
Step #1: code: 'ERR_OSSL_EVP_UNSUPPORTED'
Step #1: }
Step #1:
Step #1: Node.js v17.1.0
Finished Step #1
ERROR
ERROR: build step 1 "node" failed: step exited with non-zero status: 1
Why is it attempting to use Node.js v17.1.0 when I specified v16 in the app.yaml?
EDIT 5
I tried specifying Node v16.13.0:
steps:
- name: node: "16.13.0"
entrypoint: npm
args: ['install']
- name: node: "16.13.0"
entrypoint: npm
args: ['run', 'build']
- name: "gcr.io/cloud-builders/gcloud"
entrypoint: bash
args: [gcloud app deploy]
However, it didn't build at all:
Your build failed to run: failed unmarshalling build config cloudbuild.yaml: yaml: line 2: mapping values are not allowed in this context
So I tried adding a substitution:
---
steps:
-
args:
- install
entrypoint: npm
name: "node= $_NODE_VERSION"
-
args:
- run
- build
entrypoint: npm
name: "node= $_NODE_VERSION"
-
args:
- "gcloud app deploy"
entrypoint: bash
name: gcr.io/cloud-builders/gcloud
substitutions:
$_NODE_VERSION: v16.13.0
But I got:
Your build failed to run: generic::invalid_argument: invalid build: invalid build step name "node= ": could not parse reference: node=
And if I try node: "16.13.0", the error is similar:
Your build failed to run: failed unmarshalling build config cloudbuild.yaml: yaml: line 2: mapping values are not allowed in this context
Okay, that error was just due to spacing. I tried updating my cloudbuild.yaml as follows:
steps:
- name: node:16.13.0
entrypoint: npm
args: ['install']
- name: node:16.13.0
entrypoint: npm
args: ['run', 'build']
- name: "gcr.io/cloud-builders/gcloud"
entrypoint: bash
args: [gcloud app deploy]
Now I am able to build. However, now I get an issue with the gcloud command:
Step #2: bash: gcloud app deploy: No such file or directory
EDIT 6
I'm going in circles. My new error is the original one:
Error: Not Found
The requested URL / was not found on this server.
But at least I got the cloud build to build. So it must be my app.yaml or cloudbuild.yaml.
steps:
- name: node:16.13.0
entrypoint: npm
args: ['install']
- name: node:16.13.0
entrypoint: npm
args: ['run', 'build']
- name: "gcr.io/cloud-builders/gcloud"
entrypoint: bash
args:
- "-c"
- |
cp app.yaml ./build
cd build
gcloud app deploy
Step #2: bash: gcloud app deploy: No such file or directory
Solution
I finally got it working! I was changing directory to build, but shouldn't have been. So here are my working cloudbuild.yaml and app.yaml files:
cloudbuild.yaml
steps:
- name: node:16.13.0
entrypoint: npm
args: ['install']
- name: node:16.13.0
entrypoint: npm
args: ['run', 'build']
- name: "gcr.io/cloud-builders/gcloud"
entrypoint: bash
args:
- "-c"
- |
cp app.yaml ./build
gcloud app deploy
app.yaml
---
handlers:
-
secure: always
static_dir: build/static
url: /static
-
secure: always
static_files: build/\1
upload: build/.*\.(json|ico|js)$
url: /(.*\.(json|ico|js))$
-
secure: always
static_files: build/index.html
upload: build/index.html
url: .*
manual_scaling:
instances: 1
runtime: nodejs16
Solution
I finally got it working! I was changing directory to build, but shouldn't have been. So here are my working cloudbuild.yaml and app.yaml files:
cloudbuild.yaml
steps:
- name: node:16.13.0
entrypoint: npm
args: ['install']
- name: node:16.13.0
entrypoint: npm
args: ['run', 'build']
- name: "gcr.io/cloud-builders/gcloud"
entrypoint: bash
args:
- "-c"
- |
cp app.yaml ./build #not sure if this is necessary
gcloud app deploy
app.yaml
---
handlers:
-
secure: always
static_dir: build/static
url: /static
-
secure: always
static_files: build/\1
upload: build/.*\.(json|ico|js)$
url: /(.*\.(json|ico|js))$
-
secure: always
static_files: build/index.html
upload: build/index.html
url: .*
manual_scaling:
instances: 1
runtime: nodejs16

How to solve GAE deployment error - Error Response [9]

I am trying to deploy my react app to GAE,
I use cloud build
At the end I am receiving this error:
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build 8a36fcd1-285e-49bc-a13f-b6aef9ae5c8f status: FAILURE
npm ERR! bindings not accessible from watchpack-chokidar2:fsevents
this is my app.yaml:
runtime: nodejs14
service: ***********
basic_scaling:
max_instances: 10
idle_timeout: 5m
resources:
cpu: 1
memory_gb: 0.5
disk_size_gb: 10
handlers:
- url: /.*
script: auto
secure: always
redirect_http_response_code: 301
this is the cloudBuild.yaml
steps:
- name: node
entrypoint: npm
args: ['install']
- name: node
entrypoint: npm
args: ['run','build']
- name: 'gcr.io/cloud-builders/gcloud'
args: ['app', 'deploy']
timeout: 1200s
timeout: 1200s
Please review your Jason file over the Script part:
"scripts": { "start": "node server",*****Here should be also include the entry point, where this server is pointing.
I suggest to update this value with something like:
"scripts": { "start": "react-scripts start",*** adding this value , we were able to replicate your case without any issue.
I also recommend to review this documents to get more details about and be able to compare your settings with the ones inside on this documentation:
app-engine-react-demo[1]
package.json[2]
You can take a look to this example :
Deploying a React App to Google’s App Engine[3]
See next stackoverflow post which also mentioned similar cases:
app-engine[4]
And finally review this document where first you need to run npm build command in order to upload the files and then run the deploy.
setup.sh[5]
[1]https://github.com/mikesparr/app-engine-react-demo
[2]https://github.com/mikesparr/app-engine-react-demo/blob/master/package.json
[3]https://blog.doit-intl.com/deploying-a-react-app-to-googles-app-engine-6efa8f4732c7
[4]Deploy create-react-app on Google App Engine
[5]https://github.com/mikesparr/app-engine-react-demo/blob/master/setup.sh

CodePipeline + Beanstalk Connection Refused React + Express Apps

I'll lay this out below in detail:
Goal
A React front-end game that uses socket.io to transfer game state data between players on the Express server.
Setup
After finding that it would be impossible/a real work-around to have both React/Express on an Amplify instance, I decided to move to CodePipeline + CodeDeploy to Beanstalk.
I have the Source coming from GitHub, a CodeBuild step, and then CodeDeploy. The CodeBuild should basically just compile based on my buildspec.yml below:
version: 0.2
frontend:
phases:
preBuild:
commands:
- npm ci
build:
commands:
- npm run build
artifacts:
baseDirectory: build
files:
- '**/*'
cache:
paths:
Although this is calling npm run build, which I'm not sure A) The CodeBuild/CodeDeploy will try to do with node start or B) If this will kick off my react-scripts build in package.json:
"scripts": {
"webpack": "webpack --production",
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
I'm assuming I want it to build the webpack bundle in CodeBuild, then CodeDeploy will call the proper run command, but it seems black boxed? Is there another config file?
Runtime Errors
I get Connection Refused, although my server.js file defaults to localhost and 8080:
var hostname = process.env.IP || 'localhost';
var port = process.env.PORT || 8080;
Web.std.out.log:
Sep 20 18:10:10 ip-172-31-41-5 web: > MyApp#0.1.0 start /var/app/current
Sep 20 18:10:10 ip-172-31-41-5 web: > react-scripts start
Sep 20 18:10:12 ip-172-31-41-5 web: #033[34mℹ#033[39m #033[90m「wds」#033[39m: Project is running at http://172.31.41.5/
Sep 20 18:10:12 ip-172-31-41-5 web: #033[34mℹ#033[39m #033[90m「wds」#033[39m: webpack output is served from
Sep 20 18:10:12 ip-172-31-41-5 web: #033[34mℹ#033[39m #033[90m「wds」#033[39m: Content not from webpack is served from /var/app/current/public
Sep 20 18:10:12 ip-172-31-41-5 web: #033[34mℹ#033[39m #033[90m「wds」#033[39m: 404s will fallback to /
Sep 20 18:10:12 ip-172-31-41-5 web: Starting the development server...
Ngnix error.log: 2020/09/18 03:07:30 [error] 4436#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.31.40.133, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "172.31.41.5"
Lot of moving parts so I'm sure there are several missteps here, not sure which is which.
Based on the comments.
By default EB will look for package.json to run your application. If it is missing, and will use use script option in your package.json to start your application. For example, if you start sample node.js application that EB provides, the file is:
package.json
{
"name": "Elastic-Beanstalk-Sample-App",
"version": "0.0.1",
"private": true,
"dependencies": {},
"scripts": {
"start": "node app.js"
}
}
If you don't provide package.json nor Procfile (other option, see below) EB will expect app.js or server.js to be present to start your application.
If you don't want to use package.json, you can also use Procfile to tell EB how to start your application.
When you don't provide a Procfile, Elastic Beanstalk runs npm start if you provide a package.json file. If you don't provide that either, Elastic Beanstalk looks for the file app.js or server.js, in this order, and runs it.
Example in your case could be
web: serve -s build

Can't create .env file from cloudbuild.yaml anymore(App Engine - Standard environment)

Earlier(two months ago), I used to create .env file in App Engine through cloudbuild.yaml. Here is the file:
steps:
- name: 'gcr.io/cloud-builders/npm'
args: ['install']
- name: 'gcr.io/cloud-builders/npm'
args: ['run', 'create-env']
env:
- 'TYPE=${_TYPE}'
- name: 'gcr.io/cloud-builders/gcloud'
args: ['app', 'deploy', 'app.yaml']
timeout: "1600s"
My package.json:
"scripts": {
"start": "node app.js",
"create-env": "printenv > .env"
}
This yaml file was working as expected(creating .env file run time) earlier. I tried same file today and it did not create .env file. To cross check, I tried different name instead of .env and that worked. What's going on here? Is it no more possible?

App Engine app.yaml handlers for built React app

I'm trying to deploy production build of React app (created using create-react-app) to gcloud app engine flexible enviroment.
After running npm run build, the build folder has been created:
This is my app.yaml:
# [START runtime]
runtime: nodejs
env: flex
# [END runtime]
# [START handlers]
handlers:
- url: /
static_files: build/index.html
upload: build/index.html
- url: /
static_dir: build
# [END handlers]
When deployed to App Engine, the version configuration shows:
runtime: nodejs
api_version: '1.0'
env: flexible
threadsafe: true
handlers:
- url: /
application_readable: false
static_files: build/index.html
require_matching_file: false
upload: build/index.html
- url: '/(.*)'
application_readable: false
static_files: "build/\\1"
require_matching_file: false
upload: 'build/.*'
automatic_scaling:
min_num_instances: 2
max_num_instances: 20
cpu_utilization:
target_utilization: 0.5
The development app is being served instead of the production version from the build folder. What am I doing wrong?
Modify the scripts in package.json to have GAE use serve to serve the build dir instead of the root directory.
Upon deployment, GAE will run npm start to begin serving your application. By changing what the start script maps to, you can have the build directory served as desired. You will have to use npm run local when doing local development with this setup, as you are changing what npm start is doing.
This process requires you to run npm run build before deploying to GAE. This will serve the build directory for you, but it does not automatically run the build process for you, so that must still be done to serve your latest code in /src.
Source:
https://github.com/facebook/create-react-app/issues/2077
Code:
"scripts": {
"start": "serve -s build",
"prestart": "npm install -g serve",
"local": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test --env=jsdom",
"eject": "react-scripts eject"
}
GAE will run npm start to run your react app. Therefore you need to configure your package.json to tell GAE serve your build folder:
"scripts": {
"start": "serve -s build",
...
},
Also need to configure your .yaml file accordingly:
handlers:
- url: /
static_files: build/index.html
upload: build/index.html
- url: /(.*)$
static_files: build/\1
upload: build/(.*)
When deploying a GAE app/service the directory containing the app/service's app.yaml file being deployed is considered the app/service's top level directory and the content of that directory is what's being deployed. Which, in your case, matches what I suspect you're calling the development version.
If you want your build directory to be your app/service's top dir then you need to:
create an app.yaml file in the build dir (manually or instruct your build process to do it)
Note: you need to adjust the paths in that file, since there won't be a build directory in the app's directory anymore. Or try to create inside it a build -> . symlink to itself, possibly allowing the use of the existing app.yaml content as-is, without adjusting the paths (not 100% certain this will work, though).
deploy that build/app.yaml file, not the one from the top (bsn) dir.

Resources