The canonical example given for appengine-go module uploads implies that all modules within a given app must share the same codebase.
e.g. appcfg update app.yaml mobile-frontend.yaml my-module.yaml
In this case, all Go files in the app root directory are processed identically for each module (and therefore share the same code).
Is there any way to specify a separate root directory for each module e.g. as a sub-directory of the default app root?
TIA
As a new SO user, I was restricted by the 8-hour limit for answering my own question. Therefore I posted it initially as a comment.
So here is the link in question for anyone who runs into this problem:
https://groups.google.com/d/msg/google-appengine-go/T90jtkZ83s4/MizIcnjH4QIJ
As an aside, the new modules paradigm is vital for GAE projects. Not only does it allow for different code bases for each module but also different runtimes!!!
This means that, theoretically at least, you can leverage the best features of each language within the same app. Nice!
Finally got my app working in development and on App Engine. Thanks. To summarize the thread:
My app directory looks like this:
$ ls *
README.md dispatch.yaml
frontend:
app.yaml index.html server.go
images:
images.yaml server.go
Then to run in development:
$ goapp serve dispatch.yaml */*.yaml
and to deploy:
$ appcfg.py update_dispatch .
$ goapp deploy */*.yaml
Source code.
Related
I've recently tried getting into the whole Node ecosystem and am trying to set up some continuous deployment for my app to AWS Amplify.
For background, my project structure looks like this:
project
public
index.html
src
App.tsx/App.js
package.json
As far as I know, this is basically what create-react-app gave me to start with, and I didn't change the file structure.
For most of my time working on the app, I've been able to go to the base project directory and use
npm start
to launch the app. This will bring me to the App.tsx/js homepage.
However, when I hosted this to AWS Amplify via GitHub, the default build settings actually point to the public directory, so the published site is actually point to index.html (which is basically just an empty placeholder).
While debugging, I ran
npm build
in my root project directory, which constructed a build folder, so now the overall project looks like this:
project
build
index.html
public
index.html
src
App.tsx/App.js
package.json
Now, running
npm start
will bring me to the index.html from the build directory, instead of App.js/tsx as it used to.
The AWS setup says that it will run
npm build
so I assume that what I've done on my local machine is mirroring what the AWS server is doing behind the scenes and explains why AWS is serving the empty index.html.
I've read a few articles and watched some videos about hosting a create-react-app on AWS, and in every version, it looks like AWS will serve the App.tsx/App.js right out of the box, rather than build/index.html, and I've not been able to find a good guide on how to configure this behavior. Quite frankly, there is an overwhelming number of similar-but-slightly-different answers for questions like this, which use different combinations of package managers, packages, hosting services, all on different release versions, with different setups, and it's very difficult for me to tell which ones apply to my scenario.
So I'm hoping someone can help straighten some of this out for me, or point me towards a good resource for learning more about this type of thing. Particularly interested in learning the right way to do these things, rather than a quick hack around whatever my particular issue is.
Some specific questions...
Is deploying things from a /build folder standard convention?
Why does create-react-app create a separate /src/app.tsx and /public/index.html that seem to be competing with one another as the app's "homepage"?
Why does the behavior of
npm start
change depending on whether
npm build
has been run?
Is the correct fix here to just insert my App.tsx component into the index.html? This doesn't seem hard, but doesn't seem right either
I have seen a lot of answers discussing tweaks to webpack.config.js to solve issues like this one. My project does have webpack installed, but as best I can tell, there is no webpack.config.js anywhere. Am I expected to create this file, or should it exist already? In either case, in which directory is it supposed to live? I've seen a couple answers saying it should be in /node_modules/webpack/, but also some saying it needs to live in the same directory as package.json
Things I've tried already: Spent a bunch of time reading through other StackOverflows and watching a few videos, but as outlined above, I'm finding it difficult to tell which could apply to my situation and which are unrelated, given the huge number of unique combinations of build/packages/platforms/versions. Also spent some time monkeying around with file structure/moving code around, but not very productively.
Eventually found my issue. In the production built version of my app (aka, /build), the bundled script created by webpack was failing in the browser because exports was undefined, so index.html was being served in its vanilla state, rather than with the TSX/JSX content. I changed the "module" property in tsconfig.json from commonjs to es6 and this fixed most of the problems.
Also of note is that the reason I couldn't find my webpack.config.js is that I had hidden ALL js files in my project, so VSCode wasn't finding it. I swapped to the suggestion from this blogpost to hide only js files with a matching TS file.
For general learning about how create-react-app works, I eventually found this page, which I found helpful:
https://blog.logrocket.com/getting-started-with-create-react-app-d93147444a27/
For the basic create-react-app
npm start
Is a short command for react-scripts start that sets up the development environment and starts your development server usually localhost:3000
npm build
After you are done developing, this command short for react-scripts build correctly bundles your app for production and optimizes the build for the best performance.
The files generated in the build folder are solely the files you serve to the public folder accessible by the public URL.
In short the files in the build folder should be copied to the public folder
AWS Amplify
Provides a CI/CD process where you don't have to set all this up by yourself, as long as you have a well-configured package.json file.
There are so many methods to deploy your react app to a production server but using AWS Amplify this link might help you out: https://youtu.be/kKwyKQ8Jxd8
More on create-react-app deployment: https://create-react-app.dev/docs/deployment/
Recently tried to update my Gaelyk project (yes, it's old, but it works well and I still use it), but Google App Engine will no longer accept the update. The error message returned is "Deployments using appcfg are no longer supported. See https://cloud.google.com/appengine/docs/deprecations". The thing is, I never used appcfg to deploy my application; I used Gaelyk and Gradle. But obviously Gaelyk must have used appcfg under the covers.
I did download the replacement Google Cloud SDK, but this new tool is not similar at all to how Gaelyk and Gradle worked. Is there anything I can do to get Gaelyk to work anymore? Or is Gaelyk just dead and I need to rewrite my application (like in Node.js or something instead of Groovy).
This will be hard, however I will try to help you as possible. I think you may try to migrate it somehow to app.yaml configuration of GAE.
I am not sure what plugins are used in the project. From Gaelyk temple project I can see that it's using appengine-geb which, according to the documentation, behind the scenes, is using gradle-appengine-plugin (there is wrong link on this doc, but proper is bellow).
On the github of gradle-appengine-plugin I have found following.
There is a note:
NOTE: All App Engine users are encouraged to transition to the new
gradle plugin for their projects.
And in FAQ part there is following information:
How do I deploy with gcloud?
If you're using gcloud to deploy your application, the newest version of app deploy > doesn't support war
directories, you will need to provide it with an app.yaml OR you can
use the appengineStage task to create a directory that is deployable
in /build/staged-app
$ ./gradlew appengineStage
$ gcloud app deploy build/staged-app/app.yaml --project [app id]
--version [some version]
NOTES:
You must explicitly define all config files your want to upload
(cron.yaml, etc)
This does not work with EAR formatted projects.
I think the best option will be to migrate to new appenine plugin or if not possible try to implement is with gcloud app deploy command crating the config files manually (at least app.yaml). And for this migration I can provide you this document.
I hope you will manage somehow...
I can confirm that Serge's answer on the Gaelyk Groups site works; the same procedure that he figured out also worked for me. To summarize:
Run gradlew appengineRun as run previously with Gaelyk.
Copy all jar files inside the build\exploded-app\WEB-INF\lib folder into a \src\main\webapp\web-inf\lib folder (for me the new lib folder did not exist previously).
To deploy, use the new required gcloud tool, and instead of running gradlew appengineUpdate (which fails now), instead run
gcloud app deploy appengine-web.xml where that XML file can be found in your webapp/WEB-INF directory. I navigated to that directory to run the gcloud command, but you can use a relative path there if your working directory is elsewhere. (There are a number of optional flags associated with the gcloud app deploy command, but I didn't need any of them.)
Serge needed to use these instructions to convert datastore-indexes.xml to index.yaml and run gcloud app deploy index.yaml, however, I didn't need to do this because I had no datastores.
I'm using Google AppEngine Flexible with python environment. Right now I have two services: default and worker that share the same codebase, configured by app.yaml and worker.yaml. Now I need to install native C++ library, so I had to switch to Custom runtime and added Dockerfile.
Here is the Dockerfile generated by gcloud beta app gen-config --custom command
FROM gcr.io/google-appengine/python
LABEL python_version=python3.6
RUN virtualenv --no-download /env -p python3.6
# Set virtualenv environment variables. This is equivalent to running
# source /env/bin/activate
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH
ADD requirements.txt /app/
RUN pip install -r requirements.txt
ADD . /app/
CMD exec gunicorn --workers=3 --threads=3 --bind=:$PORT aces.wsgi
Previously my app.yaml and worker.yaml each had it's own entrypoint: config that specified the command needed to be run to start the service.
So, my question is how can I use two different commands to start the services?
EDIT 1
So far I was able to solve this by rewriting CMD line in dockerfile for each deploy of each service. However, I'm not quite satisfied with this solution.
gcloud app deploy command has --image-url flag that allows to set image url from GCR. I haven't researched that yet, but it seems that I can just upload images to GCR and use the urls since don't change that often
Yes, as you mentioned, I think using the --image-url flag, is a good option here.
Specify a custom runtime.
Build the image locally, tag it, and push it to Google Container Registry (GCR)
then, deploy your service, specifying a custom service file, and specifying the remote image on GCR using the --image-url option.
Here's an example that accomplishes different entrypoints in 2 services that share the same code:
...this is assuming that the "flex" and not "standard" app engine offering is being used.
lets say you have a: project called my-proj
with a default service that is not important
and a second service called queue-processor which is using much of the same code from the same directory.
Create a separate dockerfile for it called QueueProcessorDockerfile
and a separate app.yaml called queue-processor-app.yaml to tell google app engine what i want to happen.
QueueProcessorDockerfile
FROM node:10
# Create app directory
WORKDIR /usr/src/app
COPY package.json ./
COPY yarn.lock ./
RUN npm install -g yarn
RUN yarn
# Bundle app source
COPY . .
CMD [ "yarn", "process-queue" ]
*of course i have a "process-queue" script in my package.json
queue-processor-app.yaml
runtime: custom
env: flex
... other stuff...
...
build and tag the docker image
Check out googles guide here -> https://cloud.google.com/container-registry/docs/pushing-and-pulling
docker build -t eu.gcr.io/my-proj/queue-processor -f QueueProcessorDockerfile .
push it to GCR
docker push eu.gcr.io/my-proj/queue-processor
deploy the service, specifying which yaml config file google should use, as well as the image url you have pushed
gcloud app deploy queue-processor-app.yaml --image-url eu.gcr.io/my-proj/queue-processor
Since the Dockerfile name cannot be changed, the only way to not have to modify the Dockerfile would be to store each service in its own, separate directory. Clean separation, each service has its own Dockerfile and/or startup configuration.
But this raises a question: how to deal with the code shared by multiple services? Using symlinks (which works great for sharing code across standard env services) doesn't work for the flexible env services, see Sharing code between flexible environment modules in a GAE project.
I see a few possible approaches, none really ideal, but maybe more appealing than what you currently have:
hard-link each and every shared source code file (since hardlinking directories is not possible). A bit tedious and error-prone, but you only have to do that once per file
package and publish your shared code as an external library, added to the requirements.txt file of each service using it
split the shared code in a separate repository and have a copy of that repository in each service using it (maybe as a git submodule if using git?). You just need to ensure at the service deployment time that the shared repository is pulled at the proper version - can be quite reliably done through automation. A bit more complicated if you have uncommited changes in this repo - you'd have to patch the same changes in all services.
have multiple copies of the Dockerfiles with different names which you simply copy over instead of always editing the same file. Symlinking instead of copying might work as well, since the symlink doesn't need to be followed outside of the service directory, if it's just replicated as a symlink it'll work.
So i had a very similar issue with my Java applications. We were looking to migrate from Heroku to GAE and were attempting to simulate the Heroku Procfile with GAE services. Effectively what we did was to create separate directories in our application src/main/appengine/web and src/main/appengine/worker where each directory conainted the app.yaml and Dockerfile specific to the process. Then using the mvn appengine:deploy capabilities, we specified the -Dapp.stage.dockerDirectory and -Dapp.stage.appEngineDirecory respectively for each service we wanted to deploy. Then using just some parameters we were able to basically script out parallel deployments of each service from the same code base. Not sure if this works in your situation, but it was very useful for us: Here are the two example commands in their entirety:
Web Process:
mvn appengine:deploy -Dapp.stage.dockerDirectory=src/main/appengine/web -Dapp.stage.appEngineDirectory=src/main/appengine/web -Dapp.stage.stagingDirectory=target/appengine-web -Dapp.deploy.projectId=${project-id} -Dapp.deploy.version=${project-version}
Worker Process:
mvn appengine:deploy -Dapp.stage.dockerDirectory=src/main/appengine/worker -Dapp.stage.appEngineDirectory=src/main/appengine/worker -Dapp.stage.stagingDirectory=target/appengine-worker -Dapp.deploy.projectId=${project-id} -Dapp.deploy.version=${project-version}
Although I started development for Google App Engine using Endpoints a while ago, I hadn't noticed this - Google's ref. page for Project structure says this:
Your development file hierarchy should look like this:
MyDir/
[pom.xml]
[build.gradle]
[index.yaml]
[cron.yaml]
[dispatch.yaml]
src/main/
appengine/
app.yaml
docker/
[Dockerfile]
java/
com.example.mycode/
MyCode.java
webapp/
[index.html]
[jsp.jsp]
WEB-INF/
[web.xml]
You'll need to define an app.yaml file that looks like this:
...
Note that the app.yaml is deemed compulsory as per the docs. In my case, I spawned a backend module(through the Wizard) in Android Studio that builds on Gradle. I have been able to build and deploy this module on GAE successfully but now I needed to switch from automatic scaling to basic/manual scaling, I found this to be done through app.yaml file.
Here is the thing: I don't have an app.yaml in place and it works fine. Where is then the config info. that GAE requires to deploy the App.
Specifically,
app.yaml specifies the environment - Java. But, I found the java plugin in build.gradle for that. Aren't 2 config places for the same thing confusing?
Is it possible to ditch app.yaml entirely for equivalent config. in build.gradle?
Why is Google claiming app.yaml to be compulsory when I am able to do without it?
The App Engine Java runtime uses its own configuration schema in XML, while others are YAML.
To set the scaling elements, follow the official reference.
I am using the Go runtime of Google App Engine and have two modules. I would like to share HTML templates between them but don't the best way.
My modules are organised as below:
src/github.com/myproject/moduleone/app.yaml
src/github.com/myproject/moduleone/templates/base.html
src/github.com/myproject/moduleone/templates/homeone.html
src/github.com/myproject/moduletwo/app.yaml
src/github.com/myproject/moduletwo/templates/base.html
src/github.com/myproject/moduletwo/templates/hometwo.html
In my situation base.html is the same for moduleone and moduletwo. How can I share it between both modules without having to duplicate the file as is done now?
I would like to put base.html in a directory called src/github.com/myproject/templates but I believe neither moduleone or moduletwo would be able to access the file as it's not in the same or child directory of the module app.yaml files. Is my only option to symlink the base.html file between each module's template directory?
GAE regards each module as a standalone application (each will run in its own GAE instance). No uploadable artifacts are shared at GAE level between the modules, each such artifact needs to be separately uploaded in each module using it.
While other approaches are technically possible (as other mentioned) symlinking the files is IMHO the simplest solution to avoid code duplication in your own repo.
You could copy the templates to each module as part of a build step and gitignore the copied files.