So I'm trying to run my go app with google's app engine. When I run goapp server I get this error:
go-app-builder: Failed parsing input: app file model.go conflicts with same file imported from GOPATH
This is my project layout:
.
├── model
│ └── model.go
├── reqres
│ └── reqres.go
├── app.yaml
├── service.go
├── main.go
└── transport.go
If I run it without app engine I don't any get errors and the app runs fine.
According to my experience you get this error because your project folder is also under your GOPATH. "goapp" kind of clone your project folder and builds it against the go environment GOPATH and GOROOT... Doing so it finds duplicated symbols for all package that you have been declared under your project.
Here is the explanation in go appengine documentation
If you include your package sources in GOPATH, you must be careful not to place the source code at or below any directories in your App Engine project that contain app.yaml files. If that happens, a package could be loaded twice, once for the path relative to a module's directory, and once for the fully-qualified path. This can cause subtle problems, so the Go SDK scans your project and your GOPATH, detects this case, and reports it as an error.
Under the same link you will find some advises by google for your project structure and one of them is (your project break that guideline):
Do not include any subdirectories in a module's directory.
If you want a repository with your application definition and go packages I encourage you to adopt the folliwing structure:
projectRoot
|- modules
| |- myModule1
| | |- init.go // router pattern to handler
| | |- myModule1.yaml // configuration for the module
| |- myModule2
| |- init.go // router pattern to handler
| |- myModule2.yaml // configuration for the module
|
|- pkg
| |- myModule1
| | |- *.go // sources, subfolders(packages)
| | // with handlers and business code
| |- myModule2
| | |- *.go // sources, subfolders(packages)
// with handlers and business code
This structure is convinient and improves debugging experience as explained in the article debugging Go appengine module with visual studio code
Related
I need to some help to identify the correct solution that would help me create a seamless documentation for my friends.
We have several repositories in which a doc folder with several .MD files are going to be placed
Repo1
|- Readme.MD
|-docs
|- Installation.MD
|- Usage.MD
Repo2
|- Readme.MD
|-docs
|- Installation.MD
|- Usage.MD
Repo3
|- Readme.MD
|-docs
|- Installation.MD
|- Usage.MD
We would like to use something like vuepress to generate a static site.
If there is any tool/framework which can easily solve this issue. I would be grateful
Thanks a lot for any response,we will definitely put below what we have done
You can accomplish this using MkDocs as your static site generator and the multirepo plugin. Below are the steps to get it all setup. I assume you have Python installed and you created a Python venv.
python -m pip install git+https://github.com/jdoiro3/mkdocs-multirepo-plugin
mkdocs new my-project
cd my-project
Add the below to your newly created mkdocs.yml. This will configure the plugin.
plugins:
- multirepo:
repos:
- section: Repo1
import_url: {Repo1 url}
- section: Repo2
import_url: {Repo2 url}
- section: Repo3
import_url: {Repo3 url}
Now, you can run mkdocs serve or mkdocs build, which will build a static site with all the documentation in one site.
Here is an idea,
Initialize a repository, lets call it TheRepo
Add all depending repos (in our case Repo1, Repo2, Repo3) as git submodules to TheRepo
Now we have everything from Repo1, Repo2 and Repo3 into TheRepo, which I assume might be undesirable for a documentation website.
Create a bash script (using find, grep, rm or similar bash charm) to retain the desired *.md files and remove the undesirable source files. Here is a sample on how-to:
// remove everything else that is not markdown file
find . -type f ! -name '*.md' -delete
Initialize vuepress at TheRepo's root and let vuepress generate cosolidated documentation.
If you do a good job at step 3. you can have a seperate section/classification for each individual dependency Repo in the resulting documentation.
To refresh the contents, simple use git submodule update to update documentation and then chain it with script created in step3 as to automate the refresh process.
git submodule update && ./remove-undesirable-files.sh
I’ve been trying to use Sagemaker to run my custom MXNet training job. In all the examples I’ve seen, the code sample looks like this
estimator = MXNet(‘train.py’, role=role, other_params)
estimator.fit(inputs)
What if my train.py relies on a custom module? Given a directory structure like so
.
├── awesome
│ ├── __init__.py
│ └── lib.py
└── train.py
With my train.py file importing from awesome/lib.py, what’s the best way for me to deploy this job on Sagemaker without going through the hassle of creating a Docker container.
Note: all the code in the custom module is just regular mxnet code, organized across various files and helper methods
You can use the parameters source_dir to point to the code location, and a requirements.txt file to add dependencies. This will avoid to touch docker at all. You can see those parameters in the SDK documentation ("Use third party library"), they are available both for training and deployment. See here an mxnet deployment example with additional dependencies in a requirements.txt
https://github.com/aws-samples/sagemaker-yolov3-detection-server/blob/master/mxnet_detection_serving.ipynb
We currently host a large number of containerized websites in Kubernetes and we are exploring using Google App Engine to host these sites but we cannot figure out how to host a large number (hundreds) of websites in a single Google App Engine account.
At first glance it seemed that every website will be a separate project but then it became clear that there is a soft limit of twenty projects (and we might end up hosting hundreds of sites) so this did not seem to be the correct approach. We then explored using a single project with dispatch.yaml to route between the sites. Dispatch.yaml only allows for ten entries which mean that it will not work. Is there any other approach we are missing? Our dispatch.yaml routing looked like this:
- url: "example2.com/*"
service: my-second-website
None of the options we explored provided a scalable or viable solution. Any help or advice will be highly appreciated.
What you need is one App Engine instance with multiple services (one for each website) and multiple versions in each service (if needed).
e.g:
App Engine instance
├── website01-service
│ ├── website01-version01
│ ├── website01-version02
│ └── website01-version03
│
└── website02-service
├── website02-version01
├── website02-version02
└── website02-version03
This is an example setup from Quickstart for Python 3 in the App Engine Standard Environment documentation.
Download the files from git repository as stated in Download the Hello World app section.
Copy the hello_world sample files in two different directories. e.g. website01 and website02.
You should have something like this:
├── website01
│ ├── app.yaml
│ ├── main.py
│ ├── main_test.py
│ └── requirements.txt
└── website02
├── app.yaml
├── main.py
├── main_test.py
└── requirements.txt
In website01/app.yaml add service: website01 and in website02/app.yaml add service: website02. This will deploy each app in different App Engine service.
In website01/main.py change return 'First website!' and in website02/main.py change return 'Second website!' (This is just to confirm after deployment that 2 different websites are running).
In the /website01 directory execute $ gcloud app deploy --version website01-version01 and in /website02 directory execute $ gcloud app deploy --version website02-version01
After a successful deployment, you should see 2 different versions running in Google Cloud Console > App Engine > Services page.
When clicking on both links new tabs will open and you will see your two different websites running in the same App Engine instance with two different links. The links should appear as following:
website01 -> https://website01-dot-[PROJECT_ID].appspot.com/
website02 -> https://website02-dot-[PROJECT_ID].appspot.com/
Have you tried requesting that your project limit be increased? I think that's the only good way to make this work.
If you attempt to exceed your project limit, the console will prompt you to fill out a request form. This happens when you try to create a project but you have already reached your quota. The form will require you to specify the number of additional projects you need, along with their corresponding email accounts, billing accounts, and intended uses.
https://support.google.com/cloud/answer/6330231?hl=en
You could do this with a single project & single service/app.yaml, if your ok with the url for each website being something like :
www.website1.com/website1/
www.website2.com/website2/
www.website3.com/website3/
etc
Then you could use the handlers in app.yaml to do the routing (along with some light server code to redirect if someone tries to visit www.website1.com/website2/). The biggest issue with doing it this way is every time you'd deploy you would be deploying all 100 of your sites.
How would would you deploy the output of the angular-fullstack Yeoman generator to Azure? The generator produces output like this, i.e. two folders - client and server.
├── client
│ ├── app - All of our app specific components go in here
│ ├── assets - Custom assets: fonts, images, etc…
│ ├── components - Our reusable components, non-specific to to our app
│
├── e2e - Our protractor end to end tests
│
└── server
├── api - Our apps server api
├── auth - For handling authentication with different auth strategies
├── components - Our reusable or app-wide components
├── config - Where we do the bulk of our apps configuration
│ └── local.env.js - Keep our environment variables out of source control
│ └── environment - Configuration specific to the node environment
└── views - Server rendered views
I'd like to deploy this to Microsoft Azure, and ideally using Git deploy. How would I go about that?
Update
Here's what the deployment file structure looks like in Azure.
Here is the short version:
Create a new blank Web App in Azure.
In the Deployment options, select 'Set up continuous deployment'.
Select 'Local Git Repository'.
This will give you a Git Remote for the blank azure Web App to allow you to push to it. Which means you can do this:
git remote add azure [URL for remote repository]
git push azure master
Here is the longer version (it is for a node app - but the same concepts apply).
We recently converted our app engine project into modules as per the structure below. The problem with this new dev workflow is that we have to rebuild the EAR on every change and relaunch the app engine local dev server. This makes us loose 30s to a minute every time we make a change to the code and want to test it.
/commons
-pom.xml
/model
-pom.xml
/webapp //app engine module
-pom.xml
/apis //app engine module
-pom.xml
/ear
-pom.xml
pom.xml //main (parent) project pom
In our previous workflow, with the monolithic app, we could use app engine's hot reload functionality, where modifying code in an IDE (e.g. eclipse) would be picked up automatically.
What do you guys recommend as the best maven config and/or dev workflow in this case? Ideally, a change in any of the modules would not require a full rebuild of the project.
I am using a similar structure with a small difference. The top level directory has war and ear and then they contain their specific pom.xml. I use Eclipse for debugging, and I am able to hot deploy "most of the time" and I am not using Eclipse plugin, which (I understand) is what you want.
Directory Structure
.
|-- pom.xml
|-- README.md
|-- my-ear
| |-- devpid
| |-- pom.xml
| `-- src
| `-- main
| `-- application
| `-- META-INF
`-- my-war
|-- build
| `-- classes
| |-- main
| | |-- java
| | `-- webapp
| `-- test
| `-- java
|-- pom.xml
`-- src
|-- main
| |-- java
| | `-- com
| `-- webapp
| |-- css
| |-- favicon.ico
| |-- index.html
| |-- js
| |-- test.html
| `-- WEB-INF
`-- test
`-- java
Tools
Eclipse Luna without Google App Engine Plugin (or SDK)
Maven 3.2.1
Google App Engine SDK 1.9.6
Dev Workflow
If you already have source code, keep it somewhere else and generate a skeleton using mvn appengine command.
Run the first cut with a simple Hello World using only maven and terminal and mvn appengine:devserver command.
Once done, generate the eclipse project.
Import the eclipse project as a Maven project. It will see the jars via Maven. I won't have written this answer before Luna as it required too many tweaks. In Luna, this works automatically.
The step above will create three projects, top level, ear and war each with pom.xml - It's OK.
In eclipse, provide the output directory as war/target directory. This is the step which makes it possible to hot deploy.
In maven ear/pom.xml, add xArgs to appengine plugin for running in debug mode.
<plugin>
<groupId>com.google.appengine</groupId>
<artifactId>appengine-maven-plugin</artifactId>
<version>${appengine.target.version}</version>
<configuration>
<jvmFlags>
<jvmFlag>-Xdebug</jvmFlag>
<jvmFlag>-Xrunjdwp:transport=dt_socket,address=1044,server=y,suspend=n</jvmFlag>
</jvmFlags>
<disableUpdateCheck>true</disableUpdateCheck>
</configuration>
</plugin>
Notice the suspend=n.
Run the app engine from outside eclipse using mvn appengine:devserver from the ear directory. I use this command:
mvn appengine:devserver > ~/.logs/.appengine.devserver.logs & echo $! > devpid
Let's call this Terminal 1.
An advantage of this method is that your console is not captured by Eclipse, so you are free to use a tool of your choice to view it, like multitail etc. I use this simple tail command:
tail -f ~/.logs/.appengine.devserver.logs | sed 's/INFO/^[[0;34m&^[[0m/g;s/ERROR/^[[0;31m&^[[0m/g;s/WARN\|WARNING/^[[0;35m&^[[0m/g;s/SEVERE\|FATAL/^[[0;31;47m&^[[0m/g'
The above is a difficult to type command. Every instance of ^[ is actually Ctrl+V Esc - it is worth the effort of typing it once. But this is of course subjective and up to you.
In Eclipse, create a Debug Profile for your project under Remote Java Application - select the war project and socket attach options. This step is available on the internet at many places, here is an image nevertheless
Open another terminal, Terminal 2 in the war directory and keep it open in order to run mvn compile install when you need to.
You are good to go. You should be able to integrate your source code by just pasting it at the right place. You should also be able to use standard debugging techniques. Eclipse will compile at the right location and devserver will detect it all right. If Eclipse throws a warning, ignore it.
This works most of the time. Sometimes, you save something that breaks compilation of the whole project, or change a function name being called from a pre compiled class or simply change web.xml which is loaded at start up. Of course then hot deploy will not work.
In such a case, stop your remove debug from within eclipse, complete your tasks, run mvn compile install from Terminal 2. Devserver will autodetect.
Mostly, I hardly need to touch the tail running in Terminal 1. Devserver does not tend to need restart.
Unless I am changing web.xml or refactoring, I do not need to run mvn compile install from outside.
My reason for giving list of windows (Eclipse, Terminal 1 and Terminal 2) is just to show that Alt+Tab is actually faster than Shift+F7 from within eclipse. It is subjective and of course up to you.