I'm trying to deploy a python app to App Engine.
I've already deploy this app before but for some reason now it upload the files but then write Updating service [default].../ and never finishes.
I'm using Windows 10 and all I can see after few minutes it is running is ERROR: (gcloud.app.deploy).
The command I run is gcloud app deploy --project my-proj -v 1 --quiet server/app.yaml
I have the server side app in server sub directory.
Any ideas how to diagnose this problem?
UPDATE1
I've run the command with debug flag and I see this come every 5 seconds:
Updating service [default].../DEBUG: Received operation: [apps/my-app/operations/b0647556-93af-4fd8-b0a7-f4ae43393e1e]
DEBUG: Operation [apps/my-app/operations/b0647556-93af-4fd8-b0a7-f4ae43393e1e] not complete. Waiting 5s.
Updating service [default]...-DEBUG: Operation [apps/my-app/operations/b0647556-93af-4fd8-b0a7-f4ae43393e1e] not complete. Waiting 5s.
Updating service [default]...\DEBUG: Operation [apps/my-app/operations/b0647556-93af-4fd8-b0a7-f4ae43393e1e] not complete. Waiting 5s.
Updating service [default].../DEBUG: Operation [apps/my-app/operations/b0647556-93af-4fd8-b0a7-f4ae43393e1e] not complete. Waiting 5s.
DEBUG: Operation [apps/my-app/operations/b0647556-93af-4fd8-b0a7-f4ae43393e1e] not complete. Waiting 5s.
Updating service [default]...|DEBUG: Operation [apps/my-app/operations/b0647556-93af-4fd8-b0a7-f4ae43393e1e] not complete. Waiting 5s.
In our case, this was caused by software bug that was causing a 500 error - e.g. I believe during deploy App Engine does a simple health check to verify if the app has been successfully deployed, and if that fails it will loop forever.
I had the same problem, so the solution for me was to deploy the app from console (https://console.cloud.google.com).
I cloned my git repo in the source code folder, after installing project, I run the command: gcloud app deploy --project [google-project-name]
This was happening to me with the v2 endpoints. For backwards compatibility, I had added the old Servlet mapping of _ah/spi in my web.xml. This caused the application to get stuck at Updating service [default]... and never finish. It looks like just having the mapping for _ah/api works for backwards compatibility, so there must be some backend magic that is happening on the appengine side.
If you have a new project in my case it was a flag
I had:
"deploy": "gcloud app deploy --project quantified-self-io --quiet --promote --stop-previous-version --version=v1 --verbosity=info",
The flag --stop-previous-version was causing my issue since there was no prev version running I suppose.
I had exactly the same issue. This is what I found out:
I followed #Daniel Hernandez's suggestion and uploaded the app from a cloned git repository inside cloud shell. It worked.
Then I went ahead and cloned that same repository that was working in cloud shell, but instead I cloned it into another folder in my computer. I deployed the app without any problem.
This made me suspicious so I started looking into what files were different from these two setups. I found out that in my original setup (the one that was failing) I had two extra folders:
env/
lib/
These correspond to the local development folders for virtualenv and the extra libraries. Note that these files were "ignored" from both .gitignore and the skip_files directive in app.yaml.
I deleted these two directories and tried to upload again and it worked.
Updating Gcloud SDK fixed the issue for me: gcloud components update
As per https://stackoverflow.com/a/52428069/219657
There are probably many different things that can cause this problem, so even though there is an already accepted answer in the OP's case I think it's worth documenting other possibilities on this question.
In my case it occurred because of pip's relatively new backtracking dependency resolver which, due to some underconstrained dependencies in my requirements.txt file, was causing pip to download many versions of some packages in order to determine the right dependencies. This can be very slow and was causing the build to time out. Better constraining my dependencies resolved the issue.
You can see what the build is doing in the Cloud Logs Explorer filtered by the "Cloud Build" resource type. If there is a way to stream these logs via the CLI I'm not sure, but would love to be enlightened if so.
Related
I recently added google tag manager on a react-app I'm working on and pushed it in a branch called gtm.
AWS Amplify builds the app successfully and it's up and running. I then merged these changes in the master branch with no merge conflicts. However, in the master branch, (where it's hosted on a custom domain) the app fails to build due to this error:
Error: [BABEL] /codebuild/output/src957941865/src/webapp/src/index.js:
Cannot find module '#babel/helper-regex'
Below is the build error displayed in AWS Amplify
(1) First thing to do was obviously perform this:
yarn add #babel/helper-regex
However, deploying with the new changes only creates another
Error: [BABEL] /codebuild/output/src957941865/src/webapp/src/index.js:
but with a different babel module instead, so this creates a long chain of adding packages that don't necessarily fix the underlying issue.
(2) Second fix I did was revert to a version of the app that was successfully deployed. I initially suspected that the google tag manager was the one causing the problem. However, it wasn't the case. The same error was displayed.
(3) Third thing I did was I referred to fix suggested in this issue (https://github.com/facebook/create-react-app/issues/8062). Summarizing the thread, it says that all I needed to do was to perform the following.
Delete node_modules
Delete yarn.lock
Perform yarn install
Unfortunately, this didn't solve the issue and the build error still persists.
(4) Fourth fix I did was I checked the differences between the two branches master and gtm using git diff master..gtm and here was the result.
Seeing that there were differences, out of desperation, I downloaded the **gtm** branch and then copy pasted its contents into the master branch in windows explorer then committed the changes.
Unfortunately, this still didn't fix the issue.
The only thing I haven't tried yet is reconnecting the master branch in AWS amplify but I don't want to risk the website not working since there are active users and if I ever go through with it and it still didn't fix the issue it'll cause more inconvenience. I hope someone already experienced this before and can point me to the right direction.
I had a similar issue recently while tackling a dependency problem for babel-loader with storybook and create-react-app.
After I noticed other preview builds were working correctly, while having the same code as the master branch, I suspected the node_modules cache was the culprit.
I don't know if there's an easier way of cleaning the node_modules cache that amplify uses internally (searched a bit and found nothing), so I just added an extra command in the preBuild phase (rm -rf node_modules)
eg.
version: 1
frontend:
phases:
preBuild:
commands:
- rm -rf node_modules
- yarn install
Once the build was passing, and the cache is regenerated (this is the last step of the build process) you can simply remove the added command and everything works as expected for future builds.
I suggest you also try combining fixes (2) & (3). That is, revert to a version w/out google tag manager followed by a yarn clean-install (i.e. delete node_modules & yarn.lock -> yarn install). The reason for this is your local workspace may still have broken yarn dependencies from the gtm branch even after you revert to a known-good version.
I am deploying a app engine application through cloud shell. I have deployed it a lot of times, but currently when ever I deploy it i gets hanged at the below position and nothing happens:
myname#cloudshell:~stackdriver-metrics-export/list_metrics (my-project)$ gcloud app deploy
Services to deploy:
descriptor: [/home/myname/stackdriver-metrics-export/list_metrics/app.yaml]
source: [/home/myname/stackdriver-metrics-export/list_metrics]
target project: [my-project]
target service: [list-metrics]
target version: [20200703t225313]
target url: [http://list-metrics.my-project.uc.r.appspot.com]
Do you want to continue (Y/n)? Y
Beginning deployment of service [list-metrics]...
// it gets stuck here and nothing happens afterwards I even restarted cloudshell lot of times but the same result, there is no change in the cursor position also. No error is also thrown
Please help me as i am new to cloud shell and google cloud.
This is working now, I am not sure what is the reason, but after i try to download a project from cloudshell to my local machine, strangely I found the project in one of the cloud shell directories itself as duplicate, and I have cleaned all of that, so still not sure of the reason
Check if you have symbolic link or very large file in the same directory. That will cause the system to take along time packaging your deployment.
While I was deploying my google app engine project (Java) using gradle appengineDeploy my internet connection was interrupted.
I re-deployed the project. Although the console said BUILD SUCCESSFUL, the app engine instance no longer works. No matter how many times I re-deploy or update my application, the logs show nothing but errors:
java.lang.NoClassDefFoundError: com/google/appengine/api/ThreadManager
at
com.google.api.control.extensions.appengine.GoogleAppEngineControlFilter.createClient
(GoogleAppEngineControlFilter.java:61) at
com.google.api.control.ControlFilter.init (ControlFilter.java:141) at
org.eclipse.jetty.servlet.FilterHolder.initialize
(FilterHolder.java:139) at
org.eclipse.jetty.servlet.ServletHandler.initialize
(ServletHandler.java:873) at
org.eclipse.jetty.servlet.ServletContextHandler.startContext
(ServletContextHandler.java:349) at
org.eclipse.jetty.webapp.WebAppContext.startWebapp
(WebAppContext.java:1406) at
com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.startWebapp
(AppEngineWebAppContext.java:175) at
org.eclipse.jetty.webapp.WebAppContext.startContext
(WebAppContext.java:1368) at
org.eclipse.jetty.server.handler.ContextHandler.doStart
(ContextHandler.java:778) at
org.eclipse.jetty.servlet.ServletContextHandler.doStart
(ServletContextHandler.java:262) at
org.eclipse.jetty.webapp.WebAppContext.doStart
(WebAppContext.java:522) at
com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.doStart
(AppEngineWebAppContext.java:120) at
org.eclipse.jetty.util.component.AbstractLifeCycle.start
(AbstractLifeCycle.java:68) at
com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.createHandler
(AppVersionHandlerMap.java:240) at
com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.getHandler
(AppVersionHandlerMap.java:178) at
com.google.apphosting.runtime.jetty9.JettyServletEngineAdapter.serviceRequest
(JettyServletEngineAdapter.java:120) at
com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest
(JavaRuntime.java:747) at
com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest
(JavaRuntime.java:710) at
com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run
(JavaRuntime.java:680) at
com.google.apphosting.runtime.JavaRuntime$NullSandboxRequestRunnable.run
(JavaRuntime.java:872) at
com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run
(ThreadGroupPool.java:270) at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.ClassNotFoundException:
com.google.appengine.api.ThreadManager at
java.net.URLClassLoader.findClass (URLClassLoader.java:381) at
com.google.apphosting.runtime.ApplicationClassLoader.findClass
(ApplicationClassLoader.java:135) at java.lang.ClassLoader.loadClass
(ClassLoader.java:424) at java.lang.ClassLoader.loadClass
(ClassLoader.java:357)
Is there some way for me to clear out the partial build (or whatever there may be) in google app engine?
I tried deleting all versions and instances (you cannot delete them all, it won't let you delete the serving one).
I tried increasing the version number in appengine-web.xml.
I tried gradle clean.
I tried disabling and enabling the application in the google cloud console.
No luck.
One idea I have is maybe if I can somehow force all the files for the app to be uploaded again? Because with gradle appengineDeploy only the changed files get uploaded.
EDIT:
I managed to fix one part of this by upgrading classpath 'com.google.cloud.tools:appengine-gradle-plugin:2.2.0' in my gradle to the latest version. (Felt kind of hackish to do that and not reliable in the future if this happens again.) Maybe this flushed whatever was in app engine or something by doing that. I still have one problem remaining:
I am using the firebase admin SDK to send firebase cloud messages (authenticated with a .json credentials file) like this:
FirebaseMessaging.getInstance().send(msg);
When I do I am getting this error:
com.google.firebase.messaging.FirebaseMessagingException: Unexpected
HTTP response with status: 401; body: null
Which is strange because other parts of the firebase admin SDK are working (like writing/reading from firestore).
So there is still something weird going on and I think it has to do with the fact that as I was uploading my google app engine project the connection was interrupted.
EDIT 2:
Here is something interesting: When I deploy my application using gcloud app deploy appengine-web.xml the application again will not work. Deploying with gradle appengineDeploy does though. I also noticed that the size of the application is shown as much smaller in the GCP console when deploying with gcloud app deploy appengine-web.xml. So something is messed up here. I tried looking up some sort of gcloud command to clear cache? Or something like that but no luck.
EDIT 3:
Additional Info:
My app was already deployed and working before the failed attempt. I changed one small piece of code in a function and upon uploading the app, the connection was interrupted because the internet went out.
I am on app engine standard environment.
I am deploying my application from the macOS terminal and using android studio to develop.
I have tried the stopPreviousVersion promote and version configs in gradle (actually the first two are true by default and version gets auto-generated if you do not set it).
Running gcloud app deploy appengine-web.xml --verbosity=debug shows a lot of sensitive information but one thing I am seeing is all the files in WEB-INF are being skipped:
DEBUG: Skipping upload of [WEB-INF/....
INFO: Incremental upload skipped 100.0% of data
DEBUG: Uploading 0 files to Google Cloud Storage
DEBUG: Using [16] threads
So perhaps files are not all being uploaded? This SO post raises a similar problem but has no solution: stackoverflow.com/q/42137452/3075340
It's weird but when I do gcloud app deploy, the app won't work at all. There are run-time errors all over the place. Doing gradle appengineDeploy fixes that but I still am having the firebase-admin issue.
To actually answer your question, GAE uses a staging bucket to cache files.
You can delete this bucket, and it will get recreated next time you try to deploy.
The bucket is always named staging.[PROJECT-ID].appspot.com. It should show on the buckets overview page
Just to be clear, I'm not anywhere near sure that this will actually resolve your issue, but this will most definitely clear whatever files GAE cached
Here there was a discussion on a, more or less, similar issue on App Engine from last year.
You can use it as a source of inspiration to fix your current issue, because I think they are really similar as a concept. From there I think it may worth trying to put the appengine-api-1.0-sdk-1.9.63.jar ( or whatever your version is ) file under WEB-INF/lib.
It the same post there were some guys who found out the there was a problem with gcloud v194 and switching to another version fixed the issue.
Here there is a similar issue which was resolved by upgrading the gcloud tool.
Anyhow, this behaviour is not normal and things should not work like this. You can try to find a workaround, but my recommendation is to report the issue and let an App Engine engineer taking a look over it. There are good chances to be something internal.
I cannot get my server code to update. I'm running a PHP instance on GAE and no matter what I do, the files won't update. In the source code view, I can see the files have updated, but when I attempt to access the updated file, I'm still viewing the old version. I've also attempted disconnecting my Bitbucket repo and using the appcfg.py update project-name command, but the files aren't refreshing when I attempt to access them. I'm not sure what to do to force the changes to take place.
My app.yaml contains the following code
- url: /(.+\.php)$
script: \1
secure: always
So the files should be getting read, right?
I was able to figure out what went wrong. I downloaded my code using appcfg.py download_app -A <your_app_id> -V <your_app_version> <output-dir> and noticed that I was downloading the old versions of the files (and wasn't downloading the new files). Turns out using source control within GAE will upload new code, but won't deploy it. I attempted to use appcfg.py update project-name one more time, but it didn't work. Turns out I didn't disconnect my Bitbucket account (could have sworn that I did...). Once disconnected, I was able to update the project using appcfg.py update project-name. While I was figuring this out, I reached out to Google support and received this message:
To use the feature of push to deploy you need to spin-up the Jenkins
Instance on GCE (Google Compute Engine) and then it will take the
updated code and execute it in the environment. Go through [1] for how
to enable the Jenkins instance and its configuration according to
different run time.
In your issue, you just mirrored the code from Bit Bucket to Cloud
Repository, as it is just doing the version control for the
application not executing the application. So basically you have have
the option of using Jenkins instance as I described above to test the
different version of the code or using the appcfg.py update command
from your local repository.
I haven't attempted to install and use Jenkins since I fixed it after disconnecting my Bitbucket account), but it may help others who have run into this problem.
I have been trying out Codeship and Heroku for continuous deployment of an AngularJS application I writing at the moment. The app was created using Yeoman and uses bower and grunt. Initially I thought this seemed like a really good setup as Codeship was free to use and I was quickly able to configure this to build my AngularJS project and it offered the ability to add a deployment step after the build. There were even many PaaS providers to choose from (Heroku, S3, Google App Engine etc). However I seem to have become a bit stuck with getting the app to run on Heroku.
The problem started from the fact that all the documentation suggested that I remove the /dist path from my .gitignore so that this directory is published to Heroku post build. This was mainly from documentation that talked about publishing to Heroku from a local machine, but I figure this is all Codeship is doing under the hood anyway. I didn't want to do this as I don't believe I should be checking build output into source control. The /dist folder was added to .gitignore for a good reason. Furthermore, this kind of defeats the point of having a CI server somewhat, as I might as well just push the latest build from my machine.
After some more digging around I found out that I could add a postinstall step to my packages.json file such as bower install && grunt build which would re-run the build on Heroku and hence repopulate all the bower dependencies (something else they wanted me to check in to source control!) and the dist directory.
After giving this a try it became apparent that I would need to add bower and grunt as dependencies in packages.json, which meant moving them from devDependencies which is where they should belong!
So I now seem to be stuck. All I want to do is publish my build artefacts (/dist) the dependencies (/bower_components) and the server.js file that will run the site. Does anyone know how to achieve this with Heroku and Codeship? Alternatively has anyone had any success with this using different tools. I am looking for something that is free and I am willing to accept that it will not be production stable (won't scale to multiple servers etc), but this is fine for now as all I want to do is continuously deploy the app for internal testing and to be able to share the output with non-technical members of my team so we can discuss features we'd like to prioritise etc.
Any advice would be greatly appreciated.
Thanks
Ahoy, Marko from the Codeship crew here. Did you already send us an in app message about this? I'm sure we can get your application building on Codeship and deploying to Heroku successfully.
As a very short answer, the easiest way to get this running would be to add both bower and grunt to your dependencies in the package.json. Another possibility would be to look for a custom buildpack with both tools already installed.
And finally you could also run the tools on Codeship, add the newly installed files to the repository, commit the changes and push this new commit to Heroku. If you want to use this, you'd very probably need to force push the changes though.
Feel free to reach out to me via the in app messenger (lower right corner of the site) and I'd be happy to help you get this working!
I found two ways to get this to work.
Heroku Node Custom Buildpack
Use the mbuchetics Heroku build pack. This works by basically re-building the app once it has been pushed to Heroku.
There were a few tricks I had to employ still to make this work. In Gruntfile.jstwo new tasks needed to be configured called heroku:production and heroku:development. This is what the buildpack executes to build the app. I initially just aliased the main build task, but found that the either the buildpack or Heroku had a problem with running jshint so in the end I copied the build task and took out the parts that I didn't need.
Also in packages.json I had to add this:
"scripts": {
"postinstall": "bower cache clean && bower install"
}
This made sure the bower_components were available in Heroku.
Pros
This allowed me to keep the .gitignore file in tact so that the 'binaries' in the dist directory and the dependencies in the bower_components directory were not committed into source control.
Cons
This is basically re-building the app once it is on Heroku and I generally prefer to use the same 'binaries' throughout the entire build and deployment pipeline. That way I know that the same code that was built, is the same code that was tested and is the same code that was deployed.
It also slows down the deployment as you have to wait for the app to build twice.
CodeShip Custom Script Deployment
Not being satisfied with the fact I was building my app twice, I tried using a Custom Script pipeline in CodeShip instead of the pre-existing Heroku one. The script basically modified the .gitignore file to allow the dist folder to be committed and then pushed to the Heroku remote (which leaves the code on the origin remote unaffected by the change).
I ended up with the following bash script:
#!/bin/bash
gitRemoteName="heroku_$APP_NAME"
gitRemoteUrl="git#heroku.com:$APP_NAME.git"
# Configure git remote
git config --global user.email "you-email#example.com"
git config --global user.name "Build"
git remote add $gitRemoteName $gitRemoteUrl
# Allow dist to be pushed to heroku remote repo
echo '!dist' >> .gitignore
# Also make sure any other exclusions dont apply to that directory
echo '!dist/*' >> .gitignore
# Commit build output
git add -A .
herokuCommitMessage="Build $CI_BUILD_NUMBER for branch $CI_BRANCH. Commited by $CI_COMMITTER_NAME. Commit hash $CI_COMMIT_ID"
echo $herokuCommitMessage
git commit -m "$herokuCommitMessage"
# Must merge the last build in Heroku remote, but always chose new files in merge
git fetch $gitRemoteName
git merge "$gitRemoteName/master" -X ours -m "Merge last build and overwrite with new build"
# Branch is in detached mode so must reference the commit hash to push
git push $gitRemoteName $(git rev-parse HEAD):refs/heads/master
Pros
This only require a single build of the app and deploys the same binaries that were tested during the test phase.
Cons
I've used this script quite a few times now and it seems relatively stable. However one issue I know of is that when a new pipeline is created there will be no code on the master branch so this script fails when it tries to do the merge from the heroku remote. At the moment I get around this by doing an initial push of the master branch to Heroku before kicking off a build, but I imagine there is probably a better Git command I could run along the lines of; 'only merge this branch if it already exists'.