Local GAE Datastore is empty after restart on OSX - google-app-engine

I'm building a Google App Engine application with a Go backend + Polymer frontend. As a result, I'm using a dispatch.yaml file to serve both at the same time.
The problem I'm facing is that the datastore is empty when I restart my computer. I've tested this on both OSX 10.9.5 and 10.10.4. Both exhibit the same response upon a system reboot. Windows 7, however, seems to hold on to the data.
The documentation suggests that data should persist, since I'm not explicitly calling a clear. It's not. I've tried to set the datastore location myself using this:
dev_appserver.py --datastore_path=~/go_apps/data ~/go_apps/my_app
I'm receiving this error:
google.appengine.tools.devappserver2.errors.AppConfigNotFoundError: "/Users/anthony/go_apps/my_app is a directory but does not contain app.yaml or app.yml
Obviously, since I'm using a dispatch.yaml file, it wouldn't. So, since the backend, which handles the data, does have an app.yaml file, I try to set it there. I use this command:
dev_appserver.py --datastore_path=~/go_apps/data ~/go_apps/my_app/backend
That doesn't seem to work either, as I get this error:
sqlite3.OperationalError: unable to open database file
Okay? Well, not sure where to turn now. From what I could gather from other posts, that data is stored temporarily. But, I can't seem to set a custom, non-temporary location for the data. So, now I'm populating a datastore every time I reboot, which seems ridiculous.
* Edit *
I've tried the following, which seems like it tries to launch the app, and creates a datastore.db file at the correct location:
dev_appserver.py --datastore_path ~/go_apps/my_app/data/datastore.db ~/go_apps/my_app/dispatch.yaml ~/go_apps/my_app/backend/app.yaml ~/go_apps/my_app/frontend/app.yaml
However, I'm getting a weird error now:
/var/folders/04/3hxnpxc15wj2k4v40lkdncd00000gn/T/tmpkcQYnFappengine-go-bin/backend.go:13: can't find import: "github.com/gorilla/mux"
Does Go build to that folder temporarily? That import is definitely available, and always builds fine calling goapp serve.
Here is what my imports look like on backend.go
import (
//standard library
"fmt"
"net/http"
"time"
"log"
//third party
"github.com/gorilla/mux"
"github.com/gorilla/securecookie"
"github.com/dgrijalva/jwt-go"
"golang.org/x/crypto/bcrypt"
//my imports
"github.com/section14/go_polymer_comm_pkg/controller"
)

You have to pass the name of the file to be used as the persisted datastore, not a folder.
And next provide the folder of your app (which contains app.yaml). Don't mix the 2. So it should be something like:
dev_appserver.py --datastore_path=~/my_app/my_app.db ~/go_apps/my_app
Details can be found here:
The Go Development Server / Using the Datastore
Notes:
The default datastore file is in the temp folder, and your OS-X most likely clears that on system restart, that's why it is not preserved for you. On the other hand Windows 7 for example does not clear the temp folder on system restarts.

Got it up and running by adding both GOPATH and GOROOT environment variables to my .bash_profile. In total, these three paths (first path was already set) are needed for it to run:
# Add Google AppEngine path
export PATH=/Users/anthony/go_appengine:$PATH
# GOPATH
export GOPATH=/Users/anthony/go_appengine/gopath
export PATH=$PATH:$GOPATH
# GOROOT
export GOROOT=/Users/anthony/go_appengine/goroot
export PATH=$PATH:$GOROOT
This command is called from inside the project folder (mine resides outside of the appengine folder) for it to launch:
dev_appserver.py --datastore_path data/datastore.db dispatch.yaml backend/app.yaml frontend/app.yaml
Notice that the .yaml files are still there. It builds fine with them, and probably builds fine without them if you don't need a dispatch.yaml file.
Thanks #icza for the direction. Wanted to organize the steps in a post for easier reading.

Related

Typeorm + Firebase functions: "No connection options were found in any orm configuration files" after deployed

Env:nodejs12
Folder structure:
#root
/functions
/src
...
/models
/resolvers
index.ts
ormconfig.json
package.json
tsconfig.json
...
.firebaserc
firebase.json
Everything worked when developing in local environment. After deployed to firebase functions, No connection options were found in any orm configuration files shows up.
What might be the cause?
I'll update with more information if needed.
=========================================
update
Below is the folder structure of deployed codes. (Cloud functions can't show more than 50 files so I downloaded the source code from GCP)
As you can see the ormconfig.json does exist in the root, but somehow it cannot be located. I have to create connection manually with typeorm.createConnection({type: "postgres",...}) to make the code work.
This is likely being caused by a known bug with app-root-path (which TypeORM uses for config file resolution) when used in conjunction with Google Cloud Functions.
The workaround / fix that worked for me was to set the environment variable APP_ROOT_PATH to /workspace when I deployed my Google Cloud Function (app-root-path will short-circuit when it sees that variable).

Cannot GET index.html Azure Linux Web App

We created a Linux Web App in Microsoft Azure. The application is static written with React (html and Javascript).
We copied the code into the wwwroot folder, but the application only showing only hostingstart.html and when we try to get page index.html we have this error:
Cannot GET /index.html
We tried with a sample of Azure in GitHub (https://github.com/Azure-Samples/html-docs-hello-world) but the error is the same.
The url is this: https://consoleadmin.azurewebsites.net/index.html
Last week the application was running correctly.
We forget to do something?
MAY 2020 - You don't have to add any javascript files or config files anywhere. Let me explain.
I was facing this exact same issue and wasted 6 hours trying everything including the most popular answer to this question. While the accepted answer is a nice workaround (but requires more work than just adding the index.js file), there's something a simpler than that.
You see, when you just deploy an Azure Web App (or App Service as it is also called), two things happen:
The web app by default points to opt/startup/hostingstart.html
It also puts a hostingstart.html in home/site/wwwroot
When you deploy your code, it replaces hostingstart.html in home/site/wwwroot but the app is still pointing to opt/startup/hostingstart.html. If you want to verify this, try deleting opt/startup/hostingstart.html file and your web app will throw a "CANNOT GET/" error.
So how to change the default pointer? It's simpler than it looks:
Go to Configuration tab on your web app and add the following code to startup script:
pm2 serve /home/site/wwwroot --no-daemon
If this web app is a client-side single-page-app and you're having issues with routing, then add --spa to the above command as follows:
pm2 serve /home/site/wwwroot --no-daemon --spa
This will tell the web app to serve wwwroot folder. And that's it.
Image for reference:
Screenshot explaination
PS: If you only set the startup script without deploying your code, it will still show the hostingstart.html because by default that file lies in the wwwroot folder.
Ok you are gonna love this. This happened to me today also. Same exact thing.
I am pretty sure the azure team flipped a switch somewhere and we fell through a crack.
I found this obscure answer with no votes and it did the trick (with a little extra finagling)
BONUS! this also fixed my router issues I was having only on the deployed site (not local):
Credit: #stormwild: Default documents not serving on node web app hosted on Azure
From #stormwild's post see here:
https://blogs.msdn.microsoft.com/waws/2017/09/08/things-you-should-know-web-apps-and-linux/#NodeHome
Steps:
Go to your azure portal, select your app service and launch ssh
In ssh terminal, navigate via command line to /home/site/wwwroot
create index.js there with the following code:
var express = require('express');
var server = express();
var options = {
index: 'index.html'
};
server.use('/', express.static('/home/site/wwwroot', options));
server.listen(process.env.PORT);
NOTE: Be sure to run npm install --save express also in this folder else your app service will crash on startup
Be sure to restart your app service if it doesn't do so automagically
A workaround, I changed the webapp stack to PHP 7
Another solution would be to add a file called ecoysystem.config.js right next to your index.html file.
module.exports = {
apps: [
{
script: "npx serve -s"
}
]
};
This will tell pm2 to associate all requests to index.html as your app service starts up.
Very helpful information here: https://burkeholland.github.io/posts/static-site-azure/

Spring Devtools triggers restart when changing excluded files

I have a Spring Boot Web Project that uses Joinfaces and Primefaces. The application extends from SpringBootServletInitializer and also generates a war file.
For development I start my Project (in Eclipse) using RunAs -> Spring Boot App. I've added the spring-boot-devtools dependency as described in the Developer Tools Docu. Everything seems to work, except that changes in a xhtml file also triggers the restart.
I've added the spring.devtools.restart.additional-exclude property in my application.yml to exclude also webapp/**. But this seems to have no effect, because a change still triggers the restart.
spring:
devtools:
restart:
additional-exclude: webapp/**
My Project structure looks like:
src
-main
-java
-resources
-webapp
- *.xhtml
But maybe I do not understand how to use the DevTools correct - Starting the application as Spring Boot App might not be the correct usage? Because if I place a file e.g. foo.txt under src/main/resources and update its content the application is also restarted (using the DevTools defaults (which should exclude /resources)).
Using:
Eclipse Oxygen.3a Release (4.7.3a)
Joinfaces 3.2.1 -> Spring Boot 2.0.2.RELEASE
The solution to my main problem might be relatively easy.
I've added the webapp folder as Source Folder in Eclipse. As soon as I've removed the folder from the list of source folders, the reload was no longer triggered when I changed something in the xhtml files.
This answer gave me the idea, that the source folder might be the problem.
My exclude also did not work. What I did as a workaround is slowing the reload down like this:
poll-interval: 6000
quiet-period: 5000

Google Cloud Storage on Appengine Dev Server

There's a similar question that was recently responded to on Stackoverflow here: Google Cloud Storage Client not working on dev appserver
The solution was to either upgrade the SDK to 1.8.8 or use the previous revision of the GCS client library which didn't have the bug instead.
I'm currently using 1.8.8 and have tried downloading multiple revisions and /_ah/gcs doesn't load for me. After using up a significant number of my backend instances trying to understand how GCS and app engine work together, it'd be great if I could just test it on my local server instead!
When I visit localhost:port/_ah/gcs I get a 404 not found error.
Just a heads up, to install the library all I did was drag and drop the code into my app folder. I'm wondering if maybe I skipped a setup step? I wasn't able to find the answer in the documentation!
thanks!!
Note
To clarify this is my first week using GCS, so my first time trying to use the dev_server to host it.
I was able to find the google cloud storage files I wrote to a bucket locally at:
localhost:port/_ah/gcs/bucket_name/file_suffix
Where port is by default 8080, and the file was written to: /bucket_name/file_suffix
For those trying to understand the full process of setting up a simple python GAE app and testing local writes to google cloud storage:
1. Follow the google app engine "quickstart":
https://cloud.google.com/appengine/docs/standard/python/quickstart
2. Run a local dev server with:
dev_appserver.py app.yaml
3. If using python, follow "App Engine and Google Cloud Storage Sample":
https://cloud.google.com/appengine/docs/standard/python/googlecloudstorageclient/app-engine-cloud-storage-sample
If you run into "ImportError: No module named cloudstorage" you need to create a file named appengine_config.py
touch appengine_config.py
and add to it:
from google.appengine.ext import vendor
vendor.add('lib')
GAE runs this script automatically when starting your local dev server with dev_appserver.py app.yaml, and it is necessary to run this script for GAE to find the cloudstorage library in your lib/ folder
4. "Writing a file to cloud storage" from the same tutorial:
def create_file(self, filename):
"""Create a file."""
self.response.write('Creating file {}\n'.format(filename))
# The retry_params specified in the open call will override the default
# retry params for this particular file handle.
write_retry_params = cloudstorage.RetryParams(backoff_factor=1.1)
with cloudstorage.open(
filename, 'w', content_type='text/plain', options={
'x-goog-meta-foo': 'foo', 'x-goog-meta-bar': 'bar'},
retry_params=write_retry_params) as cloudstorage_file:
cloudstorage_file.write('abcde\n')
cloudstorage_file.write('f'*1024*4 + '\n')
self.tmp_filenames_to_clean_up.append(filename)
with cloudstorage.open(
filename, 'w', content_type='text/plain', options={
'x-goog-meta-foo': 'foo', 'x-goog-meta-bar': 'bar'},
retry_params=write_retry_params) as cloudstorage_file:
cloudstorage_file.write('abcde\n')
cloudstorage_file.write('f'*1024*4 + '\n')
Where filename is /bucket_name/file_suffix
4. After calling create_file via a route in your WSGI app, your file will be available at:
localhost:port/_ah/gcs/bucket_name/file_suffix
Where port is by default 8080, and the file was written to: /bucket_name/file_suffix
Postscript
Unfortunately, I did not find either 3) or 4) in their docs, so I hope this helps someone get set up more easily in the future.
To access gcs objects on dev_appserver, you must specify the bucket & object name, i.e. /_ah/gcs/[bucket]/[object].
The storage simulator for the local server is working in later versions of the SDK. For Java, one may choose to follow a dedicated tutorial: “App Engine and Google Cloud Storage Sample”.

Retrieve file relative to play application path

I created a Play! app and deployed it under TomCat. This works well. The only problem is the management of a properties file, currently in the conf folder right next to application.conf. But as soons as the client replaces the war file the custom properties are overwritten with the default values, resulting in errors.
Now I want to introduce a seperate properties file placed inside the webapps folder. This way I will be sure my clients will not overwrite the file 'accidentally'.
So the structure would be:
TomCat webapps:
myPlayApp
PlayConfig <-- here I want to place the config file
So I would like to retrieve the properties file by something like:
getFile("../PlayConfig/app.properties");
This obvious does not work, but I do not know how to achieve this?
I thought retrieving it by tomcat http url but the portnumber my vary, so this would also not work, I guess...
UPDATE 2012-01-25:
Actually when using the following code:
Play.applicationPath.getPath();
I get the absolute path when running the project outside tomcat (so not inside war file!)
When I deploy the same project in a TomCat server I get the following output:
W:\tomcat-5.5\webapps\MyTestProject\WEB-INF\application.
From this point on I can indeed use a relative path.
I think that when deployed in a Servlet container, play uses the /WEB-INF/application as base directory.
Try changing the path relative to this folder.

Resources