GORM Cloud SQL Connection on App Engine Using Go - google-app-engine

I'm trying to connect to a Cloud SQL database using GORM in golang.
db, _ = gorm.Open("mysql", "user:pass#cloudsql(connection:name:example)/")
if err != nil {
log.Println(err)
//panic(err)
}
When I attempt to serve the app
goapp serve appengine/
I get a runtime error
ERROR 2017-02-19 20:48:05,436 http_runtime.py:396] bad runtime process port ['\r\n']
Which I found was related to the database migration
db.AutoMigrate(&models.Event{})
If I remove the AutoMigrate, the runtime process port error goes away. However whenever I access a route (ie /events) that does a database query, the connection gets dropped, a 404 page is thrown, and an error message is logged sql: database is closed
When I run the app locally by building the package go build && ./appname and using a local MySQL server, it works fine.
Can someone please tell me how to connect to a Cloud SQL database using Go's GORM framework and App Engine?

This is due to the call to log.New in this file: https://github.com/jinzhu/gorm/blob/master/logger.go#L15
This anwser explain why dev_appserver.py gets it: https://stackoverflow.com/a/24112953/4266494
To disable this, you can either disable all GORM logging:
db.LogMode(false)
Or use an adapter the logger output: https://github.com/benguild/GAEBridge/blob/master/log/debugLevel.go
db.SetLogger(NewDebugLogger(nil)) // on application scope
db.SetLogger(NewDebugLogger(appengine.NewContext(req))) // on request scope
I'm setting a new logger with the real context
This is the only workaround I found to avoid crashes while keeping some logs, it could be awesome if one of you had a real one.

Related

Sudden datastore exceptions : DatastoreFailureException: Missing or invalid authentication

I have a Google Appengine app (Java) and the datastore was recently auto-migrated into "Cloud Firestore in Datastore mode" (completed on 6/6/2022) that has been running for years.
It has now started throwing these exceptions in new deployments without any relevant code change to trigger them:
com.google.appengine.api.datastore.DatastoreFailureException: Missing or invalid authentication.
at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:69)
at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:127)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:97)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:68)
at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:32)
at com.google.appengine.api.datastore.BaseQueryResultsSource.getIndexList(BaseQueryResultsSource.java:154)
at com.google.appengine.api.datastore.BaseQueryResultsSource.loadMoreEntities(BaseQueryResultsSource.java:187)
at com.google.appengine.api.datastore.BaseQueryResultsSource.loadMoreEntities(BaseQueryResultsSource.java:166)
at com.google.appengine.api.datastore.QueryResultIteratorImpl.ensureLoaded(QueryResultIteratorImpl.java:146)
at com.google.appengine.api.datastore.QueryResultIteratorImpl.hasNext(QueryResultIteratorImpl.java:64)
at com.googlecode.objectify.impl.KeysOnlyIterator.hasNext(KeysOnlyIterator.java:29)
at com.googlecode.objectify.impl.ChunkIterator.next(ChunkIterator.java:48)
at com.googlecode.objectify.impl.ChunkIterator.next(ChunkIterator.java:20)
at com.google.common.collect.MultitransformedIterator.hasNext(MultitransformedIterator.java:52)
at com.google.common.collect.MultitransformedIterator.hasNext(MultitransformedIterator.java:50)
at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1105)
at com.googlecode.objectify.impl.ChunkingIterator.hasNext(ChunkingIterator.java:51)
at com.google.common.collect.Iterators.addAll(Iterators.java:366)
at com.google.common.collect.Lists.newArrayList(Lists.java:163)
at com.googlecode.objectify.util.MakeListResult.translate(MakeListResult.java:22)
at com.googlecode.objectify.util.MakeListResult.translate(MakeListResult.java:12)
at com.googlecode.objectify.util.ResultTranslator.nowUncached(ResultTranslator.java:21)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
at com.googlecode.objectify.util.ResultProxy.invoke(ResultProxy.java:34)
at com.sun.proxy.$Proxy68.isEmpty(Unknown Source)
My code. After loading a list of entities, the exception is thrown on the isEmpty() call on the returned list:
List<SystemSetting> applicationSettings = ofy().load().type(SystemSetting.class).filter("name", name).list();
if (applicationSettings.isEmpty()) { // Exception thrown here
It's worth also noting that on previous versions, datastore reads and writes are still happening without any issues. It is only new deployments that exhibit this issue. However, redeploying previous versions from previous commits using the same CI service to deploy them are failing.
Is this caused by the datastore migration?
Slight update: these errors are only happening on newly deployed versions... In fact, I deployed a new version (from CI, bitbucket pipeline) that worked, but when I redeployed it with no changes it no longer works. Same app engine version id, code and users etc.
Things we've tried without success:
Re-deploying last known version
Clearing memcache
Deploying from local dev
Stopping instances
Trying alternative service account key
Expanding service account permissions
Clearing BitBucket pipeline caches (the CI builder)
Have recreated the issue with datastore api directly (no objectify)
Have redeployed with different user account - no success
Have redeployed with new service account - no success
Update 1:
If I run the following code on a local java application (pointing at live datastore using same service account) it works locally, but fails if I deploy to app engine with my project-id.
DatastoreOptions datastoreOptions = DatastoreOptions.getDefaultInstance();
Datastore datastore = datastoreOptions.getService();
// A known entity in my live datastore
Key key = datastore.newKeyFactory()
.setKind("MyEntityType")
.newKey(123123123l);
Entity retrieved = datastore.get(key);
When it fails, I can get slightly more information:
java.io.IOException: Unexpected Error code 500 trying to get security access token from Compute Engine metadata for the default service account: Could not fetch URI /computeMetadata/v1/instance/service-accounts/default/token
full stack:
com.google.cloud.datastore.DatastoreException: I/O error
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:138)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:123)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.lookup(HttpDatastoreRpc.java:173)
at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:434)
at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:431)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.datastore.DatastoreImpl.lookup(DatastoreImpl.java:430)
...
Caused by: com.google.datastore.v1.client.DatastoreException: I/O error, code=UNAVAILABLE
at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:171)
at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:117)
at com.google.datastore.v1.client.Datastore.lookup(Datastore.java:93)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.lookup(HttpDatastoreRpc.java:171)
... 74 more
Caused by: java.io.IOException: Unexpected Error code 500 trying to get security access token from Compute Engine metadata for the default service account: Could not fetch URI /computeMetadata/v1/instance/service-accounts/default/token
at com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:206)
at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:257)
at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:254)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.auth.oauth2.OAuth2Credentials$AsyncRefreshResult.executeIfNew(OAuth2Credentials.java:580)
at com.google.auth.oauth2.OAuth2Credentials.asyncFetch(OAuth2Credentials.java:220)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:170)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159)
at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc$1.initialize(HttpDatastoreRpc.java:91)
at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:95)
... 76 more
Update 2:
I've since paid for Google support and they have now confirmed it "is linked to a known issue related to the GAE standard Service Agent and access token which is affecting some users"

Can't invoke server's method from client - Error: An unexpected error occurred invoking 'SignIn' on the server

I'm build a project using SignalR.
As Front I'm using .Net Core, as Back I'm using React.
So in my localhost the project is working, but when i deploy this project i get an error.
In azure I have created:
2 web application - one used for cliend, and the other one for server.
1 Azure SignalR service (i tested it online and it worked)
1 Azure SQL Database
So when i try to invoke Method it now working and show me this error. Someone know how to solve this problem?
Please check this ASP.NET Core SignalR connection
troubleshooting | Microsoft Docs which helps to check errors
that can occur when trying to connectto an ASP.NET Core SignalR hub.
One common cause is that app enforce HTTPS by calling
UseHttpsRedirection in Startup, or enforces HTTPS via URL rewrite
rule.
So If it is the cause try to change the URL on the client side from
"http" to "https". like > .withUrl("https://xxx/HubName")
Also please go through c# - Could not connect to Azure SignalR Hub -
Stack Overflow And check if the signalR url endpoint for client
is correctly given
var signalrUrl = "https://myazuresignalr.service.signalr.net/client/?hub=yourHubName"
Please note that sending the exception this way to the client is
insecure .So try to either throw a HubException or set
EnableDetailedErrors on the server.See HandleErrors when :Use
hubs in ASP.NET Core SignalR | Microsoft Docs
Reference:
Invoke hub's method which throw ArgumentException, client gets
HubException: An unexpected error occurred · Issue azure-signalr ·
GitHub
So the problem was that my Database was empty... the migration did not upload, Update the database solved it.

Having issues getting WordPress running under GAE + Cloud SQL

I tried to set up WordPress under Google App Engine earlier tonight (following the instructions here: https://developers.google.com/appengine/articles/wordpress).
It runs fine locally, but when I push to remote I get a database error (visible at https://wp-dot-frontiermediag.appspot.com/). If we throw on a /wp_admin/install.php you get:
This either means that the username and password information in your
wp-config.php file is incorrect or we can't contact the database server
at :/cloudsql/frontiermediag:fmwp. This could mean your host's database
server is down.
Here's the relevant code in wp-config:
/** MySQL hostname */
if(isset($_SERVER['SERVER_SOFTWARE']) && strpos($_SERVER['SERVER_SOFTWARE'],'Google App Engine') !== false) {
define('DB_HOST', ':/cloudsql/frontiermediag:fmwp');
}else{
define('DB_HOST', 'localhost');
}
frontiermediag:fmwp is showing "Status Runnable" in Developers Console > Cloud SQL.
I did this once before and it worked so I'm not sure what I'm missing here. I thought it might have been because I'm using WP 3.8.1. but rolled back to 3.5.1 and same thing's happening.
Any ideas? frontiermediag is listed as an authorized application on the :fmwp ACL.
This situation happened to me earlier.However, I edited my Cloud SQL instance , and set "Preferred Location" as "Follow App Engine App" from Google Developers Console. This database connection problem was solved in my case.
I tried the instructions with wordpress 3.5.1 and the instructions seem to work for me. The code snippet you have above seems right and I am not sure what could be wrong without looking at rest of your code. Can you try the instructions from the beginning one more time with 3.5.1?
I had this issue, because "Follow App Engine App" doesn't seem to be an option for second generation instances in my case, and so the instance connection name includes the region setting.
Look at the instance details, and under properties, find "Instance connection name". That is the text that should follow :cloudsql/.

Play Siena failing to connect to MySQL on GAE

I am using play framework 1.2.7, gae module 1.6.0 and siena module 2.0.7 (also tested 2.0.6). This is a simple project that should run in play deployed on App Engine and connect to a MySQL database in Google Cloud SQL. My project runs fine locally but fails to connect to the database in production. Looking at the logs it looks like it is using the postgresql driver instead of the mysql one.
Application.conf
# db=mem
db.url=jdbc:google:mysql://PROJECT_ID:sienatest/sienatest
db.driver=com.mysql.jdbc.GoogleDriver
db.user=root
db.pass=root
This is the crash stack trace
play.Logger niceThrowable: Cannot connected to the database : null
java.lang.NullPointerException
at com.google.appengine.runtime.Request.process-a3b6145d1dbbd04d(Request.java)
at java.util.Hashtable.put(Hashtable.java:432)
at java.util.Properties.setProperty(Properties.java:161)
at org.postgresql.Driver.loadDefaultProperties(Driver.java:121)
at org.postgresql.Driver.access$000(Driver.java:47)
at org.postgresql.Driver$1.run(Driver.java:88)
at java.security.AccessController.doPrivileged(AccessController.java:63)
at org.postgresql.Driver.getDefaultProperties(Driver.java:85)
at org.postgresql.Driver.connect(Driver.java:231)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at play.modules.siena.GoogleSqlDBPlugin.onApplicationStart(GoogleSqlDBPlugin.java:103)
at play.plugins.PluginCollection.onApplicationStart(PluginCollection.java:525)
at play.Play.start(Play.java:533)
at play.Play.init(Play.java:305)
What is going on here? I am specifying the correct driver and url schema and it's using postgresql driver. Google Cloud SQL API access is enabled, the app is allowed to connect to the mysql instance, I am not using db=mem, ... I am stuck and can't figure out how to move forward! :-((
UPDATE: I thought I found the solution, but that was not the case. If I keep the %prod. prefix and create a war normally (or just don't define any DB properties), then the application will use Google DataStore instead of the Cloud SQL. If I create the war file adding --%prod at the end (or just delete the %prod. prefix in the application.conf), then it will keep failing to connect to the database showing the same initial error.
Any ideas please?
After being stuck for so long on this I just found the solution in no time after posting the question. Quite stupid actually.
The production environment properties in the application.conf file must be preceded by %prod. so the database config should read
%prod.db.url=jdbc:google:mysql://PROJECT_ID:sienatest/sienatest
%prod.db.driver=com.mysql.jdbc.GoogleDriver
%prod.db.user=root
%prod.db.pass=root
And everything runs fine.
EDIT: This is NOT the solution. The problem went away, but the app is using the DataStore instead of the Cloud SQL
At the end I ended doing a slight modification in play siena module source code and recompiling it.
In case anyone is interested, you will need to remove/comment/catch exception in this code around line 97 in GoogleSqlDBPlugin class:
// Try the connection
Connection fake = null;
try {
if (p.getProperty("db.user") == null) {
fake = DriverManager.getConnection(p.getProperty("db.url"));
} else {
fake = DriverManager.getConnection(p.getProperty("db.url"), p.getProperty("db.user"), p.getProperty("db.pass"));
}
} finally {
if (fake != null) {
fake.close();
}
}
For some reason the connection fails when initiated with DriverManager.getConnection() but it works when initiated with basicDatasource.getConnection(); which apparently is the way used by the module in the rest of the code. So if you delete the above block, and recompile the module everything will work as expected. If you are compiling with JDK 7, you will also need to implement public Logger getParentLogger() throws SQLFeatureNotSupportedException in the ProxyDriver inner class at the end of GoogleSqlDBPlugin file.
Strangely, I digged into the DriverManager.getConnection() and it looked like some postgresql driver is registered somehow, because otherwise I can't see why DriverManager.getConnection() would call to org.postgresql.Driver.connect().

Server Error in appspot.com application

Running my application on local with dev_appserver for finding event list from Google calendar and it works fine.
when I deployed through app-engine (deployment done successful) and when I run the file it gives me below error
Error: Server Error
The server encountered an error and could not complete your request.
If the problem persists, please report your problem and mention this error message and the query that caused it.
Google lot about this error still unable to find solution for the same
my app.yaml file structure.
- url: /calendar
script: webserverflow12.py
When I run this appspot.com/calendar. It is giving me mentioned error.
Any idea what changes need to do to run my application on live?
I feel that you didn't changed your client id and client secret which u registered for appspot.com domain when deploying to appengine server. So you are getting this error.
Check your application logs from your GAE admin page.
Hopefully you'll find some info there to help debug.

Resources