I can update the live datastore using the remote API but is there something similar for the local datastore ? My data is in CSV format.
When I try to connect locally using below code
String username = "test";
String password = "test";
RemoteApiOptions options = new RemoteApiOptions().server("localhost", 8888).credentials(username, password);
RemoteApiInstaller installer = new RemoteApiInstaller();
installer.install(options);
I get an exception :
Exception in thread "main" java.net.UnknownHostException: http
The exception is thrown at line :
installer.install(options);
The local server is running, am I connecting correctly ? Do I need to start the local remote_api server separately ?
I finally got this to work through alot of searching. The dev url/password is XXXX/XXXX
Taken from here : https://groups.google.com/forum/?fromgroups=#!topic/google-appengine-java/1cQWn0UEoMc
I havent been able to find this specified anywhere in the google app engine documentation.
Related
EDIT2: I have managed to get past the GlobalDatastoreConfig has already been set error. I managed to pinpoint all the locations that were getting called before the init function. They were in static space in some weird files.
I have now pointed ALL DatastoreServiceFactory.getDatastoreService() to a new static function I've created in a file called Const.java.
private static boolean hasInit = false;
public static DatastoreService getDatastoreService() {
if(!hasInit) {
try {
CloudDatastoreRemoteServiceConfig config = CloudDatastoreRemoteServiceConfig
.builder()
.appId(CloudDatastoreRemoteServiceConfig.AppId.create(CloudDatastoreRemoteServiceConfig.AppId.Location.US_CENTRAL, "gcp-project-id"))
.build();
CloudDatastoreRemoteServiceConfig.setConfig(config);
hasInit = true;
} catch (Exception ignore) {}
}
return DatastoreServiceFactory.getDatastoreService();
}
This returns no errors on the first initialisation. However, I am getting a new error now!
Dec 08, 2022 6:49:56 PM com.google.appengine.api.datastore.dev.LocalDatastoreService init
INFO: Local Datastore initialized:
Type: High Replication
Storage: C:\Users\user\dev\repo\Celbux\core\Funksi179_NSFAS_modules\classes\artifacts\Funksi179_NSFAS_modules_war_exploded\WEB-INF\appengine-generated\local_db.bin
Dec 08, 2022 6:49:56 PM com.google.appengine.api.datastore.dev.LocalDatastoreService load
INFO: Time to load datastore: 20 ms
2022-12-08 18:49:56.757:WARN:oejs.HttpChannel:qtp1681595665-26: handleException / java.io.IOException: com.google.apphosting.api.ApiProxy$CallNotFoundException: Can't make API call urlfetch.Fetch in a thread that is neither the original request thread nor a thread created by ThreadManager
2022-12-08 18:49:56.762:WARN:oejsh.ErrorHandler:qtp1681595665-26: Error page too large: 500 org.apache.jasper.JasperException: com.google.apphosting.api.ApiProxy$RPCFailedException: I/O error
Full stacktrace: https://pastebin.com/YQ2WvqzM
Pretty sure the first of the errors is invoked from this line:
DatastoreService ds = Const.getDatastoreService();
Key ConstantKey = KeyFactory.createKey("Constants", 1);
Entity Constants1 = ds.get(ConstantKey) // <-- This line.
EDIT1: I am not using Maven. Here are the .jars I have in WEB-INF/lib
appengine-api-1.0-sdk-1.9.84.jar
appengine-api-labs.jar
appengine-api-labs-1.9.76.jar
appengine-api-stubs-1.9.76.jar
appengine-gcs-client.jar
appengine-jsr107cache-1.9.76.jar
appengine-mapper.jar
appengine-testing-1.9.76.jar
appengine-tools-sdk-1.9.76.jar
charts4j-1.2.jar
guava-11.0.2.jar
javax.inject-1.jar
json-20190722.jar
Original Question:
The company that I'm working at have a legacy GCP codebase written in Java. This codebase uses the appengine-api-1.0-sdk.jar libary. Upon running this CloudDatastoreRemoteServiceConfig code in the very first place that our DatastoreService gets initialised, it says that the config has already been set.
If someone can shed light on how to get this outdated tech connected to the Cloud via localhost, I'll be most grateful!
web.xml
<filter>
<filter-name>NamespaceFilter</filter-name>
<filter-class>com.sintellec.funksi.Filterns</filter-class>
</filter>
<filter-mapping>
<filter-name>NamespaceFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
Code
public class Filterns implements javax.servlet.Filter {
public void init(FilterConfig filterConfig) {
try {
CloudDatastoreRemoteServiceConfig config = CloudDatastoreRemoteServiceConfig
.builder()
.appId(CloudDatastoreRemoteServiceConfig.AppId.create(CloudDatastoreRemoteServiceConfig.AppId.Location.US_CENTRAL, "gcp-project-id"))
.build();
CloudDatastoreRemoteServiceConfig.setConfig(config);
DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
} catch (Exception e) {
System.out.println(e);
return;
}
this.filterConfig = filterConfig;
}
}
I got this code snippet from here.
Was thinking a few ideas:
Perhaps there's GCP code that's called before our Java code which initialises the Local DB
Perhaps I need to set a global environment variable to point this old emulator to a Cloud Configuration instead
Only problem is I have no idea what to do from here, hoping someone has experience on the legacy Java library here.
To clarify; I am trying to get this outdated GCP Java codebase (appengine-api-1.0-sdk.jar) to connect to Cloud Datastore, NOT use the Local Datastore Emulator. This is so I can debug multiple applications that all access the same Cloud DB
It is very difficult to say especially with that amount of code and we can only guess but, as you indicated, probably some code is initializing your DataStore configuration, probably the SDK itself. You could try setting a breakpoint in the setConfig method of CloudDatastoreRemoteServiceConfig and analyze the call stack.
In any way, one think you could also try is not performing such as initialization in your code, delegating to Application Default Credentials the authentication of your client libraries.
For local development you have two options to configure such as Application Default Credentials.
On one hand, you can use user credentials, i.e., you can use the gcloud CLI to authenticate against GCP with an user with the required permissions to interact with the service, issuing the following command:
gcloud auth application-default login
Please, don't forget to revoke those credentials when you no longer need them:
gcloud auth application-default revoke
On the other, you can create a service account with the necessary permissions and a corresponding service account key, and download that key, a JSON file, to your local filesystem. See this for instructions specific to DataStore. Then, set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of the downloaded file with your service account key.
Again, a word of caution: take care of the downloaded service account key file and never put it under version control because anyone with that file could assume the permissions granted to the service account.
You code should work without further problem when running in GCP because probably you will be using a service that supports attaching a service account which means that Application Default Credentials are provided by the GCP services per se.
I am trying publish to google PubSub from my .NET Core App. I have already created the PubSub and a Topic in Google PubSub. I downloaded the private key json file and included in my project. I am able to read the file and create the credential but I don't see a way to pass the credential to Google's PubSub Publisher client. I have looked at the GitHub Post but unfortunately I am not seeing a property named DefaultEndPoint.
Running my code generates an error which is looking for environment variable for credential. I do not want to set the environment variable for now and if there is no alternative I will try that.
Following is my code and let me know what I am doing wrong of is something that got changed recently.
var credential = GoogleCredential.FromFile("app-services.json");
PublisherClient publisher = PublisherClient.CreateAsync(topicName).Result;
Agreed that this isn't at obvious as it should be.
using Grpc.Auth;
var credential = GoogleCredential.FromFile("app-services.json");
var createSettings = new PublisherClient.ClientCreationSettings(
credentials: credential.ToChannelCredentials());
var publisher = await PublisherClient.CreateAsync(topicName,
clientCreationSettings: createSettings);
Creating a PublisherServiceApiClient without environment variables:
var credential = GoogleCredential.FromJson(#"{""your_credentials_json_itself"":""here""}");
var channel = new Grpc.Core.Channel(PublisherServiceApiClient.DefaultEndpoint.ToString(), credential.ToChannelCredentials());
var publisher = PublisherServiceApiClient.Create(channel);
Check out the "Passing the path to the service account key in code" (for .NET) here:
https://cloud.google.com/docs/authentication/production#obtaining_and_providing_service_account_credentials_manually
I'm trying to connect to a Cloud SQL database using GORM in golang.
db, _ = gorm.Open("mysql", "user:pass#cloudsql(connection:name:example)/")
if err != nil {
log.Println(err)
//panic(err)
}
When I attempt to serve the app
goapp serve appengine/
I get a runtime error
ERROR 2017-02-19 20:48:05,436 http_runtime.py:396] bad runtime process port ['\r\n']
Which I found was related to the database migration
db.AutoMigrate(&models.Event{})
If I remove the AutoMigrate, the runtime process port error goes away. However whenever I access a route (ie /events) that does a database query, the connection gets dropped, a 404 page is thrown, and an error message is logged sql: database is closed
When I run the app locally by building the package go build && ./appname and using a local MySQL server, it works fine.
Can someone please tell me how to connect to a Cloud SQL database using Go's GORM framework and App Engine?
This is due to the call to log.New in this file: https://github.com/jinzhu/gorm/blob/master/logger.go#L15
This anwser explain why dev_appserver.py gets it: https://stackoverflow.com/a/24112953/4266494
To disable this, you can either disable all GORM logging:
db.LogMode(false)
Or use an adapter the logger output: https://github.com/benguild/GAEBridge/blob/master/log/debugLevel.go
db.SetLogger(NewDebugLogger(nil)) // on application scope
db.SetLogger(NewDebugLogger(appengine.NewContext(req))) // on request scope
I'm setting a new logger with the real context
This is the only workaround I found to avoid crashes while keeping some logs, it could be awesome if one of you had a real one.
I am using objectify 4 to write to the HRD datastore. Everything works fine in unit tests and running the application in devserver or production.
But when I try connect using the REMOTE API to the devserver datastore, an error is thrown when the code starts a XG transaction. While connecting with the Remote API, it seems to think that HRD is not enabled.
This is how I connect ...
public static void main(String[] args) {
RemoteApiOptions options = new RemoteApiOptions().server("localhost", 8888).credentials("foo", "bar");
//options = options.
RemoteApiInstaller installer = new RemoteApiInstaller();
StoredUser storedUser = null;
try {
installer.install(options);
ObjectifyInitializer.register();
storedUser = new StoredUserDao().loadStoredUser(<KEY>);
log.info("found user : " + storedUser.getEmail());
// !!! ERROR !!!
new SomeOtherDao().doSomeDataManipulationInTransaction();
} catch (Throwable e) {
e.printStackTrace();
} finally {
ObjectifyFilter.complete();
installer.uninstall();
}
}
When new SomeOtherDao().doSomeDataManipulationInTransaction() starts a transactions on multiple entity groups I get the error thrown :
transactions on multiple entity groups only allowed in High Replication applications
How can I tell the remote api that this is a HRD environment ?
If your application is using the High Replication Datastore, add an
explicit s~ prefix (or e~ prefix if your application is located in the
European Union) to the app id
For Java version, add this prefix in the application tag in the appengine-web.xml and then deploy the version where you have activated the remote_api servlet
Example
<application>myappid</application>
become
<application>s~myappid</application>
Source: https://developers.google.com/appengine/docs/python/tools/uploadingdata#Python_Setting_up_remote_api
I had 'unapplied job percentage' set to 0 and transactions using the remote api failed as if the devserver was running with Master/Slave and not HRD. Raising the 'unapplied job percentage' above zero fixed the problem.
I am using play framework 1.2.7, gae module 1.6.0 and siena module 2.0.7 (also tested 2.0.6). This is a simple project that should run in play deployed on App Engine and connect to a MySQL database in Google Cloud SQL. My project runs fine locally but fails to connect to the database in production. Looking at the logs it looks like it is using the postgresql driver instead of the mysql one.
Application.conf
# db=mem
db.url=jdbc:google:mysql://PROJECT_ID:sienatest/sienatest
db.driver=com.mysql.jdbc.GoogleDriver
db.user=root
db.pass=root
This is the crash stack trace
play.Logger niceThrowable: Cannot connected to the database : null
java.lang.NullPointerException
at com.google.appengine.runtime.Request.process-a3b6145d1dbbd04d(Request.java)
at java.util.Hashtable.put(Hashtable.java:432)
at java.util.Properties.setProperty(Properties.java:161)
at org.postgresql.Driver.loadDefaultProperties(Driver.java:121)
at org.postgresql.Driver.access$000(Driver.java:47)
at org.postgresql.Driver$1.run(Driver.java:88)
at java.security.AccessController.doPrivileged(AccessController.java:63)
at org.postgresql.Driver.getDefaultProperties(Driver.java:85)
at org.postgresql.Driver.connect(Driver.java:231)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at play.modules.siena.GoogleSqlDBPlugin.onApplicationStart(GoogleSqlDBPlugin.java:103)
at play.plugins.PluginCollection.onApplicationStart(PluginCollection.java:525)
at play.Play.start(Play.java:533)
at play.Play.init(Play.java:305)
What is going on here? I am specifying the correct driver and url schema and it's using postgresql driver. Google Cloud SQL API access is enabled, the app is allowed to connect to the mysql instance, I am not using db=mem, ... I am stuck and can't figure out how to move forward! :-((
UPDATE: I thought I found the solution, but that was not the case. If I keep the %prod. prefix and create a war normally (or just don't define any DB properties), then the application will use Google DataStore instead of the Cloud SQL. If I create the war file adding --%prod at the end (or just delete the %prod. prefix in the application.conf), then it will keep failing to connect to the database showing the same initial error.
Any ideas please?
After being stuck for so long on this I just found the solution in no time after posting the question. Quite stupid actually.
The production environment properties in the application.conf file must be preceded by %prod. so the database config should read
%prod.db.url=jdbc:google:mysql://PROJECT_ID:sienatest/sienatest
%prod.db.driver=com.mysql.jdbc.GoogleDriver
%prod.db.user=root
%prod.db.pass=root
And everything runs fine.
EDIT: This is NOT the solution. The problem went away, but the app is using the DataStore instead of the Cloud SQL
At the end I ended doing a slight modification in play siena module source code and recompiling it.
In case anyone is interested, you will need to remove/comment/catch exception in this code around line 97 in GoogleSqlDBPlugin class:
// Try the connection
Connection fake = null;
try {
if (p.getProperty("db.user") == null) {
fake = DriverManager.getConnection(p.getProperty("db.url"));
} else {
fake = DriverManager.getConnection(p.getProperty("db.url"), p.getProperty("db.user"), p.getProperty("db.pass"));
}
} finally {
if (fake != null) {
fake.close();
}
}
For some reason the connection fails when initiated with DriverManager.getConnection() but it works when initiated with basicDatasource.getConnection(); which apparently is the way used by the module in the rest of the code. So if you delete the above block, and recompile the module everything will work as expected. If you are compiling with JDK 7, you will also need to implement public Logger getParentLogger() throws SQLFeatureNotSupportedException in the ProxyDriver inner class at the end of GoogleSqlDBPlugin file.
Strangely, I digged into the DriverManager.getConnection() and it looked like some postgresql driver is registered somehow, because otherwise I can't see why DriverManager.getConnection() would call to org.postgresql.Driver.connect().