I am Using GAE and java with JDO . I have the server side code and would like to run it from command prompt, rather than initiating from a browser (as its tedious to debug server side code by running the browser every time), but how do I do that ?, what would be the starting point (start a PersistenceManager, request thru port 8888 ) ?. I am looking for some guidance.
Following is my server side code, optimize is the method i would like to call from command line, where it needs to get data from the local app engine
#SuppressWarnings("serial")
public class OptimizerServiceImpl extends RemoteServiceServlet implements
OptimizerService {
public static void main(String args[])
{
System.out.println("comes in: ");
optimize();
}
public String optimize(ModelRunDTO moDto)
{
PersistenceManager pm = PMF.get().getPersistenceManager();
Data data = pm.getData(); // gets the data thru pm
// all my logic goes here......
}
Thanks alot.
To supplement Peter's answer above, if you're just trying to test and debug your code, you may want to use a unit test. This document explains local unit testing for Java App Engine. In particular, you'll be interested in writing datastore tests, which uses an in-memory implementation of the datastore (and flushes the contents between tests). Because these tests are based on JUnit, you can run them from the command line, or through the IDE of your choice.
The only way to invoke anything on GAE is via a HTTP request. You can make HTTP requets from command line (OS-specific), for example
wget http://yourapp.appspot.com/path
Related
I have been racking my brains on this for a few weeks now, trying different variations from Google Cloud service offerings but can't seem to find the proper one.
I have a python script with dependencies etc, that I have containerized, pushed, and deploy to GCR.
The script is a bot that connects to an external websocket receiving signals perpetually to then do other processing via API against another external service.
What would be the best service offering from Google Cloud to run this?
So far, I've tried:
GCR Web Service - requires listening service (:8080) which I do not provide in this use case, and, it scales down your service when there is no traffic so no go.
GCR Job Service - Seems like the next ideal option (no HTTP port requirement) - however, since the script (my entry point), upon launch, doesn't 'return' unless it quits, the service launch just allows it to run for a minute or so, until the jobs API declares it as 'failed' - basically, it is launching it via my entry point which just executes the script as if it was running locally and my script isn't meant to return anything.
To try and get around this, I went the google's recommended way and built a main.py with they standard boilerplate, and built it as a wrapper to act as a launcher for the actual script. I did this via a simple subprocess.Popen using their sample main.py as shown below.
main.py
import json
import os
import sys
import subprocess
# Retrieve Job-defined env vars
TASK_INDEX = os.getenv("CLOUD_RUN_TASK_INDEX", 0)
TASK_ATTEMPT = os.getenv("CLOUD_RUN_TASK_ATTEMPT", 0)
# Define main script
def main():
print(f"Starting Task #{TASK_INDEX}, Attempt #{TASK_ATTEMPT}...")
subprocess.Popen(["python3", "myscript.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print(f"Completed Task #{TASK_INDEX}.")
# Start script
if __name__ == "__main__":
try:
main()
except Exception as err:
message = f"Task #{TASK_INDEX}, " \
+ f"Attempt #{TASK_ATTEMPT} failed: {str(err)}"
print(json.dumps({"message": message, "severity": "ERROR"}))
sys.exit(1) # Retry Job Task by exiting the process
My thinking being, this would allow the job to execute my script and mark the job as completed, while the actual script remains running. Also, since subprocess.Popen sets its stdout and stderr to PIPE, my thinking is it would get caught by the google logging and I would see the output.
The job runs and marks it as succeed, however, I see no indication of the actual script executing anywhere.
I had similar issue with Google Cloud functions. Jobs seemed like an ideal option since I can run on their scheduler to make sure it is launching after saying, every hour (my script uses a lock file so it doesn't run again if running).
Am I just missing the point on how these cloud services run?
Do offerings like google cloud run jobs/functions, etc meant to execute jobs that return and quit until launched again by however scheduled?
Do I need to consider Google Computing engine as an option for this use case that is, a full running VM instead of stateless/serverless options?
I am trying to use this in a containerized, scalable as needed, fashion to both make my project portable and minimize costs as much as possible given the always running nature of the job.
Lastly, I know services like pythonanywhere as I am sure others, make this kinda stuff easier, but I would like to learn how to do this via standard cloud offerings like GCR, AWS, etc.
thanks for any insight / advice!
Cloud Run best fit is for HTTP Rest APIs serving (stateless services). There are also Jobs in beta.
One of the top feature of Run is that it scales to 0, when there are not requests to your service (your service instance gets totally destroyed).
If your bot needs to stay alive "for ever", Run is not for you... (Even if you can configure Run to keep at least one instance live).
I would consider instead AppEngine or Compute.
EDIT2: I have managed to get past the GlobalDatastoreConfig has already been set error. I managed to pinpoint all the locations that were getting called before the init function. They were in static space in some weird files.
I have now pointed ALL DatastoreServiceFactory.getDatastoreService() to a new static function I've created in a file called Const.java.
private static boolean hasInit = false;
public static DatastoreService getDatastoreService() {
if(!hasInit) {
try {
CloudDatastoreRemoteServiceConfig config = CloudDatastoreRemoteServiceConfig
.builder()
.appId(CloudDatastoreRemoteServiceConfig.AppId.create(CloudDatastoreRemoteServiceConfig.AppId.Location.US_CENTRAL, "gcp-project-id"))
.build();
CloudDatastoreRemoteServiceConfig.setConfig(config);
hasInit = true;
} catch (Exception ignore) {}
}
return DatastoreServiceFactory.getDatastoreService();
}
This returns no errors on the first initialisation. However, I am getting a new error now!
Dec 08, 2022 6:49:56 PM com.google.appengine.api.datastore.dev.LocalDatastoreService init
INFO: Local Datastore initialized:
Type: High Replication
Storage: C:\Users\user\dev\repo\Celbux\core\Funksi179_NSFAS_modules\classes\artifacts\Funksi179_NSFAS_modules_war_exploded\WEB-INF\appengine-generated\local_db.bin
Dec 08, 2022 6:49:56 PM com.google.appengine.api.datastore.dev.LocalDatastoreService load
INFO: Time to load datastore: 20 ms
2022-12-08 18:49:56.757:WARN:oejs.HttpChannel:qtp1681595665-26: handleException / java.io.IOException: com.google.apphosting.api.ApiProxy$CallNotFoundException: Can't make API call urlfetch.Fetch in a thread that is neither the original request thread nor a thread created by ThreadManager
2022-12-08 18:49:56.762:WARN:oejsh.ErrorHandler:qtp1681595665-26: Error page too large: 500 org.apache.jasper.JasperException: com.google.apphosting.api.ApiProxy$RPCFailedException: I/O error
Full stacktrace: https://pastebin.com/YQ2WvqzM
Pretty sure the first of the errors is invoked from this line:
DatastoreService ds = Const.getDatastoreService();
Key ConstantKey = KeyFactory.createKey("Constants", 1);
Entity Constants1 = ds.get(ConstantKey) // <-- This line.
EDIT1: I am not using Maven. Here are the .jars I have in WEB-INF/lib
appengine-api-1.0-sdk-1.9.84.jar
appengine-api-labs.jar
appengine-api-labs-1.9.76.jar
appengine-api-stubs-1.9.76.jar
appengine-gcs-client.jar
appengine-jsr107cache-1.9.76.jar
appengine-mapper.jar
appengine-testing-1.9.76.jar
appengine-tools-sdk-1.9.76.jar
charts4j-1.2.jar
guava-11.0.2.jar
javax.inject-1.jar
json-20190722.jar
Original Question:
The company that I'm working at have a legacy GCP codebase written in Java. This codebase uses the appengine-api-1.0-sdk.jar libary. Upon running this CloudDatastoreRemoteServiceConfig code in the very first place that our DatastoreService gets initialised, it says that the config has already been set.
If someone can shed light on how to get this outdated tech connected to the Cloud via localhost, I'll be most grateful!
web.xml
<filter>
<filter-name>NamespaceFilter</filter-name>
<filter-class>com.sintellec.funksi.Filterns</filter-class>
</filter>
<filter-mapping>
<filter-name>NamespaceFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
Code
public class Filterns implements javax.servlet.Filter {
public void init(FilterConfig filterConfig) {
try {
CloudDatastoreRemoteServiceConfig config = CloudDatastoreRemoteServiceConfig
.builder()
.appId(CloudDatastoreRemoteServiceConfig.AppId.create(CloudDatastoreRemoteServiceConfig.AppId.Location.US_CENTRAL, "gcp-project-id"))
.build();
CloudDatastoreRemoteServiceConfig.setConfig(config);
DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
} catch (Exception e) {
System.out.println(e);
return;
}
this.filterConfig = filterConfig;
}
}
I got this code snippet from here.
Was thinking a few ideas:
Perhaps there's GCP code that's called before our Java code which initialises the Local DB
Perhaps I need to set a global environment variable to point this old emulator to a Cloud Configuration instead
Only problem is I have no idea what to do from here, hoping someone has experience on the legacy Java library here.
To clarify; I am trying to get this outdated GCP Java codebase (appengine-api-1.0-sdk.jar) to connect to Cloud Datastore, NOT use the Local Datastore Emulator. This is so I can debug multiple applications that all access the same Cloud DB
It is very difficult to say especially with that amount of code and we can only guess but, as you indicated, probably some code is initializing your DataStore configuration, probably the SDK itself. You could try setting a breakpoint in the setConfig method of CloudDatastoreRemoteServiceConfig and analyze the call stack.
In any way, one think you could also try is not performing such as initialization in your code, delegating to Application Default Credentials the authentication of your client libraries.
For local development you have two options to configure such as Application Default Credentials.
On one hand, you can use user credentials, i.e., you can use the gcloud CLI to authenticate against GCP with an user with the required permissions to interact with the service, issuing the following command:
gcloud auth application-default login
Please, don't forget to revoke those credentials when you no longer need them:
gcloud auth application-default revoke
On the other, you can create a service account with the necessary permissions and a corresponding service account key, and download that key, a JSON file, to your local filesystem. See this for instructions specific to DataStore. Then, set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of the downloaded file with your service account key.
Again, a word of caution: take care of the downloaded service account key file and never put it under version control because anyone with that file could assume the permissions granted to the service account.
You code should work without further problem when running in GCP because probably you will be using a service that supports attaching a service account which means that Application Default Credentials are provided by the GCP services per se.
I am using play framework 1.2.7, gae module 1.6.0 and siena module 2.0.7 (also tested 2.0.6). This is a simple project that should run in play deployed on App Engine and connect to a MySQL database in Google Cloud SQL. My project runs fine locally but fails to connect to the database in production. Looking at the logs it looks like it is using the postgresql driver instead of the mysql one.
Application.conf
# db=mem
db.url=jdbc:google:mysql://PROJECT_ID:sienatest/sienatest
db.driver=com.mysql.jdbc.GoogleDriver
db.user=root
db.pass=root
This is the crash stack trace
play.Logger niceThrowable: Cannot connected to the database : null
java.lang.NullPointerException
at com.google.appengine.runtime.Request.process-a3b6145d1dbbd04d(Request.java)
at java.util.Hashtable.put(Hashtable.java:432)
at java.util.Properties.setProperty(Properties.java:161)
at org.postgresql.Driver.loadDefaultProperties(Driver.java:121)
at org.postgresql.Driver.access$000(Driver.java:47)
at org.postgresql.Driver$1.run(Driver.java:88)
at java.security.AccessController.doPrivileged(AccessController.java:63)
at org.postgresql.Driver.getDefaultProperties(Driver.java:85)
at org.postgresql.Driver.connect(Driver.java:231)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at play.modules.siena.GoogleSqlDBPlugin.onApplicationStart(GoogleSqlDBPlugin.java:103)
at play.plugins.PluginCollection.onApplicationStart(PluginCollection.java:525)
at play.Play.start(Play.java:533)
at play.Play.init(Play.java:305)
What is going on here? I am specifying the correct driver and url schema and it's using postgresql driver. Google Cloud SQL API access is enabled, the app is allowed to connect to the mysql instance, I am not using db=mem, ... I am stuck and can't figure out how to move forward! :-((
UPDATE: I thought I found the solution, but that was not the case. If I keep the %prod. prefix and create a war normally (or just don't define any DB properties), then the application will use Google DataStore instead of the Cloud SQL. If I create the war file adding --%prod at the end (or just delete the %prod. prefix in the application.conf), then it will keep failing to connect to the database showing the same initial error.
Any ideas please?
After being stuck for so long on this I just found the solution in no time after posting the question. Quite stupid actually.
The production environment properties in the application.conf file must be preceded by %prod. so the database config should read
%prod.db.url=jdbc:google:mysql://PROJECT_ID:sienatest/sienatest
%prod.db.driver=com.mysql.jdbc.GoogleDriver
%prod.db.user=root
%prod.db.pass=root
And everything runs fine.
EDIT: This is NOT the solution. The problem went away, but the app is using the DataStore instead of the Cloud SQL
At the end I ended doing a slight modification in play siena module source code and recompiling it.
In case anyone is interested, you will need to remove/comment/catch exception in this code around line 97 in GoogleSqlDBPlugin class:
// Try the connection
Connection fake = null;
try {
if (p.getProperty("db.user") == null) {
fake = DriverManager.getConnection(p.getProperty("db.url"));
} else {
fake = DriverManager.getConnection(p.getProperty("db.url"), p.getProperty("db.user"), p.getProperty("db.pass"));
}
} finally {
if (fake != null) {
fake.close();
}
}
For some reason the connection fails when initiated with DriverManager.getConnection() but it works when initiated with basicDatasource.getConnection(); which apparently is the way used by the module in the rest of the code. So if you delete the above block, and recompile the module everything will work as expected. If you are compiling with JDK 7, you will also need to implement public Logger getParentLogger() throws SQLFeatureNotSupportedException in the ProxyDriver inner class at the end of GoogleSqlDBPlugin file.
Strangely, I digged into the DriverManager.getConnection() and it looked like some postgresql driver is registered somehow, because otherwise I can't see why DriverManager.getConnection() would call to org.postgresql.Driver.connect().
I have a simple GAE application using JPA and eclipse plugin for GAE (sdk version 1.7.2).
I only have one simple entity mapped wit 2 properties : 1 Key key ; and one String name.
I have created only one class (taken from the "Datastore Callbacks" documentation of gae)
public class PostPutCallbacks {
static Logger logger = Logger.getLogger(PostPutCallbacks.class.getName());
#PostPut
public void collectSample(PutContext context) {
logger.fine("Finished putting " + context.getCurrentElement().getKey());
}
}
The database callback is not called.
I tried with several other annotations (#PrePut #PreGet) but they still didn't work .
I've also tried with the Datastore low leve API and the callback is still not called.
I searched the web to see if there is a configuration besides the annotation processing jar manualy added for eclipse but I didn't find anything..
Can anyone give me a hint?
I want to provide more information in this thread. If you are unable to see the datastore callbacks being triggered. Try this procedure as documented. You need to do some simple project set up at Eclipse in order to make it work. Hope this helps.
I jumped a little fast to stackoverflow..I had to restart eclipse and it worked
i followed the steps that are given for cobertura report and i generated report with following steps given in the URL Cobertura on Tomcat
but now my problem is to generate cobertura report without stopping tomcat
There are only 2 Ways to create a coverage data file.
Stop Tomcat
Execute a piece of code, that tells cobertura to write the file
Regarding the Second approach: You have to call this function yourself, after your tests run. You could (for example) put this code in a servlet (which you call at the end of your tests).
If you don't stop Tomcat or execute the function, you will not get a coverage data file.
This is from the cobertura FAQ
Cobertura only writes the coverage data file when the application server shuts down. We do not want to stop our application server after running our tests.
It is possible to instruct Cobertura to write the data file. One of
your classes should call the static method
net.sourceforge.cobertura.coveragedata.ProjectData.saveGlobalProjectData().
For example, you could add something like this to a "logout" method in
your web application:
try {
String className = "net.sourceforge.cobertura.coveragedata.ProjectData";
String methodName = "saveGlobalProjectData";
Class saveClass = Class.forName(className);
java.lang.reflect.Method saveMethod = saveClass.getDeclaredMethod(methodName, new Class[0]);
saveMethod.invoke(null,new Object[0]);
} catch (Throwable t) {}