How can I externalise my selenium setup in order to make my selenium tests configurable - selenium-webdriver

I would like to externalise my selenium tests setting in order to make them more configurable.
I would like to externalise my testURL and my node URLS.
Here is my code :
public void setup () throws MalformedURLException
{ //the URL of the application to be tested
TestURL = "http://frstmwarwebsrv2.orsyptst.com:9000";
//Hub URL
BaseURL = "http://10.2.128.126";
//Node1 URL
winURL = "http://10.2.128.120:5556/wd/hub";
//Node2 URL
androidURL ="http://10.2.128.120:5555/wd/hub";
At the moment I have added this setup function in every test I would like to have it in an XML file for an example in order to make it configurable, any suggestions?
Thanks
Thanks for your help
Update :
Here is what i did so far :
Added a config.properties file with this content :
# This is my test.properties file
AppURL = http://************
HubURL= http://*****************
WinURL= http://*********/wd/hub
AndroidURL =
iOSURL
And created a classe to read properties :
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.Enumeration;
import java.util.Properties;
public class ReadPropertiesFile {
public static void main(String[] args) {
try {
File file = new File("config.properties");
FileInputStream fileInput = new FileInputStream(file);
Properties properties = new Properties();
properties.load(fileInput);
fileInput.close();
Enumeration enuKeys = properties.keys();
while (enuKeys.hasMoreElements()) {
String key = (String) enuKeys.nextElement();
String value = properties.getProperty(key);
System.out.println(key + ": " + value);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
when running this i get this error :
java.io.FileNotFoundException: config.properties (The system cannot find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(Unknown Source)
at ReadPropertiesFile.main(ReadPropertiesFile.java:15)
my properties file is under src folder

Two basic ways you could do this are:
Pass in JVM argument and access it using System.getProperty(...)
Externalize your configuration in to properties files, like here
I recently implemented the second one in my Selenium tests and can expand this answer to give more details if you need them.

In my tests, I resolved it that I created Java class called Environment to store information about given Environment:
Few snippets of code:
public enum NameOfEnvironment {
SYSTEMTEST, ACCEPTANCE
}
stores the Name of given Environment :)
public String getBaseUrl() {
switch (actualEnvironment) {
case SYSTEMTEST: {
baseUrl = getPrefix() + "172.23.32.251:9092/pages/index.html";
break;
}
will return me the URL to the environment. And on beginning of the test I have something like this:
public static final Environment USED_ENVIRONMENT = new Environment(Environment.NameOfEnvironment.SYSTEMTEST);
And later on I just call USED_ENVIRONMENT.getBaseUrl() which will return me the link which is being actual for current run
Btw, to fill in the blanks, here is the constructor f the class
public Environment(NameOfEnvironment env) {
this.actualEnvironment = env;
}

Related

How to decode base64 string before use in camel context?

I'm using a bean in my Camel Context with the class org.apache.commons.configuration.DatabaseConfiguration to get as key-value the properties saved in my DB. So, the values coming in base64, and I need to decode the values to use them on my routes. How can I do it?
Like Ocp and Kubernetes tools do it inside . It is transferred to the application by decoding the encoded text. You can still do this in camel. this example about encrypt and decrypt using jasypt lib from camel book. you must implement a basic PropertiesParser.class for base64 . PropertiesParser a basic a interface . full example https://github.com/camelinaction/camelinaction2/blob/master/chapter14/configuration/src/test/java/camelinaction/SecuringConfigTest.java
JasyptPropertiesParser jasypt = new JasyptPropertiesParser();
// and set the master password
jasypt.setPassword("supersecret");
// we can avoid keeping the master password in plaintext in the application
// by referencing a environment variable
// export CAMEL_ENCRYPTION_PASSWORD=supersecret
// jasypt.setPassword("sysenv:CAMEL_ENCRYPTION_PASSWORD");
// setup the properties component to use the production file
PropertiesComponent prop = context.getComponent("properties", PropertiesComponent.class);
prop.setLocation("classpath:rider-test.properties");
// and use the jasypt properties parser so we can decrypt values
prop.setPropertiesParser(jasypt);
return context;
You could try using Base64 Decoder from java.util package and use it with bean, exchange function or processor. Use Base64.getDecoder() or Base64.getUrlDecoder() to obtain decoder then use the decoder to decode base64 string to byte array which you can then convert to String with camels .convertBodyTo(String.class) or just creating a new instance with new String(bytes, StandardCharsets.UTF_8);
package com.example;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import java.util.Base64.Decoder;
import org.apache.camel.RoutesBuilder;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.mock.MockEndpoint;
import org.apache.camel.test.junit4.CamelTestSupport;
import org.junit.Test;
public class Base64tests extends CamelTestSupport {
#Test
public void Test() throws Exception {
MockEndpoint resultMockEndpoint = getMockEndpoint("mock:result");
resultMockEndpoint.expectedMessageCount(1);
resultMockEndpoint.message(0).body().isEqualTo("Hello World");
template.sendBody("direct:decodeBase64", "SGVsbG8gV29ybGQ=");
resultMockEndpoint.assertIsSatisfied();
}
#Override
protected RoutesBuilder createRouteBuilder() throws Exception {
return new RouteBuilder(){
#Override
public void configure() throws Exception {
from("direct:decodeBase64")
.routeId("decodeBase64")
.bean(new Base64Decoder())
.log("${body}")
.convertBodyTo(String.class)
.to("mock:result");
}
};
}
}
class Base64Decoder{
public byte[] decode(String encodedValue){
return Base64.getDecoder().decode(encodedValue);
}
}
Using exchange function:
.setBody().exchange( e -> {
String encodedBody = e.getMessage()
.getBody(String.class);
return Base64.getDecoder()
.decode(encodedBody);
})

Micronaut 3: How to use PubSubEmulatorContainer

Update: Link to repo is moved to answer because repo is now updated with code from answer below.
Problem description
Current code is working, but it is using gcloud beta emulators pubsub from google/cloud-sdk for integration tests.
Integration tests are slow due to the size of the google/cloud-sdk image
pubsub emulator has to run on a fixed port, there seems to be no way to tell Micronaut which port the emulator is running on
I'll need to set the following environment variable in maven-surefire-plugin.
<environmentVariables>
<PUBSUB_EMULATOR_HOST>localhost:8085</PUBSUB_EMULATOR_HOST>
</environmentVariables>
How this can be done in Spring Boot
According to Test Containers | Gcloud Module, the correct way of implementing integration tests with PubSubEmulatorContainer in Spring Boot is like this:
https://github.com/saturnism/testcontainers-gcloud-examples/blob/main/springboot/pubsub-example/src/test/java/com/example/springboot/pubsub/PubSubIntegrationTests.java
This will bring up the container on a random port, and that is possible because of DynamicPropertyRegistry in Spring. It seems that Micronaut is missing this possibility.
Doc: https://www.testcontainers.org/modules/gcloud/
I'm looking for a working example of a JUnit5 or Spock integration test implemented in Micronaut 3.x that is using PubSubEmulatorContainer like described in the above doc.
Related doc: https://micronaut-projects.github.io/micronaut-gcp/latest/guide/#emulator
There are some comments on GitHub around configuring TransportChannelProvider. I'm able to inject an instance and inspect it, but I still haven't found out exactly what to do.
These are the closest leads so far:
https://github.com/micronaut-projects/micronaut-gcp/issues/257
https://github.com/micronaut-projects/micronaut-gcp/pull/259
TL;DR
We'll need to start the testcontainer first, get emulator host address and then call ApplicationContext.run like this:
applicationContext = ApplicationContext.run(
["pubsub.emulator.host": emulatorHost])
Small Github repo with example code: https://github.com/roar-skinderviken/pubsub-emulator-demo
Long answer with code
I finally managed to make a working solution using Micronaut 3.0.2 and Spock. A related Micronaut PR got me on track, together with this article: Micronaut Testing Best Practices https://objectcomputing.com/files/9815/9259/7089/slide_deck_Micronaut_Testing_Best_Practices_webinar.pdf
First a PubSubEmulator class (Groovy)
package no.myproject.testframework.testcontainers
import org.testcontainers.containers.PubSubEmulatorContainer
import org.testcontainers.utility.DockerImageName
class PubSubEmulator {
static PubSubEmulatorContainer pubSubEmulatorContainer
static init() {
if (pubSubEmulatorContainer == null) {
pubSubEmulatorContainer = new PubSubEmulatorContainer(
DockerImageName.parse("gcr.io/google.com/cloudsdktool/cloud-sdk:emulators"))
pubSubEmulatorContainer.start()
}
}
}
Then a fixture for PubSubEmulator (Groovy)
package no.myproject.testframework.testcontainers
trait PubSubEmulatorFixture {
Map<String, Object> getPubSubConfiguration() {
if (PubSubEmulator.pubSubEmulatorContainer == null || !PubSubEmulator.pubSubEmulatorContainer.isRunning()) {
PubSubEmulator.init()
}
[
"pubsub.emulator-host": PubSubEmulator.pubSubEmulatorContainer.getEmulatorEndpoint()
]
}
}
Then a specification class (Groovy) that starts the container, creates a topic and a subscription.
The clue here is to pass in pubsub.emulator.host as part of the configuration when calling ApplicationContext.run.
Remaining part of the code is very similar to the Spring Boot example I linked to in my question.
package no.myproject.testframework
import com.google.api.gax.core.NoCredentialsProvider
import com.google.api.gax.grpc.GrpcTransportChannel
import com.google.api.gax.rpc.FixedTransportChannelProvider
import com.google.cloud.pubsub.v1.SubscriptionAdminClient
import com.google.cloud.pubsub.v1.SubscriptionAdminSettings
import com.google.cloud.pubsub.v1.TopicAdminClient
import com.google.cloud.pubsub.v1.TopicAdminSettings
import com.google.pubsub.v1.ProjectSubscriptionName
import com.google.pubsub.v1.PushConfig
import com.google.pubsub.v1.TopicName
import io.grpc.ManagedChannelBuilder
import io.micronaut.context.ApplicationContext
import no.myproject.configuration.GcpConfigProperties
import no.myproject.configuration.PubSubConfigProperties
import no.myproject.testframework.testcontainers.PubSubEmulatorFixture
import spock.lang.AutoCleanup
import spock.lang.Shared
import spock.lang.Specification
abstract class PubSubSpecification extends Specification
implements PubSubEmulatorFixture, EnvironmentFixture {
#AutoCleanup
#Shared
EmbeddedServer embeddedServer
#AutoCleanup
#Shared
ApplicationContext applicationContext
def setupSpec() {
// start the pubsub emulator
def emulatorHost = getPubSubConfiguration().get("pubsub.emulator-host")
// start a temporary applicationContext in order to read config
// keep any pubsub subscriptions out of context at this stage
applicationContext = ApplicationContext.run()
def gcpConfigProperties = applicationContext.getBean(GcpConfigProperties)
def pubSubConfigProperties = applicationContext.getBean(PubSubConfigProperties)
def channel = ManagedChannelBuilder.forTarget("dns:///" + emulatorHost)
.usePlaintext()
.build()
def channelProvider =
FixedTransportChannelProvider.create(GrpcTransportChannel.create(channel))
// START creating topic
def topicAdminClient =
TopicAdminClient.create(
TopicAdminSettings.newBuilder()
.setCredentialsProvider(NoCredentialsProvider.create())
.setTransportChannelProvider(channelProvider)
.build())
def topic = TopicName.of(
gcpConfigProperties.getProjectId(),
pubSubConfigProperties.getTopicName())
try {
topicAdminClient.createTopic(topic)
} catch (AlreadyExistsException) {
// this is fine, already created
topicAdminClient.getTopic(topic)
}
// START creating subscription
pubSubConfigProperties.getSubscriptionNames().forEach(it -> {
def subscription =
ProjectSubscriptionName.of(gcpConfigProperties.getProjectId(), it)
def subscriptionAdminClient =
SubscriptionAdminClient.create(
SubscriptionAdminSettings.newBuilder()
.setTransportChannelProvider(channelProvider)
.setCredentialsProvider(NoCredentialsProvider.create())
.build())
try {
subscriptionAdminClient
.createSubscription(
subscription,
topic,
PushConfig.getDefaultInstance(),
100)
System.out.println("Subscription created " + subscriptionAdminClient.getSubscription(subscription))
} catch (AlreadyExistsException) {
// this is fine, already created
subscriptionAdminClient.getSubscription(subscription)
}
})
channel.shutdown()
// stop the temporary applicationContext
applicationContext.stop()
// start the actual applicationContext
embeddedServer = ApplicationContext.run(
EmbeddedServer,
[
'spec.name' : "PubSubEmulatorSpec",
"pubsub.emulator.host": emulatorHost
],
environments)
applicationContext = embeddedServer.applicationContext
}
}
Then a factory class (Groovy) for mocking credentials
package no.myproject.pubsub
import com.google.auth.oauth2.AccessToken
import com.google.auth.oauth2.GoogleCredentials
import io.micronaut.context.annotation.Factory
import io.micronaut.context.annotation.Replaces
import io.micronaut.context.annotation.Requires
import javax.inject.Singleton
#Factory
#Requires(property = 'spec.name', value = 'PubSubEmulatorSpec')
class EmptyCredentialsFactory {
#Singleton
#Replaces(GoogleCredentials)
GoogleCredentials mockCredentials() {
return GoogleCredentials.create(new AccessToken("", new Date()))
}
}
And finally, a Spock test spec.
package no.myproject.pubsub
import no.myproject.testframework.PubSubSpecification
import java.util.stream.IntStream
class PubSubIntegrationSpec extends PubSubSpecification {
def NUMBER_OF_MESSAGES_IN_TEST = 5
def DELAY_IN_MILLISECONDS_PER_MSG = 100
def "when a number of messages is sent, same amount of messages is received"() {
given:
def documentPublisher = applicationContext.getBean(DocumentPublisher)
def listener = applicationContext.getBean(IncomingDocListenerWithAck)
def initialReceiveCount = listener.getReceiveCount()
when:
IntStream.rangeClosed(1, NUMBER_OF_MESSAGES_IN_TEST)
.forEach(it -> documentPublisher.send("Hello World!"))
// wait a bit in order to let all messages propagate through the queue
Thread.sleep(NUMBER_OF_MESSAGES_IN_TEST * DELAY_IN_MILLISECONDS_PER_MSG)
then:
NUMBER_OF_MESSAGES_IN_TEST == listener.getReceiveCount() - initialReceiveCount
}
}
The chosen answer is a good deal more complicated than necessary, and it also contains numerous typos. A better answer can be found via the Micronaut GCP codebase itself, with the key bit being:
class IntegrationTestSpec extends Specification {
static CONTAINER_PORT = -1
static CredentialsProvider CREDENTIALS_PROVIDER
static TransportChannelProvider TRANSPORT_CHANNEL_PROVIDER
static PubSubResourceAdmin pubSubResourceAdmin
static GenericContainer pubSubContainer = new GenericContainer("google/cloud-sdk:292.0.0")
.withCommand("gcloud", "beta", "emulators", "pubsub", "start", "--project=test-project",
"--host-port=0.0.0.0:8085")
.withExposedPorts(8085)
.waitingFor(new LogMessageWaitStrategy().withRegEx("(?s).*Server started, listening on.*"))
static {
pubSubContainer.start()
CONTAINER_PORT = pubSubContainer.getMappedPort(8085)
CREDENTIALS_PROVIDER = NoCredentialsProvider.create()
def host = "localhost:" + IntegrationTest.CONTAINER_PORT
ManagedChannel channel = ManagedChannelBuilder.forTarget(host).usePlaintext().build()
TRANSPORT_CHANNEL_PROVIDER =
FixedTransportChannelProvider.create(GrpcTransportChannel.create(channel))
pubSubResourceAdmin = new PubSubResourceAdmin(TRANSPORT_CHANNEL_PROVIDER, CREDENTIALS_PROVIDER)
}
}
You'd then just extend that class anywhere you wanted to make use of PubSub. The following is slightly cleaner example that I came up with which manages creating a topic as well during test startup:
#Slf4j
abstract class PubSubSpec extends Specification implements TestPropertyProvider {
static final String cloudSdkName = System.getenv('CLOUD_SDK_IMAGE') ?: "gcr.io/google.com/cloudsdktool/cloud-sdk:emulators"
static final DockerImageName cloudSdkImage = DockerImageName.parse(cloudSdkName)
static final PubSubEmulatorContainer pubsubEmulator = new PubSubEmulatorContainer(cloudSdkImage)
static {
pubsubEmulator.start()
ManagedChannel channel = ManagedChannelBuilder.forTarget(pubsubEmulator.getEmulatorEndpoint()).usePlaintext().build()
try {
TransportChannelProvider channelProvider = FixedTransportChannelProvider.create(GrpcTransportChannel.create(channel))
CredentialsProvider credentialsProvider = NoCredentialsProvider.create()
TopicAdminClient topicClient = TopicAdminClient.create(
TopicAdminSettings.newBuilder()
.setTransportChannelProvider(channelProvider)
.setCredentialsProvider(credentialsProvider)
.build()
)
TopicName topicName = TopicName.of("project-id", "project-topic")
topicClient.createTopic(topicName)
} finally {
channel.shutdown()
}
}
#Override
Map<String, String> getProperties() {
[
"pubsub.emulator.host": pubsubEmulator.getEmulatorEndpoint()
]
}
}

How to download a file with attribute(href) in chrome using selenium webdriver via js executor

I want to download a file in Selenium web driver in C#.
i have the url in the web element as the attribute href. With that url, i have to download the file through javascript executor.
I tried with js executescript to get the file in the form of bytes and then converting it to pdf file and storing in my desired location. But no luck
IWebDriver driver = new ChromeDriver();
driver.Navigate().GoToUrl("https://www.win-rar.com/predownload.html?&L=0");
string linkVal = driver.FindElement(By.LinkText("Download WinRAR")).GetAttribute("href");
var href = "https://www.win-rar.com/fileadmin/winrar-versions/winrar/winrar-x64-571.exe";
IJavaScriptExecutor js = (IJavaScriptExecutor)driver;
/// Object reposne = js.ExecuteScript("arguments[0].setAttribute(arguments[1],arguments[2])", linkVal, "href", "https://www.win-rar.com/postdownload.html?&L=0");
String script = "document.querySelector('a[href*=\"/print/\"]').setAttribute('download','name-of-the-download-file-recommend-guid-or-timestamp.pdf');";
///Object reposne = js.ExecuteAsyncScript(script , href);
var bytes = Convert.FromBase64String(reposne.ToString());
File.WriteAllBytes("F:\\file.exe", bytes);
I'm actually using RestSharp.
Like so:
public FileInfo DownloadFile(string downloadUrl)
{
RestClient rest = new RestClient();
RestRequest request = new RestRequest(downloadUrl);
byte[] downloadedfile = rest.DownloadData(request);
string tempFilePath = Path.GetTempFileName();
File.WriteAllBytes(tempFilePath, downloadedfile);
return new FileInfo(tempFilePath);
}
Consider not creating a rest client for each download request.
The RestClient also supports a base URL for convenience.
Hope this is what you wanted.
I have tested your scenario using Java, Selenium and TestNG. Where you can save the file in the root folder directory
import org.testng.annotations.Test;
import org.testng.annotations.BeforeMethod;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.testng.Assert;
import org.testng.annotations.AfterMethod;
import java.io.File;
import java.util.HashMap;
import java.util.Map;
import java.util.UUID;
public class FileDownload {
File folder;
WebDriver driver;
#SuppressWarnings("deprecation")
#BeforeMethod
public void setUp() {
//a random folder will be created 88998-98565-09792-783727-826233
folder = new File(UUID.randomUUID().toString());
folder.mkdir();
//Chrome Driver
System.setProperty("webdriver.chrome.driver", "---Your Chrome Driver or chromedriver.exe URL;--");
ChromeOptions options = new ChromeOptions();
Map<String, Object> prefs = new HashMap<String, Object>();
//When you click on download, it will ignore the popups
prefs.put("profile.default_content_settings.popups", 0);
prefs.put("download.default_directory", folder.getAbsolutePath());
options.setExperimentalOption("prefs", prefs);
DesiredCapabilities cap = DesiredCapabilities.chrome();
cap.setCapability(ChromeOptions.CAPABILITY, options);
driver = new ChromeDriver(cap);
}
#Test
public void downloadFileTest() throws InterruptedException{
driver.get("https://www.win-rar.com/predownload.html?&L=0");
driver.findElement(By.linkText("Download WinRAR")).click();
/* Depending on your download speed you have to increase or decrease the sleep value, if you give less than download time it will give error as: java.lang.AssertionError: expected [true] but found [false] */
Thread.sleep(5000);
File listOfFiles[] = folder.listFiles();
Assert.assertTrue(listOfFiles.length>0);
for(File file: listOfFiles){
Assert.assertTrue(file.length()>0);
}
}
#AfterMethod
public void tearDown() {
driver.quit();
/* You can comment below "for loop": for testing purpose where you can see the folder and file download, if you uncomment it will delete the file and folder: useful for regression you wont end up having so many files in the root folder */
for(File file: folder.listFiles()){
file.delete();
}
folder.delete();
}
}
If you are just trying to download the file, all you really need is this.
driver.FindElement(By.Id("download-button")).Click();
This will save it in your default location for your browser.
or in js executor
WebElement element = driver.findElement(By.id("download-button"));
JavascriptExecutor executor = (JavascriptExecutor)driver;
executor.executeScript("arguments[0].click();", element);
To go to a specific location, try adding chrome options:
in C#
ChromeOptions options = new ChromeOptions();
options.AddArguments("--disable-extensions");
options.AddArguments("disable-infobars");
options.AddUserProfilePreference("download.default_directory", "My Path");
options.AddUserProfilePreference("profile.password_manager_enabled", false);
options.AddUserProfilePreference("credentials_enable_service", false);
_Driver = new ChromeDriver(#"DriverPath"), options);
_wait = new WebDriverWait(_webDriver, TimeSpan.FromSeconds(10));
_Driver.Manage().Window.Maximize();
as a last resort, just move the file with:
public class MoveMyFile
{
static void Main()
{
string sourceFile = #"C:\My\File\Path\file.txt";
string destinationFile = #"C:\My\New\Path\file.txt";
// Move my file to a new home!
System.IO.File.Move(sourceFile, destinationFile);
}
}

Datastore export logic in Java

Thankfully, Google announced the export logic from cloud Datastore. I would like to set up schedule-export in my platform. However, it's not Python, but Java. So I need to use cron.xml and Java logic to design this logic.
Is there any reference to design Datastore export logic (cloud_datastore_admin.py) in Java? Especially, I need to transform this part in Java
app = webapp2.WSGIApplication(
[
('/cloud-datastore-export', Export),
], debug=True)
https://cloud.google.com/datastore/docs/schedule-export
You can create the skeleton for App Egnine by following these instructions.
Once you have the skeleton, add something like this to handle export requests:
CloudDatastoreExport.java
package com.google.example.datastore;
import com.google.appengine.api.appidentity.AppIdentityService;
import com.google.appengine.api.appidentity.AppIdentityServiceFactory;
import com.google.apphosting.api.ApiProxy;
import com.google.common.io.CharStreams;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.logging.Logger;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.json.JSONArray;
import org.json.JSONObject;
import org.json.JSONTokener;
#WebServlet(name = "CloudDatastoreExport", value = "/cloud-datastore-export")
public class CloudDatastoreExport extends HttpServlet {
private static final Logger log = Logger.getLogger(CloudDatastoreExport.class.getName());
#Override
public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
// Verify outputURL parameter
String outputUrlPrefix = request.getParameter("output_url_prefix");
if (outputUrlPrefix == null || !outputUrlPrefix.matches("^gs://.*")) {
response.setStatus(HttpServletResponse.SC_CONFLICT);
response.setContentType("text/plain");
response.getWriter().println("Error: Must provide a valid output_url_prefix.");
} else {
// Get project ID
String projectId = ApiProxy.getCurrentEnvironment().getAppId();
// Remove partition information to get plain app ID
String appId = projectId.replaceFirst("(.*~)", "");
// Get access token
ArrayList<String> scopes = new ArrayList<String>();
scopes.add("https://www.googleapis.com/auth/datastore");
final AppIdentityService appIdentity = AppIdentityServiceFactory.getAppIdentityService();
final AppIdentityService.GetAccessTokenResult accessToken =
appIdentity.getAccessToken(scopes);
// Read export parameters
// If output prefix does not end with slash, add a timestamp
if (!outputUrlPrefix.substring(outputUrlPrefix.length() - 1).contentEquals("/")) {
String timeStamp = new SimpleDateFormat("yyyyMMddHHmmss").format(new Date());
outputUrlPrefix = outputUrlPrefix + "/" + timeStamp + "/";
}
String[] namespaces = request.getParameterValues("namespace_id");
String[] kinds = request.getParameterValues("kind");
// Build export request
JSONObject exportRequest = new JSONObject();
exportRequest.put("output_url_prefix", outputUrlPrefix);
JSONObject entityFilter = new JSONObject();
if (kinds != null) {
JSONArray kindsJSON = new JSONArray(kinds);
entityFilter.put("kinds", kinds);
}
if (namespaces != null) {
JSONArray namespacesJSON = new JSONArray(namespaces);
entityFilter.put("namespaceIds", namespacesJSON);
}
exportRequest.put("entityFilter", entityFilter);
URL url = new URL("https://datastore.googleapis.com/v1/projects/" + appId + ":export");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.addRequestProperty("Content-Type", "application/json");
connection.addRequestProperty("Authorization", "Bearer " + accessToken.getAccessToken());
OutputStreamWriter writer = new OutputStreamWriter(connection.getOutputStream());
exportRequest.write(writer);
writer.close();
if (connection.getResponseCode() == HttpURLConnection.HTTP_OK) {
JSONTokener exportResponseTokens = new JSONTokener(connection.getInputStream());
JSONObject exportResponse = new JSONObject(exportResponseTokens);
response.setContentType("text/plain");
response.getWriter().println("Export started:\n" + exportResponse.toString(4));
} else {
InputStream s = connection.getErrorStream();
InputStreamReader r = new InputStreamReader(s, StandardCharsets.UTF_8);
String errorMessage =
String.format(
"got error (%d) response %s from %s",
connection.getResponseCode(), CharStreams.toString(r), connection.toString());
log.warning(errorMessage);
response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
response.setContentType("text/plain");
response.getWriter().println("Failed to initiate export.");
}
}
}
}
You can use the same cron.yaml from the docs:
cron:
- description: "Daily Cloud Datastore Export"
url: /cloud-datastore-export?namespace_id=&output_url_prefix=gs://BUCKET_NAME[/NAMESPACE_PATH]
target: cloud-datastore-admin
schedule: every 24 hours
Use gcloud to deploy the cron job:
gcloud app deploy cron.yaml
Make sure you complete this part to give GAE export and bucket permissions or else
you'll get permission denied errors:
https://cloud.google.com/datastore/docs/schedule-export#before_you_begin
The code snippet you showed is just a part of the typical GAE app skeleton specific for 1st generation standard environment python apps. You can easily recognize it in the main.py section of the python quickstart Hello World code review.
The code initializes the app variable (from the main python module, i.e. the main.py file) which is referenced in the app.yaml handler config as script: main.app.
The corresponding java app skeleton is significantly different, see the java quickstart Hello World code review. But no worries, you shouldn't need to specifically transform that code snippet, you just need to build your java app skeleton and focus on what the app handler actually does - making those POST requests to the datastore. Sorry I can't help more, but I'm not a java user.
What I really realized is that app.yaml is like Web.xml in java
and cloud-datastore-export is a servlet that communicates with gae to export data but I can't do more

how to configure a FileEntityStoreService

I'm trying to use a File EntityStore and I'm having an exception at activation because of slices being zero.
I assume it's an issue with configuration but I expected the default value to be 1.
I narrowed down to this assembly:
import org.apache.polygene.api.common.Visibility;
import org.apache.polygene.api.structure.Application;
import org.apache.polygene.bootstrap.Energy4Java;
import org.apache.polygene.entitystore.file.assembly.FileEntityStoreAssembler;
import org.apache.polygene.index.rdf.assembly.RdfNativeSesameStoreAssembler;
import org.apache.polygene.library.fileconfig.FileConfigurationAssembler;
public class FileStoreException {
public static void main(String[] args) throws Exception {
Energy4Java polygene = new Energy4Java();
Application application = polygene.newApplication(factory -> factory.newApplicationAssembly(
module -> {
new FileConfigurationAssembler()
.visibleIn(Visibility.application)
.assemble(module);
new FileEntityStoreAssembler()
.withConfig(module, Visibility.application)
.assemble(module);
new RdfNativeSesameStoreAssembler()
.withConfig(module, Visibility.application)
.assemble(module);
module.defaultServices();
})
);
application.activate();
}
}
The end of the stacktrace:
Caused by: java.lang.ArithmeticException: / by zero
at method "get" of FileEntityStoreService:FileEntityStoreService in module [Module 1] of layer [Layer 1].(:0)
at org.apache.polygene.entitystore.file.FileEntityStoreMixin.getDataFile(FileEntityStoreMixin.java:277)
at org.apache.polygene.entitystore.file.FileEntityStoreMixin.getDataFile(FileEntityStoreMixin.java:328)
at org.apache.polygene.entitystore.file.FileEntityStoreMixin.get(FileEntityStoreMixin.java:138)
at org.apache.polygene.spi.entitystore.helpers.JSONMapEntityStoreMixin.entityStateOf(JSONMapEntityStoreMixin.java:193)
... 14 more
I'm using version 3.0.0 and I'm on linux.
Adding FileConfigurationAssembler gave me the false impression that my config was done.
I struggled to find a working example of an assembly using a FileEntityStoreAssembler so here's one:
Application application = polygene.newApplication(factory -> factory.newApplicationAssembly(
module -> {
ModuleAssembly config = module.layer().module("Config");
config.defaultServices();
new MemoryEntityStoreAssembler().assemble(config);
config.entities(FileEntityStoreConfiguration.class);
new FileEntityStoreAssembler()
.withConfig(config, Visibility.application)
.assemble(module);
new RdfNativeSesameStoreAssembler()
.withConfig(config, Visibility.application)
.assemble(module);
module.defaultServices();
})
);
And to configure it:
config.forMixin(FileEntityStoreConfiguration.class)
.declareDefaults()
.directory().set("/home/user/appdata/");

Resources