Spring Boot REST: How to dynamically access the appropriate database schema specified in a client request? - database

I have one database with 3 schemas (OPS, TEST, TRAIN). All of these schemas have a completely identical table structure. Now lets say I have an endpoint /cars that accepts a query param for the schema/environment. When the user makes a GET request to this endpoint, I need the Spring Boot backend to be able to dynamically access either the OPS, TEST, or TRAIN schema based on the query param specified in the client request.
The idea is something like this where the environment is passed as a request param to the endpoint and then is somehow used in the code to set the schema/datasource that the repository will use.
#Autowired
private CarsRepository carsRepository;
#GetMapping("/cars")
public List<Car> getCars(#RequestParam String env) {
setSchema(env);
return carsRepository.findAll();
}
private setSchema(String env) {
// Do something here to set the schema that the CarsRepository
// will use when it runs the .findAll() method.
}
So, if a client made a GET request to the /cars endpoint with the env request param set to "OPS" then the response would be a list of all the cars in the OPS schema. If a client made the same request but with the env request param set to "TEST", then the response would be all the cars in the TEST schema.
An example of my datasource configuration is below. This one is for the OPS schema. The other schemas are done in the same fashion, but without the #Primary annotation above the beans.
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(
entityManagerFactoryRef = "opsEntityManagerFactory",
transactionManagerRef = "opsTransactionManager",
basePackages = { "com.example.repo" }
)
public class OpsDbConfig {
#Autowired
private Environment env;
#Primary
#Bean(name = "opsDataSource")
#ConfigurationProperties(prefix = "db-ops.datasource")
public DataSource dataSource() {
return DataSourceBuilder
.create()
.url(env.getProperty("db-ops.datasource.url"))
.driverClassName(env.getProperty("db-ops.database.driverClassName"))
.username(env.getProperty("db-ops.database.username"))
.password(env.getProperty("db-ops.database.password"))
.build();
}
#Primary
#Bean(name = "opsEntityManagerFactory")
public LocalContainerEntityManagerFactoryBean opsEntityManagerFactory(
EntityManagerFactoryBuilder builder,
#Qualifier("opsDataSource") DataSource dataSource
) {
return builder
.dataSource(dataSource)
.packages("com.example.domain")
.persistenceUnit("ops")
.build();
}
#Primary
#Bean(name = "opsTransactionManager")
public PlatformTransactionManager opsTransactionManager(
#Qualifier("opsEntityManagerFactory") EntityManagerFactory opsEntityManagerFactory
) {
return new JpaTransactionManager(opsEntityManagerFactory);
}
}

Personally, I don't feel its right to pass environment as Request Param and toggle the repository based on the value passed.
Instead you can deploy multiple instance of the service pointing to different data source and have a gate keeper(router) to route to the respective service.
By this way clients will be exposed to one gateway service which in turn routes to respective service based on input to gate keeper.

You typically don't want TEST/ACPT instances running on the very same machines because it typically gets harder to [keep under] control the extent to which load on these environments will make the PROD environment slow down.
You also don't want the setup you envisage because it makes it nigh impossible to evolve the app and/or its database structure. (You're not going to switch db schema in PROD at the very same time you're doing this in DEV are you ? Not doing that simultaneous switch is wise, but it breaks your presupposition that "all three databases have exactly the same schema".

Related

Spring Batch "Invalid object name BATCH_JOB_INSTANCE"

I've created a spring batch to query a Azure SQL server database and write the data into a CSV file. I do not have create permissions for the database. I get this error Invalid Object name BATCH_JOB_INSTANCE on running the batch. I don't want the spring batch meta-data tables to be created in the main database. Or it would be helpful if I can have them in another local or in-memory db like h2db.
I've also added spring-batch-initialize-schema=never already, which was the case with most answers to similar questions on here, but that didn't help.
Edit:
I resolved the Invalid Object name error by preventing the metadata tables from being created into the main database by extending the DefaultBatchConfigurer Class and Overriding the setDataSource method, thus having them created in the in-memory map-repository. Now I want to try two options:
How to have the meta data tables to be created in a local db or in-memory db like h2db.
Or If I have the meta data tables created already in the main database, in a different schema than my main table I'm fetching from. How to point my job to those meta-data tables in another schema, to store the job and step details data in those.
#Configuration
public class SpringBatchConfig extends DefaultBatchConfigurer{
#Override
public void setDataSource(DataSource datasource) {
}
...
My application.properties file looks like this:
spring.datasource.url=
spring.datasource.username=
spring.datasource.password=
spring.datasource.driver-class-name=com.microsoft.sqlserver.jdbc.SQLServerDriver
spring-batch-initialize-schema=never
spring.batch.job.enabled=false
spring.jpa.hibernate.ddl-auto=update
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.SQLServer2012Dialect
I've created a demo with two datasources. Batch metadata will sotre in H2 DB and the Job datasource is Azure SQL.
Here is the project structure:
We need define a DataSourceConfig class and use #Primary annotation for DataSource bean:
#Configuration
public class DataSourceConfig {
#Bean(name = "mssqlDataSource")
#ConfigurationProperties(prefix = "spring.datasource")
public DataSource appDataSource(){
return DataSourceBuilder.create().build();
}
#Bean(name = "h2DataSource")
#Primary
// #ConfigurationProperties(prefix="spring.datasource.h2")
public DataSource h2DataSource() {
return DataSourceBuilder.create()
.url("jdbc:h2:mem:thing:H2;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE")
.driverClassName("org.h2.Driver")
.username("sa")
.password("")
.build();
}
}
In the ItemReaderDbDemo class, we use #Autowired #Qualifier("mssqlDataSource") to specify the dataSource in the Spring Batch task:
#Configuration
public class ItemReaderDbDemo {
//generate task Object
#Autowired
private JobBuilderFactory jobBuilderFactory;
//Step exec tasks
//generate step Object
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
#Qualifier("mssqlDataSource")
private DataSource dataSource;
#Autowired
#Qualifier("dbJdbcWriter")
private ItemWriter<? super Todo> dbJdbcWriter;
#Bean
public Job itemReaderDbDemoJob() {
return jobBuilderFactory.get("itemReaderDbDemoJob").start(itemReaderDbStep()).build();
}
#Bean
public Step itemReaderDbStep() {
return stepBuilderFactory.get("itemReaderDbStep")
.<Todo,Todo>chunk(2)
.reader(dbJdbcReader())
.writer(dbJdbcWriter)
.build();
}
#Bean
#StepScope
public JdbcPagingItemReader<Todo> dbJdbcReader() {
JdbcPagingItemReader<Todo> reader = new JdbcPagingItemReader<Todo>();
reader.setDataSource(dataSource);
reader.setFetchSize(2);
reader.setRowMapper(new RowMapper<Todo>() {
#Override
public Todo mapRow(ResultSet rs, int rowNum) throws SQLException {
Todo todo = new Todo();
todo.setId(rs.getLong(1));
todo.setDescription(rs.getString(2));
todo.setDetails(rs.getString(3));
return todo;
}
});
SqlServerPagingQueryProvider provider = new SqlServerPagingQueryProvider();
provider.setSelectClause("id,description,details");
provider.setFromClause("from dbo.todo");
//sort
Map<String,Order> sort = new HashMap<>(1);
sort.put("id", Order.DESCENDING);
provider.setSortKeys(sort);
reader.setQueryProvider(provider);
return reader;
}
}
Here is my application.properties:
logging.level.org.springframework.jdbc.core=DEBUG
spring.datasource.driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver
spring.datasource.jdbcUrl=jdbc:sqlserver://josephserver2.database.windows.net:1433;database=<Your-Database-Name>;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;
spring.datasource.username=<Your-UserName>
spring.datasource.password=<Your-Password>
spring.datasource.initialization-mode=always
It return expected result from my Azure SQL. By the way, my Azure sql username does not have create permissions for the database.
The result shows:
How to have the meta data tables to be created in a local db or in-memory db like h2db.
You can use spring.batch.initialize-schema=embedded for that.
Or If I have the meta data tables created already in the main database, in a different schema than my main table I'm fetching from. How to point my job to those meta-data tables in another schema, to store the job and step details data in those.
spring batch works against a datasource, not a particular schema. If meta-data tables are in a different schema, then you need to create a second datasource pointing to that schema and set it on the job repository.
I know this post is a little bit old, but I'd like to give an update.
For newer versions of Spring Boot spring.batch.initialize-schema is deprecated.
I'm using Spring Boot 2.7.1 and the newer property is spring.batch.jdbc.initialize-schema.
In my case: when I was receiving the error message was due that the user did not have the CREATE TABLE permission to create the corresponding spring bacth tables.
Adding the permissions fix the issue.

Is it possible to fetch data from database before server is getting started?

I am working on a Spring project. I want to use scheduler in it and want to schedule it on a variable date. This date has to be taken from database. Is it possible to fetch data from database before server is getting started?
Two solutions come to my mind:
#PostConstruct annotated method of some #Component:
#Component
public class MyBean
{
#PostConstruct
public void init()
{
// Blocking DB call could go here
}
}
Application Events. For the ApplicationReadyEvent:
#Component
public class ApplicationReadyEventListener implements ApplicationListener<ApplicationReadyEvent>
{
#Override
public void onApplicationEvent(ApplicationReadyEvent event)
{
// DB call could go here
//
// Notice that if this is a web services application, it
// would potentially be serving requests while this method
// is being executed
}
}

Specflow-Share browser session between features if triggered between steps

I have implemented Specflow to reuse some steps across features as in this example -Specflow,Selenium-Share data between different Step definitions or classes .Since, in our project, we are integrating multiple features & reusing them. What is the best way to share browser session across features if its triggered in between steps as per the above approach?
My Scenario:
Once an application created, I need to launch new session, login different User-set different services and approve it.
But after logging in fails with below error on Step definition 4 in reused Whenstep of Given(Set the service to (.*)). That particular step is from different feature, hence the new session needs to be used in those steps. The LaunchURl method below is just launching the website with url, no new session created - This works fine
OpenQA.Selenium.WebDriverException : Unexpected error. System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it "IP here"
[Given(#"A New Application is added")]
public void GivenANewApplicationIsAdded()
{
Given("UK_The Service is set");
Given("User Navigated to New Application screen");
When("User fills up form as in data row 1");
Then("new SID generated");
}
[Given(#"New Browser Launched")]
public void GivenNewBrowserLaunched()
{
SeleniumContext sl = new SeleniumContext();
this.seleniumContext = sl;
}
[Given(#"Login is successful with ""(.*)"" and ""(.*)""")]
public void GivenLoginIsSuccessfulWithAnd(string userName, string password)
{
SuperTests spr = new SuperTests();
_driver = spr.LaunchURL(seleniumContext.WebDriver);
//seleniumContext.WebDriver = _driver;
LoginPage lg = new LoginPage(_driver);
lg.LoginProcess(userName, password);
}
[Given(#"Set the service to ""(.*)""")]
public void GivenSetTheServiceTo(string serviceId)
{
When("Select a Service from the option "+serviceId);
Then("The Services is changed to the one selected " + serviceId);
}
In other feature
[When(#"Select a Service from the option (.*)")]
public void WhenSelectAServiceFromTheOptionTestTeam(string p0)
{
HomePage mst = new HomePage(seleniumContext.WebDriver);
mst.SetServiceId(p0);
}
The 2 work around what we figured was
Create a new instance of binding class to call the methods or steps as shown below
[Given(#"Set the service to ""(.*)""")]
public void GivenSetTheServiceTo(string serviceId)
{
var serIdSteps = new UK_SetServiceIDSteps(seleniumContext);
serIdSteps.WhenUK_SelectAServiceFromTheOptionTest(serviceId);
serIdSteps.ThenUK_TheServicesIsChangedToTheOneSelected(serviceId);
}
or
tried this which worked as well- basically calling a new method to create a new session. for this I need not create any new instance for Binding class. Called the Step directly.
[Given(#"New Browser Launched")]
public void GivenNewBrowserLaunched()
{
SuperTests spr = new SuperTests();
_driver = spr.LaunchURL("Firefox");
seleniumContext.WebDriver = _driver;
}
public void GivenSetTheServiceTo(string serviceId)
{
When("UK_Select a Service from the option "+serviceId);
Then("UK_The Services is changed to the one selected " + serviceId);
}
Not sure, which is correct way of doing it? Trying to figure it out from Reusable steps point?The latter one is not advised as we need to change the type of browser to launch at multiple place.

Initialize Spring embedded database after deployment

I have an Spring MVC app with an embedded database (HSQLDB) that I want to initialize after deployment. I know that I could use an xml script to define initial data for my datasource but, as long I'm using JPA + Hibernate, I would like to use Java code. Is there a way to do this?
Heavily updated answer (it was too complex before):
All you need is to add initializing bean to your context, which will insert all the necessary data into the database:
public class MockDataPopulator {
private static boolean populated = false;
#Autowired
private SessionFactory sessionFactory;
#PostConstruct
public void populateDatabase() {
// Prevent duplicate initialization as HSQL is also initialized only once. Duplicate executions
// can happen when the application context is reloaded - e.g. when running unit tests).
if (populated) {
return;
}
// Create new persistence session
Session session = sessionFactory.openSession();
session.setFlushMode(FlushMode.ALWAYS);
// Insert mock entities
session.merge(MockDataFactory.createMyFirstObject())
session.merge(MockDataFactory.createMySeconfObject())
// ...
// Flush and close
session.flush();
session.close();
// Set initialization flag
populated = true;
}
}

Storing the Cursor for App Engine Pagination

I'm trying to implement pagination using App Engine's RPC and GWT (it's an app engine connected project).
How can I pass both the query results and the web-safe cursor object to the GWT client from the RPC?
I've seen examples using a servlet but I want to know how to do it without a servelt.
I've considered caching the cursor on the server using memcache but I'm not sure if that's appropriate or what should be used as the key (session identifier I would assume, but I'm not sure how those are handled on App Engine).
Links to example projects would be fantastic, I've been unable to find any.
OK, so the best way to do this is to store the cursor as a string on the client.
To do this you have to create a wrapper class that is transportable so you can pass back it to the client via RequestFactory that can hold the results list and the cursor string. To do that you create a normal POJO and then a proxy for it.
here's what the code looks like for the POJO:
public class OrganizationResultsWrapper {
public List<Organization> list;
public String webSafeCursorString;
public List<Organization> getList() {
return list;
}
public void setList(List<Organization> list) {
this.list = list;
}
public String getWebSafeCursorString() {
return this.webSafeCursorString;
}
public void setWebSafeCursorString(String webSafeCursorString) {
this.webSafeCursorString = webSafeCursorString;
}
}
for the proxy:
#ProxyFor(OrganizationResultsWrapper.class)
public interface OrganizationResultsWrapperProxy extends ValueProxy{
List<OrganizationProxy> getList();
void setList(List<OrganizationProxy> list);
String getWebSafeCursorString();
void setWebSafeCursorString(String webSafeCursorString);
}
set up your service and requestFactory to use the POJO and proxy respectively
// service class method
#ServiceMethod
public OrganizationResultsWrapper getOrganizations(String webSafeCursorString) {
return dao.getOrganizations(webSafeCursorString);
}
// request factory method
Request<OrganizationResultsWrapperProxy> getOrganizations(String webSafeCursorString);
Then make sure and run the RPC wizard so that your validation process runs otherwise you'll get a request context error on the server.
Here's the implementation in my data access class:
public OrganizationResultsWrapper getOrganizations(String webSafeCursorString) {
List<Organization> list = new ArrayList<Organization>();
OrganizationResultsWrapper resultsWrapper = new OrganizationResultsWrapper();
Query<Organization> query = ofy().load().type(Organization.class).limit(50);
if (webSafeCursorString != null) {
query = query.startAt(Cursor.fromWebSafeString(webSafeCursorString));
}
QueryResultIterator<Organization> iterator = query.iterator();
while (iterator.hasNext()) {
list.add(iterator.next());
}
resultsWrapper.setList(list);
resultsWrapper.setWebSafeCursorString(iterator.getCursor().toWebSafeString());
return resultsWrapper;
}
a second option would be to save the webSafeCursorString in the memcache, as you already mentioned.
my idea looks like this:
the client sends always request like this "getMyObjects(Object... myParams, int maxResults, String clientPaginationString)". the clientPaginationString is uniquely created like shown below
server receives request and looks into the memcache if there is a webSafeCursorString for the key clientPaginationString
if the server finds nothing, he creates the query and save the webSafeCursorString into memcache with the clientPaginationString as the key. -> returns the results
if the server finds the webSafeCursorString he restarts the query with it and returns the results
the problems are how to clean the memcache and how to find a unique clientPaginationString:
a unique clientPaginationString should be the current UserId + the params of the current query + timestemp. this should work just fine!
i really can't think of a easy way how to clean the memcache, however i think we do not have to clean it at all.
we could store all the webSafeCursorStrings and timestemps+params+userid in a WebSafeCursor-Class that contains a map and store all this in the memcache... and clean this Class ones in a while (timestamp older then...).
one improvement i can think of is to save the webSafeCursorString in the memcache with a key that is created on the server (userSessionId + servicename + servicemethodname + params). however, important is that the client sends an information if he is interested in a new query (memcache is overriden) or wants the next pagination results (gets webSafeCursorString from memcache). a reload of the page should work. a second tap in the browser would be a problem i think...
what would you say?

Resources