JPA + Spring how to initialize database with custom tables and data - database

In my application I use Spring context and JPA. I have some set of entities annotated with #Entity and tables for them are created automatically during system startup. Recently I started using Spring ACL, so I have to have the following additional DB schema and I don't want these tables to be mapped to the entities (simply I don't need them to, because Spring ACL manages them independently).
I want to automatically insert e.g. admin user account into the
User's entity table. How to do that properly?
I want to initialize Spring ACL custom tables during system startup, but the SQL script file does not seem to be good solution because if I use different database for production and functional testing, the different SQL dialect does not allow me to execute script properly on both engines (e.g. when using MySQL and HSQL).
At first I tried to use ServletListener that during servlet initialization check the db and adds the necessary data and schema, but this does not work for integration tests (because there are no servlet involved at all).
What I want to achieve is the Spring bean (?) that is launched after the JPA has initialized all entity tables, insert all startup data using injected DAOs and somehow creates the Spring ACL schema. Then - I want the bean to be removed from IoC (because I simly don't need it anymore). Is it possible?
Or is there any better way of doing this?

The default JPA allows you to add an SQL script upon loading the persistence.xml:
http://docs.oracle.com/javaee/7/tutorial/persistence-intro005.htm
Add this property to your persistence.xml file:
<property name="javax.persistence.sql-load-script-source"
value="META-INF/sql/data.sql" />
And fill the data.sql file with your default values.

If you are using EclipseLink you could use a SessionEventListener to execute code after JPA login. You could perform your schema creation and setup in a postLogin event.
You could use the Schema Framework in EclipseLink (org.eclipse.persistence.tools.schemaframework) to create tables and DDL in a database platform independent way. (TableDefinition, SchemaManager classes)

I use PostConstruct annotation to invoke initialize methods.
As document described: The PostConstruct annotation is used on a method that needs to be executed after dependency injection is done to perform any initialization. You may simply add a spring bin with methods has #PostConstruct annotation on it, the methods would be executed after tables are created(or we can say, they are executed after other beans are ready).
Code sample:
#Component
public class EntityLoader {
#Autowired
UserRepository userRepo;
#PostConstruct
public void initApiUserData() {
User u = new User();
// set user properties here
userRepo.save(u);
}
}

If you use hibernate, then create a sql script import.sql in the class path root. Hibernat will execute it on startup. -- This worked in former hibernate version. In the docu of the current version 4.1 I have not found any hint of this feature.
But Hibernate 4.1 has an other feature
Property: hibernate.hbm2ddl.import_files
Comma-separated names of the optional files containing SQL DML statements executed during the SessionFactory creation. This is useful for testing or demoing: by adding INSERT statements for example you can populate your database with a minimal set of data when it is deployed.
File order matters, the statements of a give file are executed before the statements of the following files. These statements are only executed if the schema is created ie if hibernate.hbm2ddl.auto is set to create or create-drop.
e.g. /humans.sql,/dogs.sql

You could try using flyway to
create the tables using the SQL DDL as an SQL migration and
possibly put some data in the tables using either SQL or Java based migrations. You might want to use the latter if in need of environment or other info.
It's a lot easier than it sounds and you will also end up with flyway itself as a bonus, if not a must.

Related

EF Core 5.0 DB first approach to access DB for read purpose

I am developing an application with Asp.Net core 5 and the application accesses and displays information from two different databases(both Sql). One database is application’s own database where all the information will be stored/added. But there is another database already exists(on-prem server) and being used by another application. I want to read some master data from this database to use in my application. So I want to connect to the second database to just read master data and display it in application pages. For the first database connectivity I have created a separate entity project using Entity Framework Core 5.0 Code first approach.
How do I access second database just for read data purpose, Which would be the feasible approach for this. I was thinking to create another entity project with EF Core 5 DB first approach, but with this approach it creates DBContext class and all DbSets objects for each table. Which is not required I feel because I just want to read 8-10 tables from the entire database
Can anyone please suggest which would be a better approach for this? DB first approach or Ado.Net Vanilla method?
Finally I am going to reference these entity projects into my Web API application for all DB operations.
Thanks!
Within the same project, next to your existing dbContext, register a new extension of db context that will encapsulate the second database that you want to read from.
Do not copy the entire project from the start, rather think that the only reason to have a second dbContext to start with, is when dealing with multiple databases.
The rest of your code can remain re-usable ( probably ) and you dont need to cope with an other project as well
EDIT:
For example you might register your current context like below:
services.AddDbContext<ApplicationDbContext>(
options => options.UseSqlServer("name=ConnectionStrings:DefaultConnection"));
You can create a new class, lets say
public class OldDataDbContext : DbContext
{
public OldDataDbContext(DbContextOptions<OldDataDbContext> options)
: base(options)....
....
....
on that new dbContext class, register all the dbSets that you are going to read from that master db.
Then register that "new implementation of db context" with the connection string for the other database.
services.AddDbContext<OldDataDbContext>(
options => options.UseSqlServer("name=ConnectionStrings:OldDatabaseConnection"));
Now next to your DefaultConnectionString appSetting ( or however else you might have called it, add an other line with the other connectionString, with keyName "OldDatabaseConnection" and you can use that new class just as you use the old one.

Laravel Dynamic Database Table Names

Is there a way to create database tables dynamically in Laravel. I have one Laravel build which has a database schema for quotes using the migrations tool. There will be several customers using the system which need to each have their own database table.
What I would like to happen is that when a function is called by the customer it will use the quotes schema to create a new table like 'customer1_quotes' and use this table for the customer in future. Additionally when migrations are run it will apply the updates to all tables with the given name structure (*_quotes).
If anyone has details to achieve this or a recommend alternative approach please message :)
Create a trait used by observers
Create a trait which loops through your customers and creates/updates the tables. You don't have to be in a migration to call DB::.
Use the trait in create/update controllers or better for model create/update observers. You could also create a console command for manual triggering or testing.
This should not be executed during maintenance. Using php artisan down should ensure no jobs are run during migrations.
The migrations for the customer{id}_quotes tables can loop through the available tables by querying table names using LIKE and/or REGEXP. See link below.
Links
Laravel Model Observers
How to dynamically set table name in Eloquent Model
Laravel's table Blueprint docs (5.8)
Get table names using LIKE or REGEXP
Optimization: Chunking results when getting query builder results
Edit: A repeatable migration probably won't work well and is confusing to others. Using a trait for flexibility to use for an observer is better for this.

Reset database after each test on Spring without using DirtiesContext

I would like to know if there is some way to reset the database after each integration test without #DirtiesContext:
#DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_EACH_TEST_METHOD)
This works but it is very slow, because the Spring context is reloaded each test.
My tests are using MockMvc, doing rest calls for an API. Like:
mockMvc.perform(put("/products/)
.header("Content-Type", "application/json")
.content(jsonPost))
.andExpect(status().isOk())
.andReturn();
So, without manual intervention (create and maintain a script to drop and create the tables), the Spring framework offer some alternative?
You can clean the tables you need by doing the following:
Inject a JdbcTemplate instance
#Autowired
private JdbcTemplate jdbcTemplate;
Use the class JdbcTestUtils to delete the records from the tables you need to.
JdbcTestUtils.deleteFromTables(jdbcTemplate, "table1", "table2", "table3");
Call this line in the method annotated with #After or #AfterEach in your test class:
#AfterEach
void tearDown() throws DatabaseException {
JdbcTestUtils.deleteFromTables(jdbcTemplate, "table1", "table2", "table3");
}
I found this approach in this blog post:
Easy Integration Testing With Testcontainers
In simple case annotate each your test class as #Transactional and transaction manager will do rollback after each #Test method. Get more information reading this.
I am a bit late to the party, but I had the same problem. All the unit tests (which could be considered integration tests) in an application I inherited took approximately 35 minutes to complete, using an embedded H2 as database for tests. All test classes where annotated by #DirtiesContext, usually method level.
So, the database was destroyed and recreated for each method. This takes time.
By removing the dirties annotation and using a database truncation class in the #Before method I now run the complete test suite in about 4 minutes.
If you have anything else than JPA stuff (not handled by the Entity manager) in your Spring context that should be removed between tests you have to do it explicitly.
I can share the DB truncation class if you like, but it is simply using the JPA meta model to find the tables to truncate. Truncation seems to be very efficient in H2. Exceptions for entities based on views, not tables, can be configured.
To make truncation easier, turn off refererential integrity before truncation and switch it back on when you're done.
You could use org.springframework.test.context.jdbc #Sql({"clear-database.sql"}) and then just write a script to clear the db.
So you'd end up with something like this:
#Test
#Sql({"classpath:sql/clear-database.sql", "classpath:sql/set-up-db.sql"}
void doThings(){
this.mockMvc.perform(etc..);
}

reinitialize code first created tables, but leave existed tables data on place with entity framework 6

Let's say I have 10 entites. 8 of them completelly new and build with EF Code-first aproach. So before I was using DropCreateDatabaseIfModelChanges initialization strategy and that's worked perfect for me.
But now I have 2 entites which build from database based on some 3rd party data, and I need this data all the time, I can't allow EF to drop this tables even if model chnages. I need something more inteligent there.
Which is correct approach in that case?
In short, I want something quite similar. I just need DbInitializer behavior, but per table basis, instead of per Database. I wan't Code-first entities work the same as before, regerating and all that stuff. But add only something custom for this specific 2 DB based entities.
You could use EF Code First Migrations
First, you need to run the Enable-Migrations command in Package Manager Console. This command will add a Migrations folder to our project. This new folder contains the Configuration class that allows you to configure how Migrations behaves for your context.
Now, after that,If you followed the required steps, you can run "update database" from the "Package Manager Console" and add the eight new tables to your DB:
Example:
Make the changes in your model (add the eight new entities)
From the Package Manager Console: Run Add-Migration [Migration Name]
Make any neccessary changes to the generated code (this is optional).
From the Package Manager Console: Run Update-Database
If you don't change or remove any property related to your existing entities, you should not loose the data that you already have in DB.
Update
To achieve what you want you can use Automated Migration. This way when you run your application, you will always get your database in the latest version because EF will do implicit migration every time it is needed - in the purest version you never need to do anything more than enabling automatic migrations.
First, you need to set the database initializer in the context class with the new db initialization strategy MigrateDatabaseToLatestVersion as shown below:
public class YourContext: DbContext
{
public YourContext(): base("DefaultConnectionString")
{
Database.SetInitializer(new MigrateDatabaseToLatestVersion<YourContext, YourProject.Migrations.Configuration>("DefaultConnectionString"));
}
}
Later, in the constructor of the Configuration class you have to enable automatic migrations:
public Configuration()
{
AutomaticMigrationsEnabled = true;
}
Now, if you are working with an existing database, before add your new eight entities, you need to do this first:
Run the Add-Migration InitialCreate –IgnoreChanges command in
Package Manager Console. This creates an empty migration with the
current model as a snapshot.
Run the Update-Database command in Package Manager Console. This
will apply the InitialCreate migration to the database. Since the
actual migration doesn’t contain any changes, it will simply add a
row to the __MigrationsHistory table indicating that this migration
has already been applied.
After that, you can apply the changes that you want to your model (adding, for example, the new eight entities), and when you execute your app again, EF will do the migrations for you.
In case that you are going to change
someting that provoke some inconsistency regarding to your database
schema that it could end in data loss, an exception will be throw.
If this exception is not thrown, you don't have to worry about loss
your data, it will remain intact in your DB.
As an aditional information, if you don't mind loose your data (which I think this is not your escenerario, but is useful to know anyway) you can set in true the AutomaticMigrationDataLossAllowed property (its default value is false), and no exception will be thrown in case you are going to loose some data in your DB in the execution of a migration.
public Configuration()
{
AutomaticMigrationsEnabled = true;
AutomaticMigrationDataLossAllowed=true;
}

Integrating GeoDjango into existing Django project

I have a Django project with multiple apps. They all share a db with engine = django.db.backends.postgresql_psycopg2. Now I want some functionality of GeoDjango and decided I want to integrate it into my existing project. I read through the tutorial, and it looks like I have to create a separate spartial database for GeoDjango. I wonder if there is anyway around. I tried to add this into one of my apps' models.py without changing my db settings :
from django.contrib.gis.db.models import PointField
class Location(models.Model):
location = PointField()
But when I run syncdb, I got this error.
File "/home/virtual/virtual-env/lib/python2.7/site-packages/django/contrib/gis/db/models/fields.py", line 200, in db_type
return connection.ops.geo_db_type(self)
Actually, as i recall, django.contrib.gis.db.backends.postgis is extension of postgresql_psycopg2 so you could change db driver in settings, create new db with spatial template and then migrate data to new db (South is great for this). By itself geodjango is highly dependent on DB inner methods thus, unfortunately, you couldn't use it with regular db.
Other way - you could make use of django's multi-db ability, and create extra db for geodjango models.
Your error looks like it comes from not changing the database extension in your settings file. You don't technically need to create a new database using the spatial template, you can simply run the PostGIS scripts on your existing database to get all of the geospatial goodies. As always, you should backup your existing database before doing this though.
I'm not 100%, but I think that you can pipe postgis.sql and spatial_ref_sys.sql into your existing database, grant permissions to the tables, and change the db setting to "django.contrib.gis.db.backends.postgis". (After you have installed the deps of course)
https://docs.djangoproject.com/en/dev/ref/contrib/gis/install/#spatialdb-template
I'd be interested to see what you find. Be careful, postgis installation can build some character but you don't want it to build too much.
From the docs (django 3.1) https://docs.djangoproject.com/en/3.1/ref/databases/#migration-operation-for-adding-extensions :
If you need to add a PostgreSQL extension (like hstore, postgis, etc.) using a migration, use the CreateExtension operation.

Resources