Can Android Room validate an imported database before it is opened? - database

Problem: I can't seem to get an imported database to fail in Android Room until first queried? I'm wanting to try/catch and validate a database and I'm not succeeding on catching it before the first query. I always thought Android Room validated the database at the moment of instance creation and build against the schema, but apparently not. So the database fails upon first query.
What I'm trying to do: This app manages multiple databases that can be shared between users. So databases can be imported or exported. I suspect that at some point, someone will attempt to import the wrong database and or structure or do something to cause it to fail. I'm trying to catch the failure at the instance / build of the database or sooner.
What I've tried: I have a try/catch/finally block at the first instance creation, but it is only failing when first queried... then it notices that a table or column is missing. I'd like to catch it sooner if possible. I've looked at the RoomDatabase methods but nothing specifically applies to validation that I see other than just letting it break.

I always thought Android Room validated the database at the moment of instance creation and build against the schema, but apparently not.
The database validation is part of the open process, which does not happen until you actually try to access the database, as opposed to when getting the instance.
I can't seem to get an imported database to fail in Android Room until first queried?
When you get the instance you can force an open by getting (or trying to get) a SupportSQLiteDatabase by using either getWritableDatabase or getReadableDatabase via the instance's openHelper.
e.g.
(Kotlin)
db = TheDatabase.getInstance(this)
try {
val supportDB = db.openHelper.writableDatabase
}
catch(e: Exception) {
....
}
(Java)
db = TheDatabase.getInstance(this);
try {
SupportSQLiteDatabase supportDB = db.getOpenHelper().getWritableDatabase();
}
catch (Exception e) {
....
}

Alternative - Self validation
You could also do your own validation and thus avoid an exception (if the validation is simple enough). You could also make corrections thus allowing perhaps minor transgressions to be acceptable.
Before getting the actual instance, you could get a validation instance (different database name) that is created as per Room and then compare the schemas yourself.
here's an example designed to detect a table missing by creating the real database with a different table name (nottablex instead of tableX).
The TableX entity :-
#Entity
class TableX {
#PrimaryKey
Long id=null;
String name;
String other;
}
No Dao's as not needed for the example.
The TheDatabase with get instance methods, one for normal, the other for getting another (validation (empty model for schema comparison)) database but as an SQLiteDatabase.
#Database(entities = {TableX.class},version = 1)
abstract class TheDatabase extends RoomDatabase {
private static volatile TheDatabase instance = null;
private static volatile TheDatabase validationInstance = null;
static TheDatabase getInstance(Context context, String databaseName) {
if (instance == null ) {
instance = Room.databaseBuilder(context,TheDatabase.class,databaseName)
.allowMainThreadQueries()
.build();
}
return instance;
}
static SQLiteDatabase getValidationInstance(Context context, String databaseName) {
// Delete DB if it exists
if (context.getDatabasePath(databaseName).exists()) {
context.getDatabasePath(databaseName).delete();
}
// Create database and close it
TheDatabase db = Room.databaseBuilder(context,TheDatabase.class,databaseName)
.allowMainThreadQueries()
.build();
db.getOpenHelper().getWritableDatabase();
db.close();
return SQLiteDatabase.openDatabase(context.getDatabasePath(databaseName).getPath(),null,SQLiteDatabase.OPEN_READWRITE);
}
}
note that this forces the open to create the model/validation database (else the openDatabase would fail).
And finally for the demo MainActivity which creates an invalid database and then goes on to do a simple validation (just check that the expected tables exist). Only opening (never in the case of the example) the database if the tables expected by room exist.
public class MainActivity extends AppCompatActivity {
public static final String DATABASE_NAME = "the_database.db";
public static final String VALIDATION_DATABASE_NAME = DATABASE_NAME + "_validation";
public static final String TAG = "DBVALIDATION";
TheDatabase db;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
createIncorrectDatabase(this,DATABASE_NAME);
if (validateDatabase(VALIDATION_DATABASE_NAME,DATABASE_NAME) < 0) {
Log.d(TAG,"DATABASE " + DATABASE_NAME + " does mot match model.");
} else {
/* Database validated OK so use it */
db = TheDatabase.getInstance(this,DATABASE_NAME);
}
}
/* Purposefully create invalid database */
private void createIncorrectDatabase(Context context, String databaseName) {
File dbfile = context.getDatabasePath(databaseName);
if (!dbfile.exists()) {
dbfile.getParentFile().mkdirs();
}
SQLiteDatabase db = SQLiteDatabase.openOrCreateDatabase(context.getDatabasePath(databaseName),null);
db.execSQL("CREATE TABLE IF NOT EXISTS nottablex(id INTEGER PRIMARY KEY,name TEXT)");
db.close();
}
#SuppressLint("Range")
private long validateDatabase(String modelDatabase, String actualDatabase) {
String sqlite_master = "sqlite_master";
/* Want to skip room_master_table and sqlite tables susch as sqlite_sequence */
/* in this example only checking tables to show the basic technique */
String wherecluase = "name NOT LIKE 'sqlite_%' AND name NOT LIKE 'room_%' AND type = 'table'";
long rv = 0;
/* Get the model/validation database */
SQLiteDatabase modelDB = TheDatabase.getValidationInstance(this,modelDatabase);
/* Only need to check if the database exists as otherwise it will be created according to Room */
if (this.getDatabasePath(actualDatabase).exists()) {
/* Open as an SQLiteDatabase so no Room open to throw an exception */
SQLiteDatabase actualDB = SQLiteDatabase.openDatabase(this.getDatabasePath(actualDatabase).getAbsolutePath(),null,SQLiteDatabase.OPEN_READWRITE);
/* Get the tables expected from the model Room database */
Cursor modelTableNames = modelDB.query(sqlite_master,null,wherecluase,null,null,null,null);
Cursor actualTableNames = null; /* prepare Cursor */
/* Loop through the tables names in the model checking if they exist */
while (modelTableNames.moveToNext()) {
/* See if the expected table exists */
actualTableNames = actualDB.query(sqlite_master,null,"name=?",new String[]{modelTableNames.getString(modelTableNames.getColumnIndex("name"))},null,null,null);
if (!actualTableNames.moveToFirst()) {
Log.d(TAG,"Table " + modelTableNames.getString(modelTableNames.getColumnIndex("name")) + " not found.");
rv = rv -1; /* Table not found so decrement rv to indicate number not found */
}
}
/* Close the actualTableNames Cursor if it was used */
if (actualTableNames != null) {
actualTableNames.close();
}
/* Close the modelTableNames Cursor */
modelTableNames.close();
/* Close the actual database so Room can use it (comment out to show results in database Inspector)*/
actualDB.close();
} else {
Log.d(TAG,"Actual Database " + actualDatabase + " does not exist. No validation required as it would be created");
}
/* Close and delete the model database (comment out to show results in database Inspector)*/
modelDB.close();
this.getDatabasePath(modelDatabase).delete();
return rv;
}
}
Result
The log includes :-
D/DBVALIDATION: Table TableX not found.
D/DBVALIDATION: DATABASE the_database.db does mot match model.
Bypassing the close and model delete the databases for the above are:-
Note in this simple example Room would actually create the TableX table rather than fail with an exception.

Related

Room Data Base Create Instance

I want to Create An Instance Of Room Data base in Composable
But
val db = Room.databaseBuilder(applicationContext, UserDatabase::class.java,"users.db").build()
is not working here not getting applicationContext
How to create an instance of context in composable
Have you tried getting the context with : val context = LocalContext.current and then adding this to get your applicationContext?
Like this: context.applicationContext or using simply val db = Room.databaseBuilder(context, UserDatabase::class.java,"users.db").build()
Room (and the underlying SQliteOpenHelper) only need the context to open the database (or more correctly to instantiate the underlying SQLiteOpenHelper).
Room/Android SQLiteOpenHelper uses the context to ascertain the Application's standard (recommended) location (data/data/<the_package_name>/databases). e.g. in the following demo (via Device Explorer):-
The database, as it is still open includes 3 files (the -wal and -shm are the Write Ahead Logging files that will at sometime be committed/written to the actual database (SQLite handles that)).
so roughly speaking Room only needs to have the context so that it can ascertain /data/data/a.a.so75008030kotlinroomgetinstancewithoutcontext/databases/testit.db (in the case of the demo).
So if you cannot use the applicationContext method then you can circumvent the need to provide the context, if using a singleton approach AND if after instantiating the singleton.
Perhaps consider this demo:-
First some pretty basic DB Stuff (table (#Entity annotated class), DAO functions and #Database annotated abstract class WITH singleton approach). BUT with some additional functions for accessing the instance without the context.
#Entity
data class TestIt(
#PrimaryKey
val testIt_id: Long?=null,
val testIt_name: String
)
#Dao
interface DAOs {
#Insert(onConflict = OnConflictStrategy.IGNORE)
fun insert(testIt: TestIt): Long
#Query("SELECT * FROM testit")
fun getAllTestItRows(): List<TestIt>
}
#Database(entities = [TestIt::class], exportSchema = false, version = 1)
abstract class TestItDatabase: RoomDatabase() {
abstract fun getDAOs(): DAOs
companion object {
private var instance: TestItDatabase?=null
/* Extra/not typical for without a context (if wanted)*/
fun isInstanceWithoutContextAvailable() : Boolean {
return instance != null
}
/******************************************************/
/* Extra/not typical for without a context */
/******************************************************/
fun getInstanceWithoutContext(): TestItDatabase? {
if (instance != null) {
return instance as TestItDatabase
}
return null
}
/* Typically the only function*/
fun getInstance(context: Context): TestItDatabase {
if (instance==null) {
instance = Room.databaseBuilder(context,TestItDatabase::class.java,"testit.db")
.allowMainThreadQueries() /* for convenience/brevity of demo */
.build()
}
return instance as TestItDatabase
}
}
}
And to demonstrate (within an activity for brevity) :-
class MainActivity : AppCompatActivity() {
lateinit var roomInstance: TestItDatabase
lateinit var dao: DAOs
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
roomInstance = TestItDatabase.getInstance(this) /* MUST be used before withoutContext functions but could be elsewhere shown here for brevity */
dao = roomInstance.getDAOs()
//dao.insert(TestIt(testIt_name = "New001")) /* Removed to test actually doing the database open with the without context */
logDataWithoutContext()
addRowWithoutContext()
addRowWithApplicationContext()
logDataWithoutContext()
}
private fun logDataWithoutContext() {
Log.d("${TAG}_LDWC","Room DB Instantiated = ${TestItDatabase.isInstanceWithoutContextAvailable()}")
for (t in TestItDatabase.getInstanceWithoutContext()!!.getDAOs().getAllTestItRows()) {
Log.d("${TAG}_LDWC_DATA","TestIt Name is ${t.testIt_name} ID is ${t.testIt_id}")
}
}
private fun addRowWithoutContext() {
Log.d("${TAG}_LDWC","Room DB Instantiated = ${TestItDatabase.isInstanceWithoutContextAvailable()}")
if (TestItDatabase.getInstanceWithoutContext()!!.getDAOs()
.insert(TestIt(System.currentTimeMillis(),"NEW AS PER ID (the time to millis) WITHOUT CONTEXT")) > 0) {
Log.d("${TAG}_ARWC_OK","Row successfully inserted.")
} else {
Log.d("${TAG}_ARWC_OUCH","Row was not successfully inserted (duplicate ID)")
}
}
private fun addRowWithApplicationContext() {
TestItDatabase.getInstance(applicationContext).getDAOs().insert(TestIt(System.currentTimeMillis() / 1000,"NEW AS PER ID (the time to seconds) WITH CONTEXT"))
}
}
The result output to the log showing that the database access, either way, worked:-
2023-01-05 12:45:39.020 D/DBINFO_LDWC: Room DB Instantiated = true
2023-01-05 12:45:39.074 D/DBINFO_LDWC: Room DB Instantiated = true
2023-01-05 12:45:39.077 D/DBINFO_ARWC_OK: Row successfully inserted.
2023-01-05 12:45:39.096 D/DBINFO_LDWC: Room DB Instantiated = true
2023-01-05 12:45:39.098 D/DBINFO_LDWC_DATA: TestIt Name is NEW AS PER ID (the time to seconds) WITH CONTEXT ID is 1672883139
2023-01-05 12:45:39.098 D/DBINFO_LDWC_DATA: TestIt Name is NEW AS PER ID (the time to millis) WITHOUT CONTEXT ID is 1672883139075
note that the shorter id was the last added but appears first due to it being selected first as it appears earlier in the index that the SQlite Query Optimiser would have used (aka the Primary Key).
basically the same date time second wise but the first insert included milliseconds whilst the insert via AddRowWithApplicationContext drops the milliseconds.

Entity Framework - Saving entity generates an Id but record isn't added to database

I'm trying to insert a new record using Entity Framework using the following code. Within the BuildRequest.Save() method in the BuildRequest class, when the 'Insert' occurs, and the db.Save() is done, a JobId is correctly generated for the BuildRequest (indicating that the Insert is being done correctly), however the record isn't added to the database.
When I check in SQL profiler, here's what the insert is trying to do:
exec sp_executesql N'INSERT [dbo].[BuildQueue]([ApplicationId], [BuildReason])
VALUES (#0, #1)
SELECT [JobId]
FROM [dbo].[BuildQueue]
WHERE ##ROWCOUNT > 0 AND [JobId] = scope_identity()',N'#0 int,#1 tinyint,#0=5819,#1=0
Here is my call to create a new 'job':
BuildRequest job = new BuildRequest(ApplicationId, Core.BuildReasons.NewApp);
job.Save()
which uses the following class:
public class BuildRequest
{
private Data.BuildQueue _buildRequest;
public BuildRequest(int applicationId, BuildReasons reason)
{
_buildRequest = new Data.BuildQueue();
ApplicationId = applicationId;
BuildReason = reason;
}
public int JobId
{
get { return _buildRequest.JobId; }
}
public int ApplicationId
{
get
{
return _buildRequest.ApplicationId;
}
set
{
_buildRequest.ApplicationId = value;
}
}
public BuildReasons BuildReason
{
get
{
return (BuildReasons)_buildRequest.BuildReason;
}
set
{
_buildRequest.BuildReason = (byte)value;
}
}
public int Save()
{
using (Core.Data.UnitOfWork db = new Data.UnitOfWork())
{
Data.BuildQueue buildRequest = db.BuildQueueRepository.Get(a => a.JobId == this.JobId).SingleOrDefault();
if (buildRequest == null)//current build request doesn't already exist
{
db.BuildQueueRepository.Insert(_buildRequest);
}
else // we're editing an existing build request
{
db.BuildQueueRepository.Update(_buildRequest);
}
db.Save();
// **At this point, _buildRequest has a JobId, but no record is added to the database
return JobId;
}
}
}
What I've checked so far:
the 'UnitOfWork' object I create ('db'), is connected to the correct database (a remote SQL server database)
the db.BuildQueueRepository.Insert(...) is correctly calling dbSet.Add(entity);
the db.Save() is correctly calling context.SaveChanges();
there's existing code elsewhere in my application that uses the same generic UnitOfWork repository, which appears almost identical to the code above, and it works perfectly
What could be happening? What could be the difference in the code that's working correctly?
Doh! There's a job that takes new records out of the queue if they're in a certain status. It turns out that this particular routine sets that status immediately and the record was being entered then immediately moved to a different table.

JPA2 CriteriaBuilder: Using LOB property for greaterThan comparison

My application is using SQLServer and JPA2 in the backend. App makes use of a timestamp column (in the SQLServer sense, which is equivalent to row version see here) per entity to keep track of freshly modified entities. NB SQLServer stores this column as binary(8).
Each entity has a respective timestamp property, mapped as #Lob, which is the way to go for binary columns:
#Lob
#Column(columnDefinition="timestamp", insertable=false, updatable=false)
public byte[] getTimestamp() {
...
The server sends incremental updates to mobile clients along with the latest database timestamp. The mobile client will then pass the old timestamp back to the server on the next refresh request so that the server knows to return only fresh data. Here's what a typical query (in JPQL) looks like:
select v from Visit v where v.timestamp > :oldTimestamp
Please note that I'm using a byte array as a query parameter and it works fine when implemented in JPQL this way.
My problems begin when trying to do the same using the Criteria API:
private void getFreshVisits(byte[] oldVersion) {
EntityManager em = getEntityManager();
CriteriaQuery<Visit> cq = cb.createQuery(Visit.class);
Root<Visit> root = cq.from(Visit.class);
Predicate tsPred = cb.gt(root.get("timestamp").as(byte[].class), oldVersion); // compiler error
cq.where(tsPred);
...
}
The above will result in compiler error as it requires that the gt method used strictly with Number. One could instead use the greaterThan method which simply requires the params to be Comparable and that would result in yet another compiler error.
So to sum it up, my question is: how can I use the criteria api to add a greaterThan predicate for a byte[] property? Any help will be greatly appreciated.
PS. As to why I'm not using a regular DateTime last_modified column: because of concurrency and the way synchronization is implemented, this approach could result in lost updates. Microsoft's Sync Framework documentation recommends the former approach as well.
I know this was asked a couple of years back but just in case anyone else stumbles upon this.. In order to use a SQLServer rowver column within JPA you need to do a couple of things..
Create a type that will wrap the rowver/timestamp:
import com.fasterxml.jackson.annotation.JsonIgnore;
import javax.xml.bind.annotation.XmlTransient;
import java.io.Serializable;
import java.math.BigInteger;
import java.util.Arrays;
/**
* A RowVersion object
*/
public class RowVersion implements Serializable, Comparable<RowVersion> {
#XmlTransient
#JsonIgnore
private byte[] rowver;
public RowVersion() {
}
public RowVersion(byte[] internal) {
this.rowver = internal;
}
#XmlTransient
#JsonIgnore
public byte[] getRowver() {
return rowver;
}
public void setRowver(byte[] rowver) {
this.rowver = rowver;
}
#Override
public int compareTo(RowVersion o) {
return new BigInteger(1, rowver).compareTo(new BigInteger(1, o.getRowver()));
}
#Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
RowVersion that = (RowVersion) o;
return Arrays.equals(rowver, that.rowver);
}
#Override
public int hashCode() {
return Arrays.hashCode(rowver);
}
}
The key here is that it implement Comparable if you want to use it in calculations (which you definitely do)..
Next create a AttributeConverter that will move from a byte[] to the class you just made:
import javax.persistence.AttributeConverter;
import javax.persistence.Converter;
/**
* JPA converter for the RowVersion type
*/
#Converter
public class RowVersionTypeConverter implements AttributeConverter<RowVersion, byte[]> {
#Override
public byte[] convertToDatabaseColumn(RowVersion attribute) {
return attribute != null ? attribute.getRowver() : null;
}
#Override
public RowVersion convertToEntityAttribute(byte[] dbData) {
return new RowVersion(dbData);
}
}
Now let's apply this RowVersion attribute/type to a real world scenario. Let's say you wanted to find all Programs that have changed on or before some point in time.
One straightforward way to solve this would be to use a DateTime field in the object and timestamp column within db. Then you would use 'where lastUpdatedDate <= :date'.
Suppose that you don't have that timestamp column or there's no guarantee that it will be updated properly when changes are made; or let's say your shop loves SQLServer and wants to use rowver instead.
What to do? There are two issues to solve.. one how to generate a rowver and two is how to use the generated rowver to find Programs.
Since the database generates the rowver, you can either ask the db for the 'current max rowver' (a custom sql server thing) or you can simply save an object that has a RowVersion attribute and then use that object's generated RowVersion as the boundary for the query to find the Programs changed after that time. The latter solution is more portable is what the solution is below.
The SyncPoint class snippet below is the object that is used as a 'point in time' kind of deal. So once a SyncPoint is saved, the RowVersion attached to it is the db version at the time it was saved.
Here is the SyncPoint snippet. Notice the annotation to specify the custom converter (don't forget to make the column insertable = false, updateable = false):
/**
* A sample super class that uses RowVersion
*/
#MappedSuperclass
public abstract class SyncPoint {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
// type is rowver for SQLServer, blob(8) for postgresql and h2
#Column(name = "current_database_version", insertable = false, updatable = false)
#Convert(converter = RowVersionTypeConverter.class)
private RowVersion currentDatabaseVersion;
#Column(name = "created_date_utc", columnDefinition = "timestamp", nullable = false)
private DateTime createdDate;
...
Also (for this example) here is the Program object we want to find:
#Entity
#Table(name = "program_table")
public class Program {
#Id
private Integer id;
private boolean active;
// type is rowver for SQLServer, blob(8) for postgresql and h2
#Column(name = "rowver", insertable = false, updatable = false)
#Convert(converter = RowVersionTypeConverter.class)
private RowVersion currentDatabaseVersion;
#Column(name = "last_chng_dt")
private DateTime lastUpdatedDate;
...
Now you can use these fields within your JPA criteria queries just like anything else.. here is a snippet that we used inside a spring-data Specifications class:
/**
* Find Programs changed after a synchronization point
*
* #param filter that has the changedAfter sync point
* #return a specification or null
*/
public Specification<Program> changedBeforeOrEqualTo(final ProgramSearchFilter filter) {
return new Specification<Program>() {
#Override
public Predicate toPredicate(Root<Program> root, CriteriaQuery<?> query, CriteriaBuilder cb) {
if (filter != null && filter.changedAfter() != null) {
// load the SyncPoint from the db to get the rowver column populated
SyncPoint fromDb = synchronizationPersistence.reload(filter.changedBeforeOrEqualTo());
if (fromDb != null) {
// real sync point made by database
if (fromDb.getCurrentDatabaseVersion() != null) {
// use binary version
return cb.lessThanOrEqualTo(root.get(Program_.currentDatabaseVersion),
fromDb.getCurrentDatabaseVersion());
} else if (fromDb.getCreatedDate() != null) {
// use timestamp instead of binary version cause db doesn't make one
return cb.lessThanOrEqualTo(root.get(Program_.lastUpdatedDate),
fromDb.getCreatedDate());
}
}
}
return null;
}
};
}
The specification above works with both the binary current database version or a timestamp.. this way I could test my stuff and all the upstream code on a database other than SQLServer.
That's it really: a) type to wrap the byte[] b) JPA converter c) use attribute in query.

Does flyway migrations support PostgreSQL's COPY?

Having performed a pg_dump of an existing posgresql schema, I have an sql file containing a number of table population statements using the copy.
COPY test_table (id, itm, factor, created_timestamp, updated_timestamp, updated_by_user, version) FROM stdin;
1 600 0.000 2012-07-17 18:12:42.360828 2012-07-17 18:12:42.360828 system 0
2 700 0.000 2012-07-17 18:12:42.360828 2012-07-17 18:12:42.360828 system 0
\.
Though not standard this is part of PostgreSQL's PLSQL implementation.
Performing a flyway migration (via the maven plugin) I get:
[ERROR] Caused by org.postgresql.util.PSQLException: ERROR: unexpected message type 0x50 during COPY from stein
Am I doing something wrong, or is this just not supported?
Thanks.
The short answer is no.
The one definite problem is that the parser is currently not able to deal with this special construct.
The other question is jdbc driver support. Could you try and see if this syntax generally supported by the jdbc driver with a single createStatement call?
If it is, please file an issue in the issue tracker and I'll extend the parser.
Update: This is now supported
I have accomplished this for Postgres using
public abstract class SeedData implements JdbcMigration {
protected static final String CSV_COPY_STRING = "COPY %s(%s) FROM STDIN HEADER DELIMITER ',' CSV ENCODING 'UTF-8'";
protected CopyManager copyManager;
#Override
public void migrate(Connection connection) throws Exception {
log.info(String.format("[%s] Populating database with seed data", getClass().getName()));
copyManager = new CopyManager((BaseConnection) connection);
Resource[] resources = scanForResources();
List<Resource> res = Arrays.asList(resources);
for (Resource resource : res) {
load(resource);
}
}
private void load(Resource resource) throws SQLException, IOException {
String location = resource.getLocation();
InputStream inputStream = getClass().getClassLoader().getResourceAsStream(location);
if (inputStream == null) {
throw new FlywayException("Failure to load seed data. Unable to load from location: " + location);
}
if (!inputStream.markSupported()) {
// Sanity check. We have to be able to mark the stream.
throw new FlywayException(
"Failure to load seed data as mark is not supported. Unable to load from location: " + location);
}
// set our mark to something big
inputStream.mark(1 << 32);
String filename = resource.getFilename();
// Strip the prefix (e.g. 01_) and the file extension (e.g. .csv)
String table = filename.substring(3, filename.length() - 4);
String columns = loadCsvHeader(location, inputStream);
// reset to the mark
inputStream.reset();
// Use Postgres COPY command to bring it in
long result = copyManager.copyIn(String.format(CSV_COPY_STRING, table, columns), inputStream);
log.info(format(" %s - Inserted %d rows", location, result));
}
private String loadCsvHeader(String location, InputStream inputStream) {
try {
return new BufferedReader(new InputStreamReader(inputStream)).readLine();
} catch (IOException e) {
throw new FlywayException("Failure to load seed data. Unable to load from location: " + location, e);
}
}
private Resource[] scanForResources() throws IOException {
return new ClassPathScanner(getClass().getClassLoader()).scanForResources(getSeedDataLocation(), "", ".csv");
}
protected String getSeedDataLocation() {
return getClass().getPackage().getName().replace('.', '/');
}
}
To use implement the class with the appropriate classpath
package db.devSeedData.dev;
public class v0_90__seed extends db.devSeedData.v0_90__seed {
}
All that is needed then is to have CSV files in your classpath under db/devSeedData that follow the format 01_tablename.csv. Columns are extracted from the header line of the CSV.

SQL Server CLR Integration enlisting in current transaction

I'm trying to use CLR integration in SQL Server to handle accessing external files instead of storing them internally as BLOBs. I'm trying to figure out the pattern I need to follow to make my code enlist in the current SQL transaction. I figured I would start with the simplest scenario, deleting an existing row, since the insert/update scenarios would be more complex.
[SqlProcedure]
public static void DeleteStoredImages(SqlInt64 DocumentID)
{
if (DocumentID.IsNull)
return;
using (var conn = new SqlConnection("context connection=true"))
{
conn.Open();
string FaceFileName, RearFileName;
int Offset, Length;
GetFileLocation(conn, DocumentID.Value, true,
out FaceFileName, out Offset, out Length);
GetFileLocation(conn, DocumentID.Value, false,
out RearFileName, out Offset, out Length);
new DeleteTransaction().Enlist(FaceFileName, RearFileName);
using (var comm = conn.CreateCommand())
{
comm.CommandText = "DELETE FROM ImagesStore WHERE DocumentID = " + DocumentID.Value;
comm.ExecuteNonQuery();
}
}
}
private class DeleteTransaction : IEnlistmentNotification
{
public string FaceFileName { get; set; }
public string RearFileName { get; set; }
public void Enlist(string FaceFileName, string RearFileName)
{
this.FaceFileName = FaceFileName;
this.RearFileName = RearFileName;
var trans = Transaction.Current;
if (trans == null)
Commit(null);
else
trans.EnlistVolatile(this, EnlistmentOptions.None);
}
public void Commit(Enlistment enlistment)
{
if (FaceFileName != null && File.Exists(FaceFileName))
{
File.Delete(FaceFileName);
}
if (RearFileName != null && File.Exists(RearFileName))
{
File.Delete(RearFileName);
}
}
public void InDoubt(Enlistment enlistment)
{
}
public void Prepare(PreparingEnlistment preparingEnlistment)
{
preparingEnlistment.Prepared();
}
public void Rollback(Enlistment enlistment)
{
}
}
When I actually try to run this, I get the following exception:
A .NET Framework error occurred during execution of user defined routine or aggregate 'DeleteStoredImages':
System.Transactions.TransactionException: The operation is not valid for the state of the transaction. ---> System.Transactions.TransactionPromotionException: MSDTC on server 'BD009' is unavailable. ---> System.Data.SqlClient.SqlException: MSDTC on server 'BD009' is unavailable.
System.Data.SqlClient.SqlException:
at System.Data.SqlServer.Internal.StandardEventSink.HandleErrors()
at System.Data.SqlServer.Internal.ClrLevelContext.SuperiorTransaction.Promote()
System.Transactions.TransactionPromotionException:
at System.Data.SqlServer.Internal.ClrLevelContext.SuperiorTransaction.Promote()
at System.Transactions.TransactionStatePSPEOperation.PSPEPromote(InternalTransaction tx)
at System.Transactions.TransactionStateDelegatedBase.EnterState(InternalTransaction tx)
System.Transactions.TransactionException:
at System.Transactions.TransactionState.EnlistVolatile(InternalTransaction tx, IEnlistmentNotification enlistmentNotification, EnlistmentOptions enlistmentOptions, Transaction atomicTransaction)
at System.Transactions.TransactionStateSubordinateActive.EnlistVolatile(InternalTransaction tx, IEnlistmentNotification enlistmentNotification, EnlistmentOptions enlistmentOptions, Transaction atomicTransaction)
at System.Transactions.Transaction.EnlistVolatile(IEnlistmentNotification enlistmentNotification, EnlistmentOptions enlistmentOptions)
at ExternalImages.StoredProcedures.DeleteTransaction.Enlist(String FaceFileName, String RearFileName)
at ExternalImages.StoredProcedures.DeleteStoredImages(SqlInt64 DocumentID)
. User transaction, if any, will be rolled back.
The statement has been terminated.
Can anyone explain what I'm doing wrong, or point me to an example of how to do it right?
You have hopefully solved this by now, but in case anyone else has a similar problem: the error message you are getting suggests that you need to start the Distributed Transaction Coordinator service on the BD009 machine (presumably your own machine).
#Aasmund's answer regarding the Distributed Transaction Coordinator might solve the stated problem, but that still leaves you in a non-ideal state: You are tying a transaction, which locks the ImagesStore table (even if it is just a RowLock), to two file system operations? And you need to BEGIN and COMMIT the transaction outside of this function (since that isn't being handled in the presented code).
I would separate those two pieces:
Step 1: Delete the row from the table
and then, IF that did not error,
Step 2: Delete the file(s)
In the scenario where Step 1 succeeds but then Step 2, for whatever reason, fails, do one or both of the following:
return an error status code and keep track of which DocumentIDs got an error when attempting to delete the file in a status table. You can use that to manually delete the files and/or debug why the error occurred.
create a process that can run periodically to find and remove unreferenced files.

Resources