I am attempting to use multiple datasources in a Grails 2.4.4 project. According to the docs, this should be possible:
http://www.grails.org/doc/2.4.4/guide/conf.html#multipleDatasources
My primary dataSource (the one I want to use for all domain classes) is using H2 at the moment, as configured by the default DataSource.groovy configuration. My second, read-only datasource is SQL Server, and I tried to declare it as follows at the top level of my DataSource.groovy config (shared by all environments):
ds {
pooled = true
dialect = "org.hibernate.dialect.SQLServer2008Dialect"
driverClassName = "net.sourceforge.jtds.jdbc.Driver"
url = "jdbc:jtds:sqlserver://myserver:1433/mydb;domain=mydomain;useNTLMv2=true;user=myuser"
dbCreate = "none"
}
(Don't let the URL throw you off - I'm just having to use Windows Auth with JTDS. I've tested this via third-party clients as well.)
I inject this into my service class and use it, and everything appears to hook up well:
def dataSource_ds
def serviceMethod(){
Sql ds = new Sql(dataSource_ds)
String query = "SELECT ... "
def results = ds.rows(query)
println "Results are ${results.size()}"
return "Some value"
}
But when I try to access this from an IntegrationSpec-backed Integration Test, I noticed that I was getting "schema not found" errors for valid schemas referred to by my query string, such as "dbo". And the stack trace of any errors from this setup looks like this:
org.h2.jdbc.JdbcSQLException: Schema "DBO" not found; SQL statement:
...
at org.h2.message.DbException.getJdbcSQLException(DbException.java:329)
at org.h2.message.DbException.get(DbException.java:169)
at org.h2.message.DbException.get(DbException.java:146)
at org.h2.command.Parser.readTableOrView(Parser.java:4774)
at org.h2.command.Parser.readTableFilter(Parser.java:1083)
at org.h2.command.Parser.parseSelectSimpleFromPart(Parser.java:1689)
at org.h2.command.Parser.parseSelectSimple(Parser.java:1796)
at org.h2.command.Parser.parseSelectSub(Parser.java:1683)
at org.h2.command.Parser.parseSelectUnion(Parser.java:1526)
at org.h2.command.Parser.parseSelect(Parser.java:1514)
at org.h2.command.Parser.parsePrepared(Parser.java:404)
at org.h2.command.Parser.parse(Parser.java:278)
at org.h2.command.Parser.parse(Parser.java:250)
at org.h2.command.Parser.prepareCommand(Parser.java:217)
at org.h2.engine.Session.prepareLocal(Session.java:414)
at org.h2.engine.Session.prepareCommand(Session.java:363)
...
Now why would THIS datasource be trying to use the H2 driver?
In case it's relevant, my Integration test looks like this:
void "serviceMethod" () {
when: "service method is called"
String response = myService.serviceMethod()
then: "we should get the appropriate text back"
response.equals("Some value")
}
If, in the Service class, I hard-code the connection using a constructor of the Groovy Sql object, the integration test works fine, and any stack traces go through the JTDS driver.But when I try to use the injected datasource, things are strange.
Any idea what I'm doing wrong here?
Just to close the loop on this and hopefully save someone pain on this oversight in the future:
Grails uses an in-memory database when running tests. Make sure to read up on the other differences between integration tests and production here:
http://www.grails.org/doc/latest/guide/testing.html#integrationTesting
This feature makes the use of external (read-only) datasources during any tests pretty interesting, but some of that is to be expected (a test which depends on an external datasource is not a very good test in the long run). I hope to refactor my app and its testing approach at some point (e.g., to use a simple DAO and mock that during the test), because I don't really care about asserting the contents of the external datasource from my app's tests.
Related
I need to avoid storing plain text passwords in config files, and so I'm storing the Postgres password externally (in AWS Secrets Manager).
Similarly to the solution provided here:
Encrypted database password in Play + Slick + HikariCP application, I've been able to override dbConfig and supply the password to my DAO classes like this:
trait MyDaoSlick extends MyTableDefinitions with HasDatabaseConfig[MyPostgresDriver] {
protected val dbConfigProvider: DatabaseConfigProvider
override protected val dbConfig: DatabaseConfig[MyPostgresDriver] = secretDbConfig(dbConfigProvider)
def secretDbConfig(dbConfigProvider: DatabaseConfigProvider): DatabaseConfig[MyPostgresDriver] = {
DatabaseConfig.forConfig[MyPostgresDriver]("", dbConfigProvider.get[MyPostgresDriver].config
.withValue("db.user", ConfigValueFactory.fromAnyRef(getUN))
.withValue("db.password", ConfigValueFactory.fromAnyRef(getPWD)))
}
}
This works great for regular DB queries, however evolutions bypass this and still expect the username and the password to be in application.conf, which kind of defeats the purpose of the password being a secret.
Any advice on how evolutions could get the DB credentials from a function?
I ran into the same issue, and I managed to resolve it like this:
Create a custom application loader, as shown here: https://www.playframework.com/documentation/2.7.x/ScalaDependencyInjection#Advanced:-Extending-the-GuiceApplicationLoader
Inside the custom loader's builder, append the DB configuration parameters for Slick:
val extra = Seq(
"slick.dbs.default.db.url" -> secrets.url,
"slick.dbs.default.db.user" -> secrets.user,
"slick.dbs.default.db.password" -> secrets.pass
)
Nothing else needs to be changed, as you've basically added the configuration needed for anything Slick, evolutions included.
On older versions of Play, we used to do this inside GlobalSettings.onLoadConfig, but, at some point, that has been deprecated in favour of DI. More details here: https://www.playframework.com/documentation/2.7.x/GlobalSettings
Our team's application development involves using Effort Testing Tool to mock our Entity Framework's DbContext. However, it seems that Effort Testing Tool needs to be see the actual SQL Server Database that the application uses in order to mock our Entity Framework's DbContext which seems to going against proper Unit Testing principles.
The reason being that in order to unit test our application code by mocking anything related to Database connectivity ( for example Entity Framework's DbContext), we should Never need a Database to be up and running.
How would I configure Effort Testing Tool to mock Entity Framework's DbContext withOut the actual SQL Server Database up and running?
*
Update:
#gert-arnold We are using Entity Framework Model First approach to implement the back-end model and database.
The following excerpt is from the test code:
connection = Effort.EntityConnectionFactory.CreateTransient("name=NorthwindModel");
jsAudtMppngPrvdr = new BlahBlahAuditMappingProvider();
fctry = new BlahBlahDataContext(jsAudtMppngPrvdr, connection, false);
qryCtxt = new BlahBlahDataContext(connection, false);
audtCtxt = new BlahBlahAuditContext(connection, false);
mockedReptryCtxt = new BlahBlahDataContext(connection, false);
_repository = fctry.CreateRepository<Account>(mockedReptryCtxt, null);
_repositoryAccountRoleMaps = fctry.CreateRepository<AccountRoleMap>(null, _repository);
The "name=NorthwindModel" pertains to our edmx file which contains information about our Database tables
and their corresponding relationships.
If I remove the "name=NorthwindModel" by making the connection like the following line of code, I get an error stating that it expects an argument:
connection = Effort.EntityConnectionFactory.CreateTransient(); // throws error
Could you please explain how the aforementioned code should be rewritten?
You only need that connection string because Effort needs to know where the EDMX file is.
The EDMX file contains all information required for creating an inmemory store with an identical schema you have in your database. You have to specify a connection string only because I thought it would be convenient if the user didn't have to mess with EDMX paths.
If you check the implementation of the CreateTransient method you will see that it merely uses the connection string to get the metadata part of it.
public static EntityConnection CreateTransient(string entityConnectionString, IDataLoader dataLoader)
{
var metadata = GetEffortCompatibleMetadataWorkspace(ref entityConnectionString);
var connection = DbConnectionFactory.CreateTransient(dataLoader);
return CreateEntityConnection(metadata, connection);
}
private static MetadataWorkspace GetEffortCompatibleMetadataWorkspace(ref string entityConnectionString)
{
entityConnectionString = GetFullEntityConnectionString(entityConnectionString);
var connectionStringBuilder = new EntityConnectionStringBuilder(entityConnectionString);
return MetadataWorkspaceStore.GetMetadataWorkspace(
connectionStringBuilder.Metadata,
metadata => MetadataWorkspaceHelper.Rewrite(
metadata,
EffortProviderConfiguration.ProviderInvariantName,
EffortProviderManifestTokens.Version1));
}
I am having issues attempting to connect to two different databases in one Qt Application. I have my information database that stores all the information collected by the application and the new Log database which allows me to track all the changes that occur to the Application, button presses, screen loads etc, for easy debugging after its release. Separately, the databases work perfectly, but when I try to use both of them, only one will work. I read that this could be because I wasn't naming the connections and obviously only the most recently connected database could use the default connection. However when I give the databases names they wont work at all, isOpen() will return true on both, but as soon as they attempt to execute a query I get the errors
"QSqlQuery::prepare: database not open"
"QSqlError(-1, "Driver not loaded", "Driver not loaded")"
My two database declarations are:
database_location = filepath.append("/logger.sqlite");
logDB = QSqlDatabase::addDatabase("QSQLITE", "LoggerDatabaseConnection");
logDB.setHostName("localhost");
logDB.setDatabaseName(database_location);
for the Logger Database connection and :
database_location = filepath.append("/db.sqlite");
db = QSqlDatabase::addDatabase("QSQLITE", "NormalDB");
db.setHostName("localhost");
db.setDatabaseName(database_location);
Also when I am running the first query on the databases to see if their tables exist I am using
QSqlQuery query("LoggerDatabaseConnection");
and likewise for the normal database, but I am still getting connection issues even after declaring the database connection to run the query on.
The database used for the application is declared as a static QSqlDatabase in a namespace to create a global effect, so everyone can access it, that was a previous programmer, and I created the Log database as Singleton with a private database connection. Like I said both versions of the code work separately but when they are together they are fighting each other. I know there is a huge debate over the proper design of Singleton vs Dependecy Injection, but again the code works separately so I am happy with how it is designed for now. If there is any missing information or if you have any ideas, please let me know. Thank you.
QSqlQuery query("LoggerDatabaseConnection");
The first parameter of the constructor is the query, not the connection name. It will use the default connection since you specified no database object.
Try something like this:
QSqlQuery query1("YourFirstQuery", db);
QSqlQuery query2("YourSecondQuery", logDB);
Important: Also do not forget to open and close the database before / after using it by calls to QSqlDatabase::open() and QSqlDatabase::close().
The correct way to have multiple databases is to not use the pointer returned from the static addConnection method. You should use the connectionName argument:
https://doc.qt.io/qt-5/qsqldatabase.html#addDatabase-1 during initilization and query usage:
example:
void MyClass::initDb(QString dbPath, QString connName)
{
// initial db usage, etc
QSqlDatabase db = QSqlDatabase::addDatabase(YOUR_DRIVER, connName);
db.setDatabaseName(dbPath);
// open it, etc
}
void MyClass::updateThing(QString val, QString name, QString connName)
{
QString q = QString("UPDATE THINGS SET val=%1 WHERE name=%2").arg(val, name);
// add the reference to your database via the connection name
QSqlDatabase db = QSqlDatabase::database(connName);
QSqlQuery query(db);
query.exec(q);
// handle the query normally, etc
}
Is there a way to dump the generated sql to the Debug log or something? I'm using it in a winforms solution so the mini-profiler idea won't work for me.
I got the same issue and implemented some code after doing some search but having no ready-to-use stuff. There is a package on nuget MiniProfiler.Integrations I would like to share.
Update V2: it supports to work with other database servers, for MySQL it requires to have MiniProfiler.Integrations.MySql
Below are steps to work with SQL Server:
1.Instantiate the connection
var factory = new SqlServerDbConnectionFactory(_connectionString);
using (var connection = ProfiledDbConnectionFactory.New(factory, CustomDbProfiler.Current))
{
// your code
}
2.After all works done, write all commands to a file if you want
File.WriteAllText("SqlScripts.txt", CustomDbProfiler.Current.ProfilerContext.BuildCommands());
Dapper does not currently have an instrumentation point here. This is perhaps due, as you note, to the fact that we (as the authors) use mini-profiler to handle this. However, if it helps, the core parts of mini-profiler are actually designed to be architecture neutral, and I know of other people using it with winforms, wpf, wcf, etc - which would give you access to the profiling / tracing connection wrapper.
In theory, it would be perfectly possible to add some blanket capture-point, but I'm concerned about two things:
(primarily) security: since dapper doesn't have a concept of a context, it would be really really easy for malign code to attach quietly to sniff all sql traffic that goes via dapper; I really don't like the sound of that (this isn't an issue with the "decorator" approach, as the caller owns the connection, hence the logging context)
(secondary) performance: but... in truth, it is hard to say that a simple delegate-check (which would presumably be null in most cases) would have much impact
Of course, the other thing you could do is: steal the connection wrapper code from mini-profiler, and replace the profiler-context stuff with just: Debug.WriteLine etc.
You should consider using SQL profiler located in the menu of SQL Management Studio → Extras → SQL Server Profiler (no Dapper extensions needed - may work with other RDBMS when they got a SQL profiler tool too).
Then, start a new session.
You'll get something like this for example (you see all parameters and the complete SQL string):
exec sp_executesql N'SELECT * FROM Updates WHERE CAST(Product_ID as VARCHAR(50)) = #appId AND (Blocked IS NULL OR Blocked = 0)
AND (Beta IS NULL OR Beta = 0 OR #includeBeta = 1) AND (LangCode IS NULL OR LangCode IN (SELECT * FROM STRING_SPLIT(#langCode, '','')))',N'#appId nvarchar(4000),#includeBeta bit,#langCode nvarchar(4000)',#appId=N'fea5b0a7-1da6-4394-b8c8-05e7cb979161',#includeBeta=0,#langCode=N'de'
Try Dapper.Logging.
You can get it from NuGet. The way it works is you pass your code that creates your actual database connection into a factory that creates wrapped connections. Whenever a wrapped connection is opened or closed or you run a query against it, it will be logged. You can configure the logging message templates and other settings like whether SQL parameters are saved. Elapsed time is also saved.
In my opinion, the only downside is that the documentation is sparse, but I think that's just because it's a new project (as of this writing). I had to dig through the repo for a bit to understand it and to get it configured to my liking, but now it's working great.
From the documentation:
The tool consists of simple decorators for the DbConnection and
DbCommand which track the execution time and write messages to the
ILogger<T>. The ILogger<T> can be handled by any logging framework
(e.g. Serilog). The result is similar to the default EF Core logging
behavior.
The lib declares a helper method for registering the
IDbConnectionFactory in the IoC container. The connection factory is
SQL Provider agnostic. That's why you have to specify the real factory
method:
services.AddDbConnectionFactory(prv => new SqlConnection(conStr));
After registration, the IDbConnectionFactory can be injected into
classes that need a SQL connection.
private readonly IDbConnectionFactory _connectionFactory;
public GetProductsHandler(IDbConnectionFactory connectionFactory)
{
_connectionFactory = connectionFactory;
}
The IDbConnectionFactory.CreateConnection will return a decorated
version that logs the activity.
using (DbConnection db = _connectionFactory.CreateConnection())
{
//...
}
This is not exhaustive and is essentially a bit of hack, but if you have your SQL and you want to initialize your parameters, it's useful for basic debugging. Set up this extension method, then call it anywhere as desired.
public static class DapperExtensions
{
public static string ArgsAsSql(this DynamicParameters args)
{
if (args is null) throw new ArgumentNullException(nameof(args));
var sb = new StringBuilder();
foreach (var name in args.ParameterNames)
{
var pValue = args.Get<dynamic>(name);
var type = pValue.GetType();
if (type == typeof(DateTime))
sb.AppendFormat("DECLARE #{0} DATETIME ='{1}'\n", name, pValue.ToString("yyyy-MM-dd HH:mm:ss.fff"));
else if (type == typeof(bool))
sb.AppendFormat("DECLARE #{0} BIT = {1}\n", name, (bool)pValue ? 1 : 0);
else if (type == typeof(int))
sb.AppendFormat("DECLARE #{0} INT = {1}\n", name, pValue);
else if (type == typeof(List<int>))
sb.AppendFormat("-- REPLACE #{0} IN SQL: ({1})\n", name, string.Join(",", (List<int>)pValue));
else
sb.AppendFormat("DECLARE #{0} NVARCHAR(MAX) = '{1}'\n", name, pValue.ToString());
}
return sb.ToString();
}
}
You can then just use this in the immediate or watch windows to grab the SQL.
Just to add an update here since I see this question still get's quite a few hits - these days I use either Glimpse (seems it's dead now) or Stackify Prefix which both have sql command trace capabilities.
It's not exactly what I was looking for when I asked the original question but solve the same problem.
I'm using Yii Framework to create my project. I need to export some data from MySQL (my project) to an external Microsoft SQL server which is on the same network.
Basically, the user needs to click on a button (which will do the export-insert) in my view and the results should be displayed - Success (if the query has been successful) or Failure (if something went wrong).
The results part is quite easy as I'll be using 'setFlash' to display the appropriate message but I want to know how to insert data into an external database through Yii.
Do you have any idea how this can be done?
Well, I agree with #SuVeRa on the first part of defining two db instances in the config.php but i don't think the sql Commands part is necessary (Plus i hate writing sql :D )
Instead you can do:
class SomeModel extends CActiveRecord
{
...
// Override the getDbConnection() function to use the ms sql db connection
public function getDbConnection()
{
return Yii::app()->ms_sql_db_connection; // The name of the connection in config.php
}
public function transfer()
{
// Here you can do all the transferring logic using normal Yii Active Record functions
}
}
Check out the docs on getDbConnection().