I have a spring boot application where I use mssql-jdbc to connect to my sql-server database and I am using Hikari connection pool. There is a line of code in my application which creates a HikaryDataSource for a list of data sources in my application. The code looks like the following
for(HikaryConfig config: configs) {
HikaryDatasource ds = new HikaryDataSource(config);
// store the ds in HashMap for later use
}
The above code works fine and I was able to fill my HashMap with HikaryDataSources
I wanted to improve the above to use java parallel-streams to create the Datasources parallely. The changed code looks like the following
configs.parallelStream(config -> {
HikaryDatasource ds = new HikaryDataSource(config);
// store the ds in HashMap for later use
});
On the above implementation, some data sources are created and some failed with a message.
IlegalStateException: Cache already created: 'database-code'.
java.lang.IllegalStateException: Cache already created: 'database-code' at org.cache2k.core.CacheManagerImpl.newCache(CacheManagerImpl.java:176) at org.cache2k.core.InternalCache2kBuilder.buildAsIs(InternalCache2kBuilder.java:266) at org.cache2k.core.InternalCache2kBuilder.build(InternalCache2kBuilder.java:202) at org.cache2k.core.Cache2kCoreProviderImpl.createCache(Cache2kCoreProviderImpl.java:228) at org.cache2k.Cache2kBuilder.build(Cache2kBuilder.java:933)
I am wandering if the parallel-stream causes it
Related
I'm trying to manage a decentralized DB around a huge number of partial DB instances. Each instance has a subset of the whole data and they are all nodes and clients, thus asking for some data the query must be spread to every (group) instance and which one have it will return the data.
Due to avoid lost of data if one instance goes down, I figured out they must replicate its contents with some others. How this scenario can be configured with Ignite?
Supose I have a table with the name and last access datetime of users in a distributed application, like ...
class UserLogOns
{
string UserName;
DateTime LastAccess;
}
Now when the program starts I prepare Ingite for work as a decentralizad DB ...
static void Main(string[] args)
{
TcpCommunicationSpi commSpi = new TcpCommunicationSpi();
// Override local port.
commSpi.LocalPort = 44444;
commSpi.LocalPortRange = 0;
IgniteConfiguration cfg = new IgniteConfiguration();
// Override default communication SPI.
cfg.CommunicationSpi = commSpi;
using (var ignite = Ignition.Start(cfg))
{
var cfgCache = new CacheConfiguration("mio");
cfgCache.AtomicityMode = CacheAtomicityMode.Transactional;
var cache = ignite.GetOrCreateCache<string, UserLogOns>(cfgCache);
cache.Put(Environment.MachineName, new UserLogOns { UserName = Environment.MachineName, LastAccess = DateTime.UtcNow });
}
}
And now ... I want to get LastAccess of other "computerB" when ever it was ..
Is this correct? How can it be implemented?
It depends on the exact use-case that you want to implement. In general, Ignite provides out of the box everything that you mentioned here.
This is a good way to start with using SQL in Ignite: https://apacheignite-sql.readme.io/docs
Create table with "template=partitioned" instead of "replicated" as it is shown in the example here: https://apacheignite-sql.readme.io/docs/getting-started#section-creating-tables, configure number of backups and select a field to be affinity key (a field that is used to map specific entries to cluster node) and just run some queries.
Also check out the concept of baseline topology if you are going to use native persistence: https://apacheignite.readme.io/docs/baseline-topology.
In-memory mode will trigger rebalance between nodes on each server topology change (node that can store data in/out) automatically.
I need to avoid storing plain text passwords in config files, and so I'm storing the Postgres password externally (in AWS Secrets Manager).
Similarly to the solution provided here:
Encrypted database password in Play + Slick + HikariCP application, I've been able to override dbConfig and supply the password to my DAO classes like this:
trait MyDaoSlick extends MyTableDefinitions with HasDatabaseConfig[MyPostgresDriver] {
protected val dbConfigProvider: DatabaseConfigProvider
override protected val dbConfig: DatabaseConfig[MyPostgresDriver] = secretDbConfig(dbConfigProvider)
def secretDbConfig(dbConfigProvider: DatabaseConfigProvider): DatabaseConfig[MyPostgresDriver] = {
DatabaseConfig.forConfig[MyPostgresDriver]("", dbConfigProvider.get[MyPostgresDriver].config
.withValue("db.user", ConfigValueFactory.fromAnyRef(getUN))
.withValue("db.password", ConfigValueFactory.fromAnyRef(getPWD)))
}
}
This works great for regular DB queries, however evolutions bypass this and still expect the username and the password to be in application.conf, which kind of defeats the purpose of the password being a secret.
Any advice on how evolutions could get the DB credentials from a function?
I ran into the same issue, and I managed to resolve it like this:
Create a custom application loader, as shown here: https://www.playframework.com/documentation/2.7.x/ScalaDependencyInjection#Advanced:-Extending-the-GuiceApplicationLoader
Inside the custom loader's builder, append the DB configuration parameters for Slick:
val extra = Seq(
"slick.dbs.default.db.url" -> secrets.url,
"slick.dbs.default.db.user" -> secrets.user,
"slick.dbs.default.db.password" -> secrets.pass
)
Nothing else needs to be changed, as you've basically added the configuration needed for anything Slick, evolutions included.
On older versions of Play, we used to do this inside GlobalSettings.onLoadConfig, but, at some point, that has been deprecated in favour of DI. More details here: https://www.playframework.com/documentation/2.7.x/GlobalSettings
I am attempting to use multiple datasources in a Grails 2.4.4 project. According to the docs, this should be possible:
http://www.grails.org/doc/2.4.4/guide/conf.html#multipleDatasources
My primary dataSource (the one I want to use for all domain classes) is using H2 at the moment, as configured by the default DataSource.groovy configuration. My second, read-only datasource is SQL Server, and I tried to declare it as follows at the top level of my DataSource.groovy config (shared by all environments):
ds {
pooled = true
dialect = "org.hibernate.dialect.SQLServer2008Dialect"
driverClassName = "net.sourceforge.jtds.jdbc.Driver"
url = "jdbc:jtds:sqlserver://myserver:1433/mydb;domain=mydomain;useNTLMv2=true;user=myuser"
dbCreate = "none"
}
(Don't let the URL throw you off - I'm just having to use Windows Auth with JTDS. I've tested this via third-party clients as well.)
I inject this into my service class and use it, and everything appears to hook up well:
def dataSource_ds
def serviceMethod(){
Sql ds = new Sql(dataSource_ds)
String query = "SELECT ... "
def results = ds.rows(query)
println "Results are ${results.size()}"
return "Some value"
}
But when I try to access this from an IntegrationSpec-backed Integration Test, I noticed that I was getting "schema not found" errors for valid schemas referred to by my query string, such as "dbo". And the stack trace of any errors from this setup looks like this:
org.h2.jdbc.JdbcSQLException: Schema "DBO" not found; SQL statement:
...
at org.h2.message.DbException.getJdbcSQLException(DbException.java:329)
at org.h2.message.DbException.get(DbException.java:169)
at org.h2.message.DbException.get(DbException.java:146)
at org.h2.command.Parser.readTableOrView(Parser.java:4774)
at org.h2.command.Parser.readTableFilter(Parser.java:1083)
at org.h2.command.Parser.parseSelectSimpleFromPart(Parser.java:1689)
at org.h2.command.Parser.parseSelectSimple(Parser.java:1796)
at org.h2.command.Parser.parseSelectSub(Parser.java:1683)
at org.h2.command.Parser.parseSelectUnion(Parser.java:1526)
at org.h2.command.Parser.parseSelect(Parser.java:1514)
at org.h2.command.Parser.parsePrepared(Parser.java:404)
at org.h2.command.Parser.parse(Parser.java:278)
at org.h2.command.Parser.parse(Parser.java:250)
at org.h2.command.Parser.prepareCommand(Parser.java:217)
at org.h2.engine.Session.prepareLocal(Session.java:414)
at org.h2.engine.Session.prepareCommand(Session.java:363)
...
Now why would THIS datasource be trying to use the H2 driver?
In case it's relevant, my Integration test looks like this:
void "serviceMethod" () {
when: "service method is called"
String response = myService.serviceMethod()
then: "we should get the appropriate text back"
response.equals("Some value")
}
If, in the Service class, I hard-code the connection using a constructor of the Groovy Sql object, the integration test works fine, and any stack traces go through the JTDS driver.But when I try to use the injected datasource, things are strange.
Any idea what I'm doing wrong here?
Just to close the loop on this and hopefully save someone pain on this oversight in the future:
Grails uses an in-memory database when running tests. Make sure to read up on the other differences between integration tests and production here:
http://www.grails.org/doc/latest/guide/testing.html#integrationTesting
This feature makes the use of external (read-only) datasources during any tests pretty interesting, but some of that is to be expected (a test which depends on an external datasource is not a very good test in the long run). I hope to refactor my app and its testing approach at some point (e.g., to use a simple DAO and mock that during the test), because I don't really care about asserting the contents of the external datasource from my app's tests.
Our team's application development involves using Effort Testing Tool to mock our Entity Framework's DbContext. However, it seems that Effort Testing Tool needs to be see the actual SQL Server Database that the application uses in order to mock our Entity Framework's DbContext which seems to going against proper Unit Testing principles.
The reason being that in order to unit test our application code by mocking anything related to Database connectivity ( for example Entity Framework's DbContext), we should Never need a Database to be up and running.
How would I configure Effort Testing Tool to mock Entity Framework's DbContext withOut the actual SQL Server Database up and running?
*
Update:
#gert-arnold We are using Entity Framework Model First approach to implement the back-end model and database.
The following excerpt is from the test code:
connection = Effort.EntityConnectionFactory.CreateTransient("name=NorthwindModel");
jsAudtMppngPrvdr = new BlahBlahAuditMappingProvider();
fctry = new BlahBlahDataContext(jsAudtMppngPrvdr, connection, false);
qryCtxt = new BlahBlahDataContext(connection, false);
audtCtxt = new BlahBlahAuditContext(connection, false);
mockedReptryCtxt = new BlahBlahDataContext(connection, false);
_repository = fctry.CreateRepository<Account>(mockedReptryCtxt, null);
_repositoryAccountRoleMaps = fctry.CreateRepository<AccountRoleMap>(null, _repository);
The "name=NorthwindModel" pertains to our edmx file which contains information about our Database tables
and their corresponding relationships.
If I remove the "name=NorthwindModel" by making the connection like the following line of code, I get an error stating that it expects an argument:
connection = Effort.EntityConnectionFactory.CreateTransient(); // throws error
Could you please explain how the aforementioned code should be rewritten?
You only need that connection string because Effort needs to know where the EDMX file is.
The EDMX file contains all information required for creating an inmemory store with an identical schema you have in your database. You have to specify a connection string only because I thought it would be convenient if the user didn't have to mess with EDMX paths.
If you check the implementation of the CreateTransient method you will see that it merely uses the connection string to get the metadata part of it.
public static EntityConnection CreateTransient(string entityConnectionString, IDataLoader dataLoader)
{
var metadata = GetEffortCompatibleMetadataWorkspace(ref entityConnectionString);
var connection = DbConnectionFactory.CreateTransient(dataLoader);
return CreateEntityConnection(metadata, connection);
}
private static MetadataWorkspace GetEffortCompatibleMetadataWorkspace(ref string entityConnectionString)
{
entityConnectionString = GetFullEntityConnectionString(entityConnectionString);
var connectionStringBuilder = new EntityConnectionStringBuilder(entityConnectionString);
return MetadataWorkspaceStore.GetMetadataWorkspace(
connectionStringBuilder.Metadata,
metadata => MetadataWorkspaceHelper.Rewrite(
metadata,
EffortProviderConfiguration.ProviderInvariantName,
EffortProviderManifestTokens.Version1));
}
We noticed a slight oddity in the Sitecore API code. The code is below for your reference. The code is trying to get a database by doing new Database(database). But randomly it was failing.
This code worked for a while with Database db = new Database(database); but started failing randomly yesterday. When we changed the code to Database db = Database.GetDatabase(database);, the code started working again. What is the difference between the two approaches and what is recommended by Sitecore?
I've seen this happen twice now - multiple times in production and a couple of times in my development environment.
public static void DeleteItem(string id, stringdatabase)
{
//get the database
Database db = new Database(database);
//get the item
item = db.GetItem(new ID(id));
if (item != null)
{
using(new Sitecore.SecurityModel.SecurityDisabler())|
{
//delete the item
item.Delete();
}
}
}
A common way you will see people get a specific database is:
Sitecore.Data.Database master = Sitecore.Configuration.Factory.GetDatabase("master");
This is equivalent to Sitecore.Data.Database.GetDatabase("master").
When you call either of these methods it will first check the cache for the database. If not found it will build up the database with all of the configuration values within the config file via reflection. Once the database is created it will be placed in the cache for future use.
When you use the constructor on the database it is simply creating a rather empty database object. I am rather suprised to hear it was working at all when you used this method.
The proper approach to get a specific database would be to use:
Sitecore.Configuration.Factory.GetDatabase("master");
// or
Sitecore.Data.Database.GetDatabase("master");
If you are looking to get the database used with the current request (aka context database) you can use Sitecore.Context.Database. You can also use Sitecore.Context.ContentDatabase.