I am trying to test connection pool using HikariCP against SnowFlake.
I have kept the pool size and idle size, both, set to 5. However I observed that while getting a connection, every time its creating a new connection. For example , If i make 8 connection, i could see 8 test sql query being executed on Snowflake (I checked it in history.). My expectation was it should not create more that 5 (pool size) connection and this is justified by only 5 test sql query on snowflake.
Does anyone have any clue?
Configuration:
micronaut:
application:
name: mnsfpoc
netty:
default:
allocator:
max-order: 3
datasources:
default:
url: jdbc:snowflake://xxx.azure.snowflakecomputing.com/?warehouse=xx&db=xx&schema=xx&role=xx
username: xxxx
password: xxxx
driverClassName: net.snowflake.client.jdbc.SnowflakeDriver
maximum-pool-size: 5
minimum-idle: 5
pool-name: mnsfconn-pool
connection-test-query: select 3
Code to get connection:
#Singleton
public class RestDao extends BaseDao {
#Inject
private DataSource dataSource;
public String greet(String name) throws Exception {
Connection conn1 = dataSource.getConnection();
close(conn1);
return "Hello " + name + "!!";
}
}
Thanks
Rakhs
Related
I have written the following small Lambda function in .NET core for testing purposes. It is triggered on a PUT in an S3 bucket, and should show the object name in CloudWatch and insert the object name into a table in an RDS instance.
Here is the code:
public async Task<string> FunctionHandler(S3Event evnt, ILambdaContext context)
{
context.Logger.LogLine("context.Logger.LogLine function begin");
var s3Event = evnt.Records?[0].S3;
if(s3Event == null)
{
context.Logger.LogLine("No S3 Events recorded");
return null;
}
try
{
var response = await this.S3Client.GetObjectMetadataAsync(s3Event.Bucket.Name, s3Event.Object.Key);
context.Logger.LogLine("S3 Key: " + s3Event.Object.Key.ToString());
var connectionString = Environment.GetEnvironmentVariable("ConnectionString");
SqlConnection c = new SqlConnection(connectionString);
c.Open();
SqlCommand cmd = new SqlCommand("INSERT INTO LambdaTestTable (ValueText) VALUES('" + s3Event.Object.Key + "')", c);
cmd.ExecuteNonQuery();
c.Close();
return response.Headers.ContentType;
}
catch(Exception e)
{
context.Logger.LogLine($"Error getting object {s3Event.Object.Key} from bucket {s3Event.Bucket.Name}. Make sure they exist and your bucket is in the same region as this function.");
context.Logger.LogLine(e.Message);
context.Logger.LogLine(e.StackTrace);
throw;
}
}
I get the following output in CloudWatch:
START RequestId: 15c5eddb-8171-11e7-af34-251688cfddb2 Version: $LATEST
Console.WriteLine function begin context.Logger.LogLine function begin
S3 Key: TaskTemplate.dp Error found Error getting object
TaskTemplate.dp from bucket api-dev-dpstorage-s3. Make sure they exist
and your bucket is in the same region as this function. A
network-related or instance-specific error occurred while establishing
a connection to SQL Server. The server was not found or was not
accessible. Verify that the instance name is correct and that SQL
Server is configured to allow remote connections. (provider: TCP
Provider, error: 40 - Could not open a connection to SQL Server)
Two errors I am unclear on here. Firstly, while CloudWatch successfully shows my object key once the file is PUT to the bucket, it then has an error saying it can't find the object? Secondly, it has a network / instance error trying to connect to the SQL RDS instance. I am using a connection string which is used successfully within an EC2 instance by another application.
UPDATE
I could never make this work with a "Windows Authentication" (domain) user. But with a "SQL Server Authentication" user everything is working like it's supposed to.
ORIGINAL QUESTION
My connectionString: Server=ip;Database=dbname;User Id=xxx\user;Password=pass;
The connection string is located in appsettings.json like this:
{
"Logging": {
"IncludeScopes": false,
"LogLevel": {
"Default": "Warning"
}
},
"ConnectionStrings": {
"ConnectionString": "Server=ip;Database=dbname;User Id=xxx\user;Password=pass;"
}
}
Then i pass it to a static class from the "Startup.cs" file, like this:
public void ConfigureServices(IServiceCollection services)
{
// Add framework services.
services.AddMvc();
Orm.DatabaseConnection.ConnectionString = Configuration["ConnectionStrings:ConnectionString"];
}
This is where I initiate the connection:
using System.Data.SqlClient;
namespace MyProject.Orm
{
public static class DatabaseConnection
{
public static string ConnectionString { get; set; }
public static SqlConnection ConnectionFactory()
{
return new SqlConnection(ConnectionString);
}
}
}
And this is my controller:
public string Get()
{
using (var databaseConnection = Orm.DatabaseConnection.ConnectionFactory())
{
var sections = databaseConnection.Query("SELECT * FROM myTable").ToList();
return sections.ToString();
}
}
Where this line:
var databaseConnection = Orm.DatabaseConnection.ConnectionFactory();
returns:
ServerVersion: "'databaseConnection.ServerVersion' threw an exception of type 'System.InvalidOperationException'"
Message: "Invalid operation. The connection is closed."
Source: "System.Data.SqlClient"
StackTrace: "at
System.Data.SqlClient.SqlConnection.GetOpenTdsConnection()\n
at
System.Data.SqlClient.SqlConnection.get_ServerVersion()"
And i get this error on new SqlConnection: "error CS0119: 'SqlConnection' is a type, which is not valid in the given context".
But the program execution doesn't stop because of these errors.
The application then hangs on the following line:
var sections = databaseConnection.Query("SELECT * FROM myTable").ToList();
I'm using Dapper as my ORM (not EntityFramework). In "myTable" sql table are only 17 rows and 5 columns so it should load fast.
I tried all kinds of different connectionStrings but it always fails. If i try the same with .NET Framework 4.5, everything works fine. The problem is .NET Core 2.0.
Any idea about fixing it is welcome. Because i spent too many hours on this already.
Try to add databaseConnection.Open().
public string Get()
{
using (var databaseConnection = new SqlConnection(#"Server=ip;Database=dbname;User Id=xxx\user;Password=pass;Pooling=false;"))
{
databaseConnection.Open();
var sections = databaseConnection.Query("SELECT * FROM myTable").ToList();
return sections.ToString();
}
}
To avoid problems with connection pool that described in comments you add Pooling=false; to connection string:
Server=ip;Database=dbname;User Id=xxx\user;Password=pass;Pooling=false;
Edit:
I hardcoded connection string and removed factory to make example smaller
Try creating a self-contained deployment, this should eliminate and strange dependency stuff. If it works then at least you know that it's due to some assembly binding type stuff.
The exception "error CS0119: 'SqlConnection' is a type, which is not valid in the given context" smells like it is.
In our application we are using Spring Boot and Tomcat connection pool manager. The connections are acquired using a specific user. So anyone who connects to the application regardless of their username, we use the application's username to persist data. However, for certain operation we want to impersonate using MSSQL command 'setuser'. How can we achieve this?
To make it more clear, what we want is some sort of interceptor so that before executing a sql statement we can execute this command:
setuser 'user_a'
and before releasing the connection we can execute:
setuser
so that impersonation resets.
Assume that in the interceptor we know whether the connection should be impersonated or not.
I solved this by using Jdbc interceptor. The method in the interceptor will be called whenever a connection is borrowed from the pool. Please check the below link:
Jdbc interceptor
Here is my code:
public class ImpersonatorJdbcInterceptor extends JdbcInterceptor {
private static final Logger logger = LoggerFactory.getLogger(ImpersonatorJdbcInterceptor.class);
/**
* This method gets invoked whenever a connection is borrowed from the pool.
* Since releasing a connection back to pool cannot be intercepted in this method we
* need to first reset the previous impersonation.
* {#inheritDoc}
*/
#Override
public void reset(ConnectionPool connectionPool, PooledConnection pooledConnection) {
Authentication auth = SecurityContextHolder.getContext().getAuthentication();
logger.debug("A connection is borrowed from the pool by {}", (auth != null ? auth.getName() : "null"));
Connection conn = pooledConnection.getConnection();
/*
* Note:
* 1 We can't impersonate current user
* 2 We can't impersonate a user when the current session is already impersonated.
* 3 'setuser' without username resets back to original user who has created the connection
* 4 The user must have the right permission to be able to impersonate
* 5 It is the caller's responsibility to close the connection; We only close the connection if this execution fails
*/
try (PreparedStatement prep = conn.prepareStatement(auth != null ? "setuser; setuser ?" : "setuser")) {
if(auth != null) {
prep.setString(1, auth.getName());
}
prep.execute();
} catch (SQLException e) {
logger.error("Impersonation failed. Please check permission for 'setuser': " + (auth != null ? auth.getName() : "null"), e);
close(conn);
throw new IllegalStateException("Oops! Cannot execute statements, please contact the administrator", e);
}
}
/**
* Closes the statement and rolls back only if something goes wrong.
* #param conn
*/
private void close(Connection conn) {
try {
if(conn != null && !conn.isClosed()) {
conn.rollback();
conn.close();
}
} catch (SQLException e) {
logger.error("Something went wrong and the connection cannot be closed", e);
//the caller will throw exception
}
}
}
I am working on some code that leverages Dropwizard that will require that I need to connect to at least two different databases (I plan to use Hibernate as well). I was unable to find any examples/documentation that will allow me to configure two different database connections in the Database block of the .yml configuration file. Is this possible in Dropwizard? If not, what are the workarounds that people have used in the past. Thank you in advanced for your help!
You can configure multiple databases in dropwizard. In the config.yml you can have multiple database configuration like this.
database1:
driverClass: org.postgresql.Driver
user: user
password: pwd
url: jdbc:postgresql://localhost:5432/db1
validationQuery: select 1
minSize: 2
maxSize: 8
database2:
driverClass: org.postgresql.Driver
user: user
password: pwd
url: jdbc:postgresql://localhost:5432/db2
validationQuery: select 1
minSize: 2
maxSize: 8
And in the config class get both config details.
public class DBConfig extends Configuration {
private DatabaseConfiguration database1;
private DatabaseConfiguration database2;
public DatabaseConfiguration getDatabase1() {
return database1;
}
public DatabaseConfiguration getDatabase2() {
return database2;
}
}
And in your service configure which Dao to use which database.
#Override
public void run(MyConfiguration configuration,
Environment environment) throws ClassNotFoundException {
...
final DBIFactory factory = new DBIFactory();
// Note that the name parameter when creating the DBIs must be different
// Otherwise you get an IllegalArgumentException
final DBI jdbi1 = factory.build(
environment, configuration.getUserDatabase(), "db1");
final DBI jdbi2 = factory.build(
environment, configuration.getItemDatabase(), "db2");
final MyFirstDAO firstDAO = jdbi1.onDemand(MyFirstDAO.class);
final MySecondDAO secondDAO = jdbi2.onDemand(MySecondDAO.class);
...
}
I am using ServiceStack and OrmLite.Oracle. I connect to an old Oracle 7.3 instance using ODBC Driver for Oracle on a Windows Server 2012 x64. ODBC is setup as an ODBC32.
I connect and query the database from my repo like this:
using (IDbConnection db = _context.DbFactory.OpenDbConnection())
{
return db.Select<T>();
}
The _context hold the OrmLiteConnectionFactory which was created like this:
DbFactory= new OrmLiteConnectionFactory(conInfo.ConnectionString,false, ServiceStack.OrmLite.Oracle.OracleDialect.Provider);
My service is running just fine and I can access and query the database, no problem.
But after a certain period of time (30 minutes or so), the connection is lost and I have to restart my service (hosted in a Windows Service) because the call to Open the connection will give me this error: unable to allocate an environment handle.
It might be a normal thing to release the handle to the connection after a while but why it simply doesn't reconnect to it? From OrmLite code, I can see that OpenDbConnection should return a new instance of its connection when the AutoDisposeConnection is set to True or if the internal ormLiteConnection is null. I guess my connection is not null but not quite alive...
private OrmLiteConnection ormLiteConnection;
private OrmLiteConnection OrmLiteConnection
{
get
{
if (ormLiteConnection == null)
{
ormLiteConnection = new OrmLiteConnection(this);
}
return ormLiteConnection;
}
}
public IDbConnection OpenDbConnection()
{
var connection = CreateDbConnection();
connection.Open();
return connection;
}
public IDbConnection CreateDbConnection()
{
if (this.ConnectionString == null)
throw new ArgumentNullException("ConnectionString", "ConnectionString must be set");
var connection = AutoDisposeConnection
? new OrmLiteConnection(this)
: OrmLiteConnection;
return connection;
}
I have tried to set the AutoDisposeConnection to True but when I do, I always get an AccessViolationException saying "Attempted to read or write protected memory. This is often an indication that other memory is corrupt.". What does that mean? Is this an OS, ODBC or OrmLite error? Any idea why this is happening?
I have to say that because I am using Oracle 7.3, I had to recompile the ServiceStack.OrmLite.Oracle.dll so it uses the System.Data.Odbc rather than System.Data.OracleClient (only compatible with v8+).
I really want to avoid to test if the connection is alive or not at every call, so any help to make this work is greatly appreciated. Thanks