Improve RESTful response time - database

I'm implementing a RESTful Service in Java EE with TomEE+(Apache CXF) which is using a database(specifically postgres). Now I've noticed that the longest time in my functions is spent in the getConnection() call for the database.
my code looks like this:
import java.sql.Connection;
import java.sql.SQLException;
import javax.annotation.Resource;
import javax.sql.DataSource;
import javax.ws.rs.Consumes;
import javax.ws.rs.Path;
import javax.ws.rs.GET;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
#Path("/test")
public class TestResource {
/* external DB resource
* configured in the resources.xml as "testDB". Either match the
* name or use the name parameter of the resource annotation.
*/
#Resource private DataSource testDB;
#Path("/hello")
#GET
#Produces("text/plain")
public String test() throws SQLException
{
Connection conn = testDB.getConnection(); //majority of time is spent in here
/*
do something.(e.g. PreparedStatement ps = conn.prepareStatement(...)
*/
conn.close();
return "world";
}
}
The database is defined in a resources.xml like so:
<?xml version="1.0" encoding="UTF-8"?>
<resources>
<Resource id="testDB" type="javax.sql.DataSource">
accessToUnderlyingConnectionAllowed = false
connectionProperties =
defaultAutoCommit = true
defaultReadOnly =
definition =
ignoreDefaultValues = false
initialSize = 0
jdbcDriver = org.postgresql.Driver
jdbcUrl = jdbc:postgresql://localhost/testdb
jtaManaged = true
maxActive = 100
maxIdle = 20
maxOpenPreparedStatements = 0
maxWaitTime = 100 millisecond
minEvictableIdleTime = 30 minutes
minIdle = 0
numTestsPerEvictionRun = 3
password = password
passwordCipher = PlainText
poolPreparedStatements = false
serviceId =
testOnBorrow = true
testOnReturn = false
testWhileIdle = false
timeBetweenEvictionRuns = -1 millisecond
userName = user
validationQuery = SELECT 1;
removeAbandoned = true
removeAbandonedTimeout = 60
logAbandoned = true
</Resource>
</resources>
So how could I reduce the time it takes to get a database connection? I'm already using the connection pooling mechanism. The only solution that comes to my mind is to make the resource class a singleton and get the connection once, but that seems counter-intuitive when many requests need to be worked on.

I would normally expect the interaction with the database to be the slowest part of the process. I would be interested in knowing how much overhead the database call is adding? How is that in relation to making the database call via a SQL tool outside of the Java code.
Also - I'm not sure if the example is just for brevity, but I would try to layer your solution so that the initial layer handles routing and validation, while the next layer handles any business logic and interacts with a database (DAO) layer.

Related

Azure Terraform import bacpac into SQL Server with public network access disabled

I have a SQL Server in Azure with public network access disabled
resource "azurerm_mssql_server" "sql_server" {
name = var.db-server-name
resource_group_name = var.resource_group
location = var.location
version = "12.0"
administrator_login = local.login
administrator_login_password = local.password
minimum_tls_version = "1.2"
public_network_access_enabled = false
tags = var.tags
}
For accessing the server, I am creating a private endpoint:
resource "azurerm_private_endpoint" "sqlserver_private_endpoint" {
name = "sqlserver-private-endpoint"
location = var.location
resource_group_name = var.resource_group
subnet_id = azurerm_subnet.db_subnet.id
private_service_connection {
name = "sqlserver-psc"
is_manual_connection = false
private_connection_resource_id = azurerm_mssql_server.sql_server.id
subresource_names = ["sqlServer"]
}
tags = var.tags
}
I am then trying to create/import a database from Blob-Storage
resource "azurerm_mssql_database" "sql_server_database" {
name = var.db-name
server_id = azurerm_mssql_server.sql_server.id
collation = "SQL_Latin1_General_CP1_CI_AS"
auto_pause_delay_in_minutes = 60
max_size_gb = 32
min_capacity = 0.5
read_replica_count = 0
read_scale = false
sku_name = "GP_S_Gen5_1"
zone_redundant = false
import {
storage_uri = var.storage-url
storage_key = var.storage-key
storage_key_type = "StorageAccessKey"
administrator_login = azurerm_mssql_server.sql_server.administrator_login
administrator_login_password = azurerm_mssql_server.sql_server.administrator_login_password
authentication_type = "Sql"
}
}
With this setup, I get the following error
Error: while import bacpac into the new database test-db (Resource Group Test-dev): Code="ImportExportJobError" Message="The ImportExport operation with Request Id '1b005b56-bccd-4484-a5e0-c2495834798a' failed due to 'The SQL instance is inaccessible because the public network interface is denied (Error 47073). Please enable public network access on the SQL Server or configure Import/Export to use Private Link per https://docs.microsoft.com/en-us/azure/azure-sql/database/database-import-export-private-link.'."
with module.sql_server.azurerm_mssql_database.sql_server_database, on Modules\SqlServer\main.tf line 69, in resource "azurerm_mssql_database" "sql_server_database":
69: resource "azurerm_mssql_database" "sql_server_database" {
This error makes sense as I have set public_network_access_enabled = false on my SQL Server.
For security reasons I would not like to set
public_network_access_enabled = true
so my question would be: is there a possibility to import the database without enabling network access on the server?
Here I found a way to import the database using PowerShell, which should create a Privatelink for importing, but using this the Database would not be created using Terraform and would not be managed through the Terraform-State...
So does someone know of a way to import the database using Terraform with public_network_access_enabled = false?
(AzureRM Provider Version: 3.31)
Using Private Link for importing seems to be still in Preview. Preview features are rarely (or never) supported with Terraform so I would search alternatives for now.
I personally would create the database with Terraform and then manually (or with some CI/CD magic) import the required data. Generally Terraform is a bit clunky tool in managing what happens inside a database and I personally like using other tools for it.

Filesystem changes using watchdog python and sent the file to SQL Server

import pandas
import os
import sqlalchemy
import sys
import time from watchdog.observers
import Observer from watchdog.events
import FileSystemEventHandler
class EventHandler(FileSystemEventHandler):
def on_any_event(self, event):
print("EVENT")
print(event.event_type)
print(event.src_path)
print()
if __name__ == "__main__":
path = 'input path here'
event_handler= EventHandler()
observer = Observer()
observer.schedule(event_handler, path, recursive=True)
print("Monitoring started")
observer.start()
try:
while(True):
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
engine = create_engine('Database information')
cursor = engine.raw_connection().cursor()
for file in os.listdir('.'):
file_basename, extension = file.split('.')
if extension == 'xlsx':
df = pd.read_excel(os.path.abspath(file))
df.to_sql(file_basename, con = engine, if_exists = 'replace')
so the first part where observer.join() ends i am okay with. but the next part where it starts with engine = create_engine("") that where i am having trouble with that part is suppose to send the file to the sql server but the code is not doing that i have researched the internet but having found anything. any help is appreciated.

Spring data JPA waiting find

I have some method which import prices for airports and save it to database through Spring data JPA repositories.
Import method is this:
#Transactional
public Future<Boolean> importFuel(File serverFile, Long providerIdLong) {
final FuelProvider fuelProvider = fuelProviderRepository
.findOne(providerIdLong);
LOG.debug("fuelProvider:" + fuelProvider.getName());
List<AirportFuel> airportFuels = processors
.get(providerIdLong).process(serverFile, fuelProvider);
if(airportFuels==null){
return new AsyncResult<>(false);
}
airportFuelRepository.deleteByFpId(providerIdLong);
airportFuelRepository.save(airportFuels);
fuelProvider.setUpdated(new Date());
fuelProviderRepository.save(fuelProvider);
return new AsyncResult<>(true);
}
For example this is read method:
List<AirportFuel> airportFuels = airportFuelRepository.findByIata(airport.getIata());
and AirportFuelRepository:
#Repository
#Transactional(readOnly = true)
public interface AirportFuelRepository extends CrudRepository<AirportFuel,
Long> {
#Transactional
#Modifying
#Query("delete from AirportFuel af where af.fpId = :#{#fpId}")
void deleteByFpId(#Param("fpId") Long fpId);
#Transactional(readOnly = true)
List<AirportFuel> findByIata(String iata);
List<AirportFuel> findByIataAndFpId(String iata, Long fpId);
}
Application is based on Spring boot, Hibernate, Spring data JPA and MS SQL.
If import method running then other method which also use airportFuelRepository waiting for end of this method and i do not know why. I suppose the reading method will work with the data before the import and will not waiting to end import method.
Thank you for advice.
If I understand correctly you want to have both methods be able to access the same repository asynchronously.
In order to do this annotate your method with the #Async annotation.
A few links to get you started:
baeldung.com
Spring Quickstart
Spring Javadoc
Dzone

JBoss 6 EAP - override HTTP response status code in a SOAP service that sends empty response back from 202 to 200

We have a SOAP web service that we are migrating from JBoss EAP 5.1 to 6.4.7 and one of the webservices returns absolutely nothing but 200 (in JBoss 5). When we migrated to 6 it still works and returns nothing but returns a 202 instead and that is going to break clients. We have no control over clients. I tried a SOAPHandler at the close method but it does nothing as it is not even called as my guess is that since there is no SOAP message going back there is nothing that triggers the handler.
I also tried to access the context directly in the web method and modif but it did nothing.
MessageContext ctx = wsContext.getMessageContext();
HttpServletResponse response = (HttpServletResponse) ctx.get(MessageContext.SERVLET_RESPONSE);
response.setStatus(HttpServletResponse.SC_OK);
I couldn't find anything in the manual.
Any direction is very much appreciated.
Here is how the port and its implementation look like:
Here is how the port and its implementation head look like:
#WebService(name = "ForecastServicePortType", targetNamespace = "http://www.company.com/forecastservice/wsdl")
#SOAPBinding(parameterStyle = SOAPBinding.ParameterStyle.BARE)
#XmlSeeAlso({
ObjectFactory.class
})
public interface ForecastServicePortType {
/**
*
* #param parameters
* #throws RemoteException
*/
#WebMethod(action = "http://www.company.com/forecast/sendForecast")
public void sendForecast(
#WebParam(name = "SendForecast", targetNamespace = "http://www.company.com/forecastservice", partName = "parameters")
SendForecastType parameters) throws RemoteException;
}
#WebService(name = "ForecastServicePortTypeImpl", serviceName = "ForecastServicePortType", endpointInterface = "com.company.forecastservice.wsdl.ForecastServicePortType", wsdlLocation = "/WEB-INF/wsdl/ForecastServicePortType.wsdl")
#HandlerChain(file = "/META-INF/handlers.xml")
public class ForecastServicePortTypeImpl implements ForecastServicePortType {
...
}
In case anybody will find this useful. Here is the solution;
Apache CXF by default uses async requests and even if the annotation #OneWay is missing it still behaves as it if was there.
So in order to disable that an interceptor needs to be created like below:
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.cxf.binding.soap.SoapMessage;
import org.apache.cxf.binding.soap.interceptor.AbstractSoapInterceptor;
import org.apache.cxf.interceptor.Fault;
import org.apache.cxf.phase.Phase;
import java.util.Arrays;
public class DisableOneWayInterceptor extends AbstractSoapInterceptor {
private static final Log LOG = LogFactory.getLog(DisableOneWayInterceptor.class);
public DisableOneWayInterceptor(){
super(Phase.PRE_LOGICAL);
addBefore(Arrays.asList(org.apache.cxf.interceptor.OneWayProcessorInterceptor.class.getName()));
}
#Override
public void handleMessage(SoapMessage soapMessage) throws Fault {
if(LOG.isDebugEnabled())
LOG.debug("OneWay behavior disabled");
soapMessage.getExchange().setOneWay(false);
}
}
And called in WebService class (annotated with #WebService) as below:
#org.apache.cxf.interceptor.InInterceptors (interceptors = {"com.mycompany.interceptors.DisableOneWayInterceptor" })

cannot connect to database using jdbc [closed]

This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 9 years ago.
Hi im pretty new to jsp and servlets so im having a problem understanding why i couldnt do a simple query. the code is for a login page where the user servlet queries the data sent by the user then matches it to the database then when it finds a match it returns login==true and there another query is executed to get the employee id number matched along with the username and password
import java.sql.Connection;
import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.sql.*;
import javax.servlet.*;
import java.io.*;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.Driver;
import java.sql.DriverManager;
import javax.servlet.http.HttpSession;
public class Login extends HttpServlet {
protected void processRequest(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException{
response.setContentType("text/html;charset=UTF-8");
PrintWriter out = response.getWriter();
try {
HttpSession session = request.getSession();
Connection conn = null;
PreparedStatement prepState = null;
Statement state = null;
ResultSet result = null;
boolean login = false;
String username = request.getParameter("username");
String password = request.getParameter("password");
String checkLogin = ("SELECT * FROM users WHERE username = ? AND password = ?");
try{
Class.forName("com.mysql.jdbc.Driver");
conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/matrix","","");
prepState = conn.prepareStatement(checkLogin);
prepState.setString(1, username);
prepState.setString(2, password);
result = prepState.executeQuery();
login = result.next();
if(login){
out.println("login");
String id = result.getString(1);
String getUser = ("SELECT empLastName, empFirstName FROM employees WHERE empID = ? ");
prepState = conn.prepareStatement(getUser);
prepState.setString(1, id);
result = prepState.executeQuery();
String first = result.getString(2);
String last = result.getString(1);
out.println("<html>");
out.println(first + " " + last);
out.println("</html>");
}
}catch(Exception e){
out.println("ERROR");
e.printStackTrace(System.err);
}
}catch(Exception db){
} finally {
out.close();
}
}
Here's the error i get
INFO: The start() method was called on component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/Matrix]] after start() had already been called. The second call will be ignored.
java.sql.SQLException: Access denied for user 'root'#'localhost' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3566)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3498)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:919)
at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4004)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1284)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2312)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2122)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:774)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:49)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:375)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:289)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at Login.processRequest(Login.java:57)
at Login.doPost(Login.java:119)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:641)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.netbeans.modules.web.monitor.server.MonitorFilter.doFilter(MonitorFilter.java:393)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:498)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:562)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:394)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:243)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:188)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
This class is wrong in so many ways:
Empty catch block. You'll never know if something goes wrong.
Open a connection in the servlet rather than using connection pool.
Hard wired driver class and connection URL.
No credentials used to connect to the database.
One class does everything. Better to have one responsibility per class.
Mingling UI (HTML generation) and database access.
You don't close result set, statement, or connection.
try nested in another try; I'd recommend just one.
I'm getting tired now.
It's not clear to me what the problem is. Did you get an error? If yes, could you post the stack trace? If not, what behavior did you observe?
Try to check your MySQL username and password. MySQL has default user name "root"
Try to change your code into this
Class.forName("com.mysql.jdbc.Driver");
conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/matrix","root","");
It means you connect to MySQL server which in localhost (the same computer), with username "root" and password empty string "" (which is the default)

Resources