So our current code loaded a csv file into a local jdbcTemplate in which then I do some querying. The issue was always performance, and we finally got access to a SQL server that could load the data. Naturally the company gets the guy with basically no database skills to set this up :P
#Autowired
DataSource dataSource;
#RequestMapping("/queryService")
public void queryService(#RequestParam("id")String id)
{
log.info("Creating tables");
jdbcTemplate.execute("DROP TABLE accounts IF EXISTS");
jdbcTemplate.execute("CREATE TABLE accounts(id VARCHAR(255), name VARCHAR(255), Organization__c VARCHAR(255)";
insertBatch(accounts,dataSource);
ArrayList<Account2> filteredaccs = filterAccount(jdbcTemplate);
.
public void insertBatch(ArrayList<Account2> accs, DataSource dataSource) {
List<Map<String, Object>> batchValues = new ArrayList<>(accs.size());
for (Account2 a : accs) {
Map<String, Object> map = new HashMap<>();
map.put("id", a.getId());
map.put("name", a.getName());
map.put("Organization__c", a.getOrganization__c());
batchValues.add(map);
}
SimpleJdbcInsert simpleJdbcInsert = new SimpleJdbcInsert(dataSource).withTableName("accounts");
int[] ints = simpleJdbcInsert.executeBatch(batchValues.toArray(new Map[accs.size()]));
}
.
public ArrayList<Account2> filterAccount(JdbcTemplate jdbcTemplate)
{
String sql= "query string";
ArrayList<Account2> searchresults = (ArrayList<Account2>) jdbcTemplate.query(sql,
new RowMapperResultSetExtractor<Account2>(new AccountRowMapper(), 130000));
return searchresults;
}
.
public class AccountRowMapper implements RowMapper<Account2> {
public Account2 mapRow(ResultSet rs, int rowNum) throws SQLException {
Account2 a = new Account2();
a.setId(rs.getString("id"));
a.setName(rs.getString("name"));
a.setOrganization__c(rs.getString("Organization__c"));
return a;
}
}
The question here is what is the quickest way for me to 'switch' over to using a SQL server to pull the data down, with the same table and rows, without changing too much of my current code?
Related
When i do JpaRepository.save(Entity e) the primary key generated with the help of hibernate sequence is saved as any random value generally starting from -43 or -42.
I am spring a spring boot project with JPA.
Below is my property file:
hibernate.dialect=org.hibernate.dialect.SQLServer2012Dialect
hibernate.hbm2ddl.auto=validate
hibernate.ejb.naming_strategy=org.hibernate.cfg.ImprovedNamingStrategy
hibernate.show_sql=false
hibernate.format_sql=true
This is my entity on which i am calling save. Sequence name - CPU_Responses_Seq is already present in DB
#Entity
#Table(name="CPU_Responses")
public class CPUResponses extends BaseEnity{
/**
*
*/
private static final long serialVersionUID = 1L;
#Id
#GeneratedValue(generator="CPUResponseSeq",strategy=GenerationType.SEQUENCE)
#SequenceGenerator(name="CPUResponseSeq",sequenceName="CPU_Responses_Seq")
#Column(name = "Response_ID", nullable=false,updatable=false)
private long responseId;
This is my persistance config class
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(basePackages= {"package path"})
#PropertySource("classpath:application.properties")
public class PersistanceConfiguration {
#Autowired
private Environment env;
public Environment getEnv() {
return env;
}
public void setEnv(Environment env) {
this.env = env;
}
#Bean
LocalContainerEntityManagerFactoryBean entityManagerFactory() throws NamingException {
LocalContainerEntityManagerFactoryBean entityManagerFactoryBean = new LocalContainerEntityManagerFactoryBean();
entityManagerFactoryBean.setDataSource(dataSource());
entityManagerFactoryBean.setJpaVendorAdapter(new HibernateJpaVendorAdapter());
entityManagerFactoryBean.setPackagesToScan("entity path");
Properties jpaProperties = new Properties();
//Configures the used database dialect. This allows Hibernate to create SQL
//that is optimized for the used database.
jpaProperties.put("hibernate.dialect", env.getRequiredProperty("hibernate.dialect"));
entityManagerFactoryBean.setJpaProperties(jpaProperties);
return entityManagerFactoryBean;
}
#Bean
public DataSource dataSource() throws NamingException {
JndiObjectFactoryBean bean = new JndiObjectFactoryBean();
bean.setJndiName("java:comp/env/jdbc/CPUDB");
bean.setProxyInterface(DataSource.class);
bean.setLookupOnStartup(false);
bean.afterPropertiesSet();
return (DataSource) bean.getObject();
}
#Bean
JpaTransactionManager transactionManager(EntityManagerFactory entityManagerFactory) {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(entityManagerFactory);
return transactionManager;
}
}
I don't know what is getting wrong. Data is getting saved in DB but with negative primary key. My sequencer in DB have min value as zero so sequencer is correct.
Kindly help
I think this is related to the changes introduced in Hibernate in their sequence generator, try to add
hibernate.id.new_generator_mappings=false
or
spring.jpa.properties.hibernate.id.new_generator_mappings=false
note that the "new generator" is not compatible with previous version so start from a clean database to avoid issues.
I'm executing queries periodically (by a scheduler) using my Spring Boot application
application.properties
src_mssqlserver_url=jdbc:sqlserver://192.168.0.1;databaseName=Test;
src_mssqlserver_username=tester
src_mssqlserver_password=tester1
src_mssqlserver_driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver
Datasource and JdbcTemplate Bean
#Primary
#Bean(name = "src_mssqlserver")
#ConfigurationProperties(prefix = "spring.ds_mssqlserver")
public DataSource srcDataSource() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(env.getProperty("src_mssqlserver_driverClassName"));
dataSource.setUrl(env.getProperty("src_mssqlserver_url"));
dataSource.setUsername(env.getProperty("src_mssqlserver_username"));
dataSource.setPassword(env.getProperty("src_mssqlserver_password"));
return dataSource;
}
#Bean(name = "srcJdbcTemplate")
public JdbcTemplate srcJdbcTemplate(#Qualifier("src_mssqlserver") DataSource dsSrcSqlServer) {
return new JdbcTemplate(dsSrcSqlServer);
}
Usage: This method is called from a scheduler with list of items to process (normally 1000 records), this process runs in an hour once.
#Autowired
#Qualifier("srcJdbcTemplate")
private JdbcTemplate srcJdbcTemplate;
public void batchInsertUsers(final List<User> users) {
String queryInsert = "INSERT INTO [User] ([Name]"
+ " , [Created_Date]"
+ " , [Notes])"
+ " VALUES (?, SYSDATETIMEOFFSET(), ?)";
srcJdbcTemplate.batchUpdate(queryInsert, new BatchPreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps, int i) throws SQLException {
User user = users.get(i);
ps.setString(1, user.getName());
ps.setString(2, user.getNotes());
}
#Override
public int getBatchSize() {
return sites.size();
}
});
I'm getting warnings from database administrator that my code keeping too much connections open. Please share some standard and workable way to handle such situation.
Thanks.
DriverManagerDataSource is NOT meant for production, it opens and closes a connection each time it needs one.
Use a connection pool like c3p0DataSource.
I have a multi-tenant application where one DB per tenant with one Master DB is configured. I load all the data sources in applications something like this :
#ConfigurationProperties(prefix = "spring.datasource")
#Bean
public DataSource dataSource() {
if(LOGGER.isInfoEnabled())
LOGGER.info("Loading datasources ...");
DataSource ds = null;
JndiDataSourceLookup dataSourceLookup = new JndiDataSourceLookup();
// load MASTER datasource
ds = dataSourceLookup.getDataSource(properties.getJndiName());
// load other TENANTs DB details
JdbcTemplate jdbcTemplate = new JdbcTemplate(ds);
List<GroupConfig> groupConfigs = jdbcTemplate.query(
"select * from master.tblTenant where IsActive=1 and ConfigCode in ('DB_URL','DATASOURCE_CLASS','USER_NAME','DB_PASSWORD') order by 2",
new ResultSetExtractor<List<GroupConfig>>() {
public List<GroupConfig> extractData(ResultSet rs) throws SQLException, DataAccessException {
List<GroupConfig> list = new ArrayList<GroupConfig>();
while (rs.next()) {
GroupConfig groupConfig = new GroupConfig();
groupConfig.setGroupConfigId(rs.getLong(1));
groupConfig.setGroupCode(rs.getString(2));
groupConfig.setConfigCode(rs.getString(3));
groupConfig.setConfigValue(rs.getString(4));
groupConfig.setIsActive(rs.getBoolean(5));
list.add(groupConfig);
}
return list;
}
});
int propCount = 1;
Map<String, Map<String, String>> groups = new HashMap<String, Map<String, String>>();
Map<String, String> temp = new HashMap<String, String>();
for (GroupConfig config : groupConfigs) {
temp.put(config.getConfigCode(), config.getConfigValue());
if (propCount % 4 == 0) {
groups.put(config.getGroupCode(), temp);
temp = new HashMap<String, String>();
}
propCount++;
}
// Create TENANT dataSource
Map<Object, Object> resolvedDataSources = new HashMap<Object, Object>();
for (String tenantId : groups.keySet()) {
Map<String, String> groupKV = groups.get(tenantId);
DataSourceBuilder dataSourceBuilder = new DataSourceBuilder(this.getClass().getClassLoader());
dataSourceBuilder.driverClassName(groupKV.get("DATASOURCE_CLASS")).url(groupKV.get("DB_URL"))
.username(groupKV.get("USER_NAME")).password(groupKV.get("DB_PASSWORD"));
//System.out.println(dataSourceBuilder.findType()); //class org.apache.tomcat.jdbc.pool.DataSource
if (properties.getType() != null) {
dataSourceBuilder.type(properties.getType());
}
if(LOGGER.isInfoEnabled())
LOGGER.info("Building datasource : "+tenantId);
resolvedDataSources.put(tenantId, dataSourceBuilder.build());
}
resolvedDataSources.put("MASTER", ds);
MultitenantDataSource dataSource = new MultitenantDataSource();
dataSource.setTargetDataSources(resolvedDataSources);
dataSource.setDataSourceLookup(dataSourceLookup);
dataSource.afterPropertiesSet();
if(LOGGER.isInfoEnabled())
LOGGER.info("Datasources initialization finished !");
return dataSource;
}
In controller I set respective data source as (similarly for TENANT datasources) :
TenantContext.setCurrentTenant("MASTER");
Issue : On server startup every thing works fine (both MASTER DB and TENANT specific queries), but once the server is idle for some time (few Hours) tenant specific calls starts failing (while MASTER DB connections still works fine) with error :
Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
Please help me to get rid off this exception. Thanks in advance.
I got the problem and solutions as well:
Why tenant connections was getting closed? Because Auto configurations(#ConfigurationProperties(prefix = "spring.datasource")) of spring-boot was not getting applied on the tenant DataSources which I was creating in code.
Resolution- I added new method to set the tomcat connection pool properties:
private DataSource buildDataSource(String driverClass, String url, String user, String pass){
PoolProperties p = new PoolProperties();
p.setUrl(url);
p.setDriverClassName(driverClass);
p.setUsername(user);
p.setPassword(pass);
p.setJmxEnabled(true);
p.setTestWhileIdle(false);
p.setTestOnBorrow(true);
p.setValidationQuery("SELECT 1");
p.setTestOnReturn(false);
p.setValidationInterval(30000);
p.setTimeBetweenEvictionRunsMillis(30000);
p.setMaxActive(100);
p.setInitialSize(10);
p.setMaxWait(10000);
p.setRemoveAbandonedTimeout(60);
p.setMinEvictableIdleTimeMillis(30000);
p.setMinIdle(10);
p.setLogAbandoned(true);
p.setRemoveAbandoned(true);
DataSource datasource = new DataSource();
datasource.setPoolProperties(p);
return datasource;
}
This solved my problem. But I'm curious to know if there is a way to apply AutoConfigurations while creating the objects in spring-boot.
I don't know why I received the error :
org.apache.cayenne.CayenneRuntimeException: [v.4.0.M5 Feb 24 2017 07:47:55] Commit Exception
[...]
Caused by: java.sql.SQLException: Procédure stockée 'auto_pk_for_table' introuvable.
[...]
I'm using Cayenne :
<dependency>
<groupId>org.apache.cayenne</groupId>
<artifactId>cayenne-server</artifactId>
<version>4.0.M5</version>
</dependency>
and JDTS for sql server :
<dependency>
<groupId>net.sourceforge.jtds</groupId>
<artifactId>jtds</artifactId>
<version>1.3.1</version>
</dependency>
The connexion is ok :
avr. 10, 2017 2:36:30 PM org.apache.cayenne.datasource.DriverDataSource getConnection
INFOS: +++ Connecting: SUCCESS.
I'm trying to create a new user (I'm starting by bascis!) so my code is :
(I cut a little bit, it's too long:!)
public abstract class _UserInfo extends CayenneDataObject {
public static final String ADDRESS_PROPERTY = "address";
public void setAddress(String address) {
writeProperty(ADDRESS_PROPERTY, address);
}
public String getAddress() {
return (String)readProperty(ADDRESS_PROPERTY);
}
}
public class UserInfo extends _UserInfo implements Serializable {
private static final long serialVersionUID = 1L;
public String address;
public String getAdress() {
return address;
}
public void setAddress(String address) {
super.setAddress(address);
}
//I have the hashcode and equals too
}
Then, I used vaadin to create my form :
public class UserAddView extends CustomComponent implements View {
private static final long serialVersionUID = 1L;
private TextField address;
private Button save;
public static final String USERVIEW = "user";
public boolean checkValidation() {
if (!checkTextFieldValid(address))
return false;
return true;
}
public boolean checkTextFieldValid(TextField element) {
if (element == null || element.isEmpty()) {
Notification.show(
"You should register a " + element.getDescription(),
Type.WARNING_MESSAGE);
return false;
}
return true;
}
public UserAddView() {
VerticalLayout mainLayout = new VerticalLayout();
mainLayout.setSizeFull();
setCompositionRoot(mainLayout);
final VerticalLayout vlayout = new VerticalLayout();
address = new TextField("Address:");
address.setDescription("Address");
vlayout.addComponent(address);
save = new Button("Save");
vlayout.addComponent(save);
mainLayout.addComponent(new HeaderMenu());
mainLayout.addComponent(vlayout);
addListeners();
}
private void addListeners() {
save.addClickListener(new ClickListener() {
private static final long serialVersionUID = 1L;
#Override
public void buttonClick(ClickEvent event) {
if (checkValidation() == true) {
ServerRuntime cayenneRuntime = ServerRuntime.builder()
.addConfig("cayenne-myapplication.xml").build();
ObjectContext context = cayenneRuntime.newContext();
UserInfo user = context.newObject(UserInfo.class);
user.setAddress(address.getValue());
user.getObjectContext().commitChanges();
Notification.show(
"Has been saved, We will send you your password by email. Your user login is: "
+ email.getValue(), Type.TRAY_NOTIFICATION);
getUI().getNavigator().navigateTo(HomepageView.MAINVIEW);
}
}
});
}
#Override
public void enter(ViewChangeEvent event) {
// TODO Auto-generated method stub
}
}
EDIT, add information : In my user object, I have a userid (primary key), in cayenne I wrote it as primary key too and in smallint. This error seems to be link... https://cayenne.apache.org/docs/3.1/api/org/apache/cayenne/dba/sybase/SybasePkGenerator.html
The error happens when you insert a new object. For each new object Cayenne needs to generate a value of the primary key. There are various strategies to do this. The default strategy depends on the DB that you are using. For SQLServer (and for Sybase, as you've discovered :)) that strategy is to use a special stored procedure.
To create this stored procedure (and other supporting DB objects), go to CayenneModeler, open your project, and select "Tools > Generate Database Schema". In "SQL Options" tab, uncheck all checkboxes except for "Create Primary Key Support". The SQL you will see in the window below the checkboxes is what you need to run on SQL server. Either do it from Cayenne modeler or copy/paste to your favorite DB management tool.
There's also an alternative that does not require a stored procedure - using DB auto-increment feature. For this you will need to go to each DbEntity in the Modeler and under the "Entity" tab select "Database-Generated" in the "Pk Generation Strategy" dropdown. This of course implies that your PK column is indeed an auto-increment in the DB (meaning you may need to adjust your DB schema accordingly).
We are developing a WPF application which can manage multiple clients. For each client we need a separate database (within the same SQL Server instance). So we need a „create database“ more than once. All individual client-databases will have all the same database structure.
We use Entity Framework Version 6.1.3, code-first and a local installation of SQL Server 2014 Express. In our „real“ business-application a new database with about 60 tables is created properly, but the performance is not so good.
Trying to isolate the problem I wrote a small sample program which creates 5 databases in a loop, 1 table per database. On my Dev-PC (Windows 10, i7-6700 3.40GHz, 16 GB RAM) the executiontime for those 5 databases is about 1 minute (11-12 secs for each database). When I debug the application, I see that there is a long wait on the line
dbContext.Database.CreateIfNotExists();
In SQL Server Profiler I see that it takes about 10 seconds per loop until the first entry arrives.
Any idea's where the time is lost? Or other suggestions for creating many databases (> 20) with the same DbContext in Entity Framework?
Here the source-code of my sample application:.
using System.ComponentModel.DataAnnotations.Schema;
using System.ComponentModel.DataAnnotations;
using System;
namespace TestEnityFramework01.Model
{
[Table("Table01")]
public class Table01 {
[Key]
[Column("ID", Order = 0, TypeName = "int")]
public int ID { get; set; }
[Column("Number01", TypeName = "int")]
public int Number01 { get; set; }
[Column("Date01")]
public DateTime? Date01 { get; set; }
[StringLength(50)]
[Column("Text01", TypeName = "nvarchar")]
public string Text01 { get; set; }
[Column("Amount01")]
public decimal Amount01 { get; set; }
[Column("Doule01")]
public double Doule01 { get; set; }
[Column("Bool01")]
public bool Bool01 { get; set; }
}
}
And the context:
using TestEnityFramework01.Model;
using System.Data.Entity;
namespace TestEnityFramework01.Context
{
class ContextA : DbContext
{
public DbSet<Table01> Tabel01 { get; set; }
public ContextA()
: base("ContextA")
{
// Don't create database automatically
Database.SetInitializer<ContextA>(null);
}
public ContextA(string pConnectionString)
: base(pConnectionString)
{
// Don't create database automatically
Database.SetInitializer<ContextA>(null);
this.Configuration.AutoDetectChangesEnabled = false;
this.Configuration.LazyLoadingEnabled = false;
this.Configuration.ValidateOnSaveEnabled = false;
}
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
base.OnModelCreating(modelBuilder);
}
}
}
The test program produces in a loop five databases:
private void button_Click(object sender, RoutedEventArgs e)
{
int _randomNumber = 0;
for (int i = 1; i <= 5; i++)
{
// create random number between 1 and 100'000
_randomNumber = new Random().Next(1, 100000);
// concat the random number to the database name
string _databaseName = "DatabaseCustomer" + _randomNumber.ToString();
// the connection-string look like this:
// "Data Source=MyPC\SQLEXPRESS;Initial Catalog=MyDatabaseName;User ID=MyUser;Password=MyPassword;Integrated Security=False;MultipleActiveResultSets=True;"
string _connectionString = GetConnectionString(_databaseName);
using (var dbContext = new ContextA(_connectionString))
{
// create the database
dbContext.Database.CreateIfNotExists();
}
}
}
**** UPDATE 16.06.2016****
When I switch the DB connection from SQL-Server 2014 Express to „SQL Server Compact Edition“, the creationtime for all 5 databases is about 1 or 2 seconds. Could it be that i use a wrong connectionstring to connect with the sql-server or my SQL-Server 2014 Express is not configured properly?
This is because the code first Approach creates a new database and establishes connection with it.
I Think you should use sql query for Creating Database.
If You Want I can help you how will you do it with better UI and Functionality design
Please reply for my answer.