I'm using sequelize, and using transactions, but I have to make a lot of inserts every night, my fear is if these inserts/changes are stored in memory until transaction is commited and can crash the server and lost it all.
Or if these changes are stored and handled by the DBMS (in this case i'm using aurora/postgresql) and i don't have to worry about nothing
Help!
I'm usings express 4, sequelize 5 and this will run maybe on a cronJob
This is an abstract example of my structure
const db = require('../database/models')
class Controller {
async test (req, res) {
let transaction = await db.sequelize.transaction()
try {
await this.storeData(req.body, transaction)
await transaction.commit()
res.status(200)
} catch (error) {
if (transaction) await transaction.rollback()
res.status(400)
}
}
async storeDate (params, transaction = null) {
// Calculation of the data to insert
var records = []
await Promise.all(records.map(async item => {
await db.MyModel.create(item, { transaction })
}
))
}
A transaction feature in Sequelize is just a wrapper over a DB transaction and, of course, a transactional DBMS has a transaction log and stores all ongoing transaction operations there.
One edge case would be if you really like to take too many objects and insert them all in one operation so I'd recommend to divide a huge amount of rows into smaller batches.
Related
As per docs SqlCommandTimeout is
This property is the cumulative time-out (for all network packets that
are read during the invocation of a method) for all network reads
during command execution or processing of the results. A time-out can
still occur after the first row is returned, and does not include user
processing time, only network read time.
For example, with a 30 second time out, if Read requires two network
packets, then it has 30 seconds to read both network packets. If you
call Read again, it will have another 30 seconds to read any data that
it requires.
I have code below that executes the stored procedure and then reads the data using SqlReader row by row.
public static async Task<IEnumerable<AvailableWorkDTO>> prcGetAvailableWork(this MyDBContext dbContext, int userID)
{
var timeout = 120
var result = new List<AvailableWorkDTO>();
using (var cmd = dbContext.Database.GetDbConnection().CreateCommand())
{
var p1 = new SqlParameter("#UserID", SqlDbType.Int)
{
Value = userID
};
cmd.CommandText = "dbo.prcGetAvailableWork";
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add(p1);
cmd.CommandTimeout = timeout;
await dbContext.Database.OpenConnectionAsync().ConfigureAwait(false);
using (var reader = await cmd.ExecuteReaderAsync().ConfigureAwait(false))
{
while (await reader.ReadAsync().ConfigureAwait(false))
{
var item = new AvailableWorkDTO();
item.ID = reader.GetInt32(0);
item.Name = reader.GetString(1);
item.Title = reader.GetString(2);
item.Count = reader.GetInt32(3);
result.Add(item);
}
}
}
return result;
}
In Sql Profiler I see only one call to stored procedure as expected. So I am guessing the stored proc executes and returns entire result set.
Questions
1>If SqlReader is reading one record at a time, where is the entire resultset is stored while reader is reading? Is it temporarily stored in SQL Server memory or Application Server memory?
2>Using EF Core is there any way to read the entire result set at once?
The resultset isn't stored anywhere, it's streamed directly to the client. As the server reads rows from disk or memory, they are fed through the query plan and out across the network. This is why you always need to make sure read as fast as possible and to dispose the reader and connection: because the query is running the whole time.
To "read the entire result set at once", you just do what you are doing now: loop the reader and add it to a List. Alternatively, you could use DataTable.Load, however I do not advise this, and it is also not async.
The reader is just an object that is capable of returning individual rows from a command. What you see in the profiler is a single execution of a command. If you also monitor SQL:Batch Completed event, you will see that that only happens when the reader is finished.
You can use a stored procedure with EF instead of ADO.Net, but I am not sure that it will be faster.
Create a special class to get data from the stored procedure, or use existing AvailableWorkDTO. This class should have all properties that select clause of your stored procedure has. You don't need to select everything in your stored procedure. Just select the properties that AvailableWorkDTO has and add NotMapped attribute
[NotMapped]
public class AvailableWorkDTO
{
.....
}
after this add this class to dbContext DbSet
public virtual DbSet<AvailableWorkDTO> AvailableWorkDTOs { get; set; }
And this is a sample function to show how to get data using the stored procedure
public async Task<IEnumerable<AvailableWorkDTO>> prcGetAvailableWork(MyDBContext dbContext, int userID)
{
var pId = new SqlParameter("#UserID", userID);
return await dbContext.Set<AvailableWorkDTO>()
.FromSqlRaw("Execute db.prcGetAvailableWork #UserID", pId)
.ToArrayAsync();
}
I am working on a Web API and Entity Framework 6 that is doing a "bulk" insert of under 500 records at any given time to a Microsoft SQL Server table. The DbContext.SaveChanges() method will insert all the records into a table in a couple seconds, so have no issues with that. However, when the method is called to insert the same number of records into the same table with a semi-extensive trigger attached to it, the process can take many minutes. The trigger has some calls to table joins and inserts into other tables and then deletes the newly inserted record.
I do not have much control of the table or the trigger, so I am looking for suggestions on how to improve performance. I made a suggestion to move the trigger to a stored procedure and have the trigger call the stored procedure, but I am uncertain if that will achieve any gains.
EDIT: As I understand my question was kind of generic, I will post some of my code in case it helps. The SQL is not mine, so I will see what I can actually post.
Here is the part of my Web API method that does the call to SaveChanges():
string[] stringArray = results[0].Split(new[] { "\r\n", "\r", "\n" }, StringSplitOptions.None);
var profileObjs = db.Set<T_ProfileStaging>();
foreach (var s in stringArray)
{
string[] columns = s.Split(new[] {",", "\t"}, StringSplitOptions.None);
if (columns.Length == 6)
{
T_ProfileStaging profileObj = new T_ProfileStaging();
profileObj.CompanyCode = columns[0];
profileObj.SubmittedBy = columns[1];
profileObj.VersionName = columns[2];
profileObj.DMName = columns[3];
profileObj.Zone = columns[4];
profileObj.DMCode = columns[5];
profileObj.ProfileName = columns[6];
profileObj.Advertiser = columns[7];
profileObj.OriginalInsertDate = columns[8];
profileObjs.Add(profileObj);
}
}
try
{
db.SaveChanges();
return Ok();
}
catch (Exception e)
{
return Content(HttpStatusCode.BadRequest, "SQL Server Insert Exception");
}
When you load with SaveChanges() EF will send each row in a separate INSERT statement. So if you have a statement trigger, it will run for each row.
To work around this you need either
use a bulk load API from the client (instead of EF's SaveChanges()) using SqlBulkCopy directly, or one of the many EF extensions that wrap it.
or
Configure EF to insert into a different table and then INSERT ... SELECT into the target table
How do I select certain set of data(rows) from sql database by the time when they are inserted? I don't see any related documents in regards to how to do this using mssql module in node.js... could anyone suggest me any reading material or something else? So my question is how to create timestamp column when data are inserted in database
Thank you
The doc seem straight forward on how to achieve it
const sql = require('mssql')
async () => {
try {
const pool = await
sql.connect('mssql://username:password#localhost/database')
const result = await sql.query`SELECT * FROM TABLE WHERE DATE BETWEEN '09/16/2010 05:00:00' and '09/21/2010 09:00:00'`
console.dir(result)
} catch (err) {
// ... error checks
}
}
Recently I started working with isolation levels, but specifying isolation level in #Transactional annotation does not seems to work in spring boot. I tried a lot of different things but I cannot get it to work, below is my code.
#Transactional(isolation=Isolation.READ_UNCOMMITTED)
public void updateWalletAddresses(List<RegisterWallet> walletMetaCollection) throws Exception{
Transaction tx =null;
Session session =null;
if(sessionFactory == null){
return;
}
session = sessionFactory.openSession();
try{
String sql = "select * from user";
SQLQuery query = session.createSQLQuery(sql);
query.addEntity(User.class);
List<User> userlist=query.list();
int i=0;
while(i<userlist.size()){
System.out.println(userlist.get(i).getEmail());
i++;
}
}catch(Exception e){
throw e;
}
}
before executing the above walletservice method I am starting a transaction
in mysql client but I am not committing it so that I have dirty data.
After that I executed above code but it does not print uncommitted data even though I specified transaction read uncommitted.
code for starting transaction in mysql is
set autocommit=0;
start transaction;
insert into user (name,email,password,roleid,username)
values("prashank","myemail#gmail.com","password",1,"prashank");
Note: as I am not committing the transaction then above insert cause dirty read problem. I am not able to read uncommitted data
Note: Similarly any other isolation level are not working .
Please help
Double check your DB engine with
SHOW TABLE STATUS LIKE 'table_name';
If is MyISAM, then transactional is not supported,
use below to create InnoDB, which support transaction, then your case should work
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL55Dialect
Also I don't think
set autocommit=0;
is required bcz start transaction will automatically disable it
Working with Express, node, the MSSQL package to create a backend for an application, and I would like to do as much processing on the server as possible before sending to the client.
I have two queries I need to run, but I need to combine the data in a specific way before sending to the client.
The first query gathers data that will be of a one-to-one relationship, and the other is a one-to-many relationship. I would like to append the one-to-Many onto the One-to-one.
First Query:
select updatedInfo.*,
nameInfo.*, nameInfo.updated as nameUpdated, nameInfo.alreadyCorrect as nameWasCorrect,
addressInfo.*, addressInfo.alreadyCorrect as addWasCorrect, addressInfo.updated as addUpdated,
phoneInfo.*, phoneInfo.alreadyCorrect as phoneWasCorrect, phoneInfo.updated as phoneUpdated,
emailInfo.*, emailInfo.alreadyCorrect as emailWasCorrect, emailInfo.updated as emailUpdated
from updatedInfo join nameInfo on updatedInfo.IndivId=nameInfo.nameInfoId
join addressInfo on updatedInfo.IndivId=addressInfo.addressInfoId
join emailInfo on updatedInfo.IndivId=emailInfo.emailInfoId
join phoneInfo on updatedInfo.IndivId=phoneInfo.phoneInfoId
where updatedInfo.correctedInFNV is not null
order by updatedInfo.IndivId
Second Query: ID is a variable passed to the query
select * from positionInfo where IndivId='${id}'
How would I go about appending the second query results to the first on the correct record?
I'm using the mssql package and using it like this:
var sqlConfig = {
server: 'IP',
database: 'db',
user: 'sweeper',
password: 'pass'
}
const connPool = new mssql.ConnectionPool(sqlConfig, err => {
console.error(err);
});
var query = {
getAllUpdatedPool: () => {
connPool.Request().query(`----first query ----`)
.then((set) => {
console.log(set);
return set;
}).catch((err) => {
console.error(err);
return err;
});
},
getPositionByIdPool: (id) => {
connPool.Request().query(`----second query-----`)
.then((set) => {
console.log(set);
return set;
}).catch((err) => {
console.error(err);
return err;
});
How should I call these to add the results of the second query to the results of the first one as an additional property? Callbacks are making this confusing.
It looks like both queries execute on the same server, have you considered using subqueries? (https://learn.microsoft.com/en-us/sql/relational-databases/performance/subqueries?view=sql-server-2017). If you can express what you're trying to do in SQL it'll probably be 1) cleaner and 2) faster to just do it with subqueries than manually merging recordsets. If they exist on different servers, you could use linked servers to achieve the same subquery result.