Get only dataValues from sequelize raw query instead of Model instance - database

I'm using sequelize and using raw query to get the datas from table. But I'm getting all of the model instances while I only need the dataValues.
My setup looks like this:
const sequelize = new Sequelize({
database: process.env.PGDATABASE,
username: process.env.PGUSER,
password: process.env.PGPASS,
host: process.env.PGHOST,
port: process.env.PGPORT,
dialect: "postgres"
});
getPostGres: () => {
return sequelize;
}
and the way I'm querying the database looks like this:
let messageRatingsArr = await getPostGres().query(
`SELECT mr.support_email, mr.support_name,
(select count(mrn."chatId") as total FROM message_ratings as mrn WHERE mrn."ratingType"='NEGATIVE' and mr.support_email = mrn.support_email) as negative,
(select count(mrp."chatId") as total FROM message_ratings as mrp WHERE mrp."ratingType"='POSITIVE' and mr.support_email = mrp.support_email) as positive,
(select count(mrm."chatId") as total FROM message_ratings as mrm WHERE mrm."ratingType"='MIXED' and mr.support_email = mrm.support_email) as mixed,
(select count(mru."chatId") as total FROM message_ratings as mru WHERE mru."ratingType"='NEUTRAL' and mr.support_email = mru.support_email) as neutral
FROM message_ratings mr
WHERE mr."createdAt" >= '${properFromDate}' AND mr."createdAt" <= '${properToDate}'
group by mr.support_email, mr.support_name
limit ${args.count} offset ${args.offset} `,
{
model: MessageRatingPG,
mapToModel: true
}
);
let messageRatings = messageRatingsArr.map(item=>{
return item.dataValues;
})
let result = connectionFromArray(messageRatings, args);
result.totalCount = messageRatings.length;
return result;
As you can see, since I'm mapping the data from the query which has all kinds of stuff like dataValues, _options, isNewRecord etc., looping through the array if I have a large data set isn't efficient, so what can I do to only get the dataValues?

From https://sequelize.org/master/manual/raw-queries.html:
In cases where you don't need to access the metadata you can pass in a query type to tell sequelize how to format the results. For example, for a simple select query you could do:
sequelize.query("SELECT * FROM `users`", { type: sequelize.QueryTypes.SELECT})
.then(users => {
// We don't need spread here, since only the results will be returned for select queries
})
Now, looking at your code, and comparing with the next paragraph in the docs:
A second option is the model. If you pass a model the returned data will be instances of that model.
// Callee is the model definition. This allows you to easily map a query to a predefined model
sequelize
.query('SELECT * FROM projects', {
model: Projects,
mapToModel: true // pass true here if you have any mapped fields
})
.then(projects => {
// Each record will now be an instance of Project
})
I'd suggest removing from your original code, the following:
{
model: MessageRatingPG,
mapToModel: true
}
and replacing it with { type: sequelize.QueryTypes.SELECT }

You have to add to your query the attribute raw. From the docs
Sometimes you might be expecting a massive dataset that you just want
to display, without manipulation. For each row you select, Sequelize
creates an instance with functions for update, delete, get
associations etc. If you have thousands of rows, this might take some
time. If you only need the raw data and don't want to update anything,
you can do like this to get the raw data.
Project.findAll({ where: { ... }, raw: true })

Related

Does mongoose have a findAndUpdate function?

I am aware that mongoose has findOneAndUpdate that can only filter one value.
How can I filter two values?
await ReceiptDetails.findAndUpdate(
{patientIDnumber:patientIDnumber,
appNum:appNum
},
{dateIssued:date,
addedItem: addedItemValue,
paymentType:paymentType,
totalAmount:totalAmount,
officialReceiptNum:officialReceiptNum,
addedProcedurePrice:addedProcedurePrice,
amountPaid:amountPaid,
}
)
These are the only option that I have.
use updateMany() for updating multiple records, api doc says response of this function has fields such as matchedCount, modifiedCount, upsertedId etc, see below:
const res = await Person.updateMany({ name: /Stark$/ }, { isDeleted: true });
res.matchedCount; // Number of documents matched
res.modifiedCount; // Number of documents modified
res.acknowledged; // Boolean indicating everything went smoothly.
res.upsertedId; // null or an id containing a document that had to be upserted.
res.upsertedCount; // Number indicating how many documents had to be upserted. Will either be 0 or 1.

Dapper One to Many Mapping Logic

The dapper tutorial gives this example to help a user with Multi Mapping (One to Many)
While this works I am curious why they have you store the orders in the dictionary but then in the end they use a linq.Distinct() and return from the list. It seems like it would be cleaner to just return the ordersDictionary.Values as the dictionary logic ensures no duplicates.
//Tutorial
using (var connection = new SqlConnection(FiddleHelper.GetConnectionStringSqlServerW3Schools()))
{
Dictionary<int,Order> orderDictionary = new Dictionary<int, Order>();
List<Order> list = connection.Query<Order, OrderDetail, Order>(sql, (order, orderDetail) =>
{
if (!orderDictionary.TryGetValue(order.OrderID, out Order orderEntry))
{
orderEntry = order;
orderEntry.OrderDetails = new List<OrderDetail>();
orderDictionary.Add(orderEntry.OrderID, orderEntry);
}
orderEntry.OrderDetails.Add(orderDetail);
return orderEntry;
}, splitOn: "OrderID")
.Distinct()
.ToList();
return list;
}
//my suggestion
using (var connection = new SqlConnection(FiddleHelper.GetConnectionStringSqlServerW3Schools()))
{
Dictionary<int,Order> orderDictionary = new Dictionary<int, Order>();
//change 1 no need to store into list here
connection.Query<Order, OrderDetail, Order>(sql, (order, orderDetail) =>
{
if (!orderDictionary.TryGetValue(order.OrderID, out Order orderEntry))
{
orderEntry = order;
orderEntry.OrderDetails = new List<OrderDetail>();
orderDictionary.Add(orderEntry.OrderID, orderEntry);
}
orderEntry.OrderDetails.Add(orderDetail);
return orderEntry;
}, splitOn: "OrderID"); //change 2 remove .Distinct().ToList()
return orderDictionary.Values.ToList(); //change 3 return dictionaryValues
}
I'm the author of this tutorial: https://dapper-tutorial.net/query#example-query-multi-mapping-one-to-many
why they have you store the orders in the dictionary
A row is returned for every OrderDetail. So you want to make sure to add the OrderDetail to the existing Order and not create a new one for every OrderDetail. The dictionary is used for performance to check if the Order has been already created or not.
it would be cleaner to just return the ordersDictionary.Values
How will your query return dictionary values?
Of course, if you are in a method such as yours, you can do
var list = orderDictionary.Values;
return list;
But how to make this Connection.Query return dictionary values? An order is returned for every row/OrderDetail, so the order will be returned multiple times.
Outside the Query, your dictionary solution works great and is even a better solution for performance, but if you want to make your Query return the distinct list of orders without using Distinct or some similar method, it's impossible.
EDIT: Answer comment
my suggestion return orderDictionary.Values.ToList(); //change 3 return dictionaryValues
Thank you for your great feedback, it's always appreciated ;)
It would be weird in a tutorial to use what the query returns when there is no relationship but use the dictionary for one to many relationships
// no relationship
var orders = conn.Query<Order>("", ...).Distinct();
// one to many relationship
conn.Query<Order, OrderDetail>("", ...);
var orders = orderDictionary.Values.ToList();
Your solution is better for performance the way you use it, there is no doubt about this. But this is how people usually use the Query method:
var orders = conn.Query("", ...).Distinct();
var activeOrders = orders.Where(x => x.IsActive).ToList();
var inactiveOrders = orders.Where(x => !x.IsActive).ToList();
They use what the Query method returns.
But again, there is nothing wrong with the way you do it, this is even better if you can do it.

Write data in to nested object in firebase firestore

I have a data structure that looks as follows:
This is the top level of the collection:
I want to write to increment the field count but I can't do it. I've tried so many different methods that I'd rather not go through all of them. The closest I've gotten was through:
const pageRef = admin.firestore().collection("pages").doc(image.page);
pageRef.set(
{
[`images.${image.position}.count`]: admin.firestore.FieldValue.increment(
1
),
},
{ merge: true }
);
But that leaves me with:
Please help. Changing the structure of pages is an option.
This is what I've tried to replicate:
Update fields in nested objects in firestore documents?
The issue is on how the point notaition is being used.
In the Post you shared the example they use is:
var setAda = dbFirestore.collection('users').doc('alovelace').update({
"first.test": "12345"
});
Applying this to your Code and model would be:
const pageRef = admin.firestore().collection("pages").doc(image.page);
pageRef.set(
{
`images[${image.position}].count`: admin.firestore.FieldValue.increment(
1
),
},
{ merge: true }
);
This will affect the element in the Array Images, the element image.position its value count.

Compare two big arrays value for value in Node.js

I have two arrays, one containing 200.000 product objects coming from a CSV file and one containing 200.000 product objects coming from a database.
Both arrays contains objects with the same fields, with one exception: the database objects have a unique ID as well.
I need to compare all 200.000 CSV objects with the 200.000 database objects. If the CSV object already exists in the database objects array I put it in an "update" array together with the ID from the match, and if it doesn't, then I put it in a "new" array.
When done, I update all the "update" objects in the database, and insert all the "new" ones. This goes fast (few seconds).
The compare step however takes hours. I need to compare three values: the channel (string), date (date) and time (string). If all three are the same, it's a match. If one of those isn't, then it's not a match.
This is the code I have:
const newProducts = [];
const updateProducts = [];
csvProducts.forEach((csvProduct) => {
// check if there is a match
const match = dbProducts.find((dbProduct) => {
return dbProduct.channel === csvProduct.channel && moment(dbProduct.date).isSame(moment(csvProduct.date), 'day') && dbProduct.start_time === csvProduct.start_time;
});
if (match) {
// we found a match, add it to updateProducts array
updateProducts.push({
id: match.id,
...csvProduct
});
// remove the match from the dbProducts array to speed things up
_.pull(dbProducts, match);
} else {
// no match, it's a new product
newProducts.push(csvProduct);
}
});
I am using lodash and moment.js libraries.
The bottleneck is in the check if there is a match, any ideas on how to speed this up?
This is a job for the Map collection class. Arrays are a hassle because they must be searched linearly. Maps (and Sets) can be searched fast. You want to do your matching in RAM rather than hitting your db for every single object in your incoming file.
So, first read every record in your database and construct a Map where the keys are objects like this {start_time, date, channel} and the values are id. (I put the time first because I guess it's the attribute with the most different values. It's an attempt to make lookup faster.)
Something like this pseudocode.
const productsInDb = new Map()
for (const entry in database) {
const key = { // make your keys EXACTLY the same when you load your Map ..
start_time: entry.start_time,
date: moment(entry.date),
entry.channel}
productsInDb.add(key, entry.id)
}
This will take a whole mess of RAM, but so what? It's what RAM is for.
Then do your matching more or less the way you did it in your example, but using your Map.
const newProducts = [];
const updateProducts = [];
csvProducts.forEach((csvProduct) => {
// check if there is a match
const key = { // ...and when you look up entries in the Map.
start_time: entry.start_time,
date: moment(entry.date),
entry.channel}
const id = productsInDb.get(key)
if (id) {
// we found a match, add it to updateProducts array
updateProducts.push({
id: match.id,
...csvProduct
});
// don't bother to update your Map here
// unless you need to do something about dups in your csv file
} else {
// no match, it's a new product
newProducts.push(csvProduct)
}
});

Two queries mysql in one json object and retrieve each details in Codeigniter

I have two tables that I want to convert them to json like this:
$first_query = $this->db->query("SELECT * FROM `parameters` WHERE patient_id=7 ORDER BY created_on DESC LIMIT 10");
$second_query = $this->db->query("SELECT * FROM `pat_details` WHERE email='".$email."' AND phone_num='".$phone_num."'");
$first_query returns the 10 set of records which should be stored in an array and $second_query returns the objects.I need to merge Two queries mysql in one object json and retrieve the details of the result.
The output should be:
[
{
"firstname":"xyz",
"id":"123456",
"mail":"xyz#gmail.com",
"parameters":[
{
"diabetic":"no",
"hypertension":"yes",
},
{
"diabetic":"no",
"hypertension":"yes",
},
{
"diabetic":"yes",
"hypertension":"no",
}
]
}
]
I am not able to aggregate these two queries into one and encode the results in json
You can use a join query or a sub query. Since the data is not relational to each other, there's not a huge performance benefit to this.
If you just want to combine the data, return or cast the data as an arrays, use array_merge() on the two arrays, and then use json_encode().
And be sure to make you queries safe from SQL injection. I'd suggest using the CodeIgniter Query Builder to make database interactions simpler and more secure.
i have found the solution
$first_query = $this->db->query("SELECT * FROM `parameters` WHERE patient_id=7 ORDER BY created_on DESC LIMIT 10");
$json= $firstquery->result();
$second_query = $this->db->query("SELECT * FROM `pat_details` WHERE email='".$email."' AND phone_num='".$phone_num."'");
$json2 = array();
foreach($secondquery->result_array() as $row){
$json2[] = array(
'name' => $row['name'],
'address' => $row['address'],
'mail' => $row['mail']
);
}
$json['parameters'] = $json2;
echo json_encode($json);

Resources