Export Excel file from database - sql-server

I'm searching way how to generate XLS (XLSX) file from SQL Server in an ASP.NET MVC application.
Now I use EPPlus where I executing SQL query and result is saved to xlsx by this library.
In this case I have performance issue. If I have many data so, generation time is longer (150 rows in average about 10 sec and it's long time).
First idea is execute query that return XML and then transform to xls by xslt template but here I have problem with open through Excel (format and extension doesn't match).
Second idea is execute query that return XSL (or XSLX) from DB but I don't know how do it because file is saved on server and I don't know how to send to client to download.
Third idea Have you someone experience with similar problems and can you help me?
Thanks for any ideas.
//EDIT
Here is short example of my code:
int row = 0;
foreach(var obj1 in objList1)
{
WriteObj1(ref row, obj1);
var objList2 = GetObj2(obj1.Id);
foreach(var obj2 in objList2)
{
WriteObj2(ref row, obj2);
var objList3 = GetObj3(obj2.Id);
foreach(var obj3 in objList3)
{
WriteObj3(ref row, obj3);
var objList4 = GetObj4(obj3.Id);
foreach(var obj4 in objList4)
{
WriteObj4(ref row, obj4);
var objList5 = GetObj5(obj4.Id);
foreach(var obj5 in objList5)
{
WriteObj5(ref row, obj5);
}
}
}
}
}
and inner method for write is this code:
// create header in Excel
...
workSheet.Cells[row, column++].Value = someValue;

You can do this with a simple INSERT INTO.
Something like INSERT INTO OPENROWSET
('Microsoft.Jet.OLEDB.4.0',
'Excel 8.0;Database=[path];','SELECT * FROM [table]')
Where [path] is the desired directory and the SELECT statement should be self explaining.
For more information on OPENROWSET you can check out the Microsoft documentation

Since your updated question differs from the original one, I'm writing a new answer:
The problem with your code are the nested loops. This way you have thousands of accesses to the file.
Instead try to generate e.g. an array with all the values and afterwards use Excel.Range to update/set the whole array at once.
Next time please use the search function first. There are dozens of (answered) questions about a slow excel insert.

Related

How to Reject rows while loading files with extra comma's in SSIS Packages

I want to reject rows while loading data from flat file to Sql server and handle the rows with extra comma's and redirect them to flat file
Example
ID,FirstName,LastName,City
1,Robert,Gurera,Phoenix
2,Tom,Bradley,Cleveland,5
3,Jack,Williams,Dallas
4,Tim,Doherthy,San franscisco,6,7
I want to reject row 2 and 4 and load them to a flat file
Any help appreciated
Thanks
This is pretty simple:
Create a Flat file source. Go to Advanced, and add a new column column 5 as shown below
Next, check in the columns. What you need to do is, use conditional split transformation to redirect rows having data in column 5 to another destination.
Make sure to add few more columns if you're sure about length coming from flat file.
If you really need to check I would check using a script component source:
Load into one column and split.
If length is correct then send the row to the data flow otherwise send it out a failure path (most likely to a flat file or something.
Let me know if you need more explanation.
I just looked at your data and you would be OK using split and just processing the left 4 columns (0-3).
Here's the code:
using System.IO;
static void Main(string[] args)
{
using(var reader = new System.IO.StreamReader(#"C:\test.csv"))
{
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
var values = line.Split(','); //Make sure single quote to represent char
if(values.length == 4)
{
Output0Buffer.AddRow();
Output0Buffer.Id=int.Parse(values[0]);
Output0Buffer.Fname = values[1];
Output0Buffer.LastName = values[2];
Output0Buffer.City = values[3];
}
else
{
BadRowBuffer.AddRow();
BadRowBuffer.BadRow = line.ToString();
}
}
}
}
You need to set up the outputs.

OrientDB - find "orphaned" binary records

I have some images stored in the default cluster in my OrientDB database. I stored them by implementing the code given by the documentation in the case of the use of multiple ORecordByte (for large content): http://orientdb.com/docs/2.1/Binary-Data.html
So, I have two types of object in my default cluster. Binary datas and ODocument whose field 'data' lists to the different record of binary datas.
Some of the ODocument records' RID are used in some other classes. But, the other records are orphanized and I would like to be able to retrieve them.
My idea was to use
select from cluster:default where #rid not in (select myField from MyClass)
But the problem is that I retrieve the other binary datas and I just want the record with the field 'data'.
In addition, I prefer to have a prettier request because I don't think the "not in" clause is really something that should be encouraged. Is there something like a JOIN which return records that are not joined to anything?
Can you help me please?
To resolve my problem, I did like that. However, I don't know if it is the right way (the more optimized one) to do it:
I used the following SQL request:
SELECT rid FROM (FIND REFERENCES (SELECT FROM CLUSTER:default)) WHERE referredBy = []
In Java, I execute it with the use of the couple OCommandSQL/OCommandRequest and I retrieve an OrientDynaElementIterable. I just iterate on this last one to retrieve an OrientVertex, contained in another OrientVertex, from where I retrieve the RID of the orpan.
Now, here is some code if it can help someone, assuming that you have an OrientGraphNoTx or an OrientGraph for the 'graph' variable :)
String cmd = "SELECT rid FROM (FIND REFERENCES (SELECT FROM CLUSTER:default)) WHERE referredBy = []";
List<String> orphanedRid = new ArrayList<String>();
OCommandRequest request = graph.command(new OCommandSQL(cmd));
OrientDynaElementIterable objects = request.execute();
Iterator<Object> iterator = objects.iterator();
while (iterator.hasNext()) {
OrientVertex obj = (OrientVertex) iterator.next();
OrientVertex orphan = obj.getProperty("rid");
orphanedRid.add(orphan.getIdentity().toString());
}

Reading HL7 data from file and inserting it into the table SQL Server

Could someone please help me on capturing the HL7 data into the SQL server using MIRTHCONNECT. I was searching for some examples and i was not able to find any tutorials which demonstrates looping through multiple segments. I was able to insert records into the database by going through the tutorials but still i`m stuck in doing the looping process.
Could someone please share me some links or give me some ideas so that i can go through those .
This is my initial thought to loop through each segment since i assume that Mirth connect reads a file line by line.
Thanks for the help
LOGIC -(I`m not sure whether this will be the right approach)
for each (seg in RAWFILE) {
if (seg.name().toString() == "MSH") {
insert into table values ();
}
if (seg.name().toString() == "PID") {
INSERT INTO TABLE2 VALUES ();
}
}
sample RAW DATA
MSH|^~&|LAB|CCF|||20040920080937||ORM^O01|42640000 009|P|2.3|
PID|||56797971||RESULTSREVIEW^TESTPATIENT^||196505 25|M||||||||||56797971|
PV1||O|UNKO^|||||
ORC|RE||A0203809||IP|||||||
OBR|1|A0203809|A0203809|400090^Complete Blood Count|||200609240000|||||||200609240847||deleted^^ ^^MD^^^^^^||||||200609241055|||P
OBX|1|ST|40010^White Blood Count (WBC) (x1000)||PENDING||||||P
OBX|2|ST|40020^Red Blood Count (RBC)||PENDING||||||P
ORC|RE||A0203809||CM|||||||
OBR|2|A0203809|A0203809|650300^Depakene (Valproic Acid) Level|||200609240000|||||||200609240847||^deleted^ ^^^MD^^^^^^||||||200609241055|||F
OBX|3|NM|65030^Depakene (Valproic Acid) Level||76.8|ug/ml|50-100||||F|||200609241054||
Sounds like you've got the db insertion working and you're having questions on how to handle repeating segments. Here is some code that I use in mirth for handling repeating segments. Of course your milage may vary but this should accomplish what you are wanting.
var segCount = 0;
// Loop through message and count number of OBX segments
for each (segment in msg.children()) {
if(segment.name() === 'OBX') {
segCount++;
}
}
// Make changes if there are OBX segments
if (segCount > 0) {
for (var i = 0; i < segCount; i++) {
tmp=msg;
// Add this segment to the database
insert into table values ();
// Here I am changing each OBX-5.1 to contain normal if OBX-3.1 is 'Some Text'
if (msg['OBX'][i]['OBX.3']['OBX.3.1'].toString() === 'Some text') {
tmp['OBX']['OBX.5']['OBX.5.1'] = 'Normal';
}
}
}
You want to pull the information from an HL7 file and insert into a DB regardless of type.
So, Create one channel with inbound message type as HL7, regardless of you taking HL7 message from file or an open TCP/IP connection.
Go to source, go to transformer, create a JS transformer, supply an HL7 message in the inbound template, and now extract information from the message and store it in variables. Something like below.
var firstname=msg[PID][PID.5][PID.5.2].toString();
A helpful tip is to drag and drop the elements from the inbound message template and store in the variable.
Now, move this variable in the channel Map, so that we can capture it in destination.
channelMap.put('first_name',firstname);
Now for the second part,
Go to destination of the same channel and create one DB writer that writes information to a DB.
Don't select Use Javascript, instead just write your INSERT query, something like below.
INSERT INTO PATIENT(first_name) VALUES (channelMap.get('first_name');
There is whole lot of documentation available at Mirth to help you with the DB writer.
Hope this helps!

Entity Framework insert (of one object) slow when table has large number of records

I have a large asp.net mvc application that runs on a database that is rapidly growing in size. When the database is empty, everything works quickly, but one of my tables now has 350K records in it and an insert is now taking 15s. Here is a snippet:
foreach (var packageSheet in packageSheets)
{
// Create OrderSheets
var orderSheet = new OrderSheet { Sheet = packageSheet.Sheet };
// Add Default Options
orderSheet.AddDefaultOptions();
orderSheet.OrderPrints.Add(
new OrderPrint
{
OrderPose = CurrentOrderSubject.OrderPoses.Where(op => op.Id == orderPoseId).Single(),
PrintId = packageSheet.Sheet.Prints.First().Id
});
// Create OrderPackageSheets and add it to the order package held in the session
var orderPackageSheet =
new OrderPackageSheet
{
OrderSheet = orderSheet,
PackageSheet = packageSheet
};
_orderPackageRepository.SaveChanges();
...
}
When I SaveChanges at this point it takes 15s the on the first loop; each iteration after is fast. I have indexed the tables in question so I believe the database is tuned properly. It's the OrderPackageSheets table that contains 350K rows.
Can anyone tell me how I can optimize this to get rid of the delay?
Thank you!
EF can be slow if you are inserting a lot of rows at same time.
context.Configuration.AutoDetectChangesEnabled = false; wont do too much for you if this is really web app
You need to share your table definition and for instance you can use Simple recovery model which will improve insert performances.
Or, as mentioned, if you need to insert a lot of rows use bulk inserts
If the number of records is too high ,You can use stored procedure instead of EF.
If you need to use EF itself ,Disable auto updating of the context using
context.Configuration.AutoDetectChangesEnabled = false;
and save the context after the loop
Check these links
Efficient way to do bulk insert/update with Entity Framework
http://weblog.west-wind.com/posts/2013/Dec/22/Entity-Framework-and-slow-bulk-INSERTs

Hitting the 2100 parameter limit (SQL Server) when using Contains()

from f in CUSTOMERS
where depts.Contains(f.DEPT_ID)
select f.NAME
depts is a list (IEnumerable<int>) of department ids
This query works fine until you pass a large list (say around 3000 dept ids) .. then I get this error:
The incoming tabular data stream (TDS) remote procedure call (RPC) protocol stream is incorrect. Too many parameters were provided in this RPC request. The maximum is 2100.
I changed my query to:
var dept_ids = string.Join(" ", depts.ToStringArray());
from f in CUSTOMERS
where dept_ids.IndexOf(Convert.ToString(f.DEPT_id)) != -1
select f.NAME
using IndexOf() fixed the error but made the query slow. Is there any other way to solve this? thanks so much.
My solution (Guids is a list of ids you would like to filter by):
List<MyTestEntity> result = new List<MyTestEntity>();
for(int i = 0; i < Math.Ceiling((double)Guids.Count / 2000); i++)
{
var nextGuids = Guids.Skip(i * 2000).Take(2000);
result.AddRange(db.Tests.Where(x => nextGuids.Contains(x.Id)));
}
this.DataContext = result;
Why not write the query in sql and attach your entity?
It's been awhile since I worked in Linq, but here goes:
IQuery q = Session.CreateQuery(#"
select *
from customerTable f
where f.DEPT_id in (" + string.Join(",", depts.ToStringArray()) + ")");
q.AttachEntity(CUSTOMER);
Of course, you will need to protect against injection, but that shouldn't be too hard.
You will want to check out the LINQKit project since within there somewhere is a technique for batching up such statements to solve this issue. I believe the idea is to use the PredicateBuilder to break the local collection into smaller chuncks but I haven't reviewed the solution in detail because I've instead been looking for a more natural way to handle this.
Unfortunately it appears from Microsoft's response to my suggestion to fix this behavior that there are no plans set to have this addressed for .NET Framework 4.0 or even subsequent service packs.
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=475984
UPDATE:
I've opened up some discussion regarding whether this was going to be fixed for LINQ to SQL or the ADO.NET Entity Framework on the MSDN forums. Please see these posts for more information regarding these topics and to see the temporary workaround that I've come up with using XML and a SQL UDF.
I had similar problem, and I got two ways to fix it.
Intersect method
join on IDs
To get values that are NOT in list, I used Except method OR left join.
Update
EntityFramework 6.2 runs the following query successfully:
var employeeIDs = Enumerable.Range(3, 5000);
var orders =
from order in Orders
where employeeIDs.Contains((int)order.EmployeeID)
select order;
Your post was from a while ago, but perhaps someone will benefit from this. Entity Framework does a lot of query caching, every time you send in a different parameter count, that gets added to the cache. Using a "Contains" call will cause SQL to generate a clause like "WHERE x IN (#p1, #p2.... #pn)", and bloat the EF cache.
Recently I looked for a new way to handle this, and I found that you can create an entire table of data as a parameter. Here's how to do it:
First, you'll need to create a custom table type, so run this in SQL Server (in my case I called the custom type "TableId"):
CREATE TYPE [dbo].[TableId] AS TABLE(
Id[int] PRIMARY KEY
)
Then, in C#, you can create a DataTable and load it into a structured parameter that matches the type. You can add as many data rows as you want:
DataTable dt = new DataTable();
dt.Columns.Add("id", typeof(int));
This is an arbitrary list of IDs to search on. You can make the list as large as you want:
dt.Rows.Add(24262);
dt.Rows.Add(24267);
dt.Rows.Add(24264);
Create an SqlParameter using the custom table type and your data table:
SqlParameter tableParameter = new SqlParameter("#id", SqlDbType.Structured);
tableParameter.TypeName = "dbo.TableId";
tableParameter.Value = dt;
Then you can call a bit of SQL from your context that joins your existing table to the values from your table parameter. This will give you all records that match your ID list:
var items = context.Dailies.FromSqlRaw<Dailies>("SELECT * FROM dbo.Dailies d INNER JOIN #id id ON d.Daily_ID = id.id", tableParameter).AsNoTracking().ToList();
You could always partition your list of depts into smaller sets before you pass them as parameters to the IN statement generated by Linq. See here:
Divide a large IEnumerable into smaller IEnumerable of a fix amount of item

Resources