I need information which should be collected in a recursion.
I have a wpf application with the following recursion:
public static int GetClusterTotalDocumentsCount(ref int count, int clusterID, int lastrunid)
{
List<Cluster> subclusters = DBHandler.GetSubClustersOperational(clusterID);
if (subclusters.Count == 0)
return count;
foreach (Cluster subclutser in subclusters)
{
int temp = DBHandler.GetClusterDocumentsCount(subclutser.ID, 0);
count += temp;
GetClusterTotalDocumentsCount(ref count, subclutser.ID, 0);
}
return count;
}
Now I want the same to be done in silverlight. instead of using ADO.net I'm consuming a WCF service for the data. Problem is, the calls are now Asynchronous so I cannot use this recursion. Any ideas?
Think about your solution - you want recursive remote operation? It means separate roundtrip to the server for each cluster in hierarchy! What about creating service operation which will simply return total and recursion will still take a place on server side inside that operation?
Edit:
Btw. doing database queries in recursion (and in sequence) is also wrong approach. SQL server 2005 and newer supports recursive queries natively so you should check them and wrap your call in single stored procedure using CTE (common table expressions) to do recursive query in single roundtrip to the database.
Related
Code Migration due to Performance Issues :-
SQL Server LIKE Condition ( BEFORE )
SQL Server Full Text Search --> CONTAINS ( BEFORE )
Elastic Search ( CURRENTLY )
Achieved So Far :-
We have a web page created in ASP.Net Core which has a Auto Complete Drop Down of 2.5+ Million Companies Indexed in Elastic Search https://www.99corporates.com/
Due to performance issues we have successfully shifted our code from SQL Server Full Text Search to Elastic Search and using NEST v7.2.1 and Elasticsearch.Net v7.2.1 in our .Net Code.
Still looking for a solution :-
If the user does not select a company from the Auto Complete List and simply enters a few characters and clicks on go then a list should be displayed which we had done earlier by using the SQL Server Full Text Search --> CONTAINS
Can we call the ASP.Net Web Service which we have created using SQL CLR and code like SELECT * FROM dbo.Table WHERE Name IN( dbo.SQLWebRequest('') )
[System.Web.Script.Services.ScriptMethod()]
[System.Web.Services.WebMethod]
public static List<string> SearchCompany(string prefixText, int count)
{
}
Any better or alternate option
While that solution (i.e. the SQL-APIConsumer SQLCLR project) "works", it is not scalable. It also requires setting the database to TRUSTWORTHY ON (a security risk), and loads a few assemblies as UNSAFE, such as Json.NET, which is risky if any of them use static variables for caching, expecting each caller to be isolated / have their own App Domain, because SQLCLR is a single, shared App Domain, hence static variables are shared across all callers, and multiple concurrent threads can cause race-conditions (this is not to say that this is something that is definitely happening since I haven't seen the code, but if you haven't either reviewed the code or conducted testing with multiple concurrent threads to ensure that it doesn't pose a problem, then it's definitely a gamble with regards to stability and ensuring predictable, expected behavior).
To a slight degree I am biased given that I do sell a SQLCLR library, SQL#, in which the Full version contains a stored procedure that also does this but a) handles security properly via signatures (it does not enable TRUSTWORTHY), b) allows for handling scalability, c) does not require any UNSAFE assemblies, and d) handles more scenarios (better header handling, etc). It doesn't handle any JSON, it just returns the web service response and you can unpack that using OPENJSON or something else if you prefer. (yes, there is a Free version of SQL#, but it does not contain INET_GetWebPages).
HOWEVER, I don't think SQLCLR is a good fit for this scenario in the first place. In your first two versions of this project (using LIKE and then CONTAINS) it made sense to send the user input directly into the query. But now that you are using a web service to get a list of matching values from that user input, you are no longer confined to that approach. You can, and should, handle the web service / Elastic Search portion of this separately, in the app layer.
Rather than passing the user input into the query, only to have the query pause to get that list of 0 or more matching values, you should do the following:
Before executing any query, get the list of matching values directly in the app layer.
If no matching values are returned, you can skip the database call entirely as you already have your answer, and respond immediately to the user (much faster response time when no matches return)
If there are matches, then execute the search stored procedure, sending that list of matches as-is via Table-Valued Parameter (TVP) which becomes a table variable in the stored procedure. Use that table variable to INNER JOIN against the table rather than doing an IN list since IN lists do not scale well. Also, be sure to send the TVP values to SQL Server using the IEnumerable<SqlDataRecord> method, not the DataTable approach as that merely wastes CPU / time and memory.
For example code on how to accomplish this correctly, please see my answer to Pass Dictionary to Stored Procedure T-SQL
In C#-style pseudo-code, this would be something along the lines of the following:
List<string> = companies;
companies = SearchCompany(PrefixText, Count);
if (companies.Length == 0)
{
Response.Write("Nope");
}
else
{
using(SqlConnection db = new SqlConnection(connectionString))
{
using(SqlCommand batch = db.CreateCommand())
{
batch.CommandType = CommandType.StoredProcedure;
batch.CommandText = "ProcName";
SqlParameter tvp = new SqlParameter("ParamName", SqlDbType.Structured);
tvp.Value = MethodThatYieldReturnsList(companies);
batch.Paramaters.Add(tvp);
db.Open();
using(SqlDataReader results = db.ExecuteReader())
{
if (results.HasRows)
{
// deal with results
Response.Write(results....);
}
}
}
}
}
Done. Got the solution.
Used SQL CLR https://github.com/geral2/SQL-APIConsumer
exec [dbo].[APICaller_POST]
#URL = 'https://www.-----/SearchCompany'
,#JsonBody = '{"searchText":"GOOG","count":10}'
Let me know if there is any other / better options to achieve this.
I've encountered a strange phenomena then investigatating a slow view of a typical ASP.NET MVC application. One of the queries is running ridiculously slow for no obvious reason. The LINQ query in question look like this (Db is DbContext):
var testResults = Db.CustomTestResults
.Include(tr => tr.TestMachine.Platform)
.Include(tr => tr.TestCase)
.Include(tr => tr.CustomTestResultAnalysis.Select(tra => tra.AnalysisOutcomeData))
.Where(tr => tr.CustomTestBuildId == testBuild.Id)
.ToList()
.AsReadOnly();
nothing special actually. Depending on filter query result set can vary in size, from 10 to 10000 records at max.
The SQL generated query (captured by LINQ debug log), executed from SSMS, runs fast, about 2 seconds for the largest sets and less than a second for smaller ones. However then run by IIS strange things happen. The queries began to run like ~1/100x slower speed. The smaller ones take ~10 seconds to execute, the larger are failing due to query execution timeout. I'm not sure if any other queries are affected, but this one is only that is dealing with large data sets, so it's most obvious to notice the problem.
As this was not confusing enough this same code was running perfectly as expected not so long ago. So the bug seems to be caused by some external factors. The database is SQL Server 2014 SP2, EF is at v6.2, IIS 7.5.
Would appreciate any ideas in what areas and how I could investigate this further.
As it turned out, the issue was in SQL Server optimizations, which start to work some time after multiple runs of the similar queries. This problem can be detected by any nonrelevant change to the original query, which fixes performance for some time.
This behaviour can be properly mitigated by controlling query command options. One of the solutions for EF is demonstrated here.
As a temporary "quick-and-dirty" solution I used this approach to randomize query each time, thus preventing optimizations by SQL Server engine:
private static IQueryable<CustomTestResult> RandomizeQuery(IQueryable<CustomTestResult> query)
{
const int minConditions = 1;
const int maxConditions = 5;
const int minId = -100;
const int maxId = -1;
var random = new Random();
var conditionsCount = random.Next(minConditions, maxConditions);
for (int i = 0; i < conditionsCount; i++)
{
var randomId = random.Next(minId, maxId);
query = query.Where(test => test.Id != randomId);
}
return query;
}
Since the SQL has not changed but it is having issues depending on what platform you run on I would start with your settings. A GREAT reference for the whys and hows is written by Erland Sommarskog this: http://www.sommarskog.se/query-plan-mysteries.html
It is long but I imagine you will find your answer in there.
I am trying to start using EF6 for a project. My database is already filled with millions of records.
I can't find right explanation how does EF send T-SQL to SQL Server? I am afraid that I am going to download bunch of data to user for no reason.
In code below I have found three way to get my data to List<> but I am not sure which is right way to do WHERE clause at SQL.
I do not want to fill client with millions of record and to query (filter) that data at client.
using (rgtBaza baza = new rgtBaza())
{
var t = baza.Database.SqlQuery<CJE_DOC>("select * from cje_doc where datum between #od and #do",new SqlParameter("od", this.dateTimePickerOD.Value.Date ) ,new SqlParameter("do", this.dateTimePickerOD.Value.Date)).ToList();
var t = baza.CJE_DOC.Where(s => s.DATUM.Value >= this.dateTimePickerOD.Value.Date && s.DATUM.Value <= this.dateTimePickerDO.Value.Date).ToList();
var query = from b in baza.CJE_DOC
where b.DATUM >= this.dateTimePickerOD.Value.Date && b.DATUM.Value <= this.dateTimePickerDO.Value.Date
select b;
var t = query.ToList();
this.dataGridViewCJENICI.DataSource = t;
}
In all 3 cases, the filtering will happen on the database side, the filtering (or WHERE clause) will not take place on the client side.
If you want to verify that this is true, especially for your last 2 options, add some logging so that you can see the generated SQL:
baza.Database.Log = s => Console.WriteLine(s);
In this case, since you are using EF already, choose the 2nd or 3rd options, they are both equivalent with different syntax. Pick your favorite syntax.
In all of those examples, EF6 will generate a SQL query including the where clause - it won't perform the where clause on the client.
It won't actually retrieve any data from the database until you iterate through the results, which in the examples above, is when you call .ToList().
EF6 would only run the filter on the client if you called something like:
baza.CJE_DOC.ToList().Where(x => x.Field == value)
In this instance, it would retrieve the entire table when you called ToList(), and then use a client-side Linq query to filter the results in the where clause.
Any of the 3 will run the query on the SQL Server.
EF relies on LINQ's deferred execution model to build up an expression tree. Once you take an action that causes the expression to be enumerated (e.g. calling ToList(), ToArray(), or any of the other To*() methods), it will convert the expression tree to SQL, send the query to the server, and then start returning the results.
One of the side effects of this is that when using the query or lambda syntax, expressions that EF does not understand how to convert to SQL will cause an exception.
If you absolutely need to use some code that EF can't handle, you can break your code into multiple segments -- filtering things down as far as possible via code that can be converted to SQL, using the AsEnumerable() method to "close off" the EF expression, and doing your remaining filtering or transformations using Linq to Objects.
I am using MS SQL server 2008 with Hibernate. the question I have is how Hibernate implements setMaxResults
Take the following simple scenario.
If I have a query that returns 100 rows and if I pass 1 to setMaxResults, will this affect the returned result from the SQL server itself(as if running a select top 1 statement) or does Hibernate get all the results first (all 100 rows in this case) and pick the top one itself?
Reason I am asking is that it would have a huge performance issue when the number of rows starts to grow.
Thank you.
Hibernate will generate a limit-type query, for all dialects which supports limit query. As the SQLServerDialect supports this (see org.hibernate.dialect.SQLServerDialect.supportsLimit(), and .getLimitString()), you will get a select top 1-query.
If you would like to be absolutly sure, you may turn on debug-logging, or enable the showSql-option and test.
May be following snippet will help. Assume we have a managed Bean class EmpBean and we want only first 5 records. So following is the code
public List<EmpBean> getData()
{
Session session = null;
try
{
session = HibernateUtil.getSession();
Query qry = session.createQuery("FROM EmpBean");
qry.setMaxResults(5);
return qry.list();
}
catch(HibernateException e)
{}
finally
{
HibernateUtil.closeSession(session);
}
return null;
}
Here getSession and closeSession are static utility methods which will take care of creating and closing session
OutOfMemoryError caused when db4o databse has 15000+ objects
My question is in reference to my previous question (above). For the same PostedMessage model and same query.
With 100,000 PostedMessage objects, the query takes about 1243 ms to return first 20 PostedMessages.
Now, I have saved 1,000,000 PostedMessage objects in db4o. The same query took 342,132 ms. Which is non-linearly high.
How can I optimize the query speed?
FYR:
The timeSent and timeReceived are Indexed fields.
I am using SNAPSHOT query mode.
I am not using TA/TP.
Do you sort the result? Unfortunatly db4o doesn't use the index for sorting / orderBy. That means it will run a regular sort algorith, with O(n*log(n)). It won't scala liniearly.
Also db4o doesn't support a TOP operator. That means even without sorting it takes quite a bit of time to copy the ids to the results set, even when you never read the entities afterwards.
So, there's no real good solution for this, except trying to use some criteria which cut down the result size.
Some adventerous people might use a different query evaluation, but personally don't recommend that.
#Gamlor No, I am not sorting at all. The code is as follows:
public static ObjectSet<PostedMessage> getMessagesBetweenDates(
Calendar after,
Calendar before,
ObjectContainer db) {
if (after == null || before == null || db == null) {
return null;
}
Query q = db.query(); //db is pre-configured to use SNAPSHOT mode.
q.constrain(PostedMessage.class);
Constraint from = q.descend("timeRecieved").constrain(new Long(after.getTimeInMillis())).greater().equal();
q.descend("timeRecieved").constrain(new Long(before.getTimeInMillis())).smaller().equal().and(from);
ObjectSet<EmailMessage> results = q.execute();
return results;
}
The arguments to this method are as follows:
after = 13-09-2011 10:55:55
before = 13-09-2011 10:56:10
And I expect only 10 PostedMessages to be returned between "after" and "before". (I am generating dummy PostedMessage with timeReceived incremented by 1 sec each.)