I believe this question is mostly related to SQL, but I will give context for what we are trying to do in case someone has done it before.
We are trying to implement DataTables JQuery plug-in with server-side processing using SQL Server for our ASP .NET Core MVC app. On top of that, we have dynamic columns, which we were able to generate for DataTables using this github code.
We have been successful in handling paging, entry #, and column searching processes (couldn't figure out global search, and we got column search to work only with SQL views). However, sorting columns are only handled partially.
The issue with column sorting is that we don't know how many and what type of columns could be returned. This means that there may be an NVARCHAR column containing strings that are currency ($3,403.09), or NVARCHARs that are regular numbers, and some of the values in those columns may be NULL. Same with dates with varying formats. We don't know how to handle sorting these NVARCHARs in SQL as the correct type instead of a string. Having said that, client-side DataTables is able to deal with identifying how data should be sorted.
I don't expect an answer, but if anyone has handled a similar situation, I'd greatly appreciate the help.
Related
I know, I know... There's no way to safely do this and I should always be using parameterized queries, but here's my situation.
I'm attempting to use Advanced Datagridview (https://github.com/davidegironi/advanceddatagridview) in a VB.Net forms application with a large dataset. The gridview control has Excel like filtering that's quite nifty but the filtering is only done on the client side so I'd need to pull down the entire table to accurately filter. This isn't practical so I need a way to pass the filtering to the server. The control is able to generate a SQL-formatted string of it's current filter, such as:
([Plant] IN ('Argile - Chester', 'Bartlett - Riverton')) AND
([VendorNum] IN ('1015')) AND
([Qty] < 22.12)
So it would be quite easy just to concatenate that onto the query populating the grid view, but that would obviously be wide open to injection attacks. Is there a way I can safely validate the string being supplied to check for any potential issues? Or can anyone think of some other way of solving this problem given my constraints?
Only other option I can think of is doing away with the built-in filtering and just using form controls to set the filter values and pass those as parameters, but I really want to make use of the built-in filtering. Another option would be to try and parse out the filter string the gridview generates and break it into individual fields to pass to parameters. But there could potentially be several fields being filtered on and numerous values. This would get dicey fast.
Thanks in advance.
In the application I am working on, we have data grids that have the capability to display custom views of the data. As a point of reference, we modeled this feature using the concept of views as they exist in SharePoint.
The custom views should have the following capabilities:
Be able to define which subset of columns (of those that are
available) should be displayed in the view.
Be able to define one or
more filters for retrieving data. These filters are not constrained
to use only the columns that are in the result set but must use one
of the available columns. Standard logical conditions and operators
apply to these filters. For example, ColumnA Equals Value1 or
ColumnB >= Value2.
Be able to define a set of columns that the data will be sorted by. This set of columns can be one or more columns
from the set of columns that will be returned in the result set.
Be
able to define a set of columns that the data will be grouped by.
This set of columns can be one or more columns from the set of
columns that will be returned in the result set.
I have application code that will dynamically generate the necessary SQL to retrieve the appropriate set of data. However, it appears to perform poorly. When I run across a poorly performing query, my first thought is to determine where indexes might help. The problem here is that I won't necessarily know which indexes need to be created as the underlying query could retrieve data in many different ways.
Essentially, the SQL that is currently being used does the following:
Creates a temporary table variable to hold the filtered data. This table contains a column for each column that should be returned in the result set.
Inserts data that matches the filter into the table variable.
Queries the table variable to determine the total number of rows of data.
If requested, determines the grouping values of the data in the table variable using the specified grouping columns.
Returns the requested page of the requested page size of data from the table variable, sorted by any specified sort columns.
My question is what are some ways that I may improve this process? For example, one idea I had was to have my table variable only contain the columns of data that are used to group and sort and then join in the source table at the end to get the rest of the displayed data. I am not sure if this would make any difference which is the reason for this post.
I need to support versions 2014, 2016 and 2017 of SQL Server in addition to SQL Azure. Essentially, I will not be able to use a specific feature of an edition of SQL Server unless that feature is available in all of the aforementioned platforms.
(This is not really an "answer" - I just can't add comments yet because my reputation score isn't high enough yet.)
I think your general approach is fine - essentially you are making a GUI generator for SQL. However a few things:
This type of feature is best suited for a warehouse or read only replica database. Do not build this on a live production transactional database. There are permutations that you haven't thought of that your users will find that will kill your database (it's also true from a warehouse standpoint, but they usually don't have response time expectations as a transactional database)
The method you described for doing paging is not efficient from a database standpoint. You are essentially querying, filtering, grouping, and sorting the same exact dataset multiple times just to cherry pick a few rows each time. If you have the data cached, that might be ok, but you shouldn't make that assumption. If you have the know how, figure out how to snapshot the entire final data set with an extra column to keep the data physically sorted in the order the user requested. That way you can quickly query the results for your paging.
If you have a Repository/DAL layer, design your solution so that in the future certain combinations of tables/columns can utilize hardcoded queries/stored procedures. There will inevitably be certain queries that pop up that cause you performance issues and you may have to build a custom solution for specific queries in order to get the desired performance that can't be obtained by your dynamic sql
We're starting to implement Unicode as we've added some international customers. There are some issues comparing character data in SSIS because of capitals, accents, and other data problems.
I've thought that the Fuzzy logic lookup could be a good solution. However, when testing this solution out, I realized that in a lot of our existing code we limit what data to process, and send in those values by parameters.
I've noticed that in the Fuzzy Lookup, I can specify the name of the table, but I can't make changes like remove a % from a field and turn it into a decimal. Any ideas how we can setup the lookup with calculated fields?
Thanks!
Create a view in your database with the proper transformation your require using a sql query.
I want auto search option in textbox and data is fetching from database. I have thousands of data in my database table (almost 8-10000 rows). I know how to achieve this but as I am fetching thousands of data, it will take a lot of time to fetch. How to achieve this without getting slow down? Should I follow any other methodology to achieve this apart from simple fetching methods? I am using Oracle SQL Developer for database.
Besides the obvious solutions involving indexes and caching, if this is web technology and depending on your tool you can sometimes set a minimum length before the server call is made. Here is a jquery UI example: https://api.jqueryui.com/autocomplete/#option-minLength
"The minimum number of characters a user must type before a search is performed. Zero is useful for local data with just a few items, but a higher value should be used when a single character search could match a few thousand items."
It depends on your web interface, but you can use two tecniques:
Paginate your data: if your requirements are to accept empty values and to show all the results load them in block of a predefined size. goggle for example paginates search results. On Oracle pagination is made using the rownum special variable (see this response). Beware: you must first issue a query with a order by and then enclose it in a new one that use rownum. Other databases that use the limit keyword behave in a different way. If you apply the pagination techique to a drop down you end up with an infinite scroll (see this response for example)
Limit you data imposing some filter that limits the number of rows returned; your search display some results only after the user typed at least n chars in the field
You can combine 1 & 2, but unless you find an existing web component (a jquery one for example) it may be a difficult task if you don't have a Javascript knowledge.
I have recently encountered a problem I have hard time working around. I have to tune a rather nasty query that extensively uses - in many forms and on several layers - XML. The problem is that while it is easy to spot the slow part of the whole process - Table Valued Function [XML Reader] - it is rather hard to map it to any specific part of the query. The properties does not indicate on which XML column does this particular object work and I'm really puzzled why is that. Generally, it is relatively easy to map object to a part of a query but these XML objects are giving me a pause.
Thanks in advance.