Two related questions. I hope that's OK. If not, please feel free to focus on the second question!
I have an Access 2019 database form that uses a macro for search functionality. The form recordsource is based on two tables using a left join. Any macro options that use ApplyFilter based on any fields in the joined tables operate correctly and quickly.
However, I need a search to use a subquery and for some reason the Macro Where Condition does not support a sub query (it shows a red exclamation point and gives an error when trying to save the macro "The 'ApplyFilter' macro action has an invalid value for the 'Where Condition' argument").
The Where Condition is:
JobPropertyID in (select PropertyID from Properties where PropertyAddress like '*' & [Address contains] & '*')
(I have tried various combinations of % and * wildcards, and quotes).
This used to work in earlier versions of Access (we upgraded from 2003 to 2019).
So, question 1 is - Is this a known limitation?
(I can work-around it by using RunCode to set the Filter and FilterOn in VBA code).
The second, and more important question relates to the performance when using a sub query. For example, this pseudo query to return jobs at matching property addresses:
select JobID, JobDescription, CompanyName from JobDetails Left Join Company on JobCompanyID = CompanyID where JobPropertyID in (select PropertyID from Properties where PropertyAddress like '*High Street*')
This does work but can take about a minute to run in Access. If I run the query in SQL Server Management Studio it shows the results in less than a second. I looked at the output in SQL Profiler and it appears that Access is requesting all rows from the joined tables and all rows from the Properties table (with no criteria being applied to either) and then, presumably, applying the filter and the sub query internally.
Is there a way to encourage Access to let SQL Server do the work?
I have tried using a pass through query, and this returns the correct results quickly, but is read only, so not suitable for a form that allows editing.
I suppose I could display the search results in subform and apply a filter to the main form from the OnCurrent event in the subform. But this seems a rather clunky solution.
Any thoughts welcome.
Access tends to like Exists over In a lot when it comes to performance/translation, and that second query can be rewritten to an Exists:
select JobID, JobDescription, CompanyName from JobDetails
Left Join Company on JobCompanyID = CompanyID
where Exists(select 1 from Properties where PropertyAddress like '*High Street*' AND JobPropertyID = PropertyID )
As for that filter, you can pass a filter string to avoid the macro from doing weird stuff:
="JobPropertyID in (select PropertyID from Properties where PropertyAddress like '*' & [Address contains] & '*')"
But you should really consider using VBA for your own sanity (working with macros is intended for beginners but quickly becomes harder than learning VBA even when doing trivial tasks), unless that's restricted in your company, in which case you will run into obstacles using Access.
Any client side filter from Access? One table - works great.
A left join - usually ok!
but, after that ? The problem is that Access (and most) ODBC connections view EACH table as a 100% separate link. While your linked tables point to SQL server? They don't have to!
So, one table could point to say a local FoxPro/dbase table.
Another table could point to a linked Excel sheet.
And another table could point to sql server.
So, if you try and use any compilex sql - involving joins, or ESPECIALLY sub-quires, then Access really can't make the assuming that these additional tables are from the same server - and they might be Excel sheets!! - So, those data sources are "often" seen and assumed by access to be seperate tables. (and while Access is amazing that you can say do sub-queries against to linked Excel sheets? Well, there no server on the other end - Access client side has to figure this out to make it work.
You have two great choices to fix this issue.
the best choice? Build that query server side (in SSMS stuido). Save it as a view. Now the joins and all that jazz will occur 100% server side. You can STILL very effective filter against that "table" in Access. You of course now will have a linked "view" in access. But access sees this as one table - filters work fantastic in this case.
And in a lot of cases, that sub-query can be re-wrtten as a left join (not always - but often so). So again, build that complex join server side, save it as view, and then from Access link to that view.
You can filter against columns in both parent and child table - it again works well.
However, if you really need that sub query and worse so that sub query needs values/paramters from the child table?
Well, then build the query but use what is called a Pass-though query in Access. When you do this, the sql is sent raw un-touched to sql server (no difference in speed, and what you put in that PT query will work 100% exactly the same if you just typed that SQL right into SSMS. So, for example, that query that ran so fast?
Take it 100% "as-is", un-changed. Build a PT query in Access, and then cut+paste in that working and speedy query you had. It will now run the same in Access. Just keep in mind that a PT query in access is read only, but often with sub-query etc., such queries would be anyway.
The next last one? Well, with a PT query in Access, you can use those to call + use + consume a stored procedure. And stored procedures are often used because this lets you EASY set parameters in say a sub query (not that you ever really needed say t-sql code, and some big procedure to run - but you need paramaters - and a SQL stored procedures give you this ability.
in VBA, to execute that stored procedure? Well, I create a working PT query, (could be anything or any T-SQL). I then use that PT query over and over in code.
eg like this:
dim rstData as DAO.RecordSet
With Currentdb.QueryDefs("MyPTQuery")
.SQL = "EXEC dbo.GetInvoices"
set rstData = .OpenRecordSet()
End if
Now above, the stored procedure (to get say invoices) did not have any parameters, but it can and could. So, say you have to pass the year, and month to get invoices.
So, your PT query could/would be used like this (from SSMS).
EXEC dbo.GetInvoices 2021, 3
So, you have to pass it the year + month. IN VBA, then we could go:
dim Mymonth as integer
dim Myyear as integer
MyMonth = Month(date) ' get current month based on today date
Myyear = Year(date) ' get current year based on today date
With Currentdb.QueryDefs("MyPTQuery")
.SQL = "EXEC dbo.GetInvoices " & MyYear & "," & MyMonth
rst = .OpenRecordSet()
End if
And you not limited to shoving the results into a recordset.
Say, we have a report - set the source of the Report to "MyPTQquery"
Then do this:
With Currentdb.QueryDefs("MyPTQuery")
.SQL = "EXEC dbo.GetInvoices " & MyYear & "," & MyMonth
End if
' now open report
docmd.OpenReport "rptInvoices",acViewePreview
So that query can be used for anything - reports, forms or a VBA recordset.
So I find that views are the most easy, perform REALLY good, and you can even have criteria against that view, and it works very well. (so this is the least amount of work). So built that messy multi-table join server side, save as view, and then link from Access client side. This works VERY well, since often you can take a REALLY big dog pile query in Access with multi-table joins - it not going to work well.
So, you take that messy query, move it to SSMS - get it working. Then save that working SQL as a view. Now re-name (or delete) the query. Say it was
QueryInvoices.
Well, now link to the view with the same name. now that VBA code, the forms, and the reports that were ALL using that messy client side query don't have to be changed!! - the name of the server side view is thus linked with that old query name. Again this gives fantastic performance.
So, the above should give you some ideas. As noted, views are your friend here. But as noted, the problem is you can't inject/have parameters in the sub query from Access being passed to a view. So, you can either build a stored procedure - use above PT Query idea.
And the other way? Well, you build the whole SQL string in VBA code, shove it into that PT query - and that also works. (but in-line messy long sql statements in VBA code can be quite a challenge. But for parameters in a sub-query, then the view trick can't be used, so you travel down the PT query road.
I am processing data from a database with millions of rows. I am pulling a batch of 1000 items from the database and processing them without a problem. I am not loading the whole entity and I am just pulling down a few columns of data for the batch.
What I want to do is mark the 1000 rows as processed with a single SQL command.
Something like:
UPDATE dbo.Clients
SET HasProcessed = 1
WHERE ClientID IN (...)
The ... is just a list of integers.
Context:
Azure SQL Server 2012
Entity Framework
Database.ExecuteSqlCommand
Ideas:
I know I could build the command as a pure string, but this would mean not using any SqlParameters and not benefiting from the query plan optimization.
Also, I found some information about table-valued parameters, but this requires creating a table type and some overhead which I would like to avoid. This is just a list of integers after all.
Question:
Is there an easy (performant) way to do this that I am overlooking either with Entity Framework or ExecuteSqlCommand?
If not and using table-valued parameters is the best way, could you provide a complete example of how to convert an integer list into the simplest type and running that with the above query?
I'm working with a new REST backend talking to a SQL Server. Our REST api allows for the caller to pass in the columns/fields they want returned (?fields=id,name,phone).
The idea seems very normal. The issue I'm bumping up against is resistance to dynamically generating the SQL statement. Any arguments passed in would be passed to the database using a parameterized query, so I'm not concerned about SQL injection.
The basic idea would be to "inject" the column-names passed in, into a SQL that looks like:
SELECT <column-names>
FROM myTable
ORDER BY <column-name-to-sort-by>
LIMIT 1000
We sanitize all column names and verify their existence in the table, to prevent SQL injection issues. Most of our programmers are used to having all SQL in static files, and loading them from disk and passing them on to the database. The idea of code creating SQL makes them very nervous.
I guess I'm curious if others actually do this? If so, how do you do this? If not, how do you manage "dynamic columns and dynamic sort-by" requests passed in?
I think a lot of people do it especially when it comes to reporting features. There are actually two things one should do to stay on the safe side:
Parameterize all WHERE clause values
Use user input values to pick correct column/table names, don't use the user values in the sql statement at all
To elaborate on item #2, I would have a dictionary where Key is a possible user input and Value is a correponding column/table name. You can store this dictionary wherever you want: config file, database, hard code, etc. So when you process user input you just check a dictionary if the Key exists and if it does you use the Value to add a column name to your query. This way you just use user input to pick required column names but don't use the actual values in your sql statement. Besides, you might not want to expose all columns. With a predefined dictionary you can easily control the list of available columns for a user.
Hope it helps!
I've done similar to what Maksym suggests. In my case, keys were pulled directly from the database system tables (after scrubbing the user request a bit for syntactic hacks and permissions).
The following query takes care of some minor injection issues through the natural way SQL handles the LIKE condition. This doesn't go as far as handling permissions on each field (as some fields are forbidden based on the log-in) but it provides a very basic way to retrieve these fields dynamically.
CREATE PROC get_allowed_column_names
#input VARCHAR(MAX)
AS BEGIN
SELECT
columns.name AS allowed_column_name
FROM
syscolumns AS columns,
sysobjects AS tables
WHERE
columns.id = tables.id AND
tables.name = 'Categories' AND
#input LIKE '%' + columns.name + '%'
END
GO
-- The following only returns "Picture"
EXEC get_allowed_column_names 'Category_,Cat%,Picture'
GO
-- The following returns both "CategoryID and Picture"
EXEC get_allowed_column_names 'CategoryID, Picture'
GO
This might seem like a silly question, but I'm surprised that I didn't find a clear answer to this already:
Is it possible to use SQL Server parameters for writing a query with dynamic column names (and table names), or does the input just need to be sanitized very carefully?
The situation is that tables and their column names (and amount of columns) are generated dynamically and there is no way to know beforehand to manually write a query. Since the tables & columns aren't known I can't use an ORM, so I'm resorting to manual queries. Usually I'd use parameters to fill in values to prevent SQL injection, however I'm pretty sure that this cannot be done the same way when specifying the table name and/or column names. I want to create generic queries for insert, update, upsert, and select, but I obviously don't want to open myself up to potential injection. Is there a best practices on how to accomplish this safely?
Just as an FYI - I did see this answer, but since there's no way for me to know the column / table names beforehand a case statement probably won't work for this situation.
Environment: SQL Server 2014 via ADO.NET (.NET 4.5 / C#)
There is no mechanism for passing table or column references to procedures. You just pass them as strings and then use dynamic SQL to build your queries. You do have to take precautions to ensure that your string parameters are valid.
One way to do this would be to validate that all table and column reference strings have valid names in sys.tables and sys.columns before building your T-SQL queries. Then you can be sure that they can be used safely.
You can also use literal parameters with dynamic sql when using the sp_executesql procedure. You can't use it to validate your table and column names, but it validates and prevents SQL injection with your other parameters.
I am working on an application where certain SQL Server queries will be limited to specific users based on a group identity.
It seems to make sense to store these within the application database where I can associate them with group names and it is easy to add/remove in future.
The queries are quite complex though and take a number of parameters such as JQuery to and from date fields, current username etc. So the query is constructed like:
MyQuery = "SELECT * FROM Table1 WHERE Username = '" + System...CurrentUser +"' AND Somedate > '" + from.text +'";
Now I'm unsure how to take this code and create an equivalent representation in the database. I thought about using specific identifiers such as %USERNAME%, %FROMDATE% etc then use a String Replace function but I'm not sure that would carry over the single quotes.
Would something like Replace("%USERNAME%", "'" + CurrentUser + "'") work?
Any better ideas!
Two things are:
Restrict some queries based on user groups.
Reasonably easy to add/remove/edit queries in the future (not hardcoded into the C#?)
First off: BIG WARNING: concatenating together your SQL statements bears a great danger of SQL injection attacks. You should avoid this technique at all costs and use parametrized queries instead.
Second of all: couldn't you just create those "complex queries" as stored procedures (with parameters) - that's what stored procedures are really good at - encapsulating complex queries into a simple call from the outside.
You could then handle permissions using standard SQL Server features - give those users (and/or groups) EXECUTE permission on that stored procedure who are allowed to call it.