I essentially want to write a bunch of commonly used queries in a web application of this format:
SELECT *
FROM secure_table
WHERE security_function(value 1, value 2) = true;
Value 1 and value 2 will have a limited enough range of values for the idea of caching the result of the security function to be potentially very useful in improving application performance. We would also need to be able to trigger a reset of the cache at will since some conditions would render the cached values out of date.
Is there an out of the box way of doing this with SQL Server (I believe we will be using the 2012 version)? I've had a google around and seen nothing concrete, some references to ASP.NET state but nothing concrete about what that actually involves, and some references to memcached, but that wouldn't seem to go down to function level, so doesn't seem suitable either.
EDIT:
So I would like the function to work something like this:
function security_function(val1, val2) {
result = getFromCache(val1, val2)
if result is empty then
result = //do big complicated query
addToCache(val1, val2, result)
end
return result
}
If you are using ASP.Net you can use the cache object to store the results of the query:
in c#:
Results GetResults(string value1, string value2)
{
string cacheItemName = "cacheItem-" + value1 + "-" + value2;
if (Cache[cacheItemName] != null)
{
return Cache[cacheItemName];
}
else
{
var result = // do big complicated query;
Cache.Insert(cacheItemName, result,
null, DateTime.Now.AddMinutes(15d), // Expire after 15 minutes
System.Web.Caching.Cache.NoSlidingExpiration);
return result;
}
}
Related
I'm setting up a fairly complex Google sheet and trying to automate some routine interpolation with a script. I now have a script that works, but I want to optimise it.
Let me briefly describe the set up with some (simple) example data:
A B C D E
1 Lookup date Result Recorded date Value
2 17/8/2018 - 31/12/2018 210
3 31/12/2018 210 31/3/2019 273
4 14/2/2019 241.5 12/6/2019 411
5 31/3/2019 273
6 12/6/2019 411
7 1/7/2019 411
In this example, I have a small number of recorded values (columns D and E) and I want to compute the value for any date (column A). Column B is the output of the script. The problem is that my script is very slow, taking quite a while on my laptop (sometimes I must refresh the page), and never fully executing on my iPad.
I think part of this may be the volume of requests: I run this script for about 200 cells in my sheet.
I will briefly explain the script (full javascript code below).
It creates a custom function getvalue(x, y, lookupdate) which, for a given x-range (col. D) y-range (col. E) and "lookup date" (eg A4) will return the correct result (eg B4). This result is either:
blank if the lookup date occurs before the first recorded date
the exact value if the lookup date equals a recorded date
an interpolated value if the lookup date is in between two recorded dates
the final recorded value if the lookup date is beyond the range of the recorded dates
Now I have optimised this somewhat. In my implementation, I actually run it as an array for 100 cells in column A (only some of which actually need to run the script). I have another simple system that basically auto-populates the date in column A as a binary flag to say the script needs to run. So using ISBLANK() as a switch, my array formula for cells B3:B103 is:
=ArrayFormula(IF(ISBLANK(A3:A103),"",getvalue(D:D,E:E,A3:A103)))
Even though the array covers 100 cells, only about 50 of them are "activated" with a date in the A column, so only about 50 of them actually need to run the getvalue function. However, as a final complication, I am actually doing this to calculate four different values for each "lookup date", running four different arrays in four columns, so that's what I say the script runs approx. 200 times.
Here is my actual script:
function getvalue(x, y, lookupdate) {
/// LOOKUP AN ARRAY
if (lookupdate.map) {
return lookupdate.map(function(v) {
return getvalue(x, y, v);
});
}
/// GET RID OF EMPTY CELLS IN COLUMN
var xf = x.filter(function(el) {
return el != "";
});
var yf = y.filter(function(el) {
return el != "";
});
/// GET RID OF HEADER ROW
xf.shift()
yf.shift()
/// SAVE THE FIRST AND LAST VALUES
var firstx = xf[0][0]
var firsty = yf[0][0]
var lastx = xf[xf.length - 1][0]
var lasty = yf[yf.length - 1][0]
/// FIGURE OUT WHAT TO RETURN
if (lookupdate < firstx) {
return "";
} else if (lookupdate.valueOf() == firstx.valueOf()) {
return firsty;
} else if (lookupdate > lastx) {
return lasty;
} else {
var check = 0, index;
for(var i = 0, iLen = xf.length; i < iLen; i++) {
if(xf[i][0] == lookupdate) {
return yf[i][0];
} else {
if(xf[i][0] < lookupdate && ((xf[i][0] - check) < (lookupdate - check))) {
check = xf[i][0];
index = i;
}
}
}
var xValue, yValue, xDiff, yDiff, xInt;
yValue = yf[index][0];
xDiff = xf[index+1][0] - check;
yDiff = yf[index+1][0] - yValue;
xInt = lookupdate - check;
return (xInt * (yDiff / xDiff)) + yValue;
}
}
The error message on the iPad is simply the cells never move past "Loading...", and on the laptop it takes much longer than expected.
The most confusing thing is that I think it has gotten worse since I set it up as an array. I previously had it where all 400 cells would run the ISBLANK() check, and then for the approx 200 triggered cells, they would individually run the script. This at least would load on the iPad.
I read on here and on general Google support that scripts will run a lot faster if you batch operations, but it seems to have gotten slower since moving from 200 single cells to 4 arrays.
Does this need to be optimised further, or is there some other reason it might be stalling on my iPad?
Is it even possible to optimise it down and do this in a single call, instead of in 4 arrays?
Accounting for the case
else if (lookupdate.valueOf() == firstx.valueOf()) return firsty;
is superfluous because it is covered already by if(xf[i][0] == lookupdate)
(xf[i][0] - check) < (lookupdate - check) can be simplified to xf[i][0] < lookupdate
You are using pure javascript code, but keep in mind that App Script has many additional functions which are handy when working with Spreadsheet.
https://developers.google.com/apps-script/reference/spreadsheet/
So, e.g. for running your function only for the populated range functions like getDataRange() or getRange() in combination with getNextDataCell() and getLastRow() will be very useful for you.
The main important point - the functionality of your script. Are you assuming that there is an approximately linear relationship between Recorded date and value, and thus interpolate the value for not recorded dates?
In this case the statistically most precise way (and the programmatically simplest one) would be to calculate your slope between the first and last x and y respectively. That is:
Result=first_y+((y_last-y_first)/(x_last-x_first)*(Lookup_Date-first_x))
If this approach is suitable for you, your code would simplify and would look in App Script something like:
function myFunction() {
var ss=SpreadsheetApp.getActiveSpreadsheet().getActiveSheet();
var Result_Range=ss.getRange("A2:B")
var limit=Result_Range.getNextDataCell(SpreadsheetApp.Direction.DOWN).getLastRow()
var Result_values=Result_Range.getValues();
var valueRange=ss.getRange("D1:E");
var values=valueRange.getValues();
var last_Index=valueRange.getNextDataCell(SpreadsheetApp.Direction.DOWN).getLastRow()
var last_y=values[last_Index-1][1];
var last_x=values[last_Index-1][0].valueOf();
var first_y=values[1][1];
var first_x=values[1][0].valueOf();
var slope=(last_y-first_y)/(last_x-first_x);
for(var i=1;i<limit;i++)
{
Result_Range.getCell(i,2).setValue(first_y+(slope*(Result_values[i-1][0].valueOf()-first_x)))
Logger.log(i)
Logger.log(Result_values[i][0].valueOf()-first_x)
}
}
In my gatling scenario, I need to check the session for a few entries that will be Vectors of numbers. I can get the Vectors if present, but when I attempt to add them using .sum I get a ClassCastException stating that java.lang.String can't be cast to java.lang.Integer
I've debugged by printing out the value retrieved from the session (Vector(100,200,300)), and even confirmed that the individual elements are Ints. However when I try to add any of them, either with something like values.sum or values(0)+values(1) I get the class cast exception
I'm setting values in the session with checks like
.check(jsonPath("$..payments..paymentAmount").findAll.optional.saveAs("payments"))
.check(jsonPath("$..receipts..receiptAmount").findAll.optional.saveAs("receipts"))
in my app these will always result in things like Vector(100, 200, 300) if the path was there
then later I want to sum all the values in these lists so I have the action
.exec(session => {
def addAmounts(listNames: Array[String]): Int = {
listNames.foldLeft(0)((acc, listName) => {
session(listName).validate[Seq[Int]] match {
case Success(ints) => ints.sum + acc
case Failure(error) => acc
}})
}
val transactionsTotal = addAmounts(Array("payments", "receipts"))
session.set("total",transactionsTotal)
}
As mentioned, this fails on the listName.sum + acc statement - since they're both Ints I'd expect there'd be no need to cast from a string
The Failure case where nothing was stored from the check works fine
I think this is a scala type inference issue - I got it working by manually casting to Int before doing addition
How can I filter properly using Objectify 4 by several parameters, considering that some of those parameters can come empty, which would mean that I don't want to filter by those?
Example:
Please consider I want to filter something like this:
releases = ofy().load().type(Release.class)
.filter("user.name", searchCriteria.getName())
.filter("category", searchCriteria.getCategory())
.filter("city", searchCriteria.getCity()).list();
In order to match with what I said above, I have now the following code, checking every time which of my parameters come empty so I don't put them on the filter in that case:
if (!nameEmpty && !categoryEmpty && !cityEmpty) {
releases = ofy().load().type(Release.class)
.filter("user.name", searchCriteria.getName())
.filter("category", searchCriteria.getCategory())
.filter("city", searchCriteria.getCity()).list();
} else if (!nameEmpty && !categoryEmpty) {
releases = ofy().load().type(Release.class)
.filter("user.name", searchCriteria.getName())
.filter("category", searchCriteria.getCategory()).list();
} else if (!nameEmpty && !cityEmpty) {
releases = ofy().load().type(Release.class)
.filter("user.name", searchCriteria.getName())
.filter("city", searchCriteria.getCity()).list();
} else if ...
...
How can I avoid this crappy way of filtering and make it with just one line (or a few) using Objectify 4?
Query<Release> query = ofy().load().type(Release.class);
if (!nameEmpty)
query = query.filter("user.name", searchCriteria.getName());
if (!categoryEmpty)
query = query.filter("category", searchCriteria.getCategory())
if (!cityEmpty)
query = query.filter("city", searchCriteria.getCity());
releases = query.list();
I am using SL 4, WCF RIA Services against Entity Framework 4.0. I have an Entity, Visit, that has a string Status field. I have a search screen where I need to display results that have StatusA or StatusB. I am struggling to find a way to specify a client-side query that specifies a collection of statuses that should be matched. If I was to write what I want in SQL it would look something like:
select * from Visit where Status in ('StatusA', 'StatusB');
Client side, it appears to be straightforward to chain Where methods for a WhereAnd effect:
var query = this.PqContext.GetVisitsQuery();
if (!string.IsNullOrEmpty(this.PracticeName))
{
query = query.Where(v => v.PracticeName.ToUpper().Contains(this.PracticeName.ToUpper()));
}
if (this.VisitDateAfter.HasValue)
{
query = query.Where(v => v.VisitDate > this.VisitDateAfter);
}
if (this.VisitDateBefore.HasValue)
{
query = query.Where(v => v.VisitDate < this.VisitDateBefore);
}
However, I can't seem to find a straightforward way to do a WhereOr style operation. I have tried this:
var statuses = new List<string>();
if (this.ShowStatusA)
{
statuses.Add("StatusA");
}
if (this.ShowStatusB)
{
statuses.Add("StatusB");
}
if (statuses.Any())
{
query = query.Where(BuildContainsExpression<Visit, string>(e => e.Status, statuses));
}
Where BuildContainsExpression looks like:
private static Expression<Func<TElement, bool>> BuildContainsExpression<TElement, TValue>(Expression<Func<TElement, TValue>> valueSelector, IEnumerable<TValue> values)
{
if (null == valueSelector)
{
throw new ArgumentNullException("valueSelector");
}
if (null == values)
{
throw new ArgumentNullException("values");
}
ParameterExpression p = valueSelector.Parameters.Single();
if (!values.Any())
{
return e => false;
}
var equals =
values.Select(
value =>
(Expression)Expression.Equal(valueSelector.Body, Expression.Constant(value, typeof(TValue))));
var body = equals.Aggregate<Expression>(Expression.Or);
return Expression.Lambda<Func<TElement, bool>>(body, p);
}
But this throws a "Bitwise operators are not supported in queries." exception. Any clues? Is there an alternative way to build an expression tree that works here or do I need to pass all the parameters over to the server and use the BuildContainsExpression there?
Your time and your guidance are much appreciated.
You can create a query method such as the following in your domain service:
GetVisitsByStatus(string[] statusList) {
// create the LINQ where clause here
}
And then from the client, call context.GetVistsByStatusQuery(string[]).
Not all of LINQ is (or even can) be exposed over the URL, so there are always cases where you need to use simple parameters, and have the middle tier construct the LINQ expressions that eventually define the query that goes to the back-end data store.
Hope that helps.
My table has a timestamp column named "RowVer" which LINQ maps to type System.Data.Linq.Binary. This data type seems useless to me because (unless I'm missing something) I can't do things like this:
// Select all records that changed since the last time we inserted/updated.
IEnumerable<UserSession> rows = db.UserSessions.Where
( usr => usr.RowVer > ???? );
So, one of the solutions I'm looking at is to add a new "calculated column" called RowTrack which is defined in SQL like this:
CREATE TABLE UserSession
(
RowVer timestamp NOT NULL,
RowTrack AS (convert(bigint,[RowVer])),
-- ... other columns ...
)
This allows me to query the database like I want to:
// Select all records that changed since the last time we inserted/updated.
IEnumerable<UserSession> rows = db.UserSessions.Where
( usr => usr.RowTrack > 123456 );
Is this a bad way to do things? How performant is querying on a calculated column? Is there a better work-around?
Also, I'm developing against Sql Server 2000 for ultimate backwards compatibility, but I can talk the boss into making 2005 the lowest common denominator.
AS Diego Frata outlines in this post there is a hack that enables timestamps to be queryable from LINQ.
The trick is to define a Compare method that takes two System.Data.Linq.Binary parameters
public static class BinaryComparer
{
public static int Compare(this Binary b1, Binary b2)
{
throw new NotImplementedException();
}
}
Notice that the function doesn't need to be implemented, only it's name (Compare) is important.
And the query will look something like:
Binary lastTimestamp = GetTimeStamp();
var result = from job in c.GetTable<tblJobs>
where BinaryComparer.Compare(job.TimeStamp, lastTimestamp)>0
select job;
(This in case of job.TimeStamp>lastTimestamp)
EDIT:
See Rory MacLeod's answer for an implementation of the method, if you need it to work outside of SQL.
SQL Server "timestamp" is only an indicator that the record has changed, its not actually a representation of Date/Time. (Although it is suppose to increment each time a record in the DB is modified,
Beware that it will wrap back to zero (not very often, admittedly), so the only safe test is if the value has changed, not if it is greater than some arbitrary previous value.
You could pass the TimeStamp column value to a web form, and then when it is submitted see if the TimeStamp from the form is different to the value in the current record - if its is different someone else has changed & saved the record in the interim.
// Select all records that changed since the last time we inserted/updated.
Is there a better work-around?
Why not have two columns, one for createddate another for lastmodifieddate. I would say that is more traditional way to handle this scenario.
Following on from jaraics' answer, you could also provide an implementation for the Compare method that would allow it to work outside of a query:
public static class BinaryExtensions
{
public static int Compare(this Binary b1, Binary b2)
{
if (b1 == null)
return b2 == null ? 0 : -1;
if (b2 == null)
return 1;
byte[] bytes1 = b1.ToArray();
byte[] bytes2 = b2.ToArray();
int len = Math.Min(bytes1.Length, bytes2.Length);
int result = memcmp(bytes1, bytes2, len);
if (result == 0 && bytes1.Length != bytes2.Length)
{
return bytes1.Length > bytes2.Length ? 1 : -1;
}
return result;
}
[DllImport("msvcrt.dll")]
private static extern int memcmp(byte[] arr1, byte[] arr2, int cnt);
}
The use of memcmp was taken from this answer to a question on comparing byte arrays. If the arrays aren't the same length, but the longer array starts with the same bytes as the shorter array, the longer array is considered to be greater than the shorter one, even if the extra bytes are all zeroes.