I'm using Gatling with the JDBC feeder and would like to dynamically add a parameter to the JDBC feeder's where clause based on the response from a previous request. Here is my example, I'm trying to do a post that will create a user, then have the feed grab the user's generated UUID using the userId returned from the create user request, then post some data with the UUID.
val dbConnectionString = "jdbc:mysql://localhost:3306/user"
val sqlQuery = "SELECT user_uuid FROM users where user_id = '${userId}'"
val sqlUserName = "dbUser"
val sqlPassword = "dbPassword"
val sqlQueryFeeder = jdbcFeeder(dbConnectionString, sqlUserName, sqlPassword, sqlQuery)
val uuidPayload = """{"userUUID":"${user_uuid}"}"""
val MyScenario = scenario("MyScenario").exec(
(pause(1, 2))
.exec(http("SubmitFormThatCreatesUserData")
.post(USER_CREATE_URL)
.body(StringBody("""{"username":"test#test.com"}""")).asJson
.header("Accept", "application/json")
.check(status.is(200))
.check(jsonPath("$..data.userId").exists.saveAs("userId")))
.feed(sqlQueryFeeder)
.exec(http("SubmitStuffWithUUID")
.post(myUUIDPostURL)
.body(uuidPayload).asJson
.header("Accept", "application/json")
.check(status.is(200)))
)
I have verified the following:
1) The user data does get inserted into the DB correctly on the form post
2) The userId is returned from that form post
3) The userId correctly saved as a Gatling session variable
4) The SQL query will execute correctly if I hard-code the user id variable
The problem I have is that when I have the Gatling ${userId} parameter on the JDBC feeder's where clause it appears the userId variable isn't used, I get an error saying java.lang.IllegalArgumentException: requirement failed: Feeder must not be empty. When I replace the ${userId} with a hard-coded userId everything works as expected. I would just like to know how I can use the userId session parameter in my JDBC feeder's where clause.
The call to jdbcFeeder(dbConnectionString, sqlUserName, sqlPassword, sqlQuery) to create a jdbc feeder only takes strings as parameters, not gatling expressions (like ${userId}).
In your scenario as posted, you are not really using feeders as intended - they are generally used to have different users pick up different values from a pool of values whereas you have a static user name and are only taking the first value returned from the db. It's generally not a good idea to have fetching of external data in the middle of your scenarios as it can make timings unpredictable.
Could you just look up and hardcode the user_uuid? The best approach would be to get all your user data and look up things like uuids in the before block of the simulation. However, you can't use the gatling DSL there.
You could also use a scala variable to store the user_uuid and define a feeder inline, but this would get messy if you do need to support multiple users
Related
I am facing an issue using Salesforce API. While querying I am getting the following exception: "The SOQL FIELDS function must have a LIMIT of at most 200". Now, I understand SF expects a max of 200 only. So, I wanted to ask how can I query when the results are more than 200?
I can only use REST API to query, but if there is another option, then please let me know and I will try to add it in my code.
Thanks in Advance
You could chunk it, SELECT FIELDS(ALL) FROM Account ORDER BY Id LIMIT 200. Read the id of last record and in next query add WHERE Id> '001...'. but that's not very effective.
Look into "describe" calls, waste 1 call to learn names of all fields you need and explicitly list them in the query instead of relying on FIELDS(ALL). You can compose SOQL up to 20k characters long and with "bulk API" queries you could fetch up to 10k records in each API call so "investing" 1 call for describes would quickly pay off.
You could even cache the describe's result in your application and fetch fresh only if something interesting changed, there's rest API header for that: https://developer.salesforce.com/docs/atlas.en-us.232.0.api_rest.meta/api_rest/sobject_describe_with_ifmodified_header.htm
Try this it is Helpful:
// Get the Map of Schema of Account SObject
Map<String, Schema.SObjectField> fieldMap = Account.sObjectType.getDescribe().fields.getMap();
// Get all of the fields on the object
Set<String> setFieldNames = fieldMap.keySet();
list<String> lstFieldNames = new List<String>(setFieldNames);
// Dynamic Query String.
List<Account> lstAccounts = Database.query('SELECT ' + String.join(lstFieldNames, ',') + ' FROM Account');
system.debug('lstAccounts'+lstAccounts);
Code Migration due to Performance Issues :-
SQL Server LIKE Condition ( BEFORE )
SQL Server Full Text Search --> CONTAINS ( BEFORE )
Elastic Search ( CURRENTLY )
Achieved So Far :-
We have a web page created in ASP.Net Core which has a Auto Complete Drop Down of 2.5+ Million Companies Indexed in Elastic Search https://www.99corporates.com/
Due to performance issues we have successfully shifted our code from SQL Server Full Text Search to Elastic Search and using NEST v7.2.1 and Elasticsearch.Net v7.2.1 in our .Net Code.
Still looking for a solution :-
If the user does not select a company from the Auto Complete List and simply enters a few characters and clicks on go then a list should be displayed which we had done earlier by using the SQL Server Full Text Search --> CONTAINS
Can we call the ASP.Net Web Service which we have created using SQL CLR and code like SELECT * FROM dbo.Table WHERE Name IN( dbo.SQLWebRequest('') )
[System.Web.Script.Services.ScriptMethod()]
[System.Web.Services.WebMethod]
public static List<string> SearchCompany(string prefixText, int count)
{
}
Any better or alternate option
While that solution (i.e. the SQL-APIConsumer SQLCLR project) "works", it is not scalable. It also requires setting the database to TRUSTWORTHY ON (a security risk), and loads a few assemblies as UNSAFE, such as Json.NET, which is risky if any of them use static variables for caching, expecting each caller to be isolated / have their own App Domain, because SQLCLR is a single, shared App Domain, hence static variables are shared across all callers, and multiple concurrent threads can cause race-conditions (this is not to say that this is something that is definitely happening since I haven't seen the code, but if you haven't either reviewed the code or conducted testing with multiple concurrent threads to ensure that it doesn't pose a problem, then it's definitely a gamble with regards to stability and ensuring predictable, expected behavior).
To a slight degree I am biased given that I do sell a SQLCLR library, SQL#, in which the Full version contains a stored procedure that also does this but a) handles security properly via signatures (it does not enable TRUSTWORTHY), b) allows for handling scalability, c) does not require any UNSAFE assemblies, and d) handles more scenarios (better header handling, etc). It doesn't handle any JSON, it just returns the web service response and you can unpack that using OPENJSON or something else if you prefer. (yes, there is a Free version of SQL#, but it does not contain INET_GetWebPages).
HOWEVER, I don't think SQLCLR is a good fit for this scenario in the first place. In your first two versions of this project (using LIKE and then CONTAINS) it made sense to send the user input directly into the query. But now that you are using a web service to get a list of matching values from that user input, you are no longer confined to that approach. You can, and should, handle the web service / Elastic Search portion of this separately, in the app layer.
Rather than passing the user input into the query, only to have the query pause to get that list of 0 or more matching values, you should do the following:
Before executing any query, get the list of matching values directly in the app layer.
If no matching values are returned, you can skip the database call entirely as you already have your answer, and respond immediately to the user (much faster response time when no matches return)
If there are matches, then execute the search stored procedure, sending that list of matches as-is via Table-Valued Parameter (TVP) which becomes a table variable in the stored procedure. Use that table variable to INNER JOIN against the table rather than doing an IN list since IN lists do not scale well. Also, be sure to send the TVP values to SQL Server using the IEnumerable<SqlDataRecord> method, not the DataTable approach as that merely wastes CPU / time and memory.
For example code on how to accomplish this correctly, please see my answer to Pass Dictionary to Stored Procedure T-SQL
In C#-style pseudo-code, this would be something along the lines of the following:
List<string> = companies;
companies = SearchCompany(PrefixText, Count);
if (companies.Length == 0)
{
Response.Write("Nope");
}
else
{
using(SqlConnection db = new SqlConnection(connectionString))
{
using(SqlCommand batch = db.CreateCommand())
{
batch.CommandType = CommandType.StoredProcedure;
batch.CommandText = "ProcName";
SqlParameter tvp = new SqlParameter("ParamName", SqlDbType.Structured);
tvp.Value = MethodThatYieldReturnsList(companies);
batch.Paramaters.Add(tvp);
db.Open();
using(SqlDataReader results = db.ExecuteReader())
{
if (results.HasRows)
{
// deal with results
Response.Write(results....);
}
}
}
}
}
Done. Got the solution.
Used SQL CLR https://github.com/geral2/SQL-APIConsumer
exec [dbo].[APICaller_POST]
#URL = 'https://www.-----/SearchCompany'
,#JsonBody = '{"searchText":"GOOG","count":10}'
Let me know if there is any other / better options to achieve this.
My code is simple:
app = Flask(__name__)
app.config.from_object('config')
db = SQLAlchemy(app)
db.metadata.reflect()
And it throws no errors. However, when I inspect the metadata after this reflection, it returns an empty immutabledict object.
The parameters in my connection string is 100% correct and the code works with non-RDS databases.
It seems to happen to others as well but I can't find a solution.
Also, I have tried to limit the reflection to specific tables using the "only" parameter in the metadata.reflect function, and this is the error I get:
sqlalchemy.exc.InvalidRequestError: Could not reflect: requested table(s) not available in mssql+pyodbc://{connection_string}: (users)
I've fixed it. The reflect() method of the SQLAlchemy class has a parameter named 'schema'. Setting this parameter, to "dbo" in my case, solved it.
I am using Flask-SQLAlchemy, which does not have the said parameter in its reflect() method. You can follow this post to gain access to that parameter and others, such as 'only'.
This error occurs when reflect is called without the schema name provided. For example, this will cause the error to happen:
metadata.reflect(only = [tableName])
It needs to be updated to use the schema of the table you are trying to reflect over like this:
metadata.reflect(schema=schemaName, only = [tableName])
You have to set schema='dbo' in parameter for reflect.
db.Model.metadata.reflect(bind=engine, schema='dbo', only=['User'])
and then create model of your table:
class User(db.Model):
__table__ = Base.metadata.tables['dbo.User']
and to access data from that table:
When you use NHibernate to "fetch" a mapped object, it outputs a SELECT query to the database. It outputs this using parameters; so if I query a list of cars based on tenant ID and name, I get:
select Name, Location from Car where tenantID=#p0 and Name=#p1
This has the nice benefit of our database creating (and caching) a query plan based on this query and the result, so when it is run again, the query is much faster as it can load the plan from the cache.
The problem with this is that we are a multi-tenant database, and almost all of our indexes are partition aligned. Our tenants have vastly different data sets; one tenant could have 5 cars, while another could have 50,000. And so because NHibernate does this, it has the net effect of our database creating and caching a plan for the FIRST tenant that runs it. This plan is likely not efficient for subsequent tenants who run the query.
What I WANT to do is force NHibernate NOT to parameterize certain parameters; namely, the tenant ID. So I'd want the query to read:
select Name, Location from Car where tenantID=55 and Name=#p0
I can't figure out how to do this in the HBM.XML mapping. How can I dictate to NHibernate how to use parameters? Or can I just turn parameters off altogether?
OK everyone, I figured it out.
The way I did it was overriding the SqlClientDriver with my own custom driver that looks like this:
public class CustomSqlClientDriver : SqlClientDriver
{
private static Regex _partitionKeyReplacer = new Regex(#".PartitionKey=(#p0)", RegexOptions.Compiled);
public override void AdjustCommand(IDbCommand command)
{
var m = _tenantIDReplacer.Match(command.CommandText);
if (!m.Success)
return;
// replace the first parameter with the actual partition key
var parameterName = m.Groups[1].Value;
// find the parameter value
var tenantID = (IDbDataParameter ) command.Parameters[parameterName];
var valueOfTenantID = tenantID.Value;
// now replace the string
command.CommandText = _tenantIDReplacer.Replace(command.CommandText, ".TenantID=" + valueOfTenantID);
}
} }
I override the AdjustCommand method and use a Regex to replace the tenantID. This works; not sure if there's a better way, but I really didn't want to have to open up NHibernate and start messing with core code.
You'll have to register this custom driver in the connection.driver_class property of the SessionFactory upon initialization.
Hope this helps somebody!
I developing web application with play framework 2.3.8 and scala, with complex architecture on backend and front-end side. As backend we use MS SQL, with many stored procedures, and called it by anorm. And here one of the problems.
I need to update some fields in database. The front end calls play framework, and recive name of the field, and value. Then I parse, field name, and then I need to generate SQL Query for update field. I need assign null, for all parameters, except recived parameter. I try to do it like that:
def updateCensusPaperXX(name: String, value: String, user: User) = {
DB.withConnection { implicit c =>
try {
var sqlstring = "Execute [ScXX].[updateCensusPaperXX] {login}, {domain}"
val params = List(
"fieldName1",
"fieldName2",
...,
"fieldNameXX"
)
for (p <- params){
sqlstring += ", "
if (name.endsWith(p))
sqlstring += value
else
sqlstring += "null"
}
SQL(sqlstring)
.on(
"login" -> user.login,
"domain" -> user.domain,
).execute()
} catch {
case e: Throwable => Logger.error("update CensusPaper04 error", e)
}
}
}
But actually that doesn't work in all cases. For example, when I try to save string, it give's me an error like:
com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect syntax near 'some phrase'
What is the best way to generate sql query using anorm with all nulls except one?
The reason this is happening is because when you write the string value directly into the SQL statement, it needs to be quoted. One way to solve this would be to determine which of the fields are strings and add conditional logic to determine whether to quote the value. This is probably not the best way to go about it. As a general rule, you should be using named parameters rather than building a string to with the parameter values. This has a few of benefits:
It will probably be easier for you to diagnose issues because you will get more sensible error messages back at runtime.
It protects against the possibility of SQL injection.
You get the usual performance benefit of reusing the prepared statement although this might not amount to much in the case of stored procedure invocation.
What this means is that you should treat your list of fields as named parameters as you do with user and domain. This can be accomplished with some minor changes to your code above. First, you can build your SQL statement as follows:
val params = List(
"fieldName1",
"fieldName2",
...,
"fieldNameXX"
)
val sqlString = "Execute [ScXX].[updateCensusPaperXX] {login}, {domain}," +
params.map("{" + _ + "}").mkString{","}
What is happening above is that you don't need to insert the values directly, so you can just build the string by adding the list of parameters to the end of your query string.
Then you can go ahead and start building your parameter list. Note, the parameters to the on method of SQL is a vararg list of NamedParameter. Basically, we need to create Seq of NamedParameters that covers "login", "domain" and the list of fields you are populating. Something like the following should work:
val userDomainParams: Seq[NamedParameter] = (("login",user.login),("domain",user.domain))
val additionalParams = params.map(p =>
if (name.endsWith(p))
NamedParameter(p, value)
else
NamedParameter(p, None)
).toSeq
val fullParams = userDomainParams ++ additionalParams
// At this point you can execute as follows
SQL(sqlString).on(fullParams:_*).execute()
What is happening here is that you building the list of parameters and then using the splat operator :_* to expand the sequence into the varargs needed as arguments for the on method. Note that the None used in the NamedParameter above is converted into a jdbc NULL by Anorm.
This takes care of the issue related to strings because you are no longer writing the string directly into the query and it has the added benefit eliminating other issues related with writing the SQL string rather than using parameters.