Apex batch not executing all records - salesforce

I'm relatively new to Apex, but I have a question about a batch job that I am creating. I am trying to insert AccountTeamMember records based on my company's territory alignment. The code seems to be working fine, but with one flaw: it is only inserting 100 AccountTeamMember records per user (it should be closer to 400, as that is how many I have loaded in my dev sandbox). Does anyone know what I can do to get an AccountTeamMember record inserted for all accounts per user, rather than 100 of the ~400? Is it something to do with the query including parent-child relationships and governor limits since it is such an even number (100)?
Here is the relevant code:
//list to hold new account teams
List<AccountTeamMember> acctMembers = new List<AccountTeamMember>();
//list to hold new account sharing rules
List<AccountShare> acctSharingRules = new List<AccountShare>();
global Database.querylocator start(Database.BatchableContext BC){
String query = 'SELECT (SELECT User__c FROM Territory_Users__r), (SELECT Account__c FROM Territory_Accounts__r) FROM Territory_Master__c';
return Database.getQueryLocator(query);}
global void execute(Database.BatchableContext BC, List<sObject> scope){
for (sObject s : scope) {
Territory_Master__c tm = (Territory_Master__c) s;
Territory_User__c[] userList = tm.getSObjects('Territory_Users__r');
Territory_Account__c[] accountList = tm.getSObjects('Territory_Accounts__r');
if (userList != null && accountList != null){
for(Territory_User__c uu : userList){
for(Territory_Account__c aa: accountList){
AccountTeamMember addRecord = new AccountTeamMember();
addRecord.AccountId = aa.Account__c;
addRecord.TeamMemberRole = 'Sales Rep';
addRecord.UserId = uu.User__c;
acctMembers.add(addRecord);
AccountShare addSharing = new AccountShare();
addSharing.AccountId = aa.Account__c;
addSharing.OpportunityAccessLevel = 'Read';
addSharing.CaseAccessLevel = 'Read';
addSharing.AccountAccessLevel = 'Edit';
addSharing.UserOrGroupId = uu.User__c;
acctSharingRules.add(addSharing);
}
}
}
}
//DML
if(acctMembers.size() > 0){
insert acctMembers;
}
if(acctSharingRules.size() > 0){
insert acctSharingRules;
}
}
Thanks,
Trey
FYI: This is the final result based on the answer to the question:
global Database.querylocator start(Database.BatchableContext BC){
String query = 'SELECT Id FROM Territory_Master__c';
return Database.getQueryLocator(query);}
global void execute(Database.BatchableContext BC, List<sObject> scope){
for(sObject s : scope){
Territory_Master__c tm = (Territory_Master__c) s;
List<Territory_User__c> userList = [SELECT User__c FROM Territory_User__c WHERE Territory_Master__c = :tm.Id];
List<Territory_Account__c> accountList = [SELECT Account__c FROM Territory_Account__c WHERE Territory_Master__c = :tm.Id];
if (userList != null && accountList != null){
for(Territory_User__c uu : userList){
for(Territory_Account__c aa: accountList){
AccountTeamMember addRecord = new AccountTeamMember();
addRecord.AccountId = aa.Account__c;
addRecord.TeamMemberRole = 'Sales Rep';
addRecord.UserId = uu.User__c;
acctMembers.add(addRecord);
acctSharingRules.add(new AccountShare(
AccountId = aa.Account__c,
OpportunityAccessLevel = 'Read',
CaseAccessLevel = 'Read',
AccountAccessLevel = 'Edit',
UserOrGroupId = uu.User__c)
);
}
}
}
}
//DML
if(acctMembers.size() > 0){
insert acctMembers;
}
if(acctSharingRules.size() > 0){
insert acctSharingRules;
}
}

I suspect that your subqueries got limited and it's your job to make sure you've finished with this record before moving on to the next one.
http://www.salesforce.com/us/developer/docs/api/Content/sforce_api_calls_soql_relationships.htm
Subquery results are like regular query results in that you might need
to use queryMore() to retrieve all the records if there are many
children. For example, if you issue a query on accounts that includes
a subquery, your client application must handle results from the
subquery as well
(the example is for Java calling Salesforce so don't copy-paste the code, try to understand the concept. As far as I know Apex doesn't have this queryMore method, it's for integrations only)
Option 1
Funny enough, it should in theory work if you'd change the for loops to this:
for(Territory_User__c uu : tm.getSObjects('Territory_Users__r')){
for(Territory_Account__c aa: tm.getSObjects('Territory_Accounts__r'))
That's because for loops written in that way should automatically call their internal queryMore().
If it won't work (I haven't tested it), you'll have to make a bit more complex changes.
Option 2
Remove the subqueries from the main query, you'll have to put them in the execute(). Something like this:
for(sObject s : scope){
Territory_Master__c tm = (Territory_Master__c) s;
for(Territory_User__c uu : [SELECT User__c FROM Territory_User__c WHERE Territory_Master__c = :tm.Id]){
for(Territory_Account__c aa: [SELECT Account__c FROM Territory_Account__c WHERE Territory_Master__c = :tm.Id]){
acctMembers.add(new AccountTeamMember(
AccountId = aa.Account__c,
TeamMemberRole = 'Sales Rep',
UserId = uu.User__c)
);
acctSharingRules.add(new AccountShare(
AccountId = aa.Account__c,
OpportunityAccessLevel = 'Read',
CaseAccessLevel = 'Read',
AccountAccessLevel = 'Edit',
UserOrGroupId = uu.User__c)
);
}
}
}
Side notes
Is there a chance you'll hit a limit of 50K rows retrieved across all objects (Territory Master/User/Account? If that's the case you might have to limit the scope of your batch job (optional second parameter passed to the Database.executeBatch()).
This trick can make your script execute bit faster and use less statements (so you won't hit another governor limit):
acctSharingRules.add(new AccountShare(
AccountId = aa.Account__c,
OpportunityAccessLevel = 'Read',
CaseAccessLevel = 'Read',
AccountAccessLevel = 'Edit',
UserOrGroupId = uu.User__c)
);

There's nothing obviously wrong with your code there, so if I were you I'd try to narrow down the cause of the problem. The source of the problem is going to be one of:
Your query is only returning 100 items in accountList or userList or both
The insert can only handle 100 items at a time and is silently failing to insert the rest
The insert is failing for the items after 100
If you haven't already done so, you should familiarize yourself with using Debug Logs in Salesforce.com (Setup | Monitoring | Debug Logs). Turn them on, then run this code with some System.debug calls in there to figure out:
How many items are in userList and accountList?
If it's more than 100, when you go to insert the share rows, are some of those inserts failing? If you use Database.insert() instead of just the insert statement then you'll get a SaveResult[] back that will tell you for each row you attempted to insert whether it was successful or errored out.
So I can't say off the top of my head why this is failing, and I don't see any particular limit that should apply here, but the above should help you debug it at least.

Related

Stored procedure - get anticipated columns before fully executing statement?

I'm working through a stored procedure and wondering if there's a way to retrieve the anticipated result column list from a sql statement before fully executing.
Scenarios:
dynamic SQL
a UDF that might vary the columns outside of our control
EX:
//inbound parameter
SET QUERY_DEFINITION_ID = 12345;
//Initial statement pulls query text from bank of queries
var sqlText = getQueryFromQueryBank(QUERY_DEFINITION_ID);
//now we run our query
var cmd = {sqlText: sqlText };
stmt = snowflake.createStatement(cmd);
What I'd like to be able to do is say "right - before you run this, give me the anticipated column list" so I can compare it to what's expected.
EX:
Expected: [col1, col2, col3, col4]
Got: [col1]
Result: Oops. Don't run.
Rationale here is that I want to short-circuit the execution if something is missing - before it potentially runs for a while. I can validate all of this after the fact, but it would be really helpful to stop early.
Any ideas very much appreciated!
This sample SP code shows how to get a list of columns that a query will project into the result before you run the query. It should only be used for large, long running queries because it will take a few seconds to get the column list.
There are a couple of caveats. 1) It will only return the names of the columns. It won't tell you how they were built, that is, whether they're aliased, direct from a table, calculated, etc. 2) The example query I used is straight from the Snowflake documentation here https://docs.snowflake.com/en/user-guide/sample-data-tpcds.html#functional-query-definition. For convenience, I minimized the query to a single line. The output of the columns includes object qualifiers in addition to the column names, so V1.I_CATEGORY, V1.D_YEAR, V1.D_MOY, etc. If you don't want them to make it easier to compare names, you can strip off the qualifiers using the JavaScript split function on the dot and take index 1 of the resulting array.
create or replace procedure EXPLAIN_BEFORE_RUNNING()
returns string
language javascript
execute as caller
as
$$
// Set the context for the session to the TPC-H sample data:
executeNonQuery("use schema snowflake_sample_data.tpcds_sf10tcl;");
// Here's a complex query from the Snowflake docs (minimized to one line for convienience):
var sql = `with v1 as( select i_category, i_brand, cc_name, d_year, d_moy, sum(cs_sales_price) sum_sales, avg(sum(cs_sales_price)) over(partition by i_category, i_brand, cc_name, d_year) avg_monthly_sales, rank() over (partition by i_category, i_brand, cc_name order by d_year, d_moy) rn from item, catalog_sales, date_dim, call_center where cs_item_sk = i_item_sk and cs_sold_date_sk = d_date_sk and cc_call_center_sk= cs_call_center_sk and ( d_year = 1999 or ( d_year = 1999-1 and d_moy =12) or ( d_year = 1999+1 and d_moy =1)) group by i_category, i_brand, cc_name , d_year, d_moy), v2 as( select v1.i_category ,v1.d_year, v1.d_moy ,v1.avg_monthly_sales ,v1.sum_sales, v1_lag.sum_sales psum, v1_lead.sum_sales nsum from v1, v1 v1_lag, v1 v1_lead where v1.i_category = v1_lag.i_category and v1.i_category = v1_lead.i_category and v1.i_brand = v1_lag.i_brand and v1.i_brand = v1_lead.i_brand and v1.cc_name = v1_lag.cc_name and v1.cc_name = v1_lead.cc_name and v1.rn = v1_lag.rn + 1 and v1.rn = v1_lead.rn - 1) select * from v2 where d_year = 1999 and avg_monthly_sales > 0 and case when avg_monthly_sales > 0 then abs(sum_sales - avg_monthly_sales) / avg_monthly_sales else null end > 0.1 order by sum_sales - avg_monthly_sales, 3 limit 100;`;
// Before actually running the query, generate an explain plan.
executeNonQuery("explain " + sql);
// Now read the column list from the explain plan from the result set.
var columnList = executeSingleValueQuery("COLUMN_LIST", `select "expressions" as COLUMN_LIST from table(result_scan(last_query_id())) where "operation" = 'Result';`);
// For now, just exit with the column list as the output...
return columnList;
// Your code here...
// Helper functions:
function executeNonQuery(queryString) {
var out = '';
cmd = {sqlText: queryString};
stmt = snowflake.createStatement(cmd);
var rs;
rs = stmt.execute();
}
function executeSingleValueQuery(columnName, queryString) {
var out;
cmd1 = {sqlText: queryString};
stmt = snowflake.createStatement(cmd1);
var rs;
try{
rs = stmt.execute();
rs.next();
return rs.getColumnValue(columnName);
}
catch(err) {
if (err.message.substring(0, 18) == "ResultSet is empty"){
throw "ERROR: No rows returned in query.";
} else {
throw "ERROR: " + err.message.replace(/\n/g, " ");
}
}
return out;
}
$$;
call Explain_Before_Running();

linq2db - server side bulkcopy

I'm trying to do a "database side" bulk copy (i.e. SELECT INTO/INSERT INTO) using linq2db. However, my code is trying to bring the dataset over the wire which is not possible given the size of the DB in question.
My code looks like this:
using (var db = new MyDb()) {
var list = db.SourceTable.
Where(s => s.Year > 2012).
GroupBy(s => new { s.Column1, s.Column2 }).
Select(g => new DestinationTable {
Property1 = 'Constant Value',
Property2 = g.First().Column1,
Property3 = g.First().Column2,
Property4 = g.Count(s => s.Column3 == 'Y')
});
db.Execute("TRUNCATE TABLE DESTINATION_TABLE");
db.BulkCopy(new BulkCopyOptions {
BulkCopyType = BulkCopyType.MultipleRows
}, list);
}
The generated SQL looks like this:
BeforeExecute
-- DBNAME SqlServer.2017
TRUNCATE TABLE DESTINATION_TABLE
DataConnection
Query Execution Time (AfterExecute): 00:00:00.0361209. Records Affected: -1.
DataConnection
BeforeExecute
-- DBNAME SqlServer.2017
DECLARE #take Int -- Int32
SET #take = 1
DECLARE #take_1 Int -- Int32
SET #take_1 = 1
DECLARE #take_2 Int -- Int32
...
SELECT
(
SELECT TOP (#take)
[p].[YEAR]
FROM
[dbo].[SOURCE_TABLE] [p]
WHERE
(([p_16].[YEAR] = [p].[YEAR] OR [p_16].[YEAR] IS NULL AND [p].[YEAR] IS NULL) AND ...
...)
FROM SOURCE_TABLE p_16
WHERE p_16.YEAR > 2012
GROUP BY
...
DataConnection
That is all that is logged as the bulkcopy fails with a timeout, i.e. SqlException "Execution Timeout Expired".
Please note that running this query as an INSERT INTO statement takes less than 1 second directly in the DB.
PS: Anyone have any recommendations as to good code based ETL tools to do large DB (+ 1 TB) ETL. Given the DB size I need things to run in the database and not bring data over the wire. I've tried pyspark, python bonobo, c# etlbox and they all move too much data around. I thought linq2db had potential, i.e. basically just act like a C# to SQL transpiler but it is also trying to move data around.
I would suggest to rewrite your query because group by can not return first element. Also Truncate is a part of the library.
var sourceQuery =
from s in db.SourceTable
where s.Year > 2012
select new
{
Source = s,
Count = Sql.Ext.Count(s.Column3 == 'Y' ? 1 : null).Over()
.PartitionBy(s.Column1, s.Column2).ToValue()
RN = Sql.Ext.RowNumber().Over()
.PartitionBy(s.Column1, s.Column2).OrderByDesc(s.Year).ToValue()
};
db.DestinationTable.Truncate();
sourceQuery.Where(s => s.RN == 1)
.Insert(db.DestinationTable,
e => new DestinationTable
{
Property1 = 'Constant Value',
Property2 = e.Source.Column1,
Property3 = e.Source.Column2,
Property4 = e.Count
});
After some investigation I stumbled onto this issue. Which lead me to the solution. The code above needs to change to:
db.Execute("TRUNCATE TABLE DESTINATION_TABLE");
db.SourceTable.
Where(s => s.Year > 2012).
GroupBy(s => new { s.Column1, s.Column2 }).
Select(g => new DestinationTable {
Property1 = 'Constant Value',
Property2 = g.First().Column1,
Property3 = g.First().Column2,
Property4 = g.Count(s => s.Column3 == 'Y')
}).Insert(db.DestinationTable, e => e);
Documentation of the linq2db project leaves a bit to be desired however, in terms of functionality its looking like a great project for ETLs (without horrible 1000s of line copy/paste sql/ssis scripts).

Multiple AggregateResult Querys

Hi guys,
I'm currently trying to join two objects in a same query or result.
My question is if it's possible to show or debug the sum of FIELD A FROM LEAD + sum of FIELD B FROM two different Objects.
Here's an example I'm working on:
Btw I really appreciate your time and comments, and if i'm making a mistake pls let me know, thank you.
public static void example() {
String sQueryOne;
String sQueryTwo;
AggregateResult[] objOne;
AggregateResult[] objTwo;
//I tried to save the following querys into a sObject List
List<SObject> bothObjects = new List<SObject>();
sQueryOne = 'Select Count(Id) records, Sum(FieldA) fieldNA From Lead';
objOne = Database.query(sQueryOne);
sQueryTwo = 'Select Count(Id) records, Sum(FieldA) fieldNB From Opportunity';
objTwo = Database.query(sQueryTwo);
bothObjects.addAll(objOne);
bothObjects.addAll(objTwo);
for(sObject totalRec : bothObjects) {
//There's a Wrapper(className) I created which contains some variables(totalSum)
className finalRes = new className();
finalRes.totalSum = (Integer.valueOf(fieldNA)) + (Integer.valueOf(fieldNB));
System.debug('The sum is: '+finalRes.totalSum);
For example if I call a System debug with the previous variable finalRes.totalSum it's just showing the first value(fieldNA) duplicated.
The following debug shows the current values of the sObject List which I want to sum for example FIELD0 = from leads, FIELD0 = from Opportunities.
}
}
You access the columns in AggregateResult by calling get('columnAlias'). If you didn't specify an alias they'll be autonumbered by SF as expr0, expr1... When in doubt you can always go System.debug(results);
Some more info: https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/langCon_apex_SOQL_agg_fns.htm
This might give you some ideas:
List<AggregateResult> results = new List<AggregateResult>{
[SELECT COUNT(Id) records, SUM(NumberOfEmployees) fieldA, SUM(AnnualRevenue) fieldB FROM Account],
[SELECT COUNT(Id) records, SUM(Amount) fieldA, SUM(TotalOpportunityQuantity) fieldB FROM Opportunity],
[SELECT COUNT(Id) records, SUM(NumberOfEmployees) fieldA, SUM(AnnualRevenue) fieldB FROM Lead]
/* hey, not my fault these are only 2 standard numeric fields on Lead.
It doesn't matter that they're identical to Account fields, what matters is what their SUM(...) aliases are
*/
};
List<Decimal> totals = new List<Decimal>{0,0,0};
for(AggregateResult ar : results){
totals[0] += (Decimal) ar.get('records');
totals[1] += (Decimal) ar.get('fieldA');
totals[2] += (Decimal) ar.get('fieldB');
}
System.debug(totals); // (636, 8875206.0, 9819762558.0) in my dev org
(I'm not saying it's perfect, your wrapper class sounds like better idea or maybe even Map<String, Decimal>. Depends what are you going to do with the results)

Transaction Management By ##Transactional Annotation

I have one requirement in which i have to insert record in one table and update the record in another table.
There are 100 records for which i have to do above insertion and updation.
But If any record fails due to any error in insertion or updation only that record should be rollback not others.Means if There are 100 records and 60 gets success and 61 gets fails due to any error then only 61 records should be rolled back not all other 60 records.
I am using Spring boot JPA and #Transactional Annotation.But if put this annotation on starting of my method,Then in case of any error it roll back all records.
I am attaching my code below.Please suggest how i can achieve this.
In below code i have persistRecord() method which have all my code for insertion and updation.I have not marked this with #Transactional annotation insteed below method is annotated with #Transactional annotation
logger.debug("insertSalesTargetData method called of service class");
String userid = (String) webSession.getAttribute("username");
logger.debug("Userid :" + userid);
HashMap<String, String> persistRecordStatus = new HashMap<String, String>();
Query qCountCheck = em.createNativeQuery(QueryConstant.CHECK_SALES_TARGET_DATA_QUERY);
List<Object> existingRecordList = null;
persistRecordStatus.put("TOTAL_RECORD", String.valueOf(xlsRowslist.size()));
int failedRecordCount = 0;
int successRecordCount = 0;
for (int i = 0; i < xlsRowslist.size(); i++) {
ArrayList rowList = xlsRowslist.get(i);
// try {
double weekNoDouble = Double.parseDouble((String) rowList.get(0));// Week No
String fromDate = (String) rowList.get(1);// fromDate
String toDate = (String) rowList.get(2);// toDate
double catCodeDouble = Double.parseDouble((String) rowList.get(3));// catCode
int catCode = (int) catCodeDouble;
int weekNo = (int) weekNoDouble;
String target = (String) rowList.get(4);// target
String salesGoalId = fromDate + toDate + catCode;
salesGoalId = salesGoalId.replace("-", "");
// Check if the sales goal id already exist in the database or not
qCountCheck.setParameter(1, salesGoalId);// SALES_GOAL_ID
existingRecordList = qCountCheck.getResultList();
logger.debug("Count List Size " + existingRecordList.size());
if (existingRecordList != null && existingRecordList.size() > 0) {
if (existingRecordList.get(0) != null) {
BigDecimal row = (BigDecimal) existingRecordList.get(0);
// try {
logger.debug("Persisting record no " + i);
persistRecord(row, salesGoalId, target, fromDate, toDate, userid, catCode, weekNo);
logger.debug("Record no " + i + " persisted");
persistRecordStatus.put("SUCCESS_RECORD", String.valueOf(successRecordCount++));
/*
* } catch (Exception e) { persistRecordStatus.put("FAILED_RECORD",
* String.valueOf(failedRecordCount++)); }
*/
}
} else {
persistRecordStatus.put("FAILED_RECORD",String.valueOf(failedRecordCount++));
}
/*
* } catch (Exception e) { logger.debug("Exception in processing record no " + i
* + " " + e.toString()); persistRecordStatus.put("FAILED_RECORD",
* String.valueOf(failedRecordCount++)); }
*/
}
return persistRecordStatus;
}
You are doing some kind of batch updates ?
In batch processing, you treat "failures" by putting all stuff involved (e.g. the input record, an error code, ...) on an error store for subsequent review by some user. And you commit that along with all the "successful" updates.
Alternatively, put each insert/update pair in its own transaction. (and still alternatively, if your DBMS / transactional framework supports savepoints, take a savepoint after each insert/update pair and upon failure, rollback to the last savepoint and then commit (and figure out a way to not lose the remaining 40 unprocessed entries from your input). In either case, don't come complaining if you get performance trouble.
There is no having things both ways in transactional batch processing.

Invalid Object Name, Temporary Table Entity Framework

I´m trying to optimize some process in my application but I´m stuck with this problem. My application is working so the entity mapping is correct. Simplifying what I´m trying to do is this:
using (var offCtx = new CheckinOfflineEntities())
{
using (var trans = offCtx.Database.BeginTransaction(IsolationLevel.Snapshot))
{
DateTime purgePivot = DateTime.Now.AddDays(-2);
count = offCtx.Database.ExecuteSqlCommand(#"select L.* into #NewLegs from InventoryLeg L where L.STDUTC >= {0}", purgePivot);
long d = offCtx.Database.SqlQuery<long>("select count(*) from #NewLegs").FirstOrDefault();
}
}
I´m selecting some data I want to delete from one table, storing it in a temporary table so that I can use this temporary table in other queries to exclude related data.
The problem is, when I try to use the temporary table I´m receiving the exception SqlException: "Invalid object name '#NewLegs'."
Thank you for your time.
You can merge the query like this.
And count returns int, not long.
COUNT always returns an int data type value. - MSDN
var query = string.Format("{0} {1}",
#"select L.* into #NewLegs from InventoryLeg L where L.STDUTC >= #STDUTC",
#"select count(*) from #NewLegs")
var d = offCtx.Database.SqlQuery<int>(query, new SqlParameter("STDUTC", purgePivot))
.FirstOrDefault();
I realy don´t know why but removing the parameter and adding it in the query solved the problem.
The code below works fine:
using (var offCtx = new CheckinOfflineEntities())
{
using (var trans = offCtx.Database.BeginTransaction(IsolationLevel.Snapshot))
{
DateTime purgePivot = DateTime.Now.AddDays(-2);
count = offCtx.Database.ExecuteSqlCommand(#"select L.* into #NewLegs from InventoryLeg L where L.STDUTC >= " + purgePivot.toString("yyyy-mm-dd HH:mm:ss");
long d = offCtx.Database.SqlQuery<long>("select count(*) from #NewLegs").FirstOrDefault();
}
}

Resources