I am running a batch to update opportunity records. I see that the query fetches around 1.1 million records but it processes only approx 100k records. I have checked the query in query editor and its working fine. Even when I process these records separately using the same code which batch is using to process its working as expected. Not sure why batch is not processing.
global class BatchAssignment implements Database.Batchable<sobject>{
global Database.Querylocator start (Database.BatchableContext BC){
return Database.getQueryLocator('SELECT id,Booking_Domain__c,Region__c,Primary_LOB__c,NA_Contract_Sales_Type__c,Owner_Sales_Group__c,Connected_Technologies_Opportunity__c FROM Opportunity where CreatedDate > LAST_N_YEARS:3 ORDER BY CreatedDate DESC');
}
global void execute (Database.BatchableContext BC, List<Opportunity> oppList) {
try{
if(!oppList.isEmpty()){
CustomBSNAReportingHandler.updateOpportunityBSNAReporting(oppList);
}
if(test.isRunningTest()) { throw new DMLException();}
}catch(Exception ex){
}
}
global void finish(Database.BatchableContext BC){
// Peform post transaction update
}
}
Is there any restriction with batch? Why its behaving like this
Is the batch job still running? Check Setup -> Jobs to see if it finished.
You have empty try-catch that swallows any exceptions. Remove it and see what kind of errors you'll get.
Yes, it will mean that if there's 1 bad Opportunity - other 199 in same execute() call will fail to update. You can mitigate that by setting smaller chunk size (Database.executeBatch(new BatchAssignment(), 10)), clever use of database.update(myrecords, false) inside that method you're calling.
Maybe even look into https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_batch_platformevents.htm for some error raising (and handling it with some monitoring software? trigger that would convert them to Tasks related to problematic opportunities?)
Related
I have a scheduled job that runs every night at midnight. It updates around 7000 records without entering any data into the records directly. About 150 records don't update as expected.
If I simply call update (or Database.update()) on these records as individual records or in a list, in an anonymous Apex block, they update as expected. When I call the batch that the scheduler calls in anonymous Apex they don't work.
The Batchable class (as well as the Schedulable class) literally has the smallest amount of code you could imagine:
global Database.QueryLocator start(Database.BatchableContext BC){
return Database.getQueryLocator([SELECT Id, Name FROM XXX ]);
}
global void execute(Database.BatchableContext BC, List<XXX> scopePolicy){
Database.update(scopePolicy, false);
}
global void finish(Database.BatchableContext BC){
}
Scheduler:
global void execute(SchedulableContext SC) {
try{
XXX xxx = new XXX();
database.executebatch(xxx, 15);
}
catch (DmlException e) {}
}
Does Batchable use a separate API?
Is it possible for the Scheduler or Batchable classes to create settings that are slightly different than if the records are updated directly?
The User that runs the Scheduled job is the same User that I have been debugging with.
(too long for a comment)
After nightly run does LastModifiedDate change on these 150? Maybe they update OK but something comes and resets them to original value (maybe you have some parent-child relations on this object and there's some cascade/rollup at play).
What field do you expect to see updated, can you put field history tracking on it. With any luck you'll see it changing state A to B and something setting it back to A.
Does the batch implements Database.Stateful?
If you run it with raw update statement, not the Database version - do you see any errors in Setup -> Apex Jobs
Is the amount of executions roughly ok? 7000 / 15 = 467 batches processed or are you ~10 batches short?
in my spring-boot application I am running a schedule job which is taking the record from one table and saving it another as archive, I just ran into a problem so now the records I have selected to be save are almost around 400,000 plus and when the job is executing it just keeps on going. Can the ja handle this much data to be saved at once? Below are my configuration and the method.
#--Additional Settings for Batch Processing --#
spring.jpa.properties.hibernate.jdbc.batch_size=5000
spring.jpa.properties.hibernate.order_inserts=true
spring.jpa.properties.hibernate.generate_statistics=true
spring.jpa.properties.hibernate.jdbc.batch_versioned_data = true
rewriteBatchedStatements=true
cachePrepStmts=true
The agentAuditTrailArchiveList object has size of 400,000 plus.
private void archiveAgentAuditTrail(int archiveTimePeriod) {
List<AgentAuditTrail> archivableAgentAuditTrails = agentAuditTrailRepository
.fecthArchivableData(LocalDateTime.now().minusDays(archiveTimePeriod));
List<AgentAuditTrailArchive> agentAuditTrailArchiveList = new ArrayList<>();
archivableAgentAuditTrails
.forEach(agentAuditTrail -> agentAuditTrailArchiveMapper(agentAuditTrailArchiveList, agentAuditTrail));
System.out.println(" agentAuditTrailArchiveList = " + agentAuditTrailArchiveList.size());
System.out.println("archivableAgentAuditTrails = " + archivableAgentAuditTrails.size());
agentAuditTrailArchiveRepository.saveAll(agentAuditTrailArchiveList);
agentAuditTrailRepository.deleteInBatch(archivableAgentAuditTrails);
}
Assuming you are actually doing this as a Spring Batch job this is already handled. You just need to set the chunk size within the batch job step config so it will only process a set number of records at a time. See the Batch step config example below.
You never want to try to do the entire job as one commit unless it is always going to be really small. Doing it all in one commit leaves you open to a laudatory list of failure conditions and data integrity problems.
#Bean
public Step someStepInMyProcess(FlatFileItemReader<MyDataObject> reader,
MyProcesssor processor, ItemWriter<MyDataObject> writer) {
return stepBuilders.get("SomeStepName")
.<MyDataObject, MyDataObject>chunk(50)
.reader( reader )
.processor( processor )
.writer(writer)
.transactionManager(transactionManager)
.build();
}
I execute the following in the anonymous execution:
Database.executeBatch(new TransferACDCAccountOppOwnerBatch(), 200);
The class is below. It uses Batchable interface. I put in a bunch of debug statements but don't see anything in the logs in the console. I also create debug logs at Debug level for Apex, etc. and also don't see any of the system.debug output. The SOQL itself I know works and should only return one row in the developer sandbox test data I setup.
Is there something about the batchable interface that doesn't allow system.debug output? I know it's asynchronous, but the job completes and I see plenty of log information. I just don't see any system.debug output.
global class TransferACDCAccountOppOwnerBatch implements Database.Batchable<sObject> {
String query;
static String type = 'Accountant';
static String stageName = 'Closed Won';
static String numberEmployees = '<5';
global TransferACDCAccountOppOwnerBatch() {
query = 'SELECT Id, Num_Active_Orgs__c, Num_Active_Paying_Orgs__c,Number_of_Bookkeeping_Clients__c, Number_Bookkeeping_Clients_SR__c,' +
'Num_Targeted_Orgs__c, AccountId, Account_State__c, Biz_Dev_Owner__c,CloseDate, IsClosed, Name, Type, OPS_ID__c, Org_Creator__c,' +
'Org_s_Geographical_State__c, OwnerId, StageName, Tier__c, IsWon,First_targeted_Org__r.NumberEmployees__c, Account.name, Account.owner.name' +
' FROM Opportunity' +
' WHERE StageName = :stageName' +
' And type = :type' +
' And CloseDate < Last_90_Days' +
' And First_targeted_Org__r.NumberEmployees__c = :numberEmployees';
}
global Database.QueryLocator start(Database.BatchableContext BC) {
System.debug('start query');
return Database.getQueryLocator(query);
}
global void execute(Database.BatchableContext BC, List<Account> scope) {
System.debug('execute batch transfer');
TransferACDCAccountOppOwnerHandler hierarchy_handler = new TransferACDCAccountOppOwnerHandler();
hierarchy_handler.setup(scope, BC.getJobId());
System.debug('after hierarchy handler setup');
// hierarchy_handler.runMatching();
// hierarchy_handler.processConsoles();
// hierarchy_handler.processGlobalConsoles();
// hierarchy_handler.commitUpdates();
}
global void finish(Database.BatchableContext BC) {
System.debug('finish bath');
}
}
Debug logs are limited to 2MB if they go over that size salesforce unhelpfully remove debug lines in an attempt to reduce the size to below the limit.
If you have a particularly noisy class you can override the logging for that class.
Check out the documentation here: https://help.salesforce.com/articleView?id=code_setting_debug_log_levels.htm&type=0&language=en_US&release=206.20
A quick and easy hack that I sometimes use would be to set your debog levels to ERROR, and then every time you want to debug you just do
System.debug(LoggingLevel.ERROR, 'Debug message goes here');
Just don't forget to clear your debug lines when you're done, so as not to make the problem worse going forward.
Turns out that in the Execute method I was passing List instead of List while the query was against Opportunity.
Also, seems like setting LoggingLevel.INFO helps at least with getting debug messages sent to developer console when running this batchable class in anonymous execution.
Thanks for the tips!
System.dubug is supported in batch apex.
As batch apex runs asynchronously, so it creates multiple logs based on your batch size. So, look at every log file that is created when batch apex completed its execution.
I was importing ttl ontologies to dbpedia following the blog post http://michaelbloggs.blogspot.de/2013/05/importing-ttl-turtle-ontologies-in-neo4j.html. The post uses BatchInserters to speed up the task. It mentions
Batch insertion is not transactional. If something goes wrong and you don't shutDown() your database properly, the database becomes inconsistent.
I had to interrupt one of the batch insertion tasks as it was taking time much longer than expected which left my database in an inconsistence state. I get the following message:
db_name store is not cleanly shut down
How can I recover my database from this state? Also, for future purposes is there a way for committing after importing every file so that reverting back to the last state would be trivial. I thought of git, but I am not sure if it would help for a binary file like index.db.
There are some cases where you cannot recover from unclean shutdowns when using the batch inserter api, please note that its package name org.neo4j.unsafe.batchinsert contains the word unsafe for a reason. The intention for batch inserter is to operate as fast as possible.
If you want to guarantee a clean shutdown you should use a try finally:
BatchInserter batch = BatchInserters.inserter(<dir>);
try {
} finally {
batch.shutdown();
}
Another alternative for special cases is registering a JVM shutdown hook. See the following snippet as an example:
BatchInserter batch = BatchInserters.inserter(<dir>);
// do some operations potentially throwing exceptions
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
batch.shutdown();
}
});
I have a similar code below that I'm trying to figure out transaction locking:
DAOT.repeatInTransaction(new Transactable() {
#Override
public void run(DAOT daot)
{
Points points = daot.ofy().find(Points.class, POINTS_ID);
// do something with points
takes_a_very_long_time_delay(); // perhaps 10 secs
daot.ofy().put(points);
}
});
The code above is executed from within a Java servlet. The operation is expected to work for 10 seconds for example. In between that time, I have a test that will invoke another servlet that will delete a Points entity, I was expecting that the delete operation would fail or at least delete the entity after the transaction above has finished.
However the entity was deleted during the period that the above code is executing. In my real application, I added exception handling to throw exception when trying to access or edit a entity that does not exist.
From there, the application is throwing "Entity not found" exception just after I executed the servlet that will delete the Entity in the code above.
Although I am using GAE Transactions already, however I think I am still missing something that's why my test fails.
Code for the delete Transaction from withing the Delete servlet:
DAOT.repeatInTransaction(new Transactable() {
#Override
public void run(DAOT daot)
{
Points points = daot.ofy().find(Points.class, POINTS_ID);
daot.ofy().delete(points);
}
});
How can I ensure that a new operation like a delete for a entity will wait until the current operation is happening on a entity during a transaction?
App Engine uses optimistic concurrency, not locking. That is, a transaction on a group of entities will not prevent other processes from modifying those entities while the transaction runs. Instead, when the transaction attempts to commit, it will check if any modifications were made while the transaction was executing, and if it has, discard any changes and run your function again from the beginning.
I assume you use objectify to work with datastore.
First you need to make sure daot.ofy() returns objectify instance with explicit transaction set (ObjectifyFactory.beginTransaction()) instead of ObjectifyFactory.begin(). Then make sure you use the same objectify instance for both find() and delete() calls (as well as for find()/put pairs).