I'm writing an Apex program that reads through a database and processes record. Each time I process a record, I want to output a message. Currently I'm using System.Debug to do this, but the debug log is cluttered with so much that this doesn't seem like the right approach.
What other ways can I generate screen or logfile output in SalesForce?
Keep using System.Debug() but when you want to view only your output messages, just filter by DEBUG. Otherwise the only other option is to create a view and then that is more clutter than what it's worth.
Please Open the Log in Raw format under Setup>> Administration Setup >> Monitoring >>Debug Logs. Under Monitored Users go to Filters and enable all the filter levels. Now use apex code as given
System.debug('StackOverflow >>1234'+ e.getMessage)
and search the detailed debug logs for StackOverflow >>1234 the unique message. It may also happen that your system.debug might not have been executed in that specific Debug logs so do not forget to check all the recent debug logs. :)
You could think about creating your own Logging__c object. And create a record for it for each record processed. You have to be creative to work round the governor limits though.
If it's not essential that you output the message in between processing each record, then you could build up a collection of Logging__c records as processing continues and then either insert them periodically, or when there's an exception in your process.
Note that if inserting them periodically, you still have to make sure the jobs not so large that you're going to hit the DML limit of 150 together with the processing work you're doing. Also, if storing the records to all be inserted at the end of processing, bear in mind the heap size is 6MB.
Alternatively, have a look at Batch Apex http://www.salesforce.com/us/developer/docs/apexcode/Content/apex_batch_interface.htm
This allows you to create a class to handle processing a job in asynchronous chunks. You can set the number of records processed in one go. So you could set this small (~20) and then insert a Logging_c record as each job record is processed to stay within the Batch Apex DML limit of 200. You can then monitor Logging_c records in real time to view progress.
Related
Imagine that we have a file, and some job that processes it and sends the data:
into the database
to an external service
Can we guarantee to process the file only once or at least to determine that something went wrong and notify the user so that he manually solved this problem?
Yes, you can.
What you can do is create a table in the database to store the name and a flag/status (if read, yes else no) of files. When process feeds the file in that location, make sure that the same process updates the name (if name is different each time) and flag/status for that file in the database. Your file read process can get the name of file from the database and dump that file in wherever you ant and when it's done, It should update the flag to read or whatever. This way, you can avoid reading the file more than one time.
I would store two tables of information in your database.
The processed file lines like you were already doing.
A record of the files themselves. Include:
the filename
whether the processing was successful, failed, partially succeeded
a SHA1 hashed checksum that can be used to check for the uniqueness of the file later
When you go to process a file, you first check whether the checksum already exists. If it does, you can stop processing and log the issue. Or you can throw that information on the file table.
Also be sure to have a foreign key association between your processed lines and your files. That way if something does go wrong, the person doing manual intervention can trace the affected lines.
Neither Usmana or Tracy answer actually guarantees that a file is not processed more than once and your job doesn't send duplicate requests to the database and the external service(#1 and #2 in your question). Both solutions suggest keeping a log and update it after all the processing is done but if an error occurs when you try to update the log at the very end, your job will try processing the file again next time it runs and will send duplicate requests to the database and external service. The only way to deal with it using the solutions Usmana and Tracy suggested is to run everything in a transaction but it's quite a challenging task in a distributing environment like yours.
A common solution to your problem is to gracefully handle duplicate requests to the database and external services. The actual implementation can vary but for example you can add a unique constraint to the database and when the job tries to insert a duplicate record an exception will be thrown which you can just ignore in the job because it means the required data is already in the db.
My answer don't mean that you don't need the log table Usmana and Tracy suggested. You do need it to keep track of processing status but it doesn't really guarantee there won't be duplicate requests to your database and external service unless you use a distributed transaction.
Hope it helps!
I need do a data migration from a legacy system, and there are 20,000+ records totally[all the data will be provided by csv file], due to some technical reasons, I must use JPA/Hibernate to import those data into your new system currently used.
When I do the importing, always encountered a transaction problem like following:
The database is MS SQL Server 2005
org.hibernate.AssertionFailure: null id in xxxx entry (don't flush the Session after an exception occurs)
And I tried following things:
Using a nested exception to importing the separate record.
Split the data to some pieces small csv file
Call entityManager.flush() manually and add Thread.sleep(10000) to sleep 10ms
Call titityManager.flush every 20 records and sleep 5ms
But unfortunately, nothing seems to help. Please help.
Thanks for reading. Any help would be much appreciated!
If mappings are correct, probably is a wrong import code like described in org.hibernate.AssertionFailure.
Also you can look about Hibernate batch processing best pratices and Stateless session.
Last option can be move to spring-batch
, Iam working on this project in hibernate, where the user enters the quantity based on this number i need to display the unique fields in the next form (For example if the user enters quantity as 5 then 5 MAC Address and DeviceSerial fields are shown in the jsp page, the other common information about the devices are entered in the same form where quantity is entered) and all this information should be put into the database . Thanks!
Please take a look at this page: webpage
You should open a transaction, save all entities (in loop), at the end commit the transaction and close the session. You should not commit for every single record but at the end.
Referenced to Goran
or
if you can use hibernateTemplate instead of session then you can save your object list by using saveOrUpdateAll(Collection entities).
Although HibernateTemplate saveOrUpdateAll does save a collection at once but, it will use the same loop logic of saving the object and flushing
You should go for Batch Insert in hibernate.
Batch Insertion is a powerful feature of hibernate particularly useful when you are importing data from other systems in batch. If you do not use batch feature of hibernate, your application's performance may decrease dramatically at the time of insertion of many records.
For more information visit hibernate tutorials.
I have a file stored on disk that can be access across multiple servers in a web farm. This file is updated as necessary based on data changes in the database. I have a database table that stores a row with a URI for this file and some hashes based off of some database tables. If the hashes don't match their respective tables, then the file need to be regenerated and a new row needs to be inserted.
How do I make it so that only 1 client regenerates this file and inserts a row?
The easiest but worst solution (because of locks) is to:
BEGIN TRANSACTION
SELECT ROW FROM TABLE (lock the table for the remainder of the transaction)
IF ROW IS OUT OF DATE:
REGENERATE FILE
INSERT ROW INTO TABLE
DO SOME STUFF WITH FILE (30s)
COMMIT TRANSACTION
However, if multiple clients execute this code, all of the subsequent clients sit for a long time while the "DO SOME STUFF WITH FILE" processes.
Is there a better way to handle this? Maybe changing the way I process the file before the commit to make it faster? I've been stumped on this for a couple days.
It sounds like you need to do your file processing asynchronously, so the file process is spun off and the transaction completes in a timely manner. There are a few ways to do that, but the easiest might be to replace the "do stuff with file" with a "insert a record into the table This_File_Needs_To_Be_Updated, then run a job every few minutes that updates each record in that table. Or HERE is some code that generates a job on the fly. Or see THIS question on Stack Overflow.
The answer depends on the details of file level processing.
If you just swap the database and file operations, you risk corruption of the file or busy waiting (depending on how exactly you open it, and what your code does when a concurrent open is rejected). Busy waiting would definitely be worse than waiting on a database lock from a throughput (or any other) perspective.
If your file processing really takes so long as to be frequently causing queueing of requests, the only solutions are to add more powerful hardware or optimize file level processing.
For example, if the file only reflects the data in the database, you might get away with not updating it at all, and having a background process that periodically regenerates its content based on the data in the database. You might need to add versioning that makes sure that whoever reads the file is not receiving stale data. If the file pointed to by the URL has a new name every time, you might need an error handler that makes sure that GET requests are not habitually receiving a 404 response on new files.
I am running a bunch of database migration scripts. I find myself with a rather pressing problem, that business is waking up and expected to see their data, and their data has not finished migrating. I also took the applications offline and they really need to be started back up. In reality "the business" is a number of companies, and therefore I have a number of scripts running SPs in one query window like so:
EXEC [dbo].[MigrateCompanyById] 34
GO
EXEC [dbo].[MigrateCompanyById] 75
GO
EXEC [dbo].[MigrateCompanyById] 12
GO
EXEC [dbo].[MigrateCompanyById] 66
GO
Each SP calls a large number of other sub SPs to migrate all of the data required. I am considering cancelling the query, but I'm not sure at what point the execution will be cancelled. If it cancels nicely at the next GO then I'll be happy. If it cancels mid way through one of the company migrations, then I'm screwed.
If I cannot cancel, could I ALTER the MigrateCompanyById SP and comment all the sub SP calls out? Would that also prevent the next one from running, whilst completing the one that is currently running?
Any thoughts?
One way to acheive a controlled cancellation is to add a table containing a cancel flag. You can set this flag when you want to cancel exceution and your SP's can check this at regular intervals and stop executing if appropriate.
I was forced to cancel the script anyway.
When doing so, I noted that it cancels after the current executing statement, regardless of where it is in the SP execution chain.
Are you bracketing the code within each migration stored proc with transaction handling (BEGIN, COMMIT, etc.)? That would enable you to roll back the changes relatively easily depending on what you're doing within the procs.
One solution I've seen, you have a table with a single record having a bit value of 0 or 1, if that record is 0, your production application disallows access by the user population, enabling you to do whatever you need to and then set that flag to 1 after your task is complete to enable production to continue. This might not be practical given your environment, but can give you assurance that no users will be messing with your data through your app until you decide that it's ready to be messed with.
you can use this method to report execution progress of your script.
the way you have it now is every sproc is it's own transaction. so if you cancel the script you will get it update only partly up to the point of the last successfuly executed sproc.
you cna however put it all in a singel transaction if you need all or nothign update.