Rails ActiveRecord - Performance - How to destroy invalid belongs_to records - query-optimization

Question

I have two models:
Asset has_many Assethistories
Assethistory belongs_to Asset
Unfortunately, when the migration was created, no foreign_key was added to Assethistories. Some Assethistories records exist where Assethistories.asset_id has a value that does not exist in Asset.id (probably someone used delete on the Asset table instead of destroy).
There are approximately 10 million Asset records and 25 million Assethistories records.
Using this query takes a VERY LONG time:
Assethistory.where("asset_id NOT IN (select id from assets)").delete_all
or in rails syntax:
Assethistory.where.not(asset_id: Asset.select(:id)).delete_all
NOTE: we can use delete_all since there are no callbacks or nested models.
In fact, just doing a COUNT of the invalid records takes VERY LONG.
Assethistory.where.not(asset_id: Asset.select(:id)).count
Is there any way to destroy the invalid records that would be more performant?

Answer 1

I've got some data which is about 20x smaller than your case, e.g 5M assethistory and 500K asset like your case, which could reproduce your issue by using where.not. By following Anti-Join pattern, where joins records that don't exist, would bring you the result you want with better performance.
Assethistory.left_outer_joins(:asset).where(assets: { id: nil })
# => 3500 rows (415.3ms)
or
Assethistory.where('NOT EXISTS (:assets)', assets: Asset.select('1').where('assets.id = assethistories.asset_id'))
# => 3500 rows (701.6ms)
By using LEFT OUTER JOIN instead of WHERE NOT, DB scans the table in a very different way. WHERE NOT loops over all rows in assethistories where assethistories.asset_id do not match all existing assets.id.
LEFT OUTER JOIN queries the records from the joined-table(assethistories and assets as a big one table) where the rows have null value in asset.id column, which is much more efficient.