Fast way to count duplicates in 30000 rows (Libreoffice Calc) - arrays

Question

Actually, I already have a partial answer!!! Conditional formatting with "Cell value is" -> "duplicate" !!!
This way a check is performed for each user's new entry in "real time".
I need to check if duplicate entries exist in 30000 rows of a column (any value, but not blanks!) . I would like to keep track of how many duplicates during the filling process.
Ok, conditional formatting is a very effective visual indication and fast anough for my needs, but as I am not able to perform a loop to check the color of the cells (found some people against this approach!! Would be so easy! ) I need to find an alternative way to count the duplicates (as a whole, no need to identify how many for each case!).
I tryed the formula:
=SUMPRODUCT((COUNTIF(F2:F30001;$F$2:$F$30001)>1))
It works, but it takes two minutes to finish.
If you want to replicate my case. My 30000 entries are formatted as: letter "A" and numbers between 100000 and 999999, e.g., A354125, A214547, etc. Copy as text the result of "=CONCATENATE("A";RANDBETWEEN(100000;999999))" to save time.
Thanks!
PS: Does anybody know the algorithm used to find the duplicates in conditional formatting (it is fast)?
A macro solution is not the best, but is acceptable! ;)

Answer 1

The =SUMPRODUCT((COUNTIF(F2:F30001;$F$2:$F$30001)>1)) must do following: Count if $F$2 is in F2:F30001, then count if $F$3 is in F2:F30001, ..., then count if $F$30001 is in F2:F30001. So it must fully loop over the array F2:F30001 with each single item.
The fastest way counting duplicates in an array is avoiding fully loop over the array with each single item. One way is sorting first. There are very fast quick sort methods. Or using collections which per definition can only have unique items.
The following code uses the second way. The keys of a Collection must be unique. Adding an item having a duplicate key fails.
Public Function countDuplicates(vArray As Variant, Optional inclusive As Boolean ) As Variant
On Error Goto wrong
If IsMissing(inclusive) Then inclusive = False
oDuplicatesCollection = new Collection
oUniqueCollection = new Collection
lCountAll = 0
For Each vValue In vArray
If contains(oUniqueCollection, CStr(vValue)) Then
On Error Resume Next
oDuplicatesCollection.Add 42, CStr(vValue)
On Error Goto 0
Else
oUniqueCollection.Add 42, CStr(vValue)
End If
lCountAll = lCountAll + 1
Next
countDuplicates = lCountAll - oUniqueCollection.Count + IIF(inclusive, oDuplicatesCollection.Count, 0)
Exit Function
wrong:
'xray vArray
countDuplicates = CVErr(123)
End Function
Function contains(oCollection As Collection, sKey As String)
On Error Goto notContains
oCollection.Item(sKey)
contains = True
Exit Function
notContains:
contains = False
End Function
The function can be called:
=COUNTDUPLICATES(F2:F30001, TRUE())
This should return the same result as your
=SUMPRODUCT((COUNTIF(F2:F30001,$F$2:$F$30001)>1))
The optional second parameter inclusive means the count includes all the values which are present multiple times. For example {A1, A2, A2, A2, A3} contains 3 times A2. Counting inclusive means the count result will be 3. Counting not inclusive means the count result will be 2. There is 2 times A2 as a duplicate.
As you see, the function contains much more information than only the count of the duplicates. The oDuplicatesCollection contains each duplicate item. The oUniqueCollection contains each unique item. So this code could also be used for getting all unique items or all duplicate items.