Improve performance finding matches in a big list/array - arrays

Question

I am building a tool to find images in a big folder of images (400k images). On that folder I have images like this:
c:\images\100001_01.jpg
c:\images\100001_05.jpg
c:\images\100001_07.jpg
c:\images\100005_05.jpg
c:\images\100010_00.jpg
Then I have my references in a text box, but only the 6 digit number:
100001
100005
100006
Etc
So I have let's say 1000 references I need to have the images for, I want to loop through all the image folder and take the file if exists. I have built this using both an array with loops and a list and getting the index. I thought getting the index of the list would be much faster, but they are actually the same.
Here are the two routines I have developed, one is using a list and then getting FindIndex to get the index. The second option is looping through all the references and at the same time looping through all images to check if any contains that reference - that is 400 million loops if I use a set of 1000 references!
Using a list takes 69 seconds, however looping through the arrays takes 64 seconds. Nevertheless taking all images on the directory using GetFile takes 120 seconds already.
Can you think of any way to make this faster?
Private Sub ExtractImagesUsingList()
Dim ListOfReferences As New List(Of String) 'the actual list of references is in a textbox, ie.: 100001, 100002, etc
For Each line In txtBox.Lines
ListOfReferences.Add(line.ToString)
Next
Dim ListOfimages As New List(Of String)
For Each file In IO.Directory.GetFiles("c:\images\")
ListOfimages.Add(file)
Next
For Each ref In ListOfReferences
Dim index As Integer = ListOfimages.FindIndex(Function(x As String) x.Contains(ref))
Next
End Sub
Private Sub ExtractImagesUsingArrayLoop()
Dim ListOfreferences As New List(Of String)'the actual list of references is in a textbox
For Each line In txtBox.Lines
If line.Length > 1 Then
ListOfReferences.Add(line.ToString)
End If
Next
Dim ArrayImages() As String = IO.Directory.GetFiles("c:\images\")
For Each reference In ListOfReferences
For Each image In ArrayImages
If image.Contains(reference) Then
Exit For ' I exist the FOR here because I am only interested in one image per reference
End If
Next
Next
End Sub

Answer 1

It's not 100% clear what you're trying to accomplish, but you could improve performance by implementing Directory.EnumerateFiles.
Additionally, you could pair this with the built-in search function using a wildcard match against the known reference key from the reference list.
Finally, if the image directory is not subject to frequent changes, you could cache the images in a dictionary to speed up future searches.
Here is a rough example of those ideas. Note I've removed any references to form controls, and assume parameters are instead being passed in.
Private _imageMap As Dictionary(Of String, ICollection(Of String))
Public ReadOnly Property ImageMap As Dictionary(Of String, ICollection(Of String))
Get
If _imageMap Is Nothing Then
_imageMap = New Dictionary(Of String, ICollection(Of String))()
End If
Return _imageMap
End Get
End Property
Public Sub RefreshImageMap()
_imageMap = Nothing
End Sub
Public Function GetImagePaths(imageFolder As String, referenceKey As String) As ICollection(Of String)
Dim imagePaths As ICollection(Of String) = Nothing
If Not ImageMap.TryGetValue(referenceKey, imagePaths) Then
imagePaths = Directory.EnumerateFiles(imageFolder, $"{referenceKey}_*.jpg").ToList()
ImageMap.Add(referenceKey, imagePaths)
End If
Return imagePaths
End Function
Also if you wanted to run multiple reference keys through a function, but also needed to keep the original reference key passed in, you could add something like this:
Public Iterator Function GetImagePaths(imageFolder As String, referenceKeys As IEnumerable(Of String)) As IEnumerable(Of KeyValuePair(Of String, ICollection(Of String)))
For Each referenceKey As String In referenceKeys
Dim imagePaths = GetImagePaths(imageFolder, referenceKey)
Yield New KeyValuePair(Of String, ICollection(Of String))(referenceKey, imagePaths)
Next
End Function
None of this is tested, and this lacks proper parameter checks on functions, but it should give you a direction to try.