Correct usage of ForEach -parallel in Powershell? - arrays

Question

I want to process a large number of URLs and grab the *.jpg file locations.
The problem is that the $entry in the second foreach is not threadsafe.
The script is firing hundreds of errors because the $entry is getting overwritten over and over.
When I move the inner foreach outside of the ForEach-Object, then its working fine but very slowly.
How can I process the split output properly within my ForEach-Object without getting these errors?
$array just contains a huge amount of URLs
$clean_img_array is the output array of the operation
$tmpArray is the reference to $clean_img_array in order to use it within a parallel ForEach
Errors:
InvalidOperation:
Line |
14 | [void]$tmpArray:clean_img_array.Add($entry);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| You cannot call a method on a null-valued expression.
Snippet:
$clean_img_array = [System.Collections.ArrayList]#();
$array | ForEach-Object -Parallel {
$web = Invoke-RestMethod $_;
$i=1;
foreach($entry in $web.Split("`"")){
echo $entry;
if($entry.IndexOf(".jpg") -ne -1 -And $entry.IndexOf("http") -ne -1){
if($entry.IndexOf("?") -ne -1){
$tmpArray = $using:clean_img_array;
[void]$tmpArray.Add($entry.Substring(0, $entry.IndexOf('?')));
}else{
$tmpArray = $using:Clean_img_array;
[void]$tmpArray:clean_img_array.Add($entry);
}
}
}
} -ThrottleLimit 20

Answer 1

Here's a simple example. Both $a and $b are arrays. $b is the result of the parallel loop. It's like example 12 in the docs.
$a = 1..10
$b = $a | foreach-object -parallel { $_ + 1 }
$b
2
3
4
5
6
7
8
9
10
11

Answer 2

Thanks for the support!
I combined the answer from #js2012 with something of my own.
The return alone did not solve the thread unsave behavior of $entry, but clearly untangled the workflow.
But I used the inline version of .foreach with the pipe line variable $_ wich happens
to be thread safe as it appears to me. Running now like a charm and also very fast for more than 2million entrys.
$array holds the URLs to be processed
$clean_img_array returns the grabbed image URLs
$clean_img_array = $array | ForEach-Object -Parallel {
$web = Invoke-RestMethod $_;
$web.Split("`"").foreach({
if($_ -ne $null){
if($_.IndexOf(".jpg") -ne -1 -And $_.IndexOf("http") -ne -1){
if($_.IndexOf("?") -ne -1){
return $_.Substring(0, $entry.IndexOf('?'));
}else{
return $_;
}
}
}
});
} -ThrottleLimit 25