Excel transpose 2D array - Out of memory issue - arrays

I have a VBA macro, where I want to write out an array to an Excel sheet.
I'm getting an "Out of memory" runtime error on some machines. I can run it easily on my development PC, but my client has issues with it.
Here I define my Values array:
Dim Values()
Dim idx As Long
idx = 0
Then I have a for cycle where I dynamically redim the array, and add my values to it:
for cycle...
ReDim Preserve Values(16, 0 To idx)
Values(0, idx) = "some text"
Values(1, idx) = "some other text"
....
Values(15, idx) = "last values for this row"
idx = idx + 1
next
Then here is where my code fails:
With ws
.Range(.Cells(1, 1), .Cells(1+ idx - 1, 16)).value = TransP(Values)
End With
Here's the TransP transposing function:
Public Function TransP(var As Variant) As Variant
Dim outP() As Variant, i As Long, j As Long
ReDim outP(LBound(var, 2) To UBound(var, 2), LBound(var, 1) To UBound(var, 1))
For i = LBound(outP) To UBound(outP)
For j = LBound(var) To UBound(var)
outP(i, j) = var(j, i)
Next
Next
TransP = outP
End Function
As I said, I can run the macro, and get something like 108770 rows. The same 108770 rows don't work on my clients PC.
I expect that the TransP function gives up on his PC, so should I split up the array into multiple smaller chunks, and write them 1 by 1?
Or my data model is not good?

You could also create a loop to write your output array row by row, it will take more time but you will most likely don't get out of memory error.
In the past when I've got out of memory issues with an arrays I just tried to perform actions using regular excel commands, in this case you could just copy range and than paste transposed values:
.PasteSpecial Paste:=xlPasteValues, Transpose:=True

Your method of appending elements to Values is inefficient because every time an element is added a new array is created and the values of the existing one copied to it. During this time twice the memory is in use, and if a large array is copied in this way multiple times in quick succession only heaven can know what demands are put on the RAM management.
The better way is to dimension the array larger than will be required (once), count the number of elements written to it and use Redim Preserve to reduce its size (once) when you are done.

I suspect this might be due to you not specifying the dimension in your TransP function. You are also defining your loops with both arrays. You've sized them identically (although with the dimensions switched) - Just use the same one to define your For loop
Public Function TransP(var As Variant) As Variant
Dim outP() As Variant, i As Long, j As Long
ReDim outP(LBound(var, 2) To UBound(var, 2), LBound(var, 1) To UBound(var, 1))
For i = LBound(outP, 1) To UBound(outP, 1)
For j = LBound(outP, 2) To UBound(outP, 2)
outP(i, j) = var(j, i)
Next
Next
TransP = outP
End Function

I don't think it's the TransP function, since that is already handling everything in a loop. I experienced the same sort of issue and there this error occured when I tried to transfer a large multidimensional array to a range.
My solution was to create a loop and do about a 1000 rows each time, but that depends on the clients pc I guess with how many rows you can do.
To take into account that var will not end on a certain step size, you could do a while loop:
i = 0
with ws
Do while i < Ubound(var)
max = application.worksheetfunction.max(1000,Ubound(var) - i) 'At some point, you could have less than 1000 left
.range(.cells(i+1,1), .cells(i+max)) = TransP(var,i,max)
i = i + 1000
Loop
In TransP you now use the lower- and upperbound of var, but if you add two variables from the loop, you can use them to only take a piece of the array.

Related

Arrays in Excel VBA. At some point it puts NA instead of the value

I am trying to make a simple simulation in Excel VBA in which we roll two dices. What is the probability of getting "1"+"2" or "1"+"3"?
This is my code:
Sub ProbabilityMeyerArray()
Dim i As Long
Dim ArrayDices(1 To 100000, 1 To 2) As Variant
Dim ArrayResult(1 To 100000) As Variant
'Simulation
For i = 1 To 100000
ArrayDices(i, 1) = WorksheetFunction.RandBetween(1, 6)
ArrayDices(i, 2) = WorksheetFunction.RandBetween(1, 6)
If (ArrayDices(i, 1) = 1 And ArrayDices(i, 2) = 3) _
Or (ArrayDices(i, 1) = 1 And ArrayDices(i, 2) = 2) _
Or (ArrayDices(i, 1) = 3 And ArrayDices(i, 2) = 1) _
Or (ArrayDices(i, 1) = 2 And ArrayDices(i, 2) = 1) Then
ArrayResult(i) = 1
Else
ArrayResult(i) = 0
End If
Next i
'print the values to cells
Range("A1:B100000").Value = ArrayDices
Range("C1:C100000").Value = WorksheetFunction.Transpose(ArrayResult)
'Calculate the probability
Probability = Application.WorksheetFunction.Sum(ArrayResult) / 100000
MsgBox "The Probability is " & Probability
End Sub
The problem is that when I print the values from arrays to the cells, then in column C I have 0 and 1 (as it should be), but then from row 34465 I get NA. Here is a screenshot:
https://ibb.co/7jsjjJC
So, for some reason it starts putting NA instead of 0 and 1. The calculation does not work properly either, because the probability is too low, and I guess this is because it only counts the first 34464 zeros and ones, while dividing with 100 000. Can you help me understand what is wrong here? It seems to be a problem with (my understanding of) arrays, since I can run a similar simulation without arrays (by simply using cells), and it works.
Thanks in advance!
As #RaymondWu said in the comments, the problem is that the Transpose function has a limit to the length of the array it can manipulate. This limit is somewhere between 65k and 66k columns.
Indeed, your code will run as expected for 65k iterations.
You can easily avoid using transpose and to be honest I don't see the reason to use it in the first place.
Instead of declaring your array as Dim ArrayResult(1 To 100000) As Variant which by default makes it 1 row x 100000 columns, you can declare it as so:
Dim ArrayResult(1 To 100000, 1 To 1) As Variant
This will make it 100000 rows x 1 columns, which can now be written in a column in excel easily like so:
Range("C1:C100000").Value = ArrayResult
Of course you will also need to change the code accordingly where needed:
ArrayResult(i,1) = 1
Else
ArrayResult(i,1) = 0
A few other tips:
Always use Option Explicit at the very top of the code. It makes the declaration of variables mandatory and it helps to avoid mistakes
Always use explicit references. When referencing Ranges the best practice is to define the worksheet to which they belong e.g. ThisWorkbook.Worksheets("Sheet1").Range("C1:C100000").Value = ArrayResult

How to Pass an Array to and from a Function?

(Fair Warning, I am self taught on VBA so I apologize in advance for any cringe-worthy coding or notations.)
I have an estimating worksheet in excel. The worksheet will have a section for the user to input variables (which will be an array). The first input variable will "reset" the remaining input variables to a standard value when the first variable is changed. The standard values for the input variables are stored in a function in a module. I am attempting to fill the input variable array with the standard values from the function and then display those values on the sheet. I was easily able to do this without arrays but have had no luck in moving everything into arrays.
This is for excel 2010. I previously did not use arrays and created a new variable when needed, however the estimating sheet has grown much larger and it would be better to use arrays at this point. I have googled this question quite a bit, played around with removing and adding parenthesis, changing the type to Variant, trying to set the input variable array to be a variable that is an array (if that makes sense?), and briefly looked into ParamArray but that does not seem applicable here.
Dim BearingDim(1 To 9, 1 To 4, 1 To 8) As Range
Dim arrBearingGeneral(1 To 5, 1 To 8) As Range
Dim Test As Variant
Private Sub Worksheet_Change(ByVal Target As Range)
'Set General Variable array to cells on the worksheet
For i = 1 To 5
For j = 1 To 8
Set arrBearingGeneral(i, j) = Cells(9 + i, 3 + j)
Next j
Next i
'Set Bearing Input Variables to Cells on the Worksheet
For p = 1 To 4
For i = 1 To 9
Select Case p
Case Is = 1
Set BearingDim(i, p, 1) = Cells(16 + i, 4)
Case Is = 2
Set BearingDim(i, p, 1) = Cells(27 + i, 4)
Case Is = 3
Set BearingDim(i, p, 1) = Cells(37 + i, 4)
Case Is = 4
Set BearingDim(i, p, 1) = Cells(49 + i, 4)
End Select
Next i
Next p
'Autopopulate standard input variables based on Bearing Type
inputMD_StdRocker BearingType:=arrBearingGeneral(1, 1), _
arrBearingDim:=BearingDim
End Sub
Sub inputMD_StdRocker(ByVal BearingType As String, ByRef _
arrBearingDim() As Variant)
Dim arrBearingDim(1 To 9, 1 To 4)
Select Case BearingType
Case Is = "MF50-I"
For j = 1 To 2
arrBearingDim(2, j) = 20
arrBearingDim(3, j) = 9
arrBearingDim(4, j) = 1.75
Next j
arrBearingDim(5, 1) = 15
'There are numerous more select case, but those were removed to keep it
'short
End Select
End Sub
The expected output is my "BearingDim" Array will have certain array index values set to a standard value from the "inputMD_StdRocker" function. Then those values will be displayed in the cell that corresponds to the array index.
Currently, I get a compile error "Type Mismatch, Array or User-Defined Type Expected". I have been able to get around the type mismatch by removing the () from "arrBearingDim()" in the function title for "inputMD_StdRocker" however, it will not pass the values back to my "BearingDim" array.
Any help would be greatly appreciated.
This is a partial answer to what (I think) is a misunderstanding you have of how to use arrays. There are a few problems in your code.
First, you're defining a two-dimensional and a three-dimensional array of Ranges when I believe you really only want to store the values captured from the worksheet. (If I'm wrong, then you are never initializing the array of Ranges, so none of the ranges in the array actually point to anything.)
Secondly, it looks as if your initial array arrBearingGeneral is always filled from the same (static) area of the worksheet. If this is so (and you really do want the values from the cells, not an array of Range objects), then you can create a memory-based array (read this website, especially section 19). So the first part of your code can be reduced to
'--- create and populate a memory-based array
Dim bearingDataArea As Range
Dim arrBearingGeneral(1 To 5, 1 To 8) As Variant
Set bearingDataArea = ThisWorkbook.Sheets("Sheet1").Range("D10:K14")
arrBearingGeneral = bearingDataArea.Value
Optionally of course you can calculate the range of your data instead of hard-coding it ("D10:K14"), but this example follows your own example.
While this isn't a complete answer, hopefully it clears up an issue to get you farther down the road.

Excel VBA Join function

I try to join the elements of a vector in vba.
First I search with a for loop specific strings.
CE_addr = FindAll(Workbooks("raw_data.xlsm").Worksheets("IQ_Key_Dev_Type").Cells(k, (p - 1)).Value, _
Worksheets("Sheet1").Range("O:O"), xlValues, xlWhole).Address
Afterwards I use the address object to get the corresponding values one column to the right.
rn = Split(CE_addr, ",")
n = UBound(rn, 1) - LBound(rn, 1)
For w = 0 To n
ReDim CE_cod(n) As Variant
CE_cod((w)) = Workbooks("raw_data.xlsm").Worksheets("CE_List").Range(rn(w))
Next w
So in the CE_cod vector I save all the corresponding values (sometimes it's just one value, sometimes more than one)
And now I want to concatenate all the contained (text) values with ";" as delimiter:
Cells(j, 3) = Join(CE_cod, ";")
Let's assume CE_cod has two entries in the first loop: ["XXX" , "YYY"]
and just one in the second loop: ["XXX"]
For some reason the result in Cell(j,3) in the first case (multiple values in CE_cod) is: ";YYY" (note that a delimiter is added, but not the expression itself).
In the second case the result is: "XXX"
I checked if the vector CE_cod contains all values with:
Cells(j, (10 + w)) = CE_cod((w))
and indeed, when I do this "XXX" and "YYY" are contained.
So why does the join function skips the first entry of the vector when there are multiple entries?
PS: I found the "FindAll" function here:
http://www.tushar-mehta.com/publish_train/xl_vba_cases/1001%20range.find%20and%20findall.shtml#_The_FindAll_function
I think your CE_cod populating loop should be closer to this.
Dim CE_cod As Variant
ReDim CE_cod(LBound(rn) to UBound(rn))
For w = LBound(rn) to UBound(rn)
'if you want to put a redim here, use Preserve
'ReDim Preserve CE_cod(w)
CE_cod(w) = Workbooks("raw_data.xlsm").Worksheets("CE_List").Range(rn(w))
Next w
The rn array is populated from a Split operation and as such, it is a zero-based, 1-D array.
Additionally, you shouldn't be using ReDim inside a loop without the Preserve parameter if you want to retain your values. In any event, there is no need to ReDim inside the loop since you know the target boundaries beforehand.

ReDim existing array with a second dimension?

I declared an array in my VBA function that has a dynamic size. As I cannot ReDim the first dimension of a two- or more-dimensional array, can I add a second dimension to a set array?
This is how I dynamically set the size of my array.
Dim nameArray() As String
Dim arrayCount As Long
For i = 1 To 100
ReDim Preserve nameArray(1 to arrayCount)
nameArray(arrayCount) = "Hello World"
arrayCount = arrayCount + 1
Next i
Now I would like to add a second dimension.
ReDim Preserve nameArray(1 To arrayCount, 1 To 5)
doesn't work.
Is there a workaround?
There isn't any built-in way to do this. Just create a new two-dimensional array and transfer the contents of your existing one-dimensional array into the first row of that new array.
This is what this function does:
Function AddDimension(arr() As String, newLBound As Long, NewUBound As Long) As String()
Dim i As Long
Dim arrOut() As String
ReDim arrOut(LBound(arr) To UBound(arr), newLBound To NewUBound)
For i = LBound(arr) To UBound(arr)
arrOut(i, newLBound) = arr(i)
Next i
AddDimension = arrOut
End Function
Example usage:
nameArray = AddDimension(nameArray, 1, 5)
There is one (also works to delete a dimension), but you will have to think in terms of worksheet dimensions...
use transposition
While I highly prefer the previous 'by hand' method from Jean-François Corbett's and I don't like to rely on Excel build-in function (especially this one!), I would just like to clarify another way for future readers coming here:
adding a dimension to a 1d line vector (a row) means transposing it in Excel
Here, nameArray(1 to arrayCount) is a row (index is a column number) and because of it, if you add a dimension it will become a column since 2d arrays are indexed as (row,column). So, you can just do this:
nameArray = Application.Worksheetfunction.Transpose(nameArray) 'transforms the array to nameArray(1 To arrayCount, 1 To 1), so then:
redim preserve nameArray(1 To arrayCount, 1 To 5)
without any other manipulation.
BUT beware of the very confusing Excel's Transpose function (at least it is the case for me!): the advantage here is that it automatically adds a dimension and redimensions the array for you.
It works as expected only because you are using a 'based 1' index '1d array'.
IF this is not the case, all indices will be shifted by 1, (that's how Transpose is build in to be coherent with cells and ranges). That is: if you start with
nameArray(0 to arrayCount)
you will end up with
nameArray(1 to arrayCount + 1, 1 to 5)
with precautions
While I am at it, it may be off-topic and their are many topics about it, but their are other traps in this Transpose function one should never be enough warned about (not to mention it can consume more time and resources than Jean-François Corbett's solution) :
• if you are dealing with '1d column' in an excel 2d array (a column), that is an array:
nameArray(1 to arrayCount, 1 to 1) [*]
and if you make the transpose of it, the column dimension will be "skipped" and the result will be:
nameArray(1 to arrayCount)
It makes sense since you will end up with an Excel row (so why bother with an extra dim?). But I have to say this is not the intuitive behaviour I would expect, which should be more something like nameArray(1 to 1, 1 to arrayCount).
Note that, a contrario, it can be used to delete a dimension and redimension the array automatically from 2d to 1d: redim preserve the last dimension to 1 to 1 and then transpose! This is the resulting array just above.
• But finally all is not lost: suppose you transpose this array of 1 line:
nameArray(0 to 0, 0 to arrayCount)
you correctly get an array of 1 column:
nameArray(1 to arrayCount + 1, 1 to 1)
(well, almost) - and look back at ref [*] now...
Thus, if you are to use this build-in function and if, moreover you need more dimensions or worst, need to compose transpositions, it can become a bit tricky...
For all this or if you simply need to know correct indexing and number of dimensions of your arrays (not only number of columns), I would suggest a very useful function from users John Coleman and Vegard, the post just below.
While all this should appear logical and trivial for people used to work with Excel sheets, it's very not the case when you are more used to matrix manipulations and I think suggesting the use of this Transpose function should come with some precisions.

Filtering 2D Arrays in Excel VBA

Using Excel and VBA, I wanted some advice on how to best filter data in an array (in the same sort of way one might use a pivot table) strictly using VBA. I am creating a UserForm that is going to make some data decisions based on currently existing data. I can visualize how to do it well enough but am not that versed in VBA programming.
Here is an example
A B C
bob 12 Small
sam 16 Large
sally 1346 Large
sam 13 Small
sally 65 Medium
bob 1 Medium
To grab the data in an Array, I could use
Dim my_array As Variant
my_array = Range("A1").CurrentRegion
Now, I am familiar with looping through 2D arrays, but I wondered: what the most effective way to filter 2D array data (without looping through the array time and again)?
For example, how do I get would be to say get this kind of data:
data_for_sally As Variant 'rows with sally as name in ColA
data_for_sally_less_than_ten As Variant ' all rows with sally's name in ColA and colB < 10
data_for_all_mediums as Variant ' all rows where ColC is Medium
Suggestions? I could work this out with a bunch of custom functions and loops but I thought there must be a better way. Thanks.
I assume you want to use VBA only.
I think it depends on several parameters, mainly on:
how often you run the same condition => do you store the result of a filter or do you recalculate every time?
how often you need to filter stuff => if often, it is worth having a proper code structure in place, if not then a one off loop is clearly the way to go.
From an OO perspective, assuming performance (speed & memory) is not an issue, I would go for the following design (I won't go into the details of the implementation, only give the general idea). Create a class (let's call it imaginatively ArrayFilter) that you could use like this.
Setup the filter
Dim filter As New ArrayFilter
With filter
.name = "sam"
.category = "Medium"
.maxValue = 10
End With
Or
filter.add(1, "sam") 'column 1
filter.add(3, "Medium") 'column 3
filter.addMax(2, 10) 'column 2
Create the filtered data set
filteredArray = getFilteredArray(originalArray, filter)
The getFilteredArray is fairly straightforward to write: you loop over the array checking if the values match the filter and put the valid lines in a new array:
If filter.isValidLine(originalArray, lineNumber) Then 'append to new array
Pros
Clean design
Reusable, especially with the second version where you use the column number. This can be used to filter any arrays really.
Filtering code is in one function that you can test
Corollary: avoid duplication of code
Cons
Filtering is recalculated every time, even if you use the same filter twice. You can store the results in a Dictionary for example - see below.
Memory: every call to the getFilteredArray creates a new array, but not sure how this can be avoided anyway
This adds quite a few lines of code, so I would do it only if it helps make the code easier to read / maintain.
ps: If you need to cache the results to improve performance, one way would be to store the results in a dictionary and add some logic to the getFilteredArray function. Note that unless your arrays are really big and/or you run the same filter a lot, this is probably not worth it.
filters.add filter, filteredArray 'filters is a dictionary
That way, when you call getFilteredArray next time, you can do something like this:
For each f in filters
'Check if all conditions in f and newFilter are the same
'If they are:
getFilteredArray = filters(f)
Exit Function
Next
'Not found in cache: compute the result
Try this
' credited to ndu
Function Filter2DArray(ByVal sArray, ByVal ColIndex As Long, ByVal FindStr As String, ByVal HasTitle As Boolean)
Dim tmpArr, i As Long, j As Long, Arr, Dic, TmpStr, Tmp, Chk As Boolean, TmpVal As Double
On Error Resume Next
Set Dic = CreateObject("Scripting.Dictionary")
tmpArr = sArray
ColIndex = ColIndex + LBound(tmpArr, 2) - 1
Chk = (InStr("><=", Left(FindStr, 1)) > 0)
For i = LBound(tmpArr, 1) - HasTitle To UBound(tmpArr, 1)
If Chk Then
TmpVal = CDbl(tmpArr(i, ColIndex))
If Evaluate(TmpVal & FindStr) Then Dic.Add i, ""
Else
If UCase(tmpArr(i, ColIndex)) Like UCase(FindStr) Then Dic.Add i, "" 'This finds only exact matches, if you need *FindStr* use: If UCase(tmpArr(i, ColIndex)) Like UCase("*" & FindStr & "*") Then Dic.Add i, ""
End If
Next
If Dic.Count > 0 Then
Tmp = Dic.Keys
ReDim Arr(LBound(tmpArr, 1) To UBound(Tmp) + LBound(tmpArr, 1) - HasTitle, LBound(tmpArr, 2) To UBound(tmpArr, 2))
For i = LBound(tmpArr, 1) - HasTitle To UBound(Tmp) + LBound(tmpArr, 1) - HasTitle
For j = LBound(tmpArr, 2) To UBound(tmpArr, 2)
Arr(i, j) = tmpArr(Tmp(i - LBound(tmpArr, 1) + HasTitle), j)
Next
Next
If HasTitle Then
For j = LBound(tmpArr, 2) To UBound(tmpArr, 2)
Arr(LBound(tmpArr, 1), j) = tmpArr(LBound(tmpArr, 1), j)
Next
End If
End If
Filter2DArray = Arr
End Function

Resources