Excel VBA: More Efficient Way of Removing Rows From Worksheet - arrays

Question

Please note the edit after the original function code block
I've got this data set in Excel that I download from my company's cost management system each month. On average, this data set is around 100,000 rows with 32 columns. One of my job functions is to filter out line items that don't belong to my work group and arrange the data in the required format for a separate analysis system. Typically, I re-arrange the columns, enter a bunch of formulas into cells, and then use a series of autofilter checks to identify line items that need to be moved to other worksheets. This normally takes me about a couple of hours tops, but it's quite arduous and I'd rather automate the process to save time and reduce chances for me to make mistakes.
So I went ahead and wrote a VBA procedure that satisfies all of the requirements and everything seems to be checking out. The only problem is that the procedure itself takes about an hour to check 10,000 line items (I stopped it at that point). Wasting 10 hours watching a progress bar tick is not going to cut it. So now I'm trying to re-think how I've written this procedure to see if there's a better way (I'm certain there is).
Here's the code as it stands (I omitted a lot of code before and after the main loop for clarity, but I left comments there so you can see what happens in a 'pseudo-code' manner. The vast majority of time is spent in that loop, so it's really my main concern):
ORIGINAL FUNCTION
Function Prepare_CICTDF()
'Rename and set worksheet
wbRawFile.Worksheets("Sheet1").Name = "Excluded"
Set wsSheet = wbRawFile.Worksheets("Excluded")
'Update progress bar
status_message = "Rearranging columns in CICT Dedicated Facility. This may take several minutes."
Call Progress_Bar(current_row, status_message)
'Rearrange columns
'Omitted to shorten code block
'Create worksheet for included rows
wbRawFile.Worksheets.Add().Name = "Self Service"
'Copy header row to other worksheets
wsSheet.Rows("4").Copy Destination:=Sheets("Self Service").Range("A4")
'Import Lookup List
Dim wbLookupList As Workbook
Set wbLookupList = Workbooks.Open("\\server\path\to\file\Dedicated Facility Lookup List.xlsx")
Dim wsLookupList As Worksheet
Set wsLookupList = wbLookupList.Worksheets("Lookup List")
wsLookupList.Copy Before:=wbRawFile.Worksheets("Excluded")
wbLookupList.Close SaveChanges:=False
'Get first and last data row
Dim FirstRow As Long
Dim LastRow As Long
FirstRow = 5
LastRow = wsSheet.UsedRange.Rows.Count - 1
'Update progress bar
status_message = "Preparing rows in CICT Dedicated Facilty."
Call Progress_Bar(current_row, status_message)
'Loop through the rows to add formulas
Dim NextBlankRow As Long
Dim RowDeleted As Boolean
Dim i As Long
i = FirstRow
'-------------------------LOOP STARTS HERE-------------------------
Do While i <= LastRow
RowDeleted = False
'Add "CICTDF" before project ID
wsSheet.Range("B" & i).Value = "CICTDF" & wsSheet.Range("B" & i).Text
'Add formula for "Total Impact" column in column T
wsSheet.Range("T" & i).FormulaR1C1 = "=IF(AND(RC[-10]=""Complete"",RC[7]=""Manual Part Number Line Item""),RC[5],IF(AND(RC[-10]=""Complete"",RC[5]=0),0,IF(RC[-10]=""Complete"",RC[5]/RC[-5]*RC[4],RC[5])))"
'Add formula for rows with blank "Cost Impact - Part" column
If wsSheet.Range("V" & i).Value = "" Then
wsSheet.Range("V" & i).FormulaR1C1 = "=IF(RC[-7]>0,RC[3]/RC[-7]*-1,0)"
End If
'Change GLOBAL SUPPLY NETWORK to GLOBAL PURCHASING
If wsSheet.Range("F" & i).Value = "GLOBAL SUPPLY NETWORK" Then
wsSheet.Range("F" & i).Value = "GLOBAL PURCHASING"
End If
'Change numbers stored as text back to numbers
wsSheet.Range("M" & i).NumberFormat = "General"
wsSheet.Range("M" & i).Value = wsSheet.Range("M" & i).Value
wsSheet.Range("P" & i).NumberFormat = "General"
wsSheet.Range("P" & i).Value = wsSheet.Range("P" & i).Value
wsSheet.Range("AB" & i).NumberFormat = "General"
wsSheet.Range("AC" & i).NumberFormat = "General"
wsSheet.Range("AD" & i).NumberFormat = "General"
wsSheet.Range("AE" & i).NumberFormat = "General"
'Insert Cab Part # Formula
wsSheet.Range("AB" & i).Formula = "=VLOOKUP(M" & i & ",'Lookup List'!A:A,1,FALSE)"
'Insert Cabs DC formula
wsSheet.Range("AC" & i).Formula = "=VLOOKUP(N" & i & ",'Lookup List'!B:B,1,FALSE)"
'Insert Cab Localization HEX & MG Formula
wsSheet.Range("AD" & i).Formula = "=VLOOKUP(B" & i & ",'Lookup List'!C:C,1,FALSE)"
'Insert Already in MOASS formula
wsSheet.Range("AE" & i).Formula = "=VLOOKUP(B" & i & ",'Lookup List'!D:D,1,FALSE)"
'Include part numbers that match the inclusion criteria
If wsSheet.Range("AB" & i).Text <> "#N/A" And wsSheet.Range("AC" & i).Text = "#N/A" And wsSheet.Range("AD" & i).Text = "#N/A" _
And wsSheet.Range("AE" & i).Text = "#N/A" And wsSheet.Range("P" & i).Value = "14" Then
NextBlankRow = Worksheets("Self Service").UsedRange.Rows.Count + 1
wsSheet.Rows(i).Copy Destination:=Worksheets("Self Service").Range("A" & NextBlankRow)
wsSheet.Rows(i).Delete
RowDeleted = True
End If
'Check if the row was included or not
If RowDeleted = True Then
LastRow = LastRow - 1
Else
i = i + 1
End If
'Update the progress completion
current_row = current_row + 1
Call Progress_Bar(current_row, status_message)
Loop
'-------------------------LOOP STOPS HERE-------------------------
'Autofilter header row in Self Service tab
Worksheets("Self Service").Range("B4:AG4").AutoFilter
'Save as new file format
Worksheets("Self Service").Select
wbRawFile.SaveAs Filename:=output_directory & "CICT 2014 Dedicated Facility.xlsx", FileFormat:=51
wbRawFile.Application.DisplayAlerts = True
wbRawFile.Close SaveChanges:=False
End Function
Basically I loop through all of the lines in the file. For each line, I enter the formulas and values that I need and then check if they satisfy the inclusion requirements. If they do, I move the line to the "Self Service" worksheet, delete the line from the "Excluded" worksheet, and move on to the next line.
After running the first 10,000 lines of data, the elapsed time was just over 58 minutes. I think most of this can be attributed to the copy/paste/delete processes at the tail end of the loop. I've read that a common suggestion is to work within arrays instead of manipulating cells/rows/ranges in Excel, but I'm not exactly sure how I would go about doing this.
----------Edit:----------
After some input from Ron Rosenfeld, I revisited my process a little bit and made a bunch of changes. In the end, the new procedure processes and prepares over 100,000 rows (of 32 columns) in just over 49 minutes. The original procedure would have taken over 9.75 hours, so the changes have resulted in a procedure that's over 10x faster than its predecessor. Rather than paste the entire code block again, I'll describe the procedure in "pseudo-code":
Rearrange columns (takes the raw server download and puts it in the order I need).
Create a new worksheet for the included rows. Note that for my purposes, I process over 100,000 rows but end up keeping only about 10,000. Thus, I made the decision to look for those that I would INCLUDE instead of those that I would EXCLUDE.
Enter formulas in first row of data and drag down the column. I used Ron's suggestion of e.g. Range("A" & FirstRow & ":A" & LastRow") = "=B1+C1" for any columns that I could.
There was one column that only needed formulas if the cell was blank. So I used the SpecialCells(xlCellTypeBlanks) method to enter these.
AutoFiltered the data so that only the rows I wanted to include were visible. Again I used the SpecialCells(xlCellTypeVisible) method to find these and stored them in an array. This array was then entered into the new worksheet.
Finally, I did a little bit of massaging format-wise to make sure everything looked consistent (since storing the values in the array lost the cell formats).
It should also be noted that I think Tim's suggestion of using SQL in this scenario could be a very efficient alternative--I simply wasn't versed well enough on the topic to try it out. I'll be looking for ways to use it in the future, though!
Thanks everyone for the help!

Answer 1

Without knowing the exact layout of your worksheets, it is hard to say. In general, with regard to values, the process of reading a large DB into an array; looping through the array to decide what rows/items to keep, the writing that back to a new sheet, is usually at least an order of magnitude (10x) faster than looping through rows. Sometimes a challenge is to figure out how big the results array needs to be. If that cannot be done with some simple formulas, I have taken the interim step of gathering each row into a collection before dimensioning the results array.
Another thought after looking at your code: why not just filter on the values for columns 15, 27-30, and then copy/paste the visible cells to your new worksheet.
After you write the data to the worksheet; you can select all the blanks in the range with the SpecialCells method and write the formula that way:
R.columns(X).SpecialCells(xlCellTypeBlanks).FormulaR1C1 = "= RC[1] + RC[2]".
To get the size of arrIncluded, it seems you could either use CountIfs; or you could add the desired rows to a Collection, then use the Count property to get the size of arrIncluded; and write the Collection to the array. I prefer the Collection method, but test to see which way is faster.