I'm currently working on an excel workbook using the following formula to copy all rows from one sheet (Creation_Series_R) to another one, excluding empty rows.
{=IFERROR(INDEX(Creation_Series_R!C:C;SMALL(IF(Creation_Series_R!$C$3:$C$20402<>"";ROW(Creation_Series_R!$C$3:$C$20402));ROW()-ROW(Creation_Series_R!$C$3)+1));"")}
And the formula works very well. Except, when I did my proof of concept I only had a few rows but with the final data, I need to work on 20400 rows... adding to the fact that I have 17 columns, and 3 similar sheets with similar formula, my workbook takes an hour to compute every time I input just one value.
This workbook is designed as a way for a client to enter data, and then it reorganize the data so that it can be imported directly in our software. I already limited the number of data the user can enter per workbook (to their very big disappointment), so I can't really reduce it to less than 20400 rows (it's only a 100 funds financial data).
Is there a way, even maybe using macro, I could do this more efficiently ?
The big block of array formulas is killing your performance (time-wise).
If your data is in column A through Q, then I would use column R as a "helper" column. In R2 insert:
=COUNTA(A2:Q2)
and copy down. The macro would:
AutoFilter column R
Hide all rows showing 0 in column R
Copy the visible rows and paste elsewhere as a block
Related
I have a workbook in which a variable number of rows of data (one per employee) are entered each week on one sheet (DATA ENTRY), and then stored on another sheet (LOG) with the help of a macro that is executed every time the file is saved.
To be able to then retrieve and review employee data for a specific week, I need a column of helper cells in which all the unique distinct dates (weeks) are listed.
I currently do this with the following array formula:
{=IFERROR(INDEX($B$2:$B$1600, MATCH(0,COUNTIF($K$1:K1, $B$2:$B$1600), 0)),"")}
This all works brilliantly, except that I found that this one specific formula slows my file down tremendously. When the file is saved (which triggers data to be copied over to the LOG sheet), it can take up to 10 seconds to process. When this array formula is disabled, it is pretty much instantaneous.
Limiting it to run over 1600 rows helped significantly (it took much longer before when I had it set to 20.000), but it is still not enough and I can't really have this check less than 1600 rows.
Any creative solutions to either make this formula run faster, or to get to the same result (a list of unique distinct dates from a large list of dates) without using an array formula?
Thanks!
You could use Power Query (Get & Transform Data) to populate your list of unique dates.
The problem I have is I need to merge two excel workbooks by common column data (Column A in both workbooks are the same data - names) but workbook 2 has more individual named data than workbook 1, in column A, so I just want the data from workbook 2 that matches the data from workbook 1 - and to skip over (leave out) the excess data in workbook 2? - if possible I'd also like to only extract certain columns from workbook 2 to merge with workbook 1 also. Is this possible? The files are too big to manually remove unnecessary rows by filter. Thanks
You could use Power Query (preferably in a third workbook.)
Have one query for each workbook/column.
The two queries can be combined into one query so all the data is together.
Preferably define the queries as Connection Only and load the resulting data directly up into a Power Pivot data model.
You have all sorts of ETL (extract, transform, load) capabilities in Power Query.
Access the data via a pivot table (in the third workbook.)
If your datasets are massive I'd upgrade to 64bit Office first.
thanks both. Only getting back to this as have been away. Jahknows, in response to your questions, the only column that actually matches is the first column (column A). And yes, I want to append the data from workbook 2 to workbook 1 - based on the names in column A matching in both workbooks - workbook 2 will have more names in column A than workbook 1. this is excess data that is not needed in the appended workbook.
Thanks TDP - I've had a quick play around with the power query and this could be the answer when I've fully figured out how to append what I want and have leave out the excess data I referred to.
Is there a way to create a running database in excel (and only in excel, without using third party programs) so for example:
-One worksheet has today's data for each person
-The additional worksheets (one per each person on the first worksheet) keeps a list of each of the past columns
-Each of the worksheets, except for the current worksheet, charts each new row of data added daily.
Here is a picture in case it helps:
This can be done in Excel, but you need to get the data architecture right.
Use ONE sheet for all raw data. Columns are Date, member, score, number of pages, number of files, notes. New data goes at the bottom of the list. You can use VBA to create a data entry form if you don't want to enter data straight into the sheet. The sheet can be hidden, if needed.
Then use ONE other sheet to create a dynamic report where you can select the time frame and the member to report on. Data is pulled from the raw data sheet and aggregated as required. Pivot tables are immensely powerful.
Using a sheet for each member would be duplication of functionality and bad data design.
Edit: a few conceptual screenshots
The raw data table. New data is added at the bottom of the table. A VBA form can ensure a pleasant user interface, so that the user never sees this table.
The report could be a pivot table grouped by date. Slicers allow the selection of specific time frames, for example a month. Another slicer allows filtering by a specific member.
It took me roughly 5 minutes to create the scenario, including making up the dummy data. With a few hours to spend, this could be made really shiny.
I have four excel sheets, One displays the data for this week (Sheet1), second displays the data from last week (sheet 2), third sheet has all the records (Sheet 3) which can be there in any sheet, i.e. sheet 1 or sheet 2 and fourth has the data with which we identify if the data should be sourced from Current Week or last week (Sheet 4, can also be mentioned as scope sheet).
For e.g. if "XXXX" mentioned in third sheet is available in the sheet4, then it will copy all the related records from Sheet 1, else the record should be picked from Sheet 2.
I have tried using vlookup; however, the output is required in Sheet 5 and the problem is that i need to have all number of records already in Sheet 5 for whole data (all records of sheet 3, if a single record is coming three times, i should have three rows in sheet 5 with the same number). I want to make my sheet dynamic and feel it can only be done using a Macro code. Can anyone help me with this.
I think this can be used using Autofilter, but i am not sure how i should put it to use, as i have around 500 records for which i want data and each record have atmost five related records in either Sheet1 or Sheet2.
It seems like you need to review your data architecture. There is a no-fail, golden rule:
Data input on one sheet.
Reporting on other sheets.
I have four excel sheets, One displays the data for this week (Sheet1), second displays the data from last week (sheet 2), third sheet has all the records (Sheet 3)
This is not good data architecture. Rather than create VBA that helps you support bad architecture, I recommend that you re-organise your file to adhere to good practice spreadsheet design.
You don't need to copy data, you just need to show the correct data in your reports. Reports can easily be created from a data input sheet. No need for VLookup. Time frames like "this week" or "last week" can be shown with filters in your report.
Hello clever people.
I manage a group workbook; each sheet has time series data, in one row per month. I receive excel sheets updated with an extra row for the latest month's data. After some rudimentary checks, I paste the new sheet over the existing sheet, so newer data is now added lower down the page.
Sometimes, a value in a row for an earlier month has changed in the imported sheet - sometimes by accident but often after validation. Obviously, when I paste on the latest sheet, only the most recent value is present - I don't necessarily need the old value, I just need to know its been changed.
I thought of performing a checksum on each row, before and after - that would do to indicate a change. Any ideas of a straightforward approach?
TIA, Paul
If values to check are numbers only you could paste it special with substraction, non-changed cells would become zeros...