Load big table into web browser using react in on-demand instantiation of table row - reactjs

Question

I'm building a Excel-like table into web browser with React.js using only <div> not <table>.
Number of columns are about 90, rows are about 24000.
As we know, it is impossible to load whole data into HTML at single web page due to performance issue.
So I decided to show partial data to user using scrolling.
The main concept is simple, build HTML near user's viewport.
Guess if user is seeing 1800th to 1900th data in single viewport. I'will load only about 1750th ~ 1950th data into HTML. If user scroll up, I'll load HTML for 1700th ~ 1750th data and remove 1900th ~ 1950th data.
I think I need to manually manipulate scroll offset for getting pos where user is at. If each row's height is same as 40px and height of viewport is 1000px, then user will see 25 items at single viewport, so I need to load about 25(front) + 25(currently seeing) + 25(end) data and if user go upside or downside, I'll load additional data and remove data which far away from user.
However, I found that, requirement for my table is not matched with this situations. Here's my situation.
First, Each row's height is not same. Basically my table will show rows of row as single row. What I mean is, table single row can be looks like below,
| Photo| ProductName | Size Pool | Stock |
.... // Below are single row
+------+---------------+-------------------+------------+
| | Boots | 110-120 | 24 | // Row header (Shows Summary of child row)
+ +---------------+-------------------+------------+
| | Boots | 110 | 16 | // Row's row #1
+ +---------------+-------------------+------------+
| | Boots | 120 | 8 | // Row's row #2
+------+------------------------------------------------+
...
+------+---------------+-------------------+------------+
| |Leather Shoe | 120 | 8 | // Row can come with no header row, only single
+------+---------------+-------------------+------------+
...
Like above, if product has more than 2 options, then it merge into rows of single row and show with summary header. And if not a option product, it shows only it's row. And if content inside the row is big, it will stretch to fit the content inside
All data came from remote DataBase which retrieve data via REST API.
DataBase scheme is like below, 2 table as example.
Table #1 ProductInfo
+--------------+------------+------------+-----------+
| GroupNumber |ProductName | Size | Stock |
+--------------+------------+------------+-----------+
| 1 | Boots | 110 | 16 |
+--------------+------------+------------+-----------+
| 1 | Boots | 120 | 8 |
+--------------+------------+------------+-----------+
| 2 |Leather Shoe| 120 | 8 |
+--------------+------------+------------+-----------+
Table #2 GroupInfo
+-----------+------------+--------------+
|GroupNumber| SizePool | ImageURL |
+-----------+------------+--------------+
| 1 | 110-120 | https://abc |
+-----------+------------+--------------+
| 2 | 120 | https://def |
+-----------+------------+--------------+
And future requirements are below, (And most of them are implemented)
Sort by each columns, multi-pivot sort by row of row OR row (Handled via SQL)
Filter data by expression (Handled by client)
Hiding, resizing, change order of column(s) (Handled by client)
Interactable component inside cell like DatePicker, Pop-up etc... (Handled by client)
I succeed to create such table with page based method. But I need scrolling viewport table.
The table contains lots of dependent value column like sum, average which are not in stored in DB except for special reason (Like performance). (Most of them are handled by DB View or Procedure including sorting, calculations etc). So overall performance is really important.
I considered few questions and way to handle this, Can you check and give me a advice?
Q1. How can I decide when data should be loaded and removed and it's amount?
Data height is not consistent, so I think I cannot use scroll offset or data number as measurement criteria. (Is it possible with predictable way?)
Is it possible to archive by accessing DOM element? I'm new to Web dev. Sorry.
Q2. I can get a data from DB in 2 different ways.
Getting ProductInfo And GroupInfo seperately [<ProductInfo>,...] And [<GroupInfo>,...]
Getting Single group which object like this { group:<GroupINfo>, values:[<ProductInfo>,...] }
which is better for performance in this case in typical situations?
Q3. If I got a data like { group:<GroupINfo>, values:[<ProductInfo>,...] }, is there any problems with performance?
Like query overhead (I need to use query joined 6 times with maximum 6 depth nested SELECT query with 30 calculated columns for single data retrieval attempt. -- Pre-calculated view or table can have problems because I have many user to use it and update frequently. So I need to worry about Mutual Exclusive at least on updating.
I'm sure that above query's performance is sufficient for cropping if I got data like [<ProductInfo>,...] And [<GroupInfo>,...]. But I think later one is better. so I need to change interface if possible.
Q4. If I crop whole data from DB and structurize at the beginning, and load and remove data only for DOM, Can it be a good way?
Of course, Q1 is my primary matter, but this also seems good except for data sync with DB (Cause other user can update value while client contain outdated data)
I considered of using Infinite-Scrolling, but this is not for my case, I need perform load data and remove data at the same time. But infinite-scrolling seems dose not support removing data from viewport. Also inconsistent row height may be a problem.

Answer 1

I found react-virtualized and it works.
It also support dynamic resizing of row and it greatly helped