Trying to reformat a very large csv with a batch file - arrays

Question

I have an application that exports data in the format:
1a,1b,1c1,1c2,1c3, ... (up to 1c100),1d1,1d2,1d3, ... (up to 1d100)
2a,2b,2c1,2c2,2c3, ... (up to 2c100),2d1,2d2,2d3, ... (up to 2d100)
etc.
and I am trying to reformat this into
1a,1b,1c1,1d1
1a,1b,1c2,1d2
.
.
1a,1b,1c100,1d100
2a,2b,2c1,2d1
2a,2b,2c2,2d2
etc.
I figured that if this can be done a row at a time I can just loop through the file. However I can't find a way of doing a single row with either tokens, a list, or even as a string function. There is too much data to process in a single operation (each value is about 12 chars). Tokens limit at (roughly) 64/202, a list at about 107/202 and a string at about 1000/2300
Does anyone know how this can be written into a new file?
I was trying things like:
#echo off
setlocal enableDelayedExpansion
set dimCnt=0
<example.csv (
set /p "dimList=" >nul
for %%D in (!dimList!) do (
set /a dimCnt+=1
set "dim[!dimCnt!]=%%D"
)
)
echo
for /l %%I in (3 1 102) do echo !dim[1]!,!dim[2]!,!dim[%%I]!
</code>
..besides the fact that I have missed out the last variable in the line (need to add 100 to it), I can't get more than about 80-110 values out of the list (I guess it depends on value string length)

Answer 1

#echo off
setlocal enableextensions enabledelayedexpansion
(for /f "tokens=1,2,* delims=," %%a in (example.csv) do (
set "data=%%c"
set "i=0"
for %%f in ("!data:,=" "!") do (
set /a "i+=1"
set "d[!i!]=%%~f"
)
set /a "end=!i!/2"
set /a "j=!end!+1"
for /l %%i in (1 1 !end!) do (
for %%j in (!j!) do echo %%a,%%b,!d[%%i]!,!d[%%j]!
set /a "j+=1"
)
)) > output.csv
endlocal
This iterates over the file, getting the first two tokens in the line (%%a and %%b), the rest of the line (%%c) is splitted and each value stored in an environment variable array (kind of). Then, the array is iterated from the start and from the middle, reading the needed values to append to %%a and %%b and generating output file.

Answer 2

#ECHO OFF
SETLOCAL
(
FOR /f "tokens=1,2,*delims=," %%a IN (u:\long.csv) DO (
SET rpta=%%a
SET rptb=%%b
CALL :rptcd %%c
)
)>newfile.txt
GOTO :EOF
:rptcd
SET /a lines=100
SET lined=%*
FOR /l %%x IN (1,1,99) DO CALL SET lined=%%lined:*,=%%
:loop
IF %lines%==0 GOTO :EOF
SET /a lines-=1
CALL SET lined=%lined:*,=%
FOR /f "delims=," %%x IN ("%lined%") DO ECHO %rpta%,%rptb%,%1,%%x&shift&GOTO loop
GOTO :eof
This should get you going - just need to change the input filename and output filename...

Answer 3

Your code does not work because SET /P cannot read more than 1023 bytes. At that point it returns the data read so far, and the next SET /P picks up where it left off. Adapting your code to compensate will be very difficult. You would be better off using FOR /F as in MC ND's answer. But beware, batch has a hard limit of 8191 characters per line in pretty much all contexts.
Better yet, you could use another scripting language like JScript, VBS, or PowerShell. Performance will be much better, and the code much more robust and far less arcane. I love working with batch, but it simply is not a good text processing language.