Batch variable unable to hold full text file - batch-file

Question

I am trying to create a batch file that will remove some special characters from a .txt file and append it to another file but the variable is unable to hold the full 1 million++ words in the text file.
Is there any way to make the variable hold all the words or at least split it into groups/sets?
The text look like this
{"One:1","two:2","three:4","four:3","five:5","EG:[512]","sets:{559,212,333,940}"};{"One:9","two:3","three:2","four:1","five:6","EG:[513]","sets:{551,215,331,944}"};...
So far I have tried using a FOR loop to append it to many different files then go through each files later using goto but it just doesn't work like it should.

Answer 1

edited - Previous answer at the bottom. As jeb comments this is faster, not sure if it will even be usable but faster (but, of course, being faster than the previous code is an easy task)
#echo off
setlocal enableextensions disabledelayedexpansion
<"in.txt" >"out.txt" call :process
goto :Eof
:process
set /p "buffer=" || goto :eof
set "buffer=%buffer:"=%"
set "buffer=%buffer:,= %"
set "buffer=%buffer:{=%"
set "buffer=%buffer:}=%"
set "buffer=%buffer:[=%"
set "buffer=%buffer:]=%"
set "buffer=%buffer:;=%"
set "buffer=%buffer::=%"
set "buffer=%buffer:0=%"
set "buffer=%buffer:1=%"
set "buffer=%buffer:2=%"
set "buffer=%buffer:3=%"
set "buffer=%buffer:4=%"
set "buffer=%buffer:5=%"
set "buffer=%buffer:6=%"
set "buffer=%buffer:7=%"
set "buffer=%buffer:8=%"
set "buffer=%buffer:9=%"
<nul set /p "=%buffer%"
goto :process
WARNING : This should not have ever been written. IT IS PAINFULLY SLOW.
Once it has been said,
#echo off
setlocal enableextensions disabledelayedexpansion
rem File configuration
set "inputFile=data.txt"
set "outputFile=out.txt"
rem Variable to hold a cariage return used to show that the
rem script is still working
set "CR="
for /f %%c in ('copy /Z "%~f0" nul') do if not defined "CR" set "CR=%%c"
rem A temporary file will be used.
for %%t in ("%temp%\%~nx0.%random%%random%.tmp") do (
echo Splitting input file into temporary file
> "%%~ft" (
( %= Split the input file into one character per line =%
cmd /u /q /c"type ""%inputFile%""" | cmd /a /q /c"find /v "" "
%= Ensure we have a terminator to empty buffer (keep reading...) =%
<nul set /p"=;"
)|( %= Remove non needed characters =%
findstr /i /r /c:"^[a-z,;]"
)
)
echo Splitting done
echo Starting to read temporary file
echo %time% Here we go ...
set "buffer=#"
rem All data to stdout will be placed into the output file
> "%outputFile%" (
for /f usebackq^ delims^=^ eol^= %%a in ("%%~ft") do (
rem Concatenate one character to the buffer
setlocal enabledelayedexpansion
for /f delims^=^ eol^= %%b in ("!buffer!") do (
endlocal
set "buffer=%%b%%a"
)
rem We will use the semicolon as a delimiter to do
rem partial processing of the input data
if "%%a"==";" (
rem Execute the required processing on the buffer
call :processSection buffer
setlocal enabledelayedexpansion
< nul (
rem Write the processed buffer to stdout
set /p "=!buffer!"
rem Show we are still working
>con set /p "=!time! ... still working ... !CR!"
)
endlocal
set "buffer=#"
)
)
)
rem Processed temporary file can be removed
) & del "%%~ft"
echo(
echo %time% ... Done
goto :eof
:processSection varName
setlocal enableextensions disabledelayedexpansion
rem Retrieve the data from the indicated variable
setlocal enabledelayedexpansion
for /f delims^=^ eol^= %%a in ("!%~1!") do (
endlocal
set "line=%%a"
)
rem The passed buffer includes an initial filler character
set "line=%line:~1%"
rem Process buffer
set "line=%line:,= %"
set "line=%line:;= %"
set "line=%line: = %"
rem Return buffer to the caller
endlocal & set "%~1=%line%"
goto :eof
It was written just for testing but, please, use any other thing.