Batch: How to remove all empty columns from a csv file - batch-file

Question

I have a CSV file like this:
P,PC,,PL,B,15feb16,P,Bay,RP,15-FEB-16,22-FEB-16,7,,,,,,11,14,138,14,16,993.42,-12,-84,-12,,,,,,,,,17,2,-10,0,0,1,1,16:05:53,15FEB16
P,PC,,PL,I,1FEB-16,P,In,RP,15-FEB-16,22-FEB-16,7,,,,,,25,5,32,5,5,-29.7,-24,-168,-24,,,,,,,,,520,14,-10,0,0,1,1,10-MAY-201606:05:53,15-FEB-16
P,PC,,PC,S,15FEB16,P,Su,RP,15-FEB-16,22-FEB-16,7,,,,,,6,5,32,56,5,4.65,0,0,0,,,,,,,,,546,0,0,0,0,1,1,10-MAY-201606:05:53,15-FEB-16
The code I have written is:
#echo off
setlocal EnableDelayedExpansion
for /F "delims=" %%a in (C:\Pca.csv) do (
set line=%%a
set line=!line:,,=, ,!
set line=!line:,,=, ,!
for /F "tokens=1,2,3* delims=," %%i in (^"!line!^") do (
echo %%i,%%l>>C:\P.csv
)
)
But it only deletes 2nd and 3rd column, no matter whether it is empty or contains data.
The sample output file should be like:
P,PC,PL,B,15feb16,P,Bay,RP,15-FEB-16,22-FEB-16,7,11,14,138,14,16,993.42,-12,-84,-12,17,2,-10,0,0,1,1,16:05:53,15FEB16
P,PC,PL,I,1FEB-16,P,In,RP,15-FEB-16,22-FEB-16,7,25,5,32,5,5,-29.7,-24,-168,-24,520,14,-10,0,0,1,1,10-MAY-201606:05:53,15-FEB-16
P,PC,PC,S,15FEB16,P,Su,RP,15-FEB-16,22-FEB-16,7,6,5,32,56,5,4.65,0,0,0,546,0,0,0,0,1,1,10-MAY-201606:05:53,15-FEB-16

Answer 1

Here is a quite comprehensive and adaptive script that removes empty columns from CSV-formatted data.
Before the code is shown, let us take a look at the help message that appears when called with /?:
"del-empty-cols-from-csv.bat"
This script removes any empty columns from CSV-formatted data. A column is con-
sidered as empty if the related fields in all rows are empty, unless the switch
/H is given, in which case the first line (so the header) is evaluated only.
Notice that fields containing white-spaces only are not considered as empty.
USAGE:
del-empty-cols-from-csv.bat [/?] [/H] csv_in [csv_out]
/? displays this help message;
/H specifies to regard the header only, that is the very first row,
to determine which columns are considered as empty; if NOT given,
the whole data, hence all rows, are taken into account instead;
csv_in CSV data file to process, that is, to remove empty columns of;
these data must be correctly formatted CSV data, using the comma as
separator and the quotation mark as text delimiter; regard that
literal quotation marks must be doubled; there are some additional
restrictions: the data must not contain any line-breaks; neither
must they contain any asterisks nor question marks;
csv_out CSV data file to write the return data to; this must not be equal
to csv_in; note that an already existing file will be overwritten
without prompt; if not given, the data is displayed on the console;
As you can read, there are two operation modes: standard (no switch) and header mode (switch /H).
Given that the following CSV data is fed into the script...:
A, ,C, ,E,F
1, , ,4,5,
1, , , ,5,
1, ,3,4, ,
...the returned CSV data in standard mode will look like...:
A,C, ,E,F
1, ,4,5,
1, , ,5,
1,3,4, ,
...and the returned CSV data in header mode (/H) will look like:
A,C,E,F
1, ,5,
1, ,5,
1,3, ,
Remind that the spaces in the above sample data must actually not be present in the files; they have just been inserted here for better illustration of the said operation modes.
Now, this is the complete code:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "OPT_HEAD=%~1"
if "%OPT_HEAD%"=="/?" (
goto :MSG_HELP
) else if /I "%OPT_HEAD%"=="/H" (
shift
) else if "%OPT_HEAD:~,1%"=="/" (
set "OPT_HEAD="
shift
) else set "OPT_HEAD="
set "CSV_IN=%~1"
if not defined CSV_IN (
>&2 echo ERROR: no input file specified!
exit /B 1
)
set "CSV_OUT=%~2"
if not defined CSV_OUT set "CSV_OUT=con"
for /F "delims==" %%V in ('2^> nul set CELL[') do set "%%V="
setlocal EnableDelayedExpansion
if not defined OPT_HEAD (
for /F %%C in ('^< "!CSV_IN!" find /C /V ""') do set "NUM=%%C"
) else set /A NUM=1
set /A LIMIT=0
< "!CSV_IN!" (
for /L %%L in (1,1,%NUM%) do (
set /P "LINE="
call :PROCESS LINE LINE || exit /B !ErrorLevel!
set /A COUNT=0
for %%C in (!LINE!) do (
set /A COUNT+=1
if not defined CELL[!COUNT!] set "CELL[!COUNT!]=%%~C"
if !LIMIT! LSS !COUNT! set /A LIMIT=COUNT
)
)
)
set "PAD=" & for /L %%I in (2,1,!LIMIT!) do set "PAD=!PAD!,"
> "!CSV_OUT!" (
for /F usebackq^ delims^=^ eol^= %%L in ("!CSV_IN!") do (
setlocal DisableDelayedExpansion
set "LINE=%%L%PAD%"
set "ROW="
set /A COUNT=0
setlocal EnableDelayedExpansion
call :PROCESS LINE LINE || exit /B !ErrorLevel!
for %%C in (!LINE!) do (
endlocal
set "CELL=%%C"
set /A COUNT+=1
setlocal EnableDelayedExpansion
if !COUNT! LEQ !LIMIT! (
if defined CELL[!COUNT!] (
for /F delims^=^ eol^= %%R in ("!ROW!,!CELL!") do (
endlocal
set "ROW=%%R"
)
) else (
endlocal
)
) else (
endlocal
)
setlocal EnableDelayedExpansion
)
if defined ROW set "ROW=!ROW:~1!"
call :RESTORE ROW ROW || exit /B !ErrorLevel!
echo(!ROW!
endlocal
endlocal
)
)
endlocal
endlocal
exit /B
:PROCESS var_return var_string
set "STRING=!%~2!"
if defined STRING (
set "STRING="!STRING:,=","!""
if not "!STRING!"=="!STRING:**=!" goto :ERR_CHAR
if not "!STRING!"=="!STRING:*?=!" goto :ERR_CHAR
)
set "%~1=!STRING!"
exit /B
:RESTORE var_return var_string
set "STRING=!%~2!"
if "!STRING:~,1!"==^""" set "STRING=!STRING:~1!"
if "!STRING:~-1!"==""^" set "STRING=!STRING:~,-1!"
if defined STRING (
set "STRING=!STRING:","=,!"
)
set "%~1=!STRING!"
exit /B
:ERR_CHAR
endlocal
>&2 echo ERROR: `*` and `?` are not allowed!
exit /B 1
:MSG_HELP
echo(
echo("%~nx0"
echo(
echo(This script removes any empty columns from CSV-formatted data. A column is con-
echo(sidered as empty if the related fields in all rows are empty, unless the switch
echo(/H is given, in which case the first line ^(so the header^) is evaluated only.
echo(Notice that fields containing white-spaces only are not considered as empty.
echo(
echo(
echo(USAGE:
echo(
echo( %~nx0 [/?] [/H] csv_in [csv_out]
echo(
echo( /? displays this help message;
echo( /H specifies to regard the header only, that is the very first row,
echo( to determine which columns are considered as empty; if NOT given,
echo( the whole data, hence all rows, are taken into account instead;
echo( csv_in CSV data file to process, that is, to remove empty columns of;
echo( these data must be correctly formatted CSV data, using the comma as
echo( separator and the quotation mark as text delimiter; regard that
echo( literal quotation marks must be doubled; there are some additional
echo( restrictions: the data must not contain any line-breaks; neither
echo( must they contain any asterisks nor question marks;
echo( csv_out CSV data file to write the return data to; this must not be equal
echo( to csv_in; note that an already existing file will be overwritten
echo( without prompt; if not given, the data is displayed on the console;
echo(
exit /B

Answer 2

assuming, your original csv looks like this:
id_users,,,quantity,,date
1,,,1,,2013
1,,,1,,2013
2,,,1,,2013
then this single line should solve your request:
(for /f "tokens=1-3 delims=," %%a in (c:\pca.csv) do echo %%a,%%b,%%c)>c:\p.csv
resulting in:
id_users,quantity,date
1,1,2013
1,1,2013
2,1,2013
The trick is: consecutive delimiters are treated as one.
Edit: another approach, as it turned out, there are much more colums, than the original question showed.
#echo off
break>out.txt
for /F "delims=" %%a in (c:\pca.csv) do call :shorten "%%a"
goto :eof
:shorten
set "line=%~1"
:remove
set "line=%line:,,=,%"
echo %line%|find ",,">nul && goto :remove
echo %line%>>c:\p.csv
break>c:\p.csv: create outputfile (overwrite if exist)
replace two consecutive commas with one;
repeat, if there are any more consecutive commas.
Write the resulting line to the outfile.