Background: OS: win10. A single root folder: "U:\11Web\gallery-dl".
Within the root I have 1400+ sub-folders (there are no sub-folders beneath those (and never will be)). (There are also no individual/extraneous files in root (and never will be)).
Each sub-folder has its own .bat file, aGallery-dl.bat.
#echo off
setlocal EnableExtensions DisableDelayedExpansion
pushd "%~dp0" || exit /B
move /Y "Folder2.jpg" "Folder.jpg"
for %%I in (.) do set "FOLDER=%%~nxI"
"%ProgramFiles(x86)%\gallery-dl\gallery-dl.exe" -d "U:\11Web\gallery-dl" --download-archive "%~dp0zzzGDB.sqlite3" "https://www.deviantart.com/%FOLDER%/gallery/all"
if not errorlevel 1 if exist "zzzGDB.sqlite3" del "Folder.jpg"
popd
endlocal
I normally run aGallery-dl.bat manually; open the next folder, double-click to run; next folder run, etc., until I'm running between 5 and 10 cmd windows. Later when they've all completed, I might go thru another 5-10 folders, and so it goes. As each aGallery-dl.bat is executed, the requested files are downloaded and when complete, some cleanup is done on Folder.jpg, and each cmd window closes.
Problem: Looking to automate the running of these a little. Would like a single batch file at the root folder that when run would
Create/update a list.txt/database file of all sub-folders. A list would work here, something simple like dir/l>list.txt but then don't know how count would be kept?
Run the aGallery-dl.bat in the first 5 sub-folders of the list, take a pause of say 15 mins (timeout?), loop and then hit the next 5 folders, and so on until 1400+ are done... Doesn't matter that if I have to reboot, or come back a week later and have to re-run that same batch file, that it starts at the first folder all over again...there is a .sqlite3 database file in each folder that retains all the previously downloaded file information. Starting at the first folder every time would be a feature, downloading only those files that have been updated.
I have no starting code/example as I can't wrap my head around it sufficiently to really begin.
Thanks in advance.
The following batch file can be used to run up to five aGallery-dl.bat parallel.
#echo off
setlocal EnableExtensions DisableDelayedExpansion
for /D %%I in ("U:\11Web\gallery-dl\*") do if exist "%%I\aGallery-dl.bat" (
start "Gallery Download %%~nxI" /MIN %ComSpec% /c "%%I\aGallery-dl.bat"
call :CheckDownloads
)
goto :EOF
:CheckDownloads
%SystemRoot%\System32\timeout.exe /T 1 >nul
for /F "delims=" %%J in ('%SystemRoot%\System32\tasklist.exe /FI "IMAGENAME eq cmd.exe" /NH /V ^| %SystemRoot%\System32\find.exe /C "Gallery Download "') do if %%J GEQ 5 goto CheckDownloads
goto :EOF
The subroutine CheckDownloads is called after starting one more command process running with a minimized window parallel to the command process which is processing this batch file for the execution of aGallery-dl.bat with a window title being Gallery Download and the folder name appended.
The subroutine waits one second before it runs one more command process in background to run TASKLIST to get a list of running cmd.exe processes which is filtered with FIND for just those command processes with Gallery Download in window title.
FIND outputs just the number of lines with the searched string which is the number of command processes running currently to do a gallery download. If this number is (greater or) equal five, the subroutine waits again one second before doing again this downloads task check.
A slightly modified version would be:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
for /D %%I in ("U:\11Web\gallery-dl\*") do if exist "%%I\aGallery-dl.bat" if exist "%%I\Folder.jpg" (
start "Gallery Download %%~nxI" /MIN %ComSpec% /c "%%I\aGallery-dl.bat"
call :CheckDownloads
)
goto :EOF
:CheckDownloads
%SystemRoot%\System32\timeout.exe /T 1 >nul
for /F "delims=" %%J in ('%SystemRoot%\System32\tasklist.exe /FI "IMAGENAME eq cmd.exe" /NH /V ^| %SystemRoot%\System32\find.exe /C "Gallery Download "') do if %%J GEQ 5 goto CheckDownloads
goto :EOF
There is additionally checked if the file Folder.jpg exists in the subfolder to omit all the subfolders on which the gallery download was done already once in the past and so the file Folder.jpg does not exist anymore in the subfolder.
The timeout value can be increased from 1 to 5 or 10 or 30 seconds to reduce the CPU usage for checking the downloads tasks.
There should not be started too many gallery downloads parallel as this could be counterproductive depending on CPU cores and download bandwidth.
The batch file posted here finishes already on up to last four gallery downloads are still running parallel.
For understanding the used commands and how they work, open a command prompt window, execute there the following commands, and read entirely all help pages displayed for each command very carefully.
call /?
echo /?
find /?
for /?
goto /?
if /?
setlocal /?
start /?
tasklist /?
timeout /?
I adapted my linked code slightly to fit your environment. (untested, so work with a copy or have a backup)
#ECHO off
setlocal
set bunch=10
for /D %%I in ("G:\tmp\*") do (
call :loop
echo processing: %%a
start "MyCommand" cmd /c "%%I\zzzGallery-dl.bat"
)
call :loop
goto :eof
:loop REM waits for available slot
for /f %%x in ('tasklist /fi "windowtitle eq MyCommand" ^| find /c "cmd.exe"') do set x=%%x
if %x% geq %bunch% goto :loop
goto :eof
%bunch% is the number of threads that run parallel. You have to experiment a bit to get the optimal number. It depends on how much load each thread generates and the specs of your computer.
Sometimes it's right in front of my face... Works one folder at a time, but works. 1400+ folders later I should be good... Thanks for your help Stephan!!
#echo off
setlocal EnableExtensions DisableDelayedExpansion
for /D %%I in ("G:\tmp\*") do ("%%I\zzzGallery-dl.bat"
endlocal
UPDATE: Do wish though that it would run 5-10 of these at a time in parallel. Sitting here watching it do these in serial one folder at a time, even after the first run-thru, gonna take days to complete 1400+ even just for updates.
Anyone able to assist in that regard would be GREATLY appreciated.
Related
I have a challenging problem.
my requirement for batch script,
check whether Firefox browser is running..
And if it is running batch script will play a music.
Script will stay awake until browsing session completes & user closes the browser. (Script will never exits ever since system booted up until the shutdown. Means always stay awake to play the music each & every time user opens up Firefox browser)
Following is the script I've written,
#echo off
: loop
tasklist /fi "imagename eq Firefox.exe" |find ":" > nul
if errorlevel 1 (start "C:\Program Files\KMPlayer 64X\KMPlayer64" "C:\Users\UserName\Desktop\Apple.mp3") else (goto loop)
// no exit
This script only runs for the first time and it does not loop. Can anybody help?
Here is one possible solution:
#echo off
title Firefox Watcher
setlocal EnableExtensions DisableDelayedExpansion
set "MusicPlay="
:Loop
%SystemRoot%\System32\tasklist.exe /FI "IMAGENAME eq Firefox.exe" /NH 2>nul | %SystemRoot%\System32\find.exe /I "firefox.exe" || goto NoFirefox
if defined MusicPlay goto WaitTime
start "" "%ProgramFiles%\KMPlayer 64X\KMPlayer64.exe" "%UserProfile%\Desktop\Apple.mp3"
if not errorlevel 1 set "MusicPlay=1"
goto WaitTime
:NoFirefox
if not defined MusicPlay goto WaitTime
%SystemRoot%\System32\taskill.exe /IM KMPlayer64.exe >nul 2>nul
set "MusicPlay="
:WaitTime
cls
%SystemRoot%\System32\choice.exe /C NY /N /T 5 /D N /M "Stop watching Firefox [N/y]?"
if not errorlevel 2 goto Loop
if defined MusicPlay %SystemRoot%\System32\taskill.exe /IM KMPlayer64.exe >nul 2>nul
endlocal
There could used for the command block below label WaitTime also just:
%SystemRoot%\System32\timeout.exe /T 5 /NOBREAK >nul
goto Loop
That results in a timeout of five seconds without a user prompt to stop watching Firefox with waiting only up to five seconds for the user input before running the check on running Firefox once again.
A wait time with CHOICE or TIMEOUT (or PING) is necessary to pause batch script execution for some time like five seconds as otherwise the Windows command processor cmd.exe would execute the commands in the batch file again and again as fast as possible which means cmd would use one core of the CPU at 100% and would make really lots of file system accesses because of starting the executable TASKLIST again and again without any longer pause between the executions. That would not be good for the entire computer performance on which the music player runs and Firefox wants also CPU performance and access to the file system.
It would be better to replace the shortcut to start Firefox by a shortcut which starts cmd.exe with the command line to start the music player as separate process, run Firefox and wait for its self-termination and then terminate the music player. That would be more efficient.
For understanding the used commands and how they work, open a command prompt window, execute there the following commands, and read entirely all help pages displayed for each command very carefully.
cls /?
choice /?
echo /?
endlocal /?
find /?
goto /?
if /?
set /?
setlocal /?
start /?
taskkill /?
tasklist /?
timeout /?
title /?
See also:
Microsoft documentation about Using command redirection operators for an explanation of >nul, 2>nul and |.
Single line with multiple commands using Windows batch file for an explanation of conditional operator ||.
In Jenkins, I have few Commands that I am trying to implement in the BAT File. So I have 2 BAT Files (RUN.BAT & CHANGE.BAT). In RUN.BAT, Commands are as follows:
CD\
E:
CD E:\RESULTS\1
mkdir 1
xcopy E:\I1\0 E:\RESULTS\1
Above Given Command would Go to the particular folder and Make a folder named as "1" & then the XCOPY line would copy the contents from folder "O" to folder "1".
Now lets come to CHANGE.BAT here the commands are:
for /f "tokens=1-7 delims=-:. " %%a in ("%date% %time%") do ren E:\RESULTS\1 %%a%%b%%c_%%d%%e%%f%%g
This command would change the folder name to the Date & Time of the current System.
Here is the issue, the above-given bat files runs completely fine when I call them on my CMD. But the same doesn't work on the JENKINS & throws me "The syntax of the command is incorrect".
What I am not able to understand is, .bat file works in CMD but the same doesn't work in Jenkins.
Kindly guide me.
The two batch files RUN.BAT and CHANGE.BAT can be replaced by a single batch file with just one command line:
#for /F "tokens=1-6 delims=/: " %%I in ('%SystemRoot%\System32\robocopy.exe "%SystemDrive%\|" . /NJH') do #%SystemRoot%\System32\robocopy.exe "E:\I1\0" "E:\RESULTS\%%I-%%J-%%K_%%L%%M%%N" /E /R:3 /W:2 /NDL /NFL /NJH /NJS & goto :EOF
ROBOCOPY is the replacement for XCOPY and is available by default since Windows Vista and Windows Server 2003.
There is first executed ROBOCOPY with an invalid source directory path to get output an error message by ROBOCOPY which contains current date and time in a region independent format. Region independent format means that the format of date and time does not depend on which country is configured for the used account which is for Jenkins running as service the built-in local system account. That is most likely the reason for the error message output by command REN. The new folder name is different on execution of the batch file with settings for system account in comparison to execution of the batch file with your user account.
Then ROBOCOPY is run once again with correct source directory path and the destination directory path with yyyy-MM-dd_hhmmss as directory name in E:\RESULTS.
ROBOCOPY creates automatically like XCOPY the entire directory tree to destination directory if the destination directory does not already exist.
The command goto :EOF is required to end the batch file execution after copying the files because of ROBOCOPY outputs a second line on error which would be otherwise processed also by FOR. It is possible to replace the predefined label :EOF by any other label which is written on next line if there are more lines to execute like:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
for /F "tokens=1-6 delims=/: " %%I in ('%SystemRoot%\System32\robocopy.exe "%SystemDrive%\|" . /NJH') do %SystemRoot%\System32\robocopy.exe "E:\I1\0" "E:\RESULTS\%%I-%%J-%%K_%%L%%M%%N" /E /R:3 /W:2 /NDL /NFL /NJH /NJS & goto FilesCopied
:FilesCopied
rem More command lines to execute.
endlocal
For completeness a solution working also on Windows XP and all later Windows versions:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
for /F "tokens=2 delims==." %%I in ('%SystemRoot%\System32\wbem\wmic.exe OS GET LocalDateTime /VALUE') do set "CurrentDateTime=%%I"
set "CurrentDateTime=%CurrentDateTime:~0,4%-%CurrentDateTime:~4,2%-%CurrentDateTime:~6,2%_%CurrentDateTime:~8,6%"
%SystemRoot%\System32\xcopy.exe "E:\I1\0" "E:\RESULTS\%CurrentDateTime%\" /I /R /Q /Y >nul
endlocal
For a full explanation what is output by ROBOCOPY on running it first with an invalid source directory path and how FOR processes this output see Time is set incorrectly after midnight. The Windows XP solution with WMIC is explained in detail in the same answer.
For understanding the used commands and how they work, open a command prompt window, execute there the following commands, and read entirely all help pages displayed for each command very carefully.
echo /?
endlocal /?
for /?
goto /?
if /?
robocopy /?
set /?
setlocal /?
wmic /?
wmic os /?
wmic os get /?
wmic os get localdatetime /?
xcopy /?
I have hundreds of sub folders coming off a c:\pdf\ folder.
I want to take the first pdf (sorted by name) in each sub folder and copy them to a single folder, say c:\QA\ so I can check the PDF for errors manually.
Can anyone help? Please!
I have this so far:
#echo off
cls
cd c:
cd c:\pdf\
for /r %%a in (*.pdf) do (
copy %%a c:\qa\%%~nxa
GOTO :EXIT
)
:Exit
Note File names are unique. Target dir is never a subdir of source. The baove code works for the first pdf in a sub folder but does not do the rest of the sub folders.
The New Technology File System (NTFS) returns a list of file names matching a wildcard pattern always sorted in alphabetical order. So for NTFS formatted partitions the following batch code is perhaps enough for this task.
#echo off
for /D %%D in ("C:\pdf\*") do if exist "%%D\*.pdf" call :CopyFirstFile "%%D"
goto :EOF
:CopyFirstFile
for %%F in ("%~1\*.pdf") do (
copy "%%F" C:\qa\
goto :EOF
)
goto :EOF
Please note that first FOR ignores subfolders with hidden attribute set. And the second FOR in subroutine CopyFirstFile ignores PDF files with hidden attribute set.
Another solution working also for File Allocation Table (FAT) partitions (FAT16, FAT32, exFAT) uses the command DIR to get the list of *.pdf files sorted by name.
#echo off
for /D %%D in ("C:\pdf\*") do if exist "%%D\*.pdf" call :CopyFirstFile "%%D"
goto :EOF
:CopyFirstFile
for /F "delims=" %%F in ('dir /A-D /B /ON "%~1\*.pdf"') do (
copy "%~1\%%F" C:\qa\
goto :EOF
)
goto :EOF
The command DIR ignores directories matching the pattern *.pdf and finds also hidden *.pdf files because of option /A-D. But COPY does not copy a hidden *.pdf file!
It would be perhaps better to use MOVE instead of COPY as on next run the batch file would copy the same files again if the folders in C:\pdf are not modified in the meantime.
The command lines below label CopyFirstFile are interpreted as subroutine called by the first FOR loop.
The first goto :EOF exits batch file processing once the first FOR loop iterating over all non hidden subdirectories of C:\pdf finished.
The second goto :EOF exits the second FOR loop and the subroutine after copying the first file found in the directory passed as first (and only) argument to the subroutine.
The third goto :EOF would not be really necessary as this command line should be never executed and is additionally also not needed when there are no more command lines in the batch file. But it is good practice to make sure that each block called as subroutine has as last line goto :EOF or alternatively exit /B which is exactly the same as goto :EOF.
For understanding the used commands and how they work, open a command prompt window, execute there the following commands, and read entirely all help pages displayed for each command very carefully.
call /?
copy /?
dir /?
echo /?
for /?
goto /?
if /?
Here's some example code for you. It looks inside each directory and any potential subdirectory below C:\PDF, it does not look at files inside C:\PDF. If you're looking for something different then update your question accordingly.
#Echo Off
For /D /R "C:\PDF" %%A In (*) Do (
For /F "Delims=" %%B In ('Dir/B/A-D/O-N "%%A\*.pdf" 2^>Nul') Do (
Set "first=%%A\%%B")
SetLocal EnableDelayedExpansion
If Exist "!first!" Copy "!first!" "C:\QA"
EndLocal)
Pause
Please don't just use it, take your time to learn it by checking various references as well.
All you need to do is to combine the features of Dir & For . Now, You can simply analyze the code and modify it according to your need.
Hint for the Code:
CD /D "C:\PDF"
DIR /s /b *.pdf >a.txt
For /f "Tokens=*" %%A In (a.txt) Do (Copy /Y "%%A" "C:\QA")
Exit
And, it is almost all you need. :)
I want to monitor my folder if new file added or not. Then If added I would like to execute some files. But I don't want to use third party app.
I have some ideas but I don't know how to do that.
This is my folder; D:\MonitoringFolder
So every hour batch file will check the files inside of it and writes them into a txt.
dir /b "D:\MonitoringFolder" > old.txt
Old.txt is --> string 1 , string 2, string 3
After one hour, batch file will check it later and writes again into another txt.
dir /b "D:\MonitoringFolder" > new.txt
New.txt is --> string 1, string 2, string 3, string 5
Then it will compare new.txt and old.txt. So string 5 added recently. It will prompt a window and says "String 5" added!. Or new file added (removed).
I want to do that If someone could show me a way to do this I would appreciate that.
Script MONITOR.cmd scheduled to run every now and then:
IF EXIST NEW.TXT DEL NEW.TXT
FOR /F "tokens=*" %%* IN ('DIR /S /B /ON "D:\MonitoringFolder"') DO ECHO "%%*">>NEW.TXT
FOR /F "tokens=*" %%* IN (NEW.TXT) DO (FIND %%* OLD.TXT >NUL || START CMD /K INSERTED.cmd %%*)
FOR /F "tokens=*" %%* IN (OLD.TXT) DO (FIND %%* NEW.TXT >NUL || START CMD /K DELETED.cmd %%*)
DEL OLD.TXT
REN NEW.TXT OLD.TXT
Script INSERTED.cmd will create new window prompting for action on appearing of a new file:
ECHO Inserted new file %1
DIR %1
PAUSE & EXIT
Script DELETED.cmd will create new window prompting for action on disappearing of an old file:
ECHO Deleted file %1
PAUSE & EXIT
Subfolders are monitored, too. It worked for me even with spaces and accented characters in filename.
Since you're already dumping the output every hour, just execute this command from the prompt:
fc /u old.txt new.txt
It will tell you, if any, which differences exist between the two files.
Maybe you're going to write batch scripts (for scanning folder and compare results) and schedule them with a scheduler like cron (Linux) or windows task scheduler every hours for e periodical checking. Some documents here : http://support.microsoft.com/kb/308569 , http://code.tutsplus.com/tutorials/scheduling-tasks-with-cron-jobs--net-8800
#ECHO OFF &SETLOCAL disableDelayedExpansion
SET "SaveFile=Save.File"
IF NOT EXIST "%SaveFile%" GOTO:cont
DIR /b /a-d | FINDSTR /vg:"%SaveFile%">nul||EXIT /b
ECHO(execute some programs here
:cont
>"%SaveFile%" DIR /b /a-d
I'm trying to check what the status is on my XenApp servers for the spoolsv.exe process. I've got the command down to run individually from my XP workstation, but can't seem to get it to iterate through a text file. Here's what I have so far, what will make this populate Servers X-XX on my CMD screen?
#echo off
FOR /F "usebackq" %%G IN ("C:\Documents and Settings\userid\Desktop\Scripts\servers.txt") DO echo tasklist /S %%G /u domain\userid | find "spoolsv.exe"
pause
I can't seem to get it to run correctly, and sometimes it will just pop up my servers.txt file in notepad and not even run. What am I missing?
As you have it presented, tasklist never runs. The "do echo tasklist..." snippet means the literal string "tasklist /S server-one..." is being echo'ed to stdout. Since none of these literal strings contain "spoolsv.exe", the "find" command won't match anything.
Try the following instead:
#echo off
FOR /F "usebackq" %%G IN ("C:\Documents and Settings\userid\Desktop\Scripts\servers.txt") DO call :RunTasklistForOneServer %%G
pause
goto :EOF
:RunTasklistForOneServer
set ServerName=%1
echo Calling server %ServerName%
tasklist /S %ServerName% /u domain\userid | find "spoolsv.exe"