I'm new in batch files and I need help extracting an URL from a single-line txt-file using Batch.
This is the curl command:
curl -X "GET" %runone% -H "accept: application/json" -o runone.txt
The output stored in the file (runone.txt) is a single line like this:
{"meta":{"terms-and-conditions":{"href":"https://apps.censored.int/datasets/licences/general/"},"license":"CC-BY-4.0","copyright":"censored"},"data":{"link":{"href":"https://apps.censored.int/webapps/opencharts/content/20220620211737-879c62d3db4f29947daaf8140a3f9261a35e4c5b.png","type":"image/png"},"attributes":{"description":"","name":"opencharts","title":"Chart"},"type":"graphical_product"},"tracker":"tracker-dd80b7a5299e46ef8aa8de4041b12aeb","uid":""}
I only want to pick the URL to the png-file from the runone.txt (in this example: https://apps.censored.int/webapps/opencharts/content/20220620211737-879c62d3db4f29947daaf8140a3f9261a35e4c5b.png) and store it to a variable in batch.
Thanks for your help.
#ECHO OFF
SETLOCAL
rem The following settings for the source directory and filename are names
rem that I use for testing and deliberately include names which include spaces to make sure
rem that the process works using such names. These will need to be changed to suit your situation.
SET "sourcedir=u:\your files"
SET "filename1=%sourcedir%\q72693254.txt"
SET "pngurl="&FOR /f "usebackqtokens=11,*delims={:}" %%b IN ("%filename1%") DO FOR %%e IN (%%c) DO IF NOT DEFINED pngurl SET "pngurl=%%~e"
ECHO pngurl=%pngurl%
GOTO :EOF
Note that if the filename does not contain separators like spaces, then %filename1% does not need to be quoted, and then the usebackq can be omitted.
Solution depends on the format of the output file being as described.
You can try to parse the JSON File and exctract a link from it using Powershell and Batch like this code below :
#echo off
Title Parse JSON File And Exctract A Link From It Using Powershell And Batch
Set "JSON_File=%~dp0runone.txt"
set psCmd="&{(GC "%JSON_File%" | ConvertFrom-Json).data.link.href}"
Call :RunPS %psCmd% URL
Echo "%URL%"
pause & Exit
::-------------------------------------------------------------------
:RunPS <PassPSCMD> <RetValue>
#for /F "usebackq tokens=*" %%i in (`Powershell %1`) do set "%2=%%i"
Goto:eof
:: End of :RunPS function
::-------------------------------------------------------------------
Since the output is JSON, I'd suggest you have a look at the JSON-parser xidel. It'll save you a lot of trouble and headache, especially because of Batch its limitations.
xidel -s --method=GET -H "accept: application/json" "%runone%" -e "$json/data/link/href"
This opens the url and extracts the url to the png-file from the JSON. As xidel can open urls, there's no need for curl, or to save the JSON to a file.
"GET" is the default method, so you can probably do without --method=GET. Maybe the header isn't necessary either.
To assign the png-file-url to a variable:
FOR /F "delims=" %A IN ('
xidel -s --method=GET -H "accept: application/json" "%runone%" -e "$json/data/link/href"
') DO SET "pngurl=%A"
or alternatively:
FOR /F "delims=" %A IN ('
xidel -s --method=GET -H "accept: application/json" "%runone%" -e "pngurl:=$json/data/link/href" --output-format^=cmd
') DO %A
(if you want use this command in a Batch-file, use %%A instead of %A)
Related
I have a Windows batch script like this:
#echo off
setlocal EnableDelayedExpansion
SET fruit1=apple
SET fruit2=banana
SET "payload={\"name\":\"value\",\"name\":\"%fruit1%^|%fruit2%\"}"
echo %payload%
curl -X POST -H "Content-Type: application/json" -d "%payload%" "<serverURL>"
endlocal
The output is:
{\"name\":\"value\",\"name\":\"apple|banana\"}
Cannot find the specified path
So Curl is not doing anything, that error message is due to the pipe I guess, while the echoed payload is displayed correctly.
As you can see I already tried escaping the pipe with ^ , and setting EnableDelayedExpansion , but nothing works.
Is there a way to pass the correct payload to curl?
I suggest to use:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "fruit1=apple"
set "fruit2=banana"
set "payload={\"name\":\"value\",\"name\":\"%fruit1%^|%fruit2%\"}"
setlocal EnableDelayedExpansion
echo !payload!
curl.exe -X POST -H "Content-Type: application/json" -d "!payload!" "<serverURL>"
endlocal
endlocal
The environment variables are defined while delayed environment variable expansion is disabled which results in processing each command line with command SET at beginning only once by the Windows command processor.
See also: How does the Windows Command Interpreter (CMD.EXE) parse scripts?
The vertical bar must be escaped nevertheless with a caret character to be interpreted as literal character and not as pipe redirection operator.
Then delayed expansion is enabled and used on referencing the string value assigned to the environment variable payload by using ! around the environment variable name instead of %.
It would be also possible to use in this special case:
#echo off
setlocal EnableExtensions EnableDelayedExpansion
set "fruit1=apple"
set "fruit2=banana"
set "payload={\"name\":\"value\",\"name\":\"%fruit1%^|%fruit2%\"}"
echo !payload!
curl.exe -X POST -H "Content-Type: application/json" -d "!payload!" "<serverURL>"
endlocal
The three lines defining the three environment variables do not contain an exclamation mark and there is also no other string get from somewhere outside the batch file. For that reason it is possible to enable delayed environment variable expansion with the second command line and let the Windows command processor double process all command lines. The environment variables fruit1 and fruit2 could be referenced in this case also with using delayed expansion by using as fifth command line:
set "payload={\"name\":\"value\",\"name\":\"!fruit1!^|!fruit2!\"}"
For understanding the used commands and how they work, open a command prompt window, execute there the following commands, and read entirely all help pages displayed for each command very carefully.
echo /?
endlocal /?
set /?
setlocal /?
curl -h or curl --help
I have a curl command:
curl -H "Accept: application/json" -H "content-type: application/x-www-form-urlencoded" -X POST -d "*****messsage_request******" https://******/api/ > "C:\Users\abc\Desktop\outputfile.json"
The output of this is in a JSON format (I get this output in one line):
[{"title_name":"abc","title_number":"1","title_des":"this is the title \r\nof the new song\r\n abc","add_comments":"Yes, I like \r\nthis song"},{"title_name":"efgh","title_number":"2","title_des":"\r\nthis is the title of the new song efgh","add_comments":"would\r\n there be parking spot\r\n"}]
I want to remove the \r\n from it. The \r\n could be anywhere.
I tried to replace:
-d
With:
--data-binary
But it didn't work.
I could use batch file or I think curl has a way to filter but I am not sure since I have not used both much.
IMO PowerShell is much better suited to process Json output, especially multiline data
## Q:\Test\2019\02\14\SO_54696007.ps1
# $Json = curl.exe --% -H "Accept: application/json" -H "content-type: application/x-www-form-urlencoded" -X POST -d "*****messsage_request******" https://******/api/
# simulateed input
$Json = '[{"title_name":"abc","title_number":"1","title_des":"this is the title \r\nof the new song\r\n abc","add_comments":"Yes, I like \r\nthis song"},{"title_name":"efgh","title_number":"2","title_des":"\r\nthis is the title of the new song efgh","add_comments":"would\r\n there be parking spot\r\n"}]'
$Table = $JSon | ConvertFrom-Json
$Table | Out-GridView
Sample output:
Any reasonable batch solution would manipulate strings within environment variable strings. But the maximum variable length that batch can process is around 8191 characters. It is conceivable that the CURL output could exceed that length, so pure batch is probably not a good option.
There are a number of scripting languages native to Windows that can easily process text of arbitrary length, such as JScript, VBScript, PowerShell.
There are also readily available 3rd party command line tools that can manipulate text, including Windows ports of unix utilities. For example sed can be found for Windows.
I would use my JREPL.BAT regular expression text processor utility, since I always have it handy and it is simple and convenient :-) It is pure script (hybrid batch/JScript) that runs natively on any Windows machine from XP onward - no 3rd party exe required.
curl --YourArguments | jrepl \r\n "" /L >"yourOutputFile.json"
or
curl --YourArguments | jrepl \r\n "" /L /O "yourOutputFile.json"
The /L option treats the search term as a string literal, which is what you want.
If you want to first capture the curl output, and then post-process with JREPL, then
curl --YourArguments >"yourOutputFile.json"
call jrepl \r\n "" /L /F "yourOutputFile.json" /O -
For a Unix-y solution, how about
curl ... things | awk "{printf("\""%s"\"",$0)}' >output
(DOS quoting gleaned from Gawk documentation; I hope it's still current.)
Here is a possible solution:
#echo off
(curl -H "Accept: application/json" -H "content-type: application/x-www-form-urlencoded" -X POST -d "*****messsage_request******" https://******/api/)>"C:\Users\abc\Desktop\outputfile.json"
for /F "delims=" %%A IN (C:\Users\abc\Desktop\outputfile.json) do (
set "line=%%A"
)
echo %line:\r\n=%
Something more general here (adapted from above):
#echo off
(curl -H "Accept: application/json" -H "content-type: application/x-www-form-urlencoded" -X POST -d "*****messsage_request******" https://******/api/)>"C:\Users\abc\Desktop\outputfile.json"
for /F "delims=" %%A IN (C:\Users\abc\Desktop\outputfile.json) do (
set "line=%%A"
call echo %line:\r\n=%
)
This, will echo output to cmd, but the file will remain with \r\n. To also modify the file, use:
#echo off
(curl -H "Accept: application/json" -H "content-type: application/x-www-form-urlencoded" -X POST -d "*****messsage_request******" https://******/api/)>"C:\Users\abc\Desktop\outputfile.json"
for /F "delims=" %%A IN ('type "C:\Users\abc\Desktop\outputfile.json"') do (
set "line=%%A"
call echo %line:\r\n=%>"C:\Users\abc\Desktop\outputfile.json"
)
However, the suggested way, is as follows:
#echo off
for /F "delims=" %%A IN ('curl -H "Accept: application/json" -H "content-type: application/x-www-form-urlencoded" -X POST -d "*****messsage_request******" https://******/api/') do (
set "line=%%A"
call echo %line:\r\n=%>"C:\Users\abc\Desktop\outputfile.json"
)
which is much sorter.
The %line:\r\n=% means to search in variable line (content of file) the string \r\n and replace it (=) with nothing, so just remove it.
Note: You have mentioned that your curl command contains &. Replace it with ^& to make it work!
I'm trying to get a string from an url using a batch file.
String example:
e-e --ser u.g --p 3 --f 0 x,ss
I am using the command below to CURL output directly to a variable:
FOR /F %%I IN ('curl.exe -s -S %URL%') DO (SET W=%%I)
The problem is, when I echo the variable [W] after the command runs, most of the string is missing...
e-e
What is the best method to get around this issue?
By default, the FOR /F command delimits the output based on a space and tab. That is stated in the help file. To keep that from happening use the DELIMS option to tell the FOR command to not use any delimiters.
FOR /F "delims=" %%I IN ('curl.exe -s -S %URL%') DO (SET W=%%I)
I'm trying to make a script to:
send a request to an url, then append (>>) the %%a variable i set in the FOR /F command IF i get a certain response from the command i run with FOR /F.
I tried with 2 scripts that are the followings:
FOR /F %%a in (usernames.txt) do (
IF "curl -k -d name=%%a https://club.pokemon.com/api/signup/verify-username" EQU "{"valid":true,"suggestions":[],"inuse":false}" %%a >> usernames1.txt
and
set /p VAR = < tmpFile
FOR /F %%a in (usernames.txt) do (
curl -k -d name=%%a https://club.pokemon.com/api/signup/verify-username
> tmpFile
IF VAR={"valid":true,"suggestions":[],"inuse":false}
%%a >> usernames1.txt)
EDIT: good enough with that script, thanks guys. but i've got another thing: can i add more than 1 variable to the script? I mean a variable like %%a, that takes every line from another txt file
First, you set VAR once, whereas your temporary file does not exist. Then you test with = instead of == and without the ! chars.
And don't put too many spaces like if you were using a real shell like bash :)
As Magoo noted, I also had to fix set /p VAR = < tmpFile to remove the extra spaces. Batch has a tendency to take them literaly.
(another example: echo foo > file: file now contains "foo ".
(needless to say that without enabledelayedexpansion it wouldn't work either because lots of things happen inside the FOR loop)
The fixed code (also had to protect the test string with extra quotes or it wouldn't work):
setlocal enabledelayedexpansion
del usernames1.txt >NUL 2>NUL
FOR /F %%a in (usernames.txt) do (
curl -k -d name=%%a https://club.pokemon.com/api/signup/verify-username > tmpFile 2>NUL
set /p VAR=<tmpFile
IF "!VAR!"=="{"valid":true,"suggestions":[],"inuse":false}" echo %%a >> usernames1.txt
)
I tested it with a random list of user names and changing "valid":true by "valid":false and the names were issued.
I have a PowerShell script that outputs a single string value. I have a cmd batch script that needs to execute the PowerShell script and place that single PowerShell output value into a variable in the batch script. I'm finding all sorts of methods for exporting to a file or reading a file but that's not what I want. Thanks!
(edit) Here's where I'm trying to use it (in response to posting the script):
#echo off
REM The next line puts the .ps1 output into the variable
REM and, obviously, this does not work
set pass_word=<C:\temp\PullPassword.ps1
tabcmd login -s "http://myserver.net" -u mylogon -p %pass_word%
( edit )
I saw the answer by the OP here Getting Powershell variable value in batch script and also looked at foxdrive's answer so I started playing with the FOR...DO statement. I thought I was pretty good with cmd and couldn't figure out why what I had wasn't working:
for /f "delims=" %%a in ('powershell . "C:\temp\PullPassword.ps1"') do set val=%%a
echo %a%
When I was looking at foxdrive's full answer in the other post it struck me: %a% was wrong, I needed %val%! Oh, the shame! The below works:
#echo off
set mypath=C:\temp\PullPassword.ps1
for /f "delims=" %%a in ('powershell . "C:\temp\PullPassword.ps1"') do set pass_word=%%a
tabcmd login -s "http://myserver.net" -u mylogon -p %pass_word%
So I'll credit where it's due and mark foxdrive's answer correct, even though it was the other post that clarified my mistake.
This may help: it expects that the powershell script outputs the text to STDOUT which is the normal place for it to appear.
#echo off
for /f "delims=" %%a in (' powershell "script.ps1" ') do set "var=%%a"