I put together a basic batch script that grabs words from a txt file and executes curl on three urls with the search_word being the search paramater. The results are placed in a html page. However, I am running into an issue.. it a slow process. How can I execute the curl simultaneously for the three urls?
#echo off
setlocal enabledelayedexpansion
set OUTPUT_FILE=test.html
for /f %%i in (search_words.txt) do (
curl -k -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0" http://www.example.com/search?q=%%i >>%OUTPUT_FILE%
curl -k -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0" http://www.example2.com/search?q=%%i >>%OUTPUT_FILE%
curl -k -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0" http://www.example3.com/search?q=%%i >>%OUTPUT_FILE%
)
Related
curl "http://192.168.0.1/goform/login" -H "Connection: keep-alive" -H "Cache-Control: max-age=0" -H "Origin: http://192.168.0.1" -H "Upgrade-Insecure-Requests: 1" -H "DNT: 1" -H "Content-Type: application/x-www-form-urlencoded" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" -H "Referer: http://192.168.0.1/login.asp" -H "Accept-Language: en-US,en;q=0.9" -H "Cookie: alertWindowStatus=donotshow" --data-raw "loginChallenge=censored just in case&loginUsername=censored&loginPassword=censored" --insecure --location
This is the code for curl that google gave me using networking monitoring when I logged in to my router. I would want curl to login to my router and change a few settings (cmd curl) in a batch file. However, even when I use what chrome gave me when I logged in, I still end up at the login screen. Why does it not curl back the home screen for my router instead of the login screen?
I got this method from: https://superuser.com/questions/1423850/how-can-i-use-curl-to-login-into-non-digest-login-sites
I am trying to scrape data from site which returns 405 not allowed and also load content using AJAX. Is there a way I can still scrape data using any method?
I solved this by using following:
$chrome_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36';
$firefox_agent = 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0';
$ie_agent = 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko';
$edge_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063';
$agents = array($chrome_agent, $firefox_agent, $edge_agent, $ie_agent);
$user_agent = array_rand(array_flip($agents));
Ref: https://github.com/jonnnnyw/php-phantomjs/issues/208
So I've been making a simple HTTP flooder with a Wget and a user agent but when flooding i realize this happens
125.27.78.172 - - [26/Apr/2016:12:38:45 -0500] "GET / HTTP/1.1" 403 4961 "-" "Wget"
And if you are confused about this error i asked my friend to flood my VPS and this happend
208.67.1.176 - - [26/Apr/2016:12:48:32 -0500] "GET / HTTP/1.0" 403 4961 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36"
Why is is it not working with me code:
sprintf(command, "wget -O /tmp/fff --header="Accept: text/html" --user-agent="Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36" http://208.67.1.176/ ");
if you're wondering how this is a flooder the main code loops it
Escape the string?
sprintf(command, "wget -O /tmp/fff --header=\"Accept: text/html\" --user-agent=\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36\" http://208.67.1.176/ ");
I have configured my solr data import handler, and now want do to full import. On Solr Wiki page I have found that one way to check my config is:
Hit _http://local_host:8983/solr/db/dataimport with a browser to verify the configuration.
Is this correct and what would parameter "db" be in previous URL? I assume it is one of parameters I entered somewhere earlier, but it is not clear which. It is not written on the page.
Thanks!
Url you are looking for is:
http://localhost:8983/solr/{collection_name}/dataimport?command=full-import
Tutorial:
http://solr.pl/en/2010/10/11/data-import-handler-%E2%80%93-how-to-import-data-from-sql-databases-part-1/
http://solr.pl/en/2010/11/01/data-import-handler-%E2%80%93-how-to-import-data-from-sql-databases-part-2/
http://solr.pl/en/2010/11/22/data-import-handler-%E2%80%93-how-to-import-data-from-sql-databases-part-3/
This command is for Full-import via URL.
curl -X POST \
http://local_host:8983/solr/db/dataimport \
-H 'Accept: application/json, text/javascript, */*; q=0.01' \
-H 'Accept-Encoding: gzip, deflate, br' \
-H 'Accept-Language: pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7' \
-H 'Cache-Control: no-cache' \
-H 'Connection: keep-alive' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'Origin: http://localhost:8983' \
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36' \
-H 'X-Requested-With: XMLHttpRequest' \
-d 'command=full-import&clean=true&commit=true&wt=json&indent=true&verbose=false&optimize=false&debug=false'
I am now writing a simple c version proxy. As am a beginning. I don not know how it works. I've got the header from request.
GET http://www.google.com/intl/zh-CN/options/ HTTP/1.1 Host:
www.google.com User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X
10.7; rv:11.0) Gecko/20100101 Firefox/11.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 Accept-Encoding:
gzip, deflate Proxy-Connection: keep-alive Cookie:
__utma=173272373.193679248.1334660049.1334660049.1334660049.1; __utmz=173272373.1334660049.1.1.utmcsr=google.com|utmccn=(referral)|utmcmd=referral|utmcct=/; PREF=ID=fa48aeea1d8f76b9:U=a085cdf552ee303a:FF=0:TM=1329307863:LM=1334660032:S=rrb7tn0TSlZxymZj; NID=57=eF8kg0Geq6xPrc4flrw-ps5WXffeSS16kpTjTUXb_7-4tw0wXKbXKeWsa2UXr7o2gvV0o4-D9nVO5JV8PBFY5wT25oesLvlg7UiDwNM1wdC90WC3IqoZi3Kk0Pyh7Aj
_
What the next step should I do. Need I modify the header and send it to the web server. I've parser the URL from the header and connected to the WEBSERVER, and I just got another header. How can I get the body of the page?
The body is separated from the headers by an empty line (technically, \r\n\r\n). Just keep reading what the remote server sends.