I am trying to do basic web automation in Python. I am running a test. And I get an error message when I run everything saying:
search.send_keys("Test")
AttributeError: 'list' object has no attribute 'send_keys'
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
url = "https://www.youtube.com"
browser = webdriver.Chrome()
browser.get(url)
time.sleep(3)
search = browser.find_elements(By.XPATH, "//*[#id='search']")
search.send_keys("Test")
How can ı fix it?
"browser.find_elements" gets all the elements with the xpath you sent, returning an array with all the elements that you can then iterate over.
You need to select an individual object instead:
search = browser.find_element(By.XPATH, "//*[#id='search']")
search.send_keys("Test")
If that code doesn't work for you, try using tools like xPath Finder to grab a unique xPath of the element you want to manipulate.
Related
Background story: I need to obtain the handles of the tagged Twitter users from an attached Twitter media. There's no current API method to do that unfortunately (see https://twittercommunity.com/t/how-to-get-tags-of-a-media-in-a-tweet/185614 and https://github.com/twitterdev/open-evolution/issues/34).
I have no other choice but to scrape, this is an example URL: https://twitter.com/justinwood_/status/1626275168157851650/media_tags. This is the page which pops up when you click on the tags link under the media of the parent Tweet: https://twitter.com/justinwood_/status/1626275168157851650/
The React generated DOM is deep and ugly, but would be scrapeable, however I do not want to log in with any account to get banned. Unfortunately when you visit https://twitter.com/justinwood_/status/1626275168157851650/media_tags in an Incognito window the popup shows up dead empty. However when I dig into the network requests the /TweetDetail GraphQL endpoint is full of messages about the anonymous page visit, fortunately it still contains the list of handles I need despite of all of this.
So what I need to have is a scraper which is able to process JavaScript, and capture the response for that specific GraphQL call. Selenium uses a headless Chrome under the hood, so it is able to process JavaScript, and Selenium-Wire offers the ability to capture the response.
Unfortunately my crafted Selenium-Wire script only has the TweetResultByRestId and UsersByRestId GraphQL requests but is missing the TweetDetail. I don't know what to tweak to make all the requests to happen. I iterated over a ton of Chrome options. Here is a variation of my script:
from seleniumwire import webdriver
from selenium.webdriver.chrome.service import Service
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--headless") # for Jenkins
chrome_options.add_argument("--disable-dev-shm-usage") # Jenkins
chrome_options.add_argument('--start-maximized')
chrome_options.add_argument('--window-size=1900,1080')
chrome_options.add_argument('--ignore-certificate-errors-spki-list')
chrome_options.add_argument('--ignore-ssl-errors')
selenium_options = {
'request_storage_base_dir': '/tmp', # Use /tmp to store captured data
'exclude_hosts': ''
}
ser = Service('/usr/bin/chromedriver')
ser.service_args=["--verbose", "--log-path=test.log"]
driver = webdriver.Chrome(service=ser, options=chrome_options, seleniumwire_options=selenium_options)
tweet_id = "1626275168157851650"
twitter_media_url = f"https://twitter.com/justinwood_/status/{tweet_id}/media_tags"
driver.get(twitter_media_url)
driver.wait_for_request("/TweetDetail", timeout=10)
Any ideas?
Apparently it looks like I'd rather need to scrape the parent Tweet URL https://twitter.com/justinwood_/status/1626275168157851650/ and right now it seems my craved GraphQL call happens. Probably I got confused while trying 100 combinations.
I am using AutoJsContext in geckobrowser with that I am using evaluate scrpit and asble to get data. But now i am using webview2 is there a how can i get this.
Gecko browser:
using (AutoJSContext context = new AutoJSContext(browser.Window))
{
var userIdResult = context
.EvaluateScript("userId", (nsIDOMWindow)browser.Window.DomWindow);
}
Above is the code I use in gecko browser. now i need to get user id from webview2. Please help me in this issue
I am not able to get any alternative
You can use CoreWebView2.ExecuteScriptAsync to inject script into the current top-level document of the WebView2 and receive the result as a JSON string. The method doesn't have a way to specify an execution context. Script is executed in the global context of the top-level document similar to running script in the console of DevTools in the browser.
For example something like the following:
var resultAsJSON = await coreWebView2.ExecuteScriptAsync('window.userId');
I am trying to download the 24-month data from www1.nseindia.com and it fails on Chrome and Firefox drivers. It just freezes after filling all the values in the required places and does not click. The webpage does not respond...
Below is the code that I am trying to execute:
import time
from selenium import webdriver
from selenium.webdriver.support.ui import Select
id_list = ['ACC', 'ADANIENT']
# Chrome
def EOD_data_Chrome():
driver = webdriver.Chrome(executable_path="C:\Py388\Test\chromedriver.exe")
driver.get('https://www1.nseindia.com/products/content/equities/equities/eq_security.htm')
s1= Select(driver.find_element_by_id('dataType'))
s1.select_by_value('priceVolume')
s2= Select(driver.find_element_by_id('series'))
s2.select_by_value('EQ')
s3= Select(driver.find_element_by_id('dateRange'))
s3.select_by_value('24month')
driver.find_element_by_name("symbol").send_keys("ACC")
driver.find_element_by_id("get").click()
time.sleep(9)
s6 = Select(driver.find_element_by_class_name("download-data-link"))
s6.click()
# FireFox(Gecko)
def EOD_data_Gecko():
driver = webdriver.Firefox(executable_path="C:\Py388\Test\geckodriver.exe")
driver.get('https://www1.nseindia.com/products/content/equities/equities/eq_security.htm')
s1= Select(driver.find_element_by_id('dataType'))
s1.select_by_value('priceVolume')
s2= Select(driver.find_element_by_id('series'))
s2.select_by_value('EQ')
s3= Select(driver.find_element_by_id('dateRange'))
s3.select_by_value('24month')
driver.find_element_by_name("symbol").send_keys("ACC")
driver.find_element_by_id("get").click()
time.sleep(9)
s6 = Select(driver.find_element_by_class_name("download-data-link"))
s6.click()
EOD_data_Gecko()
# Change the above final line to "EOD_data_Chrome()" and still it just remains stuck...
Kindly help with what is missing in that code to download the 24-month data... When I perform the same in a normal browser, with manual clicks, it is successful...
When you are manually doing it in a browser, you can change the values as below:
Set first drop down to : Security wise price volume data
"Enter Symbol" : ACC
"Select Series" : EQ
"Period" (radio button: "For Past") : 24 Months
Then click on the button, "Get Data", and in about 3-5seconds, the data loads, and then when you click on "Download file in CSV format", you can have the CSV file in your downloads
Need help using any library you know for scraping in Python: Selenium, Beautifulsoup, Requests, Scrappy, etc... Doesn't really matters unless it is python...
Edit: #Patrick Bormann, pls find the screenshot... The get data button works..
When you say that it works manually, have you try to simulate a click with action chains instead of the internal click function
from selenium.webdriver.common.action_chains import ActionChains
easy_apply = Select(driver.find_element_by_id('dateRange'))
actions = ActionChains(driver)
actions.move_to_element(easy_apply)
actions.click(easy_apply)
actions.perform()
and then you simulate a mouse movement to the specific value?
In addition, I tried it on my own and I didnt get any data when pushing on the button Get Data, as it seems to have a class of "get" as you mentioned, but this button doesnt work, but as you can see there exists a second button called full download, perhaps ypu try to use this one? Because the GetData Button doesnt work on Firefox and Chrome (when i tested it).
Did you already try to catch it through the link?
Update
As the OP asks for help in this urgent matter I delivered a working solution.
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
import time
from selenium.webdriver.support.ui import Select
chrome_driver_path = "../chromedriver.exe"
driver = webdriver.Chrome(executable_path=chrome_driver_path)
driver.get('https://www1.nseindia.com/products/content/equities/equities/eq_security.htm')
driver.execute_script("document.body.style.zoom='zoom 25%'")
time.sleep(2)
price_volume = driver.find_element_by_xpath('//*[#id="dataType"]/option[2]').click()
time.sleep(2)
date_range = driver.find_element_by_xpath('//*[#id="dateRange"]/option[8]').click()
time.sleep(2)
series = driver.find_element_by_name('series')
time.sleep(2)
drop = Select(series)
drop.select_by_value("EQ")
time.sleep(2)
driver.find_element_by_name("symbol").send_keys("ACC")
ez_download = driver.find_element_by_xpath('//*[#id="wrapper_btm"]/div[1]/div[3]/a')
actions = ActionChains(driver)
actions.move_to_element(ez_download)
actions.click(ez_download)
actions.perform()
Here you go, sorry, took a little, had to bring my son to bed...
This solution provides this output: I hope its correct. If you want to select other drop down menus you can change the string in the select (string because of too much indezes too handle) or the number in the xpath as the number highlights the index. The time is normally only for elements which need time to build themselves up on a webpage. But I made the experience that a too fast change sometimes causes errors. Feel free to change the time limit and see if it still works.
I hope you can now go on again in making some money for your living in India.
All the best Patrick,
Do not hesitate to ask if you have any questions.
UPDATE2
After one long night and another day we figured out that the Freezing originates from the website, as the website uses:
Boomerang | Akamai Developer developer.akamai.com/tools/… Boomerangis
a JavaScript library forReal User Monitoring (commonly called RUM).
Boomerang measures the performance characteristics of real-world page
loads and interactions. The documentation on this page is for mPulse’s
Boomerang. General API documentation for Boomerang can be found
atdocs.soasta.com/boomerang-api/.
.
What I discovered from the html header.
This is clearly a bot detection network/javascript. With the help of this SO post:
Can a website detect when you are using Selenium with chromedriver?
And the second paragraph from that post:https://piprogramming.org/articles/How-to-make-Selenium-undetectable-and-stealth--7-Ways-to-hide-your-Bot-Automation-from-Detection-0000000017.html
I finally solved the issue:
we changed the
var_key in chromedriver to something else like:
var key = '$dsjfgsdhfdshfsdiojisdjfdsb_';
In addition I changed the code to:
import time
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import Select
from selenium.webdriver.chrome.options import Options
options = webdriver.ChromeOptions()
chrome_driver_path = "../chromedriver.exe"
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
options.add_argument('--disable-blink-features=AutomationControlled')
driver = webdriver.Chrome(executable_path=chrome_driver_path, options=options)
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.get('http://www1.nseindia.com/products/content/equities/equities/eq_security.htm')
driver.execute_script("document.body.style.zoom='zoom 25%'")
time.sleep(5)
price_volume = driver.find_element_by_xpath('//*[#id="dataType"]/option[2]').click()
time.sleep(3)
date_range = driver.find_element_by_xpath('//*[#id="dateRange"]/option[8]').click()
time.sleep(5)
series = driver.find_element_by_name('series')
time.sleep(3)
drop = Select(series)
drop.select_by_value("EQ")
time.sleep(4)
driver.find_element_by_name("symbol").send_keys("ACC")
actions = ActionChains(driver)
ez_download = driver.find_element_by_xpath('/html/body/div[2]/div[3]/div[2]/div[1]/div[3]/div/div[1]/form/div[2]/div[3]/p/img')
actions.move_to_element(ez_download)
actions.click(ez_download)
actions.perform()
#' essential because the button has to be loaded
time.sleep(5)
driver.find_element_by_class_name('download-data-link').click()
The code finally worked and the OP is happy.
I have edited the chromedriver.exe using hex editor and replaced cdc_ with dog_ and saved it. Then executed the below code using chrome driver.
import selenium
from selenium import webdriver
from selenium.webdriver.support.select import Select
import time
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument("--disable-blink-features")
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options)
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'})
print(driver.execute_script("return navigator.userAgent;"))
# Open the website
driver.get('https://www1.nseindia.com/products/content/equities/equities/eq_security.htm')
symbol_box = driver.find_element_by_id('symbol')
symbol_box.send_keys('20MICRONS')
driver.implicitly_wait(10)
#rd_period=driver.find_element_by_id('rdPeriod')
#rd_period.click()
list_daterange=driver.find_element_by_id('dateRange')
list_daterange=Select(list_daterange)
list_daterange.select_by_value('24month')
driver.implicitly_wait(10)
btn_getdata=driver.find_element_by_xpath('//*[#id="get"]')
btn_getdata.click()
driver.implicitly_wait(100)
print("Clicked button")
lnk_downloadData=driver.find_element_by_xpath('/html/body/div[2]/div[3]/div[2]/div[1]/div[3]/div/div[3]/div[1]/span[2]/a')
lnk_downloadData.click()
This code is working fine as of now. But the problem is that - this is not a permanent solution. NSE keeps on updating the logic to detect BOT execution in a better way. Like NSE, we will also have update our code. Please let me know if this code is not working. Will figure out some other solution.
Hellow there! I am using the wdio/cli so I created the wdio.conf.js with this command, then I start doing the test. But the issues is when have more than one test in a single or multiple test files.
In the test file I have something like this:
beforeEach(async function() {
$('~home').waitForDisplayed(81000, false);
});
Where home tag is a tag in the first view when the app runs in first screen. And appear this error:
element ("~home") still not displayed after 10000ms
So need to do kind of driver.resetApp()/ But dont know how to do it, what import do I need to do etc.
Did you try resetApp? You can't user driver as "main object" - Everything is under browser variable. Try this
//async
await browser.resetApp();
//sync
browser.resetApp();
Check Appium doc + wdio documentation.
Hy, I have some problems with the Go endpoints and Dart client library.
I use the Go library https://github.com/crhym3/go-endpoints and the dart generator https://github.com/dart-lang/discovery_api_dart_client_generator
The easy examples works fine. But they show never how to use time.Time.
In my project, I have a struct with a field:
Created time.Time `json:"created"`
The output in the explorer looks like this:
"created": "2014-12-08T20:42:54.299127593Z",
When i use it in the dart client library, I get the error
FormatException: Invalid date format 2014-12-08T20:53:56.346129718Z
Should I really format every time fields in the go app (Format Timestamp in outgoing JSON in Golang?)?
My research come to that the dart accept something:
t.Format(time.RFC3339) >> 2014-12-08T20:53:56Z
Second problem, if comment out the Created field or leave it blank. I get a other error:
The null object does not have a method 'map'.
NoSuchMethodError: method not found: 'map' Receiver: null Arguments:
[Closure: (dynamic) => dynamic]
But I can't figure it out which object is null. I'm not sure if I'm using the Dart client correct
import 'package:http/browser_client.dart' as http;
...
var nameValue = querySelector('#name').value;
var json = {'name':nameValue};
LaylistApi api = new LaylistApi(new http.BrowserClient());
api.create(new NewLayListReq.fromJson(json)).then((LayList l) {
print(l);
}).catchError((e) {
querySelector('#err-message').innerHtml=e.toString();
});
Does anyone know of a larger project on github with Go endpoint and Dart?
Thanks for any advice
UPDATE[2014-12-11]:
I fixed the
NoSuchMethodError
with the correct discovery url https://constant-wonder-789.appspot.com/_ah/api/discovery/v1/apis/greeting/v1/rest
The problem with the time FormatExcetion still open, but I'm one step further. If i create a new item, it doesn' work. But if I load the items from the datastore and send it back, this works.
I guess this can be fixed with implementing Marshaler interface, thanks Alex. I will update my source soon.
See my example:
http://constant-wonder-789.appspot.com/
The full source code:
https://github.com/cloosli/greeting-example