I am trying to scrape a React based website with selenium and python, and i came to a point where, although i can retrieve all that is being 'seen' by Google DevTools Inspector, i am not being able to find the link to the next page i needed to scrape. I could, i guess, do this, in a way where i could click every single button to the next page, although, i was kind of curious to know why selenium has a problem is seeing this particular key and how to workaround this, since i have to build a database and any 'extra' request will add add up exponentially.
Google DevTools Inspect View
As you can see, there is no 'key' or href class anywhere on the tab, but if i look in React DevTools, it is there:
React DevTools Inspect View
So my question is: is there any way i can retrieve those 'keys'?
Are there any better tools to do this job?
Thank you in advance!
Related
I'm working through a React video from 2019 and their React Developer Tools extension (in Safari) looks like this:
However, my React Developer Tools in both Firefox and Chrome extensions look like this:
One can see here that the presenter in the video is getting much more information about the React component than I am getting:
Is there a way to see the elements as in the old version of the extension? Or does one have to use React Developer Tools in a different way now?
Inline props in tree have been removed for better performances, however, you can still get an element's details by selecting it:
Also, for a more detailed DOM tree view, it can be displayed by clicking on the eye icon on the top-right which says "Inspect the matching DOM element".
Another thing I noticed is that selecting a component automatically make the $r variable refers to it.
appium logs 2nd part
appium logs first part
UI Automator showing element properties
UI Automator showing all element properties
Summary:
Appium is unable to identify any element on particular page of My Mobile native app. I have used all the possible locators for different elements shown on that page with help of UI Automate Viewer.
Used different locators (by id/xpath/contains/accessibility/UIAutomate).
Verified that the page belongs to native app only with no Web View.
applied different types of sufficient wait for the page.
driver.get.pagesource not working for the page.
Can any one please help in same, the below ways I have opted for finding the solutions but nothing has worked till now.
You say a native apps, but seen from the picture, that you facing with like a Maps application (this is web apps, my assumse), often in testing things like this happen, like you have to do a switch context. When you are on this page please try this :
driver.context("WEBVIEW_1");
And this is for switch again to native :
driver.context("NATIVE_APP");
Please read this reference and this.
I'm trying to scrape a web made with Angularjs, using java.
I use Selenium and a ChromeDriver to tramping the web, and, to know the next step I use devtools of Chrome.
In a 'traditional' website I can guess easily the id of the item or where a button goes, looking in Elements or Source tag but, how can I do it in a web made with angular?
I mean, where I can found the id, href property or where a button goes in an angular web? Can I find it using devtools of chrome or I need to install something?
Thanks
First of all, what your doing is probably illegal.
But to give you the benefit of the doubt, and guess that you are doing to a website that belongs to you but you didn't write, or don't have the code, or you have permission from the website owner you have two options.
Extension.
You can use this chrome extension, which lets you inspect your AngularJS in the chrome debugger tools.
Then you can check where the ng-click leads to, and look for that function on the scope.
Console.
Select the element you are interested in inspecting, and since $0 returns the current selected element in the DOM you can write in the console:
angular.element($0).scope()
which will return you an object with all the data on the current scope.
Note that you might have to go up the $parent, to find the function you are looking for.
P.S. if you are looking for how to do the same thing in Angular, you can use the following extension (thanks #user1767316) or in the console ng.probe($0).componentInstance
I am making a website using AngularJS, I am curious to know that is there any disadvantage of hash in url with respect to seo ?
e.g. http://www.website.com/#about-us
I'll appreciate any contribution.
Thanks
If we go back to the basics, HASH # means a DIV ID in your HTML, and to talk in more details Google ignores anything after the HASH.
Example, this page www.mydomain.com is similar to www.mydomain.com/#about-us
This is an advanced technique some marketers are using it to track their campaign without using parameters like UTMs to avoid content duplication.
To make sure your page is loading without any errors, try to disable the JS from your browsers using "Web Developer Tool" and then load your page, i think you will get a white page without content and this is the way Google and most of the search engines see your pages.
Also there is another way to test it by going to Search Console "Webmaster tool" and use fetch as Google, here you will see exactly how Google view your page.
Is there a possible way to automate the click event of angularjs button, which navigates to a new page in turn.
Something like following is what the original browser code is.
"data-ng-click="services.urls.openPageUrlWithId(plusPageId)"
I am able to get to the point where the click event comes into picture, but it is not getting navigated to a different screen.
Any solution to this problem ? Thanks.
Is this for end-to-end testing purposes?
If so, the angular team themselves maintain a fantastic tool called protractor https://angular.github.io/protractor/#/
It provides smart abstractions on top of WebDriverJS/Selenium to allow you to easily find elements by their angular bindings and click through as you have described.