I am scraping a website that uses React for the front-end. So far it seems that I have to use their search form in order to arrive at the results page.
The problem is that the site clears out the search form's selected options from a dropdown (its state) every time the page is refreshed and therefore it makes scraping significantly slower. I know that it's working as intended, but if there was a way I could directly manipulate the state then it could speed up my scraper as opposed to re-selecting all the choices from the little buttons.
I don't think it uses any type of persistent storage or local storage at all, for every selection, otherwise the form probably wouldn't be cleared on refresh.
I can see that the years options for the form are always present in a data-attribute (data-years=["2017", "2016", ...]) but only for the years. And when a year (or any option from the dropdowns) is selected, a hidden field is populated with a value such as <input type="hidden" name="year" value="2017">.
Is this all that React uses for temporary storage (aka. state)—hidden fields?
And for the second part of my question, what type of event is fired off when there is a state change? How could I trigger it manually? When I select a year, for example, I want the form to give me the options for the next dropdown—given the year.
React does not use the DOM at all to maintain state. The example you provided is simply a poorly written React app. Normally everything will kept in memory (closured code so nothing in window/global) and React will update the DOM as she wants. :)
This means I don't think you'll be able to read/detect React instrinsic state changes from the outside. Interactive scraping should work like a user using the page, without any hint of what tech it's really using.
Depending on the technology you're using for scraping, you could indeed simulate or generate the real DOM events. When we need to write some end to end tests for a React app using the ubiquitous Selenium server, we normally have to manually click on buttons, options and so on and allow time for the React app to react accordingly and do its magic (like fetching more data and updating the page) and afterwards read document contents to verify everything was working. It's basically "scraping" with a desired output to verify, your test assertions.
If you're scraping static pages only (curl style: fetch the HTML and work your way with the original HTML response), I don't think you'll be able to handle a Javascript form. You need your scraper to be interactive.
Something like PhantomJS apart from the mentioned Selenium/WebDriver may help.
Related
I am new to react and developing the react application, where I want that user can save his/her selected values on the frontend and later on can also see those values via clicking a button and user can see what values have been selected previously. now the question is how to provide this functionality, I have studied that either it could be done by local-storage or by the data binding concept in react, but I have no clue which one is best to implement in my scenario.
Let me explain to you with the help of a diagram.
(The first Image with Indicator Generation)This is the main page of my React Application, under the section of "Question & Indicator" there is a thing called Associated Indicator, these are the values which user have select itself on the Frontend side.
(The second image with detailed user-controls) This is the page where all the user-controls are defined, here user can select the values and at the end when user clicked "Associate" it will be associated and the values have been shown under the section of Associated Indicators, from that user when clicking any of the associated indicators it will show all the values selected by user on front end.
Cheers.
Preserving data in a web app is not related to React -or any other front end technology specifically.
Data binding is a concept on how to update your views when certain data changes, and it is not related at all to how data should be saved. But related to how modern front end technologies like react and angular render your components.
You can save data in a web app using different methods, each one is suitable for certain use cases - and so you should use the one that suits you.
For example you can use any of the following:
Cookies: old way of saving data on the front end, can stay infinitely but can also be deleted if the user decided to delete them or clear data from his browser. Cookies Get send to the server-side with each request done with the browser, has limitations with size.
IndexedDB: This one is literally like a database inside the browser, really fast when querying data, also has bigger Size limitations than both Cookies, and LocalStorage. However, this is a really low level API. Won't recommend to use it, unless you really need its features. Can also stay infinitely, and can also be deleted by user.
LocalStorage: New browser API to persist data, much more easier to handle than Cookies And IndexedDB, Can stay infinitely, but can also be deleted by the user. Has bigger size limits than Cookies, but less than IndexedDb.
SessionStorage: Data saved for each tab, and data cleared when page session ends.
Finally If You want data that persist for forever and the user can't delete it, you have to do your own solution in the backend.
I have a table having millions of records. I am using Sails js as my server side code , React js to render data in view and Mysql as my DBMS. So what is the best method to retrieve the data in faster manner.
Like the end user does not feel like getting a huge amount of data which affects UI as well.
Shall I bring only 50 records first and show the pagination in bottom using pagination logic and then using socket.io fetch the rest in background ?
Or any good way of handling it ?
This really depends on how you expect your user to go through the data.
You will probably want an API call for getting only the first page of data (likely in such a way that you can fetch any page: api/my-data/<pagesize>/<pagenumber>).
Then it depends on what you expect your user to do. Is he going to click through every page to see all the data? In that case, it seems ok to load all the others as well as you mentioned. This seems unlikely to me, however.
If you expect your user to only view a few pages, you could load the data for the next page in the background (api/my-data/<pagesize>/<currentpage+1>), and then load the next page every time the user navigates.
Then you probably still need to support jumping to a certain page number, where you will need to check if you have the data or not, and then show a loading state (or nothing) while the data is being fetched.
All this said I don't see why you would need socket.io instead of normal requests (you really only need socket.io if the server needs to be abled to make 'requests' to the client so to speak)
I have a simple app built with AngularJS routes which is loading the controller and template for each path. I have a register form and login form on separates paths/templates. Say I go to the login form (/#/login) and enter my username/password, if I then hit "Register" (redirects me to /#/register), and then I hit back in my browser, it will return me to /#/login but the form will now be empty; the information I typed in has been removed.
Expected behaviour would be that the form data is still there.
Anyway to make that happen (without manually caching the data in a service)?
I'm guessing when the page changes, Angular is tossing the old template data and reloading the template again. Is there a way to instead cache that page template/DOM and reload it when the user returns to that path (instead of downloading and showing new template file)?
Well, this is a bit tricky. The browser should implement this kind of feature out of the box. Firefox started doing some work around this "issue" but I don't really know the current status of it.
Alternatively you can use a bit of javascript with LocalStorage to make this works. You're using AngularJS you can create a Directive that encapsulates this feature to be used on multiple places.
Basically you need to create a mechanism that translate an field to and unique-identifier and a value. Every time the user type on the field, you update the store. If the user "finish" the interaction on the form, you clean the value from the store.
You can also grab a jQuery plugin and just create a directive that uses the plugin.
https://github.com/kugaevsky/jquery-phoenix (never tested it).
TL:DR
There's nothing you can't do using a DOM property/attribute or something similar.
You'll need to get your hands dirty on some javascript to make this happen.
I've read a lot of threads here but can't find a real answer.
I'm building a desktop app that first loads a lot of json records (let's call them "cards"). Then the user can filter them down with by using many checkboxes, so he can "sum/substract" options (read: query vars).
I'm also trying to use routes.
So, I can have for example (and I don't really know if I'm doing it the right way):
app.com/#/cards/?near_address=London
app.com/#/cards/?near_address=London&cat=concerts
app.com/#/cards/?near_address=London&cat=concerts&day=8
app.com/#/cards/?near_address=London&radius=10000
[...]
Basically, every time the user change filters, I should add/remove query vars.
I could do it in many ways, for examble using some simple "state" json object, but I'm not sure it would be a good practice, mostly because I'm not really sure if I can instead do it maybe only by using routes (I'm quite new to the routes concept).
Is there any "good practice" in doing this kind of things with backbone?
thank you
Using routes to store state is actually a very good idea. That's what routes are there for, to store state, but it's up to you how granular you would want to go with them. some developers choose to go route per page and some choose to get more granular like your example and that's perfectly fine too. You just have to be careful not to go overboard and make your URLs too long and cryptic.
Using routes to store states gives your a few really important benefits:
Your pages will be refreshable. There is nothing more annoying than refreshing a page and losing all your progress and be take back to the start page of the app.
Your URIs are sharable. I could create a filter see the result and send the uri to anyone and they would see the same page as I was.
They make your life easier as a dev. which goes back to your pages being refreshable, they allow you to change your styles or scripts files and refresh the page and see the updated page without having to navigate through your app to get back to the same view over and over again.
Well, as technology progresses, issues we solved long ago crop up again.
Back in the dark ages, when PHP and ASP were considered awesome, we always had a problem with view states. If you had a page with say a dozen select combo boxes on it, your user chooses some combination and hits next, then realizes they screwed up and hit the back button on the browser, the combo boxes would be back in the default state, usually with option[0] selected. In order to prevent this, we had to write boatloads of boilerplate code that would save the state of those combo boxes to a cookie, or session variable, or something so that when the user hits the back button, we can reload the combo boxes back to the state they were in when they left.
This problem was compounded even further if you had a datagrid on the screen. Because then you would have to come up with some slick way of saving that grid somewhere to prevent from having to hit the database again.
Then came the light. Browser developers realized that most web developers were on the verge of going back to writing terminal programs in Cobol due to this issue and added UI caching to the browsers. This allowed us webdevs to not have to worry about this anymore except in odd situations.
So, life was good. Then someone came up with the bright idea of trying to replicate GWT without all the hassle and the web explodes with all these javascript frameworks. The one im dealing with specifically at the moment is AngularJS 1.2.10 with Angular-UI. I have until Friday (most likely wednesday tho) to make an initial assessment on if this technology is a viable alternative to our current standard (thats pretty much universally hated) JSF.
So, i follow some guides, pound my head against the desk a few times, and I have an angular app with 3 actual HTML pages, each HTML page with 2 views.
Before you go there, understand we can't use it unless we can do multi-page JS apps. Some of the applications that this will be worked into have been in development for a decade or more and its simply not financially practical to scrap an the entire UI and start over again. We would instead be doing things like taking these 50 struts pages and converting them to angular/rest and linking them seamlessly back into the remaining 800 struts pages of the application.
So in my exercise of playing with this, I encounter my old nemesis. Back button view state issues.
I have been playing with the UI-route system. The fact that I can deep link using the route system solves part of my problems. But, if say I have a search page like this:
view-search
combo: search type [member,nonmember]
combo: result type [detail,summary]
combo: search state {all the states]
textbox: contract number
etc etc etc
And various combinations of combo box selections and text entries comes up with a list of 1000 people. Now the user selects one of those people on the data grid and it takes you to view-detail. Well the fact that you can use routing to do something like index.html#detail/bob is cool, but if the user realizes thats the wrong bob and hits the back button, they get a blank search screen again and they have to enter everything over and worse yet, send another search to the database to rebuild the datagrid. Some of these screens have 50 or more options to choose from when searching for data so trying to put all of them into the URL routing sounds completely impractical to me.
Now in my research I found this post:
Preserve state with Angular UI-Router
And that has promise mainly because I have a view state object that I can store into a Redis database or a session EJB for cases when the user actually jumps out of angular and into the legacy Struts application, then back buttons back into the angular application, but the fact still remains that on some of these pages, that is a huge amount of boilerplate code that we would have to write in order to make it work.
I don't really mind the idea of having to manually save off the view state object and read it back in from a Redis server or something anytime a user enters or leaves an HTML page in the system. What i'm really looking for is a way to automatically generate the object that is to be saved without having to write volumes of boiler code.
Is this possible? I keep reading the ui-route documentation but it doesn't look like this is addressed, at least not that i've translated yet.
If this is possible, what controls should I be looking at?
thanks
-------------- Edit
I just thought of something. There is one central scope to each of the single page applications. (Im basically going to be building a multiple single page apps and hooking them together) So if i use a naming convention, something like this
$scope.viewstate.view-search.searchType
$scope.viewstate.view-search.resultType
$scope.viewstate.view-search.searchState
Then the viewstate object should simply be a js array and when I create a function to move to struts.do, i can simply save that array off to the Redis server as a nested map object. Then when my user back buttons back into the angular app, i can capture that using the route system and retrieve that viewstate object from Redis and insert it back into my scope, thereby rebuilding the scope for the entire single page app in one shot.
Would that work?
I believe that you have a very complicated issue of trying to keep the view states between your varying pages with the amount of data in your pages. I think that the only real effective way to do this is to write an angular service that you can then pass to your various pages. as You already know the service is a singleton that you can use in various controllers and could be utilized to maintain the view state as you described. here take a look that this link and see if it will help: http://txt.fliglio.com/2013/05/angularjs-state-management-with-ui-router/
After some thought what you suggest in your edit might work, but I would still use a service to retrieve that array of data, as it would make it easier to reinsert in to angular scope
I am exploring something similar for an Angular app that I am writing. Keeping a user login during a page refresh is easy. Displaying the state on the page after a refresh is an entirely different problem.
How long must the state be persisted? I'm evaluating two possibilities.
First, saving the state (current form values or whatever) to the database. As the page changes, incrementally save the state to the database. On a browser refresh check the database for saved values.
Second is to use local browser storage. This is 5 megs of storage. 5 megs is a lot of text. Again this data would incrementally be saved into storage. When the browser refreshed, simply load data from localStorage.