I am trying to generate a sitemap for my react js website using Sitemap Generator extension of googlechrome. (https://chrome.google.com/webstore/detail/sitemap-generator/hcnjemngcihnhncobgdgkkfkhmleapah?utm_source=chrome-ntp-icon). It says it works with reactjs applications. But in my case it doesnt move beyond the base URL. I tried with multiple extensions same behavior is seen with them as well. Check the image below where it shows its stuck on the base link.
Sitemap generator Output Screen
Any idea why this might be happening.
This was happening because I was saving my access_token in localStorage and all the pages were protected. The crawler was not able to access the token and therefore was not able to crawl through the other pages. I solved this by storing the token in redux store. So that it would be available to the crawler as well.
This is a solution I could think of. Please in case you have a better solution to this please let it be known here :) Especially if the project doesn't have redux store how would we solve this. Thank you.
Related
Does anyone have any pointers on how to go about adding a /docs page for website documentation to a next.js app? I've looked up Docusaurus but it seems like it's already a react app itself. Is there a way to integrate it inside an existing next.js app or are there other solutions?
Many Thanks
One idea might be to intercept the request and send the html file that docusaurus builds out, and putting all other files in the public folder.
https://medium.com/wesionary-team/render-html-file-in-pages-of-next-js-59281c46c05
Also checkout this discussion about it.
https://github.com/vercel/next.js/discussions/12373
I have done this with React apps using express. But never with Next. At first it looks like it would be possible with multi-zone in Next but that doesn't seem to do the job. So my other recommendation would be to try to use a docs.domain.com instead and host it separately. Then you have a /docs url or a button that redirects to the doc domain instead.
Firebase has free hosting and allows you to setup multiple sites. So it should be fast to test this setup there
I'm going to actively try to get this to work with Next myself but I do not think it will work because of how they are developed. So I would do the above recommendation and if I find a workaround, I'll post an update.
The app
The application was made using ReactJS, React Router Dom, Styled Components and Redux ducks.
The backend we consume is also made by us using Amazon Amplify and GraphQL.
The goal
We need to define the meta tags of one of the application pages so that it is possible to share personalized links to users
in social networks using OpenGraphic meta tags and the like.
The problem
The project was made in ReactJS and ReactJS has only one HTML page as root (/public/index.html), in this way, everything is generated with Javascript in a root tag, and when it arrives in the browser it is transpiled, as we already know. The problem is that the crawlers responsible for understanding the meta tags are not able to understand Javascript and end up not finding the dynamic data that I am defining on the page that I need to share the link on. They understand that there is one html file and only.
Attempts to resolve the issue
1) Define the meta tags in the /public/index.html file itself
This solution doesn't work because the data we are using is dynamic and the index.html file is a static file
2) Using react-helmet
The solution allows meta tags to be defined, but as already mentioned, crawlers don't understand JS. So, despite being on the page, the meta tags do not appear when sharing the link.
3) Using some SSR technology
This is a possible solution, but we were unable to integrate any SSR Framework into React. And it is not feasible to change the base technology of the project. We can't just switch from React to Next, for example, as the project is already complete.
4) Using a small server made with express.js along with the React application to replace the meta tags in index.html with string.replace() simulating something like an SSR
This solution works, but it causes two requests to be made every time the page is accessed, once by express.js and once on the front-end side by React. Due to the number of requests increasing, this solution was discarded. But if necessary, you can do it. In this case it is also necessary to check if Amplify can keep the application and the small server running in the same project.
5) Using react-snap with react-helmet
React-snap allows you to create html snapshots of the pages of a React project based on their routes and links, this added to react-helmet generates a perfect solution for links to be treated well by web crawlers when they are shared. But the solution doesn't work with dynamic routes. For example, /your-route/:id is a dynamic route that expects an id to be fully defined. React-snap gets lost when trying to create a snapshot of a route that only exists when the id is set. Unfortunately, this solution doesn't work.
These were the solutions we used to try to solve the problem, but it was not possible yet. Probably attempt 4 would be the most ideal to solve the problem. But we are looking for the best way that will not generate reworks and future problems. If someone knows a better way to do that, would help us a lot!
I'm pretty new to react and building out a little prototype using Firebase as a backend. One of the primary functionalities involves a user writing a post in an editor, which is saved to firebase. On submit in the editor, I am trying to create a new standalone page for the post with the firebase uid as the ending part of the new unique URL.
The problem I'm having is figuring out a way to create the new page on submit. I haven't been able to find any documentation for a similar problem like this specific to react or firebase, and was just wondering on a high-level what a good approach to executing this might be? Thanks
The newly launched Firebase Hosting + Cloud Functions integration can help here. The first bullet point in the documentation looks like it describes your use case exactly.
You also mentioned React. There is a handy sample project showing how to implement an isomorphic React app with Firebase.
I tried using OWASP in my angularjs web app, however.. none of my routing pages ever get lists in the spider.. can someone give me a detailed explanation on how to do this? I just want to see the vulnerabilites of all routing URLS in my angularjs app
ZAP will spider from the base URL you give it. Make sure that it can get to your routing pages by traversing the paths of available links that start on your base URL. If ZAP can't get to them by traversing your site from the base URL, create a simple sitemap or listing of all of your URLs and link/insert that on your home page.
Also, this is a handy way to get ZAP up and running via Docker.
https://github.com/zaproxy/zaproxy/wiki/Docker
HTH
AngularJS makes heavy use of JavaScript, which the 'traditional' spider doesnt handle very well.
Instead use the ajax spider: https://github.com/zaproxy/zap-core-help/wiki/HelpAddonsSpiderAjaxConcepts this is slower but will handle any JS as it launches browsers to crawl the site.
Have read up on angular and generating snapshots for the crawlers
Just want to know if the snapshot needs to have CSS or just plain and img tags for Facebook and Google etc. to work.
Any good examples of how to implement the framework with expression engine or Wordpress.
As I understand it you would actually create two versions of every page. Is that correctly understood ?
Cheers looking forward on trying it out .
As I understand it it's enough to modify the htaccess file to show the snapshots instead of the angular once when the request is coming from Facebook or Google etc.
Thanks