I have a site that is using prerender.io. Last year it was all working smoothly. This year I had to change the site and I moved it from a .net project to a pure AngularJs application (using yeo angular).
This was done a couple of months ago. What I have recently found, is that prerender.io is not caching my pages anymore.
I found that my web.config has changed. I added this rule:
<rule name="SEO" stopProcessing="true">
<match url="^(?!.*?(\.js|\.css|\.xml|\.html|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.woff|\.ttf|\.m4v|\.svg|\.torrent))(.*)" ignoreCase="false" />
<conditions logicalGrouping="MatchAny">
<add input="{HTTP_USER_AGENT}" pattern="baiduspider|facebookexternalhit|Facebot|twitterbot|googlebot" />
<add input="{QUERY_STRING}" pattern="(.*)_escaped_fragment_=(.*)" ignoreCase="false" />
</conditions>
<action type="Rewrite" url="http://service.prerender.io/http://www.kudos-sports.com/{R:0}" appendQueryString="false" />
<serverVariables>
<set name="HTTP_X_PRERENDER_TOKEN" value="****" />
</serverVariables>
</rule>
This is now allowing prerender to cache my site, but it doesn't cache it properly.
If you go to https://www.kudos-sports.com/?_escaped_fragment_= and have a look, you can see that it just pulls in the text that is on the index page.
It doesn't load the ui-view contents :(
Does anyone know why?
Google has already deprecated _escaped_fragment_ support.
Reference:
https://webmasters.googleblog.com/2015/10/deprecating-our-ajax-crawling-scheme.html
https://medium.com/finnovate-io/googlebot-no-longer-picking-up-content-in-prerender-io-pages-ae21d9710459
Related
I have the following in my web.config
<configuration>
<system.webServer>
<httpErrors errorMode="Detailed" />
<rewrite>
<rules>
<rule name="Angular" stopProcessing="true">
<match url="^(.*)$" ignoreCase="false" />
<conditions logicalGrouping="MatchAll">
<add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" pattern="" ignoreCase="false" />
</conditions>
<action type="Rewrite" url="index.html" appendQueryString="true"/>
</rule>
<rule name="Redirect to www" stopProcessing="true">
<match url="(.*)" />
<conditions trackAllCaptures="false">
<add input="{HTTP_HOST}" pattern="^mydomain.com$" />
</conditions>
<action type="Redirect"
url="{MapProtocol:{HTTPS}}://www.mydomain.com/{R:1}" />
</rule>
</rules>
<rewriteMaps>
<rewriteMap name="MapProtocol">
<add key="on" value="https" />
<add key="off" value="http" />
</rewriteMap>
</rewriteMaps>
</rewrite>
</system.webServer>
</configuration>
I have the following in my assets in my angular.json
"assets": [
"src/favicon.ico",
"src/assets",
"src/web.config"
],
But my app does not redirect.
The issues is i have a App Service managed cert setup for www.mydomain.com
I have two custom domains setup
www.mydomain.com (this one allows me to add the binding to the app service cert)
mydomain.com (is unbound as I cant add a cert)
So I thought I could fix it with the redirect to always go to www. but it does not seem to work
UPDATE
You can RewriterConfig to set url rewrite function in web.config file. More details you can see my answer in anothor post .
PRIVIOUS
The Web Server integrated by App Service cannot have full control, which is one of the attributes of PaaS products.
App Service can only use some functions of IIS. It needs to be configured in the web.config file under /site/wwwroot (that is, the root directory of the project you develop).
All available IIS functions can only be configured through the web.config file.
solution:
• Try to add redirect rule in the web config file. If it fails, you can only use the rewrite feature of the application gateway.
• The following are some documents of Application Gateway about rewrite feature for your reference.
Redirection
Redirect web traffic
Troubleshoot--App service issues
I am using react-router and have put it in it, but I noticed that when I start at the landing page and go through all my routes it works. If I just type in a path url I get a 404.
Am I guessing this has something to do with the server not knowing how to pass it off to my client routes?
I found this post but I am a bit confused with the answers as some say to update the web.config file but I don't have one as it is a Reactjs only.
Edit
Here is what I have now for my rewrite
I am now getting a white page when I try to navigate to a URL that is not the index page.
When I look at my console. I see
Uncaught SyntaxError: Unexpected token <
which goes to main.js and complains about this "<!DOCTYPE html>" it almost seems as my index.html was merged into the javascript file.
The key to getting React-Router to work with IIS is to setup URL Rewrite rules. After looking at how some others have setup AngularJS SPA apps with IIS, I have come up with the following solution.
Download and install URL Rewrite on your server (development and production)
Setup rule to catch any url routes that ARE NOT files and ARE NOT directories. Optionally, you can negate any actual directories that you want serve regularly. More info can be found here on Microsoft's Documentation
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<rewrite>
<rules>
<rule name="ReactRouter Routes" stopProcessing="true">
<match url=".*" />
<conditions logicalGrouping="MatchAll">
<add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
<add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" />
<add input="{REQUEST_URI}" pattern="^/(docs)" negate="true" />
</conditions>
<action type="Rewrite" url="index.html" />
</rule>
</rules>
</rewrite>
</system.webServer>
</configuration>
In your base html page (index.html) add a tag with a href attribute to your app.
<base href='/path/to/my/app'/>
To expand a little bit you do need the URL rewrite package as Mohit explained but you don't need to write your own web.config file, you can do it straight from IIS Manager if you want.
Once you have URL Rewrite installed, you may have to reboot or restart IIS, you should see it on the dashboard for the sites. I do it at the site level and not the server level.
In there you are going to add a rule.
It must Match the pattern for the request URL using regular expressions with the pattern .*
This will match all urls sent by the browser.
In the conditions you will need two conditions that must Match All. There are two inputs you need:
{REQUEST_FILENAME} is Not a File
{REQUEST_FILENAME} is NOT a Directory
No pattern is needed on these.
Finally in the Action section you need to Rewrite with the rewrite url to be / appending the query string. This will keep everything being sent to react-router and not absorbed by the server.
At this point you might want to stop processing more rules unless you have other unrelated business logic that also needs to run.
This will configure IIS to send all requests to the root of the site which should be one of the default documents as setup in IIS, most likely index.html. From there React-router will pick it up.
Some remarks to Mohit Tilwani answer
say your url is
http://localhost/mySubDir
in web.config add rules to prevent rewrite of js, css files:
<system.webServer>
<rewrite>
<rules>
<rule name="ReactRouter Routes" stopProcessing="true">
<match url=".*" />
<conditions logicalGrouping="MatchAll">
<add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
<add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" />
<add input="{REQUEST_URI}" pattern="^/(docs)" negate="true" />
<add input="{URL}" negate="true" pattern="\.js$" />
<add input="{URL}" negate="true" pattern="\.css$" />
</conditions>
<action type="Rewrite" url="index.html" />
</rule>
</rules>
</rewrite>
</system.webServer>
in index.html
<base href="%PUBLIC_URL%/" />
in package.json add the line:
"homepage": "http://localhost/mySubDir",
in index.js you add the subDir part
const baseUrl = '/mySubDir/';
const rootElement = document.getElementById('root');
ReactDOM.render(
<BrowserRouter basename={baseUrl}>
<App />
</BrowserRouter>
</ThemeProvider>
</Suspense>
</StoreContext.Provider>
,
rootElement);
I have this web.config file into some of my PROD pages using Plesk as a Host. I've manually added this web.config to the React page root folder, hopefully, this might work with any IIS Server:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.web>
<compilation defaultLanguage="c#" />
</system.web>
<system.webServer>
<httpErrors errorMode="Custom" existingResponse="Replace">
<remove statusCode="404" subStatusCode="-1" />
<error statusCode="404" path="/" responseMode="ExecuteURL" />
</httpErrors>
<tracing>
<traceFailedRequests>
<add path="*">
<traceAreas>
<add provider="ASP" areas="" verbosity="Verbose" />
<add provider="ASPNET" areas="AppServices,Infrastructure,Module,Page" verbosity="Verbose" />
<add provider="ISAPI Extension" areas="" verbosity="Verbose" />
<add provider="WWW Server" areas="Authentication,Cache,CGI,Compression,FastCGI,Filter,Module,RequestNotifications,RequestRouting,Rewrite,Security,StaticFile,WebSocket" verbosity="Verbose" />
</traceAreas>
<failureDefinitions statusCodes="500" />
</add>
</traceFailedRequests>
</tracing>
</system.webServer>
</configuration>
After updating the web.config, I had a different issue when refreshing the page it was lead to a blank page with a console error as
here
Able to resolve the issue by adding the below tag to the Head section of index.html
<base href="https://mydomain/" target="_blank">
We have a website coded with AngularJS. Since much of the site is generated via javascript, search engines can't index the pages. So, we have setup Prerender.io to index our site. It does that fine and I can view the rawhtml from their site.
If I enter the following into a browser, Prerender.io will display the rawhtml correctly:
http://service.prerender.io/http://www.swiftlearning.com/?_escaped_fragment_=/login
The problem is that I can not get IIS 6 to send the bot requests to Prerender.
When I take this URL: http://www.swiftlearning.com/#!/login and replace the #! with ?_escaped_fragment_= resulting in http://www.swiftlearning.com/?_escaped_fragment_=/login
IIS returns the initial web site page and displays the following URL:
http://www.swiftlearning.com/?_escaped_fragment_=/login#!/home
I have setup Wireshark to capture the traffic. It shows that the request comes in and then the response comes from the website, not prerender.io.
I created a web.config file (with what I have already found on StackOverflow) with the following configuration.
<?xml version="1.0"?>
<configuration>
<system.web>
<httpProtocol>
<customHeaders>
<add name="X-Prerender-Token" value="dTaPu5H97XTS618Y8edm" />
</customHeaders>
</httpProtocol>
<httpModules>
<add name="Prerender" type="Prerender.io.PrerenderModule, Prerender.io, Version=1.0.0.2, Culture=neutral, PublicKeyToken=null"/>
</httpModules>
<rewrite>
<rules>
<!--# Only proxy the request to Prerender if it's a request for HTML-->
<rule name="Prerender" stopProcessing="true">
<match url="^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent))(.*)" ignoreCase="false" />
<conditions logicalGrouping="MatchAny">
<add input="{HTTP_USER_AGENT}" pattern="baiduspider|facebookexternalhit|twitterbot" />
<add input="{QUERY_STRING}" pattern="_escaped_fragment_" ignoreCase="false" />
</conditions>
<action type="Rewrite" url="http://service.prerender.io/http://{HTTP_HOST}{REQUEST_URI}" appendQueryString="false" />
</rule>
<rule name="Redirect To Index" stopProcessing="true">
<match url=".*" />
<conditions>
<add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
<add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" />
</conditions>
<action type="Rewrite" url="/" />
</rule>
</rules>
</rewrite>
</system.web>
</configuration>
I hope that I haven't confused the issue with my explanation. Any help will be appreciated.
Thanks,
Dana
Well, it turns out that IIS 6 does not handle redirects. Redirects became possible with IIS 7. So, it boils down to upgrading our server, which for various reasons is highly suggestible, or some hack.
Thanks for looking at the question.
I can not select this as the answer until tomorrow.
Dana
I've a issue with Cakephp Version 3.2 on IIS. But, Cakephp Version 2.2 is working well in the same setup environment. Not sure why?
Here's my web.config file.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<rewrite>
<rules>
<rule name="Exclude direct access to webroot/*"
stopProcessing="true">
<match url="^webroot/(.*)$" ignoreCase="false" />
<action type="None" />
</rule>
<rule name="Rewrite routed access to assets(img, css, files, js, favicon)"
stopProcessing="true">
<match url="^(img|css|files|js|favicon.ico)(.*)$" />
<action type="Rewrite" url="webroot/{R:1}{R:2}"
appendQueryString="false" />
</rule>
<rule name="Rewrite requested file/folder to index.php"
stopProcessing="true">
<match url="^(.*)$" ignoreCase="false" />
<action type="Rewrite" url="index.php"
appendQueryString="true" />
</rule>
</rules>
</rewrite>
</system.webServer>
</configuration>
The above content was copied from the Official Cakephp forum.
Whatever we type in URL, I get the below response
Note :
I tried all the related links in SO & I posted this after trying all the provided solutions but none of them worked.
More Information :
IIS Ver - 10
Cakephp - 3.2
I repeat again, Cakephp 2.2 works well.
I would like to answer my own question on how i solved the problem. The problem is not with web.config but the real problem is in .gitignore.
By Default it adds vendor directory to .gitignore. Since it doesn't added to git, those errors were occurred.
URL not redirecting to Prerender.io when sharing the url from facebook.
My site is http://www.vbuy.in/ , which is in demo and I am trying to share one of the url in facebook for example "http://www.vbuy.in/product/samsung-galaxy-grand-2-black-/1975"
Title and Description are showing from the home page (I added all those supported ng-content, and in script i registed the viewtitle etc..
I wrote a URL rewrite and also tested in https://developers.facebook.com/tools/debug/og/object/.
But, for me it looks like its not redirecting to prerender.io url. What Am I missing here? . I use IIS and also downloaded ARR and enabled proxy there.
I am having tough times for the last one week with SEO & social sharing part. Even before starting the project we were aware of SEO issues in angular js, but we thought when we go
live, we can use prerender or any pre render static html support. ( we didnt evaluated in depth earlier, due to time & resource constraints)
My code .. (Which is copied from other Stack overflow solutions.
<httpProtocol>
<customHeaders>
<add name="X-Prerender-Token" value="XXXXX....." />
</customHeaders>
</httpProtocol>
<rewrite>
<rules>
<rule name="Prerender" stopProcessing="true">
<match url="^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent))(.*)" ignoreCase="false" />
<conditions logicalGrouping="MatchAny">
<add input="{HTTP_USER_AGENT}" pattern="googlebot|baiduspider|facebookexternalhit|twitterbot" />
<add input="{QUERY_STRING}" pattern="_escaped_fragment_" ignoreCase="false" />
</conditions>
<action type="Rewrite" url="http://service.prerender.io/http://www.vbuy.in/{R:1}" />
</rule>