Is it possible for Google to render SPAs (angular in particular) without the use of a headless browser?
I have built a service purely in angular but I would like to make it possible to apply SEO.
Now Google crawlers are able to render AJAX (Angular) sites: http://googlewebmastercentral.blogspot.com/2015/10/deprecating-our-ajax-crawling-scheme.html
But you can implement AJAX crawling scheme (deprecated): https://developers.google.com/webmasters/ajax-crawling/docs/learn-more
First of all, no, you can't render an Angular page without the aid of an actual browser. It's something that is coming in Angular 2 but for now we are stuck using PhantomJS or some headless Chrome or something if we want to create HTML snapshots.
However, as Alex already pointed out that is no longer needed for SEO reasons!
But, wait, there's more
The reason I write my own answer is to point out that it IS still needed for Facebook, Twitter, LinkedIn etc :(
If a user shares a link to your site on Facebook (or Twitter, etc) their crawler will try to read your page and generate a preview. Unlike search engines these crawlers will not run Javascript, so the preview of your Angular page will look all broken.
If you want your previews to be dynamic you will still need the html snapshots for now.
Related
I tried using OWASP in my angularjs web app, however.. none of my routing pages ever get lists in the spider.. can someone give me a detailed explanation on how to do this? I just want to see the vulnerabilites of all routing URLS in my angularjs app
ZAP will spider from the base URL you give it. Make sure that it can get to your routing pages by traversing the paths of available links that start on your base URL. If ZAP can't get to them by traversing your site from the base URL, create a simple sitemap or listing of all of your URLs and link/insert that on your home page.
Also, this is a handy way to get ZAP up and running via Docker.
https://github.com/zaproxy/zaproxy/wiki/Docker
HTH
AngularJS makes heavy use of JavaScript, which the 'traditional' spider doesnt handle very well.
Instead use the ajax spider: https://github.com/zaproxy/zap-core-help/wiki/HelpAddonsSpiderAjaxConcepts this is slower but will handle any JS as it launches browsers to crawl the site.
Is it safe to use<link rel="canonical" ref="{{canonical_url}}" /> in an angularjs page, or is there a preferred way to handle this?
Google is now automatically crawling and rendering javascript, which is great for angularjs sites. But, I'm concerned that Googlebot may not wait for the rendering when deciding about canonical pages, and I don't want to mess up our site by having Googlebot think all pages are "url".
Using Google's Webmaster Tools, I can see that Google's crawler can render the pages just fine, but I'm not sure how to tell how it's dealing with a canonical tag. Other reading implies that Googlebot stops reading/rendering a page if it sees a canonical tag for another page it has already processed.
As #JB Nizet mentioned, as I witnessed, Google as of Sep 25 2015 executes JS, render it properly in webmaster but more like "for demo". En mass production links still not executed RELIABLY by Google ( sometime not at all, sometime quite good).
So we still have to use prerender.io (which is quite good). But attention: use the one with phantomjs2, phantomjs1 is not good at parsing/reading angular apps.
IMPORTANT things: they do support httpCode, and all your double curly braces.
Good luck!
Source: my painful days setting up SEO for our angular platform
Google will render and index plain SPA without any static html snapshots.
BUT: i have a project with change over to https. usually i dont redirect with 301 but just add https in the canonical tag. on this site google has not yet recognized the change.
Same time on another project, same changeover to https with same code but with html snapshots included:
google has recognized the change to https and right version is in in the search results.
i wrote my own directive to have a nice seo header with canonical tags. its better to use your pages head tag as directive call.
https://github.com/w11k/w11k-angular-seo-header
If you want to prerender your site do not use a moron service like prerender but simply add a task to your build process. Setup with Grunt/Gulp is about 15 mins.. once!
I'm deciding whether it's safe to develop my client-facing app in AngularJS using pushState.
I've read that when using pushState in an AngularJS app, we don't need to worry about Googlebot because it can now execute enough JS to produce an HTML snippet for itself. But then I wonder about Bing, Facebook and other bots and scrapers. The tutorials I've seen for making AngularJS SEO-friendly all deal with apps that use hashbangs (#!). These don't apply to me since I'm not using hashbangs.
Does anyone have insight into this problem? What are some methods for ensuring an AngularJS app that uses pushState is SEO-friendly and Social-scraper-friendly? If you use a service like Seo4Ajax or prerender.io I'd appreciate your thoughts on it.
Note: As I understand it, when developing single page apps in the last couple of years it has been necessary to send HTML snippets to SEO crawlers. This was accomplished by using hashbangs and a meta tag that let Google, Bing and Facebook know that it needed to replace the bang (!) with an _escaped_string when making a request. On the server you'd listed for requests with _escaped_string and deliver the appropriate HTML snippet using a tool to generate HTML snippets like phantomJS.
Now that we have pushState, I don't see how we indicate to javascript-less bots what part of the URL to rewrite with an _escaped_string or even if it's necessary. I'm having trouble finding any information beyond "you're site will be okay with google ;)".
Here are some other SO questions that are similar but have gone unanswered.
Angularjs vs SEO vs pushState
.htaccess for SEO bots crawling single page applications without hashbangs
Here's a solution I posted in that question and am considering for myself in case I want to send HTML snippets to bots. This would be a solution for a Symfony2 backend:
Use prerender or another service to generate static snippets of all your pages. Store them somewhere accessible by your router.
In your Symfony2 routing file, create a route that matches your SPA. I have a test SPA running at localhost.com/ng-test/, so my route would look like this:
# Adding a trailing / to this route breaks it. Not sure why.
# This is also not formatting correctly in StackOverflow. This is yaml.
NgTestReroute:
----path: /ng-test/{one}/{two}/{three}/{four}
----defaults:
--------_controller: DriverSideSiteBundle:NgTest:ngTestReroute
--------'one': null
--------'two': null
--------'three': null
--------'four': null
----methods: [GET]
In your Symfony2 controller, check user-agent to see if it's googlebot or bingbot. You should be able to do this with the code below, and then use this list to target the bots you're interested in (http://www.searchenginedictionary.com/spider-names.shtml)...
if(strstr(strtolower($_SERVER['HTTP_USER_AGENT']), "googlebot"))
{
// what to do
}
If your controller finds a match to a bot, send it the HTML snippet. Otherwise, as in the case with my AngularJS app, just send the user to the index page and Angular will correctly do the rest.
Supposedly, Bing also supports pushState. For Facebook, make sure your website takes advantage of Open Graph META tags.
Just really curious about this,is github coded using angularJS in the front-end? I've noticed that the pages almost never reload, just a bunch of behind the scene requests. It acts more of a web app than a web page.
I don't have anything to do with this site, but I believe it serves your purpose:
http://builtwith.com/github.com
It allows you to enter any site, and it attempts to scrape production details for you.
No, you can see by yourself:
Angular.js is just a very good framework around javascript, whatever you can do with angular you can do with vanilla javascript or with other framework (but with more pain).
They use "PJAX" for the seamless transitions:
pjax is a jQuery plugin that uses ajax and pushState to deliver a fast browsing experience with real permalinks, page titles, and a working back button.
https://github.com/defunkt/jquery-pjax
(seems its now pjax2: https://github.com/github/pjax)
Github uses github so you can check out a lot of their libraries at https://github.com/github
Have read up on angular and generating snapshots for the crawlers
Just want to know if the snapshot needs to have CSS or just plain and img tags for Facebook and Google etc. to work.
Any good examples of how to implement the framework with expression engine or Wordpress.
As I understand it you would actually create two versions of every page. Is that correctly understood ?
Cheers looking forward on trying it out .
As I understand it it's enough to modify the htaccess file to show the snapshots instead of the angular once when the request is coming from Facebook or Google etc.
Thanks