crawling all links of same domain in Nutch - solr

Question

Can anyone tel me how to crawl all other pages of same domain.
For example i'm feeding a website http://www.techcrunch.com/ in seed.txt.
Following property is added in nutch-site.xml
<property>
<name>db.ignore.internal.links</name>
<value>false</value>
<description>If true, when adding new links to a page, links from
the same host are ignored. This is an effective way to limit the
size of the link database, keeping only the highest quality
links.
</description>
</property>
And following is added in regex-urlfilter.txt
accept anything else
+.
Note: if i add http://www.tutorialspoint.com/ in seed.txt, I'm able to crawl all other pages but not techcrunch.com's pages though it has got many other pages too.
Please help..?

Answer 1

In nutch-default.xml set db.ignore.external.links to true and 'db.ignore.external.links.mode' to byDomain. Like this :
<property>
<name>db.ignore.external.links</name>
<value>true</value>
</property>
<property>
<name>db.ignore.external.links.mode</name>
<value>byDomain</value>
</property>
By default db.ignore.external.links.mode is set to byHost. Which means while crawing http://www.techcrunch.com/ the URL http://subdomain1.techcrunch.com will get treated as EXTERNAL and hence will be ignored. But you want sudomain1 pages to be crawled too - hence keep db.ignore.external.links.mode to byDomain
No work around required in regex-urlfilter.txt. Use regex-urlfilter.txt for some complex situation

Answer 2

I think you are using the wrong property, first use db.ignore.external.links in nutch-site.xml
<property>
<name>db.ignore.external.links</name>
<value>true</value>
<description>If true, outlinks leading from a page to external hosts
will be ignored. This will limit your crawl to the host on your seeds file.
</description>
</property>
b) Then you could also use a regex in regex-urlfilter.txt to limit the domains crawled to just techcrunch.
+^(http|https)://.*techcrunch.com/
However I think that your issue is that Nutch obeys the robots.txt file and in this case techcrunch has a Crawl-delay value of 3600!! see robots.txt. The default value of fetcher.max.crawl.delay is 30 seconds making Nutch dismiss all the pages from techcrunch.
From fetcher.max.crawl.delay in nutch-default
"If the Crawl-Delay in robots.txt is set to greater than this value (in
seconds) then the fetcher will skip this page, generating an error report.
If set to -1 the fetcher will never skip such pages and will wait the
amount of time retrieved from robots.txt Crawl-Delay, however long that
might be."
You may want to play with the fetcher.threads.fetch and fetcher.threads.per.queue values to speed up your crawl. You could also take a look at this and play with the Nutch code.. or you may even want to use a different approach to crawl sites with long crawl delays.
Hope this is useful to you.
Cheers!