How to index a NFS mount using Nutch? - solr

Question

I am trying to build a search tool hosted on a CentOS 7 machine which should index and search the directories of the mounted NFS export. I found that Nutch+Solr is the best bet for this. I have had a hard time configuring the url for this since this will not search any http locations.
The mount is located on /mnt
So my seeds.txt looks like this:
[root#sauron bin]# cat /root/Desktop/apache-nutch-1.13/urls/seed.txt
file:///mnt
and my regex-urlfilter.txt has the same site plus allowing file protocol
# skip file: ftp: and mailto: urls
-^(http|https|ftp|mailto):
# skip image and other suffixes we can't yet parse
# for a more extensive coverage use the urlfilter-suffix plugin
#-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|wmf|WMF|zip|ZIP|ppt|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$
# skip URLs containing certain characters as probable queries, etc.
-[?*!#=]
# skip URLs with slash-delimited segment that repeats 3+ times, to break loops
-.*(/[^/]+)/[^/]+\1/[^/]+\1/
# accept anything else
+^file:///mnt
However when I try to bootstrap from the initial seed list there are no updates done:
[root#sauron apache-nutch-1.13]# bin/nutch inject crawl/crawldb urls
Injector: starting at 2017-06-12 00:07:49
Injector: crawlDb: crawl/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Injector: overwrite: false
Injector: update: false
Injector: Total urls rejected by filters: 1
Injector: Total urls injected after normalization and filtering: 0
Injector: Total urls injected but already in CrawlDb: 0
Injector: Total new urls injected: 0
Injector: finished at 2017-06-12 00:10:27, elapsed: 00:02:38
I have also tried changing the seeds.txt to the following with no luck:
file:/mnt
file:////<IP>:<export_path>
Please let me know if I am doing something wrong here.

Answer 1

From the URI point of view a file system is not really that different for Nutch, you just need to enable the protocol-file plugin, and configure the regex-urlfilter.txt like:
+^file:///mnt/directory/
-.
In this case you prevent it from indexing the parent directories of the one that you've specified.
Keep in mind that since you've already have the NFS share mounted locally it would work as a normal local file system. More information could be found in https://wiki.apache.org/nutch/FAQ#How_do_I_index_my_local_file_system.3F.