Debugging solr rss DataImportHandler - solr

Question

I have an existing collection, to which I want to add an RSS importer. I've copied what I could gleam from the example-DIH/solr/rss code.
The details are below, but the bottom line is that everything seems to run, but it always says "Fetched: 0" (and I get no documents). There are no exceptions in the tomcat log.
Questions:
Is there a way to turn up debugging on rss importers?
Can I see solr's actual request and response?
What would cause the request to succeed, but no rows to be fetched?
Is there a tutorial for adding an RSS DIH to an existing collection?
Thanks!
My solrconfig.xml file contains the requestHandler:
<requestHandler name="/dataimport"
class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">rss-data-config.xml</str>
</lst>
</requestHandler>
And rss-data-config.xml:
<dataConfig>
<dataSource type="URLDataSource" />
<document>
<entity name="slashdot"
pk="link"
url="http://rss.slashdot.org/Slashdot/slashdot"
processor="XPathEntityProcessor"
forEach="/rss/channel | /rss/item"
transformer="DateFormatTransformer">
<field column="source_name" xpath="/rss/channel/title" commonField="true" />
<field column="title" xpath="/rss/item/title" />
<field column="link" xpath="/rss/item/link" />
<field column="body" xpath="/rss/item/description" />
<field column="date" xpath="/rss/item/date" dateTimeFormat="yyyy-MM-dd'T'HH:mm:ss" />
</entity>
</document>
</dataConfig>
and from schema.xml:
<fields>
<field name="title" type="text_general" required="true" indexed="true" stored="true"/>
<field name="link" type="string" required="true" indexed="true" stored="true"/>
<field name="source_name" type="text_general" required="true" indexed="true" stored="true"/>
<field name="body" type="text_general" required="false" indexed="false" stored="true"/>
<field name="date" type="date" required="true" indexed="true" stored="true" />
<field name="text" type="text_general" indexed="true" stored="false" multiValued="true"/>
<field name="_version_" type="long" indexed="true" stored="true"/>
<fields>
When I run the dataimport from the admin web page, it all seems to go well. It shows "Requests: 1" and there are no exceptions in the tomcat log:
Mar 12, 2013 9:02:58 PM org.apache.solr.handler.dataimport.DataImporter maybeReloadConfiguration
INFO: Loading DIH Configuration: rss-data-config.xml
Mar 12, 2013 9:02:58 PM org.apache.solr.handler.dataimport.DataImporter loadDataConfig
INFO: Data Configuration loaded successfully
Mar 12, 2013 9:02:58 PM org.apache.solr.handler.dataimport.DataImporter doFullImport
INFO: Starting Full Import
Mar 12, 2013 9:02:58 PM org.apache.solr.handler.dataimport.SimplePropertiesWriter readIndexerProperties
INFO: Read dataimport.properties
Mar 12, 2013 9:02:59 PM org.apache.solr.handler.dataimport.DocBuilder execute
INFO: Time taken = 0:0:0.693
Mar 12, 2013 9:02:59 PM org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [articles] webapp=/solr path=/dataimport params={optimize=false&clean=false&indent=true&commit=false&verbose=true&entity=slashdot&command=full-import&debug=true&wt=json} {} 0 706

Answer 1

Your problem here is due to your rss-data-config.xml and the defined xpaths.
If you open the url http://rss.slashdot.org/Slashdot/slashdot in Internet Explorer and hit F12 for developer tools it will show you the structure of the HTML.
You can see that the node <item> is a child of <channel> and not <rss>. So your config should look as follows:
<?xml version="1.0" encoding="UTF-8" ?>
<dataConfig>
<dataSource type="URLDataSource" />
<document>
<entity name="slashdot"
pk="link"
url="http://rss.slashdot.org/Slashdot/slashdot"
processor="XPathEntityProcessor"
forEach="/rss/channel | /rss/channel/item"
transformer="DateFormatTransformer">
<field column="source_name" xpath="/rss/channel/title" commonField="true" />
<field column="title" xpath="/rss/channel/item/title" />
<field column="link" xpath="/rss/channel/item/link" />
<field column="body" xpath="/rss/channel/item/description" />
<field column="date" xpath="/rss/channel/item/date" dateTimeFormat="yyyy-MM-dd'T'HH:mm:ss" />
</entity>
</document>
</dataConfig>

Answer 2

Which Solr version are you using ?
For 3.X you have the debug feature with DIH which will help you debug step by step.
Its missing in 4.X probably check SOLR-4151

Answer 3

The following data-config.xml file does the work for Slashdot (Solr 4.2.0)
<dataConfig>
<dataSource type="HttpDataSource" />
<document>
<entity name="slashdot"
pk="link"
url="http://rss.slashdot.org/Slashdot/slashdot"
processor="XPathEntityProcessor"
forEach="/rss/channel/item"
transformer="DateFormatTransformer">
<field column="title" xpath="/rss/channel/item/title" />
<field column="link" xpath="/rss/channel/item/link" />
<field column="description" xpath="/rss/channel/item/description" />
<field column="creator" xpath="/rss/channel/item/creator" />
<field column="item-subject" xpath="/rss/channel/item/subject" />
<field column="slash-department" xpath="/rss/channel/item/department" />
<field column="slash-section" xpath="/rss/channel/item/section" />
<field column="slash-comments" xpath="/rss/channel/item/comments" />
<field column="date" xpath="/rss/channel/item/date" dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss'Z'" />
</entity>
</document>
Notice the extra 'Z' on the dateTimeFormat, which is necessary according to "schema.xml"
Quoting schema.xml
The format for this date field is of the form 1995-12-31T23:59:59Z, and
is a more restricted form of the canonical representation of dateTime
http://www.w3.org/TR/xmlschema-2/#dateTime
The trailing "Z" designates UTC time and is mandatory.
Optional fractional seconds are allowed: 1995-12-31T23:59:59.999Z
All other components are mandatory.

Answer 4

Update your rss-data-config.xml as below
'<dataConfig>
<dataSource type="URLDataSource" />
<document>
<entity name="slashdot"
pk="link"
url="http://rss.slashdot.org/Slashdot/slashdot"
processor="XPathEntityProcessor"
forEach="/RDF/channel | /RDF/item"
transformer="DateFormatTransformer">
<field column="source" xpath="/RDF/channel/title" commonField="true" />
<field column="source-link" xpath="/RDF/channel/link" commonField="true" />
<field column="subject" xpath="/RDF/channel/subject" commonField="true" />
<field column="title" xpath="/RDF/item/title" />
<field column="link" xpath="/RDF/item/link" />
<field column="description" xpath="/RDF/item/description" />
<field column="creator" xpath="/RDF/item/creator" />
<field column="item-subject" xpath="/RDF/item/subject" />
<field column="slash-department" xpath="/RDF/item/department" />
<field column="slash-section" xpath="/RDF/item/section" />
<field column="slash-comments" xpath="/RDF/item/comments" />
<field column="date" xpath="/RDF/item/date" dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss" />
</entity>
</document>
</dataConfig>'
It worked for me