Solr - WordDelimiterFactory with Custom Tokenizer to split only on Boundires - solr

Question

I have configured WordDelimiterFilterFactory for custom tokenizers for & and - which is working fine.
And for few tokenizer (like . _ :) we need to split on boundries only. And not to split if in between of word.
e.g.
test.com (should tokenized to test.com)
newyear. coming (should tokenized to newyear and coming)
new_car (should tokenized to new_car)
..
..
I checked that types can be used in Solr.WordDelimiterFilterFactory are LOWER, UPPER, ALPHA, DIGIT, ALPHANUM, SUBWORD_DELIM . there's no description available for use of each type. as per name suggest , i thought type SUBWORD_DELIM may fulfill my need, but it doesn't seem to work.
Below is defination for text field
<fieldType name="text_general_preserved" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="false" />
<filter class="solr.WordDelimiterFilterFactory"
splitOnCaseChange ="0"
splitOnNumerics ="0"
stemEnglishPossessive ="0"
generateWordParts="1"
generateNumberParts="1"
catenateWords="0"
catenateNumbers="0"
catenateAll="0"
preserveOriginal="0"
protected="protwords_general.txt"
types="wdfftypes_general.txt"
/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="false" />
<filter class="solr.WordDelimiterFilterFactory"
splitOnCaseChange ="0"
splitOnNumerics ="0"
stemEnglishPossessive ="0"
generateWordParts="1"
generateNumberParts="1"
catenateWords="0"
catenateNumbers="0"
catenateAll="0"
preserveOriginal="0"
protected="protwords_general.txt"
types="wdfftypes_general.txt"
/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
below is wdfftypes_general.txt content
& => ALPHA
- => ALPHA
_ => SUBWORD_DELIM
: => SUBWORD_DELIM
. => SUBWORD_DELIM
Can anybody suggest me how can i set configuration for Solr.WordDelimiterFilterFactory to fulfill my requirement.
Thanks.

Answer 1

Based on the documentation for WordDelimiterFilterFactory, the SUBOWRD_DELIM settings in the wdfftypes.txt file only impact the behavior based on the splitOnCaseChange and splitOnNumerics settings. Therefore, I would add : _ . as ALPHA entries in the wdfftypes.txt file and add a new PatternReplaceCharFilterFactory after the WordDelimiterFilterFactory in your fieldType to remove those leading or trailing character from any tokens.