Extracting date from TextField and sorting with it in Solr - solr

Question

I have a a scheme like this in Solr:
<?xml version="1.0" encoding="UTF-8"?>
<schema name="dovecot-fts" version="3.0">
<fieldType name="ytext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="3" maxGramSize="20"/>
<filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
<filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1"/>
<filter class="solr.FlattenGraphFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/>
<filter class="solr.FlattenGraphFilterFactory"/>
<filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
<filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
</fieldType>
<field name="hdr" type="ytext" indexed="true" stored="true"/>
<uniqueKey>id</uniqueKey>
</schema>
And a record looks like this:
{
"id": "339/9821f61c4fa04b62fa030s002df11e39/user#example.com",
"hdr": "...................Some header information...................Date: Fri, 23 Sep 2022 15:24:43 +0300...................Some other header information..................."
}
Now, I want an additional field containing the date in the HDR field to sorting results. To achieve this I tried copying and manipulate hdr field with PatternTokenizerFactory and I can't.
<fieldType name="ts" class="solr.TextField" positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.PatternTokenizerFactory" pattern="Date:\s(\w{3},\s\d{2}\s\w{3,4}\s\d{4}\s\d{2}:\d{2}:\d{2}\s\+\d{4})" group="1"/>
</analyzer>
</fieldType>
<field name="hdr" type="ytext" indexed="true" stored="true"/>
<field name="received" type="ts" indexed="true" stored="false" required="false"/>
<copyField source="hdr" dest="received"/>
So I'm waiting for your help, thanks.

Answer 1

In your schema, you are creating a copy field named received and copying the content of hdr into it. The received field has a specific text analysis but this doesn't change the value of the field. It only changes the way it is indexed. Moreover, you cannot sort documents using a text field.
To achieve your goal you need to transform the value of hdr and copy it into another field of type StrField or Date. It is important to use a non-text type if you want to sort using that field. You can use a custom Update Request Processor Chain: https://solr.apache.org/guide/solr/latest/configuration-guide/update-request-processors.html.
You must insert your custom update chain into the solrconfig.xml. Your best chance is to use the following request processors:
solr.CloneFieldUpdateProcessorFactory
solr.RegexReplaceProcessorFactory
This is an example that you can use as a starting point:
<updateRequestProcessorChain name="extract_hdr_date">
<processor class="solr.CloneFieldUpdateProcessorFactory">
<str name="source">hdr</str>
<str name="dest">hdr_date</str>
</processor>
<processor class="solr.RegexReplaceProcessorFactory">
....
</processor>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
Given that you want to use that field for sorting, you might find it useful to enable docvalues on hdr_date: https://solr.apache.org/guide/solr/latest/indexing-guide/docvalues.html