automatic language detect in solr 4.5.1 during indexing time - solr

Question

I need your helps.
I want to detect Korean and English language during indexing time in solr.
My solr directory structure is
/opt/tmocat7/webapps/solr (solr webapp)
/usr/share/solr/collection1 (solr core)
/usr/share/solr/lib/langid (lib for langid)
First, I copy some libraries(jsonic-1.2.7.jar, langdetect-1.1-20120112.jar, solr-langid-4.5.1.jar) into specific directory(/usr/share/solr/lib/langid) - my solr is located
My solrconfig.xml is
<lib dir="../lib/langid/" regex=".*\.jar" />
<requestHandler name="/update" class="solr.UpdateRequestHandler">
<lst name="defaults">
<str name="update.chain">dedupe</str>
<str name="update.chain">uuid</str>
<str name="update.chain">langid</str>
</lst>
</requestHandler>
<updateRequestProcessorChain name="langid">
<processor class="org.apache.solr.update.processor.LangDetectLanguageIdentifierUpdateProcessorFactory">
<bool name="langid">true</bool>
<str name="langid.fl">title,content,comment</str>
<str name="langid.langField">lang</str>
<str name="langid.langsField">langs</str>
<str name="langid.lcmap">ko:ko kor:ko en_GB:en en_US:en</str>
<str name="langid.whitelist">ko,en</str>
<bool name="langid.map">true</bool>
<str name="langid.map.fl">title,content,comment</str>
<bool name="langid.map.keepOrig">true</bool>
<bool name="langid.map.individual">true</bool>
<str name="langid.fallback">ko</str>
<str name="langid.map.lcmap">ko:ko kor:ko en_GB:en en_US:en</str>
</processor>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
and schema.xml is
<field name="lang" type="string" indexed="true" stored="true" multiValued="false" />
<field name="langs" type="string" indexed="true" stored="true" multiValued="true" />
<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
<field name="title" type="text_ko" indexed="true" stored="true" multiValued="true"/>
<field name="content" type="text_ko" indexed="true" stored="true" multiValued="true"/>
<field name="comment" type="text_ko" indexed="true" stored="true" multiValued="true" />
<field name="site" type="text_general" indexed="true" stored="true" multiValued="false"/>
<field name="page" type="text_general" indexed="true" stored="true" multiValued="false"/>
<field name="fileloc" type="text_general" indexed="true" stored="true"
multiValued="false"/>
<field name="filename" type="text_general" indexed="true" stored="true"
multiValued="false" />
<field name="storeddate" type="date" indexed="true" stored="true" multiValued="false"/>
<!-- for english web data-->
<field name="title_en" type="text_en" indexed="true" stored="true" multiValued="true" />
<field name="content_en" type="text_en" indexed="true" stored="true" multiValued="true" />
<field name="comment_en" type="text_en" indexed="true" stored="true" multiValued="true" />
<field name="title_ko" type="text_ko" indexed="true" stored="true" multiValued="true"/>
<field name="content_ko" type="text_ko" indexed="true" stored="true" multiValued="true"/>
<field name="comment_ko" type="text_ko" indexed="true" stored="true" multiValued="true" />
<copyField source="title" dest="title_en"/>
<copyField source="content" dest="content_en"/>
<copyField source="comment" dest="comment_en"/>
<copyField source="title" dest="title_ko"/>
<copyField source="content" dest="content_ko"/>
<copyField source="comment" dest="comment_ko"/>
I read a some books and searching web to get a information about detecting language in solr, but can't detect language.
What is my fault?
For more information, add my post.sh and log
This is post.sh
#!/bin/sh
FILES=$*
URL=http://locahost:port/solr/collection1/update
for f in $FILES; do
echo Posting file $f to $URL
curl $URL --data-binary #$f -H 'Content-type:application/xml'
echo
done
#send the commit command to make sure all the changes are flushed and visible
curl $URL --data-binary '<commit/>' -H 'Content-type:application/xml'
echo
some part of tomcat logs during indexing
70634079 [http-bio-7070-exec-38] TRACE org.apache.solr.handler.UpdateRequestHandler – body
70634079 [http-bio-7070-exec-38] DEBUG org.apache.solr.update.processor.LogUpdateProcessor – PRE_UPDATE add{,id=2f2323f4f7966e0d} {{params({params(),defaults(update.chain=dedupe&update.chain=uuid&update.chain=langid)}),defaults(wt=xml)}}
70634125 [http-bio-7070-exec-38] TRACE org.apache.solr.update.UpdateLog – TLOG: added id 2f2323f4f7966e0d to tlog{file=/usr/share/solr/collection1/data/tlog/tlog.0000000000000000129 refcount=1} LogPtr(29407) map=614254179
70634125 [http-bio-7070-exec-38] DEBUG org.apache.solr.update.processor.LogUpdateProcessor – PRE_UPDATE FINISH {{params({params(),defaults(update.chain=dedupe&update.chain=uuid&update.chain=langid)}),defaults(wt=xml)}}
70634126 [http-bio-7070-exec-38] INFO org.apache.solr.update.processor.LogUpdateProcessor – [collection1] webapp=/solr path=/update params={} {add=[2f2323f4f7966e0d (1473490520171872256)]} 0 68
70634146 [http-bio-7070-exec-33] TRACE org.apache.solr.handler.UpdateRequestHandler – body
70634146 [http-bio-7070-exec-33] DEBUG org.apache.solr.update.processor.LogUpdateProcessor – PRE_UPDATE add{,id=329ee20831e1a0c7} {{params({params(),defaults(update.chain=dedupe&update.chain=uuid&update.chain=langid)}),defaults(wt=xml)}}
70634148 [http-bio-7070-exec-33] TRACE org.apache.solr.update.UpdateLog – TLOG: added id 329ee20831e1a0c7 to tlog{file=/usr/share/solr/collection1/data/tlog/tlog.0000000000000000129 refcount=1} LogPtr(46005) map=614254179
70634148 [http-bio-7070-exec-33] DEBUG org.apache.solr.update.processor.LogUpdateProcessor – PRE_UPDATE FINISH {{params({params(),defaults(update.chain=dedupe&update.chain=uuid&update.chain=langid)}),defaults(wt=xml)}}
70634148 [http-bio-7070-exec-33] INFO org.apache.solr.update.processor.LogUpdateProcessor – [collection1] webapp=/solr path=/update params={} {add=[329ee20831e1a0c7 (1473490520241078272)]} 0 2
I can't find any other warn or error.
I need your advice
Thanks all

Answer 1

I think you use /update/extract instead of /update
In Solr 5.3.1, it works fine when I use with /update/extract.
Here's the full config:
<requestHandler name="/update/extract"
startup="lazy"
class="solr.extraction.ExtractingRequestHandler" >
<lst name="defaults">
<str name="lowernames">true</str>
<str name="uprefix">ignored_</str>
<!-- capture link hrefs but ignore div attributes -->
<str name="captureAttr">true</str>
<str name="fmap.a">links</str>
<str name="fmap.div">ignored_</str>
<str name="update.chain">langid</str>
</lst>

Answer 2

Thanks for the question and the great answers, they helped me configure my system appropriately. I don't know how I managed to get the JAR file solr-langdetect.*.*.*.jar into my lib directory, but each time when I started solr it would show me the following error:
org.apache.solr.common.SolrException: com.cybozu.labs.langdetect.DetectorFactory.loadProfile(Ljava/util/List;)V
After removing that JAR file everything worked fine. The other three JAR files mentioned in the question (jsonic-*.*.*.jar, langdetect-*.*.jar, solr-langid-*.*.*.jar) are however required.