I've got a number of different cores, each with its own schema, but they all share the same field types. I'd like to remove the duplication of the field type declarations and do something like this in my schema.xml files:
<?xml version="1.0" encoding="UTF-8" ?>
<schema name="foo" version="1.5">
<fields>
<field name="_version_" ...
<field name="id" ...
...
</fields>
<uniqueKey>id</uniqueKey>
<include "/path/to/field_types.xml">
</schema>
I don't see any mechanism in the docs to accomplish this however. I found one post referring to this:
<xi:include href="/path/to/field_types.xml" />
But that gives me a launch error: The prefix "xi" for element "xi:include" is not bound.
Anybody have an idea how to perform this type of raw include?
From this past Solr Issue - SOLR-3087, it looks like <xi:include> is the correct syntax, you just need to include the xi namespace reference inline.
<?xml version="1.0" encoding="UTF-8" ?>
<schema name="foo" version="1.5">
<fields>
<field name="_version_" ...
<field name="id" ...
...
</fields>
<uniqueKey>id</uniqueKey>
<xi:include href="/path/to/field_types.xml" xmlns:xi="http://www.w3.org/2001/XInclude"/>
</schema>
Another clean solution at this problem is add resources as external entities:
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE schema
[
<!ENTITY schemafieldtypes SYSTEM "schemafieldtypes.xml">
]>
then, in the xml, you can add everywhere this:
&schemafieldtypes;
Related
I am building a block moodle plugin. For the plugin I have created three tables:
'block_learning_strategizer':contains just one field-id
'ls_basic': containing 3 fields-id,lp_name,description
'ls_path_details': containing 9 fields.
The definition is done through install.xml(under blocks/learning_strategizer/db)
The XML is as below:
<?xml version="1.0" encoding="UTF-8" ?>
<XMLDB PATH="blocks/learning_strategizer/db" VERSION="20120122" COMMENT="XMLDB file for Moodle blocks/learning_strategizer"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="../../../lib/xmldb/xmldb.xsd"
>
<TABLES>
<TABLE NAME="block_learning_strategizer" COMMENT="Default for block_learning_strategizer" NEXT="ls_basic">
<FIELDS>
<FIELD NAME="id" TYPE="int" LENGTH="10" NOTNULL="true" SEQUENCE="true"/>
</FIELDS>
<KEYS>
<KEY NAME="primary" TYPE="primary" FIELDS="id"/>
</KEYS>
</TABLE>
<TABLE NAME="ls_basic" COMMENT="Table contains name and description of learning paths" NEXT="ls_path_details">
<FIELDS>
<FIELD NAME="id" TYPE="int" LENGTH="10" NOTNULL="true" SEQUENCE="true"/>
<FIELD NAME="lp_name" TYPE="char" LENGTH="255" NOTNULL="true" SEQUENCE="false" PREVIOUS="id" NEXT="description"/>
<FIELD NAME="description" TYPE="text" NOTNULL="false" SEQUENCE="false" PREVIOUS="lp_name"/>
</FIELDS>
<KEYS>
<KEY NAME="primary" TYPE="primary" FIELDS="id"/>
</KEYS>
</TABLE>
<TABLE NAME="ls_path_details" COMMENT="Table contains details of created learning paths" PREVIOUS="ls_basic">
<FIELDS>
<FIELD NAME="id" SEQUENCE="true" TYPE="int" NOTNULL="true" LENGTH="10" NEXT="lp_id"/>
<FIELD NAME="lp_id" SEQUENCE="false" TYPE="int" LENGTH="10" NOTNULL="true" NEXT="id" PREVIOUS="course" />
<FIELD NAME="course" SEQUENCE="false" TYPE="int" LENGTH="10" NOTNULL="true" NEXT="section" PREVIOUS="lp_id"/>
<FIELD NAME="section" SEQUENCE="false" TYPE="int" NOTNULL="true" LENGTH="10" NEXT="req" PREVIOUS="course"/>
<FIELD NAME="req" SEQUENCE="false" TYPE="int" NOTNULL="true" LENGTH="2" NEXT="inc" PREVIOUS="section"/>
<FIELD NAME="inc" SEQUENCE="false" TYPE="int" NOTNULL="true" LENGTH="2" NEXT="modid" PREVIOUS="req"/>
<FIELD NAME="modid" SEQUENCE="false" TYPE="int" NOTNULL="true" LENGTH="3" NEXT="seqno" PREVIOUS="inc"/>
<FIELD NAME="seqno" SEQUENCE="false" TYPE="int" NOTNULL="true" LENGTH="10" NEXT="filename" PREVIOUS="modid"/>
<FIELD NAME="filename" SEQUENCE="false" TYPE="text" NOTNULL="true" LENGTH="255" PREVIOUS="seqno"/>
</FIELDS>
<KEYS>
<KEY NAME="primary" TYPE="primary" FIELDS="id" NEXT="lp_id"/>
<KEY NAME="lp_id" TYPE="foreign" FIELDS="lp_id" REFTABLE="ls_basic" REFFIELDS="id" PREVIOUS="primary" />
</KEYS>
</TABLE>
</TABLES>
</XMLDB>
However when I am trying to insert records I am getting an error :"Table "ls_basic" does not exist"
I checked XMLDB editor from Site administration and I could see the tables have been made.
I havent included an upgrade.php but as far as I know that file is optional.
It would be really helpful if someone could point out why do i get this error?
When a plugin is first installed, Moodle parses the install.xml file and uses this to create the database tables required for your plugin.
After the first installation of the plugin, Moodle does not look at the install.xml file again. Instead it relies on checking the upgrade.php file, at the point where your plugin's version number (in version.php) changes, in order to find out how to transform the previous database structure to match the new structure.
If your plugin is still under local development, you can get Moodle to re-parse the install.xml file by using the 'uninstall' feature in the 'Plugins' part of the 'Site administration' area. This will remove all the data for the plugin, then, if the code for the plugin still exists on the server, will immediately offer to re-install the plugin (which will create all the tables in install.xml).
If your plugin is already in use, or you do not want to lose any existing data, then you will need to use the XMLDB editor to generate the relevant lines of code to go in your upgrade.php file (and increase your plugin's version number to match).
See https://docs.moodle.org/dev/Upgrade_API for more details.
I would also suggest that this is a good time to fix your database tables to match the Moodle coding guidelines:
Variable names should not have _ characters in them - this applies to database field names as well (although _ in database table names are fine).
Plugin database tables should all start with the name of the plugin ('block_learning_stategizer' in this case) - if you end up using Travis CI to automatically check your plugins, then it will complain about your database table names.
Plugin names are strongly discouraged from having _ in them (other than between the plugin type and the rest of the name) - there have been a number of bugs over the years caused by Moodle core getting stuck on names that break this rule. It may be a good idea to rename your plugin 'block_learningstrategizer' now, before you hit any problems.
Apache camel bean validation:
I have one field dependent on another can some one help me how to Validate e.g. If manufacturer field data is SPECIFIC then licensePlate field is mandatory.
<?xml version="1.0" encoding="UTF-8"?>
<constraint-mappings xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://jboss.org/xml/ns/javax/validation/mapping validation-mapping-1.0.xsd"
xmlns="http://jboss.org/xml/ns/javax/validation/mapping">
<default-package>org.apache.camel.component.bean.validator</default-package>
<bean class="CarWithoutAnnotations" ignore-annotations="true">
<field name="manufacturer">
<constraint annotation="javax.validation.constraints.NotNull" />
</field>
<field name="licensePlate">
<constraint annotation="javax.validation.constraints.NotNull" />
</field>
</bean>
</constraint-mappings>
Please use org.hibernate.validator.constraints.ScriptAssert for cross field validation.
I am trying to create a very simple search index on solr 4.5.1 with just two fields 'id' and 'name' by using a csv file. Running on Windows 8.
When I run:
curl http://localhost:8983/solr/update/csv --data-binary #mydata.csv -H
"Content-type:text/plain; charset=utf-8"
I get the error: Document is missing mandatory uniqueKey field: id
When I copy/paste the content of the file into the csv import function in the solr-admin ui (documents->document type:csv) then it works.
What am I missing? Thx for any help!
My schema.xml:
<fields>
<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
<!-- points to the root document of a block of nested documents -->
<field name="_root_" type="string" indexed="true" stored="false"/>
<field name="name" type="text_en" indexed="true" stored="true"/>
<field name="_version_" type="long" indexed="true" stored="true" multiValued="true"/>
</fields>
<uniqueKey>id</uniqueKey>
The simplest csv file I tried is:
id,name
LXOxjksM2z, The simplest cookbook you can ever find
I do not have windows 8 , but i tried with windows 7. I installed 4.5.1 and ran your curl on windows/cygwin as:
/cygdrive/c/Installs/solr-4.5.1/solr-4.5.1/example/exampledocs
$ curl http://localhost:8983/solr/update/csv --data-binary #test.csv -H "Content-type:text/plain; charset=utf-8"
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">17</int></lst>
</response>
Where test.csv is:
id,name
LXOxjksM2z, The simplest cookbook you can ever find
Solr indexed properly and i was able to query from admin for id:LXOxjksM2z and found 1 document as :
"response": {
"numFound": 1,
"start": 0,
"docs": [
{
"id": "LXOxjksM2z",
"name": " The simplest cookbook you can ever finds",
"_version_": 1452541521661264000
}
]
}
}
Not sure what is wrong on windows 8.
It's all about file encoding, it works only with ANSI encoding
here is the same issue
Apach Solr import index from csv(UTF-8) error: undefined field
in my db-data-config.xml i have configured two datasource, each with his parameter name,
for example:
<dataSource name="test1"
type="JdbcDataSource"
driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost/firstdb"
user="username1"
password="psw1"/>
<dataSource name="test2"
type="JdbcDataSource"
driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost/seconddb"
user="username2"
password="psw2"/>
<document name="content">
<entity name="news" datasource="test1" query="select...">
<field column="OTYPE_ID" name="otypeID" />
<field column="NWS_ID" name="cntID" />
....
</entity>
<entity name="news_update" datasource="test2" query="select...">
<field column="OTYPE_ID" name="otypeID" />
<field column="NWS_ID" name="cntID" />
....
</entity>
</document>
</dataConfig>
but when in solr from dataimport i execute the second entity-name-query it launch an exception:
"Table 'firstdb.secondTable' doesn't exist\n\tat"
could someone help me? thank you in advance
A think that your query for news_update is wrong. You must have an error on name of table.
I'm pretty sure this question showed up on the solr-user mailing list. The answer given there was that you are using datasource in your entity tags instead of dataSource. It's case sensitive. If I recall the thread correctly, changing this solved your problem.
I don't know java, I don't know XML, and I don't know Lucene. Now that that's out of the way. I have been working to create a little project using apache solr/lucene. My problem is that I am unable to index the xml files. I think I understand how its supposed to work but I could be wrong. I am not sure what information is required for you to help me so I will just post the code.
<dataConfig>
<dataSource type="FileDataSource" encoding="UTF-8" />
<document>
<!-- This first entity block will read all xml files in baseDir and feed it into the second entity block for handling. -->
<entity name="AMMFdir" rootEntity="false" dataSource="null"
processor="FileListEntityProcessor"
fileName="^*\.xml$" recursive="true"
baseDir="C:\Documents and Settings\saperez\Desktop\Tomcat\apache-tomcat-7.0.23\webapps\solr\data\AMMF_New"
>
<entity
processor="XPathEntityProcessor"
name="AMMF"
pk="AcquirerBID"
datasource="AMMFdir"
url="${AMMFdir.fileAbsolutePath}"
forEach="/AMMF/Merchants/Merchant/"
transformer="DateFormatTransformer, RegexTransformer"
>
<field column="AcquirerBID" xpath="/AMMF/Merchants/Merchant/AcquirerBID" />
<field column="AcquirerName" xpath="/AMMF/Merchants/Merchant/AcquirerName" />
<field column="AcquirerMerchantID" xpath="/AMMF/Merchants/Merchant/AcquirerMerchantID" />
</entity>
</entity>
</document>
Example xml file
<?xml version="1.0" encoding="utf-8"?>
<AMMF xmlns="http://tempuri.org/XMLSchema.xsd" Version="11.2" CreateDate="2011-11-07T17:05:14" ProcessorBINCIB="422443" ProcessorName="WorldPay" FileSequence="18">
<Merchants Count="153">
<Merchant ChangeIndicator="A" LocationCountry="840">
<AcquirerBID>10029881</AcquirerBID>
<AcquirerName>WorldPay</AcquirerName>
<AcquirerMerchantID>*</AcquirerMerchantID>
<Merchant ChangeIndicator="A" LocationCountry="840">
<AcquirerBID>10029882</AcquirerBID>
<AcquirerName>WorldPay2</AcquirerName>
<AcquirerMerchantID>Hello World!</AcquirerMerchantID>
</Merchant>
</Merchants>
I have this in schema.
<field name="AcquirerBID" type="string" indexed="true" stored="true" required="true" />
<field name="AcquirerName" type="string" indexed="true" stored="true" />
<field name="AcquirerMerchantID" type="string" indexed="true" stored="true"/>
I have this in config.
<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler" default="true" >
<lst name="defaults">
<str name="config">AMMFconfig.xml</str>
</lst>
</requestHandler>
The sample XML is not well formed. This might explain errors indexing the files:
$ xmllint sample.xml
sample.xml:13: parser error : expected '>'
</Merchants>
^
sample.xml:14: parser error : Premature end of data in tag Merchants line 3
sample.xml:14: parser error : Premature end of data in tag AMMF line 2
Corrected XML
Here's what I think your sample data should look like (Didn't check the XSD file)
<?xml version="1.0" encoding="utf-8"?>
<AMMF xmlns="http://tempuri.org/XMLSchema.xsd" Version="11.2" CreateDate="2011-11-07T17:05:14" ProcessorBINCIB="422443" ProcessorName="WorldPay" FileSequence="18">
<Merchants Count="153">
<Merchant ChangeIndicator="A" LocationCountry="840">
<AcquirerBID>10029881</AcquirerBID>
<AcquirerName>WorldPay</AcquirerName>
<AcquirerMerchantID>*</AcquirerMerchantID>
</Merchant>
<Merchant ChangeIndicator="A" LocationCountry="840">
<AcquirerBID>10029882</AcquirerBID>
<AcquirerName>WorldPay2</AcquirerName>
<AcquirerMerchantID>Hello World!</AcquirerMerchantID>
</Merchant>
</Merchants>
</AMMF>
Alternative solution
I know you said you're not a programmer, but this task is significantly simpler, if you use the solrj interface.
The following is a groovy example which indexes your example XML
//
// Dependencies
// ============
import org.apache.solr.client.solrj.SolrServer
import org.apache.solr.client.solrj.impl.CommonsHttpSolrServer
import org.apache.solr.common.SolrInputDocument
#Grapes([
#Grab(group='org.apache.solr', module='solr-solrj', version='3.5.0'),
])
//
// Main
// =====
SolrServer server = new CommonsHttpSolrServer("http://localhost:8983/solr/");
def i = 1
new File(".").eachFileMatch(~/.*\.xml/) {
it.withReader { reader ->
def ammf = new XmlSlurper().parse(reader)
ammf.Merchants.Merchant.each { merchant ->
SolrInputDocument doc = new SolrInputDocument();
doc.addField("id", i++)
doc.addField("bid_s", merchant.AcquirerBID)
doc.addField("name_s", merchant.AcquirerName)
doc.addField("merchantId_s", merchant.AcquirerMerchantID)
server.add(doc)
}
}
}
server.commit()
Groovy is a Java scripting language that does not require compilation. It would be just as easy to maintain as a DIH config file.
To figure out how DIH XML import works, I suggest you first carefully read this chapter in DIH wiki: http://wiki.apache.org/solr/DataImportHandler#HttpDataSource_Example.
Open the Slashdot link http://rss.slashdot.org/Slashdot/slashdot in your browser, then right click on the page and select View source. There's the XML file used in this example.
Compare it with XPathEntityProcessor configuration in DIH example and you'll see how easy it is to import any XML file in Solr.
If you need more help just ask...
Often the best thing to do is NOT use the DIH. How hard would it be to just post this data using the API and a custom script in a language you DO know?
The benefit of this approach is two-fold:
You learn more about your system, and know it better.
You don't spend time trying to understand the DIH.
The downside is that you're re-inventing the wheel a bit, but the DIH is quite a thing to understand.