Im making an index in solr from db in the following way:
<document name="Index">
<entity name="c" query="SELECT * FROM C">
<field column="Name" name="name"/>
</entity>
<entity name="p" query="SELECT * FROM P">
<field column="Name" name="name"/>
</entity>
</document>
Is it possible to have a static field that is set for each row that signify what type is returned to client so that one can make a call to the right database table based on that information from the json result?
That is a field that has no column in the table
<field name="id" value="1"/>
Or is there another way to solve this?
<document name="Index">
<entity name="c" transformer="TemplateTransformer" query="SELECT * FROM C">
<field column="Name" name="name"/>
<field column="id" template="1"/>
</entity>
<entity name="p" transformer="TemplateTransformer" query="SELECT * FROM P">
<field column="Name" name="name"/>
<field column="id" template="1"/>
</entity>
</document>
You can add a column to your SQL query that contains static data like this:
<document name="Index">
<entity name="c" query="SELECT *, 'foo' as NameFromC FROM C">
<field column="NameFromC" name="name"/>
</entity>
<entity name="p" query="SELECT *, 'bar' as NameFromP FROM P">
<field column="NameFromP" name="name"/>
</entity>
</document>
If you try to add a field with only name and template attributes, Solr will throw an error saying Field must have a column attribute.
Related
I've simple DataImportHandler, that is working on my local system and not on my Server. Both the versions of Solr are same i.e Solr 4.6.0.
I've tried these configurations for DataImportHandler:
Configuration 1:
<dataConfig>
<dataSource type="JdbcDataSource"
driver="org.postgresql.Driver"
url="jdbc:postgresql://HOST:5432/mydb"
user="admin"
password="admin" />
<script><![CDATA[
function generate_resource_uri(row) {
row.put('resource_uri', '/api/v1/product/' + row.get('id') + '/');
return row;
}
]]></script>
<document>
<entity name="products_product"
query="SELECT id, image_url, impression_url, product_url, manufacturer_name, discount_percentage, short_description, merchant_name, product_name, sku, long_description, date_modified, merchant_id, commission, keywords, product_id, retail_price, date_created FROM products_product"
transformer="script:generate_resource_uri" >
<entity name="source" query="select title from products_source where id = '${products_product.id}'"
processor="CachedSqlEntityProcessor">
<field column="title" name="source"/>
</entity>
<entity name="currency" query="select code from products_currency where id = '${products_product.id}'"
processor="CachedSqlEntityProcessor">
<field column="code" name="currency"/>
</entity>
<entity name="category" query="select title from products_category where id = '${products_product.id}'"
processor="CachedSqlEntityProcessor">
<field column="title" name="category"/>
</entity>
</entity>
</document>
</dataConfig>
Configuration 2:
<dataConfig>
<dataSource type="JdbcDataSource"
driver="org.postgresql.Driver"
url="jdbc:postgresql://HOST:5432/mydb"
user="admin"
password="123456"/>
<script><![CDATA[
function generate_resource_uri(row) {
row.put('resource_uri', '/api/v1/product/' + row.get('id') + '/');
return row;
}
]]></script>
<document>
<entity name="products_product"
query="SELECT id, image_url, impression_url, product_url, manufacturer_name, discount_percentage, short_description, merchant_name, product_name, sku, long_description, date_modified, merchant_id, commission, keywords, product_id, retail_price, date_created FROM products_product"
transformer="script:generate_resource_uri" >
<entity name="source" query="select title from products_source where id = '${products_product.id}'"
cachePk="id" cacheLookup="products_product.id" cacheImpl="SortedMapBackedCache">
<field column="title" name="source"/>
</entity>
<entity name="currency" query="select code from products_currency where id = '${products_product.id}'"
cachePk="id" cacheLookup="products_product.id" cacheImpl="SortedMapBackedCache">
<field column="code" name="currency"/>
</entity>
<entity name="category" query="select title from products_category where id = '${products_product.id}'"
cachePk="id" cacheLookup="products_product.id" cacheImpl="SortedMapBackedCache">
<field column="title" name="category"/>
</entity>
</entity>
</document>
</dataConfig>
Locally I've have approx 2K rows, which is indexed properly, and all the child entity show up.
On server, the fields from the child entities are not showing up i.e source, category and currency. The server has approx 6M rows, its a silly doubt but I hope that memory is not the issue. My server is running on m1.medium EC2 instance, Ubuntu 12.04LTS.
Thanks in Advance :)
Im making an index in solr from db in the following way:
<document name="Index">
<entity name="c" query="SELECT * FROM C">
<field column="Name" name="name"/>
</entity>
<entity name="p" query="SELECT * FROM P">
<field column="Name" name="name"/>
</entity>
</document>
Is it possible to have a static field that is set for each row that signify what type is returned to client so that one can make a call to the right database table based on that information from the json result?
That is a field that has no column in the table
<field name="id" value="1"/>
Or is there another way to solve this?
<document name="Index">
<entity name="c" transformer="TemplateTransformer" query="SELECT * FROM C">
<field column="Name" name="name"/>
<field column="id" template="1"/>
</entity>
<entity name="p" transformer="TemplateTransformer" query="SELECT * FROM P">
<field column="Name" name="name"/>
<field column="id" template="1"/>
</entity>
</document>
You can add a column to your SQL query that contains static data like this:
<document name="Index">
<entity name="c" query="SELECT *, 'foo' as NameFromC FROM C">
<field column="NameFromC" name="name"/>
</entity>
<entity name="p" query="SELECT *, 'bar' as NameFromP FROM P">
<field column="NameFromP" name="name"/>
</entity>
</document>
If you try to add a field with only name and template attributes, Solr will throw an error saying Field must have a column attribute.
Im successfully able to index pdf,doc,ppt,etc files using the Data Import Handler in solr 4.3.0 .
My data-config.xml looks like this -
<dataConfig>
<dataSource name="bin" type="BinFileDataSource" />
<document>
<entity name="f" dataSource="null" rootEntity="false"
processor="FileListEntityProcessor"
baseDir="C:\Users\aroraarc\Desktop\Impdo"
fileName=".*\.(DOC)|(PDF)|(pdf)|(doc)|(docx)|(ppt)|(pptx)|(xls)|(xlsx)|(txt)" onError="skip"
recursive="true">
<field column="fileAbsolutePath" name="path" />
<field column="fileSize" name="size" />
<field column="fileLastModified" name="lastmodified" />
<field column="file" name="fileName"/>
<entity name="tika-test" dataSource="bin" processor="TikaEntityProcessor"
url="${f.fileAbsolutePath}" format="text" onError="skip">
<field column="Author" name="author" meta="true"/>
<field column="title" name="title" meta="true"/>
<field column="text" name="content"/>
</entity>
</entity>
</document>
</dataConfig>
However in the fileName field i want to insert the pure file name without the extension. Eg - Instead of 'HelloWorld.txt' I want only 'HelloWorld' to be inserted in the fileName field. How do I achieve this?
Thanks in advance!
Check ScriptTransformer to replace or change the value before it is indexed.
Example -
Data Config - Add custom field -
<script><![CDATA[
function changeFileName(row){
var fileName= row.get('fileName');
// Replace or remove the extension .. e.g. from last index of .
file_name_new = file_name.replace ......
row.put(fileName, row.get('file_name_new'));
return row;
}
]]></script>
Entity mapping -
<entity name="f" transformer="script:changeFileName" ....>
......
</entity>
I am in the process of redesigning one of our companies site. My boss wants to play around with the idea of replacing all of our navigation with a search box.. the search box should be able to query any of our tables of unrelated data.
So right now I am trying it with 5 tables.
Products
Manufacturers
Category
Ingredients
Uses
So should be able to lookup a product name, a manufacturer name, a category name, an ingredient name, or a use name
When I retrieve the results. if the user clicked on a manufacturer search result.. It will take them to a manufacturer page that lookups all products for that manufacturer.
When clicks on a product page.. link will take them to that actual product information.
Ingredient will take them to a page that will show all products containing that ingredient.
Anyways here is my data config
<dataConfig>
<dataSource driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost:3306/xxx" user="xxx" password="xxx" />
<document>
<entity name="manufacturer" transformer="TemplateTransformer" pk="manNum"
query="SELECT manNum, manName FROM manufacturer
WHERE active = 1">
<field column="id" name="id" template="MAN-${manNum}" />
<field column="type" template="manufacturer" name="type"/>
<field column="manName" name="text"/>
<field column="manNum" name="manNum"/>
</entity>
<entity name="product" transformer="TemplateTransformer"
query="SELECT products.prodNum, products.prodName as text, m.manName FROM products JOIN man m USING (manNum)
WHERE products.active = 1
AND (hideWeb = 0 or hideWeb IS NULL)">
<field column="id" template="PROD-${products.prodNum}" name="id"/>
<field column="type" template="product" name="type"/>
<field column="text" name="text"/>
<field column="manName" name="manName"/>
</entity>
<entity name="ingredients" transformer="TemplateTransformer" pk="id"
query="SELECT id, text FROM inglist WHERE sort != ''">
<field column="id" name="id" template="ING-${inglist.id}"/>
<field column="type" template="ingredient" name="type"/>
<field column="text" name="text" />
</entity>
<entity name="uses" transformer="TemplateTransformer" pk="id"
query="SELECT id, text FROM useslist">
<field column="id" name="id" template="USE-${id}"/>
<field column="type" template="use" name="type"/>
<field column="text" name="text"/>
</entity>
<entity name="categories" transformer="TemplateTransformer" pk="id"
query="SELECT id, textShow as text FROM categorylist">
<field column="id" name="id" template="CATEGORY-${id}"/>
<field column="type" template="category" name="type"/>
<field column="text" name="text"/>
</entity>
</document>
</dataConfig>
And my schema..
<fields>
<field name="id" type="string" indexed="true" stored="true"/>
<field name="text" indexed="true" stored="true" type="text"/>
<field name="type" type="string" indexed="false" stored="true"/>
<field name="manName" type="text" indexed="false" stored="true"/>
<field name="manNum" type="string" indexed="false" stored="false"/>
</fields>
Now perhaps I am not doing this the right way... and there may be a better way to handle this.
Anyways the problem I am running into right now is that I am getting the error missing required field "id". Now products query and manufacturer query does not have an id column in the select.. but I thought the transform query should take care of it? If I do the select prodNum as id .. then all the ids are overwritting each other.
Now I could probably concat it in the actual query.. and will do so as a last resort, but would like to know what I am doing wrong with this solution.
EDIT
Nevermind, it was just a noob issue, for some reason I was thinking that the template variable was refering to the table name in the SQL not the entity name,
So I replaced all of the
With
And it worked.
Prefixing the table-specific ID with a distinct character or string is a good idea. I do it in the SQL, which allows me to check the behavior outside of Solr.
select
concat('b',cast(b.id as char)) as id,
...
It Was a noob issue,
for some reason I was thinking that the template variable was refering to the table name in the SQL not the entity name.
I do it like this:
<entity name="GG-Boryslaw-1939-Phonebook"
transformer="TemplateTransformer,DateFormatTransformer"
pk="id"
query="SELECT * FROM boryslaw_1939_phonebook">
<field column="record_id" template="GG-Boryslaw-1939-Phonebook-${GG-Boryslaw-1939-Phonebook.id}" />
<field column="record_type" template="phonebook" />
<field column="record_source" template="Boryslaw Phonebook (1939)" />
<field column="record_date" template="${GG-Boryslaw-1939-Phonebook.Year}" dateTimeFormat="yyyy" />
...etc...
</entity>
I'm developing search engine using Solr and I've been successful in indexing data from one table using DIH (Dataimport Handler). What I need is to get search result from 5 different tables. I couldn't do this without help.
if we assume x table with x rows, there should be x * x documents from each table, which lead to, 5x documents if I have 5 tables as total. In dataconfig.xml, I created 5 seperate entities in single document as shown below. the result from indexed data when I query *:* is only 6 of the entity users and 3 from entity classes which is the number of users total rows which is 9.
Clearly, this way didn't work for me, so how can I achieve this using only single core?
note: I followed DIHQuickStart and DIH tutorial which didn't help me.
<document>
<!-- Users -->
<entity dataSource="JdbcDataSource" name=" >
<field column="name" name="name" sourceColName="name" />
<field column="username" name="username" sourceColName="username"/>
<field column="email" name="email" sourceColName="email" />
<field column="country" name="country" sourceColName="country" />
</entity>
<!-- Classes -->
<entity dataSource="JdbcDataSource" name="classes" >
<field column="code" name="code" sourceColName="code" />
<field column="title" name="title" sourceColName="title" />
<field column="description" name="description" sourceColName="description" />
</entity>
<!-- Schools -->
<entity dataSource="JdbcDataSource" name="schools" >
<field column="school_name" name="school_name" sourceColName="school_name" />
<field column="country" name="country" sourceColName="country" />
<field column="city" name="city" sourceColName="city" />
</entity>
<!-- Resources -->
<entity dataSource="JdbcDataSource" name="resources" >
<field column="title" name="title" sourceColName="title" />
<field column="description" name="description" sourceColName="description" />
</entity>
<!-- Tasks -->
<entity dataSource="JdbcDataSource" name="tasks" >
<field column="title" name="title" sourceColName="title" />
<field column="description" name="description" sourceColName="description" />
</entity>
</document>
you need to look at the structures of your tables then either create queries with joins or creat nested entities like this for example
<dataConfig>
<dataSource driver="org.hsqldb.jdbcDriver" url="jdbc:hsqldb:/temp/example/ex" user="sa" />
<document name="schools">
<entity name="school" query="select * from schools s ">
<field column="school_name" name="school_Name" />
<entity name="school_class" query="select * from schools_classes sc where sc.school_name = '${school.school_name}'">
<field column="class_code" name="class_code" />
<entity name="class" query="select class_name from classes c where c.class_name= '${school_class.class_code}'">
<field column="class_name" name="class_name" />
</entity>
</entity>
</entity>
</document>
</dataConfig>