I've created an external cache dictionary which I load on start by setting dictionaries_lazy_load as False. The external dictionary is present in system.dictionaries as follows:
database ──┬─name─────────────┬─────────────────────────────────uuid─┬─status─┬─origin──────────────────────────────────────────┬─type──┬─key────┬─attribute.names────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─attribute.types───────────────────────────────────────────────────────────────────────────────────────────────────────┬─bytes_allocated─┬─query_count─┬─hit_rate─┬─element_count─┬─load_factor─┬─source──────────────────────────────────────────────────┬─lifetime_min─┬─lifetime_max─┬──loading_start_time─┬─last_successful_update_time─┬─loading_duration─┬─last_exception─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ benchmark │ cacheDictionary │ 00000000-0000-0000-0000-000000000000 │ LOADED │ /etc/clickhouse-server/benchmark_dictionary.xml │ Cache │ UInt64 │ ['date','integers','filterColumn','ramdom0','ramdom1','ramdom2','ramdom3','ramdom4','ramdom5','ramdom6','ramdom7','ramdom8','ramdom9'] │ ['Date','UInt64','Float64','String','String','String','String','String','String','String','String','String','String'] │ 26481680 │ 0 │ nan │ 0 │ 0 │ Executable: cat /var/lib/clickhouse/user_files/test.csv │ 0 │ 0 │ 2021-01-12 20:50:36 │ 2021-01-12 20:50:36 │ 0.009 │
but when I run the query SELECT dictGet('benchmark.cacheDictionary', 'random1', toUInt64(64)),
I get the error
Received exception from server (version 20.8.3): Code: 36. DB::Exception: Received from localhost:9000. DB::Exception: external dictionary 'benchmark.cacheDictionary' not found.
Any thoughts? I've added the xml code below.
<yandex>
<dictionary>
<name>cacheDictionary </name>
<database>benchmark</database>
<source>
<executable>
<command>cat /var/lib/clickhouse/user_files/test.csv</command>
<format>CSV</format>
</executable>
</source>
<layout>
<cache>
<size_in_cells>130000</size_in_cells>
</cache>
</layout>
<lifetime>0</lifetime>
<structure>
<id>
<name>referentialKey</name>
</id>
<attribute>
<name>date</name>
<type>Date</type>
<null_value></null_value>
</attribute>
<attribute>
<name>integers</name>
<type>UInt64</type>
<null_value></null_value>
</attribute>
<attribute>
<name>filterColumn</name>
<type>Float64</type>
<null_value></null_value>
</attribute>
<attribute>
<name>ramdom0</name>
<type>String</type>
<null_value></null_value>
</attribute>
<attribute>
<name>ramdom1</name>
<type>String</type>
<null_value></null_value>
</attribute>
<attribute>
<name>ramdom2</name>
<type>String</type>
<null_value></null_value>
</attribute>
<attribute>
<name>ramdom3</name>
<type>String</type>
<null_value></null_value>
</attribute>
<attribute>
<name>ramdom4</name>
<type>String</type>
<null_value></null_value>
</attribute>
<attribute>
<name>ramdom5</name>
<type>String</type>
<null_value></null_value>
</attribute>
<attribute>
<name>ramdom6</name>
<type>String</type>
<null_value></null_value>
</attribute>
<attribute>
<name>ramdom7</name>
<type>String</type>
<null_value></null_value>
</attribute>
<attribute>
<name>ramdom8</name>
<type>String</type>
<null_value></null_value>
</attribute>
<attribute>
<name>ramdom9</name>
<type>String</type>
<null_value></null_value>
</attribute>
</structure>
</dictionary>
</yandex>
XML dictionaries are exists out of database.
XML Dictionary: SELECT dictGet('cacheDictionary'
CREATE Dictionary: SELECT dictGet('benchmark.cacheDictionary'
https://clickhouse.tech/docs/en/sql-reference/functions/ext-dict-functions/#dictget
Attention
dict_name parameter must be fully qualified for
dictionaries created with DDL queries. Eg. ..*
Related
I follow this post[1] as a guide to build an example of query an array of UDT in WSO2 DSS. In the post just query an UDT, my config try to query an UDT array.
I created this in my DB, a dummy PROCEDURE to try this:
create or replace
TYPE "LIST_CUSTOMERS" IS TABLE OF customer_t
CREATE OR REPLACE
PROCEDURE getCustomer2(listcust OUT list_customers) IS
cust customer_t;
cust2 customer_t;
BEGIN
listcust := list_customers();
cust := customer_t(1, 'prabath');
cust2 := customer_t(2, 'jorge');
listcust.extend;
listcust(1) := cust;
listcust.extend;
listcust(2) := cust2;
END;
My DS is this:
<?xml version="1.0" encoding="UTF-8"?>
<data name="UDTSample2">
<config id="default">
<property name="org.wso2.ws.dataservice.driver">oracle.jdbc.driver.OracleDriver</property>
<property name="org.wso2.ws.dataservice.protocol">jdbc:oracle:thin:#localhost:1521:DBMB</property>
<property name="org.wso2.ws.dataservice.user">****</property>
<property name="org.wso2.ws.dataservice.password">****</property>
</config>
<query id="q3" useConfig="default">
<sql>call getCustomer2(?)</sql>
<result element="customers">
<element name="customer" arrayName="custArray" column="cust" optional="true"/>
</result>
<param name="cust" paramType="ARRAY" sqlType="ARRAY" type="OUT" structType="LIST_CUSTOMERS" />
</query>
<operation name="op3">
<call-query href="q3" />
</operation>
</data>
ant return:
<customers xmlns="http://ws.wso2.org/dataservice">
<customer>{1,prabath}</customer>
<customer>{2,jorge}</customer>
</customers>
but I want something like this:
<customers xmlns="http://ws.wso2.org/dataservice">
<customer>
<id>1</id>
<name>prabath<name>
</customer>
<customer>
<id>2</id>
<name>Jorge<name>
</customer>
</customers>
How can I accomplish this?
[1] http://prabathabey.blogspot.com/2012/05/query-udtsuser-defined-types-with-wso2.html
Not sure whether this kind of transformation can be done at DSS level because DSS gives back what it recieves from database. Better use WSO2 esb for this kind of transformation.
By the moment it's not possible to accomplish this scenario using just DSS. the DS response must be send to the WSO2 ESB to do the corresponding transformation before send the response to the client. A JIRA was created to do this in the future https://wso2.org/jira/browse/DS-1104
As a workaround, you can use a procedure returning a sys_refcursor.
It would look like this:
PROCEDURE getCustomer_CUR(cur_cust OUT SYS_REFCURSOR)
l_cust LIST_CUSTOMERS;
IS
-- Retrieve cust list:
getCustomer2(l_cust);
OPEN cur_cust for
select cast(multiset(select * from TABLE(l_cust)) as customer_t) from dual;
...
END;
Then you can do your DSS mapping something like:
<sql>call getCustomer_CUR(?)</sql>
<result element="customers">
<element arrayName="custArray" name="Customers">
<element column="custArray[0]" name="col0" xsdType=.../>
...
</element>
</result>
<param name="cust" sqlType="ORACLE_REF_CURSOR" type="OUT"/>
It is tedious but it works.
I am trying to read the xml and storing it in SQL server.
DECLARE #xml XML
SET #xml =
'<report>
<personal>
<search>
<subject>
<name>SearchName</name>
</subject>
</search>
</personal>
<personal>
<search>
<subject>
<name>SearchName</name>
</subject>
</search>
<result>
<history>
<name>HistoryName</name>
</history>
</result>
</personal>
<personal>
<search>
<subject>
<name>SearchName</name>
</subject>
</search>
<result>
<history>
<dob>HistoryDOB</dob>
</history>
</result>
</personal>
</report>
'
What i am trying here is - selecting the name but condition here is
if <personal> contains <result> then select the name under history/name
if <personal> doesn't contain <result> select the name under subject/name
if <personal> contain <result>BUT name is not there then enter null
I am using below query
SELECT
COALESCE(
A.Search.value('(result/history/name)[1]','varchar(max)'),
A.Search.value('(search/subject/name)[1]','varchar(max)')
) AS Name
FROM #xml.nodes('/report/personal') as A(Search)
It is returning
SearchName
HistoryName
SearchName
But it is failing in 3rd condition.
Just tweak the second values call to specifically request what you've specified - that you'll only take a search/subject for nodes with no result:
SELECT
COALESCE(
A.Search.value('(result/history/name)[1]','varchar(max)'),
A.Search.value('(search[not(../result)]/subject/name)[1]','varchar(max)')
) AS Name
FROM #xml.nodes('/report/personal') as A(Search)
Result:
Name
------------------
HistoryName
SearchName
NULL
You can use exist method to chech if <personal> contains <result> or not.
Using it your algorithm can be straightforward translated to query as:
select
case
when A.Search.exist('result') = 1
then A.Search.value('(result/history/name)[1]','varchar(max)')
else A.Search.value('(search/subject/name)[1]','varchar(max)')
end as Name
FROM #xml.nodes('/report/personal') as A(Search)
My xml file looks something like this:
<PackageRuntimeContext xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<UserToken>
<Id>449694</Id>
</UserToken>
<Addresses>
<Address>
<LastSeen xsi:nil="true" />
<UniqueID>9afd29f6-f4fe-4a91-aade-da8a3fcdc358</UniqueID>
<IsPrimary>true</IsPrimary>
<Id>0</Id>
<OrderID>0</OrderID>
<SubjectId>0</SubjectId>
<AddressLine1>123 Main St.</AddressLine1>
<City>louisville</City>
<State>KY</State>
<ZipCode>40206</ZipCode>
</Address>
<Address>
<LastSeen xsi:nil="true" />
<UniqueID>0ae8014e-a950-48f3-8ee6-3526a7f3a50d</UniqueID>
<IsPrimary>true</IsPrimary>
<Id>0</Id>
<OrderID>0</OrderID>
<SubjectId>0</SubjectId>
<AddressLine1>789 Elm St.</AddressLine1>
<City>louisville</City>
<State>KY</State>
<ZipCode>40206</ZipCode>
</Address>
<Address>
<LastSeen xsi:nil="true" />
<UniqueID>b1bcc271-bec8-432f-b968-25430ba63b95</UniqueID>
<IsPrimary>false</IsPrimary>
<Id>0</Id>
<OrderID>0</OrderID>
<SubjectId>0</SubjectId>
<AddressLine1>456 Oak St.</AddressLine1>
<City>louisville</City>
<State>KY</State>
<ZipCode>40206</ZipCode>
</Address>
</Addresses>
I want to get the <Id> number 449694, and with it, the 3 (or whatever) subsequent <UniqueID> numbers under Addresses/Address so it looks something like this:
IDNumber UniqueID
======== ========
449694 9afd29f6-f4fe-4a91-aade-da8a3fcdc358
449694 0ae8014e-a950-48f3-8ee6-3526a7f3a50d
449694 b1bcc271-bec8-432f-b968-25430ba63b95
The code If found here (How to query values from xml nodes?) directed me to write something like this:
SELECT
t.p.value('(./UserToken/Id)[1]', 'int') [IdNumber],
t.p.value('(./Addresses/Address/UniqueID)[1]', 'varchar(max)') [Context]
FROM product.PackageRuntimeState prs WITH(NOLOCK)
CROSS APPLY prs.Context.nodes('/PackageRuntimeContext') t(p)
My results were:
IDNumber UniqueID
======== ========
449694 9afd29f6-f4fe-4a91-aade-da8a3fcdc358
449694 b8439471-d4b9-46db-9321-b6175e1b8fb4 (this is from ANOTHER record)
449694 b8439471-d4b9-46db-9321-b6175e1b8fb4 (this too is from another record)
What do I need to do to my code to get the subsequent UniqueID nodes from my xml file?
Thanks!
Drop down one more level. You need to list the direct decendants of <Addresses>, not <PackageRuntimeContext>
SELECT
t.p.value('(../../UserToken/Id)[1]', 'int') [IdNumber],
t.p.value('(./UniqueID)[1]', 'varchar(max)') [Context]
FROM product.PackageRuntimeState prs WITH(NOLOCK)
CROSS APPLY prs.Context.nodes('/PackageRuntimeContext/Addresses/Address') t(p)
I am trying to generate an XML file which looks like following.
<products>
<product>
<sku>12345</sku>
<attributes>
<attribute>
<name>AttributeName</name>
<row>
<name>ProdName</name>
<value>someProduct</value>
</row>
</attribute>
<attribute>
<name>AttributeName</name>
<row>
<name>color</name>
<value>Blue</value>
</row>
</attribute>
</attributes>
</product>
</products>
With following SQL query I could generate something very close but need to fine tune it.
bcp "SELECT RTRIM(LTRIM(sku)) as sku,
(SELECT 'AttributeName' AS [name],'ProdName' AS [row/name],ProdName AS [row/value] FOR XML PATH ('attribute'), Type),
(SELECT 'AttributeName' AS [name],'color' AS [row/name], color AS [row/value] FOR XML PATH ('attribute'), Type)
FROM Tbl_Product WHERE (SKU = 12345) FOR XML PATH ('product'), ROOT('products'),Type" queryout "..\Desktop\sample.xml" -c -T
<products>
<product>
<sku>12345</sku>
<!-- need <attributes> here -->
<attribute>
<name>AttributeName</name>
<row>
<name>ProdName</name>
<value>someProduct</value>
</row>
</attribute>
<attribute>
<name>AttributeName</name>
<row>
<name>color</name>
<value>Blue</value>
</row>
</attribute>
<!-- end <attributes> here-->
</product>
</products>
Is there a way to nest all <attribute> nodes inside <attributes>.
Thanks in advance!
I wasn't aware of String concatenation (+) in SQL. It took me over a week to figure that one little thing out. Here's what solved my problem. Might help someone else in future.
bcp "SELECT RTRIM(LTRIM(sku)) as sku, (SELECT 'AttributeName' AS [attribute/name],
'ProdName' AS [attribute/row/name],
ProdName AS [attribute/row/value],
+'',
'AttributeName' AS [attribute/name],
'color' AS [attribute/row/name],
color AS [attribute/row/value]
FROM Tbl_Product Where (SKU = TP.SKU)
FOR XML PATH (''), ROOT('attributes'), Type)
FROM Tbl_Product TP WHERE (SKU = 12345)
FOR XML PATH ('product'), ROOT('products'),Type" queryout "..\Desktop\sample.xml" -c -T
In one of my sql scripts, I need to execute a stored procedure with the following xml string
<Collection xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<Field>
<Attributes>
<Attribute Name="CODE1" IsRequired="true" Order="1" IsVisible="true"/>
<Attribute Name="CODE2" IsRequired="true" Order="2" IsVisible="true"/>
</Attributes>
<Rows>
<Row ProductState="5">
<Items>
<Item Name="PROD1" SendCustomer="false"/>
<Item Name="PROD2" SendCustomer="false"/>
</Items>
</Row>
</Rows>
</Field>
</Collection>
I get the Attribute and the Item information from different tables. I am writing a generic function in which you pass an ID and returns this XML string that is used by the SQL script to execute the stored procedure
Sometimes, I need to override the attribute values of some elements like SendCustomer. My initial thought was to deserialize this to a temp table, update the temp table with the override value and then serialize it back to XML.
So, essentially, the entire process boils down to:
Query tables, serialize to XML in the function
Deserialze XML, store in temp table
Override values if necessary
Serialze from table to XML again
Is there a more elegant way in sql server 2005 to do this entire process?
The XML datatype actually can be modified using XQuery. See the modify() method.
declare #x XML;
select #x = N'<Collection xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<Field>
<Attributes>
<Attribute Name="CODE1" IsRequired="true" Order="1" IsVisible="true"/>
<Attribute Name="CODE2" IsRequired="true" Order="2" IsVisible="true"/>
</Attributes>
<Rows>
<Row ProductState="5">
<Items>
<Item Name="PROD1" SendCustomer="false"/>
<Item Name="PROD2" SendCustomer="false"/>
</Items>
</Row>
</Rows>
</Field>
</Collection>';
set #x.modify(N'replace value of
(/Collection/Field/Rows/Row/Items/Item[#Name="PROD2"]/#SendCustomer)[1]
with "true"');
select #x;