XML Attributes to SQL - sql-server

I've seen posts that are similar to what I'm about to ask, but I can't find anything that actually solves my problem. I've seen (and duplicated) getting XML tags into SQL, but not getting the attributes of the tags into SQL.
Background: We use a program that runs "events" based on each event's schedule. The schedules are in XML. Ultimately, I'm trying to compare the last time a given event ran to the last time it should have run, based on its schedule. Once I get the schedules out of XML and into a table (or tables), I'm confident I can take it from there. But, I'm STRUGGLING with that first step.
Below is the XML I'm trying to get into a table (or tables). Any help would be greatly appreciated!!
<Schedule LastModified="2016-06-27T21:02:10.6041531Z" TimeZone="(UTC-06:00) Central Time (US & Canada)" ConvertedToUTC="True" Type="Weekly">
<Beginning StartDate="2016-05-26T22:26:00.0000000" />
<Block BlockType="AllDay" Interval="10" IntervalType="Minute" SetType="Inclusive" Start="15:00:00" End="17:00:00" Duration="02:00:00" />
<Interval Type="Weekly" RecurEveryX="1" Sunday="False" Monday="True" Tuesday="True" Wednesday="True" Thursday="True" Friday="True" Saturday="False" />
<Ending Type="NoEndDate" />
</Schedule>

Don't know, where you have your issues... Reading this XML is rather trivial (did not code all of the values, but you'll get the idea):
DECLARE #mockup TABLE(YourXML XML);
INSERT INTO #mockup VALUES
(N'<Schedule LastModified="2016-06-27T21:02:10.6041531Z" TimeZone="(UTC-06:00) Central Time (US & Canada)" ConvertedToUTC="True" Type="Weekly">
<Beginning StartDate="2016-05-26T22:26:00.0000000" />
<Block BlockType="AllDay" Interval="10" IntervalType="Minute" SetType="Inclusive" Start="15:00:00" End="17:00:00" Duration="02:00:00" />
<Interval Type="Weekly" RecurEveryX="1" Sunday="False" Monday="True" Tuesday="True" Wednesday="True" Thursday="True" Friday="True" Saturday="False" />
<Ending Type="NoEndDate" />
</Schedule>');
SELECT m.YourXML.value(N'(/Schedule/#LastModified)[1]',N'datetime') AS Schedule_LastModified
,m.YourXML.value(N'(/Schedule/#TimeZone)[1]',N'nvarchar(max)') AS Schedule_TimeZone
,m.YourXML.value(N'(/Schedule/Beginning/#StartDate)[1]',N'datetime') AS Beginning_StartDate
,m.YourXML.value(N'(/Schedule/Block/#BlockType)[1]',N'nvarchar(max)') AS Block_BlockType
,m.YourXML.value(N'(/Schedule/Block/#Interval)[1]',N'int') AS Block_Interval
,m.YourXML.value(N'(/Schedule/Interval/#Type)[1]',N'nvarchar(max)') AS Interval_Type
FROM #mockup AS m;
If this is not your solution, please edit your question: Add your own attempt, the wrong output and the expected output. Your explanation did not make is all clear to me...

Related

Disable clickhouse logs in the system database

In clickhouse, there is a database called system where logs are stored.
My problem is that after installing clickhouse, the volume of the system database has increased a day, and I sent a photo of it, and if I only use it for 30 days, I have to allocate nearly 30 gigs of space on the server just for the system database, which costs It will be high.
Especially the two tables trace_log and part_log take a lot of space
How to disable the logs in the system database?
I have already seen the link below and did everything and it didn't work (link).
The following command does not work to prevent system database logs:
set log_queries = 0;
And also the following code does not work for me:
cat /etc/clickhouse-server/users.d/log_queries.xml
<?xml version="1.0" ?>
<yandex>
<users>
<default>
<log_queries>0</log_queries>
</default>
</users>
</yandex>
I even went to this path sudo nano /etc/clickhouse-server/config.xml
and I entered the following values, but it didn't work:
<logger>
<level>none</level>
<output>null</output>
</logger>
In addition, I restarted clickhouse every time to apply the changes
It is interesting here that when I do not insert any data into my database in my codes, the system database increases in size for no reason.
I searched a lot and did a lot of tests, but I didn't get any results. Thank you for your guidance
https://kb.altinity.com/altinity-kb-setup-and-maintenance/altinity-kb-system-tables-eat-my-disk/
You can disable all / any of them
Do not create log tables at all (a restart is needed for these changes to take effect).
$ cat /etc/clickhouse-server/config.d/z_log_disable.xml
<?xml version="1.0"?>
<clickhouse>
<asynchronous_metric_log remove="1"/>
<metric_log remove="1"/>
<query_thread_log remove="1" />
<query_log remove="1" />
<query_views_log remove="1" />
<part_log remove="1"/>
<session_log remove="1"/>
<text_log remove="1" />
<trace_log remove="1"/>
<crash_log remove="1"/>
<opentelemetry_span_log remove="1"/>
<zookeeper_log remove="1"/>
</clickhouse>
And you need to drop existing tables
drop table system.trace_log;
...
The settings you referenced control the query_log table, more details are available here:
https://clickhouse.com/docs/en/operations/system-tables/query_log/
Note that it is not recommended to turn off the query_log because information in this table is important for solving issues.
The trace_log and part_log tables are different and shouldn't be enabled by default, you can locate these blocks in your config.xml and comment them:
<trace_log>
<database>system</database>
<table>trace_log</table>
<partition_by>toYYYYMM(event_date)</partition_by>
<flush_interval_milliseconds>7500</flush_interval_milliseconds>
</trace_log>
and
<part_log>
<database>system</database>
<table>part_log</table>
<partition_by>toMonday(event_date)</partition_by>
<flush_interval_milliseconds>7500</flush_interval_milliseconds>
</part_log>
Reference:
https://clickhouse.com/docs/en/operations/server-configuration-parameters/settings#server_configuration_parameters-trace_log
https://clickhouse.com/docs/en/operations/server-configuration-parameters/settings/#server_configuration_parameters-part-log

Having an issue with XML containing nested objects to POCO (de)serialisation (VB.Net)

I've created an SP in SQL Server that returns as XML. I decided to do this as the information has contacts and addresses in it and I wanted to reduce the amount of data I get.
<Accounts>
<Account>
<Company />
<ContactNumber />
<Addresses>
<Address>
<Line1 />
....
</Address>
<Addresses>
<Contacts>
<Contact>
<Name />
....
</Contact>
<Account>
</Accounts>
I have found SqlCommand.ExecuteXmlReader but I'm confused as to how to serialise this into my POCO. Can someone point me at what my next step is. (The POCO was created by the Insert XML as a class menu item in VS2019).
My Google fu is letting me down as I'm not sure what I should be looking for to help understand how to serialize the XML into something that will allow me to go with Foreach Account in AccountsClass type logic.
Any help is greatly appreciated.
PS The XML above is just a sample to show what I'm doing. The actual data is over 70 fields and with two FK joins the initial 40000 rows is well in excess of 1.8 million once selected as a normal dataset.
EDIT: Just in case someone stumbles on this and are in the same situation I was in.
When preparing a sample record for the Past XML to class make sure you have more than one record if you are expecting something similar to my example above. (The class changes to support more than one record.)
You get very different results when searching for deserialize when doing your research. This small change resulted in me figuring out the issue.

How to retrieve multiple XML siblings - SQL Server

I have searched thoroughly and not found any examples or guidance on how I can query and retrieve the values of multiple XML siblings.
There are plenty of examples on how to retrieve a single sibling value.
For example given the following XML fragment:
<StoreSurvey>
<AnnualSales>800000</AnnualSales>
<AnnualRevenue>80000</AnnualRevenue>
<BankName>United Security</BankName>
<BusinessType>BM</BusinessType>
<YearOpened>1996</YearOpened>
<Specialty>Mountain</Specialty>
<SquareFeet>21000</SquareFeet>
<Brands>2</Brands>
<Internet>ISDN</Internet>
<NumberEmployees>13</NumberEmployees>
<Products Type="Bikes">
<Product>Mountain</Product>
<Product>Road</Product>
<Product>Racing</Product>
</Products>
<Products Type="Clothes">
<Product>Jerseys</Product>
<Product>Jackets</Product>
<Product>Shorts</Product>
</Products>
</StoreSurvey>
If I want to retrieve the value of <AnnualSales>, I can execute the following statement in query analyzer:
SELECT Survey_untyped.query('/StoreSurvey/AnnualSales')
FROM Stores;
The result set will properly display
<AnnualSales>800000</AnnualSales>
However, what if I want to retrieve both <AnnualSales> AND <AnnualRevenue>? How would I do that?
The query should provide a result set looking like:
<AnnualSales>800000</AnnualSales>
<AnnualRevenue>80000</AnnualRevenue>
Or what if I want three of the sibling values “BankName” in addition to those values? Result set would look like this:
<AnnualSales>800000</AnnualSales>
<AnnualRevenue>80000</AnnualRevenue>
<BankName>United Security</BankName>
Does anyone know the answer to that?
This XPath, which uses the self:: axis,
/StoreSurvey/*[self::AnnualSales or self::AnnualRevenue or self::BankName]
will select
<AnnualSales>800000</AnnualSales>
<AnnualRevenue>80000</AnnualRevenue>
<BankName>United Security</BankName>
from your XML, as requested.

Can You Optimize XML Operations in SQL Server?

I am generating & sending XML EVENTS from the database through a SQL BROKER using SQL CLR - and it works great. However, I am looking at the SQL PLAN and am a little shocked at some of the statistics. Small transformations seem to cost quite a bit of CPU TIME.
All the examples I see online optimize the TABLE the XML sits in by adding an index (etc)...but there is no table for me (I am simply generating the XML).
As such...
Q: Is there a way to "optimize" these kind of "generational" statements?
Maybe some approaches are better than others?
I have yet to see anything online about this
Thanks.
SAMPLES OF EXPENSIVE STATEMENTS:
DECLARE #CurrentId UNIQUEIDENTIFIER = (SELECT #Event.value('(/Event/#auditId)[1]', 'UNIQUEIDENTIFIER'));
SET #Event.modify('replace value of (/Event/#auditId)[1][1] with sql:variable("#NewId")');
EVENT XML:
An event would look like...
<Event auditId="FE4D0A4C-388B-E611-9B4D-0050569B733D" force="false" CreatedOn="2016-10-05T20:14:20.020">
<DataSource machineName="ABC123">DatabaseName</DataSource>
<Topic>
<Filter>TOPIC/ENTITY/ACTION</Filter>
</Topic>
<Name>Something.Created</Name>
<Contexts>
<Context>
<Name>TableName</Name>
<Key>
<IssueId>000</IssueId>
</Key>
</Context>
</Contexts>
</Event>
An XML index will not help you with this (Read this). There are very rare situations, where this kind of index would help you. The effect is high, if you read from your XML with full-path. In the moment you are using XQuery, any kind of navigation, it makes things even worse.
.modify() is quite heavy. In this special case it could be faster to rebuild the XML as such (you know more about it than the engine does):
DECLARE #xml XML=N'
<Event auditId="FE4D0A4C-388B-E611-9B4D-0050569B733D" force="false" CreatedOn="2016-10-05T20:14:20.020">
<DataSource machineName="ABC123">DatabaseName</DataSource>
<Topic>
<Filter>TOPIC/ENTITY/ACTION</Filter>
</Topic>
<Name>Something.Created</Name>
<Contexts>
<Context>
<Name>TableName</Name>
<Key>
<IssueId>000</IssueId>
</Key>
</Context>
</Contexts>
</Event>';
DECLARE #NewId UNIQUEIDENTIFIER=NEWID();
SELECT #NewId AS [#auditId]
,e.value('#force','nvarchar(max)') AS [#force] --read this as string to avoid expensive conversions
,e.value('#CreatedOn','nvarchar(max)') AS [#CreatedOn] --same here
,e.query('*') AS [node()] --read "as-is"
FROM #xml.nodes('/Event') AS A(e)
FOR XML PATH('Event');
There is - for sure! - no general approach to get XML things faster. If this existed, it would be the one and only ...
I'd monitor the system and pick out the most expensive calls and try to modify them one by one...

Where is the data saved, from the Datatable-items, in ECM eRoom-database?

I am trying to retrieve data out of the ECM eRoom Database (which isn't documented, as I know of).
I have an eRoom with an custom "Database", some Fields.
When I query the objects table I find the "Database" row
select * from[dbo].[Objects] where internalId = 1234567
and the Rows for the entries
select top 10 * from[dbo].[Objects] where parentInternalId = 1234567
but I don't find any field with the values of the entries, only an Column with NonSearchableProperties., that is only full with Hex Data.
My Question(s),
how could i retrieve the values?
Is it possible to retrieve them with mere SQL?
What is the simplest way?
This is not the silver bullet, but it is okay for my usecase
After long goolging and alot of test scripts, i found some answers, but probably due to the fact the the system is soon end-of-life and that the documentation is not easy to read, here are my finding.
Is it possible to retrieve them with mere SQL?
As far as I could find out, no! (please correct me If I'm wrong)
how could i retrieve the values?
With the eRoom API(on the Server there are some Sample programms to query the data/objects <installation-path>\eRoom Server\Toolkit\Samples, with c++, vb, vbscript, ... all a bit to much overhead), or with the eRoom XML Query Language(exql) over soap calls.
What is the simplest way?
After alot of tests, searching in forums and many tests with soap ui. I found out that queries with exql seem to be the simplest way to retrieve Data, if you understand the structure.
Here some ressource that were helpful:
(very) Basic info of exql from the manufacturer https://eroom.asce.org/eRoomHelp/en/XML_Help/Introduction.htm
(disclaimer: I didn't find it helpful, but it shows at least some basics)
short 9 page Developer guide https://developer-content.emc.com/developer/downloads/eRoomXMLCapabilitiesUseCaseProgramDashboard.pdf (the last example on page 8, helped me understand how to setup the query, with alot of fantasy)
But for this to work, don't forget, to activate Allow XML queries and commands from external applications in the Site Settings
TIP 1:
you always can go go deeper you just need to know the right xml-element under. <Database>, <Cells> and <DBCell> can help you go deeper
TIP 2:
don't query to much data since this query likely run into timeouts
Update 1:
Just to save time for anyone who is looking, this "query" returns all rows (properties) for a Database(s) created in an eRoom Root.
(don't forget to set facility and room in the Url ex. http://server/eroomxml/facilities/TEST/Rooms/TestRoom, although it could be set in the query)
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:er="http://www.eroom.com/eRoomXML/2003/700">
<soapenv:Header/>
<soapenv:Body>
<er:ExecuteXMLCommand>
<er:eRoomXML>
<er:command er:select="HomePage/Items">
<er:getproperties>
<er:Item>
<Database>
<Rows>
<Item>
<Cells>
<DBCell>
<Content>
</Content>
</DBCell>
</Cells>
</Item>
</Rows>
</Database>
</er:Item>
</er:getproperties>
</er:command>
</er:eRoomXML>
</er:ExecuteXMLCommand>
</soapenv:Body>
</soapenv:Envelope>

Resources