Database, relational query - database

Question

Chemicals have a trade name (which is what it is commonly referred to as) and an actual chemical name. I need to look up a trade name, find its proper chemical name, then get properties of that chemical. For example:
$tradeName = "Voranol"
if $tradeName == "Voranol" then
$productName = "Polyether Polyol"
$flare = "List I"
$bay = "1"
$listPos = 3
EndIf
I have an .au3 file that contains a ton of products. It works fine but it's kind of tedious to do if $tradeName == "Name1" or $tradeName == "Name2" or $tradeName == "Name4" or $tradeName == "Name5". I will also need to be able to edit it programmaticly. This is what I'm using now:
if $product == "dowanol pmb" or $product == "dowanol pma" or $product == "dipropylene glycol" or $product == "dowanol pnp" or $productCheck2 == "dowanol pmb" or $productCheck2 == "dowanol pma" or $productCheck2 == "dipropylene glycol" or $productCheck2 == "dowanol pnp" then
$result = "DIPROPYLENE GLYCOL"
$bay = 4
$useFlare = 1
$listPos = 2
elseif $product == "glycol blend" then
$result = "GLYCOL"
$bay = 3
$useFlare = 2
$listPos = 1
elseif $product == "isopar e" or $product == "isopar c" or $product == "isopar h" or $productCheck2 == "isopar h" then
$result = "PETROLEUM NAPTHA"
$bay = 5
$useFlare = 0
$listPos = 1
EndIf
; Note: 0 = No Flare, 1 = Normal Flare, 2 = CAS Flare

Answer 1

Another alternative to the database option is to use an XML document to store the product (chemical) information.
You could easily query the XML with XPath to get the product name/properties. Maintaining the file would be fairly easy too. You could even create an XML Schema to validate against to be sure your file is still valid after modifications.
The processing of the XML in AutoIt can be done a few different ways:
Creating an MSXML object
Running a command line tool like xmlstarlet
Using a XPath/XQuery/XSLT processor from the command line (i.e. java to run Saxon)
Running something like Saxon is probably overkill for what you need.
MSXML wouldn't be too bad and there should be multiple UDFs that already exist.
Xmlstarlet would be my vote. (Note: I haven't used xmlstarlet in this fashion before. I'm a huge fan of Saxon and use it almost exclusively. Specifically for AutoIt, I've used a combination of MSXML and Saxon; Saxon to do the transforms of complicated data into a smaller, simpler subset and then MSXML to do simple xpath queries on that subset. However, if I was going to do something like this, I'd give xmlstarlet a serious look.)
Also, if your data grows to the point that a single XML file does not make sense, you could always split it into a collection of smaller files; individual products maybe. You might also reach a point that it might make sense to load those files into an actual XML database (eXistdb is an option).
Here's a simple example of what your XML (schema and instance) might look like:
XSD (products.xsd)
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">
<xs:element name="products">
<xs:complexType>
<xs:sequence>
<xs:element maxOccurs="unbounded" ref="product"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="product">
<xs:complexType>
<xs:sequence>
<xs:element ref="name"/>
<xs:element ref="tradenames"/>
</xs:sequence>
<xs:attribute name="bay" use="required" type="xs:integer"/>
<xs:attribute name="listpos" use="required" type="xs:integer"/>
<xs:attribute name="useflare" use="required" type="xs:integer"/>
</xs:complexType>
</xs:element>
<xs:element name="tradenames">
<xs:complexType>
<xs:sequence>
<xs:element maxOccurs="unbounded" ref="name"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="name" type="xs:string"/>
</xs:schema>
XML (products.xml)
<products xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="products.xsd">
<product bay="4" useflare="1" listpos="2">
<name>DIPROPYLENE GLYCOL</name>
<tradenames>
<name>dowanol pmb</name>
<name>dowanol pma</name>
<name>dipropylene glycol</name>
<name>dowanol pnp</name>
</tradenames>
</product>
<product bay="3" useflare="2" listpos="1">
<name>GLYCOL</name>
<tradenames>
<name>glycol blend</name>
</tradenames>
</product>
<product bay="5" useflare="0" listpos="1">
<name>PETROLEUM NAPTHA</name>
<tradenames>
<name>isopar e</name>
<name>isopar c</name>
<name>isopar h</name>
</tradenames>
</product>
</products>
Here's a little bit of AutoIt that uses MSXML to demonstrate loading the XML (validates against XSD) and searching for products that have a trade name that contains "glycol".
AutoIt
;~ AutoIt Version: 3.3.14.2
;String to search on.
$searchString = "glycol"
ConsoleWrite("Search string: '" & $searchString & "'" & #CRLF)
;XPath for searching trade names. Search string is injected (code injection; escaping of strings would be a very good idea!).
$xpath_tradename = "/products/product[tradenames/name[contains(.,'" & $searchString & "')]]"
ConsoleWrite("XPath: '" & $xpath_tradename & "'" & #CRLF)
$msxml = ObjCreate('MSXML2.DOMDocument.6.0')
If IsObj($msxml) Then
$msxml.async = False
$msxml.validateOnParse = True
$msxml.resolveExternals = True
$msxml.setProperty("SelectionLanguage", "XPath")
$msxml.load('products.xml')
If $msxml.parseError.errorCode = 0 Then
$prods = $msxml.SelectNodes($xpath_tradename)
If IsObj($prods) And $prods.Length > 0 Then
ConsoleWrite("Number of products found: '" & $prods.Length & "'" & #CRLF)
For $prod In $prods
ConsoleWrite(#CRLF & "------ PRODUCT ------" & #CRLF)
ConsoleWrite("Product name: '" & $prod.SelectSingleNode('name').text & "'" & #CRLF)
ConsoleWrite("Product bay: '" & $prod.getAttribute('bay') & "'" & #CRLF)
Next
ConsoleWrite(#CRLF)
Else
ConsoleWrite("PRODUCT NOT FOUND" & #CRLF)
EndIf
Else
MsgBox(17, 'Error', 'Error opening XML file: ' & #CRLF & #CRLF & $msxml.parseError.reason)
SetError($msxml.parseError.errorCode)
EndIf
EndIf
Console Output
Search string: 'glycol'
XPath: '/products/product[tradenames/name[contains(.,'glycol')]]'
Number of products found: '2'
------ PRODUCT ------
Product name: 'DIPROPYLENE GLYCOL'
Product bay: '4'
------ PRODUCT ------
Product name: 'GLYCOL'
Product bay: '3'

Answer 2

The best database solution in Autoit is SQLite.
If you want to do it like a pro, you should use SQLite.
#include <SQLite.au3>
#include <SQLite.dll.au3>
Local $hQuery, $aRow
_SQLite_Startup()
ConsoleWrite("_SQLite_LibVersion=" & _SQLite_LibVersion() & #CRLF)
_SQLite_Open()
; Without $sCallback it's a resultless statement
_SQLite_Exec(-1, "Create table tblTest (a,b int,c single not null);" & _
"Insert into tblTest values ('1',2,3);" & _
"Insert into tblTest values (Null,5,6);")
Local $d = _SQLite_Exec(-1, "Select rowid,* From tblTest", "_cb") ; _cb will be called for each row
Func _cb($aRow)
For $s In $aRow
ConsoleWrite($s & #TAB)
Next
ConsoleWrite(#CRLF)
; Return $SQLITE_ABORT ; Would Abort the process and trigger an #error in _SQLite_Exec()
EndFunc ;==>_cb
_SQLite_Close()
_SQLite_Shutdown()
; Output:
; 1 1 2 3
; 2 5 6
As a simpler solution I would recommend using INI file as database.
[Voranol]
ProductName=Polyether Polyol
Flare=List I
Bay=1
ListPos=3
[Chem2]
ProductName=..
...
And then
; Read the INI section names. This will return a 1 dimensional array.
Local $aArray = IniReadSectionNames($sFilePath)
; Check if an error occurred.
If Not #error Then
; Enumerate through the array displaying the section names.
For $i = 1 To $aArray[0]
MsgBox($MB_SYSTEMMODAL, "", "Section: " & $aArray[$i])
Next
EndIf
Now there is a limit in windows on INI file size(32kb).
That means, certain Autoit INI functions will not work if that limit is breached.
This can be solved by using your own INI functions:
Func _IniReadSectionNamesEx($hFile)
Local $iSize = FileGetSize($hFile) / 1024
If $iSize <= 31 Then
Local $aNameRead = IniReadSectionNames($hFile)
If #error Then Return SetError(#error, 0, '')
Return $aNameRead
EndIf
Local $aSectionNames = StringRegExp(#CRLF & FileRead($hFile) & #CRLF, '(?s)\n\s*\[(.*?)\]s*\r', 3)
If IsArray($aSectionNames) = 0 Then Return SetError(1, 0, 0)
Local $nUbound = UBound($aSectionNames)
Local $aNameReturn[$nUbound + 1]
$aNameReturn[0] = $nUbound
For $iCC = 0 To $nUBound - 1
$aNameReturn[$iCC + 1] = $aSectionNames[$iCC]
Next
Return $aNameReturn
EndFunc
Func _IniReadSectionEx($hFile, $vSection)
Local $iSize = FileGetSize($hFile) / 1024
If $iSize <= 31 Then
Local $aSecRead = IniReadSection($hFile, $vSection)
If #error Then Return SetError(#error, 0, '')
Return $aSecRead
EndIf
Local $sFRead = #CRLF & FileRead($hFile) & #CRLF & '['
$vSection = StringStripWS($vSection, 7)
Local $aData = StringRegExp($sFRead, '(?s)(?i)\n\s*\[\s*' & $vSection & '\s*\]\s*\r\n(.*?)\[', 3)
If IsArray($aData) = 0 Then Return SetError(1, 0, 0)
Local $aKey = StringRegExp(#LF & $aData[0], '\n\s*(.*?)\s*=', 3)
Local $aValue = StringRegExp(#LF & $aData[0], '\n\s*.*?\s*=(.*?)\r', 3)
Local $nUbound = UBound($aKey)
Local $aSection[$nUBound +1][2]
$aSection[0][0] = $nUBound
For $iCC = 0 To $nUBound - 1
$aSection[$iCC + 1][0] = $aKey[$iCC]
$aSection[$iCC + 1][1] = $aValue[$iCC]
Next
Return $aSection
EndFunc

Answer 3

I need to look up a trade name, find its proper chemical name, then get properties of that chemical.
Relational query (SQLite)
SQL as understood by SQLite.
If requirements are limited to inserts, edits and queries, an SQLite database management tool may suffice. AutoIt SQLite functions are documented as per User-defined function-reference:
(Create and) open database using _SQLite_Open().
Create tables , indexes and view(s) (and insert records) using _SQLite_Exec().
Queries using _SQLite_GetTable2d() return results as two-dimensional array.
Structure
SQLite data types. Inserting records to product and substance table may be automated.
Table product
Defines trade names.
id
name
1
dowanol pmb
2
dowanol pma
3
dipropylene glycol
4
dowanol pnp
5
glycol blend
6
isopar e
7
isopar c
8
isopar h
9
Polyether Polyol
10
Voranol
Creation:
CREATE TABLE IF NOT EXISTS product (
id INTEGER PRIMARY KEY,
name TEXT UNIQUE
);
Insert record:
INSERT INTO product (name) VALUES ('dowanol pmb');
Table substance
Defines chemical names (and properties).
id
name
flare
bay
1
Polyether Polyol
1
1
2
DIPROPYLENE GLYCOL
1
4
3
GLYCOL
2
3
4
PETROLEUM NAPHTA
0
5
Creation:
CREATE TABLE IF NOT EXISTS substance (
id INTEGER PRIMARY KEY,
name TEXT UNIQUE,
flare INTEGER,
bay INTEGER
);
Insert record:
INSERT INTO substance (name, flare, bay) VALUES ('Polyether Polyol', 1, 1);
Table relation
Defines relations between product and substance table's records (using foreign keys).
id
product
substance
1
1
2
2
2
2
3
3
2
4
4
2
5
5
3
6
6
4
7
7
4
8
8
4
9
9
4
10
10
1
Creation:
CREATE TABLE IF NOT EXISTS relation (
id INTEGER PRIMARY KEY,
product INTEGER REFERENCES product(id),
substance INTEGER REFERENCES substance(id),
UNIQUE( /* Constraint applies to *combination* of columns. */
product,
substance
)
);
Insert record:
INSERT INTO relation (
product,
substance
) VALUES (
(SELECT id FROM product WHERE name = 'dowanol pmb'),
(SELECT id FROM substance WHERE name = 'DIPROPYLENE GLYCOL')
);
Or just:
INSERT INTO relation (product, substance) VALUES (0, 1);
View view_relation
View (stored query), joining (combines) product and substance tables' fields, as per relation table's records (acting as virtual table, negating need to include underlying JOIN operators every query).
name_product
name_substance
flare
bay
dowanol pmb
DIPROPYLENE GLYCOL
1
4
dowanol pma
DIPROPYLENE GLYCOL
1
4
dipropylene glycol
DIPROPYLENE GLYCOL
1
4
dowanol pnp
DIPROPYLENE GLYCOL
1
4
glycol blend
GLYCOL
2
3
isopar e
PETROLEUM NAPTHA
0
5
isopar c
PETROLEUM NAPTHA
0
5
isopar h
PETROLEUM NAPTHA
0
5
Polyether Polyol
PETROLEUM NAPTHA
0
5
Voranol
Polyether Polyol
1
1
Creation:
CREATE VIEW IF NOT EXISTS view_relation AS
SELECT
product.name AS name_product,
substance.name AS name_substance,
substance.flare,
substance.bay
FROM
relation
LEFT OUTER JOIN product ON relation.product = product.id
LEFT OUTER JOIN substance ON relation.substance = substance.id
ORDER BY
product.id ASC
;
Query by trade name
name_product
name_substance
flare
bay
Voranol
Polyether Polyol
1
1
Query:
SELECT
*
FROM
view_relation
WHERE
name_product = 'Voranol'
;
Or (without view):
SELECT
product.name AS name_product,
substance.name AS name_substance,
substance.flare,
substance.bay
FROM
relation
WHERE
product.name = 'Voranol'
LEFT OUTER JOIN product ON relation.product = product.id
LEFT OUTER JOIN substance ON relation.substance = substance.id
;
Query by substance
name_product
name_substance
flare
bay
dowanol pmb
DIPROPYLENE GLYCOL
1
4
dowanol pma
DIPROPYLENE GLYCOL
1
4
dipropylene glycol
DIPROPYLENE GLYCOL
1
4
dowanol pnp
DIPROPYLENE GLYCOL
1
4
Query:
SELECT
*
FROM
view_relation
WHERE
name_substance = 'DIPROPYLENE GLYCOL'
;
Or (without view):
SELECT
product.name AS name_product,
substance.name AS name_substance,
substance.flare,
substance.bay
FROM
relation
WHERE
substance.name = 'DIPROPYLENE GLYCOL'
LEFT OUTER JOIN product ON relation.product = product.id
LEFT OUTER JOIN substance ON relation.substance = substance.id
;
AutoIt
Example creating (and inserting records to) a single table and query it:
#include <SQLite.au3>
#include <SQLite.dll.au3>
#include <Array.au3>
Global Const $g_iRecords = 50
Global Const $g_sFileDb = #ScriptDir & '\example.sqlite'
Global $g_aCreate = [ _
'CREATE TABLE IF NOT EXISTS example (id INTEGER PRIMARY KEY, name TEXT UNIQUE);' & #LF, _
'--CREATE INDEX IF NOT EXISTS index_example_name ON example(name);' & #LF _
]
Global Const $g_sInsert = 'INSERT INTO example (name) VALUES (%i);\n'
Global Const $g_sTransact1 = 'BEGIN TRANSACTION;' & #LF
Global Const $g_sTransact2 = 'END TRANSACTION;' & #LF
Global Const $g_sQuery = 'SELECT * FROM example ORDER BY id ASC;'
Global Const $g_sMsgError = 'sqlite.dll not found.' & #LF
Global $g_hDb = 0
Main()
Func Main()
Local $sQuery = ''
Local $iResultCol = 0
Local $iResultRow = 0
Local $aResult
For $i1 = 0 To UBound($g_aCreate, 1) -1
$sQuery &= $g_aCreate[$i1]
Next
For $i1 = 1 To $g_iRecords
$sQuery &= StringFormat($g_sInsert, _SQLite_FastEscape('example' & $i1))
Next
$sQuery = $g_sTransact1 & $sQuery & $g_sTransact2
ConsoleWrite($sQuery)
_SQLite_Startup()
If #error Then
ConsoleWrite($g_sMsgError)
Exit
EndIf
_SQLite_Open($g_sFileDb)
_SQLite_Exec($g_hDb, $sQuery)
_SQLite_GetTable2d($g_hDb, $g_sQuery, $aResult, $iResultRow, $iResultCol)
_SQLite_Close($g_hDb)
_SQLite_Shutdown()
_ArrayDisplay($aResult)
Exit
EndFunc