Dynamically build a SQL Insert statement based on results from a DataView - sql-server

I have a legacy data logging industrial app that I'm writing a new interface for. The program lets you select points on devices, save those to a profile, then select devices to apply that profile for. When you apply the profile it create a table for each device using the devices unique ID as the table name and creates columns for each point of data you will be logging using the unique point ID. For example I select 3 points of information to datalog and it saves those three as a Profile (into it's own table) and then the point into the Points table tagged with that Profile:
PointID PointName ProfileID
33 Temp23 1
34 Hum14 1
35 Stat 1
I then select a couple devices and apply that profile which saves to the Device table:
DeviceID DeviceName ProfileID
5 NWUnit 1
6 NEUnit 1
After it saves the devices it creates the table per device such as:
Table Name: DEV5
Column 1: PNT1 - Float
Column 2: PNT2 - Float
Column 3: PNT3 - Bit
As you can see the table names are directly related to the device ID and the column names directly related to the point ID. I can add/remove points form the profile, it adds/deletes columns as needed. Apply a different profile and the DEV tables get deleted and recreated. Everything works as expected like the old program that's being replaced.
Now I need to actually do the data logging. I created a simple view:
SELECT dbo.Devices.DeviceID, dbo.Points.PointName, dbo.Points.PointID
FROM dbo.Devices LEFT OUTER JOIN
dbo.Points ON dbo.Devices.ProfileID = dbo.Points.ProfileID
Again so far so good:
DeviceID PointName PointID
5 Temp23 33
5 Hum14 34
5 Stat 35
6 Temp23 33
6 Hum14 34
6 Stat 35
I take this and I throw it in a DataTable, do a Columns.Add("Value") to it to get a blank column, then go through a data retrieval. When it's done I now have the table with the retrieved value:
DeviceID PointName PointID Value
5 Temp23 33 72.34
5 Hum14 34 43.8
5 Stat 35 1
6 Temp23 33 76.80
6 Hum14 34 54.2
6 Stat 35 0
And that's where I'm stuck. I need to take this info, use the DeviceID for the table name and the PointID for the column name, and insert the data. In otherwords I need this:
Dim myParamList As New Dictionary(Of String, Object) From {
{"#SampleTime", Date.Now},
{"#DevTable", "Dev" & r.Item("DeviceID")},
HOW DO I CYCLE THROUGH TO GET THE COLUMNS HERE?
}
UpdateDatabase(MySQLConnection, "INSERT INTO #DevTable (SampleTime, AND HERE?) VALUES (#SampleTime, AND HERE)", myParamList)
I cannot figure out the cycling through part. I thought I should use a Count + Group By to find out how many rows have the same device ID, like DeviceID 5 has 3 rows, and use that to cycle through that number of times but I'm just stuck trying to figure out how.
Any suggestions on the best way to do this?

So after struggling with trying to do a GroupBy on a dataview I decided to just do another database query with a Count(*) and GroupBy DeviceID to grab my unique DeviceIDs:
DeviceID RowCount
5 3
6 3
I then used that to loop through the device ID's and used the ID to filter myView as needed. Then I dynamically created a parameterized SQL string and update the database:
For Each r As DataRow In DevIDDataset.Tables("DeviceIDs").Rows
myView.RowFilter = "DeviceID=" & r.Item("DeviceID")
Dim myParamList As New Dictionary(Of String, Object) From {
{"#SampleTime", Date.Now}
}
Dim myFields As String = "SampleTime"
Dim myValues As String = "#SampleTime"
For Each row As DataRowView In myView
Dim myPointID As String = row.Item("PointID")
myFields += ",obj" & myPointID
myParamList.Add("#obj" & myPointID, row.Item("RetrievedValue"))
myValues += ",#obj" & myPointID
Next
UpdateDatabase(MySQLConnection, "INSERT INTO dev" & r.Item("DeviceID") & " (" & myFields & ") VALUES (" & myValues & ")", myParamList)
Next
Not pretty but it does what it needs to do and I can't think of any other way to do it.

Related

Flink : How to save list of rows in Database

Right now I am reading rows from a file and saving in database using the below code:
String strQuery = "INSERT INTO public.alarm (id, name, marks) VALUES (?, ?, ?)";
JDBCOutputFormat jdbcOutput = JDBCOutputFormat.buildJDBCOutputFormat()
.setDrivername("org.postgresql.Driver")
.setDBUrl("jdbc:postgresql://localhost:5432/postgres?user=michel&password=polnareff")
.setQuery(strQuery)
.setSqlTypes(new int[] { Types.INTEGER, Types.VARCHAR, Types.INTEGER}) //set the types
.finish();
DataStream<Row> rows = FilterStream
.map((tuple)-> {
Row row = new Row(3);
row.setField(0, tuple.f0);
row.setField(1, tuple.f1);
row.setField(2, tuple.f2);
return row;
});
rows.writeUsingOutputFormat(jdbcOutput);
env.execute();
}
}
The above is working fine and it picks rows one by one from a file and saves it in the database.
For example:
If the file contains:
1, mark, 20
then database entry will look like:
id name marks
------------------
1 mark 20
Now the requirement is for every row, I have to create 2 different rows and it should look like below:
For example:
If the file contains:
1, mark, 20
then database entry should look like this:
id name marks
------------------
1 mark-1 20
1 mark-2 20
Now I should return List instead of row and datastream variable should look like DataStream<List<Row>> rows.
What should I change in JDBCOutputFormat variable in order to achieve this?

Calculate Sum and Insert as Row

Using SSIS I am bringing in raw text files that contain this in the output:
I use this data later to report on. The Key columns get pivoted. However, I don't want to show all those columns individually, I only want to show the total.
To accomplish this my idea was calculate the Sum on insert using a trigger, and then insert the sum as a new row into the data.
The output would look something like:
Is what I'm trying to do possible? Is there a better way to do this dynamically on pivot? To be clear I'm not just pivoting these rows for a report, there are other ones that don't need the sum calculated.
Using derived column and Script Component
You can achieve this by following these steps:
Add a derived column (name: intValue) with the following expression:
(DT_I4)(RIGHT([Value],2) == "GB" ? SUBSTRING([Value],1,FINDSTRING( [Value], " ", 1)) : "0")
So if the value ends with GB then the number is taken else the result is 0.
After that add a script component, in the Input and Output Properties, click on the Output and set the SynchronousInput property to None
Add 2 Output Columns outKey , outValue
In the Script Editor write the following script (VB.NET)
Private SumValues As Integer = 0
Public Overrides Sub PostExecute()
MyBase.PostExecute()
Output0Buffer.AddRow()
Output0Buffer.outKey = ""
Output0Buffer.outValue = SumValues.ToString & " GB"
End Sub
Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
Output0Buffer.AddRow()
Output0Buffer.outKey = Row.Key
Output0Buffer.outValue = Row.Value
SumValues += Row.intValue
End Sub
I am going to show you a way but I don't recommend adding total to the end of the detail data. If you are going to report on it show it as a total.
After source add a data transformation:
C#
Add two columns to your data flow: Size int and type string
Select Value as readonly
Here is the code:
string[] splits = Row.value.ToString().Split(' '); //Make sure single quote for char
int goodValue;
if(Int32.TryParse(splits[0], out goodValue))
{
Row.Size = goodValue;
Row.Type = "GB";
}
else
{
Row.Size = 0;
Row.Type="None";
}
Now you have the data with the proper data types to do arithmatic in your table.
If you really want the data in your format. Add a multicast and an aggregate and SUM(Size) and then merge back into your original flow.
I was able to solve my problem in another way using a trigger.
I used this code:
INSERT INTO [Table] (
[Filename]
, [Type]
, [DeviceSN]
, [Property]
, [Value]
)
SELECT ms.[Filename],
ms.[Type],
ms.[DeviceSN],
'Memory Device.Total' AS [Key],
CAST(SUM(CAST(left(ms.[Value], 2) as INT)) AS VARCHAR) + ' GB' as 'Value'
FROM [Table] ms
JOIN inserted i ON i.Row# = ms.Row#
WHERE ms.[Value] like '%GB'
GROUP BY ms.[filename],
ms.[type],
ms.[devicesn]

Removing the repeating elements from a row in a squlite table

Please let me know if there is any query where in I remove the repeating entries in a row.
For eg: I have a table which has name with 9 telephone numbers:
Name Tel0 Tel1 Tel2 Tel3 Tel4 Tel5 Tel6 Tel7 Tel8
John 1 2 2 2 3 3 4 5 1
The final result should be as shown below:
Name Tel0 Tel1 Tel2 Tel3 Tel4 Tel5 Tel6 Tel7 Tel8
John 1 2 3 4 5
regards
Maddy
I fear that it will be more complicated to keep this format than to split the table in two as I suggested. If you insist on keeping the current schema then I would suggest that you query the row, organise the fields in application code and then perform an update on the database.
You could also try to use SQL UNION operator to give you a list of the numbers, a UNION by default will remove all duplicate rows:
SELECT Name, Tel FROM
(SELECT Name, Tel0 AS Tel FROM Person UNION
SELECT Name, Tel1 FROM Person UNION
SELECT Name, Tel2 FROM Person) ORDER BY Name ;
Which should give you a result set like this:
John|1
John|2
You will then have to step through the result set and saving each number into a separate variable (skipping those variables that do not exist) until the "Name" field changes.
Tel1 := Null; Tel2 := Null;
Name := ResultSet['Name'];
Tel0 := ResultSet['Tel'];
ResultSet.Next();
if (Name == ResultSet['Name']) {
Tel1 := ResultSet['Tel'];
} else {
UPDATE here.
StartAgain;
}
ResultSet.Next();
if (Name == ResultSet['Name']) {
Tel2 := ResultSet['Tel'];
} else {
UPDATE here.
StartAgain;
}
I am not recommending you do this, it is very bad use of a relational database but once implemented in a real language and debugged that should work.

Hive query, better option to self join

So I am working with a hive table that is set up as so:
id (Int), mapper (String), mapperId (Int)
Basically a single Id can have multiple mapperIds, one per mapper such as an example below:
ID (1) mapper(MAP1) mapperId(123)
ID (1) mapper(MAP2) mapperId(1234)
ID (1) mapper(MAP3) mapperId(12345)
ID (2) mapper(MAP2) mapperId(10)
ID (2) mapper(MAP3) mapperId(12)
I want to return the list of mapperIds associated to each unique ID. So for the above example I would want the below returned as a single row.
1, 123, 1234, 12345
2, null, 10, 12
The mapper Strings are known, so I was thinking of doing a self join for every mapper string I am interested in, but I was wondering if there was a more optimal solution?
If the assumption that the mapper column is distinct with respect to a given ID is correct, you could collect the mapper column and the mapperid column to a Map using brickhouse collect. You can clone the repo from that link and build the jar with Maven.
Query:
add jar /complete/path/to/jar/brickhouse-0.7.0-SNAPSHOT.jar;
create temporary function collect as 'brickhouse.udf.collect.CollectUDAF';
select id
,id_map['MAP1'] as mapper1
,id_map['MAP2'] as mapper2
,id_map['MAP3'] as mapper3
from (
select id
,collect(mapper, mapperid) as id_map
from some_table
group by id
) x
Output:
| id | mapper1 | mapper2 | mapper3 |
------------------------------------
1 123 1234 12345
2 10 12

Loops through Access Table and for each Column with Data

I think this should be simple, but I can't find the right way to do it. I have a table with an ID number column, and 10 rows following it labeled Question #1, Question #2, and so forth.
There are no duplicate ID numbers, but each ID number could have more than one row of questions.
I would like to take the ID row and for each different question where applicable create a new row with the same ID. So if an ID number has a question listed under the Question #1 and Question #2, Id like to create a duplicate for that ID number and have have both questions listed under one column Lets call it "Total Questions", and grouped by that ID number. This can be done by creating a new table.
Example:
From:
+-------+---------------------------+---------------------------+
| ID | Question #1 | Question #2 |
+-------+---------------------------+---------------------------+
| 11111 | Was it notated correctly? | Was it completed on time? |
+-------+---------------------------+---------------------------+
To:
+-------+-------------------------------------+
| ID | Total Questions |
+-------+-------------------------------------+
| 11111 | Was it notated correctly? |
| 11111 | Was it completed on time? |
+-------+-------------------------------------+
A simple solution using DAO
sub SomeProcedure()
Dim db as DAO.Database, recIn as DAO.Recordset, recOut as DAO.Recordset
Set db = currentdb()
Set recIn = db.openRecordset("yourQuestionsInputTable", dbOpenDynaset, dbReadOnly)
Set recOut = db.openRecordset("yourQuestionsOutputTable", dbOpenDynaset, dbEditAdd)
with recIn
.moveFirst
do
for i = 1 to .Fields.count
if left(.Fields(i).Name, 8) = "Question" then
recOut.addNew
recOut.Fields("Id") = .fields("Id")
recOut.Fields("Total Questions") = .Fields(i)
recOut.update
end if
next i
.moveNext
loop until .EOF
end with
recIn.close
recOut.close
db.close
end sub
The explanation:
What I'm doing is:
Read each record from the input table
For each column wich name begins with "Question", create a new record in the output table, with the Id of the input table, and the value of the selected column.
This is just a draft. You'll need to tweak the code to fit your needs.
Hope this helps.
Alternatives
After thinking a little, I may have an alternative to the problem you mention in your comments.
I think you can change the loop like this:
' You'll need a variable of type Field
Dim f as DAO.Field ' Check if this is right
' Some code
with recIn
.moveFirst
do
for f in .Fields
if left(f.Name, 8) = "Question" then
recOut.addNew
recOut.Fields("Id") = .Fields("Id").Value
recOut.Fields("Total Questions") = .Fields(f.Name).Value
recOut.update
end if
next f
.moveNext
loop until .EOF
end with
' More code
Instead of iterating on the Fields collection with an index, this will iterate with any Field member in it. That should avoid the "Item not found in collection" issue.
Warning: Not tested
Try a couple queries like this:
SELECT ID, Question1 AS TotalQuestions
INTO NewTable
FROM OriginalTable;
SELECT ID, Question2 AS TotalQuestions
INTO NewTable
FROM OriginalTable;

Resources