I'm using PostgreSQL to coordinate a large scale simulation which involves initializing various components of my application via arrays of integers. In particular, I have the notion of a "Controller", where each controller requires a variable number of parameters to be initialized.
I have a job table which stores the controller_id and a controller_parameters foreign key for actually linking to the set of parameters we want. My idea to start with was to do the following:
Use the controller_id to dynamically choose a table name from which to select the initialization parameters. Each of these tables would have a controller_parameters column that links the actual initialization data to the source table.
Once we know the table, run a SELECT * FROM #someController_parameters_table p WHERE p.controller_parameters = controller_parameters LIMIT 1;
Put these into a custom type which has an integer array field to be returned to the client.
The main problem is that this has Dynamic SQL, which I hear is not a good thing to do.
My proposed change is to have a new table, let's say controller_parameters which has the columns (controller_id, parameters_id, parameters[]). The third column stores the initialization parameters for an individual controller and parameter set.
Pros of this design are that we move back into static SQL land, which is good. Cons are that, when I generate the actual parameters to insert to the individual parameters table, I usually use a cross join to get all of the permutations of the individual parameters, and insert them accordingly to individual tables. I personally don't know how to take a cross-joined table row and convert it to an int[], so that's a potential roadblock.
Thoughts?
You can use array_agg to take the result of a group and turn it into an array.
select controller_id, parameters_id, array_agg(parameter)
from ...
group by controller_id, parameters_id
Here's postgresql docs on aggregate functions:
https://www.postgresql.org/docs/9.5/static/functions-aggregate.html
Related
I have a table with about 50,000 records (a global index of corporate and government bonds).
I would like the user to be able to filter this master index firstly into a smaller subset index (based on permanent logic), and then apply further run time criteria that vary each time.
For example, let's say the user wanted to start from one of many subset indices of bonds, let's say of government bonds only, rather than government and corporate bonds, and also only wanted the US$ government bond index specifically. This would be a permanently defined subset of the master index, with a where clause something like "[Level1]='Government' AND [Currency]='USD' AND [CountryCode]='US'"
At run time, the user would additionally request additional criteria, say for example "AND [IssueSize] > 1,000,000,000 AND [Yield] > 0.0112".
I initially thought of having a separate table that stored the different criteria for these permanent sub-indices as where clauses, for example it might have columns "IndexCode, IndexLogic", and using the example above the values would be "UST", "[Level1]='Government' and [Currency]='USD' AND [CountryCode]='US'", and there would be dozens of rows in this table defining commonly used bond indices.
I had originally thought of creating a dynamic string at run-time, where the user supplies their choice of sub-index code ('UST' in the example above), which then adds the relevant where conditions, and any additional criteria passed as separate parameters, and then doing an exec(#tsql) type command. I had also thought of perhaps having a where clause that was a function call, but this seems very inefficient?
Is the dynamic string method the best way of doing this, or is there a better way involving some kind of 'eval' function equivalent which can take a field value and use that as a where clause?
The problem here is you don't know in advance what the filtered index is.
A solution I have used in this instance, where the filtered index can often change is to grab the definition of the filter back into the client app, and use that to dynamically generate the SQL batch. You can also do this with dynamic SQL in a stored procedure:
SELECT ISNULL(
(SELECT i.filter_definition
FROM sys.indexes i
WHERE i.object_id = OBJECT_ID(#tablename) AND
i.name = #indexname AND has_filter = 1),
'(1=1)');
You pass the table name, and the index name, and you get back the exact definition for the index. This has the benefit of if the index is dropped, the condition becomes (1=1) i.e. every row. You can change this to (1=0) to return nothing instead.
Then you concat this into your dynamic query like so:
SELECT *
FROM table
WHERE regular_conditions_here
AND concated_filter_here
If you are joining other tables, I would advise you to subquery the filter, otherwise you many get column clash as there are no aliases.
SELECT *
FROM (SELECT * FROM table WHERE concated_filter_here) table
JOIN othertables etc
WHERE regular_conditions_here
In SQL we can write query like:
Select field1,field2,field3,field4,field5,field6,field7
from table1 t1,table2 t2,table3 t3
where t1.field1 = t3.field3 and
t2.field2 = 'USD'
In Qlikview, I have created QVD's for 6 tables, now I want to create a single QVD of these 6 QVD's. Unfortunately these tables don't contain primary keys. So I cant use join. I have tried like this also:
fact:
load *
from
[D:\path\fact*.qvd](qvd);
//To store all qvd's into one qvd.
store fact into [D:\path\facttable.qvd];
This query creates a facttable but only with 2 columns, these columns are of first fact table. Diagram shows it much clear:
As it internally gives the name of all the facts table with fact, fact-1, fact-2 and so on and I have written the query store fact into [D:\path\facttable.qvd]; and in this diagram fact table contains only two columns so it creates the fact table with two columns only.
Please let me know the solution that how can we write this query in Qlikview or how can we create a fact table using all the QVDS?
Thanks in advance.
Since every qvd contains different field names it will create several tables with synthetic keys when you load *.
You can use Concatenate Load to stack each qvd onto one fact table. One simple example would be to first create a Fact table by:
Fact:
Load * INLINE [
dummyField
];
Now you can concatenate the qvd's onto that Fact table:
concatenate(Fact)
load *
from
[D:\path\fact*.qvd](qvd);
//To store all qvd's into one qvd.
store Fact into [D:\path\facttable.qvd];
//drop the dummy field.
drop field dummyField;
The question is how database design should I apply for this situation:
main table:
ID | name | number_of_parameters | parameters
parameters table:
parameter | parameter | parameter
Number of elements in parameters table does not change. number_of_parameters cell defines how many parameters tables should be stored in next cell.
I have problems to move from object thinking to database design. So when we talk about object one row has as much parameters as number_of_parameters says.
I hope that description of requirements is clear. What is the correct way to design such database. If someone can provide some SQL statments to obtain it it would be nice. But the main goal of this question is to understand how to make such architecture.
I want to use SQLite to create this database.
The relational way is to have two tables. The main table has an ID, name and as many other universally-present parameters as possible. The parameters table contains a mapping from an ID in the main table to a parameter name and a parameter value; the main table ID should be a foreign key, and the combination of ID and name should be unique.
The number of parameters can be found by just counting the number of rows with a particular ID.
If you can serialize the data whiile saving to the database and deserialize it back when you get the record it will work. You can get total number of objects in serialized container and save the count to the number_of_parameters field and serialized data in parameters field.
There isn't one perfect correct way, but if you want to use a relational database, you preferably have relational tables.
If you have a key-value database, you place your serialized data as a document attached to your key.
If you want a hybrid solution, both human editable and single table, you can serialize your data to a human-readable format such as yaml, which sees heavy usage in configuration sections of open source projects.
I'm new to complex database design. I'm currently into a project where the user should be able to retrieve Instructions based on a combination of 18 columns. So my parameter Table has the following columns
Job
State
Manager
ProcessCode
ProcessType
(rest of the columns truncated).
InstructionID (FK of Instruction Table)
When adding / Modifying the instruction, he can choose multiple options in each of the above parameters. The Stored Procedure will store data in all combinations possible, in order facilitate easy retrieval, as during search (retrieval) only one option will be chosen in each of the columns.
There can be multiple instructions for same combination and the same instruction can apply to multiple combinations.
I have somehow created the SP for adding instruction, but am now struck with modification. When my Webpage passes the new combination to SP, what is the best way to update the Table?
I could delete all existing rows and create new rows for new combination, but I wanted to maintain the created date and created user columns. Further, there is a requirement to maintain history of these in a separate history table.
Sorry for the length of the question... And, Thank you for help.
If you're trying to retrieve data based on a combination of parameters then you can set the parameters to have the default value of NULL e.g.
CREATE PROC spProcName
#FieldName INT = NULL
The only other thing to do is set the WHERE section of the statement to look at the parameter values and compare them to see if they or null or not e.g.
WHERE ((FieldName = #FieldName) OR (#FieldName IS NULL))
Use this for querying the tables and use standard update queries in a similar fashion using the default parameter value of null but setting the value like this:
FieldName = ISNULL(#FieldName, FieldName)
Which lets you update only given parameters.
Hope this is something you are after, I can give a full example if needed.
What you have is many-to-many relationship, so I would suggest you use:
I have a table that holds information about a particular Object, Say Item and has columns
ItemID, ItemName, price, ItemListingType.....LastOrderDate
One of the bits of information, ItemListingType could be one of 10 different types
such as:
private, gov, non-gov, business... etc (strings) and could be extended to more types in future.
Should I be using a column inside table ITEM or should I Use a separate table with two columns and put a foreign key in Item table to reference that (a one to many relationship)? Like:
ListingTypeID int
ListingTypeName varchar(MAX)
EDIT: how many values for a column, you will consider to use another table for that
2, 4 or what ?
Thanks
Use a separate table to store this kind of reference data. This is a tenet of normalization and will also enable easier caching because you are separating read-only and read-write data. my two cents...
Separate table.
What if you have a listing type not yet used?
Or delete the last item with type x?
Or need to change a value?
These are insert, update and delete anomalies, which is one reason for normalisation
I would definitely go for a "lookup" style column; that way you are not stumped when there future additions to the list of permissible listing types. You are also reducing redundancy and making it easier to change the designation of aparticular type of listing (if "gov" changes to "government agencies", then you only have to change it in one place).
You should do it with the second table that holds the ListingTypes and link to the id of that table from the one with the Objects...
Take a look at Relational Database and Relational model.
In situations like this I ask myself:
Can the Item have an undetermined number of Listing types? If yes, different table.
Do the specs say that there will never be more than 3 types? Depends. Sometimes I'll still go with a separate table, sometimes not. You get a feel for this after a while.
Will the Item ALWAYS have a single listing type? If yes, same table, single column.
Now to take matters one step further.
If an Item has zero or more listing types AND those listing types are actually shared (in other words two items could have the same listing type, then we have 3 tables: Items, ListingTypes, and a cross reference table to support a many to many relationship.
Clasically, you should use an extra table, because you won't have any duplication that way. It will also allow you to change the value for this listing in a single place. However, if you are very very sure that no types will be added, keep the column.