I have a report i need to create that has a severe performance problem.
I need to create a catalogue of all courses in our database.
Here is the simplified data model:
Organizational Unit --> contains multiple Courses --> which contains Multiple Activities;
Each activity contains the following:
A list of attached links
A list of prerequisite activities
A list of additional property - value pairs (Cataloging information)
A list of required resource types and quantity for each resource type
A list of Training Objectives
and I wish to create a report that will group everything to look something like that:
After creating the straightforward query which joins all tables together i got almost 6 million rows because of different cartesian products that occur due to joining the activities table with all the 1-many relation table for getting the attached links resources etc...
I was thinking to avoid that in several ways:
sub reports that will list the different lists of items foreach activity.
create an XML field foreach of the described above lists and parse it using VB in my report
use multiple datasets in the report and somehow use lookup functions to list the different values.
Results so far:
sub-report proved to be very inefficient with regards to performance and it took 50% more that what it took with the original 6 million row query.
the xml fields are very efficent DB-wise but it will be trouble to format the data using VB i would very much like to avoid that if possible,
I cannot seem to find the right way to use lookupSet to get a list of attachment names and their links next to them.
so my questions are:
what is the best practice when displaying an entity with a lot of 1-many relations that need to be displayed when dealing with a lot of data and SSRS 2008-r2.
is there a way to join data using lookup function and somehow create "nested tables" that will list 1-many relations
any other suggestion would be very appreciated.
Can you create a drill down report? This looks like a good option to have activities listed and the courses being summarized and then drillable with the detail separated, or similar.
Or you can cache your report if the data change is unimportant?
6m records is a lot for a ssrs report!
Related
I'm trying to learn Datascript in the context of LogSeq, and I've stumbled into something I'm not sure how to solve.
The Fundamental Problem
I'm trying to query for a subset of entities that have NOT been referenced by attributes on a different group of filtered entities.
The Background
General LogSeq schema: https://github.com/logseq/logseq/blob/master/deps/db/src/logseq/db/schema.cljs
LogSeq Documentation for Datascript: https://docs.logseq.com/#/page/advanced%20queries
I've got a set of entities, with :block/properties like so:
tags:: contact
list:: C
Other entities have :block/refs to these pages.
I'm trying to create a query that shows me the contacts in a given list (A|B|C) that have NO notes within the past two weeks.
In SQL this would be a straightforward left join, but I'm having trouble translating that to Datalog since the information is in two different entity groups (instead of attributes all on the same entity). I assume there's some sort of not-join to filter out the contacts that have recent refs, but since that data is in other entities, I'm not sure how to structure the query since my implicit joins knock out either one group or the other.
I should add, because this is in LogSeq, I can't do two separate queries and join them in code. It has to be in one go.
Thank you!
I have been tasked with create a tracker for company work flow.
I have 10 tables with data in them. There are attributes all the tables have in common. I made a table with those attributes, giving those records a unique ID that could join them to the unique attribute records of the original tables. I am also linking a personnel table to the original tables. All of these tables exist on my SQL Server back end. I Have made a query in Access that displays all the information I was given. I'm going to use the forms in access as a front end to display, edit, and add records.
The issue I am encountering is that I can not write to a query that has externally linked tables. I have spent a lot of time normalizing this data and I know I can get around it by making tables with redundant attributes in SQL and not writing to the query, but rather write to the linked tables instead. Just wondering if there is a way around this.
Thanks
In general, even without linked tables, such queries are NOT updatable.
The general approach when working with multiple tables is to use sub forms for the child tables. That way, each form is only bound to one table. (You are free even to bind such forms directly to the linked table).
Thus, you might say have a customer table and then a table of invoices. So your main form will display the one customer and is bound to that one table.
In the sub form, you can then display all of the invoices.
So to combine multiple tables into a form or screen that allows users to update the data, or add more data, you don’t build some query that joins all the data together, but simply combine several forms into one form. But each of those separate forms will display data from the given related child table.
Here is a typical invoice form thus built in Access:
The top part is the “main” form of an invoice. It is bound to the customer table – one record. Then the multiple lines of detail is the invoice details table. So the form does NOT use queries, but each part of the form and sub forms are bound to the given related tables. You are binding each form directly to the linked table (or tables if you need to show related data like above).
This approach allows you to cobble together a set of forms that edit related tables, but each form is bound directly to the linked table.
So the fact of linked tables or not is moot – such queries are not used to edit data and such queries from link or non-linked tables are NOT updatable.
So your form + sub forms will follow the pattern of related tables that you need to work together as one whole view and means to edit data. You don’t need nor want to use a query to fill these forms.
You most certainly will provide some kind of “search” form, or some means to pull up say one customer invoice, and that invoice along with its sub forms will display the related data, and also allow editing of that data.
So I've done extensive searching on this and I can't seem to find a good solution that actually applies to my situation.
I have a list of projects in a table, then a list of people. I want to assign multiple people to one project. Seems pretty common. Obviously, I can't make multiple columns on my projects table for each person, as the people will change fairly frequently.
I need to display this information very quickly in a continuous list of projects (the ultimate way would be a multiple-select combobox as a listbox is too tall, but they don't exist outside of the dreaded lookup fields)
I can think of two ways:
- Store multiple employee IDs delimited by commas in one field in my projects table (I know this goes against good database design). Would require some code to store and retrieve the data.
- Have a separate table for employees assigned to projects (ID, ProjectID, EmployeeID). One to many relationship between projects table and this new table. One to many relationship between employees table and this new table. If a project has 3 employees assigned, it would store 3 records in this table. It seems a bit odd joining both tables in this way, and would also require code to get it to store and retrieve into a control like the one mentioned above).
Does anyone know if there is a better way (including displaying in an easy control) or how you usually tackle this problem?
The usual way to tackle this problem would be with a Junction Table. This is what you describe where you have a separate table maybe called EmployeeProject which has an EmployeeProjectID(PK), EmployeeID(FK) and ProjectID(FK).
In this way you model a Many-to-Many relationship where each project can have many employees involved and each employee can be involved in many projects. It's not actually all that difficult to do the SQL etc. required to pull the information back together again for display.
I would definitely stay away from storing comma-delimited values as this becomes significantly more complicated when you want to display or manipulate the data.
There's a good guide here: http://en.tekstenuitleg.net/articles/software/create-a-many-to-many-relationship-in-access but if you google "many to many junction table" or similar, there are thousands of pages/articles about implementation.
I am going to create a new project, where I need users to view their friends activities and actions just like Facebook and LinkedIn.
Each user is allowed to do 5 different types of activities, each activity have different attributes, for example activity X can be public/private for while activity Y will be assigned to categories. Some of actions include 1 users others have 2 or 3 ...etc. Eventually I have to aggregate all these 5 different types of activities on the news feed page.
How can I design a database that is efficient?
I have 3 designs in mind, please let me know your thoughts. Any new ideas will be greatly appreciated!
1- Separate tables: since there are nearly 3-4 different columns for each activity, it would be logical to separate each activity to its own table.
Pros: Clean database, and easy to develop.
Cons: It will need to query the database 5 times and aggregate results to make a single newsfeed page.
2- One big table: This table will hold all activities with many unused columns. A new numeric column will be added called "type" which will indicate the type of activity. Some attributes could be combined in an HStore field (since we are using Postgres), others will be queried a lot so I dont think it is a good thing to include them as in an HStore field.
Pros: Easy to pull newsfeed.
Cons: Lots of read/writes on the same table, the code will be a bit messier so is the database.
3- Hybrid: A solution would be to make one table containing all the newsfeed, with a polymorphic association to other tables that contain details of each specific activity.
Pros: Tidy code and database, easy to add new activities.
Cons: JOIN ALL THE TABLES to make a single newsfeed! Still better than making 5 different queries.
As I am writing this post I am starting to lean towards solution number 2. Please advise!
Thanks
I would consider a graph database for this. Neo4j. It will add very flexible attributes on either nodes (users) or links (types of relations).
For small sets and few joins, SQL databases are faster and more appropriate. But if your starting point is 5 table joins, graph databases seem simpler and offer similar performance (if not better).
Probably a noob question, but I'll go for it nevertheless.
For sake of example, I have a Person table, a Tag table and a ContactMethod table. A Person will have multiple Tag records and multiple ContactMethod records associated with them.
I'd like to have a forgiving search which will search among several fields from each table. So I can find a person by their email (via ContactMethod), their name (via Person) or a tag assigned to them.
As a complete noob to FTS, two approaches come to mind:
Build some complex query which addresses each field individually
Build some sort of lookup table which concatenates the fields I want to index and just do a full-text query on that derived table.
(Feel free to edit for clarity; I'm not in it for the rep points.)
If your sql server supports it you can create an indexed view and full text search that; you can use containstable(*,'"chris"') to read all the columns.
If it doesn't support it as the fields are all coming from different tables I think for scalability; if you can easily populate the fields into a single row per record in a separate table I would full text search that rather than the individual records. You will end up with a less complex FTS catalog and your queries will not need to do 4 full text searches at a time. Running lots of separate FTS queries over different tables at the same time is a ticket to query performance issues in my experience. The downside with doing this is you lose the ability to search for Surname on its own; if that is something you need you might need to look at an alternative.
In our app we found that the single table was quicker (we can't rely on customers having enterprise sql at hand); so we populate the data with spaces into an FTS table through an update sp then our main contact lookup runs a search over the list. We have two separate searches to handle finding things with precision (i.e. names or phone numbers) or just for free text. The other nice thing about the table is it is relatively easy and low cost to add further columns to the lookup (we have been asked for social security number for example; to do it we just added the column to the update SP and we were away with little or no impact.
One possibility is to make a view which has these columns: PersonID, ContentType, Content. ContentType would be something like "Email", "PhoneNumber", etc... and Content would hold that. You'd be searching on the Content column, and you'd be able to see what the person's ID is. I'm not 100% sure how full text search works though, so I'm not sure if you could use that on a view.
The FTS can search multiple fields out-of-the-box. The CONTAINS predicate accepts a list of columns to search. Also CONTAINSTABLE.