Apologies for the long winded question - I am experienced with the basics but this is first time working with datasets and databases (previously applications involved records, arrays and text files). My question is one of best practice and what is the best way to implement this, but to answer you will need to know what I am trying to achieve...
I am building an application that allows a telephone operator to take messages, write them into a form and save them to a DB. The main form contains a grid showing the messages already in the DB, and I have a 'message form' which is a VCL form containing edit, combo and checkboxes for all the respective fields the operator must log, some are mandatory, some are optional. There are lots of little helpers and automation's that run depending on the user input, but my question is related to the underlying data capture and storage.
There are two actions the user can perform:
Create new/blank message by clicking a button
Edit message by clicking on respective line in grid
Both of these actions cause an instance of the form to be created and initialized, and in the case of EDIT, the fields are then populated with the respective data from the database. In the case of NEW a blank dataset/record is loaded - read on...
The message form captures four distinct groups of information, so for each one I have defined a specific record structure in a separate unit (lets call them Group1, Group2, Group3 and Group4) each containing different numbers/types of elements. I have then defined a fifth record structure with four elements - each element being one of the previous four defined record structures - this is called TMessageDetails.
Unlike perhaps other applications, I am allowing the user to have up to 6 instances of the message form open at any one time - and each one can be in either NEW or EDIT mode. The only restriction is that two forms in EDIT mode cannot be editing the same message - the main form prevents this.
To manage these forms, I have another record (TFormDetails) with elements such as FormName (each form is given a unique name when created), an instance of TMessageDetails, FormTag and some other bits. I then have an array of TFormDetails with length 6. Each time a form is opened a spare 'slot' in this array is found, a new form created, the TMessageDetails record initialized (or data loaded into it from the DB) and a pointer to this record is given to the form. The form is then opened and it loads all the data from the TMessageDetails record into the respective controls. The pointer is there so that when the controls on the form make changes to the record elements, the original record is edited and I don't end up with a 'local' copy behind the form and out of sync with the original.
For the DB interaction I have four FDQuery components (one per group) on the message form, each pointing to a corresponding table (one per group) in an SQLite DB.
When loading a message I have a procedure that uses FDQuery1 to get a row of data from Table1, and then it copies the data to the Group1 record (field by field) in the respective TMessageDetails record (stored in the TFormDetails array) for that form. The same then happens with FDQuery2, 3, 4 etc...
Saving is basically the same but obviously in reverse.
The reason for the four FDQuery's is so that I can keep each dataset open after loading, which then gives me an open dataset to update and post to the DB when saving. The reason for copying to the records is mainly so that I can reference the respective fields elsewhere in the code with shorter names, and also when the VCL control tries to change a field in the dataset the changes don't 'stick' (the data I try and save back to the DB is the same as what I loaded), whereas in a record they do. The reason for breaking the records down into groups is there are places where the data in one of the groups may need to be copied to somewhere else, but not the whole message. It was also more natural to me to use records than datasets.
So my question is...
Is my use of; the record structures, a global TFormsDetails array, pointers, four FDQuery's per form (so up 6 forms open means up to 24 datasets open), and copying between records and datasets on save/load; a good way to implement what I am trying to achieve?
OR
Is there a way I can replace the records with datasets (making copying from FDQuery easier/shorter surely?) but still store them in an a global 'message form' array so I can keep track of them. Should I also try and reduce the instances of FDQuery and number of potential open datasets by having say one FDQuery component, and re-using it to load the tables into other global datasets etc?
My current implementation works just fine and there is no noticeable lag/hang when saving/loading, I just can't find much info on what is considered best practice for my needs (namely having multiple instances of the same form open - other example refer to ShowModal and only having one dataset to worry about) so I'm not sure if I'm leaving myself open to problems like memory leaks (I understand the 'dangers' of using pointers), performance issues or just general bad practice.
Currently using RAD 10.3 and the latest version of SQLite.
I don't know if what I'll say is the "best" practice but it is how I would do it.
The form would have his own TFDQuery using a global FDConnection. When several instance of the form are created, you have a FDQuery for each instance.
You add an 'Execute' method in the form. That method will create the FDQuery, get all data from the dataset with one or more queries, populate the form and show the form. Execute method receive argument (or a form's property) such as primary key to be able to get the data. If the argument is empty, then it is for a new record.
If the form has to update the grid on the fly, then an event will be used. The main form (containing the grid) will install an event handler and update the grid accordingly to the data given by the form.
When the form is done, it will use the primary key to store data back to the database.
Related
I'm new to Redshift and am looking at the best way to store event data. The data consists of an identifier, time and JSON metadata about the current state.
I'm considering three approaches:
Create a table for each event type with a column for each piece of data.
Create a single table for events and store metadata as a JSON field.
Create a single table with a column for every possible piece of data I might want to store.
The advantage of #1 is I can filter on all data fields and the solution is more flexible. The disadvantage is every time I want to add a new event I have to create a new table.
The advantage of #2 is I can put all types of events into a single table. The disadvantage is to filter on any of the data in the metadata I would need to use a JSON function on every row.
The advantage of #3 is I can easily access all the fields without running a function and don't have to create a new table for each type. The disadvantage is whoever is using the data needs to remember which columns to ignore.
Is one of these ways better than the others or am I missing something entirely?
This is a classic dilemma. After thinking for a while, in my company we ended up keeping the common properties of the events in separate columns and the unique properties in the JSON field. Examples of the common properties:
event type, timestamp (every event has it)
URL (this will be missing for backend events and mobile app events but is present for all frontend events and is worth to have in a separate column)
client properties: device, browser, OS (will be missing in backend but present in mobile app events and frontend events)
Examples of unique properties (no such properties in other events):
test name and variant in AB test event
product name or ID in purchase event
Borderline between common and unique property is your own judgement based on how many events share this property and how often will this property be used in the analytics queries to filter or group data. If some property is just "nice-to-have" and it is not involved in regular analysis use cases (yeah, we all love to store anything that is trackable just in case) the burden of maintaining a separate column is an overkill.
Also, if you have some unique property that you use extensively in the queries there is a hacky way to optimize. You can place this property at the beginning of your JSON column (yes, in Python JSON is not ordered but in Redshift it is a string, so the order of keys can be fixed if you want) and use LIKE with a wildcard only at the end of the field:
select *
from event_table
where event_type='Start experiment'
and event_json like '{"test_name":"my_awesome_test"%' -- instead of below
-- and json_extract_path_text(event_json,'test_name')='my_awesome_test'
LIKE used this way works much faster than JSON lookup (2-3x times faster) because it doesn't need to scan every row, decode JSON, find the key and check the value but it just checks if the string starts with a substring (much cheaper operation).
I'm working with an existing MS SQL database and ASP.NET web application. An update is needed in a table, but in order to add the new data and have it display correctly in the site, I need to be able to take a series of existing records, essentially "push them down", and then add the new data in the open space created.
Is there a cleaner, more efficient method than by just creating a new record that's a copy of the last related record, and then essentially doing a copy-and-paste for the remaining records until I reach the insertion point? There are quite a few records to move and I'd prefer something that isn't as mind-numbing and potentially error-prone as that.
I know the existing site and database isn't designed optimally for inserting new data into this table unless it's added to the end, but reconfiguring the database and stored procedures is not an option I presently have.
-- EDIT --
For additional requested information...
Screen shot of table definition:
Screen shot of some table data (filtered by TemplateID):
When looking at the table data, there are a couple other template ID values that bring back a bit more complex data. The issue is that this data needs to maintain this order, which happens to be the order in which it has been entered, since it gets returned and displayed in the shown order. The new data needs to be entered prior to one of these lettered subject headers. Honestly, I think this is not the best way to do this, but I had no hand in the design. It was created by a different company, and mine was hired to handle updates and maintenance after the creators became unpleasant to work with. A different template ID value brings back two levels of headers, which doesn't make my task any easier or alterations much cleaner considering the CS code that calls the stored procedures is completely separated from the code that builds the contents of the pages, and the organizational structure i tough to follow. There are some very poor naming conventions in places.
At any rate, there needs to be an insertion into this group of data under the "A" header value. The same needs to occur with another chunk associated with a different template ID, and there is another main header below the insertion point:
I need to develop an application where I need to handle more than 30 forms. Those forms have different numbers of fields with different data types. After storage, I need to do advanced search over the forms. A full-text-search may be needed for fields with a specific name shared among forms. Expected data size is ~50k forms with ~500k form fields. PostgreSQL is going to be used.
The solutions that I came up with:
1. Encoding form fields into a JSON String
Problems: Performing full-text-search for data with a specific field name can be cumbersome. Also when I need to read or update the data in the form, I need to perform decode and encode.
2. Creating a table having data fields as much as the number of the inputs in a form
Problems: As the form mapping classes are going to be ready, I can actually map form fields to database fields for each of those forms. For search, I may need to write different rules for each of those forms as the mapping for fields will change radically.
3. Keeping fields in a different table with a foreign-key to the form table
Problems: Maintaining the form data is still an issue but I don't know about speed. I expect it to run at least as fast as the previous way. Generating the search query from Java/Hibernate will be a bit harder than the previous way.
As I don't have any experience in handling such a case, I need your help and suggestions.
I have custom object 'Subject_c' with 3 fields and I have created those objects by uploading a CSV file. Subject_c has a lookup relationship with Leads (Its general for the same user regardless of what lead he is viewing). I am able to insert a related list and I can see that the objects are created under Data Management/Storage Usage. But it shows blank under related list.
You're saying that the custom object has lookup to Lead but then you say Subjects are generic and somehow should be displayed on every Lead page? I don't think it'll work.
Stuff appears on related list only when field Subject_c.Lead_c will be populated with "this" Lead's Id. (please note I've made best guess at the field name). So you'd need to insert separate data for each Lead which can quickly blow your storage usage and will be a pain in the a$$ to maintain later. Is it only for displaying? Or do you plan to later capture some kind of survey results for each Lead?
If it's just for display I think you'll need to embed a Visualforce page in the Lead page layout to achieve that in a saner way. The subjects are specific to current viewing user? Or it's more like a general list, just 3 subjects for whole organisation?
P.S. "object" is like a table in normal database. I think you mixed a bit the difference between table and records / rows of data stored in it.
I have two tables: stores and users. Every user is assigned to a store. I thought "What if I could just save all the users assigned to a store as a json object and save that json object in a field of a store." So in other words, user's data will be stored in a field instead of it's own table. There will be around 10 people to a store. I would like to know which method will require the least amount of processing for the server.
Most databases are relational, meaning there's no reason to be putting multiple different fields in one column. Besides being more work for you having to put them together and take them apart, you'd be basically ignoring the strength of the database.
If you were ever to try to access the data from another app, you'd have to make yourself go through additional steps. It also limits sorting and greatly adds to your querying difficulties (i.e. can't say where field = value because one field contains many values)
In your specific example, if the users at a store change, rather than being able to do a very efficient delete from the users table (or modify which store they're assigned to) you'd have to fetch the data and edit it, which would double your overhead.
Joins exist for a reason, and they are efficient. So, don't fear them!