I have a table containing user input which needs to be optimized.
I have some ideas about how to solve this but i would really appreciate your input on this. The table that needs optimization is called Value in the structure below.
All tables mentioned below has integer primary keys called Id.
Specs: Ms Sql Server 2008, Linq2Sql, asp.net website, C#.
The current structure looks as follows:
Page -> Field -> FieldControl -> ValueGroup -> Value
Page
A pages is a container for one or more Fields.
Field
A field is a container for one or more FieldControls such as a textbox or dropdown-options.
Relationships: PageId
FieldControl
If a Field is of the type 'TextBox' then a single FieldControl is created for the Field.
If a Field is of the type 'DropDown' then one FieldControl per dropdown option is created for the Field containing the option text.
Relationships: FieldId
ValueGroup
Each time a user fills in Fields within a Page and saves it, a new ValueGroup (Id) is created to keep track of user input that is relevant to that save. When a user wants to
look at a previously filled in form, the valuegroup is used to load the Values into the FieldControls of that previously filled in instance.
Relationships: None
Value
The actual input of a FieldControl. If the user typed 'Hello' in a TextBox then 'Hello' would be stored in a row in this table followed by a reference back to which FieldControl 'Hello' was inputted for. A ValueGroup is linked to values in order to group them to keep track of which save/instance they belong to as described in ValueGroup.
Relationships: ValueGroupId, FieldControlId
The problem
If 100.000 Pages are fully filled in, containing 10 TextBoxes each then we get 100.000 * 10 records in the Values table meaning we quickly reach one million records making it really slow as it is now. The user can create as many different pages with as many different Fields as he/she likes and all these values are stored in the Values table. The way i use this data is by either displaying a gridview with pagination that displays all records for a single Pagetype, or when looking at a specific Page instance (Values grouped by ValueGroupId).
Some ideas that i have:
Good indexing should be very important when optimizing the Values table.
Should i perhaps add a foreign key directly back to Page from Value, ending up with indexing by (Id, PageId, ValueGroup) allowing the gridview to retrieve values that are only relevant for one Page?
Should i look into partitioning the table and if so, how would you recommend that i do this? I was thinking that partitioning by Page, hence getting chunks of values that are only relevant to a certain page would be wise in this case right? How would the script/schema look for something like that where pages could be created/removed at any time by the users.
PS. There should be a badge on this forum for all people that finished reading this long post, and i hope ive made myself clear :)
Just to close this post. Correct indexing solved all performance problems.
This may be slightly off-topic, but why? Is this data that you need to access in real-time, or is it for some later processing? Could you perhaps pack the data into a single row and then unpack it later?
Generic
You say it is slow now and that can be many reasons for that other than the database
like low memory, high CPU, disk fragmentation, network load, sockets problems etc etc.
This should show up on a system monitor
Try for instance Sysinternals (now MS) tool: http://live.sysinternals.com/procexp.exe
But if that is all under control then back to the database.
Database index
One million records is not "that much" and should not be a problem.
An index should do the trick if you don't have any indexes right now.
You should probably set indexes on all tables if you haven't done so already.
I tried to do a database model, is this right:
http://www.freeimagehosting.net/image.php?a39cf99ae5.png
Table structure (?)
Page -> Field -> FieldControl -> ValueGroup -> Value
The table structure looks like it may not be the optimal one but it is hard to say exactly when I don't know how the application works.
Do all tables have the foreign keys of
the table above ?
Is this somewhat similar to your code ?
Pseudo code:
1. Get page info. Gives key "page-id"
2. Get all "Field":s marked with that "page-id".
Gives keys "field-id" & "fieldcontrol-id"
3. Loop trough all fields-id:s and get the FieldControl for each one
4. Loop trough all fields-id:s and get all ValueGroup:s.
Gives a list of "valuegroup-id":s keys
5. Loop trough all ValueGroup:s and get all fields
Related
I need to use SSRS to create many different reports, and I have been trying to find the best way for me to easily create them as need, and for users to navigate them and use them for their needs.
To give you and idea of the two sets of data I am dealing with:
EDI file from our customer
Raw data output from hardware configuration
Now the EDI data is fairly consistent, so these columns are static.
The hardware data is usually a massive list of different configuration. I receive them in different flat files formats and using SSIS or other tools I get the data into Key Value Pairs. Now in a report, I use matrix to keep EDI columns static, it matches with the hardware on serial number, and Hardware data pivots.
So the report does not break, and so I don't give the user too much information, it matches up on another table where I specify what keys I want to be columns.
Here is a small example of one of my reports:
The green columns are EDI, while the orange is the hardware.
My question is, is there a better way for me to be doing this? Some reports can get complicated like needing total for certain hardware (counting hardrive space, ram total etc.) which is difficult to do dynamically.
I have tried creating in reports in this fashion, with these parameters:
This way I can create the Key columns per project and user can select what report they want to run. The default is All Data.
Is there a better way for me to create these reports? SSRS really doesn't seem to play well with dynamic pivots.
Is there a better tool that will handle these reports dynamically, or let users pick and choose what they want to see in a report?
I can't visualise your data but if I understand correctly, you could have a dropdown list showing all the unique values that are in the column you are using in the column group. Set this to be multi-value and then simply have the WHERE clause read something like
SELECT * FROM myTable WHERE myColumnGroupField IN (#myColumnChoiceParameter)
This way the user could select whichever columns they would like.
You could extend this by adding another parameter that has some preset groups of columns (I think you might have one of these already if I understand correctly) that would set the default value of the main #myColumnChoiceParameter parameter.
If you want something more flexible then you might want to look at Power BI but depending on how you intend to deploy that might not be a simple option.
You cannot dynamically create columns in SSRS but you can control the visibility of the columns.
1) Create a list in table that contains the names of all the columns that yo want to toggle and include a column titled 'All'.
2) Create a parameter that is based on this table and make sure multi-select is turned on.
3) Right click on every column that you want to toggle, select visibility and then create a condition that checks if the user either selected All or selected the column from the parameter list.
4) Train users that by selecting and deselecting from the dropdown they control whats visible.
BACK END - SQL Server
FRONT END - Access 2010 (2000 format)
The system stores and retrieves data about technical documents. Broadly, there are three tables A, B and C, each of which maintains data about a different type of document.
The ACCESS front end provides a Search Form and Data Entry/Edit form (bound to the underlying table) for each document type. In all three document types, when adding a new record, the user will open the Search form and press a button called "Add". This opens the Data Entry/Edit form and in the Form_Load event is the line
DoCmd.GoToRecord , , acNewRec
When the data entry is complete, the user presses a "Close" button which simply runs the code
DoCmd.Close
As I said, the design and code of the objects relating to the three document types is, for all intents and purposes, identical. However, while for tables A and B the process of adding a new record is seamless and extremely quick, for table C it has proved impossible to add a new record via the ACCESS UI. The edit form will open correctly to add the data, but when the user presses the "Close" button the form hangs, and eventually returns to the Search form without the new record having been added.
It is possible to bypass the UI by opening ACCESS while holding down the SHIFT key, opening the linked table, and adding new records directly. While this is acceptable as an interim measure, it is unacceptable in the long term. It should be noted that the system is about ten years old, and has been working entirely correctly for about nine of those years (apart from minor glitches moving between different versions of ACCESS).
Unfortunately this system is owned and operated by a major global corporation and it is very difficult for me, a subcontracted supplier, to get access to the SQL Server box to run diagnosis (SQL Profiler would be a good starting point). My gut feeling is that there is a subtle difference in the permissions model for that particular table but I don't know.
The situation is further complicated by the fact that I have a copy of the system at my work and I cannot reproduce the problem. Of course, there are bound to be subtle differences between the two architectures (for example, I don't know for certain what version of SQL Server it's running on, but I believe it's 2000, nor do I know how completely it is patched or updated) but the facts are that for one particular table bound to one particular form, it is not possible to add records, whereas for other tables there is no such problem.
I would be grateful if anyone has any ideas about how to go about diagnosing this or even solving it (if anyone has come across the same problem before).
Many thanks
Edward
As a general rule when you encounter problems to update a table, then this tends to suggests that the table does not have a PK or the form the query is based on does not have a PK exposed.
The next thing I would ensure is the table has a time stamp column as Access uses this to test for record changes behind the scenes.
Next up I would check the default locking for the form (while these settings generally don't effect odbc, they should be checked).
Next up is to check if the table has any "bit" column (true/false) and ensure that the defaults for such controls are set SQL SERVER side (they should default to 0). This null bits issue will cause updates to fail if not addressed.
I would also check if the form in question is based on a query or if the data source is set directly to the table. As noted the PK auto number ID of that table in query should be INTEGER value sql side – big int is NOT supported.
So check default values (both in sql table and on the form (those controls) to ensure nothing be set that would prevent the update.
I'm currently struggling with Oracle Apex.
I'm trying to create an application which enables customers to place their order. Therefore I create a report which lists the available products. Furthermore the report contains a column (the SQL query for that is simply '0' as "Quantity") which displays a text box. In this text boy the customer should be able to insert the required quantity.
I've create a screenshot to make it easier to follow me:
After the customer has fill out the form, the "Place Order" button will purchase the wished items then.
My question is now, how is it possible to read out in which text boxes did the user filled in a number and also to which product belongs it!
An easier solution would be to recreate the region but choose Form Region and then Tabular Form Region and then the wizard will help take care of the DML for you. But you need to use specific table columns for this to work.
To answer your question more directly - the input items defined in reports that are posted to the server can be accessed in PL/SQL as a set of "Global Arrays". These are defined as PL/SQL tables in the package apex_application with the names g_f01 through g_f50.
To be sure which of these arrays to use for the quantity text box you can look at the html of the page for the name attribute of the input tag. If it is f01 then you would be able to process the results by accessing each position or element in apex_application.g_f01.
To link the input with the table you would need some sort of key. If you use the wizard to build a Tabular Form all this headache is taken care of for you though.
I have a SQL database where one of my fields 'Status' has one of 10 entries in it (not from a dropdown list). I let some people have access to the data through an Access database, they can add and modify entries.
My question is, in Access, how can I make it so that the 'Status' field is a drop-down list from which the users can choose from (the 10 entries which are already in the SQL database)? It would be easier for them and also mean that mistakes cannot be made.
Many thanks
Scott
The usual way to do this is to use a combo box on a form with the row source taken from the look-up table and the bound column set to the field (column) of the table to be updated.
In Access you can add lookup information to a column. That will automatically display a dropdown list.
Step 1: Start the lookup wizard:
Step 2: After the wizard, the lookup settings should look like this:
Step 3: When your users open a table, they should see the dropdown box:
In addition to the solution described by Andomar you must not use another table as the source for your lookup. You can also provide the lookup-values in a list, which is hardcoded in the table-definition. This is fine for simple scenarios where the lookup is something that is not likely to be changed.
Several issues here:
table datasheets are not suitable user interface for users.
you can create a saved QueryDef and if you view the properties of a field, the second tab is just like the second tab in table design, and allows you to use a combo box as your display type in your query. I would generally recommend against this, as, like table datasheets, a QueryDef is not a proper UI element, and if you use the saved query in other queries, you can run into the same problems that cause lookups in table fields to be such as bad idea.
you're building a UI, so use the tools that Access provides you for building a UI. That means a form. Your form's recordsource would have the bare data, and you'd create a combo box on your form that is bound to the field in the table behind your form, and displays the values from the lookup tables. There's a wizard to step you through this. If you like the look of datasheets so much (column sizing, sorting, show/hiding are all features that are nice in datasheets), you can set your form to display as a datasheet.
My advice is that for building a user interface, use the tools Access provides for creating user interface. In my opinion, a dropdown list belongs in a form, and nowhere else. While I occasionally might add one to a query for quick-and-dirty editing, I would never do that in objects that users are going to use.
I have a huge database (800MB) which consists of a field called 'Date Last Modified' at the moment this field is entered as a text data type but need to change it to a Date/Time field to carry out some queries.
I have another exact same database but with only 35MB of data inside it and when I change the data type it works fine, but when I try to change data type on big database it gives me an error:
Micorosoft Office Access can't change the data type.
There isn't enough disk space or memory
After doing some research some sites mentioned of changing the registry file (MaxLocksPerFile) tried that as well, but no luck :-(
Can anyone help please?
As John W. Vinson says here, the problem you're running into is that Access wants to hold a copy of the table while it makes the changes, and that causes it to exceed the maximum allowable size of an Access file. Compacting and repairing might help get the file under the size limit, but it didn't work for me.
If, like me, you have a lot of complex relationships and reports on the old table that you don't want to have to redo, try this variation on #user292452's solution instead:
Copy the table (i.e. 'YourTable') then paste Structure Only back
into your database with a different name (i.e. 'YourTable_new').
Copy YourTable again, and paste-append the data to YourTable_new.
(To paste-append, first paste, and select Append Data to Existing
Table.)
You may want to make a copy of your Access database at this point,
just in case something goes wrong with the next part.
Delete all data in YourTable using a delete query---select all
fields, using the asterisk, and then run with default settings.
Now you can change the fields in YourTable as needed and save
again.
Paste-append the data from YourTable_new to YourTable, and check
that there were no errors from type conversion, length, etc.
Delete YourTable_new.
One relatively tedious (but straightforward) solution would be to break the big database up into smaller databases, do the conversion on the smaller databases, and then recombine them.
This has an added benefit that if, by some chance, the text is an invalid date in one chunk, it will be easier to find (because of the smaller chunk sizes).
Assuming you have some kind of integer key on the table that ranges from 1 to (say) 10000000, you can just do queries like
SELECT *
INTO newTable1
FROM yourtable
WHERE yourkey >= 0 AND yourkey < 1000000
SELECT *
INTO newTable2
FROM yourtable
WHERE yourkey >= 1000000 AND yourkey < 2000000
etc.
Make sure to enter and run these queries seperately, since it seems that Access will give you a syntax error if you try to run more than one at a time.
If your keys are something else, you can do the same kind of thing, but you'll have to be a bit more tricky about your WHERE clauses.
Of course, a final thing to consider, if you can swing it, is to migrate to a different database that has a little more power. I'm guessing you have reasons that this isn't easy, but with the amount of data you're talking about, you'll probably be running into other problems as well as you continue to use Access.
EDIT
Since you are still having some troubles, here is some more detail in the hopes that you'll see something that I didn't describe well enough before:
Here, you can see that I've created a table "OutputIDrive" similar to what you're describing. I have an ID tag, though I only have three entries.
Here, I've created a query, gone into SQL mode, and entered the appropriate SQL statement. In my case, because my query only grabs value >= 0 and < 2, we'll just get one row...the one with ID = 1.
When I click the run button, I get a popup that tells/warns me what's going to happen...it's going to put a row into a new table. That's good...that's what we're looking for. I click "OK".
Now our new table has been created, and when I click on it, we can see that our one line of data with ID = 1 has been copied over to this new table.
Now you should be able to just modify the table name and the number values in your SQL query, and run it again.
Hopefully this will help you with whatever tripped you up.
EDIT 2:
Aha! This is the trick. You have to enter and run the SQL statements one at a time in Access. If you try to put multiple statements in and run them, you'll get that error. So run the first one, then erase it and run the second one, etc. and you should be fine. I think that will do it! I've edited the above to make it clearer.
Adapted from Karl Donaubauer's answer on an MSDN post:
Switch to immediate window (Ctl + G)
Execute the following statement:
DBEngine.SetOption dbMaxLocksPerFile, 200000
Microsoft has a KnowledgeBase article that addresses this problem directly and describes the cause:
The page locks required for the transaction exceed the MaxLocksPerFile value, which defaults to 9500 locks. The MaxLocksPerFile setting is stored in the Windows registry.
The KnowledgeBase article says it applies to Access 2002 and 2003, but it worked for me when changing a field in an .mdb from Access 2013.
It's entirely possible that in a database of that size, you've got text data that won't convert to a valid Date/Time.
I would suggest (and you may hate me for this) that you export all those prospective date values from "Big" and go through them (perhaps in Excel) to see which ones are not formatted the way you'd expect.
Assuming that the error message is accurate, you're running up against a disk or memory limitation. Assuming that you have more than a couple of gigabytes free on your disk drive, my best guess is that rebuilding the table would put the database (including work space) over the 2 gigabyte per file limit in Access.
If that's the case you'll need to:
Unload the data into some convenient format and load it back in to an empty database with an already existing table definition.
Move a subset of the data into a smaller table, change the data type in the smaller table, compact and repair the database, and repeat until all the data is converted.
If the error message is NOT correct (which is possible), the most likely cause is a bad or out-of-range date in your text-date column.
Copy the table (i.e. 'YourTable') then paste just its structure back into your database with a different name (i.e. 'YourTable_new').
Change the fields in the new table to what you want and save it.
Create an append query and copy all the data from your old table into the new one.
Hopefully Access will automatically convert the old text field directly to the correct value for the new Date/Time field. If not, you might have to clear out the old table and re-append all the data and use a string to date function to convert that one field when you do the append.
Also, if there is an autonumber field in the old table this might not work because there is no way to ensure that the old autonumber values will line up with the new autonumber values that get assigned.
You've been offered a bunch of different ways to get around the disk space error message.
Have you tried adding a new field to your existing table using Date data type and then updating the field with the value the existing string date field? If that works, you can then delete the old field and rename the new one to the old name. That would probably take up less temp space than doing a direct conversion from string to date on a single field.
If it still doesn't work, you may be able to do it with a sceond table with two columns, the first long integer (make it the primary key), the second, date. Then append the PK and string date field to this empty table. Then add a new date field to the existing table, and using a join, update the new field with the values from the two-column table.
This may run into the same problem. It depends on number of things internal to the Jet/ACE database engine over which we have no real control.