Can I use variable table name for db mapped objects? For example, there are n objects of the same structure and I want to store it in different tables, for raising performance on some operations.
Let's say I've got class defined as:
class Measurement(models.Model):
slave_id = models.IntegerField()
tag = models.CharField(max_length=40)
value = models.CharField(max_length=16)
timestamp = models.DateTimeField()
class Meta:
db_table = 'measurements'
Now all objects are stored into table 'measurements'. I would like to make table name dependant on 'slave_id' value. For example, to handle data from tables 'measurements_00001', 'measurements_00002' etc...
Is it possible to achieve this using Django ORM model or the only solution is to drop to SQL level?
In the vast majority of cases, this shouldn't buy you any performance advantage. Any RDBMS worth its salt should handle immense tables effortlessly.
If it's needed, there could be some sharding of the table. Again, managed by the DB server; at SQL level (and ORM) it should be seen as a single table. Ideally, the discrimination should be automatically handled; if not, most RDBMS let you specify it at table definition time (or sometimes tune with ALTER TABLE)
If you choose to define the sharding method, each RDBMS has it's own non-standard methods. Best not to tie your Python code to that; do the tuning once on the DB server instead.
Related
I have an application that requires a database containing a set of products where each product can have a set of tables. The end-user should be able to add new products and define new tables for a product. So each table has a set of columns that are specified by the user. The user can then fill the tables with rows of data. Each table belongs to exactly one product.
The end-user should also be able to view the tables as they were at a specific point in time (at a certain transaction).
How would I go about making a schema for this in Datomic so that querying it would be as efficient as possible?
I would go with 4 entity types: products, tables, columns, and rows.
The relationship between products and tables is best handled by a :table/product to-one ref attribute, but a :product/tables to-many component ref attribute could also work (the latter does not enforce the one-to-many relationship).
Likewise, I would use either a :column/table or :table/columns attribute. I would also have a :column/name string attribute and maybe a :column/type enumerated attribute.
The hardest part is to model rows.
One tempting solution is to just create an attribute per column - I actually think it's bad idea, Datomic attributes are not intended for such a dynamic use. In particular, schema attributes are stored in a cache on the Peer that's not meant to grow big. (I may be wrong about this, so it'd be nice if someone in the Datomic team could confirm.)
Instead, I would have a few dozens reusable :row/cell-0, :row/cell-1, :row/cell-2, etc. 'cell position' attributes, that are shared across all tables. Each actual column would be mapped to a at creation time by a to-one :column/position attribute.
If the rows can have several data types, it's a bit more difficult, you'd have to basically make an attribute for each (type,position) pair.
Then each row basically consist of a :row/table attribute and the above cell position attributes.
Here's a Datalog query that would let you read the whole table
[:find ?row ?column-name ?val :in $ ?table :where
[?column :column/table ?table]
[?row :row/table ?table]
[?row ?pos ?val]
[?column :column/position ?pos]
[?column :column/name ?column-name]]
Note that all of the above is only useful if you want to query the table with Datalog directly against your Datomic db. But it can be also completely fine to serialize your tables and store them as blobs - especially if they're small; later, you pull out the blob, deserialize it, then you can query with Datalog too. And if tables are to coarse for this use, maybe you can do it with rows.
So, I have a relatively common task, and hope to get some suggestions here.
Idea is that I have a small database in mind, database will have at least 2 types of tables:
dictionary-table - it will have just the id and few columns of text
aggregation-table - it should combine different dictionary entries into some aggregation, so it will be basically mapping id's of different dictionary entries all together.
So, what I hoped to do is to have some software that will help me to fill database easily. I will add data to dictionary-tables, and will say that 'this particular column of my aggregation table can have values only from this dictionary-table', so I would type words, and it will just add id's from dictionary-table instead. You know, like a relationships in database.
Except that in the end I want it to be a plain sqlite database, and sqlite doesn't support relationships.
So what I want is some cool high-level GUI tool that will simplify the way I input data to database, and will help me to maintain the data when DB grows in future, but also be able to export to a simple SQLite.
I tried: SQliteBrowser, SqliteAdmin, Libreoffice Base + Sqlite ODBC. Neither supports what I want.
Anything else worth checking out?
How about PhpLiteAdmin? - https://code.google.com/p/phpliteadmin/
It allows you to directly add/modify the structure and data of an sqlite database but also allows import and export of tables, structure, indexes, and data (SQL, CSV). If you're dealing with thousands of entries then this may be the important feature for whatever tool you use.
There's no installation and it's open-source
you said
...be a plain sqlite database, and sqlite doesn't support relationships.
But sqlite supports relationships. By default it is disabled. you can enable it with
sqlite> PRAGMA foreign_keys = ON;
now you can code your requirement by having proper foreign key.
you said
I will add data to dictionary-tables,
Instead of multiple dictionary tables, just have one dictionary table and add one more column in it as dictionary_name
your aggregate table can simply have foreign key referring the dictionary table
I'm creating an Account table in my project's database. Each account has A LOT of properties:
login
email
password
birthday
country
avatarUrl
city
etc.
Most of them are nullable. My question is, how should I design this in database?
Should it be one table with all those properties? Or maybe should I create two tables, like AccountSet, and AccountInfoSet, where I would store all those 'advanced' user's settings? And last, but not least: if this should be two tables, what kind of relation should be between those tables?
If this is a relational database, then I definitely would not store those properties as fields in the Account table. Some reasons why:
Once your application goes to production (or maybe it's already there), the schema maintenance will become a nightmare. You will absolutely add more properties and having to constantly touch that table in production will be painful.
You will most likely end up with orphaned fields. I've seen this many times where you'll introduce a property and then stop using it, but it's baked into your schema and you might be too scared to remove it.
Ideally you want to avoid having such sparse data in a table (lots of fields with lots of nulls).
My suggestion would be to do what you're already thinking about and that's to introduce a property table for Accounts. You called it AccountInfoSet.
The table should look like this:
AccountId int,
Property nvarchar(50),
Value nvarchar(50)
(Of course you'll set the data types and sizes as you see fit.)
Then you'll join to the AccountInfoSet table and maybe pivot on the "advanced" properties - turn the rows into columns with a query.
In .NET you can also write a stored procedure that returns two queries with one call and look at the tables in the DataSet object.
Or you could just make two separate calls. One for Account and one for the properties.
Lots of ways to get the information out, but make sure you don't just add fields to Account if you're using a relational database.
I'm trying to figure out how to implement this relationship in coldfusion. Also if anyone knows the name for this kind of relationship I'd be curious to know it.
I'm trying to create the brown table.
Recreating the table from the values is not the problem, the problem that I've been stuck with for a couple of days now is how to create an editing environment.
I'm thinking that I should have a table with all the Tenants and TenantValues (TenantValues that match TenantID I'm editing) and have the empty values as well (the green table)
any other suggestions?
The name of this relationship is called an Entity Attribute Value model (EAV). In your case Tenant, TenantVariable, TenantValues are the entity, attribute and value tables, respectively. EAV is attempt to allow for the runtime definition or entities and is most found in my experience backing content managements systems. It has been referred to an as anti pattern database model because you lose certain RDBMS advantages, while gaining disadvantages such as having to lock several tables on delete or save. Often a suitable persistence alternative is a NoSQL solution such as Couch.
As for edits, the paradigm I typically see is deleting all the value records for a given ID and inserting inside a loop, and then updating the entity table record. Do this inside of a transaction to ensure consistency. The upshot of this approach is that it's must easier to figure out than delta detection algorithm. Another option is using the MERGE statement if your database supports it.
You may want to consider an RDF Triple Store for this problem. It's an alternative to Relational DBs that's particularly good for sparse categorical data. The data is represented as triples - directed graph edges consisting of a subject, an object, and the predicate that describes the property connecting them:
(subject) (predicate) (object)
Some example triples from your data set would look something like:
<Apple> rdf:type <Red_Fruit>
<Apple> hasWeight "1"^^xsd:integer
RDF triple stores provide the SPARQL query language to retrieve data from your store much like you would use SQL.
Is there a paradigm in which I can change a data-key name in one place and one place only, and have it properly be dealt with by both the application and database?
I have resorted most recently to using class constants to map to database field names, but
I still have to keep those aligned with the raw database keys.
What I mean is, using PHP as an example, right now I might use
$infoToUpdateUser[ User::FIELD_FIRST_NAME ]
This means that when I change it at the constant, I don't have to search through the code to change all references to that field.
Another area this crops up in is in referencing fields. Due to some early poor design decisions, I have, for example, these sorts of tables:
( table name : primary_key )
cats : cat_id
dogs : dog_id
parrots : bird_id (remember, poor design, thus the mismatch between parrots / bird_id)
lizards: lizard_id
etc
Then let's say I have a series of form classes that update records.
AnimalForm
DogForm extends AnimalForm
CatForm extends AnimalForm
ParrotForm extends AnimalForm
etc
Now I want to update a record in the SQL database using an update function in the parent class, AnimalForm, so I don't have to replicate code in 20 subclasses.
However I do not know of a way to generalize the update query, so currently each subclass has an idFieldName member variable, and the parent class inserts that into the query, like
"UPDATE " . $this->table . " SET <data> WHERE " . $this->idFieldName
It seems sloppy to do it this way but I can't think of a better solution at this point.
Is there a design model or paradigm that links together or abstracts data-key names to be shared as a reference by both a database and an application?
What you are looking for is called an Object-Relational Mapping layer.
An ORM separates the concerns of data access from business logic by mapping a relational database into an object model. Since the ORM does all the translation, if you change the name of a database table or column, you only have to tell the ORM once, and it will properly apply that change to all of your code.
Since you indicate that you are using PHP, here is a question that addresses ORM libraries in PHP. Additional information about ORM technologies can be found in Wikipedia.