I am using peewee ORM and have a table with a composite, non-integer primary key. The key is composed of a timestamp and an integer foreign key field. An example table and the related model are shown below. I have many tables like this for different data sets I'm working with.
class fof2_site_hourly_manual(Base):
descriptor = TextField(null=True)
qualifier = TextField(null=True)
source = ForeignKeyField(db_column='source_id', rel_model=ionosonde)
valid_time = DateTimeField()
value = FloatField(null=True)
class Meta:
db_table = 'fof2_site_hourly_manual'
primary_key = CompositeKey('valid_time','source')
class ionosonde(Base):
name = TextField()
latitude = FloatField()
longitude = FloatField()
instrument_type = TextField()
class Meta:
db_table = 'ionosonde'
Now, I can insert rows with no problem using either the save() method of an instance or the class.insert().execute() approach.
However, I can't update rows by any means I've found. Using either save() or class.update().execute() I get 'duplicate primary key' errors. Fine, of course there's a duplicate, that's why I'm trying to update.
It gets worse, I get the same error just trying to examine a row using class.get(). I can read rows using select().
Is this a known limitation in peewee? The docs do state that support for composite primary keys is pretty basic, but don't explicitly state what the limitations are.
Edit: Here are all the things I have tried so far. I have the fields and values of the product I want to insert or update a row for in a dictionary, lets call it data. For inserts:
obj = class(**data)
obj.save(force_insert=True)
or
class.insert(**data).execute()
both work. However, if the row already exists (the primary key matches the relevant fields in data) then I've tried
obj = class(**data)
obj.save()
and
class.update(**data).execute()
this last one is probably not expected to work (no where clause) but I've included in for completeness in terms of what I've tried. In desperation, I attempted to implement a select, delete then insert loop, but that also fails since:
obj = class.get(**data)
and
obj = class.get_or_create(**data)
also give duplicate key errors if the row exists! That one really bakes my noodle, I'm not even trying to write to the DB!
I can read existing data out using
query = class.select()
and constructing where clauses using the known primary key fields.
Edit:
Ha! Just tried using select() to instantiate an object for a given row, then use delete_instance() to delete it (to then re-insert with the updated data). However, it deleted the entire contents of the table! It really seems as though peewee does not support composite keys, or am I doing something very wrong?
Related
I'm trying to update (MS Docs) multiple rows without knowing the primary key and without fetching each row first as I might have hundreds or thousands of rows to update.
Scenario - I have a webhook that's coming into my controller and it provides me with a list if Stripe transfer ids (ex. "12345", "67890", "111111"). I have a table called Transfer that has this column 'StripeTransferId' but it's not the primary key for the table.
Problem - I need to loop through each entry from the webhook and update each entity in my DB without knowing the primary key and without fetching each row before hand, as there might be thousands of rows to update.
Here is an example of what I need to do
[HttpPost("stripe")]
public async Task<IActionResult> StripeWebhook()
{
// parse and read data from webhook
// get a list of stripeTransferIds from the webhook data
// loop through each id and update corresponding row in db
foreach(string stripeTransferId in stripeTransferIds) {
// change the status of the transfer in the DB to complete and update
var existingTransfer = new Core.Entities.Transfer { TransferStatus = TransferStatus.Complete, StripeTransferId = transferId };
_context.Attach(existingTransfer);
_context.Entry(existingTransfer).State = EntityState.Modified;
}
_context.SaveChangesAsync();
}
Doing this I get an error below saying it's trying to insert and not update the row
Message [string]:"Cannot insert duplicate key row in object 'dbo.Transfers' with unique index 'IX_Transfers_StripeTransferId'. The duplicate key value is (tr_1KNMXEH6wPz6LzpKxzl1Q39h).\nThe statement has been terminated."
FYI - If I comment out the line that attaches the entity I get an error saying 0 rows were affected.
I have a table called Transfer that has this column 'StripeTransferId' but it's not the primary key for the table. . . . I need to loop through each entry from the webhook and update each entity in my DB without knowing the primary key and without fetching each row
If StripeTransferId has a unique index then you can treat it as the key. Just configure the entity with that as the Key, and your update using a stub entity will work. EF doesn't care which unique index, unique constraint, or Primary Key constraint you use for the entity key.
Or use a Raw SQL Query to perform the update, which is probably your best bet. Especially since you might be able to send all the (StripeTransferId, State) pairs to the database in one call using a Table-Valued parameter, JSON, XML, or a comma-delimited string (depending on your RDBMS).
I hope that you use await in your original code. Since attaching is not working I would try this
var existingTransfers = await _context.Set<Transfer>().Where(t=>
stripeTransferIds.Contains (t.StripeTransferId ).ToListAsync();
foreach(var item in existingTransfers)
item.TransferStatus = TransferStatus.Paid;
await _context.SaveChangesAsync();
this way you can put everything in one transaction. If you have a very large quantiy of Ids maybe it makes a sense to user raw sql string or even a stored procedure to increase performance
So I need to find a way to track modifications made to database rows. I'm thinking the best way to do this is to store object deltas for the row after the modification has been made to the row. So if I had this as a table:
CREATE TABLE global.users
(
id serial NOT NULL,
username text,
password text,
email text,
admin boolean, --
CONSTRAINT users_pkey PRIMARY KEY (id)
) WITH (
OIDS=FALSE
);
ALTER TABLE global.users
OWNER TO postgres;
COMMENT ON COLUMN global.users.admin IS '';
So say I updated the username field from "support" to "admin", then I'd save a JSON object like, say:
{
"username": ["support", "admin"]
}
associated with a specific row ID.
So my question is as follows: what's a nice way to organize these objects in Postgres? I'm currently debating between either a) having a deltas table for every existing table in the database (so this object would go into a table called global.users_delta or similar) or b) having a global deltas table which holds all deltas for all objects and tracks which table each is associated with.
I haven't really been able to identify any best practices for doing this sort of thing as yet, so even some direction towards preexisting documentation would be nice.
EDIT: An added requirement here is the issue of how to deal with the related data. So say, something belongs to a category, which is stored in another table. Usually that would be referenced by ID, so the delta would track the change in category ID's numeric value. That value needs to be labeled somehow, but that label can't necessarily be retroactively applied (in case the other linked value changes say). Should the labeled value be stored or should the raw value be stored? Or maybe both?
I have an SQL Server table structure where in some columns are foreign keys that refer to different look up tables. These columns are created as NOT NULL and with DEFAULT value = 1 as the look up value for ID = 1 is the default value I want to assign in case NULL is passed while saving a record.
However, I got below error while trying to Add/Update/Delete records of this child table.
An attempt was made to remove a relationship between a (*) and a (*). However, one of the relationship's foreign keys (X) cannot be set to null.
I tried to search a lot but I got solution to change LINQ to SQL code or XML manually for resolving this. I cannot do that because in future when ever the table gets changed, I'll have to make sure the manual change is applied every time.
Also, I can not change my columns to accept NULL because that will impact other applications using the same table through LINQ To SQL.
I need more manageable solution for this.
I assume that you are trying to "reset" a reference to its default value by assigning null to it, expecting L2S to adopt the database default. However, as for L2S, you're only nullifying a foreign key. It's not aware of any defaults in the database.
So you have to change the reference by setting it to the default lookup object yourself. You could do that by adding a Reset() method to your classes, so you don't have to scatter this code all over the place.
I got this resolved by assigning default values to the Table class properties from Code it self. What I understood is this. LINQ to SQL does not take into consideration the default value set from the DB. Also, the column is NOT NULL. So being INT columns the default value that .NET assigned was 0 to these properties. Now, I don't have any master entry with ID = 0 in my database. Also, since the DB did not get a NULL value for this column, it does not attempt to apply the Default value at all.
So what I did was, before saving the record, if I find that if the control which is bound to this property is having a NULL value (Nothing is selected/ control is invisible etc) I assign the Default value from the code it self. This eliminates the need to fall back on the database to assign the default.
Now, everything is working fine. Thought to share this as I could not get any satisfactory answers from anywhere.
I have this really strange problem where my entity framework query isn't enumerating correctly.
The SQL Server table I'm using has a table with a Sku field, and the column is "distinct". It isn't a key, but it doesn't contain any duplicate values. Using actual SQL with where, distinct and group by cluases I have confirmed this.
However, when I do this:
// Not good
foreach(var product in dc.Products)
or
// Not good
foreach(var product in dc.Products.ToList())
or
// Not good
foreach(var product in dc.Products.OrderBy(p => p.Sku))
the first two objects that are returned ARE THE SAME!!!
The third item was technically the second item in the table, but then the fourth item was the first row from the table again!!!
The only solution I have found is to use the Distinct extension method, which shouldn't really do anything in this situation:
// Good
foreach(var product in dc.Products.ToList().Distinct())
Another weird thing about this is that the count of the resulting queries is the same!!! So whether or not the resulting enumerable has the correct results or duplicates, I always get the number of rows in the actual table! (No I don't have a limit clause anywhere).
What could possibly cause this!?!?!?
Make sure you have a Primary Key for the table.
I'm having a little bit of trouble saving data to a database. Basically, I have a main table that has associations to other tables (Example Below).
Tbl_Listing
ID
UserID - Associated to ID in User Table
CategoryID - Associated to ID in Category Table
LevelID - Associated to ID in Level Table.
Name
Address
Normally, it's easy for me to add data to the DB (using Entity Framework). However, I'm not sure how to add data to the fields with associations. The numerous ID fields just need to hold an int value that corresponds with the ID in the associated table.
For example; when I try to access the column in the following manner I get a "Object reference not set to an instance of an object." error.
Listing NewListing = new Listing();
NewListing.Tbl_User.ID = 1;
NewListing.Tbl_Category.ID = 2;
...
DBEntities.AddToListingSet(NewListing);
DBEntities.SaveChanges();
I am using NewListing.Tbl_User.ID instead of NewListing.UserID because the UserID field is not available through intellisense.
If I try and create an object for each related field I get a "The relationship between the two objects cannot be defined because they are attached to different ObjectContext objects." error.
With this method, I am trying to add the object without the .ID shown above - example NewListing.User = UserObject.
I know this should be simple as I just want to reference the ID from the associated table in the main Listing's table. Any help would be greatly appreciated.
Thanks in advance,
-S
In general, with Entity Framework you don't use the ID:s of foreign keys, instead you use references. Instead of setting the ID, you set the property of the associated table.
In your case, it would be something like this:
Listing newListing = new Listing();
newListing.Tbl_User = DBEntities.Users.Single(u=>u.ID == 1);
newListing.Tbl_Category = DBEntities.Categories.Single(c=>c.ID == 2);
...
DBEntities.AddToListingSet(newListing);
DBEntities.SaveChanges();
In Entity Framework 4 (coming with .NET 4.0) you'll be able to use a simpler syntax more like what you expected.
You can have a look at this article on msdn where you can read more about this!