I am hitting a road block with preloading and associations
type Entity struct {
ID uint `gorm:"primary_key"`
Username string
Repositories []*Repository `gorm:"many2many:entity_repositories"`
}
type Repository struct {
ID uint `gorm:"primary_key"`
Name string
Entities []*Entity `gorm:"many2many:entity_repositories"`
}
With small users numbers the preloading is fine using the below
db.Preload("Repositories").Find(&list)
Also tried
db.Model(&User{}).Related(&Repository{}, "Repositories").Find(&list)
The preload seems to a select * entities and then a inner join using a SELECT * FROM "repositories" INNER JOIN "entity_repositories" ON "entity_repositories"."repository_id" = "repositories"."id" WHERE ("entity_repositories"."entity_id" IN ('1','2','3','4','5','6','7','8','9','10'))
As the number of user's increases this is no longer maintainable as it hits a sqlite limit (dev). I've tried numerous permutations! .. Realistically i guess i just want it to do something like
SELECT entities.*, repositories.*
FROM entities
JOIN entity_repositories ON entity_repositories.entity_id = entities.id
JOIN repositories ON repositories.id = entity_repositories.repository_id
ORDER BY entities.id
And fill in the model for me ..
Am I doing something obviously wrong or?
Unfortunately that's just the way GORM handles preloading.
go-pg has slightly better queries, but doesn't have the same functionality as GORM. It still will do multiple queries in some cases.
I would honestly recommend just using query building with raw SQL, especially if you know what your models will look like at compile time. I ended up going with this approach in my project, despite the fact that I didn't know that my models were going to look like.
Can try the Joins(), which can perform JOIN syntax in SQL
Related
I have inherited a very old DB and I'm just starting out with .NET Core 3 (preview) to see if I can create a proof-of-concept WebApi that uses EF Core to query the DB.
So my problem is that in this DB, much of the data is untrimmed (so has lots of white space around values). There's a mix of varchar(n) and char(n) fields. And no, I'm not in a position to re-engineer this.
So my challenge is that the WebApi will receive a query which will contain a List<People>, and I need to return all relevant data about those people.
If I were doing this in SQL I guess I would do something like:
SELECT *
FROM Table1
INNER JOIN Table2
ON RTRIM(LTRIM(Table1.[P1])) = RTRIM(LTRIM(Table2.[P9]))
I need to be able to do something similar using Entity Framework where I would join the DB table with the .NET List<People>. However, I'm a) not sure how best to go about this with the necessary trims and b) the number of rows in this DB is very large and so it has to be a set-based operation performed on the SQL Server - I can't bring everything back to the DB and join in-memory.
[EDIT (24th June)]
To add some examples of what I've tried, and to re-focus the question onto the approach I should be employing:
1 - I attempted this using a JOIN
var y = from a in query
join b in request.Items
on new { a.Prop1, a.Prop2 } equals new { b.Prop1, b.Prop2 }
select new { a.Prop1, a.Prop2, a.Prop3 };
return await y.ToListAsync().ConfigureAwait(false);
The compile error returned is:
The type of one of the expressions in the join clause is incorrect.
Type inference failed in the call to 'Join'.
Not sure why, but I suspect that this is because I'm trying to join strings to DB char fields.
Also tried:
var y = from a in query
from b in request.Items
where a.Prop1 == b.Prop1 && a.Prop2 == b.Prop2
select new {a.Prop1, a.Prop2, a.Prop3};
return await y.ToListAsync().ConfigureAwait(false);
This compiles, but gives run-time error:
System.InvalidOperationException: 'collection selector was not
NavigationExpansionExpression'
So, yes, I do need to be able to ensure that I'm trimming the data when doing the "join", but I think the main problem is that this needs to be a server-side operation (because of the size of the data sets) but I'd doing a join with a DB table against an in-memory List on the Client.
Not sure what the best approach is here (with EF Core 3).
I'm writing an application that use Hibernate to query the database (SQL Server).
Now I'm querying a link table for all Items.
The query looks like:
"FROM UserRole ur join ur.platformUser join ur.role join ur.company"
I need all the UserRoles objects in a list but when I query the above query I got a arrao of objects with UserRole, Role, Company and PlatformUser objects in it.
I only need the UserRole objects with the other objects in the UserRole object.
How can I solve this in Hiernate that I can cast the result to for ex. Arraylist<UserRole>?
I tried following syntax:
Query query = session.createSQLQuery("select * FROM UserRole ur join PlatformUser pu ON pu.userId = ur.userId join [Role] r ON r.roleId = ur.roleId join [Company] c ON c.companyId = ur.companyId").addEntity(UserRole.class);
With this line I got a List of UserRoles but all the underlying objects are NULL.
It could be that you've mapped the fields in the UserRole as lazy loaded? (Or haven't specified, I believe lazy loading is the default)
Try writing something along the lines of
Object obj = userRoles.get(0).getName(); //Or any of the previously null valued fields you have in there
Make sure you're writing this still within session, preferably right after the query.list(); itself
That should lazy load the value, and if that was indeed the case, look into initializing hibernate objects.
edit:
If you're looking for a more proper way of doing this, there's actually a hibernate method that requests to initialize a proxy object, I'm unsure why it's better than just getting the thing you want to initialize from code, but it sure is prettier.
Hibernate.initialize(Object initializeMe)
But as far as I know, it's a shallow method, meaning it won't load entities inside of this one. To achieve that you'll need to either do it by hand, or make a generic method that'll load everything, think reflection & recursive.
I am after a database system to store my email messages (over 4 million now) which would allow me to search over emails easily programmatically, so far I only found these two, and alternative is come up with my own schemea and sync the IMAP/Pop3 folders as the emails come in. However reading this : DBMAIL for search seems due to the way dbmail keeps the emails, searching over the emails is not very possible.
My main goal is to be able to read the emails and search through them lets say using python.
I'm looking for something almost exactly the same, storage and search across a huge corpus of emails and attachments without and clear winners. Would love to compare notes. Appears that archiveopteryx are dbmail the two front runners with mature db schemas after a quick googling. Both seem to be suites optimized around mta integration delivery/pop/imap type services, though I'm still investigating both.
I also found this
http://www.flaterco.com/aemail/aedocs.html
but it seems unmaintained since about 2003.
I can't seem to figure out how to send a PM or connect with people on this service yet.
Some other alternative choices:
http://sqmail.sourceforge.net/ is written in Python but unsupported since a long time.
http://www.manitou-mail.org/ which is written in Perl (by me!) with an optional GUI in Qt/C++.
A little bit late, but you can do your searching easily by creating views in your database and then do your searching.
/* to, from and subject*/
create view as EmailHeaders
select
`m`.`physmessage_id` AS `physmessage_id`,
`ph`.`internal_date` AS `internal_date`,
max(if(`n`.`headername` = 'from',`v`.`headervalue`,NULL)) AS `From`,
max(if(`n`.`headername` = 'to',`v`.`headervalue`,NULL)) AS `To`,
max(if(`n`.`headername` = 'subject',`v`.`headervalue`,NULL)) AS `Subject`
from
(
(
(`dbmail_messages` `m`
join `dbmail_header` `h` on (`m`.`physmessage_id` = `h`.`physmessage_id`)
)
join `dbmail_physmessage` `ph` on(`m`.`physmessage_id` = `ph`.`id`)
)
join `dbmail_headername` `n` on(`h`.`headername_id` = `n`.`id`)
)
join `dbmail_headervalue` `v` on(`h`.`headervalue_id` = `v`.`id`))
where
`n`.`headername` = 'to'
or `n`.`headername` = 'from'
or `n`.`headername` = 'subject'
group by `m`.`physmessage_id`
You can also create views to disaplay messages, but is not efficient (memory consuming) but you can use something like this
select
physmessage_id,
sum(prts.size) emailSize
from dbmail_partlists list, dbmail_mimeparts prts, EmailHeaders eh
where list.part_id=prts.id
and list.is_header=0
and eh.physmessage_id=list.physmessage_id
and prts like '%your_word_here%'
group by list.physmessage_id
For small servers these queries are enough, but for large ones, you may need to employ other methods, but essentially this is it.
Using the Toxi solution, how should I select the tags for a certain "Bookmark" (to keep the Delicious theme):
I can either:
1) Join in a single query:
select bookmark.title, bookmark.url,
(
SELECT group_concat( tags.name ) as tagNames
FROM taggings INNER JOIN tags
ON taggings.tagId_fk=tags.tagId
WHERE taggings.bookmarkId_fk = bookmarks.bookmarkId_fk
)
from bookmarks
where bookmarks.id=1 ;
^^ That gives
title url tagNames
A bkmrk http://url.com tag1,tag2,tag3
2) Use two queries: one to retrieve the bookmark id's to display, then another to retrieve the tags for those bookmarks. The results can then be merged in PHP.
So really this question is: In general efficiency/database load-wise is it better to do more joining in a single query or multiple queries?
How do you make that kind of decision? Or do you simply not think about it until load causes a problem?
Server side is more efficient.
In both cases, the server must read all of the tags.
If you bring them to PHP, then they must all travel over the wire and PHP has to fiddle with them.
If you do them on the server, the finished answer (smaller) comes over the wire ready for PHP to pass it up to the UI.
I have a basic "property bag" table that stores attributes about my primary table "Card." So when I want to start doing some advanced searching for cards, I can do something like this:
SELECT dbo.Card.Id, dbo.Card.Name
FROM dbo.Card
INNER JOIN dbo.CardProperty ON dbo.CardProperty.IdCrd = dbo.Card.Id
WHERE dbo.CardProperty.IdPrp = 3 AND dbo.CardProperty.Value = 'Fiend'
INTERSECT
SELECT dbo.Card.Id, dbo.Card.Name
FROM dbo.Card
INNER JOIN dbo.CardProperty ON dbo.CardProperty.IdCrd = dbo.Card.Id
WHERE (dbo.CardProperty.IdPrp = 10 AND (dbo.CardProperty.Value = 'Wind' OR dbo.CardProperty.Value = 'Fire'))
What I need to do is to extract this idea into some kind of stored procedure, so that ideally I can pass in a list of property/value combinations and get the results of the search.
Initially this is going to be a "strict" search meaning that the results must match all elements in the query, but I'd also like to have a "loose" query so that it would match any of the results in the query.
I can't quite seem to wrap my head around this one. My previous version of this was to do generate some massive SQL query to execute with a lot of AND/OR clauses in it, but I'm hoping to do something a little more elegant this time. How do I go about doing this?
it seems to me that you have an EAV model here.
if you're using sql server 2005 and up i'd suggest you use XML datatype for this:
http://weblogs.sqlteam.com/mladenp/archive/2006/10/14/14032.aspx
makes searching and stuff much easier with built in xml querying capabilities.
if you can't change your model then look at this:
http://weblogs.sqlteam.com/davidm/articles/12117.aspx