Yii MANY_MANY on clause - database

I have the following tables in my database.
subscriber(
subscriber_id int,
status char(1),
...
)
sendlist(
sendlist_id int,
...
)
subscriber_list(
subscriber_id int,
sendlist_id int,
status char(1)
)
in my SendList model I have a relation defined as follows:
'Subscribers'=>array(self::MANY_MANY, 'Subscriber', 'subscriber_list(sendlist_id, subscriber_id)',"on"=>"Subscribers_Subscribers.status='a' and Subscribers.status='a'")
I have tried setting both the on and condition clause, and one or the other, however the sole Subscriber in the table at present (linked to this sendlist) is ALWAYS returned, regardless of whether it's status in the subscriber table, or the subscriber_list table is set to 's'.
When I check the DB - the query that the join supposedly gives something along the lines of:
SELECT `Subscribers`.`subscriber_id` AS `t1_c0`, `Subscribers`.`full_phone` AS `t1_c1`, `Subscribers`.`status` AS `t1_c2`, `Subscribers`.`contact` AS `t1_c3`, `Subscribers`.`client_id` AS `t1_c4`, `Subscribers`.`date_joined` AS `t1_c5` FROM `subscriber` `Subscribers`
INNER JOIN `subscriber_list` `Subscribers_Subscribers` ON (`Subscribers_Subscribers`.`sendlist_id`=7075) AND (`Subscribers`.`subscriber_id`=`Subscribers_Subscribers`.`subscriber_id`) AND (Subscribers.status='a' and Subscribers_Subscribers.status='a')
WHERE (Subscribers.status='a' and Subscribers_Subscribers.status='a');
returns an empty set (as it should). But when I print_r the $list->Subscribers array, I get the following:
Array(
[0] => Subscriber Object
(
...
[_attributes:CActiveRecord:private] => Array
(
[subscriber_id] => 2043221
[full_phone] => 447944426885
[status] => s
[contact] => 0
[client_id] => 14002
[date_joined] =>
)
so it recognises that the subscriber is status 's', but still loads it!
The code that causes the issue is here:
$client = new Client();
$client->save(false);
//ensure that we are working with a clean subscriber set.
$subscribers = Subscriber::model()->findAll();
foreach($subscribers as $subscriber){
$subscriber->delete();
}
//create a new subscriber with a set of filter options.
$subscriber = new Subscriber();
$subscriber->client_id = $client->client_id;
$subscriber->full_phone = "44712345678";
$subscriber->country = "England";
$subscriber->gender = "Male";
$subscriber->save(false);
//create a new list with a set of matching options
$list = new SendList();
$list->client_id = $client->client_id;
$list->gender = "=Male";
$list->save(false);
//confirm that the subscriber can be added to the list.
self::assertTrue($list->canImport($subscriber));
$list->import();
//confirm that the subscriber has been added to the list.
self::assertEquals(1, $list->active_subscriber_count);
self::assertEquals(1, sizeof($list->Subscribers));
//remove all subscribers (set status to 's')
$list->remove();
//check that the list no longer records the subscribers presence.
self::assertEquals(0, $list->active_subscriber_count);
self::assertEquals(1, $list->stopped_subscriber_count);
//check that the subscriber has no lists associated with it.
$subscriber->refresh();
self::assertEquals('s', $subscriber->status, sizeof($subscriber->SendLists));
//attempts to refresh the list. Tested without these with the same result.
$list_id = $list->sendlist_id;
unset($list);
unset($subscriber);
$list2 = SendList::model()->findByPk($list_id);
$list2->refresh();
//check that the the list has no subscribers - this fails.
self::assertEquals(0, sizeof($list2->Subscribers));
Have I missed something silly?

Related

linq2db - server side bulkcopy

I'm trying to do a "database side" bulk copy (i.e. SELECT INTO/INSERT INTO) using linq2db. However, my code is trying to bring the dataset over the wire which is not possible given the size of the DB in question.
My code looks like this:
using (var db = new MyDb()) {
var list = db.SourceTable.
Where(s => s.Year > 2012).
GroupBy(s => new { s.Column1, s.Column2 }).
Select(g => new DestinationTable {
Property1 = 'Constant Value',
Property2 = g.First().Column1,
Property3 = g.First().Column2,
Property4 = g.Count(s => s.Column3 == 'Y')
});
db.Execute("TRUNCATE TABLE DESTINATION_TABLE");
db.BulkCopy(new BulkCopyOptions {
BulkCopyType = BulkCopyType.MultipleRows
}, list);
}
The generated SQL looks like this:
BeforeExecute
-- DBNAME SqlServer.2017
TRUNCATE TABLE DESTINATION_TABLE
DataConnection
Query Execution Time (AfterExecute): 00:00:00.0361209. Records Affected: -1.
DataConnection
BeforeExecute
-- DBNAME SqlServer.2017
DECLARE #take Int -- Int32
SET #take = 1
DECLARE #take_1 Int -- Int32
SET #take_1 = 1
DECLARE #take_2 Int -- Int32
...
SELECT
(
SELECT TOP (#take)
[p].[YEAR]
FROM
[dbo].[SOURCE_TABLE] [p]
WHERE
(([p_16].[YEAR] = [p].[YEAR] OR [p_16].[YEAR] IS NULL AND [p].[YEAR] IS NULL) AND ...
...)
FROM SOURCE_TABLE p_16
WHERE p_16.YEAR > 2012
GROUP BY
...
DataConnection
That is all that is logged as the bulkcopy fails with a timeout, i.e. SqlException "Execution Timeout Expired".
Please note that running this query as an INSERT INTO statement takes less than 1 second directly in the DB.
PS: Anyone have any recommendations as to good code based ETL tools to do large DB (+ 1 TB) ETL. Given the DB size I need things to run in the database and not bring data over the wire. I've tried pyspark, python bonobo, c# etlbox and they all move too much data around. I thought linq2db had potential, i.e. basically just act like a C# to SQL transpiler but it is also trying to move data around.
I would suggest to rewrite your query because group by can not return first element. Also Truncate is a part of the library.
var sourceQuery =
from s in db.SourceTable
where s.Year > 2012
select new
{
Source = s,
Count = Sql.Ext.Count(s.Column3 == 'Y' ? 1 : null).Over()
.PartitionBy(s.Column1, s.Column2).ToValue()
RN = Sql.Ext.RowNumber().Over()
.PartitionBy(s.Column1, s.Column2).OrderByDesc(s.Year).ToValue()
};
db.DestinationTable.Truncate();
sourceQuery.Where(s => s.RN == 1)
.Insert(db.DestinationTable,
e => new DestinationTable
{
Property1 = 'Constant Value',
Property2 = e.Source.Column1,
Property3 = e.Source.Column2,
Property4 = e.Count
});
After some investigation I stumbled onto this issue. Which lead me to the solution. The code above needs to change to:
db.Execute("TRUNCATE TABLE DESTINATION_TABLE");
db.SourceTable.
Where(s => s.Year > 2012).
GroupBy(s => new { s.Column1, s.Column2 }).
Select(g => new DestinationTable {
Property1 = 'Constant Value',
Property2 = g.First().Column1,
Property3 = g.First().Column2,
Property4 = g.Count(s => s.Column3 == 'Y')
}).Insert(db.DestinationTable, e => e);
Documentation of the linq2db project leaves a bit to be desired however, in terms of functionality its looking like a great project for ETLs (without horrible 1000s of line copy/paste sql/ssis scripts).

Select specific columns with repository in Entity Framework Core

I have a big table with a binary column for picture. I need to show contents of this table in a view and make it searchable. I have tried only selecting a subset of columns that are needed in this. However, the generated SQL always has all the columns of the table in the generated query.
public IQueryable<ApplicantDTO> GetApplicantQueryable()
{
return DataContext.Applicants
.Include(a => a.Nationality)
.Select(x => new ApplicantDTO
{
Id = x.Id,
BirthDate = x.BirthDate,
Gender = x.Gender,
FirstName = x.FirstName,
LastName = x.LastName,
OtherName = x.OtherName,
MobileNo = x.MobileNo,
Age = x.Age,
Nationality = x.Nationality.National,
Admitted = x.admitted,
Completed = x.Completed
})
.Where(a => a.Admitted == false && a.Completed == true)
.OrderBy(a => a.LastName)
.AsNoTracking();
}
But instead of just specifying the above rows, the generated SQL from profiler is
SELECT
[a].[Id], [a].[BirthDate], [a].[BirthPlace], [a].[CompleteDate],
[a].[Completed], [a].[ContentType], [a].[CreateDate],
[a].[Denomination], [a].[Disability], [a].[Email],
[a].[FirstName], [a].[Gender], [a].[HomeTown], [a].[LastName],
[a].[MarryStatus], [a].[MatureApp], [a].[MobileNo], [a].[NationalityID],
[a].[OtherName], [a].[Passport], [a].[Pin], [a].[PostalAddress],
[a].[Region], [a].[Religion], [a].[ResAddress], [a].[SerialNo],
[a].[Title], [a].[VoucherID], [a].[admitted], [a.Nationality].[National]
FROM
[Applicants] AS [a]
INNER JOIN
[Nationality] AS [a.Nationality] ON [a].[NationalityID] = [a.Nationality].[Id]
WHERE
([a].[admitted] = 0)
AND ([a].[Completed] = 1)
ORDER BY
[a].[LastName]
With all the underlying columns all included in the query.
I tried putting it in an anonymous type before casting it to the ApplicantDTO but still the same effect.
What's wrong?

database not inserting using PDO prepared statements

I have a paypal form which is submitted and working, this is part of the ipn, which means I can not see the error reporting on page which means I am a little blind on what is the issue.
Everything looks fine I think.
Here is the code:
$newcustom = rtrim($_POST['custom'], ',');
$buyingarray = explode('~', $newcustom);
$name = $buyingarray[1];
$phone = $buyingarray[2];
$email = $buyingarray[3];
$comments = $buyingarray[4];
$date = $buyingarray[5];
$time = explode(",",$tt[6]);
$person = explode(",",$tt[7]);
$orderID = $tt[0];
$booking_date = strtotime($date);
$nowitsdate = date('Y-m-d G:i:s', strtotime("now"));
$addreservation = $pdo->prepare("INSERT INTO reservations (id, dateCreated, name, email, phone, comments, status, eventID, voucherCode, voucherplace, OrderID) VALUES ('', :dateCreated, :name, :email, :phone, :comments, 1, NULL, '', 'PayPal Purchase', :OrderID)");
$addreservation->execute(array(':dateCreated' => $nowitsdate,':name' => $name,':email' => $email,':phone' => $phone,':comments' => $comments,':OrderID' => $orderID));
$addreservation_num = $addreservation->rowCount();
if($addreservation_num == 0){ $inserterror = $pdo->query("INSERT INTO testipn (id, testing, testing2) VALUES ('','Input into reservations died','')"); exit();}
Now I know that everything is ok with the details coming in as I have input each of the variables into the testipn row which is telling me if an error occurred when the ipn is being used.
As far as I can see everything looks fine yet it still is not finding the problem.
Now the only thing I can think it can be is the NULL within the eventID, maybe that is causing an issue?
If a ID is not present which it wont be within this table, then it needs to be NULL.
The rows effected is 0 which means my error row in the database says: Input into reservations died
Thanks for any input into finding a solution :)

Cannot understand how will Entity Framewrok generate a SQL statement for an Update operation using timestamp?

I have the following method inside my asp.net mvc web application :
var rack = IT.ITRacks.Where(a => !a.Technology.IsDeleted && a.Technology.IsCompleted);
foreach (var r in rack)
{
long? it360id = technology[r.ITRackID];
if (it360resource.ContainsKey(it360id.Value))
{
long? CurrentIT360siteid = it360resource[it360id.Value];
if (CurrentIT360siteid != r.IT360SiteID)
{
r.IT360SiteID = CurrentIT360siteid.Value;
IT.Entry(r).State = EntityState.Modified;
count = count + 1;
}
}
IT.SaveChanges();
}
When I checked SQL Server profiler I noted that EF will generated the following SQL statement:
exec sp_executesql N'update [dbo].[ITSwitches]
set [ModelID] = #0, [Spec] = null, [RackID] = #1, [ConsoleServerID] = null, [Description] = null, [IT360SiteID] = #2, [ConsoleServerPort] = null
where (([SwitchID] = #3) and ([timestamp] = #4))
select [timestamp]
from [dbo].[ITSwitches]
where ##ROWCOUNT > 0 and [SwitchID] = #3',N'#0 int,#1 int,#2 bigint,#3 int,#4 binary(8)',#0=1,#1=539,#2=1502,#3=1484,#4=0x00000000000EDCB2
I can not understand the purpose of having the following section :-
select [timestamp]
from [dbo].[ITSwitches]
where ##ROWCOUNT > 0 and [SwitchID] = #3',N'#0 int,#1 int,#2 bigint,#3 int,#4 binary(8)',#0=1,#1=539,#2=1502,#3=1484,#4=0x00000000000EDCB2
Can anyone advice?
Entity Framework uses timestamps to check whether a row has changed. If the row has changed since the last time EF retrieved it, then it knows it has a concurrency problem.
Here's an explanation:
http://www.remondo.net/entity-framework-concurrency-checking-with-timestamp/
This is because EF (and you) want to update the updated client-side object by the newly generated rowversion value.
First the update is executed. If this succeeds (because the rowversion is still the one you had in the client) a new rowversion is generated by the database and EF retrieves that value. Suppose you'd immediately want to make a second update. That would be impossible if you didn't have the new rowversion.
This happens with all properties that are marked as identity or computed (by DatabaseGenertedOption).

CAKEPHP Group by problem in paginate

CAKEPHP Group by problem in paginate..
This is my table structure
Friends Table
`id` int(11) NOT NULL AUTO_INCREMENT,
`user1_id` int(11) NOT NULL DEFAULT '0',--> UserFrom
`user2_id` int(11) NOT NULL DEFAULT '0',---> UserTo
I need to get all the unique records of one user and his user1_id = 100
There are lot of duplicate values in user2_id. I need to get the unique values
While i trying this code it returns only first 12 values(according to limit).
If i commented the group by line then all records are displaying (including duplicate values)
$this->paginate = array(
'conditions' => $conditions,
'contain'=>array(
'UserFrom'=>array(
'fields'=>array(
'UserFrom.id',
'UserFrom.first_name',
'UserFrom.last_name',
),
),
'UserTo'=>array(
'fields'=>array(
'UserTo.id',
'UserTo.first_name',
'UserTo.last_name',
)
)
),'limit' => 12,
'order' => array('Friend.id' => 'desc'),
'recursive'=>0,
'fields'=>array('DISTINCT Friend.user2_id','Friend.*'),
'group'=>array('UserTo.id'),
);
This is my sql query on that page
SELECT COUNT(*) AS count FROM friends AS Friend LEFT JOIN users AS UserTo ON (Friend.user2_id = UserTo.id) LEFT JOIN users AS UserFrom ON (Friend.user1_id = UserFrom.id) WHERE UserFrom.nick_name = 'shyam' AND Friend.status = 'friend' GROUP BY UserTo.id
SELECT DISTINCT Friend.user2_id, Friend.*, UserTo.id, UserTo.first_name, UserTo.last_name, UserTo.nick_name, UserTo.name, UserTo.icon_medium, UserTo.icon_big, UserFrom.id, UserFrom.first_name, UserFrom.last_name, UserFrom.nick_name, UserFrom.name FROM friends AS Friend LEFT JOIN users AS UserTo ON (Friend.user2_id = UserTo.id) LEFT JOIN users AS UserFrom ON (Friend.user1_id = UserFrom.id) WHERE UserFrom.nick_name = 'shyam' AND Friend.status = 'friend' GROUP BY UserTo.id ORDER BY Friend.id desc LIMIT 12
1.First count query returning count properly
2.Second query if i put limit it not working properly. i.e there are 144 records in my table i need to display 12 per page.. only first page coming if i use group by and limit
You should have either DISTINCT or GROUP BY, not both - they're duplicating each other and probably causing problems with your query. Remove one of them and retry.
Note: If you want to manually request pages other than 1st with $this->paginate(), add 'page' parameter with page number to your $this->paginate array, like this:
$this->paginate = array_merge($this->paginate, array('page' => $pageNumber);
However, normally you wouldn't want to do this, as paginator does this automatically (passing page variable via URL parameter to your action) as long as you use pagination controls in your view (see http://book.cakephp.org/view/1233/Pagination-in-Views).

Resources