I was just wondering why updating a record (flush) slows down over time.
For example, if i want to check mulitple records and compare them with eachother and after comparisation update a column like setCompared(true);. The first 1000 records are fine and fast but after that, the flush seems like it slows down after each flush. Is there a reason for that, like maybe it builds up or something. Do i need to clear() after flush().
Does anyone have a explaination for that?
You should do something like this according to the Doctrine documentation) :
<?php
$batchSize = 20;
for ($i = 1; $i <= 10000; ++$i) {
$user = new CmsUser;
$user->setStatus('user');
$user->setUsername('user' . $i);
$user->setName('Mr.Smith-' . $i);
$em->persist($user);
if (($i % $batchSize) === 0) {
$em->flush();
$em->clear(); // Detaches all objects from Doctrine!
}
}
$em->flush(); // Persist objects that did not make up an entire batch
$em->clear();
The following code shows an example for inserting 10000 objects with a batch size of 20. This means that we insert them 20 by 20 with the flush().
Related
i'm trying to add xlsx data to my data base as entities i wrote this script
$batchCount = 10000;
$i = 0;
foreach($rows as $row) {
if (is_numeric($row[0])){
$i++;
$cart = new Carte();
//doing something
$this->em->persist($cart);
if (($i % $batchCount) == 0) {
$this->em->flush();
$this->em->clear();
}
}
}
But for a adding 50,000 line in database its taking to long
I'd love if some one have fast method to add large data to database using Symfony
I have a product controller and when I'm saving a new product I want to save some records to another related controller to make a record of what categories the product is associated with.
My code I'm using is:
$this->Product->create();
if ($this->Product->save($this->request->data)) {
$newProductId = $this->Product->getInsertID();
//associate products with categories
foreach($categoriesToSave as $key=>$value) {
$saveArray['CategoriesProduct'] = array('category_id'=>$value, 'product_id'=>$newProductId);
$this->Product->CategoriesProduct->create();
$this->Product->CategoriesProduct->save($saveArray);
$this->Product->CategoriesProduct->clear();
}
}
For some reason though, even if $categoriesToSave has 10 items in it, only the very last one is being saved. So it's obviuosly creating only the one new CategoriesProduct item and saving each record over the top of the last instead of create()-ing a new one.
Can anyone explain what I'm doing wrong and how I can make this work?
The way I would do it would be like this:
//Add a counter
$c = 0
foreach($categoriesToSave as $key=>$value) {
$saveArray[$c]['CategoriesProduct']['category_id'] = $value;
$saveArray[$c]['CategoriesProduct']['product_id'] = $newProductId;
$c++;
}
$this->Product->CategoriesProduct->saveMany($saveArray);
I don't think that is the only way to do it but that should work just fine.
Good luck!
I have I problem that I hope someone can help me with. I thought this code was right, but it will not work. Below is my code, it is a function for my CakePHP 2.2.2 site, the main aim of the code is to produce a menu system from database results. The problem is with my foreach loop, it will not loop. All $Key does is return the value of 2 (three records within the table at this time). So When I display / echo / debug $Menu, the only result I get is the last result stored within the database.
I know the SQL command is right, if that is debuged / echoed then all three results are displayed. The idea of this loop was to get it to count the results, so that I could run a check on a selected field. Where I am going wrong?
function MenuSystem() {
$this->loadModel('Menu');
$MenuSQL = $this->Menu->find('all', array('conditions' => array('active' => true)));
foreach ($MenuSQL as $Key=>$Value) {
$MenuSystem = $MenuSQL[$Key];
$this->Menu = $MenuSystem;
}
}
Many Thanks,
Glenn.
UPDATE :::
Below is my function, now my foreach loop now works, don't know what I was doing wrong, but I know think its working. You can see the print_r command that I am using for testing, if I use that, then all links from my database are printed / echoed on the screen and all works. But if I try and call the $this->Menu from another controller, then only the last record is echoed on the screen. I have moved the $this->Menu outside of the foreach loop, but it made no difference, with it inside the loop or outside, it still only echoes the last record and not all three. So what I am doing wrong?
function MenuSystem() {
$this->loadModel('Menu');
$SiteBase = '/projects/cake/';
$MenuSQL = $this->Menu->find('all', array('conditions' => array('active' => true)));
foreach ($MenuSQL as $key => $Value) {
$MenuAccessLevel = $MenuSQL[$key]['Menu']['roles_id'];
if ($MenuAccessLevel == 1) {
$Path = $MenuSQL[$key]['Menu']['path'];
$Title = $MenuSQL[$key]['Menu']['title'];
$MenuSys = "<a href=\" " . $SiteBase . $Path . " \">" . $Title ."";
} else {
print ("Admin");
}
//print_r($MenuSys);
} //End of Foreach Loop
$this->Menu = $MenuSys;
} //End of function MenuSystem
So When I display / echo / debug $Menu, the only result I get is the last result stored within the database.
You're setting the value of $this->Menu within the foreach, so when the foreach is complete it will take the last value iterated over.
If you want to find the number of records matching a condition, try:
$menuCount = $this->Menu->find('count', array(
'conditions'=>array('active'=>true)
));
$this->set(compact('menuCount'));
Edit: also, by setting the value of $this->Menu within the foreach, you're overwriting the Menu model variable. This is not a good idea.
Edit2: to get the counts of rows as grouped by some value, try:
$this->Menu->virtualFields = array('count' => 'count(*)');
$counts = $this->Menu->find('all', array(
'group'=>'Role',
'fields'=>array('Role', 'count'),
));
This generates SQL to have the results grouped by the Role column. Returned fields are the name of the role, and the number of rows having that value.
If you wanted to do it with a foreach loop instead, it might look like:
$menus = $this->Menu->find('all', array('fields'=>array('id', 'Role')));
$counts = array('user'=>0, 'admin'=>0);
foreach ($menus as $menu) {
$role = $menu['Menu']['Role'];
$counts[$role] += 1;
}
Again, a whole day and I am stuck again.
I need to use an array of words or var that contains forbidden words that cannot appear in an email address.
Either:
$baddies = 'smtp mailer sysop';
or
#baddies = qw(smtp mailer sysop);
or
#baddies = qw/smtp mailer sysop/;
There are more bad words in the array too, about two dozen.
I am not running the latest version of perl so ~~ and so on are not supported.
I have a loop going on that sends the bands schedule out.
In that loop I need to check to see if the email contains any of those words.
I realize there may be some good emails that contain a match but, that is fine.
I have tried literally dozens of examples after I gave up trying to figure it out.
Latest was:
####FYI## $uaddress is from the foreach $uaddress(#addresses){ loop.
my %params = map { $uaddress => 1 } #baddies;
if(exists($params{$uaddress})) {
print "yep, it's there"; #for testing
push(#failed,"$uaddress is restricted<br />");
But, everything I tried just does not do what I need.
I even tried =~ and so on.
I AM REALLY feeling stupid about now..
I need another lesson here folks.. Thanks in advance.
Update: I also tried:
$baddies = 'smtp mailer sysop';
my #baddies = split / /, $baddies;
# iterate through the array
foreach (#baddies) {
if($_ =~ $uaddress) #I also reversed that {
print qq~$uaddress contains $_~;
}
}
Through trial, error and dumb luck, I haphazardly got this below to work:
foreach $uaddress(#addressList) {
$uaddress =~ s/\s//g;
#baddies = qw(smtp mailer sysop);
my #failed;
foreach (#baddies) {
if($uaddress =~ m/$_/i) {
push(#failed,"$uaddress is restricted because it found \"$_\" in the address<br />");
}
}
##stuff happens with emails that passed like Email::Verify
}
ok sorry, wrote too little info here and i have done some research:
The question: What is the easiest way to write a model function that creates an unique code (5 characters) for each row in a cake model/table that doesnt already have a code?
//create a unique code
//set the random id length
function generateCode()
{
while($i > 0)
{
$random_id_length = 7;
//generate a random id encrypt it and store it in $rnd_id
$rnd_id = crypt(uniqid(rand(),1));
//to remove any slashes that might have come
$rnd_id = strip_tags(stripslashes($rnd_id));
//Removing any . or / and reversing the string
$rnd_id = str_replace(".","",$rnd_id);
$rnd_id = strrev(str_replace("/","",$rnd_id));
//finally I take the first 7 characters from the $rnd_id
$rnd_id = substr($rnd_id,0,$random_id_length);
//check if the code is unique
$i = $this->checkCode($rnd_id);
}
return $rnd_id;
}
//check to see if the code isnt already in the database
function checkCode($code)
{
$conditions = array("uniqueCode" => $code));
$result = $this->Visitors->count('all', array('conditions' => $conditions));
return $result;
}
//check how many has a code
function check()
{
$conditions = array("NOT" => array ("uniqueCode" => null));
$result = $this->Visitors->count('all', array('conditions' => $conditions));
return $result;
}
function update()
{
//update until everyone has a code
while( $i > 0)
{
$this->generateCode()
// get next visitors that DONT have a code.
$conditions = array ("uniqueCode" => null));
$visitor = $this->Visitors->find('first', array('conditions' => $conditions));
$i = $this->check();
}
echo 'Done';
}
Cant make it work properly and it must be an easier way?
How about an id column? :]
If You want it to be like a hash thing, I think use this:
substr(str_shuffle("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789,;:!?.$/*-+&#_+;./*&?$-!,"), 0, 5);
and you shouldn't worry about checking whether it's unique. Or do it, so create a hash like that and try to find it in the table, if it's not in there - your good to go.
in your Model, you can do something like this
<?php
function beforeSave() {
if (empty($this->data[$this->alias]['id'])) {
$this->data[$this->alias]['id'] = base_convert(uniqid(), 16, 36);
}
return $this->data;
}
?>
This will generate a unique id of 10 chars. You can figure out how to create a unique id of 5 chars, maybe converting the string to a higher base, but I find it difficult because you'll have to make your own function, base_convert has a limit of 36
One thing to think about is the length of the code. A five character code, assuming you use upper and lower case characters in it, has 380,204,032 possible unique codes. If you create these codes truly randomly, you'll rather soon hit the point where you're creating duplicates. There's a 50% chance of collision after you have generated 22,959 codes.
So, since for each new code you need to check whether it already exists or not,
(BTW, a much more elegant check is this):
do {
$code = /* create code */;
} while (!$this->isUnique(array('code' => $code)));
over time you'll hit more and more already existing codes, slowing down the generation of each code, to the point where you may have to loop 100 or more times to find a unique code (99% chance of collision at only 59,177 codes).
If you're just talking about a few thousand records max, I wouldn't worry about it. If your dataset could/is supposed to grow beyond that though, you should think either about a different scheme for your codes (UUIDs come to mind), generate codes sequentially or pre-generate a list of all possible codes, shuffle it and use up codes one by one.
There are a gazillion ways to do this and all you have to do is google the appropriate words. I found it in the PHP manual, so that would be a good place to start.
To make it unique, it would be based on date and would incorporate some sort of hashing.
Do some searching, reading and learning.