Related
CakePHP 3.7
I'm trying to generate a query which uses a WHERE...OR pattern. The equivalent in MySQL - which executes and gives the results I want is:
SELECT * FROM groups Groups WHERE (regulation_id = 1 AND label like '%labelling%') OR (id IN(89,1,8,232,228,276,268,294));
I've read the Advanced Conditions (https://book.cakephp.org/3.0/en/orm/query-builder.html#advanced-conditions) part of the documentation but can't generate that query.
Assume the Table class is Groups I have this:
$Groups = TableRegistry::getTableLocator()->get('Groups');
$groups_data = $Groups->find('all')->where(['regulation_id' => 1);
$groups_data = $groups_data->where(['label LIKE' => '%labelling%']);
This produces the first segment of the WHERE statement, i.e.
SELECT * FROM groups Groups WHERE (regulation_id = 1 AND label like '%labelling%')
However I can't see how to attach the OR condition, especially since orWhere() is deprecated.
So I've tried this - which is even given as an example in the docs:
$in_array = [89,1,8,232,228,276,268,294]; // ID's for IN condition
$groups_data = $groups_data->where(['OR' => ['id IN' => $in_array]]);
But this just appends an AND to the inside of my existing SQL:
SELECT * FROM groups Groups WHERE (regulation_id = 1 AND label like '%labelling%' AND id IN(89,1,8,232,228,276,268,294);
Which does not yield the correct results as the syntax isn't what's required to run this query.
How do you "move out" of the WHERE and append an OR condition like in the vanilla query?
I made several attempts using QueryExpression as per the docs, but all of these produced PHP Fatal Errors saying something to do with the Table class - I doubt this was on the right lines anyway.
"moving out" is a little tricky, you have to understand that internally the conditions are pushed into a \Cake\Database\Expression\QueryExpression object which by default uses AND to concatenate the statements, so whatever you push on to that, will be added using AND.
When you create OR statements, being it implicitly with the shown nested array syntax, or explicitly by using the expression builder, this creates a separate, self-contained expression, where its parts are being concatenated using OR, it will compile itself (and since there's only one condition, you don't see any OR's), and the result will be used in the parent expression, which in your case is the main/base expression object for the queries where clause.
Either pass the whole thing at once (being it via array syntax or expressions), eg:
$groups_data->where([
'OR' => [
'AND' => [
'regulation_id' => 1,
'label LIKE' => '%labelling%'
],
'id IN' => $in_array
]
]);
and of course you could build that array dynamically if required, or, if you for some reason need to use separate calls to where(), you could for example overwrite the conditions (third parameter of where()), and include the current ones where you need them:
$groups_data->where(
[
'OR' => [
$groups_data->clause('where'),
'id IN' => $in_array
]
],
[],
true
);
I know this issue is old but maybe someone is looking. Here is my solution:
protected $_hardValues= array(
'company_id' => $company_from_session;
);
function beforeFind($event=null, $query = null, $options = null, $primary = true){
$conds = [];
$columns = $this->getSchema()->columns();
foreach( $this->_hardValues as $field => $value){
if( !is_null($value) && in_array($field, $columns) ){
$conds[$this->_alias . '.' . $field] = $value;
}
}
if( empty( $conds)) return true;
$where = $query->clause('where'); //QueryExpression object;
if( empty( $where)){
$query->where($conds);
}else{
$where->add($conds);
}
}
As of CakePHP 4.x, the documented way of doing this is:
$query = $articles->find()
->where([
'author_id' => 3,
'OR' => [['view_count' => 2], ['view_count' => 3]],
]);
See documentation
I have table call users , like
id name amount created
1 a 100 6-16-2016
2 b 200 5-16-2016
I need max amount full row, I have tried below code but getting syntax error.
$user = $this->Users->find('all',[
'fields' => array('MAX(Users.amount) AS amount'),
]);
simplest way
$user = $this->Users->find('all',[
'fields' => array('amount' => 'MAX(Users.id)'),
]);
using select instead of an options array
$user = $this->Users->find()
->select(['amount' => 'MAX(Users.id)']);
making use of cake SQL functions
$query = $this->Users->find();
$user = $query
->select(['amount' => $query->func()->max('Users.id')]);
the above three all give the same results
if you want to have a single record you have to call ->first() on the query object:
$user = $user->first();
$amount = $user->amount;
Simplest method using CakePHP 3:
$this->Model->find('all')->select('amount')->hydrate(false)->max('amount')
Will result in an array containing the maximum amount in that table column.
Is is better to disable hydration before getting the results, to get the results as an array, instead of entity. Below is a complete example of how to get the results in an array, with distinct steps:
//create query
$query = $this->Users->find('all',[
'fields' => array('amount' => 'MAX(Users.id)'),
]);
$query->enableHydration(false); // Results as arrays instead of entities
$results = $query->all(); // Get ResultSet that contains array data.
$maxAmount = $results->toList(); // Once we have a result set we can get all the rows
Inside a cell I need to access the TreeOptions model.
So I've wrote this :
$this->loadModel( 'TreeOptions' );
$i = $this->TreeOptions->find( 'all' );
But when I do the foreach like this :
foreach( $i as $row )
debug( $row->description );
It only returns the last record of the result.
The only way I've found to make it work as desired is adding the limit clause :
$i = $this->TreeOptions->find( 'all', [ 'limit' => 200 ] );
And then, I can get the whole set of records.
What am I missing ?
Thanks.
Regards.
In your first snippet, the variable $i, is a state where the query has not yet run. See the excerpt from CakePHP 3 Cookbook: Retrieving Data & Results — Using Finders to Load Data:
// Find all the articles.
// At this point the query has not run.
$query = $articles->find('all');
// Iteration will execute the query.
foreach ($query as $row) {
}
// Calling all() will execute the query
// and return the result set.
$results = $query->all();
// Once we have a result set we can get all the rows
$data = $results->toArray();
// Converting the query to an array will execute it.
$results = $query->toArray();
I want to know item count with DynamoDB querying.
I can querying for DynamoDB, but I only want to know 'total count of item'.
For example, 'SELECT COUNT(*) FROM ... WHERE ...' in MySQL
$result = $aws->query(array(
'TableName' => 'game_table',
'IndexName' => 'week-point-index',
'KeyConditions' => array(
'week' => array(
'ComparisonOperator' => 'EQ',
'AttributeValueList' => array(
array(Type::STRING => $week)
)
),
'point' => array(
'ComparisonOperator' => 'GE',
'AttributeValueList' => array(
array(Type::NUMBER => $my_point)
)
)
),
));
echo Count($result['Items']);
this code gets the all users data higher than my point.
If count of $result is 100,000, $result is too much big.
And it would exceed the limits of the query size.
I need help.
With the aws dynamodb cli you can get it via scan as follows:
aws dynamodb scan --table-name <TABLE_NAME> --select "COUNT"
The response will look similar to this:
{
"Count": 123,
"ScannedCount": 123,
"ConsumedCapacity": null
}
notice that this information is in real time in contrast to the describe-table api
You can use the Select parameter and use COUNT in the request. It "returns the number of matching items, rather than the matching items themselves". Important, as brought up by Saumitra R. Bhave in a comment, "If the size of the Query result set is larger than 1 MB, then ScannedCount and Count will represent only a partial count of the total items. You will need to perform multiple Query operations in order to retrieve all of the results".
I'm Not familiar with PHP but here is how you could use it with Java. And then instead of using Count (which I am guessing is a function in PHP) on the 'Items' you can use the Count value from the response - $result['Count']:
final String week = "whatever";
final Integer myPoint = 1337;
Condition weekCondition = new Condition()
.withComparisonOperator(ComparisonOperator.EQ)
.withAttributeValueList(new AttributeValue().withS(week));
Condition myPointCondition = new Condition()
.withComparisonOperator(ComparisonOperator.GE)
.withAttributeValueList(new AttributeValue().withN(myPoint.toString()))
Map<String, Condition> keyConditions = new HashMap<>();
keyConditions.put("week", weekCondition);
keyConditions.put("point", myPointCondition);
QueryRequest request = new QueryRequest("game_table");
request.setIndexName("week-point-index");
request.setSelect(Select.COUNT);
request.setKeyConditions(keyConditions);
QueryResult result = dynamoDBClient.query(request);
Integer count = result.getCount();
If you don't need to emulate the WHERE clause, you can use a DescribeTable request and use the resulting item count to get an estimate.
The number of items in the specified table. DynamoDB updates this value approximately every six hours. Recent changes might not be reflected in this value.
Also, an important note from the documentation as noted by Saumitra R. Bhave in the comments on this answer:
If the size of the Query result set is larger than 1 MB, ScannedCount and Count represent only a partial count of the total items. You need to perform multiple Query operations to retrieve all the results (see Paginating Table Query Results).
Can be seen from UI as well.
Go to overview tab on table, you will see item count. Hope it helps someone.
I'm too late here but like to extend Daniel's answer about using aws cli to include filter expression.
Running
aws dynamodb scan \
--table-name <tableName> \
--filter-expression "#v = :num" \
--expression-attribute-names '{"#v": "fieldName"}' \
--expression-attribute-values '{":num": {"N": "123"}}' \
--select "COUNT"
would give
{
"Count": 2945,
"ScannedCount": 7874,
"ConsumedCapacity": null
}
That is, ScannedCount is total count and Count is the number of items which are filtered by given expression (fieldName=123).
Replace the table name and use the below query to get the data on your local environment:
aws dynamodb scan --table-name <TABLE_NAME> --select "COUNT" --endpoint-url http://localhost:8000
Replace the table name and remove the endpoint url to get the data on production environment
aws dynamodb scan --table-name <TABLE_NAME> --select "COUNT"
If you happen to reach here, and you are working with C#, here is the code:
var cancellationToken = new CancellationToken();
var request = new ScanRequest("TableName") {Select = Select.COUNT};
var result = context.Client.ScanAsync(request, cancellationToken).Result;
totalCount = result.Count;
If anyone is looking for a straight forward NodeJS Lambda count solution:
const data = await dynamo.scan({ Select: "COUNT", TableName: "table" }).promise();
// data.Count -> number of elements in table.
I'm posting this answer for anyone using C# that wants a fully functional, well-tested answer that demonstrates using query instead of scan. In particular, this answer handles more than 1MB size of items to count.
public async Task<int> GetAvailableCount(string pool_type, string pool_key)
{
var queryRequest = new QueryRequest
{
TableName = PoolsDb.TableName,
ConsistentRead = true,
Select = Select.COUNT,
KeyConditionExpression = "pool_type_plus_pool_key = :type_plus_key",
ExpressionAttributeValues = new Dictionary<string, AttributeValue> {
{":type_plus_key", new AttributeValue { S = pool_type + pool_key }}
},
};
var t0 = DateTime.UtcNow;
var result = await Client.QueryAsync(queryRequest);
var count = result.Count;
var iter = 0;
while ( result.LastEvaluatedKey != null && result.LastEvaluatedKey.Values.Count > 0)
{
iter++;
var lastkey = result.LastEvaluatedKey.Values.ToList()[0].S;
_logger.LogDebug($"GetAvailableCount {pool_type}-{pool_key} iteration {iter} instance key {lastkey}");
queryRequest.ExclusiveStartKey = result.LastEvaluatedKey;
result = await Client.QueryAsync(queryRequest);
count += result.Count;
}
_logger.LogDebug($"GetAvailableCount {pool_type}-{pool_key} returned {count} after {iter} iterations in {(DateTime.UtcNow - t0).TotalMilliseconds} ms.");
return count;
}
}
DynamoDB now has a 'Get Live Item Count' button in the UI. Please note the production caveat if you have a large table that will consume read capacity.
In Scala:
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder
import com.amazonaws.services.dynamodbv2.document.DynamoDB
val client = AmazonDynamoDBClientBuilder.standard().build()
val dynamoDB = new DynamoDB(client)
val tableDescription = dynamoDB.getTable("table name").describe().getItemCount()
Similar to Java in PHP only set Select PARAMETER with value 'COUNT'
$result = $aws->query(array(
'TableName' => 'game_table',
'IndexName' => 'week-point-index',
'KeyConditions' => array(
'week' => array(
'ComparisonOperator' => 'EQ',
'AttributeValueList' => array(
array(Type::STRING => $week)
)
),
'point' => array(
'ComparisonOperator' => 'GE',
'AttributeValueList' => array(
array(Type::NUMBER => $my_point)
)
)
),
'Select' => 'COUNT'
));
and acces it just like this :
echo $result['Count'];
but as Saumitra mentioned above be careful with resultsets largers than 1 MB, in that case use LastEvaluatedKey til it returns null to get the last updated count value.
Adding some additional context to this question. In some circumstances it makes sense to Scan the table to obtain the live item count. However, if this is a frequent occurrence or if you have large tables then it can be expensive from both a cost and performance point of view. Below, I highlight 3 ways to gain the item count for your tables.
1. Scan
Using a Scan requires you to read every item in the table, this works well for one off queries but it is not scalable and can become quite expensive. Using Select: COUNT will prevent returning data, but you must still pay for reading the entire table.
Pros
Gets you the most recent item count ("live")
Is a simple API call
Can be run in parallel to reduce time
Cons
Reads the entire dataset
Slow performance
High cost
CLI example
aws dynamodb scan \
--table-name test \
--select COUNT
2. DescribeTable
DynamoDB DescribeTable API provides you with an estimated value for ItemCount which is updated approx. every 6 hours.
The number of items in the specified table. DynamoDB updates this value approximately every six hours. Recent changes might not be reflected in this value. Ref.
Calling this API gives you an instant response, however, the value of the ItemCount could be up to 6 hours stale. In certain situations this value may be adequate.
Pros
Instant response
No cost to retrieve ItemCount
Can be called frequently
Cons
Data could be stale by up to 6 hours.
CLI Example
aws dynamodb describe-table \
--table-name test \
--query Table.ItemCount
DescribeTable and CloudWatch
As previously mentioned DescribeTable updates your tables ItemCount approx. every 6 hours. We can obtain that value and plot it on a custom CloudWatch graph which allows you to monitor your tables ItemCount over time, providing you historical data.
Pros
Provides historical data
Infer how your ItemCount changes over time
Reasonably easy to implement
Cons
Data could be stale by up to 6 hours.
Implementation
Tracking DynamoDB Storage History with CloudWatch showcases how to automatically push the value of DescribeTable to CloudWatch periodically using EventBridge and Lambda, however, it is designed to push TableSizeBytes instead of ItemCount. Some small modifications to the Lambda will allow you to record ItemCount.
Change this line from TableSizeBytes to ItemCount
Remove line 18 to line 27
You could use dynamodb mapper query.
PaginatedQueryList<YourModel> list = DymamoDBMapper.query(YourModel.class, queryExpression);
int count = list.size();
it calls loadAllResults() that would lazily load next available result until allResultsLoaded.
Ref: https://docs.amazonaws.cn/en_us/amazondynamodb/latest/developerguide/DynamoDBMapper.Methods.html#DynamoDBMapper.Methods.query
This is how you would do it using the DynamoDBMapper (Kotlin syntax), example with no filters at all:
dynamoDBMapper.count(MyEntity::class.java, DynamoDBScanExpression())
$aws = new Aws\DynamoDb\DynamoDbClient([
'region' => 'us-west-2',
'version' => 'latest',
]);
$result = $aws->scan(array(
'TableName' => 'game_table',
'Count' => true
));
echo $result['Count'];
len(response['Items'])
will give you the count of the filtered rows
where,
fe = Key('entity').eq('tesla')
response = table.scan(FilterExpression=fe)
I used scan to get total count of the required tableName.Following is a Java code snippet for same
Long totalItemCount = 0;
do{
ScanRequest req = new ScanRequest();
req.setTableName(tableName);
if(result != null){
req.setExclusiveStartKey(result.getLastEvaluatedKey());
}
result = client.scan(req);
totalItemCount += result.getItems().size();
} while(result.getLastEvaluatedKey() != null);
System.out.println("Result size: " + totalItemCount);
This is solution for AWS JavaScript SDK users, it is almost same for other languages.
Result.data.Count will give you what you are looking for
apigClient.getitemPost({}, body, {})
.then(function(result){
var dataoutput = result.data.Items[0];
console.log(result.data.Count);
}).catch( function(result){
});
I am still new to Cakephp I have 3 find queries to the same model and I have tried to combine them to make one. Finding it hard with the cake way of accessing the data. Especial that they have different find condition. Maybe it cant be done.
Cakephp version 2.3.5
// Count Total Members
$totalMemebers = $this->Member->find('count');
$this->set('totalMemebers', $totalMemebers);
// SUM total points gained for the last 7 days (positive values)
$this->Member->Point->virtualFields['gainedTotal'] = 'SUM(Point.points)';
$gainedTotal = $this->Member->Point->find('all', array(
'recursive'=> -1,
'fields' => array('gainedTotal'),
'conditions'=>array(
'Point.points >'=>0,
'Point.date >' => date('Y-m-d', strtotime("-1 weeks")))
)
);
$this->set('gainedTotal', $gainedTotal);
// SUM total points redeemed for the last 7 days (negative values)
$this->Member->Point->virtualFields['redeemedTotal'] = 'SUM(Point.points)';
$redeemedTotal = $this->Member->Point->find('all', array(
'recursive'=> -1,
'fields' => array('redeemedTotal'),
'conditions'=>array(
'Point.points <'=>0),
'Point.date >' => date('Y-m-d', strtotime("-1 weeks"))
)
);
$this->set('redeemedTotal', $redeemedTotal);
Because all queries use different conditions, it will not be possible to combine them to a single query. This is not a limitation of CakePHP, using regular SQL will not allow you to do so either, unless you're going to use sub queries or some calculated fields (SELECT CASE WHEN points > 0 AND date > .... THEN points ELSE 0 END CASE AS gainedPoint) but this will offer no benefits (performance wise)
Optimizations
To optimise (and clean up) your code, there are some improvements possible;
First of all, to get the value of a single field, you may use Model::field(). This will return just the plain value of a field, not an array containing the value.
For example:
$this->Member->Point->virtualFields['gainedTotal'] = 'SUM(Point.points)';
$gainedTotal = $this->Member->Point->field('gainedTotal', array(
'Point.points >'=>0,
'Point.date >' => date('Y-m-d', strtotime("-1 weeks"))
));
$this->set('gainedTotal', $gainedTotal);
Move data-related code to your model
This is something you should do in general. Move those queries to your Models. This will make your code cleaner and better to maintain. Also, your queries will be easier to re-use and finally, you will be able to Unit-test the queries.
class Point extends AppModel
{
// general stuff/properties here
/**
* Returns the SUM total points gained for
* the last 7 days (positive values)
*
* #return int Gained total for all members
*/
public function gainedTotal()
{
this->virtualFields['gainedTotal'] = 'SUM(points)';
$gainedTotal = $this->field('gainedTotal', array(
'points >'=>0,
'date >' => date('Y-m-d', strtotime("-1 weeks"))
));
// remove the virtualField to prevent it being used
// in subsequent queries
unset($this->virtualFields['gainedTotal']);
return (int)$gainedTotal;
}
}
And, inside your Controller, simply do this:
$this->set('gainedTotal', $this->Member->Point->gainedTotal());
This will clearly reduce the amount of code inside your Controller ('lean controllers, fat models'), which is always good practice.
Add relevant indexes to your database
If performance is important, you can massively improve the performance of these queries by adding the right indexes to your database, especially when calculating totals for a large number of records.
Describing this in detail is beyond the scope of this question, but here are some pointers; How MySQL uses indexes and CREATE INDEX SYNTAX