I'm currently writing a survey style app that has some models:
Assessments
Recommendations
An assessment has many recommendations and is joined by the assessments_recommendations table, and I store data in the join table which are effectively answers.
One such answer is user impact. This field is an integer value between 0 and 10 and is different for each assessment->recommendation.
What I want to be able to do is send a count and categorisation of that field into a view so that I can chart it using chart.js. For example, I would like:
0-3 = Low
4-7 = Moderate
8-10 = High
As far as I can tell, I need to pass something along the lines of:
$counts [4, 6, 8]
$legend ['Low', 'Moderate', 'High']
So in the above example, 4 assessment->recommendations are classed as low, 6 are moderate, 8 are high.
Does anyone have any advice on how to do this in my controller? I've been reading up on collections but I'm not sure that is the correct way of doing this. I had issues as my data is stored in the _joinData part of the array and I couldn't figure out how to access it.
Any advice would be much appreciated.
Thanks!
Edit 1
This is the controller code I have been playing around with to get an initial collection working. The col variable passes through nicely and I can see this in the view. I get 'unknown getter' error on the join data line though. It isn't entirely clear to me from the docs what variables are what in the example in the book, so I've been very much trying every combo I can think of to try and get it to work before even trying to move on with the complete scenario.
public function view($id = null)
{
$this->loadModel ('Assessments');
$this->loadModel ('Recommendations');
$query = $this->Assessments->find('all', [
'conditions' => ['Assessments.id =' => $id],
'contain' => ['Recommendations']
]);
$assessment = $query->first();
$collection = new Collection($assessment->ToArray());
$countHigh = $collection->countBy(function ($assessment) {
return $assessment->_joinData->_user_impact > 7 ? 'High' : 'Error';
});
$this->set('assessments', $assessment);
$this->set('col', $collection);
$this->set('user_impact_high', $countHigh);
}
Edit 2
So I'm testing this some more. Trying to simplify it. Even using the groupBy function in it's simple form is failing. The code below generates the following error.
public function view($id = null)
{
$this->loadModel ('Assessments');
$this->loadModel ('Recommendations');
$query = $this->Assessments->find('all', [
'conditions' => ['Assessments.id =' => $id],
'contain' => ['Recommendations']
]);
$assessment = $query->first();
$collection = new Collection($assessment->ToArray());
$test = $collection->groupBy('client_id');
$this->set('assessments', $assessment);
$this->set('col', $collection);
$this->set('test', $test);
}
Error:
Cannot use object of type Cake\I18n\FrozenTime as array
I am using CakePHp 2.5 and need to remove some results from the query:
$this->paginate = $paginate;
$results = $this->paginate('services');
foreach($results as $key=>$data )
{
if( empty( $dado['services']['service_id'] ) )
{
unset($results[$key]);
}
The result count will keep the original count.
Is there a way to subtract, the paginate query count, when i do unset the query results?
Looking at the Paginator class can not see if there is a property with the result count information.
Don't post-process the results of an sql call.
If you post-process the results of the sql call, unless the results fit in one page the count is always going to be inaccurate - because you'd be removing results from the current page, yet other results you would remove would still be present in the other pages - affecting the count.
Make paginate return what you want
Instead, just make the database give you the results you want, and none you don't:
$this->paginate['conditions'] = ['service_id IS NOT NULL'];
$results = $this->paginate('services');
If service id can be null or 0, you can account for that using a greater-than comparison:
$this->paginate['conditions'] = ['service_id >' => 0];
$results = $this->paginate('services');
In cakephp 2, I was able to use a virtualField for this, but seems impossible in 3. I have been struggling with this for two days without luck from the internet or the cakephp manual. I get no errors, but I do get a blank return.
My code in my controller looks like this:
if (isset($this->request->data['location']) && (isset($this->request->data['radius']))){
$radius = $this->request->data['radius'];
$location = $this->request->data['location'];
$address = $location; // Google HQ
$HttpSocket = new Client();
$geocode = $HttpSocket->get('http://maps.google.com/maps/api/geocode/json?address='.$address.'&sensor=false');
$geocode = $geocode->json;
if ($geocode['status'] == "OK"){
$lat = $geocode['results'][0]['geometry']['location']['lat'];
$lon = $geocode['results'][0]['geometry']['location']['lng'];
$R = 6371; // earth's mean radius, km
// first-cut bounding box (in degrees)
$maxLat = $lat + rad2deg($radius/$R);
$minLat = $lat - rad2deg($radius/$R);
// compensate for degrees longitude getting smaller with increasing latitude
$maxLon = $lon + rad2deg($radius/$R/cos(deg2rad($lat)));
$minLon = $lon - rad2deg($radius/$R/cos(deg2rad($lat)));
$conditions[] = ['Venues.lat' => "'BETWEEN '".$minLat."' AND '".$maxLat."'"];
$conditions[] = ['Venues.lon' => "'BETWEEN :'".$minLon."' AND :'".$maxLon."'"];
}
$this->paginate =[
'limit' => 10,
'order' => ['Quads.date' => 'asc'],
'conditions' => $conditions,
'contain' => [
'Performers' => ['Genres'],
'Users' => ['Profiles'],
'Venues' => ['fields' => [
'name',
'id',
'verified',
'address1',
'city',
'zip_code'], 'States'=>['Countries']],
'Categories',
'Likes' => ['Users' => ['Profiles']]]];
$quads = $this->paginate();
Impossible is (nearly) nothing. The old virtual fields concept is gone, right, the new ORM is flexible enough so that this isn't necessary anymore.
Your problem is that you are defining the conditions the wrong way, what you are doing there by specifying key => value sets, is creating ordinary operator conditions, where the value is going to be escaped/casted according to the column type. In case you really don't receive any errors, I'd assume that the lat/lan columns are of a numeric type, so your BETWEEN ... strings do end up as numbers, and the conditions will look something like
Venus.lat = 0 AND Venus.lon = 0
Also note that you are creating a nested array, ie
[
['keyA' => 'value'],
['keyB' => 'value']
]
and while this works, you may run into further problems in case you're not aware of it, so you'd better stick with
[
'keyA' => 'value',
'keyB' => 'value'
]
unless there's actually a technical reason to use nested conditions.
tl;dr use expressions
That being said, you can use expressions to build the proper conditions, like
$conditions[] = $this->Quads->Venues
->query()->newExpr()->between('Venues.lat', $minLat, $maxLat);
$conditions[] = $this->Quads->Venues
->query()->newExpr()->between('Venues.lon', $minLon, $maxLon);
This will safely create proper conditions like
Venus.lat BETWEEN a AND b AND Venus.lon BETWEEN x AND Y
Note that it is advised to create the expressions via the table that holds the columns (VenuesTable in this case), as you'd otherwise have to manually specify the column type (see the fourth argument of QueryExpression::between()) in order for the correct casting/escaping to be applied!
See also
Cookbook > Database Access & ORM > Query Builder > Advanced Conditions
Cookbook > Database Access & ORM > Query Builder > Using SQL Functions
API > \Cake\Database\QueryExpression::between()
I want to know item count with DynamoDB querying.
I can querying for DynamoDB, but I only want to know 'total count of item'.
For example, 'SELECT COUNT(*) FROM ... WHERE ...' in MySQL
$result = $aws->query(array(
'TableName' => 'game_table',
'IndexName' => 'week-point-index',
'KeyConditions' => array(
'week' => array(
'ComparisonOperator' => 'EQ',
'AttributeValueList' => array(
array(Type::STRING => $week)
)
),
'point' => array(
'ComparisonOperator' => 'GE',
'AttributeValueList' => array(
array(Type::NUMBER => $my_point)
)
)
),
));
echo Count($result['Items']);
this code gets the all users data higher than my point.
If count of $result is 100,000, $result is too much big.
And it would exceed the limits of the query size.
I need help.
With the aws dynamodb cli you can get it via scan as follows:
aws dynamodb scan --table-name <TABLE_NAME> --select "COUNT"
The response will look similar to this:
{
"Count": 123,
"ScannedCount": 123,
"ConsumedCapacity": null
}
notice that this information is in real time in contrast to the describe-table api
You can use the Select parameter and use COUNT in the request. It "returns the number of matching items, rather than the matching items themselves". Important, as brought up by Saumitra R. Bhave in a comment, "If the size of the Query result set is larger than 1 MB, then ScannedCount and Count will represent only a partial count of the total items. You will need to perform multiple Query operations in order to retrieve all of the results".
I'm Not familiar with PHP but here is how you could use it with Java. And then instead of using Count (which I am guessing is a function in PHP) on the 'Items' you can use the Count value from the response - $result['Count']:
final String week = "whatever";
final Integer myPoint = 1337;
Condition weekCondition = new Condition()
.withComparisonOperator(ComparisonOperator.EQ)
.withAttributeValueList(new AttributeValue().withS(week));
Condition myPointCondition = new Condition()
.withComparisonOperator(ComparisonOperator.GE)
.withAttributeValueList(new AttributeValue().withN(myPoint.toString()))
Map<String, Condition> keyConditions = new HashMap<>();
keyConditions.put("week", weekCondition);
keyConditions.put("point", myPointCondition);
QueryRequest request = new QueryRequest("game_table");
request.setIndexName("week-point-index");
request.setSelect(Select.COUNT);
request.setKeyConditions(keyConditions);
QueryResult result = dynamoDBClient.query(request);
Integer count = result.getCount();
If you don't need to emulate the WHERE clause, you can use a DescribeTable request and use the resulting item count to get an estimate.
The number of items in the specified table. DynamoDB updates this value approximately every six hours. Recent changes might not be reflected in this value.
Also, an important note from the documentation as noted by Saumitra R. Bhave in the comments on this answer:
If the size of the Query result set is larger than 1 MB, ScannedCount and Count represent only a partial count of the total items. You need to perform multiple Query operations to retrieve all the results (see Paginating Table Query Results).
Can be seen from UI as well.
Go to overview tab on table, you will see item count. Hope it helps someone.
I'm too late here but like to extend Daniel's answer about using aws cli to include filter expression.
Running
aws dynamodb scan \
--table-name <tableName> \
--filter-expression "#v = :num" \
--expression-attribute-names '{"#v": "fieldName"}' \
--expression-attribute-values '{":num": {"N": "123"}}' \
--select "COUNT"
would give
{
"Count": 2945,
"ScannedCount": 7874,
"ConsumedCapacity": null
}
That is, ScannedCount is total count and Count is the number of items which are filtered by given expression (fieldName=123).
Replace the table name and use the below query to get the data on your local environment:
aws dynamodb scan --table-name <TABLE_NAME> --select "COUNT" --endpoint-url http://localhost:8000
Replace the table name and remove the endpoint url to get the data on production environment
aws dynamodb scan --table-name <TABLE_NAME> --select "COUNT"
If you happen to reach here, and you are working with C#, here is the code:
var cancellationToken = new CancellationToken();
var request = new ScanRequest("TableName") {Select = Select.COUNT};
var result = context.Client.ScanAsync(request, cancellationToken).Result;
totalCount = result.Count;
If anyone is looking for a straight forward NodeJS Lambda count solution:
const data = await dynamo.scan({ Select: "COUNT", TableName: "table" }).promise();
// data.Count -> number of elements in table.
I'm posting this answer for anyone using C# that wants a fully functional, well-tested answer that demonstrates using query instead of scan. In particular, this answer handles more than 1MB size of items to count.
public async Task<int> GetAvailableCount(string pool_type, string pool_key)
{
var queryRequest = new QueryRequest
{
TableName = PoolsDb.TableName,
ConsistentRead = true,
Select = Select.COUNT,
KeyConditionExpression = "pool_type_plus_pool_key = :type_plus_key",
ExpressionAttributeValues = new Dictionary<string, AttributeValue> {
{":type_plus_key", new AttributeValue { S = pool_type + pool_key }}
},
};
var t0 = DateTime.UtcNow;
var result = await Client.QueryAsync(queryRequest);
var count = result.Count;
var iter = 0;
while ( result.LastEvaluatedKey != null && result.LastEvaluatedKey.Values.Count > 0)
{
iter++;
var lastkey = result.LastEvaluatedKey.Values.ToList()[0].S;
_logger.LogDebug($"GetAvailableCount {pool_type}-{pool_key} iteration {iter} instance key {lastkey}");
queryRequest.ExclusiveStartKey = result.LastEvaluatedKey;
result = await Client.QueryAsync(queryRequest);
count += result.Count;
}
_logger.LogDebug($"GetAvailableCount {pool_type}-{pool_key} returned {count} after {iter} iterations in {(DateTime.UtcNow - t0).TotalMilliseconds} ms.");
return count;
}
}
DynamoDB now has a 'Get Live Item Count' button in the UI. Please note the production caveat if you have a large table that will consume read capacity.
In Scala:
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder
import com.amazonaws.services.dynamodbv2.document.DynamoDB
val client = AmazonDynamoDBClientBuilder.standard().build()
val dynamoDB = new DynamoDB(client)
val tableDescription = dynamoDB.getTable("table name").describe().getItemCount()
Similar to Java in PHP only set Select PARAMETER with value 'COUNT'
$result = $aws->query(array(
'TableName' => 'game_table',
'IndexName' => 'week-point-index',
'KeyConditions' => array(
'week' => array(
'ComparisonOperator' => 'EQ',
'AttributeValueList' => array(
array(Type::STRING => $week)
)
),
'point' => array(
'ComparisonOperator' => 'GE',
'AttributeValueList' => array(
array(Type::NUMBER => $my_point)
)
)
),
'Select' => 'COUNT'
));
and acces it just like this :
echo $result['Count'];
but as Saumitra mentioned above be careful with resultsets largers than 1 MB, in that case use LastEvaluatedKey til it returns null to get the last updated count value.
Adding some additional context to this question. In some circumstances it makes sense to Scan the table to obtain the live item count. However, if this is a frequent occurrence or if you have large tables then it can be expensive from both a cost and performance point of view. Below, I highlight 3 ways to gain the item count for your tables.
1. Scan
Using a Scan requires you to read every item in the table, this works well for one off queries but it is not scalable and can become quite expensive. Using Select: COUNT will prevent returning data, but you must still pay for reading the entire table.
Pros
Gets you the most recent item count ("live")
Is a simple API call
Can be run in parallel to reduce time
Cons
Reads the entire dataset
Slow performance
High cost
CLI example
aws dynamodb scan \
--table-name test \
--select COUNT
2. DescribeTable
DynamoDB DescribeTable API provides you with an estimated value for ItemCount which is updated approx. every 6 hours.
The number of items in the specified table. DynamoDB updates this value approximately every six hours. Recent changes might not be reflected in this value. Ref.
Calling this API gives you an instant response, however, the value of the ItemCount could be up to 6 hours stale. In certain situations this value may be adequate.
Pros
Instant response
No cost to retrieve ItemCount
Can be called frequently
Cons
Data could be stale by up to 6 hours.
CLI Example
aws dynamodb describe-table \
--table-name test \
--query Table.ItemCount
DescribeTable and CloudWatch
As previously mentioned DescribeTable updates your tables ItemCount approx. every 6 hours. We can obtain that value and plot it on a custom CloudWatch graph which allows you to monitor your tables ItemCount over time, providing you historical data.
Pros
Provides historical data
Infer how your ItemCount changes over time
Reasonably easy to implement
Cons
Data could be stale by up to 6 hours.
Implementation
Tracking DynamoDB Storage History with CloudWatch showcases how to automatically push the value of DescribeTable to CloudWatch periodically using EventBridge and Lambda, however, it is designed to push TableSizeBytes instead of ItemCount. Some small modifications to the Lambda will allow you to record ItemCount.
Change this line from TableSizeBytes to ItemCount
Remove line 18 to line 27
You could use dynamodb mapper query.
PaginatedQueryList<YourModel> list = DymamoDBMapper.query(YourModel.class, queryExpression);
int count = list.size();
it calls loadAllResults() that would lazily load next available result until allResultsLoaded.
Ref: https://docs.amazonaws.cn/en_us/amazondynamodb/latest/developerguide/DynamoDBMapper.Methods.html#DynamoDBMapper.Methods.query
This is how you would do it using the DynamoDBMapper (Kotlin syntax), example with no filters at all:
dynamoDBMapper.count(MyEntity::class.java, DynamoDBScanExpression())
$aws = new Aws\DynamoDb\DynamoDbClient([
'region' => 'us-west-2',
'version' => 'latest',
]);
$result = $aws->scan(array(
'TableName' => 'game_table',
'Count' => true
));
echo $result['Count'];
len(response['Items'])
will give you the count of the filtered rows
where,
fe = Key('entity').eq('tesla')
response = table.scan(FilterExpression=fe)
I used scan to get total count of the required tableName.Following is a Java code snippet for same
Long totalItemCount = 0;
do{
ScanRequest req = new ScanRequest();
req.setTableName(tableName);
if(result != null){
req.setExclusiveStartKey(result.getLastEvaluatedKey());
}
result = client.scan(req);
totalItemCount += result.getItems().size();
} while(result.getLastEvaluatedKey() != null);
System.out.println("Result size: " + totalItemCount);
This is solution for AWS JavaScript SDK users, it is almost same for other languages.
Result.data.Count will give you what you are looking for
apigClient.getitemPost({}, body, {})
.then(function(result){
var dataoutput = result.data.Items[0];
console.log(result.data.Count);
}).catch( function(result){
});
Can't make sort links work. (concrete or virtual fields).
Vitual fields for my sum() field on this action:
$this->Qca->virtualFields['comps'] = 'Sum(CASE WHEN Qca.qca_tipcode = 1 THEN 1 END)';
$this->Qca->virtualFields['production'] = 'Sum(qca_end - qca_start)';
$this->Qca->virtualFields['idle'] = 'Sum(Qca.qca_durend)';
My find(), works fine:
$hoursvalues = $this->Qca->find('all', array('conditions' => $conditions,
'fields' => array('Qca.dir_id', 'Qca.name', 'Sum(CASE WHEN Qca.qca_tipcode = 1 THEN 1 END) AS Qca__comps', 'Sum(qca_end - qca_start) as Qca__production', 'Sum(Qca.qca_durend) as Qca__idle'),
'group' => array('Qca.dir_id')
)
);
and then:
$this->paginate('Qca' );
$this->set('hoursvalues', $hoursvalues);
What extra settings does $this->paginate('Qca' ); needs? Please note I have all data I need via find().
What is it I'm missing that sorting does not work for either concrete or virtual fields?
Thansk a lot!
Carlos
Part of your problem is the following:
$this->paginate('Qca');
$this->set('hoursvalues', $hoursvalues);
$this->paginate() returns an array with sorted values. What you need to do is specify your extra settings in $this->paginate array and then
$this->set('hoursvalues', $this->paginate('Qca'));
For your other fields, making them virtualFields will make them easier to work with.