Related
Suppose I have the following struct:
mutable struct Car
load
locale
availability
odometer
end
And I have created an array:
fleet = Vector{Car}(undef, num_cars)
for i in 1:num_cars
a, b, c, d = rand(4)
fleet[i] = Car(a, b, c, d)
end
How can I find the smallest (with findmin or similar) or biggest (with?) value from, for example, the odometer of all the cars in my array?
Basically, I want to be able to use conditional statements with struct arrays, e.g.: For every element in my struct array, get the ones that a data satisfy a condition and, from those, get the minimum value of another data.
Thanks!
Finding the minimum value is pretty straightforward, using the minimum function, with a mapping argument:
julia> minimum(x->x.odometer, fleet)
0.08468003971220694
If you also want the index of the minimum, you can use the findmin function. Unfortunately, this does not, for some reason, support a function argument, so you have to create a temporary array, and apply findmin to that:
julia> findmin(getfield.(fleet, :odometer))
(0.08468003971220694, 1)
You can also use getproperty instead of getfield, they do the same thing for your struct, I'm not certain which is preferable. Probably, the most idiomatic solution would be to define accessor functions instead of using the field values directly. For example:
odometer(car::Car) = car.odometer
minimum(odometer, fleet)
findmin(odometer.(fleet))
For maximum values, use maximum and findmax.
Julia is about performance.
You should avoid using untyped structs hence your type definition should be:
mutable struct Car
load::Float64
locale::Float64
availability::Float64
odometer::Float64
end
The code for creating Vector can be shorter:
cars = [Car(rand(4)...) for _ in 1:5]
The most efficient way to find the index of the minimum element is:
julia> Base.isless(c1::Car,c2::Car) = c1.odometer < c2.odometer
julia> findmin(cs)
(Car(0.7623514815463603, 0.7523019237133661, 0.37422766075278413, 0.49830949323733464), 3)
I'm having a hard time in realizing how to use the repeated field rule.
for example, this is my .proto:
message Test
{
repeated float value = 1;
}
now, I'm initialize a new Test object:
Test test = test_init_zero()
Finally, I want to assign some values. For example:
float values[] = { 1.0, 2.2, 5.5, 7.13 }
My question is how can I assign them?
is it like
test.value = values
//or
test.value[0] = values[0] //... etc.
and then, how do I read them back?
This depends on how you define the repeated field inside the proto file. According to nanopb docs, you either just specify the repeated field like you did, and then use a callback function to handle each item separately during encoding/decoding, or you use nanopb-specific settings so have a fixed length array:
Strings, bytes and repeated fields of any type map to callback functions by default.
If there is a special option (nanopb).max_size specified in the .proto file, string maps to null-terminated char array and bytes map to a structure containing a char array and a size field.
If (nanopb).fixed_length is set to true and (nanopb).max_size is also set, then bytes map to an inline byte array of fixed size.
If there is a special option (nanopb).max_count specified on a repeated field, it maps to an array of whatever type is being repeated. Another field will be created for the actual number of entries stored.
For example, byte arrays need to use max_size:
required bytes data = 1 [(nanopb).max_size = 40, (nanopb).fixed_length = true];
And this would create the following field, when compiled using nanopb:
// byte arrays get a special treatment in nanopb
pb_byte_t data[40];
Or, for a float, you would use max_count according to rule 4.:
repeated float data = 1 [(nanopb).max_count = 40];
And then you'll get:
size_t data_count;
float data[40];
If you simply define a repeated field like you did, then nanopb will create a callback function:
// repeated float value = 1;
pb_callback_t value;
Which means you will have to provide your own function which will handle each incoming item:
yourobject.value.arg = &custom_args;
yourobject.value.funcs.decode = custom_function_for_decoding;
Say, we have a class Car
classdef Car < handle
properties
wheels = 4;
end
methods
function obj = Car()
end
end
end
We can create ten cars like so
cars = Car.empty();
for ii = 1:10
cars(end+1) = Car;
end
However, one can make an empty array of cars first
>> cars = Car.empty(0,10)
cars =
0x10 Car array with properties:
wheels
What I do not understand: If one now puts a single car in it, the array seems to shrink to a single element
>> cars(1) = Car
cars =
Car with properties:
wheels: 4
So, does allocating such an empty array make any sense? What are the use cases?
First of all, note that your question is not directly related to custom classes - you can do this with any MATLAB variable type. For example,
>> a = double.empty(0,10)
a =
Empty matrix: 0-by-10
So your questions are
What I do not understand: If one now puts a single car in it, the array seems to shrink to a single element
Well no, it has grown to a single element. I guess there is an ambiguity here, in that you might expect it to grow not to a 1x1 array but to a 1x10 array, with the other nine elements that weren't directly assigned being set to default (i.e. zero in the case of doubles or other numbers, or to a default element in the case of a custom class). However, I think the only thing to say there is that that's not what MATLAB does.
does allocating such an empty array make any sense? What are the use cases?
The use cases are very few in general, really just edge cases. The capability of having arrays of with zero-length dimensions is obviously required to be there for consistency, but an array of 0x10 is rarely much different in behaviour from an array of 0x0.
If I have an array that may vary in height but will always be 10 wide, I might predefine it as 0x10 rather than 0x0, just to leave myself a reminder in the code of that fact.
Also note that there are differences in behaviour between [] and double.empty(0,0). For example
>> a = rand(4,2)
a =
0.83625 0.19468
0.58508 0.12698
0.44332 0.8509
0.51858 0.3673
>> a(2,:) = []
a =
0.83625 0.19468
0.44332 0.8509
0.51858 0.3673
>> a(2,:) = double.empty(0,0)
Subscripted assignment dimension mismatch.
>> a(2,:) = double.empty(0,2)
Improper assignment with rectangular empty matrix.
This is because = [] is a special piece of MATLAB syntax that is used for deleting rows, rather than literally constructing the empty array and then assigning it. So there's another use case there, i.e. preventing accidental deletion of rows.
This is not a problem related to object programming but a normal behavior which is more general in Matlab.
For instance:
>> a = NaN(0,10);
>> a(1) = 5;
>> a
a =
5
When you define an array of size 0xn Matlab does not allocate any memory to it, since there is no element. So a call to empty(0,10) should not be considered as a pre-allocation.
When you define the first element of the array, you force a resizing, and during resizing Matlab always adopts the minimal possible size for the array.
The empty syntax is useful to define an array that will be filled by aggregation in a loop with the end+1 syntax, for instance:
a = NaN(0,2);
for i = 1:10
a(end+1,:) = [i i^2];
end
Best,
I have a table schema which includes an int array column, and a custom aggregate function which sums the array contents. In other words, given the following:
CREATE TABLE foo (stuff INT[]);
INSERT INTO foo VALUES ({ 1, 2, 3 });
INSERT INTO foo VALUES ({ 4, 5, 6 });
I need a "sum" function that would return { 5, 7, 9 }. The PL/pgSQL version, which works correctly, is as follows:
CREATE OR REPLACE FUNCTION array_add(array1 int[], array2 int[]) RETURNS int[] AS $$
DECLARE
result int[] := ARRAY[]::integer[];
l int;
BEGIN
---
--- First check if either input is NULL, and return the other if it is
---
IF array1 IS NULL OR array1 = '{}' THEN
RETURN array2;
ELSEIF array2 IS NULL OR array2 = '{}' THEN
RETURN array1;
END IF;
l := array_upper(array2, 1);
SELECT array_agg(array1[i] + array2[i]) FROM generate_series(1, l) i INTO result;
RETURN result;
END;
$$ LANGUAGE plpgsql;
Coupled with:
CREATE AGGREGATE sum (int[])
(
sfunc = array_add,
stype = int[]
);
With a data set of about 150,000 rows, SELECT SUM(stuff) takes over 15 seconds to complete.
I then re-wrote this function in C, as follows:
#include <postgres.h>
#include <fmgr.h>
#include <utils/array.h>
Datum array_add(PG_FUNCTION_ARGS);
PG_FUNCTION_INFO_V1(array_add);
/**
* Returns the sum of two int arrays.
*/
Datum
array_add(PG_FUNCTION_ARGS)
{
// The formal PostgreSQL array objects:
ArrayType *array1, *array2;
// The array element types (should always be INT4OID):
Oid arrayElementType1, arrayElementType2;
// The array element type widths (should always be 4):
int16 arrayElementTypeWidth1, arrayElementTypeWidth2;
// The array element type "is passed by value" flags (not used, should always be true):
bool arrayElementTypeByValue1, arrayElementTypeByValue2;
// The array element type alignment codes (not used):
char arrayElementTypeAlignmentCode1, arrayElementTypeAlignmentCode2;
// The array contents, as PostgreSQL "datum" objects:
Datum *arrayContent1, *arrayContent2;
// List of "is null" flags for the array contents:
bool *arrayNullFlags1, *arrayNullFlags2;
// The size of each array:
int arrayLength1, arrayLength2;
Datum* sumContent;
int i;
ArrayType* resultArray;
// Extract the PostgreSQL arrays from the parameters passed to this function call.
array1 = PG_GETARG_ARRAYTYPE_P(0);
array2 = PG_GETARG_ARRAYTYPE_P(1);
// Determine the array element types.
arrayElementType1 = ARR_ELEMTYPE(array1);
get_typlenbyvalalign(arrayElementType1, &arrayElementTypeWidth1, &arrayElementTypeByValue1, &arrayElementTypeAlignmentCode1);
arrayElementType2 = ARR_ELEMTYPE(array2);
get_typlenbyvalalign(arrayElementType2, &arrayElementTypeWidth2, &arrayElementTypeByValue2, &arrayElementTypeAlignmentCode2);
// Extract the array contents (as Datum objects).
deconstruct_array(array1, arrayElementType1, arrayElementTypeWidth1, arrayElementTypeByValue1, arrayElementTypeAlignmentCode1,
&arrayContent1, &arrayNullFlags1, &arrayLength1);
deconstruct_array(array2, arrayElementType2, arrayElementTypeWidth2, arrayElementTypeByValue2, arrayElementTypeAlignmentCode2,
&arrayContent2, &arrayNullFlags2, &arrayLength2);
// Create a new array of sum results (as Datum objects).
sumContent = palloc(sizeof(Datum) * arrayLength1);
// Generate the sums.
for (i = 0; i < arrayLength1; i++)
{
sumContent[i] = arrayContent1[i] + arrayContent2[i];
}
// Wrap the sums in a new PostgreSQL array object.
resultArray = construct_array(sumContent, arrayLength1, arrayElementType1, arrayElementTypeWidth1, arrayElementTypeByValue1, arrayElementTypeAlignmentCode1);
// Return the final PostgreSQL array object.
PG_RETURN_ARRAYTYPE_P(resultArray);
}
This version takes only 800 ms to complete, which is.... much better.
(Converted to a stand-alone extension here: https://github.com/ringerc/scrapcode/tree/master/postgresql/array_sum)
My question is, why is the C version so much faster? I expected an improvement, but 20x seems a bit much. What's going on? Is there something inherently slow about accessing arrays in PL/pgSQL?
I'm running PostgreSQL 9.0.2, on Fedora Core 8 64-bit. The machine is a High-Memory Quadruple Extra-Large EC2 instance.
Why?
why is the C version so much faster?
A PostgreSQL array is its self a pretty inefficient data structure. It can contain any data type and it's capable of being multi-dimensional, so lots of optimisations are just not possible. However, as you've seen it's possible to work with the same array much faster in C.
That's because array access in C can avoid a lot of the repeated work involved in PL/PgSQL array access. Just take a look at src/backend/utils/adt/arrayfuncs.c, array_ref. Now look at how it's invoked from src/backend/executor/execQual.c in ExecEvalArrayRef. Which runs for each individual array access from PL/PgSQL, as you can see by attaching gdb to the pid found from select pg_backend_pid(), setting a breakpoint at ExecEvalArrayRef, continuing, and running your function.
More importantly, in PL/PgSQL every statement you execute is run through the query executor machinery. This makes small, cheap statements fairly slow even allowing for the fact that they're pre-prepared. Something like:
a := b + c
is actually executed by PL/PgSQL more like:
SELECT b + c INTO a;
You can observe this if you turn debug levels high enough, attach a debugger and break at a suitable point, or use the auto_explain module with nested statement analysis. To give you an idea of how much overhead this imposes when you're running lots of tiny simple statements (like array accesses), take a look at this example backtrace and my notes on it.
There is also a significant start-up overhead to each PL/PgSQL function invocation. It isn't huge, but it's enough to add up when it's being used as an aggregate.
A faster approach in C
In your case I would probably do it in C, as you have done, but I'd avoid copying the array when called as an aggregate. You can check for whether it's being invoked in aggregate context:
if (AggCheckCallContext(fcinfo, NULL))
and if so, use the original value as a mutable placeholder, modifying it then returning it instead of allocating a new one. I'll write a demo to verify that this is possible with arrays shortly... (update) or not-so-shortly, I forgot how absolute horrible working with PostgreSQL arrays in C is. Here we go:
// append to contrib/intarray/_int_op.c
PG_FUNCTION_INFO_V1(add_intarray_cols);
Datum add_intarray_cols(PG_FUNCTION_ARGS);
Datum
add_intarray_cols(PG_FUNCTION_ARGS)
{
ArrayType *a,
*b;
int i, n;
int *da,
*db;
if (PG_ARGISNULL(1))
ereport(ERROR, (errmsg("Second operand must be non-null")));
b = PG_GETARG_ARRAYTYPE_P(1);
CHECKARRVALID(b);
if (AggCheckCallContext(fcinfo, NULL))
{
// Called in aggregate context...
if (PG_ARGISNULL(0))
// ... for the first time in a run, so the state in the 1st
// argument is null. Create a state-holder array by copying the
// second input array and return it.
PG_RETURN_POINTER(copy_intArrayType(b));
else
// ... for a later invocation in the same run, so we'll modify
// the state array directly.
a = PG_GETARG_ARRAYTYPE_P(0);
}
else
{
// Not in aggregate context
if (PG_ARGISNULL(0))
ereport(ERROR, (errmsg("First operand must be non-null")));
// Copy 'a' for our result. We'll then add 'b' to it.
a = PG_GETARG_ARRAYTYPE_P_COPY(0);
CHECKARRVALID(a);
}
// This requirement could probably be lifted pretty easily:
if (ARR_NDIM(a) != 1 || ARR_NDIM(b) != 1)
ereport(ERROR, (errmsg("One-dimesional arrays are required")));
// ... as could this by assuming the un-even ends are zero, but it'd be a
// little ickier.
n = (ARR_DIMS(a))[0];
if (n != (ARR_DIMS(b))[0])
ereport(ERROR, (errmsg("Arrays are of different lengths")));
da = ARRPTR(a);
db = ARRPTR(b);
for (i = 0; i < n; i++)
{
// Fails to check for integer overflow. You should add that.
*da = *da + *db;
da++;
db++;
}
PG_RETURN_POINTER(a);
}
and append this to contrib/intarray/intarray--1.0.sql:
CREATE FUNCTION add_intarray_cols(_int4, _int4) RETURNS _int4
AS 'MODULE_PATHNAME'
LANGUAGE C IMMUTABLE;
CREATE AGGREGATE sum_intarray_cols(_int4) (sfunc = add_intarray_cols, stype=_int4);
(more correctly you'd create intarray--1.1.sql and intarray--1.0--1.1.sql and update intarray.control. This is just a quick hack.)
Use:
make USE_PGXS=1
make USE_PGXS=1 install
to compile and install.
Now DROP EXTENSION intarray; (if you already have it) and CREATE EXTENSION intarray;.
You'll now have the aggregate function sum_intarray_cols available to you (like your sum(int4[]), as well as the two-operand add_intarray_cols (like your array_add).
By specializing in integer arrays a whole bunch of complexity goes away. A bunch of copying is avoided in the aggregate case, since we can safely modify the "state" array (the first argument) in-place. To keep things consistent, in the case of non-aggregate invocation we get a copy of the first argument so we can still work with it in-place and return it.
This approach could be generalised to support any data type by using the fmgr cache to look up the add function for the type(s) of interest, etc. I'm not particularly interested in doing that, so if you need it (say, to sum columns of NUMERIC arrays) then ... have fun.
Similarly, if you need to handle dissimilar array lengths, you can probably work out what to do from the above.
PL/pgSQL excels as server-side glue for SQL elements. Procedural elements and lots of assignments are not among its strengths. Assignments, tests or looping are comparatively expensive and only warranted if they help to take shortcuts one could not achieve with just SQL. The same logic implemented in C will always be faster, but you seem to be well aware of that ...
Typically, a pure SQL solution is faster. Compare this simple, equivalent solution in your test setup:
SELECT array_agg(a + b)
FROM (
SELECT unnest('{1, 2, 3 }'::int[]) AS a
, unnest('{4, 5, 6 }'::int[]) AS b
) x;
You can wrap it into a simple SQL function. Or, for best performance, integrate it into your big query directly:
SELECT tbl_id, array_agg(a + b)
FROM (
SELECT tbl_id
, unnest(array1) AS a
, unnest(array2) AS b
FROM tbl
ORDER BY tbl_id
) x
GROUP BY tbl_id;
Set-returning functions only work in parallel in a SELECT if the number of returned rows is identical. I.e.: works only for arrays of equal length. The behavior has eventually been sanitized in Postgres 10. See:
What is the expected behaviour for multiple set-returning functions in SELECT clause?
It's generally best to test with a current version of Postgres. At least update to the latest point release (9.0.15 at the time of writing). Might be part of the explanation for the big difference in performance.
Postgres 9.4
Performance improvements for array handling.
There is a cleaner solution for unnesting in parallel now:
Unnest multiple arrays in parallel
I need to write a program to find the mode. Or the most occurrence of an integer or integers.
So,
1,2,3,4,1,10,4,23,12,4,1 would have mode of 1 and 4.
I'm not really sure what kind of algorithm i should use. I'm having a hard time trying to think of something that would work.
I was thinking of a frequency table of some sort maybe where i could go through array and then go through and create a linked list maybe. If the linked doesn't contain that value add it to the linked, if it does then add 1 to the value.
So if i had the same thing from above. loop through
1,2,3,4,1,10,4,23,12,4,1
Then list is empty so add node with number = 1 and value = 1.
2 doesnt exist so add node with number = 2 and value = 1 and so on.
Get to the 1 and 1 already exists so value = 2 now.
I would have to loop through the array and then loop through linked list everytime to find that value.
Once i am done then go through the linked list and create a new linked list that will hold the modes. So i set the head to the first element which is 1. Then i go through the linked list that contains the occurences and compare the values. If the occurences of the current node is > the current highest then i set the head to this node. If its = to the highest then i add the node to the mode linked list.
Once i am done i loop through the mode list and print the values.
Not sure if this would work. Does anyone see anything wrong with this? Is there an easier way to do this? I was thinking a hash table too, but not really sure how to do that in C.
Thanks.
If you can keep the entire list of integers in memory, you could sort the list first, which will make repeated values adjacent to each other. Then you can do a single pass over the sorted list to look for the mode. That way, you only need to keep track of the best candidate(s) for the mode seen up until now, along with how many times the current value has been seen so far.
The algorithm you have is fine for a homework assignment. There are all sorts of things you could do to optimise the code, such as:
use a binary tree for efficiency,
use an array of counts where the index is the number (assuming the number range is limited).
But I think you'll find they're not necessary in this case. For homework, the intent is just to show that you understand how to program, not that you know all sorts of tricks for wringing out the last ounce of performance. Your educator will be looking far more for readable, structured, code than tricky optimisations.
I'll describe below what I'd do. You're obviously free to use my advice as much or as little as you wish, depending on how much satisfaction you want to gain at doing it yourself. I'll provide pseudo-code only, which is my standard practice for homework questions.
I would start with a structure holding a number, a count and next pointer (for your linked list) and the global pointer to the first one:
typedef struct sElement {
int number;
int count;
struct sElement *next;
} tElement;
tElement first = NULL;
Then create some functions for creating and using the list:
tElement *incrementElement (int number);
tElement *getMaxCountElement (void);
tElement *getNextMatching (tElement *ptr, int count);
Those functions will, respectively:
Increment the count for an element (or create it and set count to 1).
Scan all the elements returning the maximum count.
Get the next element pointer matching the count, starting at a given point, or NULL if no more.
The pseudo-code for each:
def incrementElement (number):
# Find matching number in list or NULL.
set ptr to first
while ptr is not NULL:
if ptr->number is equal to number:
return ptr
set ptr to ptr->next
# If not found, add one at start with zero count.
if ptr is NULL:
set ptr to newly allocated element
set ptr->number to number
set ptr->count to 0
set ptr->next to first
set first to ptr
# Increment count.
set ptr->count to ptr->count + 1
def getMaxCountElement (number):
# List empty, no mode.
if first is NULL:
return NULL
# Assume first element is mode to start with.
set retptr to first
# Process all other elements.
set ptr to first->next
while ptr is not NULL:
# Save new mode if you find one.
if ptr->count is greater than retptr->count:
set retptr to ptr
set ptr to ptr->next
# Return actual mode element pointer.
return retptr
def getNextMatching (ptr, number):
# Process all elements.
while ptr is not NULL:
# If match on count, return it.
if ptr->number is equal to number:
return ptr
set ptr to ptr->next
# Went through whole list with no match, return NULL.
return NULL
Then your main program becomes:
# Process all the numbers, adding to (or incrementing in) list .
for each n in numbers to process:
incrementElement (n)
# Get the mode quantity, only look for modes if list was non-empty.
maxElem = getMaxCountElement ()
if maxElem is not NULL:
# Find the first one, whil exists, print and find the next one.
ptr = getNextMatching (first, maxElem->count)
while ptr is not NULL:
print ptr->number
ptr = getNextMatching (ptr->next, maxElem->count)
If the range of numbers is known in advance, and is a reasonable number, you can allocate a sufficiently large array for the counters and just do count[i] += 1.
If the range of numbers is not known in advance, or is too large for the naive use of an array, you could instead maintain a binary tree of values to maintain your counters. This will give you far less searching than a linked list would. Either way you'd have to traverse the array or tree and build an ordering of highest to lowest counts. Again I'd recommend a tree for that, but your list solution could work as well.
Another interesting option could be the use of a priority queue for your extraction phase. Once you have your list of counters completed, walk your tree and insert each value at a priority equal to its count. Then you just pull values from the priority queue until the count goes down.
I would go for a simple hash table based solution.
A structure for hash table containing a number and corresponding frequency. Plus a pointer to the next element for chaining in the hash bucket.
struct ItemFreq {
struct ItemFreq * next_;
int number_;
int frequency_;
};
The processing starts with
max_freq_so_far = 0;
It goes through the list of numbers. For each number, the hash table is looked up for a ItemFreq element x such that x.number_ == number.
If no such x is found, then a ItemFreq element is created as { number_ = number, frequency_ = 1} and inserted into the hash table.
If some x was found then its frequency_ is incremented.
If frequency_ > max_freq_so_far then max_freq_so_far = frequency
Once traversing through the list of numbers of complete, we traverse through the hash table and print the ItemFreq items whose frequency_ == max_freq_so_far
The complexity of the algorithm is O(N) where N is the number of items in the input list.
For a simple and elegant construction of hash table, see section 6.6 of K&R (The C Programming Language).
This response is a sample for the idea of Paul Kuliniewicz:
int CompInt(const void* ptr1, const void* ptr2) {
const int a = *(int*)ptr1;
const int b = *(int*)ptr2;
if (a < b) return -1;
if (a > b) return +1;
return 0;
}
// This function leave the modes in output and return the number
// of modes in output. The output pointer should be available to
// hold at least n integers.
int GetModes(const int* v, int n, int* output) {
// Sort the data and initialize the best result.
qsort(v, v + n, CompInt);
int outputSize = 0;
// Loop through elements while there are not exhausted.
// (look there is no ++i after each iteration).
for (int i = 0; i < n;) {
// This is the begin of the new group.
const int begin = i;
// Move the pointer until there are no more equal elements.
for (; i < n && v[i] == v[begin]; ++i);
// This is one-past the last element in the current group.
const int end = i;
// Update the best mode found until now.
if (end - begin > best) {
best = end - begin;
outputSize = 0;
}
if (end - begin == best)
output[outputSize++] = v[begin];
}
return outputSize;
}