I'm not trying to replicate the usual question about C not being able to return arrays but to dig a bit more deeply into it.
We cannot do this:
char f(void)[8] {
char ret;
// ...fill...
return ret;
}
int main(int argc, char ** argv) {
char obj_a[10];
obj_a = f();
}
But we can do:
struct s { char arr[10]; };
struct s f(void) {
struct s ret;
// ...fill...
return ret;
}
int main(int argc, char ** argv) {
struct s obj_a;
obj_a = f();
}
So, I was skimming the ASM code generated by gcc -S and seems to be working with the stack, addressing -x(%rbp) as with any other C function return.
What is it with returning arrays directly? I mean, not in terms of optimization or computational complexity but in terms of the actual capability of doing so without the struct layer.
Extra data: I am using Linux and gcc on a x64 Intel.
First of all, yes, you can encapsulate an array in a structure, and then do anything you want with that structure (assign it, return it from a function, etc.).
Second of all, as you've discovered, the compiler has little difficulty emitting code to return (or assign) structures. So that's not the reason you can't return arrays, either.
The fundamental reason you cannot do this is that, bluntly stated, arrays are second-class data structures in C. All other data structures are first-class. What are the definitions of "first-class" and "second-class" in this sense? Simply that second-class types cannot be assigned.
(Your next question might be, "Other than arrays, are there any other second-class data types?", and I think the answer is "Not really, unless you count functions".)
Intimately tied up with the fact that you can't return (or assign) arrays is that there are no values of array type, either. There are objects (variables) of array type, but whenever you try to take the value of one, you immediately get a pointer to the array's first element. [Footnote: more formally, there are no rvalues of array type, although an object of array type can be thought of as an lvalue, albeit a non-assignable one.]
So, quite aside from the fact that you can't assign to an array, you can't even generate a value to try to assign. If you say
char a[10], b[10];
a = b;
it's as if you had written
a = &b[0];
So we've got an array on the left, but a pointer on the right, and we'd have a massive type mismatch even if arrays somehow were assignable. Similarly (from your example) if we try to write
a = f();
and somewhere inside the definition of function f() we have
char ret[10];
/* ... fill ... */
return ret;
it's as if that last line said
return &ret[0];
and, again, we have no array value to return and assign to a, merely a pointer.
(In the function call example, we've also got the very significant issue that ret is a local array, perilous to try to return in C. More on this point later.)
Now, part of your question is probably "Why is it this way?", and also "If you can't assign arrays, why can you assign structures containing arrays?"
What follows is my interpretation and my opinion, but it's consistent with what Dennis Ritchie describes in his paper The Development of the C Language.
The non-assignability of arrays arises from three facts:
C is intended to be syntactically and semantically close to the machine hardware. An elementary operation in C should compile down to one or a handful of machine instructions taking one or a handful of processor cycles.
Arrays have always been special, especially in the way they relate to pointers; this special relationship evolved from and was heavily influenced by the treatment of arrays in C's predecessor language B.
Structures weren't initially in C.
Due to point 2, it's impossible to assign arrays, and due to point 1, it shouldn't be possible anyway, because a single assignment operator = shouldn't expand to code that might take N thousand cycles to copy an N thousand element array.
And then we get to point 3, which really ends up leading to a contradiction.
When C got structures, they initially weren't fully first-class either, in that you couldn't assign or return them. But the reason you couldn't was simply that the first compiler wasn't smart enough, at first, to generate the code. There was no syntactic or semantic roadblock, as there was for arrays.
And the goal all along was for structures to be first-class, and this was achieved relatively early on. The compiler caught up, and learned how to emit code to assign and return structures, shortly around the time that the first edition of K&R was going to print.
But the question remains, if an elementary operation is supposed to compile down to a small number of instructions and cycles, why doesn't that argument disallow structure assignment? And the answer is, yes, it's a contradiction.
I believe (though this is more speculation on my part) that the thinking was something like this: "First-class types are good, second-class types are unfortunate. We're stuck with second-class status for arrays, but we can do better with structs. The no-expensive-code rule isn't really a rule, it's more of a guideline. Arrays will often be large, but structs will usually be small, tens or hundreds of bytes, so assigning them won't usually be too expensive."
So a consistent application of the no-expensive-code rule fell by the wayside. C has never been perfectly regular or consistent, anyway. (Nor, for that matter, are the vast majority of successful languages, human as well as artificial.)
With all of this said, it may be worth asking, "What if C did support assigning and returning arrays? How might that work?" And the answer will have to involve some way of turning off the default behavior of arrays in expressions, namely that they tend to turn into pointers to their first element.
Sometime back in the '90's, IIRC, there was a fairly well-thought-out proposal to do exactly this. I think it involved enclosing an array expression in [ ] or [[ ]] or something. Today I can't seem to find any mention of that proposal (though I'd be grateful if someone can provide a reference). At any rate, I believe we could extend C to allow array assignment by taking the following three steps:
Remove the prohibition of using an array on the left-hand side of an assignment operator.
Remove the prohibition of declaring array-valued functions. Going back to the original question, make char f(void)[8] { ... } legal.
(This is the biggie.) Have a way of mentioning an array in an expression and ending up with a true, assignable value (an rvalue) of array type. For the sake of argument I'll posit a new operator or pseudofunction called arrayval( ... ).
[Side note: Today we have a "key definition" of array/pointer correspondence, namely that:
A reference to an object of array type which appears in an expression decays (with three exceptions) into a pointer to its first element.
The three exceptions are when the array is the operand of a sizeof operator, or a & operator, or is a string literal initializer for a character array. Under the hypothetical modifications I'm discussing here, there would be a fourth exception, namely when the array was an operand of this new arrayval operator.]
Anyway, with these modifications in place, we could write things like
char a[8], b[8] = "Hello";
a = arrayval(b);
(Obviously we would also have to decide what to do if a and b were not the same size.)
Given the function prototype
char f(void)[8];
we could also do
a = f();
Let's look at f's hypothetical definition. We might have something like
char f(void)[8] {
char ret[8];
/* ... fill ... */
return arrayval(ret);
}
Note that (with the exception of the hypothetical new arrayval() operator) this is just about what Dario Rodriguez originally posted. Also note that — in the hypothetical world where array assignment was legal, and something like arrayval() existed — this would actually work! In particular, it would not suffer the problem of returning a soon-to-be-invalid pointer to the local array ret. It would return a copy of the array, so there would be no problem at all — it would be just about perfectly analogous to the obviously-legal
int g(void) {
int ret;
/* ... compute ... */
return ret;
}
Finally, returning to the side question of "Are there any other second-class types?", I think it's more than a coincidence that functions, like arrays, automatically have their address taken when they are not being used as themselves (that is, as functions or arrays), and that there are similarly no rvalues of function type. But this is mostly an idle musing, because I don't think I've ever heard functions referred to as "second-class" types in C. (Perhaps they have, and I've forgotten.)
Footnote: Because the compiler is willing to assign structures, and typically knows how to emit efficient code for doing so, it used to be a somewhat popular trick to co-opt the compiler's struct-copying machinery in order to copy arbitrary bytes from point a to point b. In particular, you could write this somewhat strange-looking macro:
#define MEMCPY(b, a, n) (*(struct foo { char x[n]; } *)(b) = \
*(struct foo *)(a))
that behaved more or less exactly like an optimized in-line version of memcpy(). (And in fact, this trick still compiles and works under modern compilers today.)
What is it with returning arrays directly? I mean, not in terms of optimization or computational complexity but in terms of the actual capability of doing so without the struct layer.
It has nothing to do with capability per se. Other languages do provide the ability to return arrays, and you already know that in C you can return a struct with an array member. On the other hand, yet other languages have the same limitation that C does, and even more so. Java, for instance, cannot return arrays, nor indeed objects of any type, from methods. It can return only primitives and references to objects.
No, it is simply a question of language design. As with most other things to do with arrays, the design points here revolve around C's provision that expressions of array type are automatically converted to pointers in almost all contexts. The value provided in a return statement is no exception, so C has no way of even expressing the return of an array itself. A different choice could have been made, but it simply wasn't.
For arrays to be first-class objects, you would expect at least to be able to assign them. But that requires knowledge of the size, and the C type system is not powerful enough to attach sizes to any types. C++ could do it, but doesn't due to legacy concerns—it has references to arrays of particular size (typedef char (&some_chars)[32]), but plain arrays are still implicitly converted to pointers as in C. C++ has std::array instead, which is basically the aforementioned array-within-struct plus some syntactic sugar.
Bounty hunting.
The authors of C did not aspire to be language or type system designers. They were tool designers. C was a tool to make system programming easier.
ref: B Kernighan on Pascal Ritchie on C
There was no compelling case for C to do anything unexpected; especially as UNIX and C were ushering in the era of least surprise. Copying arrays, and making complex syntax to do so when it was the metaphorical equivalent of having a setting to burn the toast did not fit the C model.
Everything in C, the language, is effectively constant time, constant size. C, the standard, seems preoccupied with doing away with this core feature which made C so popular; so expect the, uh, standard C/2023.feb07 to feature a punctuation nightmare that enables arrays as r-values.
The decision of the C authors makes eminent sense if you view the programming world pragmatically. If you view it as a pulpit for treasured beliefs, then get onboard for C/2023.feb07 before C/2023.feb08 nullifies it.
I'm afraid in my mind it's not so much a debate of first or second class objects, it's a religious discussion of good practice and applicable practice for deep embedded applications.
Returning a structure either means a root structure being changed by stealth in the depths of the call sequence, or a duplication of data and the passing of large chunks of duplicated data. The main applications of C are still largely concentrated around the deep embedded applications. In these domains you have small processors that don't need to be passing large blocks of data. You also have engineering practice that necessitates the need to be able to operate without dynamic RAM allocation, and with minimal stack and often no heap. It could be argued the return of the structure is the same as modification via pointer, but abstracted in syntax... I'm afraid I'd argue that's not in the C philosophy of "what you see is what you get" in the same way a pointer to a type is.
Personally, I would argue you have found a loop hole, whether standard approved or not. C is designed in such a way that allocation is explicit. You pass as a matter of good practice address bus sized objects, normally in an aspirational one cycle, referring to memory that has been allocated explicitly at a controlled time within the developers ken. This makes sense in terms of code efficiency, cycle efficiency, and offers the most control and clarity of purpose. I'm afraid, in code inspection I'd throw out a function returning a structure as bad practice. C does not enforce many rules, it's a language for professional engineers in many ways as it relies upon the user enforcing their own discipline. Just because you can, doesn't mean you should... It does offer some pretty bullet proof ways to handle data of very complex size and type utilising compile time rigour and minimising the dynamic variations of footprint and at runtime.
Is there any advantage to using pointer notation over array notation? I realize that there may be some special cases where pointer notation is better, but it seems to me that array notation is clearer. My professor told us that he prefers pointer notation "because it's C", but it's not something he will be marking. And I know that there are differences with declaring strings as character arrays vs declaring a pointer as a string - I'm just talking about in general looping through an array.
If you write a straightforward loop, both array and pointer forms typically compile to the same machine code.
There are differences in especially non-constant loop exit conditions, but it only matters if you are trying to optimize the loop for a specific compiler and architecture.
So, how about we consider a real world example that relies on both?
These types implement a double-precision floating-point matrix of dynamically determined size, with separate reference-counted data storage:
struct owner {
long refcount;
size_t size;
double data[]; /* C99 flexible array member */
};
struct matrix {
long rows;
long cols;
long rowstep;
long colstep;
double *origin;
struct owner *owner;
};
The idea is that when you need a matrix, you describe it using a local variable of type struct matrix. All data referred to is stored in dynamically allocated struct owner structures, in the C99 flexible array member. After you no longer need the matrix, you must explicitly "drop" it. This allows multiple matrices refer to the same data: you can even have separate row, column, or diagonal vectors, with any change to one immediately reflected in the all others (because they refer to the same data values).
When a matrix is associated with data, either by creating an empty matrix, or by referring to existing data referred to by another matrix, the owner structure refcount is incremented. Whenever a matrix is dropped, the referred to owner structure refcount is decremented. The owner structure is freed, when the refcount drops to zero. This means you only need to remember to "drop" each matrix you used, and the data referred to will be correctly managed and released as soon as possible (unneeded), but never too early.
This all assumes a single-threaded process; multithreaded handling is quite a bit more complicated.
To access element in matrix struct matrix m, row r, column c, assuming 0 <= r < m.rows and 0 <= c < m.cols, you use m.origin[r*m.rowstep + c*m.colstep].
If you want to transpose a matrix, you simply swap m.rows and m.cols, and m.rowstep and m.colstep. All that changes, is the order in which the data (stored in the owner structure) is read.
(Note that origin points to the double which appears at row 0, column 0, in the matrix; and that rowstep and colstep can be negative. This allows all kinds of weird "views" to the otherwise dull regular data, like mirrors and diagonals and so on.)
If we did not have the C99 flexible array member -- say, we only had pointers, and no array notation at all --, the owner structure data member would have to be a pointer. It would mean an additional redirection at the hardware level (slowing down the data accesses a bit). We would either need to allocate the memory pointed by data separately, or use tricks to point to an address following the owner structure itself, but suitably aligned for a double.
Multidimensional arrays do have their uses -- basically, when the sizes of all dimensions (or all but one dimension) are known --, and it's nice for the compiler to take care of the indexing, but it does not have to mean they are always easier than methods using pointers. For example, in the above matrix structure case, we can always define some helper preprocessor macros, like
#define MATRIXELEM(m, r, c) ((m).origin[(r)*(m).rowstep + (c)*(m).colstep])
which admittedly has the downside that it evaluates the first parameter, m, three times. (It means that MATRIXELEM(m++,0,0) would actually try to increment m three times.) In this particular case, m is normally a local variable of struct matrix type, which should minimize surprises. One could have e.g.
struct matrix m1, m2;
/* Stuff that initializes m1 and m2, and makes sure they point
to valid matrix data */
MATRIXELEM(m1, 0, 0) = MATRIXELEM(m2, 0, 0);
The "extra" parentheses in such macros ensure that if you use a calculation, for example i + 4*j as row, the index calculation is correct ((i + 4*j)*m.rowstep and not i + 4*j*m.rowstep). In preprocessor macros, those parentheses are not really "extra" at all. In addition to ensure the correct calculation, having the "extra" parentheses also tell other programmers that the macro writer has been careful in avoiding such arithmetic-related bugs. (I for one consider it "good form" to put the parentheses there, even in cases where they are not needed for syntax unambiquity, if it conveys that "assurance" to other developers reading the code.)
And this, after all this text, leads to my most important point: Some things are easier expressed and understood by us human programmers using array notation than pointer notation, and vice versa. "Foo"[1] is pretty obviously equal to 'o', whereas *("Foo"+1) is not nearly as obvious. (Then again, neither is 1["foo"], but you can blame the C standardization folks for that.)
Based on the examples above, I consider the two notations complementary; they do have large overlap especially in simple loops -- in which case it is okay to just pick one --, but being able to utilize both notations and pick one not based on ones proficiency in one but based on ones opinion on as to what makes most sense wrt. readability and maintainability, is an important skill for any C programmer, in my not very humble opinion.
Actually if you, say, pass an array argument to a function in C, you actually pass a pointer to its beginning. This doesn't really passes an array in a common sense, first, because passing an array would include passing its actual length, second, because passing an array (as a value) would imply its copying.
In other word, you really pass an iterator pointing to an array beginning (like std::vector::begin() in C++) but you pretend that you pass the array itself. It's very confusing in fact. So, using pointers represents things those are really happening in a much more clear way, and it definitely should be preferred.
There may be some advantages of array notation too but I don't think they overweight the drawbacks mentioned. First, using array notation emphasizes the difference between pointer to a single value and pointer to continuous block. And then, you may specify an expected size of passed array for your own reference. But that size isn't actually passed to expressions or functions or somehow checked which fact is very confusing.
In some cases, one knows at compile time what a particular piece of algorithmic data looks like, and as such might wish to convey this information to the compiler. This question is about how one might best achieve that.
By way of example, consider the following example of a sparse matrix multiplication in which the matrix is constant and known at compile time:
matrix = [ 0, 210, 0, 248, 137]
[ 0, 0, 0, 0, 239]
[ 0, 0, 0, 0, 0]
[116, 112, 0, 0, 7]
[ 0, 0, 0, 0, 165]
In such a case, a fully branchless implementation could be written to implement the matrix vector multiplication for an arbitrary input vector:
#include <stdio.h>
#define ARRAY_SIZE 8
static const int matrix[ARRAY_SIZE] = {210, 248, 137, 239, 116, 112, 7, 165};
static const int input_indices[ARRAY_SIZE] = {1, 3, 4, 4, 0, 1, 4, 4};
static const int output_indices[ARRAY_SIZE] = {0, 0, 0, 1, 3, 3, 3, 4};
static void matrix_multiply(int *input_array, int *output_array)
{
for (int i=0; i<ARRAY_SIZE; ++i){
output_array[output_indices[i]] += (
matrix[i] * input_array[input_indices[i]]);
}
}
int main()
{
int test_input[5] = {36, 220, 212, 122, 39};
int output[5] = {0};
matrix_multiply(test_input, output);
for (int i=0; i<5; ++i){
printf("%d\n", output[i]);
}
}
which prints the correct result for the matrix-vector multiplication (81799, 9321, 0, 29089, 6435).
Further optimisations can be envisaged that build on data specific knowledge about the memory locality of reference.
Now, clearly this is an approach which can be used, but it starts getting unwieldy when the size of the data gets big (say ~100MB in my case) and also in any real world situation would depend on meta-programming to generate the associated data dependent knowledge.
Does the general strategy of baking in data specific knowledge have mileage as regards optimisation? If so, what is the best approach to do this?
In the example given, on one level the whole thing than be reduced to knowledge about ARRAY_SIZE with the arrays set at runtime. This leads me to think the approach is limited (and is really a data structures problem), but I'm very interested to know if the general approach of data derived compile-time optimisations is useful in any situation.
I don't think this is a very good answer to this question but I'm going to try offering it anyway. It's also more of a search for the same basic answer.
I work in 3D VFX including raytracing where it's not uncommon to take a fairly modest input with data structures that build in under a second, and then do a monumental amount of processing subsequently to the point where a user might wait hours for a quality production render in a difficult lighting situation.
In theory at least, this could go so much faster if we could make these "data-specific optimizations". Variables could turn into literal constants, significantly less branching could be required, data that is known to always have an upper bound of 45 elements could be allocated on the stack instead of heap or use another form of memory preallocated in advance, locality of reference could be exploited to a greater deal than ever before, vectorization could be applied more easily, achieving both thread-safety and efficiency could be a lot easier, etc.
Where this gets awkward for me is that this requires information about user inputs which can only be provided after the usual notion of "compile-time". So a lot of my interest here relates to code-generation techniques while the application is running.
Now, clearly this is an approach which can be used, but it starts
getting unwieldy when the size of the data gets big (say ~100MB in my
case) and also in any real world situation would depend on
meta-programming to generate the associated data dependent knowledge.
I think beyond that, if the data size gets excessive, then we do often need a good share of branching and variables just to avoid generating so much code that we start becoming bottlenecked by icache misses.
Yet even the ability to turn a dozen variables accessed frequently into compile-time constants and allowing a handful of data structures to exploit greater knowledge of the specified input (and with the aid of an aggressive optimizer) may yield great mileage here, especially considering how well optimizers do provided they have the necessary information provided in advance.
Some of this could be tackled normally with increasingly elaborate and generalized code, metaprogramming techniques, etc, yet there's a peak to how far we can go there: an optimizer can only optimize as much as the information is has available in advance. The difficulty here is providing that information in a practical way. And, as you already guessed, this can quickly get unwieldy, difficult to maintain, and productivity starts to become just as great (if not greater) of a concern than efficiency.
So the most promising techniques to me revolve about code-generation techniques tuned for a specific problem domain, but not for a specific input (optimizing for the specific input will lean more on the optimizer, the code generation is there so that we can provide more of that info needed for the optimizer more easily/appropriately). A modest example that already does something like this is Open Shading Language, where it uses JIT compilation that exploits this idea to a modest level:
OSL uses the LLVM compiler framework to translate shader networks into
machine code on the fly (just in time, or "JIT"), and in the process
heavily optimizes shaders and networks with full knowledge of the
shader parameters and other runtime values that could not have been
known when the shaders were compiled from source code. As a result, we
are seeing our OSL shading networks execute 25% faster than the
equivalent shaders hand-crafted in C! (That's how our old shaders
worked in our renderer.)
While a 25% improvement over handwritten code is modest, that's still a big deal in a production renderer, and it seems like we could go far beyond that.
The use of nodes as a visual programming language also offers a more restrictive environment that helps reduce human errors, allows expressing solutions at a higher-level, seeing the results of changes made on the fly (instant turnaround), etc. -- so it adds not only efficiency but that productivity we need to avoid getting lost in such optimizations. Maintaining and building the code generator could be a little complex, but it only needs to have the minimal amount of code required and doesn't scale in complexity with the amount of code generated using it.
So apologies -- this isn't exactly an answer to your question as a comment, but I think we're searching for a similar thing.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
From the number of questions posted here, it's clear that people have some pretty fundemental issues when getting their heads around pointers and pointer arithmetic.
I'm curious to know why. They've never really caused me major problems (although I first learned about them back in the Neolithic). In order to write better answers to these questions, I'd like to know what people find difficult.
So, if you're struggling with pointers, or you recently were but suddenly "got it", what were the aspects of pointers that caused you problems?
When I first started working with them, the biggest problem I had was the syntax.
int* ip;
int * ip;
int *ip;
are all the same.
but:
int* ip1, ip2; //second one isn't a pointer!
int *ip1, *ip2;
Why? because the "pointer" part of the declaration belongs to the variable, and not the type.
And then dereferencing the thing uses a very similar notation:
*ip = 4; //sets the value of the thing pointed to by ip to '4'
x = ip; //hey, that's not '4'!
x = *ip; //ahh... there's that '4'
Except when you actually need to get a pointer... then you use an ampersand!
int *ip = &x;
Hooray for consistency!
Then, apparently just to be jerks and prove how clever they are, a lot of library developers use pointers-to-pointers-to-pointers, and if they expect an array of those things, well why not just pass a pointer to that too.
void foo(****ipppArr);
to call this, I need the address of the array of pointers to pointers to pointers of ints:
foo(&(***ipppArr));
In six months, when I have to maintain this code, I will spend more time trying to figure out what all this means than rewriting from the ground up.
(yeah, probably got that syntax wrong -- it's been a while since I've done anything in C. I kinda miss it, but then I'm a bit of a massochist)
I suspect people are going a bit too deep in their answers. An understanding of scheduling, actual CPU operations, or assembly-level memory management isn't really required.
When I was teaching, I found the following holes in students' understanding to be the most common source of problems:
Heap vs Stack storage. It is simply stunning how many people do not understand this, even in a general sense.
Stack frames. Just the general concept of a dedicated section of the stack for local variables, along with the reason it's a 'stack'... details such as stashing the return location, exception handler details, and previous registers can safely be left till someone tries to build a compiler.
"Memory is memory is memory" Casting just changes which versions of operators or how much room the compiler gives for a particular chunk of memory. You know you're dealing with this problem when people talk about "what (primitive) variable X really is".
Most of my students were able to understand a simplified drawing of a chunk of memory, generally the local variables section of the stack at the current scope. Generally giving explicit fictional addresses to the various locations helped.
I guess in summary, I'm saying that if you want to understand pointers, you have to understand variables, and what they actually are in modern architectures.
Proper understanding of pointers requires knowledge about the underlying machine's architecture.
Many programmers today don't know how their machine works, just as most people who know how to drive a car don't know anything about the engine.
When dealing with pointers, people that get confused are widely in one of two camps. I've been (am?) in both.
The array[] crowd
This is the crowd that straight up doesn't know how to translate from pointer notation to array notation (or doesn't even know that they are even related). Here are four ways to access elements of an array:
array notation (indexing) with the
array name
array notation (indexing) with the
pointer name
pointer notation (the *) with the
pointer name
pointer notation (the *) with the
array name
int vals[5] = {10, 20, 30, 40, 50};
int *ptr;
ptr = vals;
array element pointer
notation number vals notation
vals[0] 0 10 *(ptr + 0)
ptr[0] *(vals + 0)
vals[1] 1 20 *(ptr + 1)
ptr[1] *(vals + 1)
vals[2] 2 30 *(ptr + 2)
ptr[2] *(vals + 2)
vals[3] 3 40 *(ptr + 3)
ptr[3] *(vals + 3)
vals[4] 4 50 *(ptr + 4)
ptr[4] *(vals + 4)
The idea here is that accessing arrays via pointers seems pretty simple and straightforward, but a ton of very complicated and clever things can be done this way. Some of which can leave experienced C/C++ programmers befuddled, let alone inexperienced newbies.
The reference to a pointer and pointer to a pointer crowd
This is a great article that explains the difference and which I'll be citing and stealing some code from :)
As a small example, it can be very difficult to see exactly what the author wanted to do if you came across something like this:
//function prototype
void func(int*& rpInt); // I mean, seriously, int*& ??
int main()
{
int nvar=2;
int* pvar=&nvar;
func(pvar);
....
return 0;
}
Or, to a lesser extent, something like this:
//function prototype
void func(int** ppInt);
int main()
{
int nvar=2;
int* pvar=&nvar;
func(&pvar);
....
return 0;
}
So at the end of the day, what do we really solve with all this gibberish? Nothing.
Now we have seen the syntax of
ptr-to-ptr and ref-to-ptr. Are there
any advantages of one over the other?
I am afraid, no. The usage of one of
both, for some programmers are just
personal preferences. Some who use
ref-to-ptr say the syntax is "cleaner"
while some who use ptr-to-ptr, say
ptr-to-ptr syntax makes it clearer to
those reading what you are doing.
This complexity and the seeming (bold seeming) interchangeability with references ,which is often another caveat of pointers and an error of newcomers, makes understanding pointers hard. It's also important to understand, for completion's sake, that pointers to references are illegal in C and C++ for confusing reasons that take you into lvalue-rvalue semantics.
As a previous answer remarked, many times you'll just have these hotshot programmers that think they are being clever by using ******awesome_var->lol_im_so_clever() and most of us are probably guilty of writing such atrocities at times, but it's just not good code, and it's certainly not maintainable.
Well this answer turned out to be longer than I had hoped...
I blame the quality of reference materials and the people doing the teaching, personally; most concepts in C (but especially pointers) are just plain taught badly. I keep threatening to write my own C book (titled The Last Thing The World Needs Is Another Book On The C Programming Language), but I don't have the time or the patience to do so. So I hang out here and throw random quotes from the Standard at people.
There's also the fact that when C was initially designed, it was assumed you understood machine architecture to a pretty detailed level just because there was no way to avoid it in your day-to-day work (memory was so tight and processors were so slow you had to understand how what you wrote affected performance).
There is a great article supporting the notion that pointers are hard on Joel Spolsky's site - The Perils of JavaSchools.
[Disclaimer - I am not a Java-hater per se.]
Most things are harder to understand if you're not grounded in the knowledge that's "underneath". When I taught CS it got a lot easier when I started my students on programming a very simple "machine", a simulated decimal computer with decimal opcodes whose memory consisted of decimal registers and decimal addresses. They would put in very short programs to, for example, add a series of numbers to get a total. Then they would single step it to watch what was happening. They could hold down the "enter" key and watch it run "fast".
I'm sure almost everyone on SO wonders why it is useful to get so basic. We forget what it was like not knowing how to program. Playing with such a toy computer puts in place concepts without which you can't program, such as the ideas that computation is a step-by-step process, using a small number of basic primitives to build up programs, and the concept of memory variables as places where numbers are stored, in which the address or name of the variable is distinct from the number it contains. There is a distinction between the time at which you enter the program, and the time at which it "runs". I liken learning to program as crossing a series of "speed bumps", such as very simple programs, then loops and subroutines, then arrays, then sequential I/O, then pointers and data structure. All of these are much easier to learn by reference to what a computer is really doing underneath.
Finally, when getting to C, pointers are confusing though K&R did a very good job of explaining them. The way I learned them in C was to know how to read them - right to left. Like when I see int *p in my head I say "p points to an int". C was invented as one step up from assembly language and that's what I like about it - it is close to that "ground". Pointers, like anything else, are harder to understand if you don't have that grounding.
I didn't get pointers until I read the description in K&R. Until that point, pointers didn't make sense. I read a whole bunch of stuff where people said "Don't learn pointers, they are confusing and will hurt your head and give you aneurysms" so I shied away from it for a long time, and created this unnecessary air of difficult-concept.
Otherwise, mostly what I thought was, why on earth would you want a variable that you have to go through hoops to get the value of, and if you wanted to assign stuff to it, you had to do strange things to get values to go into them. The whole point of a variable is something to store a value, I thought, so why someone wanted to make it complicated was beyond me. "So with a pointer you have to use the * operator to get at its value??? What kind of goofy variable is that?", I thought. Pointless, no pun intended.
The reason it was complicated was because I didn't understand that a pointer was an address to something. If you explain that it is an address, that it is something that contains an address to something else, and that you can manipulate that address to do useful things, I think it might clear up the confusion.
A class that required using pointers to access/modify ports on a PC, using pointer arithmetic to address different memory locations, and looking at more complicated C-code that modified their arguments disabused me of the idea that pointers were, well, pointless.
Here's a pointer/array example that gave me pause. Assume you have two arrays:
uint8_t source[16] = { /* some initialization values here */ };
uint8_t destination[16];
And your goal is to copy the uint8_t contents from source destination using memcpy(). Guess which of the following accomplish that goal:
memcpy(destination, source, sizeof(source));
memcpy(&destination, source, sizeof(source));
memcpy(&destination[0], source, sizeof(source));
memcpy(destination, &source, sizeof(source));
memcpy(&destination, &source, sizeof(source));
memcpy(&destination[0], &source, sizeof(source));
memcpy(destination, &source[0], sizeof(source));
memcpy(&destination, &source[0], sizeof(source));
memcpy(&destination[0], &source[0], sizeof(source));
The answer (Spoiler Alert!) is ALL of them. "destination", "&destination", and "&destination[0]" are all the same value. "&destination" is a different type than the other two, but it is still the same value. The same goes for the permutations of "source".
As an aside, I personally prefer the first version.
I should start out by saying that C and C++ were the first programming languages I learned. I started with C, then did C++ in school, a lot, and then went back to C to become fluent in it.
The first thing that confused me about pointers when learning C was the simple:
char ch;
char str[100];
scanf("%c %s", &ch, str);
This confusion was mostly rooted in having been introduced to using reference to a variable for OUT arguments before pointers were properly introduced to me. I remember that I skipped writing the first few examples in C for Dummies because they were too simple only to never get the first program I did write to work (most likely because of this).
What was confusing about this was what &ch actually meant as well as why str didn't need it.
After I became familiar with that I next remember being confused about dynamic allocation. I realized at some point that having pointers to data wasn't extremely useful without dynamic allocation of some type, so I wrote something like:
char * x = NULL;
if (y) {
char z[100];
x = z;
}
to try to dynamically allocate some space. It didn't work. I wasn't sure that it would work, but I didn't know how else it might work.
I later learned about malloc and new, but they really seemed like magical memory generators to me. I knew nothing about how they might work.
Some time later I was being taught recursion again (I'd learned it on my own before, but was in class now) and I asked how it worked under the hood -- where were the separate variables stored. My professor said "on the stack" and lots of things became clear to me. I had heard the term before and had implemented software stacks before. I had heard others refer to "the stack" long before, but had forgotten about it.
Around this time I also realized that using multidimensional arrays in C can get very confusing. I knew how they worked, but they were just so easy to get tangled up in that I decided to try to work around using them whenever I could. I think that the issue here was mostly syntactic (especially passing to or returning them from functions).
Since I was writing C++ for school for the next year or two I got a lot of experience using pointers for data structures. Here I had a new set of troubles -- mixing up pointers. I would have multiple levels of pointers (things like node ***ptr;) trip me up. I'd dereference a pointer the wrong number of times and eventually resort to figuring out how many * I needed by trial and error.
At some point I learned how a program's heap worked (sort of, but good enough that it no longer kept me up at night). I remember reading that if you look a few bytes before the pointer that malloc on a certain system returns, you can see how much data was actually allocated. I realized that the code in malloc could ask for more memory from the OS and this memory was not part of my executable files. Having a decent working idea of how malloc works is a really useful.
Soon after this I took an assembly class, which didn't teach me as much about pointers as most programmers probably think. It did get me to think more about what assembly my code might be translated into. I had always tried to write efficient code, but now I had a better idea how to.
I also took a couple of classes where I had to write some lisp. When writing lisp I wasn't as concerned with efficiency as I was in C. I had very little idea what this code might be translated into if compiled, but I did know that it seemed like using lots of local named symbols (variables) made things a lot easier. At some point I wrote some AVL tree rotation code in a little bit of lisp, that I had a very hard time writing in C++ because of pointer issues. I realized that my aversion to what I thought were excess local variables had hindered my ability to write that and several other programs in C++.
I also took a compilers class. While in this class I flipped ahead to the advanced material and learned about static single assignment (SSA) and dead variables, which isn't that important except that it taught me that any decent compiler will do a decent job of dealing with variables which are no longer used. I already knew that more variables (including pointers) with correct types and good names would help me keep things straight in my head, but now I also knew that avoiding them for efficiency reasons was even more stupid than my less micro-optimization minded professors told me.
So for me, knowing a good bit about the memory layout of a program helped a lot. Thinking about what my code means, both symbolically and on the hardware, helps me out. Using local pointers that have the correct type helps a lot. I often write code that looks like:
int foo(struct frog * f, int x, int y) {
struct leg * g = f->left_leg;
struct toe * t = g->big_toe;
process(t);
so that if I screw up a pointer type it is very clear by the compiler error what the problem is. If I did:
int foo(struct frog * f, int x, int y) {
process(f->left_leg->big_toe);
and got any pointer type wrong in there, the compiler error would be a whole lot more difficult to figure out. I would be tempted to resort to trial and error changes in my frustration, and probably make things worse.
Looking back, there were four things that really helped me to finally understand pointers. Prior to this, I could use them, but I did not fully understand them. That is, I knew if I followed the forms, I would get the results I desired, but I did not fully understand the 'why' of the forms. I realize that this is not exactly what you have asked, but I think it is a useful corollary.
Writing a routine that took a pointer to an integer and modified the integer. This gave me the necessary forms upon which to build any mental models of how pointers work.
One-dimensional dynamic memory allocation. Figuring out 1-D memory allocation made me understand the concept of the pointer.
Two-dimensional dynamic memory allocation. Figuring out 2-D memory allocation reinforced that concept, but also taught me that the pointer itself requires storage and must be taken into account.
Differences between stack variables, global variables and heap memory. Figuring out these differences taught me the types of memory to which the pointers point/refer.
Each of these items required imagining what was going on at a lower level--building a mental model that satisfied every case I could think of throwing at it. It took time and effort, but it was well worth it. I am convinced that to understand pointers, you have to build that mental model on how they work and how they are implemented.
Now back to your original question. Based on the previous list, there were several items that I had difficulty in grasping originally.
How and why would one use a pointer.
How are they different and yet similar to arrays.
Understanding where the pointer information is stored.
Understanding what and where it is the pointer is pointing at.
I had my "pointer moment" working on some telephony programs in C. I had to write a AXE10 exchange emulator using a protocol analyser that only understood classic C. Everything hinged on knowing pointers. I tried writing my code without them (hey, I was "pre-pointer" cut me some slack) and failed utterly.
The key to understanding them, for me, was the & (address) operator. Once I understood that &i meant the "address of i" then understanding that *i meant "the contents of the address pointed to by i" came a bit later. Whenever I wrote or read my code I always repeated what "&" meant and what "*" meant and eventually I came to use them intuitively.
To my shame, I was forced into VB and then Java so my pointer knowledge is not as sharp as it once was, but I am glad I am "post-pointer". Don't ask me to use a library that requires me to understand **p, though.
The main difficulty with pointers, at least to me, is that I didn't start with C. I started with Java. The whole notion of pointers were really foreign until a couple of classes in college where I was expected to know C. So then I taught myself the very basics of C and how to use pointers in their very basic sense. Even then, every time I find myself reading C code, I have to look up pointer syntax.
So in my very limited experience(1 year real world + 4 in college), pointers confuse me because I've never had to really use it in anything other than a classroom setting. And I can sympathize with the students now starting out CS with JAVA instead of C or C++. As you said, you learned pointers in the 'Neolithic' age and have probably been using it ever since that. To us newer people, the notion of allocating memory and doing pointer arithmetic is really foreign because all these languages have abstracted that away.
P.S.
After reading the Spolsky essay, his description of 'JavaSchools' was nothing like what I went through in college at Cornell ('05-'09). I took the structures and functional programming (sml), operating systems (C), algorithms (pen and paper), and a whole slew of other classes that weren't taught in java. However all the intro classes and electives were all done in java because there's value in not reinventing the wheel when you are trying to do something higher leveled than implementing a hashtable with pointers.
Here is a non-answer:
Use cdecl (or c++decl) to figure it out:
eisbaw#leno:~$ cdecl explain 'int (*(*foo)(const void *))[3]'
declare foo as pointer to function (pointer to const void) returning pointer to array 3 of int
They add an extra dimension to the code without a significant change to the syntax. Think about this:
int a;
a = 5
There's only one thing to change: a. You can write a = 6 and the results are obvious to most people. But now consider:
int *a;
a = &some_int;
There are two things about a that are relevant at different times: the actual value of a, the pointer, and the value "behind" the pointer. You can change a:
a = &some_other_int;
...and some_int is still around somewhere with the same value. But you can also change the thing it points to:
*a = 6;
There's a conceptual gap between a = 6, which has only local side effects, and *a = 6, which could affect a bunch of other things in other places. My point here is not that the concept of indirection is inherently tricky, but that because you can do both the immediate, local thing with a or the indirect thing with *a... that might be what confuses people.
I had programmed in c++ for like 2 years and then converted to Java(5 years) and never looked back. However, when I recently had to use some native stuff, I found out (with amazement) that I hadn't forgotten anything about pointers and I even find them easy to use. This is a sharp contrast to what I experienced 7 years ago when I first tried to grasp the concept. So, I guess understanding and liking is a matter of programming maturity ? :)
OR
Pointers are like riding a bike, once you figure out how to work with them, there's no forgetting it.
All in all, hard to grasp or not, the whole pointer idea is VERY educational and I believe it should be understood by every programmer regardless if he programs on a language with pointers or not.
I think one reason C pointers are difficult is that they conflate several concepts which are not really equivalent; yet, because they are all implemented using pointers, people can have a hard time disentangling the concepts.
In C, pointers are used to, amoung other things:
Define recursive data structures
In C you'd define a linked list of integers like this:
struct node {
int value;
struct node* next;
}
The pointer is only there because this is the only way to define a recursive data structure in C, when the concept really has nothing to do with such a low-level detail as memory addresses. Consider the following equivalent in Haskell, which doesn't require use of pointers:
data List = List Int List | Null
Pretty straightforward - a list is either empty, or formed from a value and the rest of the list.
Iterate over strings and arrays
Here's how you might apply a function foo to every character of a string in C:
char *c;
for (c = "hello, world!"; *c != '\0'; c++) { foo(c); }
Despite also using a pointer as an iterator, this example has very little in common with the previous one. Creating an iterator that you can increment is a different concept from defining a recursive data structure. Neither concept is especially tied to the idea of a memory address.
Achieve polymorphism
Here is an actual function signature found in glib:
typedef struct g_list GList;
void g_list_foreach (GList *list,
void (*func)(void *data, void *user_data),
void* user_data);
Whoa! That's quite a mouthful of void*'s. And it's all just to declare a function that iterates over a list that can contain any kind of thing, applying a function to each member. Compare it to how map is declared in Haskell:
map::(a->b)->[a]->[b]
That's much more straightforward: map is a function that takes a function which converts an a to a b, and applies it to a list of a's to yield a list of b's. Just like in the C function g_list_foreach, map doesn't need to know anything in its own definition about the types to which it will be applied.
To sum up:
I think C pointers would be a lot less confusing if people first learned about recursive data structures, iterators, polymorphism, etc. as separate concepts, and then learned how pointers can be used to implement those ideas in C, rather than mashing all of these concepts together into a single subject of "pointers".
I think it requires a solid foundation, probably from the machine level, with introduction to some machine code, assembly, and how to represent items and data structure in RAM. It takes a little time, some homework or problem solving practice, and some thinking.
But if a person knows high level languages at first (which is nothing wrong -- a carpenter uses an ax. a person who needs to split atom uses something else. we need people who are carpenters, and we have people who study atoms) and this person who knows high level language is given a 2 minute introduction to pointers, and then it is hard to expect him to understand pointer arithmetics, pointers to pointers, array of pointers to variable size strings, and array of array of characters, etc. A low-level solid foundation can help a lot.
Pointers are difficult because of the indirection.
Pointers (along with some other aspects of low-level work), require the user to take away the magic.
Most high level programmers like the magic.
Pointers are a way of dealing with the difference between a handle to an object and an object itself. (ok, not necessarily objects, but you know what I mean, as well as where my mind is)
At some point, you probably have to deal with the difference between the two. In modern, high-level language this becomes the distinction between copy-by-value and copy-by-reference. Either way, it is a concept that is often difficult for programmers to grasp.
However, as has been pointed out, the syntax for handling this problem in C is ugly, inconsistent, and confusing. Eventually, if you really attempt to understand it, a pointer will make sense. But when you start dealing with pointers to pointers, and so on ad nauseum, it gets really confusing for me as well as for other people.
Another important thing to remember about pointers is that they're dangerous. C is a master programmer's language. It assumes you know what the heck you're doing and thereby gives you the power to really mess things up. While some types of programs still need to be written in C, most programs do not, and if you have a language that provides a better abstraction for the difference between an object and its handle, then I suggest you use it.
Indeed, in many modern C++ applications, it is often the case that any required pointer arithmetic is encapsulated and abstracted. We don't want developers doing pointer arithmetic all over the place. We want a centralized, well tested API that does pointer arithmetic at the lowest level. Making changes to this code must be done with great care and extensive testing.
The problem I have always had (primarily self-taught) is the "when" to use a pointer. I can wrap my head around the syntax for constructing a pointer but I need to know under which circumstances a pointer should be used.
Am I the only one with this mindset? ;-)
Once upon a time... We had 8 bit microprocessors and everyone wrote in assembly. Most processors included some type of indirect addressing used for jump tables and kernels. When higher level languages came along we add a thin layer of abstraction and called them pointers. Over the years we have gotten more and more away from the hardware. This is not necessarily a bad thing. They are called higher level languages for a reason. The more I can concentrate on what I want to do instead of the details of how it is done the better.
It seems many students have a problem with the concept of indirection, especially when they meet the concept of indirection for the first time. I remember from back when I was a student that out of the +100 students of my course, only a handful of people really understood pointers.
The concept of indirection is not something that we often use in real life, and therefore it's a hard concept to grasp initially.
I have recently just had the pointer click moment, and I was surprised that I had been finding it confusing. It was more that everyone talked about it so much, that I assumed some dark magic was going on.
The way I got it was this. Imagine that all defined variables are given memory space at compile time(on the stack). If you want a program that could handle large data files such as audio or images, you wouldn't want a fixed amount of memory for these potential structures. So you wait until runtime to assign a certain amount of memory to holding this data(on the heap).
Once you have your data in memory, you don't want to be copying that data all around your memory bus every time you want to run an operation on it. Say you want to apply a filter to your image data. You have a pointer that starts at the front of the data you have assigned to the image, and a function runs across that data, changing it in place. If you didn't know what you we're doing, you would probably end up making duplicates of data, as you ran it through the operation.
At least that's the way I see it at the moment!
Speaking as a C++ newbie here:
The pointer system took a while for me to digest not necessarily because of the concept but because of the C++ syntax relative to Java. A few things I found confusing are:
(1) Variable declaration:
A a(1);
vs.
A a = A(1);
vs.
A* a = new A(1);
and apparently
A a();
is a function declaration and not a variable declaration. In other languages, there's basically just one way to declare a variable.
(2) The ampersand is used in a few different ways. If it is
int* i = &a;
then the &a is a memory address.
OTOH, if it is
void f(int &a) {}
then the &a is a passed-by-reference parameter.
Although this may seem trivial, it can be confusing for new users - I came from Java and Java's a language with a more uniform use of operators
(3) Array-pointer relationship
One thing that's a tad bit frustrating to comprehend is that a pointer
int* i
can be a pointer to an int
int *i = &n; //
or
can be an array to an int
int* i = new int[5];
And then just to make things messier, pointers and array are not interchangeable in all cases and pointers cannot be passed as array parameters.
This sums up some of the basic frustrations I had with C/C++ and its pointers, which IMO, is greatly compounded by the fact that C/C++ has all these language-specific quirks.
I personally did not understand the pointer even after my post graduation and after my first job. The only thing I was knowing is that you need it for linked list, binary trees and for passing arrays into functions. This was the situation even at my first job. Only when I started to give interviews, I understand that the pointer concept is deep and has tremendous use and potential. Then I started reading K & R and writing own test program. My whole goal was job-driven.
At this time I found that pointers are really not bad nor difficult if they are been taught in a good way. Unfortunately when I learn C in graduation, out teacher was not aware of pointer, and even the assignments were using less of pointers. In the graduate level the use of pointer is really only upto creating binary trees and linked list. This thinking that you don't need proper understanding of pointers to work with them, kill the idea of learning them.
Pointers.. hah.. all about pointer in my head is that it give a memory address where the actual values of whatever its reference.. so no magic about it.. if you learn some assembly you wouldn't have that much trouble learning how pointers works.. come on guys... even in Java everything is a reference..
The main problem people do not understand why do they need pointers.
Because they are not clear about stack and heap.
It is good to start from 16bit assembler for x86 with tiny memory mode. It helped many people to get idea of stack, heap and "address". And byte:) Modern programmers sometimes can't tell you how many bytes you need to address 32 bit space. How can they get idea of pointers?
Second moment is notation: you declare pointer as *, you get address as & and this is not easy to understand for some people.
And the last thing I saw was storage problem: they understand heap and stack but can't get into idea of "static".