Related
In Go there are various ways to return a struct value or slice thereof. For individual ones I've seen:
type MyStruct struct {
Val int
}
func myfunc() MyStruct {
return MyStruct{Val: 1}
}
func myfunc() *MyStruct {
return &MyStruct{}
}
func myfunc(s *MyStruct) {
s.Val = 1
}
I understand the differences between these. The first returns a copy of the struct, the second a pointer to the struct value created within the function, the third expects an existing struct to be passed in and overrides the value.
I've seen all of these patterns be used in various contexts, I'm wondering what the best practices are regarding these. When would you use which? For instance, the first one could be ok for small structs (because the overhead is minimal), the second for bigger ones. And the third if you want to be extremely memory efficient, because you can easily reuse a single struct instance between calls. Are there any best practices for when to use which?
Similarly, the same question regarding slices:
func myfunc() []MyStruct {
return []MyStruct{ MyStruct{Val: 1} }
}
func myfunc() []*MyStruct {
return []MyStruct{ &MyStruct{Val: 1} }
}
func myfunc(s *[]MyStruct) {
*s = []MyStruct{ MyStruct{Val: 1} }
}
func myfunc(s *[]*MyStruct) {
*s = []MyStruct{ &MyStruct{Val: 1} }
}
Again: what are best practices here. I know slices are always pointers, so returning a pointer to a slice isn't useful. However, should I return a slice of struct values, a slice of pointers to structs, should I pass in a pointer to a slice as argument (a pattern used in the Go App Engine API)?
tl;dr:
Methods using receiver pointers are common; the rule of thumb for receivers is, "If in doubt, use a pointer."
Slices, maps, channels, strings, function values, and interface values are implemented with pointers internally, and a pointer to them is often redundant.
Elsewhere, use pointers for big structs or structs you'll have to change, and otherwise pass values, because getting things changed by surprise via a pointer is confusing.
One case where you should often use a pointer:
Receivers are pointers more often than other arguments. It's not unusual for methods to modify the thing they're called on, or for named types to be large structs, so the guidance is to default to pointers except in rare cases.
Jeff Hodges' copyfighter tool automatically searches for non-tiny receivers passed by value.
Some situations where you don't need pointers:
Code review guidelines suggest passing small structs like type Point struct { latitude, longitude float64 }, and maybe even things a bit bigger, as values, unless the function you're calling needs to be able to modify them in place.
Value semantics avoid aliasing situations where an assignment over here changes a value over there by surprise.
Passing small structs by value can be more efficient by avoiding cache misses or heap allocations. In any case, when pointers and values perform similarly, the Go-y approach is to choose whatever provides the more natural semantics rather than squeeze out every last bit of speed.
So, Go Wiki's code review comments page suggests passing by value when structs are small and likely to stay that way.
If the "large" cutoff seems vague, it is; arguably many structs are in a range where either a pointer or a value is OK. As a lower bound, the code review comments suggest slices (three machine words) are reasonable to use as value receivers. As something nearer an upper bound, bytes.Replace takes 10 words' worth of args (three slices and an int). You can find situations where copying even large structs turns out a performance win, but the rule of thumb is not to.
For slices, you don't need to pass a pointer to change elements of the array. io.Reader.Read(p []byte) changes the bytes of p, for instance. It's arguably a special case of "treat little structs like values," since internally you're passing around a little structure called a slice header (see Russ Cox (rsc)'s explanation). Similarly, you don't need a pointer to modify a map or communicate on a channel.
For slices you'll reslice (change the start/length/capacity of), built-in functions like append accept a slice value and return a new one. I'd imitate that; it avoids aliasing, returning a new slice helps call attention to the fact that a new array might be allocated, and it's familiar to callers.
It's not always practical follow that pattern. Some tools like database interfaces or serializers need to append to a slice whose type isn't known at compile time. They sometimes accept a pointer to a slice in an interface{} parameter.
Maps, channels, strings, and function and interface values, like slices, are internally references or structures that contain references already, so if you're just trying to avoid getting the underlying data copied, you don't need to pass pointers to them. (rsc wrote a separate post on how interface values are stored).
You still may need to pass pointers in the rarer case that you want to modify the caller's struct: flag.StringVar takes a *string for that reason, for example.
Where you use pointers:
Consider whether your function should be a method on whichever struct you need a pointer to. People expect a lot of methods on x to modify x, so making the modified struct the receiver may help to minimize surprise. There are guidelines on when receivers should be pointers.
Functions that have effects on their non-receiver params should make that clear in the godoc, or better yet, the godoc and the name (like reader.WriteTo(writer)).
You mention accepting a pointer to avoid allocations by allowing reuse; changing APIs for the sake of memory reuse is an optimization I'd delay until it's clear the allocations have a nontrivial cost, and then I'd look for a way that doesn't force the trickier API on all users:
For avoiding allocations, Go's escape analysis is your friend. You can sometimes help it avoid heap allocations by making types that can be initialized with a trivial constructor, a plain literal, or a useful zero value like bytes.Buffer.
Consider a Reset() method to put an object back in a blank state, like some stdlib types offer. Users who don't care or can't save an allocation don't have to call it.
Consider writing modify-in-place methods and create-from-scratch functions as matching pairs, for convenience: existingUser.LoadFromJSON(json []byte) error could be wrapped by NewUserFromJSON(json []byte) (*User, error). Again, it pushes the choice between laziness and pinching allocations to the individual caller.
Callers seeking to recycle memory can let sync.Pool handle some details. If a particular allocation creates a lot of memory pressure, you're confident you know when the alloc is no longer used, and you don't have a better optimization available, sync.Pool can help. (CloudFlare published a useful (pre-sync.Pool) blog post about recycling.)
Finally, on whether your slices should be of pointers: slices of values can be useful, and save you allocations and cache misses. There can be blockers:
The API to create your items might force pointers on you, e.g. you have to call NewFoo() *Foo rather than let Go initialize with the zero value.
The desired lifetimes of the items might not all be the same. The whole slice is freed at once; if 99% of the items are no longer useful but you have pointers to the other 1%, all of the array remains allocated.
Copying or moving the values might cause you performance or correctness problems, making pointers more attractive. Notably, append copies items when it grows the underlying array. Pointers to slice items from before the append may not point to where the item was copied after, copying can be slower for huge structs, and for e.g. sync.Mutex copying isn't allowed. Insert/delete in the middle and sorting also move items around so similar considerations can apply.
Broadly, value slices can make sense if either you get all of your items in place up front and don't move them (e.g., no more appends after initial setup), or if you do keep moving them around but you're confident that's OK (no/careful use of pointers to items, and items are small or you've measured the perf impact). Sometimes it comes down to something more specific to your situation, but that's a rough guide.
If you can (e.g. a non-shared resource that does not need to be passed as reference), use a value. By the following reasons:
Your code will be nicer and more readable, avoiding pointer operators and null checks.
Your code will be safer against Null Pointer panics.
Your code will be often faster: yes, faster! Why?
Reason 1: you will allocate less items in the heap. Allocating/deallocating from stack is immediate, but allocating/deallocating on Heap may be very expensive (allocation time + garbage collection). You can see some basic numbers here: http://www.macias.info/entry/201802102230_go_values_vs_references.md
Reason 2: especially if you store returned values in slices, your memory objects will be more compacted in memory: looping a slice where all the items are contiguous is much faster than iterating a slice where all the items are pointers to other parts of the memory. Not for the indirection step but for the increase of cache misses.
Myth breaker: a typical x86 cache line are 64 bytes. Most structs are smaller than that. The time of copying a cache line in memory is similar to copying a pointer.
Only if a critical part of your code is slow I would try some micro-optimization and check if using pointers improves somewhat the speed, at the cost of less readability and mantainability.
Three main reasons when you would want to use method receivers as pointers:
"First, and most important, does the method need to modify the receiver? If it does, the receiver must be a pointer."
"Second is the consideration of efficiency. If the receiver is large, a big struct for instance, it will be much cheaper to use a pointer receiver."
"Next is consistency. If some of the methods of the type must have pointer receivers, the rest should too, so the method set is consistent regardless of how the type is used"
Reference : https://golang.org/doc/faq#methods_on_values_or_pointers
Edit : Another important thing is to know the actual "type" that you are sending to function. The type can either be a 'value type' or 'reference type'.
Even as slices and maps acts as references, we might want to pass them as pointers in scenarios like changing the length of the slice in the function.
A case where you generally need to return a pointer is when constructing an instance of some stateful or shareable resource. This is often done by functions prefixed with New.
Because they represent a specific instance of something and they may need to coordinate some activity, it doesn't make a lot of sense to generate duplicated/copied structures representing the same resource -- so the returned pointer acts as the handle to the resource itself.
Some examples:
func NewTLSServer(handler http.Handler) *Server -- instantiate a web server for testing
func Open(name string) (*File, error) -- return a file access handle
In other cases, pointers are returned just because the structure may be too large to copy by default:
func NewRGBA(r Rectangle) *RGBA -- allocate an image in memory
Alternatively, returning pointers directly could be avoided by instead returning a copy of a structure that contains the pointer internally, but maybe this isn't considered idiomatic:
No such examples found in the standard libraries...
Related question: Embedding in Go with pointer or with value
Regarding to struct vs. pointer return value, I got confused after reading many highly stared open source projects on github, as there are many examples for both cases, util I found this amazing article:
https://www.ardanlabs.com/blog/2014/12/using-pointers-in-go.html
"In general, share struct type values with a pointer unless the struct type has been implemented to behave like a primitive data value.
If you are still not sure, this is another way to think about. Think of every struct as having a nature. If the nature of the struct is something that should not be changed, like a time, a color or a coordinate, then implement the struct as a primitive data value. If the nature of the struct is something that can be changed, even if it never is in your program, it is not a primitive data value and should be implemented to be shared with a pointer. Don’t create structs that have a duality of nature."
Completedly convinced.
Say I wanted to store objects statically inside their own class. Like this:
public class Foo
{
private static int instance_id = 0;
public static List<Foo> instances = new List<Foo>();
public Foo()
{
instances[instance_id++] = this;
}
}
Why?
I don't need to create unique array structures outside the class (one will do).
I want to map each object to a unique id according to their time of birth.
I will only have one thread with the class in use. Foo will only exist as one set in the program.
I did searching, but could find no mention of this data structure. Is this bad practice? If so, why? Thank you.
{please note, this question is not specific to any language}
There are a couple of potential problems I can see with this setup.
First, since you only have a single array of objects, if you need to update the code so that you have lots of different groups of objects in different contexts, you'll need to do a significant rewrite so that each object ends up getting associated with a different context. Depending on your setup this may not be a problem, but I suspect that in the long term this decision may come back to haunt you.
Second, this approach assumes that you never need to dispose of any objects. Imagine that you want to update your code so that you do a number of different simulations and aggregate the results. If you do this, then you'll end up having your giant array storing pointers to objects you're not using. This means that you'll (1) have a memory leak and (2) have to update all your looping code to skip over objects you no longer care about.
Third, this approach makes it the responsibility of the class, rather than the client, to keep track of all the instances. In some sense, if the purpose of what you're doing is to make it easier for clients to have access to a global list of all the objects that exist, you may want to consider just putting a different list somewhere else that's globally accessible so that the objects themselves aren't the ones responsible for keeping track of themselves.
I would recommend using one of a number of alternate approaches:
Just have the client do this. If the client needs to keep track of all the instances, just have them always create the array they need and populate it. That way, if multiple clients need different arrays, they can do so. You also avoid the memory leak issues if you do this properly.
Have each object take, as part of its constructor, a context in which to be constructed. For example, if all of these objects are nodes in a quadtree, have them take a pointer to the quadtree in which they'll live as a constructor parameter, then have the quadtree object store the list of the nodes in it. After all, it seems like it's really the quadtree's responsibility to keep track of everything.
Keep doing what you're doing, but using something with weak references. For example, you might consider using some variation on a WeakHashMap so that you do store everything, but if the objects are no longer needed, you at least don't have a memory leak.
I was reading an article about array vs list, and the author says that an array is worse than a list, because (among other things) an array is a list of variables, but to me a list is also a list of variables. I mean, I can still do list[3] = new Item().
Actually, I have always somehow saw a List<T> like a wrapper for an array that allows me to use it easily without caring about handling its structure.
What are the internal differences between a T[] and a List<T> in terms of heap/stack memory usage?
Since an array is a static structure, after the initialization, it allocates the memory that you've demanded.
int arr[5];
For example here there are 5 int objects created in memory. But when you use lists, according to its implementation, it gives you first an array with predefined capacity. And while you are adding your elements, if you exceed the capacity then it scales up. In some implementations it just doubles its size, or in some implementations it enlarges itself when the granted capacity is half full.
The author's point about a "list of variables" wasn't about memory. It's that an array contains your internal variables, and returning it allows them to be reassigned by the caller. It comes down to this:
Only pass out an array if it is wrapped up by a read-only object.
If you pass out an internal List<T>, you have the same problem, but here's the key:
We have an extensibility model for lists because lists are classes. We
have no ability to make an “immutable array”. Arrays are what they are
and they’re never going to change.
And, at the time the article was written, the IReadOnlyList interface didn't exist yet (.NET 4.5), though he probably would have mentioned it if it had. I believe he was advocating implementing an IList<T> that would simply throw an exception if you tried to use the setter. Of course, if the user doesn't need the ability to access elements by index, you don't need a list interface at all -- you can just wrap it in a ReadOnlyCollection<T> and return it as an IEnumerable<T>.
My project has classes which, unavoidably, contain hundreds upon hundreds of variables that I'm always having to keep straight. For example, I'm always having to keep track of specific kinds of variables for a recurring set of "items" that occur inside of a class, where placing those variables between multiple classes would cause a lot of confusion.
How do I better sort my variables to keep from going crazy, especially when it comes time to save my data?
Am I missing something? Actionscript is an Object Oriented language, so you might have hundreds of variables, but unless you've somehow treated it like a grab bag and dumped it all in one place, everything should be to hand. Without knowing what all you're keeping track of, it's hard to give concrete advice, but here's an example from a current project I'm working on, which is a platform for building pre-employment assessments.
The basic unit is a Question. A Question has a stem, text that can go in the status bar, a collection of answers, and a collection of measures of things we're tracking about what the user does in that particular type of questions.
The measures are, again, their own type of object, and come in two "flavors": one that is used to track a time limit and one that isn't. The measure has a name (so we know where to write back to the database) and a value (which tells us what). Timed ones also have a property for the time limit.
When we need to time the question, we hand that measure to yet another object that counts the time down and a separate object that displays the time (if appropriate for the situation). The answers, known as distractors, have a label and a value that they can impart to the appropriate measure based on the user selection. For example, if a user selects "d", its value, "4" is transferred to the measure that stores the user's selection.
Once the user submits his answer, we loop through all the measures for the question and send those to the database. If those were not treated as a collection (in this case, a Vector), we'd have to know exactly what specific measures are being stored for each question and each question would have a very different structure that we'd have to dig through. So if looping through collections is your issue, I think you should revisit that idea. It saves a lot of code and is FAR more efficient than "var1", "var2", "var3."
If the part you think is unweildy is the type checking you have to do because literally anything could be in there, then Vector could be a good solution for you as long as you're using at least Flash Player 10.
So, in summary:
When you have a lot of related properties, write a Class that keeps all of those related bits and pieces together (like my Question).
When objects have 0-n "things" that are all of the same or very similar, use a collection of some sort, such as an Array or Vector, to allow you to iterate through them as a group and perform the same operation on each (for example, each Question is part of a larger grouping that allows each question to be presented in turn, and each question has a collection of distractors and another of measures.
These two concepts, used together, should help keep your information tidy and organized.
While I'm certain there are numerous ways of keeping arrays straight, I have found a method that works well for me. Best of all, it collapses large amounts of information into a handful of arrays that I can parse to an XML file or other storage method. I call this method my "indexed array system".
There are actually multiple ways to do this: creating a handful of 1-dimensional arrays, or creating 2-dimensional (or higher) array(s). Both work equally well, so choose the one that works best for your code. I'm only going to show the 1-dimensional method here. Those of you who are familiar with arrays can probably figure out how to rewrite this to use higher dimensional arrays.
I use Actionscript 3, but this approach should work with almost any programming or scripting language.
In this example, I'm trying to keep various "properties" of different "activities" straight. In this case, we'll say these properties are Level, High Score, and Play Count. We'll call the activities Pinball, Word Search, Maze, and Memory.
This method involves creating multiple arrays, one for each property, and creating constants that hold the integer "key" used for each activity.
We'll start by creating the constants, as integers. Constants work for this, because we never change them after compile. The value we put into each constant is the index the corresponding data will always be stored at in the arrays.
const pinball:int = 0;
const wordsearch:int = 1;
const maze:int = 2;
const memory:int = 3;
Now, we create the arrays. Remember, arrays start counting from zero. Since we want to be able to modify the values, this should be a regular variable.
Note, I am constructing the array to be the specific length we need, with the default value for the desired data type in each slot. I've used all integers here, but you can use just about any data type you need.
var highscore:Array = [0, 0, 0, 0];
var level:Array = [0, 0, 0, 0];
var playcount:Array = [0, 0, 0, 0];
So, we have a consistent "address" for each property, and we only had to create four constants, and three arrays, instead of 12 variables.
Now we need to create the functions to read and write to the arrays using this system. This is where the real beauty of the system comes in. Be sure this function is written in public scope if you want to read/write the arrays from outside this class.
To create the function that gets data from the arrays, we need two arguments: the name of the activity and the name of the property. We also want to set up this function to return a value of any type.
GOTCHA WARNING: In Actionscript 3, this won't work in static classes or functions, as it relies on the "this" keyword.
public function fetchData(act:String, prop:String):*
{
var r:*;
r = this[prop][this[act]];
return r;
}
That queer bit of code, r = this[prop][this[act]], simply uses the provided strings "act" and "prop" as the names of the constant and array, and sets the resulting value to r. Thus, if you feed the function the parameters ("maze", "highscore"), that code will essentially act like r = highscore[2] (remember, this[act] returns the integer value assigned to it.)
The writing method works essentially the same way, except we need one additional argument, the data to be written. This argument needs to be able to accept any
GOTCHA WARNING: One significant drawback to this system with strict typing languages is that you must remember the data type for the array you're writing to. The compiler cannot catch these type errors, so your program will simply throw a fatal error if it tries to write the wrong value type.
One clever way around this is to create different functions for different data types, so passing the wrong data type in an argument will trigger a compile-time error.
public function writeData(act:String, prop:String, val:*):void
{
this[prop][this[act]] = val;
}
Now, we just have one additional problem. What happens if we pass an activity or property name that doesn't exist? To protect against this, we just need one more function.
This function will validate a provided constant or variable key by attempting to access it, and catching the resulting fatal error, returning false instead. If the key is valid, it will return true.
function validateName(ID:String):Boolean
{
var checkthis:*
var r:Boolean = true;
try
{
checkthis = this[ID];
}
catch (error:ReferenceError)
{
r = false;
}
return r;
}
Now, we just need to adjust our other two functions to take advantage of this. We'll wrap the function's code inside an if statement.
If one of the keys is invalid, the function will do nothing - it will fail silently. To get around this, just put a trace (a.k.a. print) statement or a non-fatal error in the else construct.
public function fetchData(act:String, prop:String):*
{
var r:*;
if(validateName(act) && validateName(prop))
{
r = this[prop][this[act]];
return r;
}
}
public function writeData(act:String, prop:String, val:*):void
{
if(validateName(act) && validateName(prop))
{
this[prop][this[act]] = val;
}
}
Now, to use these functions, you simply need to use one line of code each. For the example, we'll say we have a text object in the GUI that shows the high score, called txtHighScore. I've omitted the necessary typecasting for the sake of the example.
//Get the high score.
txtHighScore.text = fetchData("maze", "highscore");
//Write the new high score.
writeData("maze", "highscore", txtHighScore.text);
I hope ya'll will find this tutorial useful in sorting and managing your variables.
(Afternote: You can probably do something similar with dictionaries or databases, but I prefer the flexibility with this method.)
This question has spawned out of this one. Working with lists of structs in cocoa is not simple. Either use NSArray and encode/decode, or use a C type array and lose the commodities of NSArray. Structs are supposed to be simple, but when a list is needed, one would tend to build a class instead.
When does using lists of structs make sense in cocoa?
I know there are already many questions regarding structs vs classes, and I've read users argue that it's the same answer for every language, but at least cocoa should have its own specific answers to this, if only because of KVC or bindings (as Peter suggested on the first question).
Cocoa has a few common types that are structs, not objects: NSPoint, NSRect, NSRange (and their CG counterparts).
When in doubt, follow Cocoa's lead. If you find yourself dealing with a large number of small, mostly-data objects, you might want to make them structs instead for efficiency.
Using NSArray/NSMutableArray as the top-level container, and wrapping the structs in an NSValue will probably make your life a lot easier. I would only go to a straight C-type array if you find NSArray to be a performance bottleneck, or possibly if the array is essentially read-only.
It is convenient and useful at times to use structs, especially when you have to drop down to C, such as when working with an existing library or doing system level stuff. Sometimes you just want a compact data structure without the overhead of a class. If you need many instances of such structs, it can make a real impact on performance and memory footprint.
Another way to do an array of structs is to use the NSPointerArray class. It takes a bit more thought to set up but it works pretty much just like an NSArray after that and you don't have to bother with boxing/unboxing or wrapping in a class so accessing the data is more convenient, and it doesn't take up the extra memory of a class.
NSPointerFunctions *pf = [[NSPointerFunctions alloc] initWithOptions:NSPointerFunctionsMallocMemory |
NSPointerFunctionsStructPersonality |
NSPointerFunctionsCopyIn];
pf.sizeFunction = keventSizeFunction;
self.pending = [[NSPointerArray alloc] initWithPointerFunctions:pf];
In general, the use of a struct implies the existence of a relatively simple data type that has no logic associated with it nor should have any logic associated with it. Take an NSPoint for instance - it is merely a (x,y) representation. Given this, there are also some issues that arise from it's use. In general, this is OK for this type of data as we usually observe for a change in the point rather than the y-coordinate of a point (fundamentally, (0,1) isn't the same as (1,1) shifted down by 1 unit). If this is an undesirable behavior, it may be a better idea to use a class.