var buf bytes.Buffer
var outputBuffer [100]byte
b := []byte(`{"Name":"Wednesday","Age":6,"Parents":["Gomez","Morticia"],"test":{"prop1":1,"prop2":[1,2,3]}}`)
w := zlib.NewWriter(&buf)
r, _ := zlib.NewReader(&buf)
w.Write(b)
w.Flush()
r.Read(outputBuffer)//cannot use outputBuffer (type [100]byte) as type []byte in function argument
fmt.Println(outputBuffer)
what can I do to make this right? thanks
well you tried to use an array as a slice. It expected a []byte and you gave it a [100]byte. A []byte has a dynamic width, while a [100]byte is always 100 bytes. An array's size is a part of its type; a [1]int is a different type from a [2]int. That's why almost everything operates on slices.
But that's not the only thing. When you call Read on an io.Reader directly, it fills in the target slice up to its current width, without expanding it. If you had made your output slice 10 bytes wide (make([]byte, 10)), the output you would see would be {"Name":"W.
var in bytes.Buffer
b := []byte(`{"Name":"Wednesday","Age":6,"Parents":["Gomez","Morticia"],"test":{"prop1":1,"prop2":[1,2,3]}}`)
w := zlib.NewWriter(&in)
w.Write(b)
w.Close()
var out bytes.Buffer
r, _ := zlib.NewReader(&in)
io.Copy(&out, r)
os.Stdout.Write(out.Bytes())
but at this point, you might as well just pass os.Stdout into io.Copy, just like they do in the standard library docs. The only difference is we have kept a copy of the output format, but... what if the output is so large that you don't want to hold it in memory? That's why io.Copy takes an interface: you can take compressed data, and write an uncompressed version of it directly to any output stream, including stdout but also including things like files, unix sockets, or network sockets.
Try one of the following:
r.Read([]byte(outputBuffer))
or
r.Read(outputBuffer[0:100])
I believe it has to do with Go's concept of slices versus fixed size arrays. You can read more about these concepts here (the sections about Arrays and Slices):
http://www.golang-book.com/6
There are other ways as well, for instance making your outputBuffer a proper slice initially instead of a fixed length array, but with the pointer above you should be able to figure it out.
Related
I am using TwinCat, when making a program to read my XML file.
I need to read an array of points in the XML file. but the amount of points can variate. Is there a method to create an array depending on the XML data.
You can't define a dynamic array in most of the Codesys based systems. However, it is possible to create functions and blocks with to handle dynamic size arrays through VAR_IN_OUT if required (TwinCAT 3 - see this link).
NOTE: Actually in newest TwinCAT 3 versions it's also possible to allocate dynamic memory with __NEW command (see this link). I haven't tested it yet but there is an example how to create a byte array size of 25 at runtime. This could be a great way but there could be some problems as you need to delete the memory used after no more needed with __DELETE function.
It's a common convention in these systems to create an array that is as big as will ever be needed. So think a theoretical maximum of how many points you will ever have in the file and create an array of that size. Then you just have to know how many points you have added for later use. This method is very safe to use, so I would suggest you to start with it instead of the __NEW.
Small simple code to help you understand:
PROGRAM PRG_XmlTest
VAR CONSTANT
MAXIMUM_ARRAY_SIZE : UINT := 9999;
END_VAR
VAR
PointArray : ARRAY[0..MAXIMUM_ARRAY_SIZE] OF REAL;
PointsInArray : UINT;
i : UINT;
END_VAR
//Deleting old data before loading
MEMSET(
destAddr := ADR(PointArray),
fillByte := 0,
n := SIZEOF(PointArray)
);
PointsInArray := 0;
//Loading your XML. NOTE: This is not a working code
WHILE Xml.HasPoints() DO
IF PointsInArray < MAXIMUM_ARRAY_SIZE THEN
PointArray[PointsInArray] := XmlReader.GetPoint();
PointsInArray := PointsInArray + 1;
ELSE
//Overflow, array is full. Do something
EXIT;
END_IF
END_WHILE
//To do something with your data.
//NOTE: Using MIN to prevent faults if PointsInArray has too large value
FOR i := 0 TO MIN(PointsInArray, MAXIMUM_ARRAY_SIZE) DO
DoSomething(PointArray[i]);
END_FOR
Say I have the following code:
a := []int{1,2,3}
i := 0
var mu = &sync.Mutex{}
for i < 10 {
go func(a *[]int) {
for _, i := range a {
mu.Lock()
fmt.Println(a[0])
mu.Unlock()
}
}(&a)
i++
}
The array is a shared resource and is being read from in the loop. How do I protect the array in the loop header and do I need to? Also is it necessary to pass the array to the goroutine as a pointer?
First, some Go terminology:
[]int{1, 2, 3} is a slice, not an array. An array would be written as [...]int{1, 2, 3}.
A slice is a triplet of (start, length, capacity) and points to an underlying array (usually heap-allocated, but this is an implementation detail that the language completely hides from you!)
Go's memory model allows any number of readers or (but not and) at most one writer to any given region in memory. The Go memory model (unfortunately) doesn't specifically call out the case of accessing multiple indices into the same slice concurrently, but it appears to be fine to do so (i.e. they are treated as distinct locations in memory, as would be expected).
So if you're just reading from it, it is not necessary to protect it at all.
If you're reading and writing to it, but the goroutines don't read and write to the same places as each other (for example, if goroutine i only reads and writes to position i) then you also don't need synchronization. Moreover, you could either synchronize the entire slice (which means fewer mutexes, but much higher contention) or you could synchronize the individual positions in the slice (which means much lower contention but many more mutexes and locks acquired and released).
But since Go allows functions to capture variables in scope (that is, they are closures) there's really no reason to pass the array as a pointer at all:
Your code would thus be most idiomatically be written as:
a := []int{1,2,3}
for i := 0; i < 10; i++
for i < 10 {
go func() {
for _, i := range a {
fmt.Println(a[0])
}
}()
}
I'm not really sure what the above code is supposed to be for- since it's going to print out a[0] 10 times in various goroutines, which makes it look like it's not even using the slice in a meaningful way.
First you shuold know a := []int{1,2,3} is not an array, it is a slice.
A slice literal is like an array literal without the length.
This is an array literal:
[3]bool{true, true, false}
And this creates the same array as above, then builds a slice that
references it:
[]bool{true, true, false}
Types with empty [], such as []int are actually slices, not arrays. In Go, the size of an array is part of the type, so to actually have an array you would need to have something like [16]int, and the pointer to that would be *[16]int.
Q: is it necessary to pass the array to the goroutine as a pointer?
A: No. From https://golang.org/doc/effective_go.html#slices
If a function takes a slice argument, changes it makes to the elements
of the slice will be visible to the caller, analogous to passing a
pointer to the underlying array.
I am trying to write a Go program which uses mmap to map a very large file containing float32 values into memory. Here is my attempt (inspired by a previous answer, error handling omitted for brevity):
package main
import (
"fmt"
"os"
"syscall"
"unsafe"
)
func main() {
fileName := "test.dat"
info, _ := os.Stat(fileName)
fileSize := info.Size()
n := int(fileSize / 4)
mapFile, _ := os.Open(fileName)
defer mapFile.Close()
mmap, _ := syscall.Mmap(int(mapFile.Fd()), 0, int(fileSize),
syscall.PROT_READ, syscall.MAP_SHARED)
defer syscall.Munmap(mmap)
mapArray := (*[n]float32)(unsafe.Pointer(&mmap[0]))
for i := 0; i < n; i++ {
fmt.Println(mapArray[i])
}
}
This fails with the following error message:
./main.go:21: non-constant array bound n
Since n is determined by the length of the file (not known at compile time), I cannot replace n with a constant value in the cast. How do I convert mmap into an array (or slice) of float32 values?
You first convert to an array of a type with a static length that can fit your data, then slice that array to the correct length and capacity.
mapSlice := (*[1 << 30]float32)(unsafe.Pointer(&mmap[0]))[:n:n]
Unfortunately you can't get a pointer to an array in your case. This is because n is not a constant value (i.e. it's determined at runtime with fileSize/4). (Note If fileSize were constant, you could get an array.)
There are safe and unsafe alternatives though.
The safe, or some might call the "right" way -- this requires a copy, but you have control over the endianness. Here's an example:
import (
"encoding/binary"
"bytes"
"unsafe" // optional
)
const SIZE_FLOAT32 = unsafe.Sizeof(float32(0)) // or 4
bufRdr := bytes.NewReader(mmap)
mapSlice := make([]float32, len(mmap)/SIZE_FLOAT32) // = fileSize/4
err := binary.Read(bufRdr, binary.LittleEndian, mapSlice) // could pass &mapSlice instead of mapSlice: same result.
// mapSlice now can be used like the mapArray you wanted.
There are a couple ways to do this unsafely, but with Go 1.17 it's pretty simple.
mapSlice := unsafe.Slice((*float32)(unsafe.Pointer(&mmap[0])), len(mmap)/SIZE_FLOAT32)
You could also use reflect.SliceHeader. There are lots of nuances to be careful of here to prevent garbage collector issues:
var mapSlice []float32 // mapSlice := []float32{} also works (important thing is that len and cap are 0)
// newSh and oldSh are here for readability (i.e. inlining these variables is ok, but makes things less readable IMO)
newSh := (*reflect.SliceHeader)(unsafe.Pointer(&mapSlice))
oldSh := (*reflect.SliceHeader)(unsafe.Pointer(&mmap))
// Note: order of assigning Data, Cap, Len is important (due to GC)
newSh.Data = oldSh.Data
newSh.Cap = oldSh.Cap/SIZE_FLOAT32
newSh.Len = oldSh.Len/SIZE_FLOAT32
runtime.KeepAlive(mmap) // ensure `mmap` is not freed up until this point.
The final unsafe way I can think of is given in #JimB's answer -- cast an mmap's Data to an unsafe.Pointer, then cast that to an arbitrarily large pointer to array, and then finally slice that array specifying to desired size and capacity.
Some time ago I needed to convert Array of Byte type to TBytes, which was done with help of delphi gurus here on SO;
Specifically, I needed to convert data so that I could extract what UDPServer gave me on ServerUDPRead in Indy 10.
This function was made by #David Heffernan, and is posted on this topic: Delphi XE3 indy compatibility issue between tbytes and tidbytes
So, I'm using
function CopyBytes(const Bytes: array of Byte): TBytes;
var
Count: Integer;
begin
Count := Length(Bytes);
SetLength(Result, Count);
if Count > 0 then
Move(Bytes[0], Result[0], Length(Bytes));
end;
to convert this to TBytes and as one I can then send this type via Client UDP SendBuffer.
However, I need to make some modifications to data between they are forwarded;
I read first line of array of byte which the ServerUDPRead delivers, to a string with this:
var FirstString: string;
FirstString := PAnsiChar(#AData[0]);
Where AData is array of byte;
Now, how could I do reverse conversion, so that I can put my own string to this AData array of byte instead of the one that is currently there, but without modifying any other data inside of the array, and then convert it to TBytes?
Is there a way to put something like AData[0]:=PAnsiChar(mystring); (this one is wrong, of course...) and then convert it with CopyBytes to TBytes, or maybe convert first to TBytes and then replace it there...?
Either way would be useful.
Indy has BytesToString() and ToBytes() functions for converting between String and TIdBytes. And if you are using Indy 10.6+, the IIdTextEncoding interface has GetBytes() and GetString() overloads that take PByte parameters, so you can use TBytes and array of Byte arrays without having to copy to/from TIdBytes.
I am trying to write an application that reads RPM files. The start of each block has a Magic char of [4]byte.
Here is my struct
type Lead struct {
Magic [4]byte
Major, Minor byte
Type uint16
Arch uint16
Name string
OS uint16
SigType uint16
}
I am trying to do the following:
lead := Lead{}
lead.Magic = buffer[0:4]
I am searching online and not sure how to go from a slice to an array (without copying). I can always make the Magic []byte (or even uint64), but I was more curious on how would I go from type []byte to [4]byte if needed to?
The built in method copy will only copy a slice to a slice NOT a slice to an array.
You must trick copy into thinking the array is a slice
copy(varLead.Magic[:], someSlice[0:4])
Or use a for loop to do the copy:
for index, b := range someSlice {
varLead.Magic[index] = b
}
Or do as zupa has done using literals. I have added onto their working example.
Go Playground
You have allocated four bytes inside that struct and want to assign a value to that four byte section. There is no conceptual way to do that without copying.
Look at the copy built-in for how to do that.
Try this:
copy(lead.Magic[:], buf[0:4])
Tapir Liui (auteur de Go101) twitte:
Go 1.18 1.19 1.20 will support conversions from slice to array: golang/go issues 46505.
So, since Go 1.18,the slice copy2 implementation could be written as:
*(*[N]T)(d) = [N]T(s)
or, even simpler if the conversion is allowed to present as L-values:
[N]T(d) = [N]T(s)
Without copy, you can convert, with the next Go 1.17 (Q3 2021) a slice to an array pointer.
This is called "un-slicing", giving you back a pointer to the underlying array of a slice, again, without any copy/allocation needed:
See golang/go issue 395: spec: convert slice x into array pointer, now implemented with CL 216424/, and commit 1c26843
Converting a slice to an array pointer yields a pointer to the underlying array of the slice.
If the length of the slice is less than the length of the array,
a run-time panic occurs.
s := make([]byte, 2, 4)
s0 := (*[0]byte)(s) // s0 != nil
s2 := (*[2]byte)(s) // &s2[0] == &s[0]
s4 := (*[4]byte)(s) // panics: len([4]byte) > len(s)
var t []string
t0 := (*[0]string)(t) // t0 == nil
t1 := (*[1]string)(t) // panics: len([1]string) > len(s)
So in your case, provided Magic type is *[4]byte:
lead.Magic = (*[4]byte)(buffer)
Note: type aliasing will work too:
type A [4]int
var s = (*A)([]int{1, 2, 3, 4})
Why convert to an array pointer? As explained in issue 395:
One motivation for doing this is that using an array pointer allows the compiler to range check constant indices at compile time.
A function like this:
func foo(a []int) int
{
return a[0] + a[1] + a[2] + a[3];
}
could be turned into:
func foo(a []int) int
{
b := (*[4]int)(a)
return b[0] + b[1] + b[2] + b[3];
}
allowing the compiler to check all the bounds once only and give compile-time errors about out of range indices.
Also:
One well-used example is making classes as small as possible for tree nodes or linked list nodes so you can cram as many of them into L1 cache lines as possible.
This is done by each node having a single pointer to a left sub-node, and the right sub-node being accessed by the pointer to the left sub-node + 1.
This saves the 8-bytes for the right-node pointer.
To do this you have to pre-allocate all the nodes in a vector or array so they're laid out in memory sequentially, but it's worth it when you need it for performance.
(This also has the added benefit of the prefetchers being able to help things along performance-wise - at least in the linked list case)
You can almost do this in Go with:
type node struct {
value int
children *[2]node
}
except that there's no way of getting a *[2]node from the underlying slice.
Go 1.20 (Q1 2023): this is addressed with CL 430415, 428938 (type), 430475 (reflect) and 429315 (spec).
Go 1.20
You can convert from a slice to an array directly with the usual conversion syntax T(x). The array's length can't be greater than the slice's length:
func main() {
slice := []int64{10, 20, 30, 40}
array := [4]int64(slice)
fmt.Printf("%T\n", array) // [4]int64
}
Go 1.17
Starting from Go 1.17 you can directly convert a slice to an array pointer. With Go's type conversion syntax T(x) you can do this:
slice := make([]byte, 4)
arrptr := (*[4]byte)(slice)
Keep in mind that the length of the array must not be greater than the length of the slice, otherwise the conversion will panic.
bad := (*[5]byte)(slice) // panics: slice len < array len
This conversion has the advantage of not making any copy, because it simply yields a pointer to the underlying array.
Of course you can dereference the array pointer to obtain a non-pointer array variable, so the following also works:
slice := make([]byte, 4)
var arr [4]byte = *(*[4]byte)(slice)
However dereferencing and assigning will subtly make a copy, since the arr variable is now initialized to the value that results from the conversion expression. To be clear (using ints for simplicity):
v := []int{10,20}
a := (*[2]int)(v)
a[0] = 500
fmt.Println(v) // [500 20] (changed, both point to the same backing array)
w := []int{10,20}
b := *(*[2]int)(w)
b[0] = 500
fmt.Println(w) // [10 20] (unchanged, b holds a copy)
One might wonder why the conversion checks the slice length and not the capacity (I did). Consider the following program:
func main() {
a := []int{1,2,3,4,5,6}
fmt.Println(cap(a)) // 6
b := a[:3]
fmt.Println(cap(a)) // still 6
c := (*[3]int)(b)
ptr := uintptr(unsafe.Pointer(&c[0]))
ptr += 3 * unsafe.Sizeof(int(0))
i := (*int)(unsafe.Pointer(ptr))
fmt.Println(*i) // 4
}
The program shows that the conversion might happen after reslicing. The original backing array with six elements is still there, so one might wonder why a runtime panic occurs with (*[6]int)(b) where cap(b) == 6.
This has actually been brought up. It's worth to remember that, unlike slices, an array has fixed size, therefore it needs no notion of capacity, only length:
a := [4]int{1,2,3,4}
fmt.Println(len(a) == cap(a)) // true
You might be able to do the whole thing with one read, instead of reading individually into each field. If the fields are fixed-length, then you can do:
lead := Lead{}
// make a reader to dispense bytes so you don't have to keep track of where you are in buffer
reader := bytes.NewReader(buffer)
// read into each field in Lead, so Magic becomes buffer[0:4],
// Major becomes buffer[5], Minor is buffer[6], and so on...
binary.Read(reader, binary.LittleEndian, &lead)
Don't. Slice itself is suffice for all purpose. Array in go lang should be regarded as the underlying structure of slice. In every single case, use only slice. You don't have to array yourself. You just do everything by slice syntax. Array is only for computer. In most cases, slice is better, clear in code. Even in other cases, slice still is sufficient to reflex your idea.