Max size of slice of string in Go - arrays

I want to input 200000 space separated strings of arbitary numbers. When taking the input using bufio.Reader it only take a few of them. Here is the code:
package main
import (
"bufio"
"fmt"
"io"
"os"
"strings"
)
func main() {
reader := bufio.NewReaderSize(os.Stdin, 1024*1024)
scoresTemp := strings.Split(readLine(reader), " ")
fmt.Println(scoresTemp)
fmt.Println("---")
fmt.Println(len(scoresTemp))
}
func readLine(reader *bufio.Reader) string {
str, _, err := reader.ReadLine()
if err == io.EOF {
return ""
}
return strings.TrimRight(string(str), "\r\n")
}
The slice length should be 200000, but it only take 410 items. If I increase the size of reader, it would be the same. What is the max size of slice of strings in Go?, How to work on it?

I believe you have an issue with your input rather than on your Go code. I've tried your code on my local machine and got this result:
$ for((i=0;i<200000;i++)) do echo -n "x "; done | go run main.go
...
---
200001

Your code doesn't iterate on the input very well. The problem isn't with slices.
Try the following code if it does what you want.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
inputScanner := bufio.NewScanner(os.Stdin)
inputScanner.Split(bufio.ScanWords)
scoresTemp := make([]string, 0, 200000)
for inputScanner.Scan() {
scoresTemp = append(scoresTemp, inputScanner.Text())
}
fmt.Println(scoresTemp)
fmt.Println("---")
fmt.Println(len(scoresTemp))
}
For the explanation:
bufio.Scanner helps "scanning" a certain input and splitting it in whichever way you like. By default it splits be new lines. Which brings us to number 2.
bufio.ScanWords is a function that splits bytes on spaces (including new-lines). This defines the behaviour of inputScanner.
Next comes the slice of strings where we store our data. It's initialized with 0 elements, and capacity of 200,000 strings. This optimizes allocation times.
PRINT !!

Related

How to declare a 2d array of unknown size

I did this for the implementation.
csvData := make([][]string, 100)
for i := range csvData {
csvData[i] = make([]string, 100)
}
But I want to remove 100 since I don't know the actual size of the array. I want to push some string type elements into the 2d array. Basically I am reading a CSV file and want to extract its elements and store them in a 2d array. What would be a simple way to do so?
Here a working example, it creates a 10 by 10 slice.
package main
import (
"fmt"
)
func main() {
myData := "data"
var nestedSlice [][]string
nestedSize := 10
for i:=0;i<nestedSize; i++ {
jSlice := []string{}
for j:=0;j<nestedSize; j++ {
jSlice = append(jSlice, fmt.Sprintf("row:%v-col:%v-%v", i, j, myData))
}
nestedSlice = append(nestedSlice, jSlice)
}
fmt.Printf("result: %v\n", nestedSlice)
}
And here is the output showing the 10x10 slice.
result: [[row:0-col:0-data row:0-col:1-data row:0-col:2-data row:0-col:3-data row:0-col:4-data row:0-col:5-data row:0-col:6-data row:0-col:7-data row:0-col:8-data row:0-col:9-data] [row:1-col:0-data row:1-col:1-data row:1-col:2-data row:1-col:3-data row:1-col:4-data row:1-col:5-data row:1-col:6-data row:1-col:7-data row:1-col:8-data row:1-col:9-data] [row:2-col:0-data row:2-col:1-data row:2-col:2-data row:2-col:3-data row:2-col:4-data row:2-col:5-data row:2-col:6-data row:2-col:7-data row:2-col:8-data row:2-col:9-data] [row:3-col:0-data row:3-col:1-data row:3-col:2-data row:3-col:3-data row:3-col:4-data row:3-col:5-data row:3-col:6-data row:3-col:7-data row:3-col:8-data row:3-col:9-data] [row:4-col:0-data row:4-col:1-data row:4-col:2-data row:4-col:3-data row:4-col:4-data row:4-col:5-data row:4-col:6-data row:4-col:7-data row:4-col:8-data row:4-col:9-data] [row:5-col:0-data row:5-col:1-data row:5-col:2-data row:5-col:3-data row:5-col:4-data row:5-col:5-data row:5-col:6-data row:5-col:7-data row:5-col:8-data row:5-col:9-data] [row:6-col:0-data row:6-col:1-data row:6-col:2-data row:6-col:3-data row:6-col:4-data row:6-col:5-data row:6-col:6-data row:6-col:7-data row:6-col:8-data row:6-col:9-data] [row:7-col:0-data row:7-col:1-data row:7-col:2-data row:7-col:3-data row:7-col:4-data row:7-col:5-data row:7-col:6-data row:7-col:7-data row:7-col:8-data row:7-col:9-data] [row:8-col:0-data row:8-col:1-data row:8-col:2-data row:8-col:3-data row:8-col:4-data row:8-col:5-data row:8-col:6-data row:8-col:7-data row:8-col:8-data row:8-col:9-data] [row:9-col:0-data row:9-col:1-data row:9-col:2-data row:9-col:3-data row:9-col:4-data row:9-col:5-data row:9-col:6-data row:9-col:7-data row:9-col:8-data row:9-col:9-data]]
Depending on your use case there might be ways to optimise the for loop so its not n^2 time complexity but that's a different topic completely.
When creating an array or slice backed by new array you must supply a size, otherwise there is no way for the compiler to know how large to make it. But when declaring a variable to hold an array created elsewhere, just declare the var as a slice, type [][]string, which is the type of cvsData in your example.

Count similar array value

I'm trying to learn Go (or Golang) and can't seem to get it right. I have 2 texts files, each containing a list of words. I'm trying to count the amount of words that are present in both files.
Here is my code so far :
package main
import (
"fmt"
"log"
"net/http"
"bufio"
)
func stringInSlice(str string, list []string) bool {
for _, v := range list {
if v == str {
return true
}
}
return false
}
func main() {
// Texts URL
var list = "https://gist.githubusercontent.com/alexcesaro/c9c47c638252e21bd82c/raw/bd031237a56ae6691145b4df5617c385dffe930d/list.txt"
var url1 = "https://gist.githubusercontent.com/alexcesaro/4ebfa5a9548d053dddb2/raw/abb8525774b63f342e5173d1af89e47a7a39cd2d/file1.txt"
//Create storing arrays
var buffer [2000]string
var bufferUrl1 [40000]string
// Set a sibling counter
var sibling = 0
// Read and store text files
wordList, err := http.Get(list)
if err != nil {
log.Fatalf("Error while getting the url : %v", err)
}
defer wordList.Body.Close()
wordUrl1, err := http.Get(url1)
if err != nil {
log.Fatalf("Error while getting the url : %v", err)
}
defer wordUrl1.Body.Close()
streamList := bufio.NewScanner(wordList.Body)
streamUrl1 := bufio.NewScanner(wordUrl1.Body)
streamList.Split(bufio.ScanLines)
streamUrl1.Split(bufio.ScanLines)
var i = 0;
var j = 0;
//Fill arrays with each lines
for streamList.Scan() {
buffer[i] = streamList.Text()
i++
}
for streamUrl1.Scan() {
bufferUrl1[j] = streamUrl1.Text()
j++
}
//ERROR OCCURRING HERE :
// This code if i'm not wrong is supposed to compare through all the range of bufferUrl1 -> bufferUrl1 values with buffer values, then increment sibling and output FIND
for v := range bufferUrl1{
if stringInSlice(bufferUrl1, buffer) {
sibling++
fmt.Println("FIND")
}
}
// As a testing purpose thoses lines properly paste both array
// fmt.Println(buffer)
// fmt.Println(bufferUrl1)
}
But right now, my build doesn't even succeed. I'm only greeted with this message:
.\hello.go:69: cannot use bufferUrl1 (type [40000]string) as type string in argument to stringInSlice
.\hello.go:69: cannot use buffer (type [2000]string) as type []string in argument to stringInSlice
bufferUrl1 is an array: [4000]string. You meant to use v (each
string in bufferUrl1). But in fact, you meant to use the second
variable—the first variable is the index which is ignored in the code
below using _.
type [2000]string is different from []string. In Go, arrays and slices are not the same. Read Go Slices: usage and internals. I've changed both variable declarations to use slices with the same initial length using make.
These are changes you need to make to compile.
Declarations:
// Create storing slices
buffer := make([]string, 2000)
bufferUrl1 := make([]string, 40000)
and the loop on Line 69:
for _, s := range bufferUrl1 {
if stringInSlice(s, buffer) {
sibling++
fmt.Println("FIND")
}
}
As a side-note, consider using a map instead of a slice for buffer for more efficient lookup instead of looping through the list in stringInSlice.
https://play.golang.org/p/UcaSVwYcIw has the fix for the comments below (you won't be able to make HTTP requests from the Playground).

Jump to specific line in file in Go

In Go is it possible to jump to particular line number in a file and delete it? Something like linecache in python.
I'm trying to match some substrings in a file and remove the corresponding lines. The matching part I've taken care of and I have an array with line numbers I need to delete but I'm stuck on how to delete the matching lines in the file.
This is an old question, but if anyone is looking for a solution I wrote a package that handles going to any line in a file. Link here. It can open a file and seek to any line position without reading the whole file into memory and splitting.
import "github.com/stoicperlman/fls"
// This is just a wrapper around os.OpenFile. Alternatively
// you could open from os.File and use fls.LineFile(file) to get f
f, err := fls.OpenFile("test.log", os.O_CREATE|os.O_WRONLY, 0600)
defer f.Close()
// return begining line 1/begining of file
// equivalent to f.Seek(0, io.SeekStart)
pos, err := f.SeekLine(0, io.SeekStart)
// return begining line 2
pos, err := f.SeekLine(1, io.SeekStart)
// return begining of last line
pos, err := f.SeekLine(0, io.SeekEnd)
// return begining of second to last line
pos, err := f.SeekLine(-1, io.SeekEnd)
Unfortunately I'm not sure how you would delete, this just handles getting you to the correct position in the file. For your case you could use it to go to the line you want to delete and save the position. Then seek to the next line and save that as well. You now have the bookends of the line to delete.
// might want lineToDelete - 1
// this acts like 0 based array
pos1, err := f.SeekLine(lineToDelete, io.SeekStart)
// skip ahead 1 line
pos2, err := f.SeekLine(1, io.SeekCurrent)
// pos2 will be the position of the first character in next line
// might want pos2 - 1 depending on how the function works
DeleteBytesFromFileFunction(f, pos1, pos2)
Based on my read of the linecache module it takes a file and explodes it into an array based on '\n' line endings. You could replicate the same behavior in Go by using strings or bytes. You could also use the bufio library to read a file a line by line and only store or save the lines you want.
package main
import (
"bytes"
"fmt"
)
import "io/ioutil"
func main() {
b, e := ioutil.ReadFile("filename.txt")
if e != nil {
panic(e)
}
array := bytes.Split(b, []byte("\n"))
fmt.Printf("%v", array)
}
I wrote a small function that allowing you remove from a file a specific line.
package main
import (
"io/ioutil"
"os"
"strings"
)
func main() {
path := "path/to/file.txt"
removeLine(path, 2)
}
func removeLine(path string, lineNumber int) {
file, err := ioutil.ReadFile(path)
if err != nil {
panic(err)
}
info, _ := os.Stat(path)
mode := info.Mode()
array := strings.Split(string(file), "\n")
array = append(array[:lineNumber], array[lineNumber+1:]...)
ioutil.WriteFile(path, []byte(strings.Join(array, "\n")), mode)
}

Golang How to read input filename in Go

I would like to run my go file on my input.txt where my go program will read the input.txt file when I type in go run command ie:
go run goFile.go input.txt
I don't want to put input.txt in my goFile.go code since my go file should run on any input name not just input.txt.
I try ioutil.ReadAll(os.Stdin) but I need to change my command to
go run goFile.go < input.txt
I only use package fmt, os, bufio and io/ioutil. Is it possible to do it without any other packages?
Please take a look at the package documentation of io/ioutil which you are already using.
It has a function exactly for this: ReadFile()
func ReadFile(filename string) ([]byte, error)
Example usage:
func main() {
// First element in os.Args is always the program name,
// So we need at least 2 arguments to have a file name argument.
if len(os.Args) < 2 {
fmt.Println("Missing parameter, provide file name!")
return
}
data, err := ioutil.ReadFile(os.Args[1])
if err != nil {
fmt.Println("Can't read file:", os.Args[1])
panic(err)
}
// data is the file content, you can use it
fmt.Println("File content is:")
fmt.Println(string(data))
}
Firs you check for the provided argument. If the first argument satisfy the condition of an input file, then you use the ioutil.ReadFile method, providing parameter the os.Args result.
package main
import (
"fmt"
"os"
"io/ioutil"
)
func main() {
if len(os.Args) < 1 {
fmt.Println("Usage : " + os.Args[0] + " file name")
os.Exit(1)
}
file, err := ioutil.ReadFile(os.Args[1])
if err != nil {
fmt.Println("Cannot read the file")
os.Exit(1)
}
// do something with the file
fmt.Print(string(file))
}
Another possibility is to use:
f, err := os.Open(os.Args[0])
but for this you need to provide the bytes lenght to read:
b := make([]byte, 5) // 5 is the length
n, err := f.Read(b)
fmt.Printf("%d bytes: %s\n", n, string(b))
For running .go file from command-line by input parameter like file (for example abc.txt).We need use mainly os, io/ioutil, fmt packages. Additionally for reading command line parameters we use
os.Args Here is example code
package main
import (
"fmt"
"os"
"io/ioutil"
)
func main() {
fmt.Println(" Hi guys ('-') ")
input_files := os.Args[1:]
//input_files2 := os.Args[0];
//fmt.Println("if2 : ",input_files2)
if len(input_files) < 1{
fmt.Println("Not detected files.")
}else{
fmt.Println("File_name is : ",input_files[0])
content, err := ioutil.ReadFile(input_files[0])
if err != nil {
fmt.Println("Can't read file :", input_files[0],"Error : ",err)
}else {
fmt.Println("Output file content is(like string type) : \n",string(content))//string Output
fmt.Println("Output file content is(like byte type) : \n",content)//bytes Output
}
}
}
Args holds command line arguments, including the command as Args[0].
If the Args field is empty or nil, Run uses {Path}.
In typical use, both Path and Args are set by calling Command.
Args []string
function. This function return back array on string type https://golang.org/pkg/os/exec/.Args hold the command-line arguments, starting with the program name. In this case short way to take filename from command-line is this functions os.Args[1:] . And here is output
elshan_abd$ go run main.go abc.txt
Hi guys ('-')
File_name is : abc.txt
Output file content is(like string type) :
aaa
bbb
ccc
1234
Output file content is(like byte type) :
[97 97 97 10 98 98 98 10 99 99 99 10 49 50 51 52 10]
Finally we need for reading content file this function
func ReadFile(filename string) ([]byte, error) source is https://golang.org/pkg/io/ioutil/#ReadFile

Convert string to array of single digit integers in Go

Spent several hours trying to find a solution to the 'simple' problem of converting a string of digits to an array of single digit integers in Go. Tried many different approaches, but always ran into a problem. Here is the last one tried. It builds, but gives index out of range at runtime, on the line indicated. There will probably be a few AHs that will mark me down for asking stupid questions, but I couldn't find a solution in the first 50 results of multiple Google searches. So come on dude, mark me down, you're the man. To the other 99%: Thanks for your patience and help.
package main
import (
"fmt"
"strconv"
"strings"
)
func main() {
s := "876567747896354336739443262"
var dstr []string = strings.SplitAfterN(s,"",len(s))
var dint []int
for i := 0; i < len(s); i++ {
dint[i], _ = strconv.Atoi(dstr[i]) //index out of range at runtime
fmt.Printf("dstr[%v] is: %s\n", i, dstr[i])
fmt.Printf("dint[%v] is: %v\n", i, dint[i])
}
}
Here is the answer. Morphiax showed me the error in my ways. Thanks dude, appreciated. It turns out that this is wrong: var dint []int . I should have given the size when creating the array, when not initializing it: var dint [27]int . Here's full code again:
package main
import (
"fmt"
"strconv"
"strings"
)
func main() {
s := "876567747896354336739443262"
var dstr []string = strings.SplitAfterN(s,"",len(s))
var dint [27]int
for i := 0; i < len(s); i++ {
dint[i], _ = strconv.Atoi(dstr[i])
fmt.Printf("dstr[%v] is: %s\n", i, dstr[i])
fmt.Printf("dint[%v] is: %v\n", i, dint[i])
}
}

Resources