Using goroutines to iterate through file indefinitely - loops

I'm new to Go so please excuse my ignorance. I'm attempting to iterate through a bunch of wordlists line by line indefinitely with goroutines. But when trying to do so, it does not iterate or stops half way through. How would I go about this in the proper manner without breaking the flow?
package main
import (
"bufio"
"fmt"
"os"
)
var file, _ = os.Open("wordlist.txt")
func start() {
scanner := bufio.NewScanner(file)
for scanner.Scan() {
fmt.Println(scanner.Text())
}
}
func main(){
for t := 0; t < 150; t++ {
go start()
fmt.Scanln()
}
}
Thank you!

You declare file as a global variable. Sharing read/write file state amongst multiple goroutines is a data race and will give you undefined results.
Most likely, reads start where the last read from any of the goroutines left off. If that's end-of-file, it likely continues to be end-of-file. But, since the results are undefined, that's not guaranteed. Your erratic results are due to undefined behavior.
Here's a revised version of your program that declares a local file variable and uses a sync.Waitgroup to synchronize the completion of all the go start() goroutines and the main goroutine. The program checks for errors.
package main
import (
"bufio"
"fmt"
"os"
"sync"
)
func start(filename string, wg *sync.WaitGroup, t int) {
defer wg.Done()
file, err := os.Open(filename)
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
lines := 0
scanner := bufio.NewScanner(file)
for scanner.Scan() {
lines++
}
if err := scanner.Err(); err != nil {
fmt.Println(err)
return
}
fmt.Println(t, lines)
}
func main() {
wg := &sync.WaitGroup{}
filename := "wordlist.txt"
for t := 0; t < 150; t++ {
wg.Add(1)
go start(filename, wg, t)
}
wg.Wait()
}

Related

Go WriteString function panicking?

func FileFill(filename string) error {
f, err := os.Open("file.txt")
if err != nil {
panic("File not opened")
}
defer f.Close()
for i := 0; i < 10; i++ {
//I know this should have some error checking here
f.WriteString("some text \n")
}
return nil
}
Hi, I'm new to learning Go and I've been trying out some small use cases to learn it a bit better. I made this function to fill 10 lines of a file with "some text". When I tried this with error checking, the program panicked at the WriteString line. Am I misunderstanding something fundamental here? I looked at the documentation and I can't figure out why it doesn't like this. Thanks.
Need to use a function with write or append permission:
package main
import "os"
func main() {
f, err := os.Create("file.txt")
if err != nil {
panic(err)
}
defer f.Close()
for range [10]struct{}{} {
f.WriteString("some text\n")
}
}
https://golang.org/pkg/os#Create
// Choose the permit you want with os.OpenFile flags
file, err := os.OpenFile(path, os.O_RDWR, 0644)
// or crate new file if not exist
file, err = os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0755)
// or append new data into your file with O_APPEND flag
file, err = os.OpenFile(path, os.O_APPEND, 0755)
docs: https://pkg.go.dev/os#OpenFile

I get an error that I don't know for what reason. a file disappears when I run the program

I'm coding in Go, and I created a file handler and a program that prints the value of that file.
However, the file that should be created with file.Filename is deleted when I run it.
I don't know what the reason is, even if I try to debug, the answer doesn't come out, and even if I google it, I don't get the answer.
(64bit windows 10 (WSL2))
package main
import (
"fmt"
"io"
"io/ioutil"
"os"
"github.com/labstack/echo"
)
func checkErr(err error) {
if err != nil {
panic(err)
}
}
func readFile(filename string) string {
data, err := ioutil.ReadFile(filename)
checkErr(err)
return string(data)
}
func main() {
e := echo.New()
e.POST("/file", func(c echo.Context) error {
file, err := c.FormFile("file")
checkErr(err)
src, err := file.Open()
checkErr(err)
defer src.Close()
dst, err := os.Create(file.Filename)
checkErr(err)
defer dst.Close()
_, err = io.Copy(dst, src)
checkErr(err)
data := readFile(file.Filename)
fmt.Println(data)
return c.String(200, "sd")
})
e.Logger.Fatal(e.Start(":5000"))
}
I'm guessing that your file exists, but the code that you wrote is reading the file before the changes are "flushed to disk".
Right here:
defer dst.Close()
_, err = io.Copy(dst, src)
Should Close() or Sync() your writer as soon as possible, otherwise you may read before the write is finished. And since your readFile() function isn't re-using the file, you might as well just close (not Sync()) it immediately, not deferred
Try this:
_, err = io.Copy(dst, src)
dst.Close()
if err != nil {
}
There could be an error while copying, but we still want to Close() the file (if there wasn't an error during the os.Create, os.Open, or os.OpenFile...

File scanner loop does not execute

I'm getting the last line of a text file, and try to read it.
get last line:
func getLastLine(file *os.File) (result int) {
s := bufio.NewScanner(file)
result = 0
for s.Scan() {
result++
}
err := s.Err()
if err != nil {
log.Fatal(err)
}
return
}
read file:
func readFileFrom(file *os.File) {
s := bufio.NewScanner(file)
for s.Scan() {
fmt.Println(s.Text())
}
err := s.Err()
if err != nil {
log.Fatal(err)
}
}
If i write this in main.go:
getLastLine(file)
readFileFrom(file)
It will not execute the block:
for s.Scan() {
fmt.Println(s.Text())
}
If I remove the line getLastLine(file), the reading works as expected.
I think it's because 2 Scanners are accessing the same file.
os.File maintains the position where the next read or write operation will work. Reading from / writing to the file updates this position.
If you use a single file, passing it to getLastLine() will read it till its end, so its pointer will point to the end of the file. Now passing it to readFileFrom() will not read and print anything because there is no more data after the end of the file (that's the definition of the "end").
You need to either rewind the pointer using File.Seek(), or you need to close and reopen it. Obviously just rewinding is more efficient. To set the pointer to the file start:
if _, err := file.Seek(0, io.SeekStart); err != nil {
panic(err)
}
So do this between the 2 function calls:
getLastLine(file)
if _, err := file.Seek(0, io.SeekStart); err != nil {
panic(err)
}
readFileFrom(file)
Also note that if you would open the file twice, you would not need to rewind it, and you could also run the 2 functions concurrently without interfering with each other, because they only read the file and each os.File has its own pointer.
file1, err := os.Open("a.txt")
// handle err
defer file1.Close()
file2, err := os.Open("a.txt")
// handle err
defer file2.Close()
wg := sync.WaitGroup()
wg.Add(1)
go func() {
defer wg.Done()
getLastLine(file1)
}()
readFileFrom(file2)
wg.Wait() // Wait for getLastLine() to complete

Goroutines sharing an array channel : trying to solve data race

I try to write a complex program with parallel goroutines. It is my first program with channels ;) Each goroutine returns an array and, unfortunately, the result is "random". If I run 10 times the program, I have 10 different results :(
This is an over simplification of my program, the results is good (maybe because it is too simple) but when I run it with -race argument, there is 4 data races.
I tried to had a close() function but it did not worked.
May you help me to find the mistake ? Thank you very much in advance !
package main
import "fmt"
import "sync"
import "strconv"
func cat_strings(a int, b string) []string{
var y []string
j := strconv.Itoa(a)
y = append(y, j)
y = append(y, b)
return y
}
func main() {
var slice []string
var wg sync.WaitGroup
var x []string
queue := make(chan []string, 10)
wg.Add(10)
for i := 0; i < 10; i++ {
go func(i int) {
defer wg.Done()
x = cat_strings(i, "var")
queue <- x
}(i)
}
//close(queue)
go func() {
defer wg.Done()
for t := range queue {
slice = append(slice, t...)
}
}()
wg.Wait()
fmt.Println(slice)
}
There's two pieces to this fix, don't share the slices between goroutines, and then range over queue synchronously in main.
import (
"fmt"
"strconv"
"sync"
)
func cat_strings(a int, b string) []string {
var y []string
j := strconv.Itoa(a)
y = append(y, j)
y = append(y, b)
return y
}
func main() {
var slice []string
var wg sync.WaitGroup
queue := make(chan []string, 10)
wg.Add(10)
for i := 0; i < 10; i++ {
go func(i int) {
defer wg.Done()
queue <- cat_strings(i, "var")
}(i)
}
go func() {
wg.Wait()
close(queue)
}()
for t := range queue {
slice = append(slice, t...)
}
fmt.Println(slice)
}
There's no reason for the extra x slice you're sharing between goroutines. If each goroutine needs another slice, define a new one for each. Sharing a single slice is always going to require extra synchronization.
The other race is between the goruoutine appending from queue to the slice slice, and the final fmt.Println. There's no reason for those be concurrent since you don't want to print until all values have been read, so finish the for-range loop entirely before printing the final value.

How to write the output of this statement into a file in Golang

I'm trying to write the output of the statement below into a text file but I can't seem to find out if there is a printf function that writes directly to a text file. For example if the code below produces the results [5 1 2 4 0 3] I would want to read this into a text file for storage and persistence. Any ideas please?
The code I want to goto the text file:
//choose random number for recipe
r := rand.New(rand.NewSource(time.Now().UnixNano()))
i := r.Perm(5)
fmt.Printf("%v\n", i)
fmt.Printf("%d\n", i[0])
fmt.Printf("%d\n", i[1])
You can use fmt.Fprintf together with an io.Writer, which would represent a handle to your file.
Here is a simple example:
func check(err error) {
if err != nil {
panic(err)
}
}
func main() {
f, err := os.Create("/tmp/yourfile")
check(err)
defer f.Close()
w := bufio.NewWriter(f)
//choose random number for recipe
r := rand.New(rand.NewSource(time.Now().UnixNano()))
i := r.Perm(5)
_, err = fmt.Fprintf(w, "%v\n", i)
check(err)
_, err = fmt.Fprintf(w, "%d\n", i[0])
check(err)
_, err = fmt.Fprintf(w, "%d\n", i[1])
check(err)
w.Flush()
}
More ways of writing to file in Go are shown here.
Note that I have used panic() here just for the sake of brevity, in the real life scenario you should handle errors appropriately (which in most cases means something other than exiting the program, what panic() does).
This example will write the values into the output.txt file.
package main
import (
"bufio"
"fmt"
"math/rand"
"os"
"time"
)
func main() {
file, err := os.OpenFile("output.txt", os.O_WRONLY|os.O_CREATE, 0666)
if err != nil {
fmt.Println("File does not exists or cannot be created")
os.Exit(1)
}
defer file.Close()
w := bufio.NewWriter(file)
r := rand.New(rand.NewSource(time.Now().UnixNano()))
i := r.Perm(5)
fmt.Fprintf(w, "%v\n", i)
w.Flush()
}
Use os package to create file and then pass it to Fprintf
file, fileErr := os.Create("file")
if fileErr != nil {
fmt.Println(fileErr)
return
}
fmt.Fprintf(file, "%v\n", i)
This should write to file.

Resources