Is there any way to make goroutine execute one after another ( one by one) if it was the same function?
I didn't mean to use goroutine firstly. However, "os/exec" function in TCP will cause the tcp forced to stop. Thus I use goroutine to avoid crashing. But I still want them to execute by order, instead of concurrently. Here is my code.
func handleTCP(conn net.Conn) {
defer conn.Close()
fmt.Println("handle TCP function")
for {
wg := new(sync.WaitGroup)
wg.Add(1)
go func() {
cmdArgs := []string{temp_str, test_press, gh, "sample.csv"}
cmd := exec.Command("calib.exe", cmdArgs...)
wg.Done()
}()
}
}
You can use locks to limit access to a resource to one thread at a time as follows
i := 1
iLock = &sync.Mutex{}
for {
go func() {
iLock.Lock()
fmt.Printf("Value of i: %d\n", i)
iLock.Unlock()
}
}
More examples here
Not sure what you are using the WaitGroup for. If it is to achieve the sequentiality you desire, then it can be removed.
Try to put the lock into function, it makes theirs execution sequential. But remember about wg.Done() must be under defer at first line of the function.
Something like this:
var mu sync.Mutex
func handleTCP(conn net.Conn) {
defer conn.Close()
fmt.Println("handle TCP function")
for {
wg := new(sync.WaitGroup)
wg.Add(1)
go func() {
defer wg.Done()
mu.Lock()
defer mu.UnLock()
cmdArgs := []string{temp_str, test_press, gh, "sample.csv"}
cmd := exec.Command("calib.exe", cmdArgs...)
}()
}
}
Related
I'm new to Go so please excuse my ignorance. I'm attempting to iterate through a bunch of wordlists line by line indefinitely with goroutines. But when trying to do so, it does not iterate or stops half way through. How would I go about this in the proper manner without breaking the flow?
package main
import (
"bufio"
"fmt"
"os"
)
var file, _ = os.Open("wordlist.txt")
func start() {
scanner := bufio.NewScanner(file)
for scanner.Scan() {
fmt.Println(scanner.Text())
}
}
func main(){
for t := 0; t < 150; t++ {
go start()
fmt.Scanln()
}
}
Thank you!
You declare file as a global variable. Sharing read/write file state amongst multiple goroutines is a data race and will give you undefined results.
Most likely, reads start where the last read from any of the goroutines left off. If that's end-of-file, it likely continues to be end-of-file. But, since the results are undefined, that's not guaranteed. Your erratic results are due to undefined behavior.
Here's a revised version of your program that declares a local file variable and uses a sync.Waitgroup to synchronize the completion of all the go start() goroutines and the main goroutine. The program checks for errors.
package main
import (
"bufio"
"fmt"
"os"
"sync"
)
func start(filename string, wg *sync.WaitGroup, t int) {
defer wg.Done()
file, err := os.Open(filename)
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
lines := 0
scanner := bufio.NewScanner(file)
for scanner.Scan() {
lines++
}
if err := scanner.Err(); err != nil {
fmt.Println(err)
return
}
fmt.Println(t, lines)
}
func main() {
wg := &sync.WaitGroup{}
filename := "wordlist.txt"
for t := 0; t < 150; t++ {
wg.Add(1)
go start(filename, wg, t)
}
wg.Wait()
}
In the below code:
package main
import "fmt"
func main() {
for i := 0; i <= 9; i++ {
go func() {
fmt.Println(i)
}()
}
}
Output:
code$
code$ go install github.com/myhub/cs61a
code$ bin/cs61a
code$
Above program does not provide any output.
1) Is their a data race for single memory location i among 10 go-routines?
2) Why above code does not print value of free variable i?
Is there a data race?
Yes, confirm it by running go run -race example.go. The main goroutine writes i, and the other goroutines read it without any synchronization. See Passing parameters to function closure; Why do these two for loop variations give me different behavior? and Register multiple routes using range for loop slices/map.
Why above code does not print anything?
Because when the main goroutine ends, your program ends as well. It does not wait for other non-main goroutines to finish. See No output from goroutine
Fix
Make a copy of the loop variable, and use that in the closures, and use a sync.WaitGroup to wait for the launched goroutines to end:
var wg sync.WaitGroup
for i := 0; i <= 9; i++ {
i2 := i
wg.Add(1)
go func() {
defer wg.Done()
fmt.Println(i2)
}()
}
wg.Wait()
This will output (try it on the Go Playground):
9
0
2
1
6
5
3
7
8
4
An alternative is to pass i as a parameter to the launched function:
var wg sync.WaitGroup
for i := 0; i <= 9; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
fmt.Println(i)
}(i)
}
wg.Wait()
Try this one on the Go Playground.
I have some .log files and want to monitor if any data is appended to any of them to collect that data into a DB.
How do I open an opened-for-writing file and how do I monitor for new lines/changes?
Thank you.
I wrote this small piece of code that will monitor for file change and if detected, run the function you passed to it.
func watchFileAndRun(filePath string, fn func()) error {
defer func() { r := recover(); if r != nil { logCore("ERROR","Error:watching file:", r) } }()
fn()
initialStat, err := os.Stat(filePath)
checkErr(err)
for {
stat, err := os.Stat(filePath)
checkErr(err)
if stat.Size() != initialStat.Size() || stat.ModTime() != initialStat.ModTime() {
fn()
initialStat, err = os.Stat(filePath)
checkErr(err)
}
time.Sleep(10 * time.Second)
}
return nil
}
To call this function where loadEnv is a function:
go watchFileAndRun("config.json", loadEnv)
Some Notes:
go watchFileAndRun() will run the routing asynchronously. IE in the background.
checkErr is my own function you can simply put a standard
if err != nil {fmt.printf("Something went wrong",err)} in its place.
and the defer first line is to prevent the program crashing the main if something went wrong. You could nest it deeper if you wanted to make this function more resilient.
Hope it helps someone. and its smaller than the nsnotify plug in which to be frank I could get working either.
I'm using Gorilla to enable session variables on Google App Engine. So far I imported "github.com/gorilla/sessions" only, but Gorilla's page says:
If you aren't using gorilla/mux, you need to wrap your handlers with
context.ClearHandler as or else you will leak memory! An easy way to
do this is to wrap the top-level mux when calling http.ListenAndServe:
http.ListenAndServe(":8080",
context.ClearHandler(http.DefaultServeMux))
My question is how do I adapt this for the App Engine Standard Environment, which as far as I know does not use http.ListenAndServe by default. My code looks like this:
package test
import (
"fmt"
"net/http"
"github.com/gorilla/sessions"
)
func init() {
http.HandleFunc("/", handler)
}
func handler(w http.ResponseWriter, r *http.Request) {
var cookiestore = sessions.NewCookieStore([]byte("somesecret"))
session, _ := cookiestore.Get(r, "session")
session.Values["foo"] = "bar"
fmt.Fprintf(w, "session value is %v", session.Values["foo"])
}
Would I get a memory leak this way?
The doc you quoted tells everything you need to do: wrap your handlers using context.ClearHandler(). Since you "only" have a handler function and not an http.Handler, you may use the http.HandlerFunc adapter to get a value that implements http.Handler:
func init() {
http.Handle("/", context.ClearHandler(http.HandlerFunc(handler)))
}
You need to do this for every handler you register. That's why the doc mentions that it's easier to just wrap the root handler you pass to http.ListenAndServe() (and that way you don't have to wrap the other, non-top-level handlers). But this isn't the only way, just the easiest / shortest.
If you don't call http.ListenAndServe() yourself (as in App Engine), or you don't have a single root handler, then you need to wrap all handlers manually that you register.
Note that the handler returned by context.ClearHandler() does nothing magical, all it does is call context.Clear() after calling the handler you pass. So you may just as easily call context.Clear() in your handler to achieve the same effect.
If you do so, one important thing is to use defer as if for some reason context.Clear() would not be reached (e.g. a preceding return statement is encountered), you would again leak memory. Deferred functions are called even if your function panics. So it should be done like this:
func handler(w http.ResponseWriter, r *http.Request) {
defer context.Clear(r)
var cookiestore = sessions.NewCookieStore([]byte("somesecret"))
session, _ := cookiestore.Get(r, "session")
session.Values["foo"] = "bar"
fmt.Fprintf(w, "session value is %v", session.Values["foo"])
}
Also note that the session store creation should only be done once, so move that out from your handler to a global variable. And do check and handle errors to save you some headache. So the final suggested code is this:
package test
import (
"fmt"
"net/http"
"log"
"github.com/gorilla/context"
"github.com/gorilla/sessions"
)
func init() {
http.Handle("/", context.ClearHandler(http.HandlerFunc(handler)))
}
var cookiestore = sessions.NewCookieStore([]byte("somesecret"))
func handler(w http.ResponseWriter, r *http.Request) {
session, err := cookiestore.Get(r, "session")
if err != nil {
// Handle error:
log.Printf("Error getting session: %v", err)
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
session.Values["foo"] = "bar"
fmt.Fprintf(w, "session value is %v", session.Values["foo"])
}
Gorilla also offers a context.Clear() option that you can put in your request handler as per the docs here:
https://godoc.org/github.com/gorilla/context#Clear
So your handler func would become:
func handler(w http.ResponseWriter, r *http.Request) {
var cookiestore = sessions.NewCookieStore([]byte("somesecret"))
session, _ := cookiestore.Get(r, "session")
session.Values["foo"] = "bar"
fmt.Fprintf(w, "session value is %v", session.Values["foo"])
context.Clear(r)
}
If I run below example on Windows I will quickly hit TCP connection limit (which I set to 64k) and get error: dial tcp 127.0.0.1:3306: connectex: Only one usage of each socket address (protocol/network address/port) is normally permitted.
I see all this TIME_WAIT states waiting for there lifetime to end with: netstat -ano|findstr 3306
Why aren't connections closed immediately?
The code:
package main
import (
_ "github.com/go-sql-driver/mysql"
"github.com/jmoiron/sqlx"
"log"
"sync"
)
var (
db_instance *sqlx.DB
wg sync.WaitGroup
)
func main() {
db, err := sqlx.Connect("mysql", "user:pass#/table")
if err != nil {
log.Fatalln(err)
}
defer db.Close()
db_instance = db
for {
for l := 0; l < 50; l++ {
wg.Add(1)
go DB_TEST()
}
wg.Wait()
}
}
func DB_TEST() {
defer wg.Done()
var s string
err := db_instance.QueryRow("SELECT NOW()").Scan(&s)
if err != nil {
log.Println(err)
return
}
log.Println(s)
}
Drafting answer from my comments discussion with #Glavić.
Utilize the SetMaxOpenConns and SetMaxIdleConns settings to keep TIME_WAIT status and connections under control. If needed use SetConnMaxLifetime too, generally it's not needed.