I'm making a function like ioutil.ReadDir() but recursively due I want all the files in folders and subfolders and ioutil.ReadDir() just do it in the specified folder but I don't know how to append items to an array of []os.FileInfo that I've created.
This is what I have:
func GetFilesRecursively(searchDirectory string) (foundFileList []os.FileInfo, errorGenerated error){
fileList := []os.FileInfo{}
allFilesAndFolders := []string{}
//Get all the files and directories
err := filepath.Walk(searchDirectory, func(path string, f os.FileInfo, err error) error {
allFilesAndFolders = append(allFilesAndFolders, path)
return nil
})
// Remove directories due those are also added into the array and we don't need them
for _, file := range allFilesAndFolders{
fileInfo, _ := os.Stat(file)
if (!fileInfo.Mode().IsDir()){
fileList = append(fileList, file) //error here!!
}
}
return fileList, err
}
The error is in the comments in the code snippet above
How could I do that?
You try to append a string to an array of os.FileInfo. That is a type error.
Here is a version of your code that does what you want and looks more Go-like, with shorter variable names, unnamed return types (their types perfectly decribe what they do) and using var to create an empty slice.
func GetFilesRecursively(root string) ([]os.FileInfo, error) {
var files []os.FileInfo
// walk all directories but only collect files
err := filepath.Walk(root, func(path string, f os.FileInfo, err error) error {
if !f.IsDir() {
files = append(files, f)
}
return nil
})
return files, err
}
Related
I want to find all files matching a specific pattern in a directory recursively (including subdirectories). I wrote the code to do this:
libRegEx, e := regexp.Compile("^.+\\.(dylib)$")
if e != nil {
log.Fatal(e)
}
files, err := ioutil.ReadDir("/usr/lib")
if err != nil {
log.Fatal(err)
}
for _, f := range files {
if libRegEx.MatchString(f.Name()) {
println(f.Name())
}
}
Unfortunately, it only searches in /usr/bin, but I also want to search for matches in its subdirectories. How can I achieve this? Thanks.
The standard library's filepath package includes Walk for exactly this purpose: "Walk walks the file tree rooted at root, calling walkFn for each file or directory in the tree, including root." For example:
libRegEx, e := regexp.Compile("^.+\\.(dylib)$")
if e != nil {
log.Fatal(e)
}
e = filepath.Walk("/usr/lib", func(path string, info os.FileInfo, err error) error {
if err == nil && libRegEx.MatchString(info.Name()) {
println(info.Name())
}
return nil
})
if e != nil {
log.Fatal(e)
}
Starting with Go 1.16 (Feb 2021), you can use filepath.WalkDir:
package main
import (
"io/fs"
"path/filepath"
)
func walk(s string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if ! d.IsDir() {
println(s)
}
return nil
}
func main() {
filepath.WalkDir("..", walk)
}
If you are looking for something that doesn't use walk, I found this project
The main recursive algorithm seems effective despite using strings. It basically amounts to the below code and kinda reminds me of merge sort and other recursive algorithms:
func processed(fileName string, processedDirectories []string) bool {
for i := 0; i < len(processedDirectories); i++ {
if processedDirectories[i] != fileName {
continue
}
return true
}
return false
}
func listDirContents(path string, dirs []string) {
files, _ := ioutil.ReadDir(path)
for _, f := range files {
var newPath string
if path != "/" {
newPath = fmt.Sprintf("%s/%s", path, f.Name())
} else {
newPath = fmt.Sprintf("%s%s", path, f.Name())
}
if f.IsDir() {
if !processed(newPath, dirs) {
dirs = append(dirs, newPath)
listDirContents(newPath, dirs)
}
} else {
fmt.Println(newPath)
}
}
}
That actually prints all found paths starting from the provided directory and includes all sub-directories. Therefor you would have to check if the path contains your target string instead of just printing the path with fmt.Println() statements.
After trying it out vs the find command, it scanned my /home directory in about .8s... the find command found the same files but did it in about .3s (a full .5s faster than the above algorithm).
You can use all the files in the directory using following code:
files, err := ioutil.ReadDir(dirPath)
check(err)
for _, file := range files {
fmt.Println(dirPath + file.Name())
}
The code is using the io/ioutil package to read all the files in the given directory and then looping through them to print there names.
I don't really know how to formulate the question, but here it is.
I'm using fsnotify to watch some directories for changes and when a file changes, I sync the change to another directory. But I want to add newly created directories to the watch too and it's not really working.
Here's my code:
func Watcher() {
watcher, err := fsnotify.NewWatcher()
defer watcher.Close()
done := make(chan bool)
go func() {
for {
select {
case event := <-watcher.Events:
if file.Mode().IsDir() {
err = os.Mkdir(dest, 0755)
err = watcher.Add(dest)
}
case err := <-watcher.Errors:
log.Println("error:", err)
}
}
}()
dirs, err := readLines("dirs")
for _, el := range dirs {
err = watcher.Add(el)
}
check(err)
<-done
}
The function is much longer, but I've deleted the non-important parts. Everything works, except the err = watcher.Add(dest).
How can I make it watch more directories?
It was working just fine, but I got some variables wrong. Should have been watcher.Add(event.Name) instead of watcher.Add(dest).
I'm trying to learn Go (or Golang) and can't seem to get it right. I have 2 texts files, each containing a list of words. I'm trying to count the amount of words that are present in both files.
Here is my code so far :
package main
import (
"fmt"
"log"
"net/http"
"bufio"
)
func stringInSlice(str string, list []string) bool {
for _, v := range list {
if v == str {
return true
}
}
return false
}
func main() {
// Texts URL
var list = "https://gist.githubusercontent.com/alexcesaro/c9c47c638252e21bd82c/raw/bd031237a56ae6691145b4df5617c385dffe930d/list.txt"
var url1 = "https://gist.githubusercontent.com/alexcesaro/4ebfa5a9548d053dddb2/raw/abb8525774b63f342e5173d1af89e47a7a39cd2d/file1.txt"
//Create storing arrays
var buffer [2000]string
var bufferUrl1 [40000]string
// Set a sibling counter
var sibling = 0
// Read and store text files
wordList, err := http.Get(list)
if err != nil {
log.Fatalf("Error while getting the url : %v", err)
}
defer wordList.Body.Close()
wordUrl1, err := http.Get(url1)
if err != nil {
log.Fatalf("Error while getting the url : %v", err)
}
defer wordUrl1.Body.Close()
streamList := bufio.NewScanner(wordList.Body)
streamUrl1 := bufio.NewScanner(wordUrl1.Body)
streamList.Split(bufio.ScanLines)
streamUrl1.Split(bufio.ScanLines)
var i = 0;
var j = 0;
//Fill arrays with each lines
for streamList.Scan() {
buffer[i] = streamList.Text()
i++
}
for streamUrl1.Scan() {
bufferUrl1[j] = streamUrl1.Text()
j++
}
//ERROR OCCURRING HERE :
// This code if i'm not wrong is supposed to compare through all the range of bufferUrl1 -> bufferUrl1 values with buffer values, then increment sibling and output FIND
for v := range bufferUrl1{
if stringInSlice(bufferUrl1, buffer) {
sibling++
fmt.Println("FIND")
}
}
// As a testing purpose thoses lines properly paste both array
// fmt.Println(buffer)
// fmt.Println(bufferUrl1)
}
But right now, my build doesn't even succeed. I'm only greeted with this message:
.\hello.go:69: cannot use bufferUrl1 (type [40000]string) as type string in argument to stringInSlice
.\hello.go:69: cannot use buffer (type [2000]string) as type []string in argument to stringInSlice
bufferUrl1 is an array: [4000]string. You meant to use v (each
string in bufferUrl1). But in fact, you meant to use the second
variable—the first variable is the index which is ignored in the code
below using _.
type [2000]string is different from []string. In Go, arrays and slices are not the same. Read Go Slices: usage and internals. I've changed both variable declarations to use slices with the same initial length using make.
These are changes you need to make to compile.
Declarations:
// Create storing slices
buffer := make([]string, 2000)
bufferUrl1 := make([]string, 40000)
and the loop on Line 69:
for _, s := range bufferUrl1 {
if stringInSlice(s, buffer) {
sibling++
fmt.Println("FIND")
}
}
As a side-note, consider using a map instead of a slice for buffer for more efficient lookup instead of looping through the list in stringInSlice.
https://play.golang.org/p/UcaSVwYcIw has the fix for the comments below (you won't be able to make HTTP requests from the Playground).
I'm currently learning how to develop with Go (or golang) and I have a strange issue:
I try to create a script looking inside an HTML file in order to get all the sources of each tags.
The goal of the script is to merge all the retrieved files.
So, that's for the story: for now, I'm able to get the content of each JavaScript files but... I can't concatenate them...
You can see below my script:
//Open main file
mainFilePath := "/path/to/my/file.html"
mainFileDir := path.Dir(mainFilePath)+"/"
mainFileContent, err := ioutil.ReadFile(mainFilePath)
if err == nil {
mainFileContent := string(mainFileContent)
var finalFileContent bytes.Buffer
//Start RegExp searching for JavaScript src
scriptReg, _ := regexp.Compile("<script src=\"(.*)\">")
scripts := scriptReg.FindAllStringSubmatch(mainFileContent,-1)
//For each SRC found...
for _, path := range scripts {
//We open the corresponding file
subFileContent, err := ioutil.ReadFile(mainFileDir+path[1])
if err == nil {
//And we add its content to the "final" variable
fmt.Println(finalFileContent.Write(subFileContent))
} else {
fmt.Println(err)
}
}
//Try to display the final result
// fmt.Println(finalFileContent.String())
fmt.Printf(">>> %#v", finalFileContent)
fmt.Println("Y U NO WORKS? :'(")
} else {
fmt.Println(err)
}
So, each fmt.Println(finalFileContent.Write(subFileContent)) display something like 6161 , so I assume the Write() method is correctly executed.
But fmt.Printf(">>> %#v", finalFileContent) displays nothing. Absolutely nothing (even the ">>>" are not displayed!) And it's the same for the commented line just above.
The funny part is that the string "Y U NO WORK ? :'(" is correctly displayed...
Do you know why?
And do you know how to solve this issue?
Thanks in advance!
You are ignoring some errors. What are your results when you run the following version of your code?
package main
import (
"bytes"
"fmt"
"io/ioutil"
"path"
"regexp"
)
func main() {
//Open main file
mainFilePath := "/path/to/my/file.html"
mainFileDir := path.Dir(mainFilePath) + "/"
mainFileContent, err := ioutil.ReadFile(mainFilePath)
if err == nil {
mainFileContent := string(mainFileContent)
var finalFileContent bytes.Buffer
//Start RegExp searching for JavaScript src
scriptReg, _ := regexp.Compile("<script src=\"(.*)\">")
scripts := scriptReg.FindAllStringSubmatch(mainFileContent, -1)
//For each SRC found...
for _, path := range scripts {
//We open the corresponding file
subFileContent, err := ioutil.ReadFile(mainFileDir + path[1])
if err == nil {
//And we add its content to the "final" variable
// fmt.Println(finalFileContent.Write(subFileContent))
n, err := finalFileContent.Write(subFileContent)
fmt.Println("finalFileContent Write:", n, err)
} else {
fmt.Println(err)
}
}
//Try to display the final result
// fmt.Println(finalFileContent.String())
// fmt.Printf(">>> %#v", finalFileContent)
n, err := fmt.Printf(">>> %#v", finalFileContent)
fmt.Println()
fmt.Println("finalFileContent Printf:", n, err)
fmt.Println("Y U NO WORKS? :'(")
} else {
fmt.Println(err)
}
}
UPDATE:
The statement:
fmt.Println("finalFileContent Printf:", n, err)
Outputs:
finalFileContent Printf: 0 write /dev/stdout: winapi error #8
or
finalFileContent Printf: 0 write /dev/stdout: Not enough storage is available to process this command.
From MSDN:
ERROR_NOT_ENOUGH_MEMORY
8 (0x8)
Not enough storage is available to process this command.
The formatted output to the Windows console overflows the buffer (circa 64KB).
There is a related Go open issue:
Issue 3376: windows: detect + handle console in os.File.Write
This question already has answers here:
How to check if a file exists in Go?
(14 answers)
Closed 2 years ago.
I want to check the existence of file ./conf/app.ini in my Go code,
but I can't find a good way to do that.
I know there is a method of File in Java: public boolean exists(), which returns true if the file or directory exists.
But how can this be done in Go?
// exists returns whether the given file or directory exists
func exists(path string) (bool, error) {
_, err := os.Stat(path)
if err == nil { return true, nil }
if os.IsNotExist(err) { return false, nil }
return false, err
}
Edited to add error handling.
You can use this :
if _, err := os.Stat("./conf/app.ini"); err != nil {
if os.IsNotExist(err) {
// file does not exist
} else {
// other error
}
}
See : http://golang.org/pkg/os/#IsNotExist
More of an FYI, since I looked around for a few minutes thinking my question be a quick search away.
How to check if path represents an existing directory in Go?
This was the most popular answer in my search results, but here and elsewhere the solutions only provide existence check. To check if path represents an existing directory, I found I could easily:
path := GetSomePath();
if stat, err := os.Stat(path); err == nil && stat.IsDir() {
// path is a directory
}
Part of my problem was that I expected path/filepath package to contain the isDir() function.
Simple way to check whether file exists or not:
if _, err := os.Stat("/path/to/whatever"); os.IsNotExist(err) {
// path/to/whatever does not exist
}
if _, err := os.Stat("/path/to/whatever"); err == nil {
// path/to/whatever exists
}
Sources:
mattes made this gist on Aug 6 '14
Sridhar Ratnakumar made this answer on Sep 21 '12
I use the following function to check my directories for any errors. It's very similar to previous answers, but I think not nesting ifs makes the code more clear. It uses go-homedir to remove ~ from directory paths and pkg/errors to return nicer error messages, but it would be easy to take them out if you don't need their functionality.
// validateDirectory expands a directory and checks that it exists
// it returns the full path to the directory on success
// validateDirectory("~/foo") -> ("/home/bbkane/foo", nil)
func validateDirectory(dir string) (string, error) {
dirPath, err := homedir.Expand(dir)
if err != nil {
return "", errors.WithStack(err)
}
info, err := os.Stat(dirPath)
if os.IsNotExist(err) {
return "", errors.Wrapf(err, "Directory does not exist: %v\n", dirPath)
}
if err != nil {
return "", errors.Wrapf(err, "Directory error: %v\n", dirPath)
}
if !info.IsDir() {
return "", errors.Errorf("Directory is a file, not a directory: %#v\n", dirPath)
}
return dirPath, nil
}
Also, to repeat #Dave C's comment, if the reason you're checking a directory's existence is to write a file into it, it's usually better to simply try to open it an deal with errors afterwards:
// O_EXCL - used with O_CREATE, file must not exist
file, err := os.OpenFile(fileName, os.O_RDWR|os.O_CREATE|os.O_EXCL, 0666)
if err != nil {
return errors.WithStack(err)
}
defer file.Close()
There is simple way to check whether your file exists or not:
if _, err := os.Stat("./conf/app.ini"); err != nil {
if os.IsNotExist(err) {
..... //Shows error if file not exists
} else {
..... // Shows success message like file is there
}
}