I want to delete thelast N bytes from file in Go,
Actually, this is already implemented is the os.Truncate() function. But this function takes the new size. So to use this, you have to first get the size of the file. For that, you may use os.Stat().
Wrapping it into a function:
func truncateFile(name string, bytesToRemove int64) error {
fi, err := os.Stat(name)
if err != nil {
return err
}
return os.Truncate(name, fi.Size()-bytesToRemove)
}
Using it to remove the last 5000 bytes:
if err := truncateFile("C:\\Test.zip", 5000); err != nil {
fmt.Println("Error:", err)
}
Another alternative is to use the File.Truncate() method for that. If we have an os.File, we may also use File.Stat() to get its size.
This is how it would look like:
func truncateFile(name string, bytesToRemove int64) error {
f, err := os.OpenFile(name, os.O_RDWR, 0644)
if err != nil {
return err
}
defer f.Close()
fi, err := f.Stat()
if err != nil {
return err
}
return f.Truncate(fi.Size() - bytesToRemove)
}
Using it is the same. This may be preferable if we're working on a file (we have it opened) and we have to truncate it. But in that case you'd want to pass os.File instead of its name to truncateFile().
Note: if you try to remove more bytes than the file currently has, truncateFile() will return an error.
I want to find all files matching a specific pattern in a directory recursively (including subdirectories). I wrote the code to do this:
libRegEx, e := regexp.Compile("^.+\\.(dylib)$")
if e != nil {
log.Fatal(e)
}
files, err := ioutil.ReadDir("/usr/lib")
if err != nil {
log.Fatal(err)
}
for _, f := range files {
if libRegEx.MatchString(f.Name()) {
println(f.Name())
}
}
Unfortunately, it only searches in /usr/bin, but I also want to search for matches in its subdirectories. How can I achieve this? Thanks.
The standard library's filepath package includes Walk for exactly this purpose: "Walk walks the file tree rooted at root, calling walkFn for each file or directory in the tree, including root." For example:
libRegEx, e := regexp.Compile("^.+\\.(dylib)$")
if e != nil {
log.Fatal(e)
}
e = filepath.Walk("/usr/lib", func(path string, info os.FileInfo, err error) error {
if err == nil && libRegEx.MatchString(info.Name()) {
println(info.Name())
}
return nil
})
if e != nil {
log.Fatal(e)
}
Starting with Go 1.16 (Feb 2021), you can use filepath.WalkDir:
package main
import (
"io/fs"
"path/filepath"
)
func walk(s string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if ! d.IsDir() {
println(s)
}
return nil
}
func main() {
filepath.WalkDir("..", walk)
}
If you are looking for something that doesn't use walk, I found this project
The main recursive algorithm seems effective despite using strings. It basically amounts to the below code and kinda reminds me of merge sort and other recursive algorithms:
func processed(fileName string, processedDirectories []string) bool {
for i := 0; i < len(processedDirectories); i++ {
if processedDirectories[i] != fileName {
continue
}
return true
}
return false
}
func listDirContents(path string, dirs []string) {
files, _ := ioutil.ReadDir(path)
for _, f := range files {
var newPath string
if path != "/" {
newPath = fmt.Sprintf("%s/%s", path, f.Name())
} else {
newPath = fmt.Sprintf("%s%s", path, f.Name())
}
if f.IsDir() {
if !processed(newPath, dirs) {
dirs = append(dirs, newPath)
listDirContents(newPath, dirs)
}
} else {
fmt.Println(newPath)
}
}
}
That actually prints all found paths starting from the provided directory and includes all sub-directories. Therefor you would have to check if the path contains your target string instead of just printing the path with fmt.Println() statements.
After trying it out vs the find command, it scanned my /home directory in about .8s... the find command found the same files but did it in about .3s (a full .5s faster than the above algorithm).
You can use all the files in the directory using following code:
files, err := ioutil.ReadDir(dirPath)
check(err)
for _, file := range files {
fmt.Println(dirPath + file.Name())
}
The code is using the io/ioutil package to read all the files in the given directory and then looping through them to print there names.
I am trying to use a file instead of a DB to get a prototype up and running. I have a program that (1) reads existing content from the file to a map, (2) takes JSON POSTs that add content to the map, (3) on exit, writes to the file.
First, the file is not being created. Then I created an empty file. It is not being written to.
I am trying to read the file, determine if there is existing content. If there is not existing content, create a blank map. If there is existing content, unmarshal it into a new map.
func writeDB() {
eventDBJSON, err := json.Marshal(eventDB)
if err != nil {
panic(err)
}
err2 := ioutil.WriteFile("/Users/sarah/go/dat.txt", eventDBJSON, 0777)
if err2 != nil {
panic(err2)
}
}
func main() {
dat, err := ioutil.ReadFile("/Users/sarah/go/dat.txt")
if err != nil {
panic(err)
}
if dat == nil {
eventDB = DB{
events: map[string]event{},
}
} else {
if err2 := json.Unmarshal(dat, &eventDB); err2 != nil {
panic(err2)
}
}
router := httprouter.New()
router.POST("/join", JoinEvent)
router.POST("/create", CreateEvent)
log.Fatal(http.ListenAndServe(":8080", router))
defer writeDB()
}
There is no way for the server to ever reach defer writeDB().
http.ListenAndServe blocks, and if it did return anything, you log.Fatal that, which exits your app at that point.
You can't intercept all ways an app can exit, getting SIGKILL, machine loss of power, etc.
I'm assuming you really just want to write some code, bounce the server, repeat
If that's the case, then Ctrl-C is good enough.
If you want to write your file on Ctrl-C, look at the signal package.
Also, defer on the last line of a function really has no purpose as defer basically means "do this last".
you can use (*os.File).Stat() to get a file's FileInfo which contain its size
file, err := os.Open( filepath )
if err != nil {
// handle error
}
fi, err := file.Stat()
if err != nil {
// handle error
}
s := fi.Size()
Go's standard library does not have a function solely intended to check if a file exists or not (like Python's os.path.exists). What is the idiomatic way to do it?
To check if a file doesn't exist, equivalent to Python's if not os.path.exists(filename):
if _, err := os.Stat("/path/to/whatever"); errors.Is(err, os.ErrNotExist) {
// path/to/whatever does not exist
}
To check if a file exists, equivalent to Python's if os.path.exists(filename):
Edited: per recent comments
if _, err := os.Stat("/path/to/whatever"); err == nil {
// path/to/whatever exists
} else if errors.Is(err, os.ErrNotExist) {
// path/to/whatever does *not* exist
} else {
// Schrodinger: file may or may not exist. See err for details.
// Therefore, do *NOT* use !os.IsNotExist(err) to test for file existence
}
Answer by Caleb Spare posted in gonuts mailing list.
[...] It's not actually needed very often and [...] using os.Stat is
easy enough for the cases where it is required.
[...] For instance: if you are going to open the file, there's no reason to check whether it exists first. The file could disappear in between checking and opening, and anyway you'll need to check the os.Open error regardless. So you simply call os.IsNotExist(err) after you try
to open the file, and deal with its non-existence there (if that requires special handling).
[...] You don't need to check for the paths existing at all (and you shouldn't).
os.MkdirAll works whether or not the paths already exist. (Also you need to check the error from that call.)
Instead of using os.Create, you should use os.OpenFile(path, os.O_RDWR|os.O_CREATE|os.O_EXCL, 0666) . That way you'll get an error if the file already exists. Also this doesn't have a race condition with something else making the file, unlike your version which checks for existence beforehand.
Taken from: https://groups.google.com/forum/#!msg/golang-nuts/Ayx-BMNdMFo/4rL8FFHr8v4J
The first thing to consider is that it is rare that you would only want to check whether or not a file exists. In most situations, you're trying to do something with the file if it exists. In Go, any time you try to perform some operation on a file that doesn't exist, the result should be a specific error (os.ErrNotExist) and the best thing to do is check whether the return err value (e.g. when calling a function like os.OpenFile(...)) is os.ErrNotExist.
The recommended way to do this used to be:
file, err := os.OpenFile(...)
if os.IsNotExist(err) {
// handle the case where the file doesn't exist
}
However, since the addition of errors.Is in Go 1.13 (released in late 2019), the new recommendation is to use errors.Is:
file, err := os.OpenFile(...)
if errors.Is(err, os.ErrNotExist) {
// handle the case where the file doesn't exist
}
It's usually best to avoid using os.Stat to check for the existence of a file before you attempt to do something with it, because it will always be possible for the file to be renamed, deleted, etc. in the window of time before you do something with it.
However, if you're OK with this caveat and you really, truly just want to check whether a file exists without then proceeding to do something useful with it (as a contrived example, let's say that you're writing a pointless CLI tool that tells you whether or not a file exists and then exits ¯\_(ツ)_/¯), then the recommended way to do it would be:
if _, err := os.Stat(filename); errors.Is(err, os.ErrNotExist) {
// file does not exist
} else {
// file exists
}
You should use the os.Stat() and os.IsNotExist() functions as in the following example:
func Exists(name string) (bool, error) {
_, err := os.Stat(name)
if err == nil {
return true, nil
}
if errors.Is(err, os.ErrNotExist) {
return false, nil
}
return false, err
}
edit1: fixed issue of returning true when under some circumstances.
edit2: switched to using errors.Is() from os.IsNotExist(), which many say is a best-practice and here
What other answers missed, is that the path given to the function could actually be a directory. Following function makes sure, that the path is really a file.
func fileExists(filename string) bool {
info, err := os.Stat(filename)
if os.IsNotExist(err) {
return false
}
return !info.IsDir()
}
Another thing to point out: This code could still lead to a race condition, where another thread or process deletes or creates the specified file, while the fileExists function is running.
If you're worried about this, use a lock in your threads, serialize the access to this function or use an inter-process semaphore if multiple applications are involved. If other applications are involved, outside of your control, you're out of luck, I guess.
The example by user11617 is incorrect; it will report that the file exists even in cases where it does not, but there was an error of some other sort.
The signature should be Exists(string) (bool, error). And then, as it happens, the call sites are no better.
The code he wrote would better as:
func Exists(name string) bool {
_, err := os.Stat(name)
return !os.IsNotExist(err)
}
But I suggest this instead:
func Exists(name string) (bool, error) {
_, err := os.Stat(name)
if os.IsNotExist(err) {
return false, nil
}
return err != nil, err
}
_, err := os.Stat(file)
if err == nil {
log.Printf("file %s exists", file)
} else if os.IsNotExist(err) {
log.Printf("file %s not exists", file)
} else {
log.Printf("file %s stat error: %v", file, err)
}
basicly
package main
import (
"fmt"
"os"
)
func fileExists(path string) bool {
_, err := os.Stat(path)
return !os.IsNotExist(err)
}
func main() {
var file string = "foo.txt"
exist := fileExists(file)
if exist {
fmt.Println("file exist")
} else {
fmt.Println("file not exists")
}
}
run example
other way
with os.Open
package main
import (
"fmt"
"os"
)
func fileExists(path string) bool {
_, err := os.Open(path) // For read access.
return err == nil
}
func main() {
fmt.Println(fileExists("d4d.txt"))
}
run it
Best way to check if file exists:
if _, err := os.Stat("/path/to/file"); err == nil || os.IsExist(err) {
// your code here if file exists
}
The function example:
func file_is_exists(f string) bool {
_, err := os.Stat(f)
if os.IsNotExist(err) {
return false
}
return err == nil
}
Let's look at few aspects first, both the function provided by os package of golang are not utilities but error checkers, what do I mean by that is they are just a wrapper to handle errors on cross platform.
So basically if os.Stat if this function doesn't give any error that means the file is existing if it does you need to check what kind of error it is, here comes the use of these two function os.IsNotExist and os.IsExist.
This can be understood as the Stat of the file throwing error because it doesn't exists or is it throwing error because it exist and there is some problem with it.
The parameter that these functions take is of type error, although you might be able to pass nil to it but it wouldn't make sense.
This also points to the fact that IsExist is not same as !IsNotExist, they are way two different things.
So now if you want to know if a given file exist in go, I would prefer the best way is:
if _, err := os.Stat(path/to/file); !os.IsNotExist(err){
//TODO
}
As mentioned in other answers, it is possible to construct the required behaviour / errors from using different flags with os.OpenFile. In fact, os.Create is just a sensible-defaults shorthand for doing so:
// Create creates or truncates the named file. If the file already exists,
// it is truncated. If the file does not exist, it is created with mode 0666
// (before umask). If successful, methods on the returned File can
// be used for I/O; the associated file descriptor has mode O_RDWR.
// If there is an error, it will be of type *PathError.
func Create(name string) (*File, error) {
return OpenFile(name, O_RDWR|O_CREATE|O_TRUNC, 0666)
}
You should combine these flags yourself to get the behaviour you are interested in:
// Flags to OpenFile wrapping those of the underlying system. Not all
// flags may be implemented on a given system.
const (
// Exactly one of O_RDONLY, O_WRONLY, or O_RDWR must be specified.
O_RDONLY int = syscall.O_RDONLY // open the file read-only.
O_WRONLY int = syscall.O_WRONLY // open the file write-only.
O_RDWR int = syscall.O_RDWR // open the file read-write.
// The remaining values may be or'ed in to control behavior.
O_APPEND int = syscall.O_APPEND // append data to the file when writing.
O_CREATE int = syscall.O_CREAT // create a new file if none exists.
O_EXCL int = syscall.O_EXCL // used with O_CREATE, file must not exist.
O_SYNC int = syscall.O_SYNC // open for synchronous I/O.
O_TRUNC int = syscall.O_TRUNC // truncate regular writable file when opened.
)
Depending on what you pick, you will get different errors.
Below is an example which will either truncate an existing file, or fail when a file exists.
openOpts := os.O_RDWR|os.O_CREATE
if truncateWhenExists {
openOpts |= os.O_TRUNC // file will be truncated
} else {
openOpts |= os.O_EXCL // file must not exist
}
f, err := os.OpenFile(filePath, openOpts, 0644)
// ... do stuff
This is how I check if a file exists in Go 1.16
package main
import (
"errors"
"fmt"
"io/fs"
"os"
)
func main () {
if _, err:= os.Stat("/path/to/file"); errors.Is(err, fs.ErrNotExist){
fmt.Print(err.Error())
} else {
fmt.Print("file exists")
}
}
Here is my take on a file exists method. It also checks that the file is not a directory and in case of an error, returns it as well.
// FileExists checks if a file exists (and it is not a directory).
func FileExists(filePath string) (bool, error) {
info, err := os.Stat(filePath)
if err == nil {
return !info.IsDir(), nil
}
if errors.Is(err, os.ErrNotExist) {
return false, nil
}
return false, err
}
I'm currently learning how to develop with Go (or golang) and I have a strange issue:
I try to create a script looking inside an HTML file in order to get all the sources of each tags.
The goal of the script is to merge all the retrieved files.
So, that's for the story: for now, I'm able to get the content of each JavaScript files but... I can't concatenate them...
You can see below my script:
//Open main file
mainFilePath := "/path/to/my/file.html"
mainFileDir := path.Dir(mainFilePath)+"/"
mainFileContent, err := ioutil.ReadFile(mainFilePath)
if err == nil {
mainFileContent := string(mainFileContent)
var finalFileContent bytes.Buffer
//Start RegExp searching for JavaScript src
scriptReg, _ := regexp.Compile("<script src=\"(.*)\">")
scripts := scriptReg.FindAllStringSubmatch(mainFileContent,-1)
//For each SRC found...
for _, path := range scripts {
//We open the corresponding file
subFileContent, err := ioutil.ReadFile(mainFileDir+path[1])
if err == nil {
//And we add its content to the "final" variable
fmt.Println(finalFileContent.Write(subFileContent))
} else {
fmt.Println(err)
}
}
//Try to display the final result
// fmt.Println(finalFileContent.String())
fmt.Printf(">>> %#v", finalFileContent)
fmt.Println("Y U NO WORKS? :'(")
} else {
fmt.Println(err)
}
So, each fmt.Println(finalFileContent.Write(subFileContent)) display something like 6161 , so I assume the Write() method is correctly executed.
But fmt.Printf(">>> %#v", finalFileContent) displays nothing. Absolutely nothing (even the ">>>" are not displayed!) And it's the same for the commented line just above.
The funny part is that the string "Y U NO WORK ? :'(" is correctly displayed...
Do you know why?
And do you know how to solve this issue?
Thanks in advance!
You are ignoring some errors. What are your results when you run the following version of your code?
package main
import (
"bytes"
"fmt"
"io/ioutil"
"path"
"regexp"
)
func main() {
//Open main file
mainFilePath := "/path/to/my/file.html"
mainFileDir := path.Dir(mainFilePath) + "/"
mainFileContent, err := ioutil.ReadFile(mainFilePath)
if err == nil {
mainFileContent := string(mainFileContent)
var finalFileContent bytes.Buffer
//Start RegExp searching for JavaScript src
scriptReg, _ := regexp.Compile("<script src=\"(.*)\">")
scripts := scriptReg.FindAllStringSubmatch(mainFileContent, -1)
//For each SRC found...
for _, path := range scripts {
//We open the corresponding file
subFileContent, err := ioutil.ReadFile(mainFileDir + path[1])
if err == nil {
//And we add its content to the "final" variable
// fmt.Println(finalFileContent.Write(subFileContent))
n, err := finalFileContent.Write(subFileContent)
fmt.Println("finalFileContent Write:", n, err)
} else {
fmt.Println(err)
}
}
//Try to display the final result
// fmt.Println(finalFileContent.String())
// fmt.Printf(">>> %#v", finalFileContent)
n, err := fmt.Printf(">>> %#v", finalFileContent)
fmt.Println()
fmt.Println("finalFileContent Printf:", n, err)
fmt.Println("Y U NO WORKS? :'(")
} else {
fmt.Println(err)
}
}
UPDATE:
The statement:
fmt.Println("finalFileContent Printf:", n, err)
Outputs:
finalFileContent Printf: 0 write /dev/stdout: winapi error #8
or
finalFileContent Printf: 0 write /dev/stdout: Not enough storage is available to process this command.
From MSDN:
ERROR_NOT_ENOUGH_MEMORY
8 (0x8)
Not enough storage is available to process this command.
The formatted output to the Windows console overflows the buffer (circa 64KB).
There is a related Go open issue:
Issue 3376: windows: detect + handle console in os.File.Write