I have a Go project where I want to read a HCL file. This HCL file contains variables. However, I cannot parse it and I get the following error message:
Variables not allowed; Variables may not be used here., and 1 other diagnostic(s)
My Go Code:
package main
import (
"fmt"
"log"
"github.com/hashicorp/hcl/v2/hclsimple"
)
type Config struct {
Hello string `hcl:"hello"`
World string `hcl:"world"`
Message string `hcl:"message"`
}
func main() {
var config Config
err := hclsimple.DecodeFile("test.hcl", nil, &config)
if err != nil {
log.Fatalf("Failed to load configuration: %s", err)
}
fmt.Println(config.Message)
}
My HCL File
hello = "hello"
world = "world"
message = "hello ${world}"
What am I doing wrong? Is my HCL syntax perhaps not correct?
Is my HCL syntax perhaps not correct?
It's syntactically valid but doesn't work the way you're expecting it to. HCL doesn't allow for referencing arbitrary values defined elsewhere in the HCL file. It allows only for referencing variables which are exposed by the parser. For example, this gives the expected output:
ectx := &hcl.EvalContext{Variables: map[string]cty.Value{"world": cty.StringVal("world")}}
err := hclsimple.DecodeFile("test.hcl", ectx, &config)
The documentation doesn't make this especially clear, but the relevant reference would be here: https://github.com/hashicorp/hcl/blob/main/guide/go_expression_eval.rst#expression-evaluation-modes
I am using Zap logger and I want to limit the information that gets logged. For example , I have below code snippet
func (m *mountCommand) Execute(args []string) error {
filelogger.Info("First log", zap.Strings("input args", args))
And the log output is as below
{"level":"info","ts":"2017-11-16T10:04:40.225Z","msg":"First log","input args":["/var/lib/kubelet/pods/74785895-cab5-11e7-88ed-ce1c8b57856c/volumes/xyz-xandndnd",
"{\"kubernetes.io/secret/access-key\":\"Qdfnnfbbdnsjnxni8ehh=\",\"kubernetes.io/secret/secret-key\":\"GGHNHwsffUIJMNBNBVV==\",
\"parallel-count\":\"5\",\"region\":\"iam-standard\"}"]}
How do I just prevent the access-key and secret-key values from getting added to the log due to sensitivity of the data.
I tried many string manipulation functions in Go so that I can just remove those contents before using the zap object.There seems no simple function to get this done in a straight forward simple way.
Your current args is a slice of the following strings:
/var/lib/kubelet/pods/74785895-cab5-11e7-88ed-ce1c8b57856c/volumes/xyz-xandndnd
{"kubernetes.io/secret/access-key":"Qdfnnfbbdnsjnxni8ehh=","kubernetes.io/secret/secret-key":"GGHNHwsffUIJMNBNBVV==","parallel-count":"5","region":"iam-standard"}
The input argument that contains the security-sensitive data is at index 1, and it's a JSON text.
You should not "string-manipulate" a JSON text. First you should unmarshal it into a Go value, then manipulate it, then marshal it back.
This is how this can be done:
// Make a copy of args:
args2 := append([]string{}, args...)
// Unmarshal:
var m map[string]interface{}
if err := json.Unmarshal([]byte(args2[1]), &m); err != nil {
panic(err)
}
// Modify:
m["kubernetes.io/secret/access-key"] = "XXX"
m["kubernetes.io/secret/secret-key"] = "YYY"
// Marshal:
s2, err := json.Marshal(m)
if err != nil {
panic(err)
}
args2[1] = string(s2)
// Verify:
fmt.Println(args2[1])
// Now use args2 to log
filelogger.Info("First log", zap.Strings("(masked) input args", args2))
The "Verify:" is obviously not needed, it's just for us to see the result. Output on the Go Playground:
{"kubernetes.io/secret/access-key":"XXX","kubernetes.io/secret/secret-key":"YYY","parallel-count":"5","region":"iam-standard"}
In your solution you should also add slice index checks to avoid run-time panic.
I've got the following code in Python:
if not os.path.exists(src): sys.exit("Does not exist: %s" % src)
if os.path.exists(dst): sys.exit("Already exists: %s" % dst)
os.rename(src, dst)
From this question, I understand that there is no direct method to test if a file exists or doesn't exist.
What is the proper way to write the above in Go, including printing out the correct error strings?
Here is the closest I've gotten:
package main
import "fmt"
import "os"
func main() {
src := "a"
dst := "b"
e := os.Rename(src, dst)
if e != nil {
fmt.Println(e.(*os.LinkError).Op)
fmt.Println(e.(*os.LinkError).Old)
fmt.Println(e.(*os.LinkError).New)
fmt.Println(e.(*os.LinkError).Err)
}
}
From the availability of information from the error, where it effectively doesn't tell you what the problem is without you parsing an English freeformat string, it seems to me that it is not possible to write the equivalent in Go.
The code you provide contains a race condition: Between you checking for dst to not exist and copying something into dst, a third party could have created the file dst, causing you to overwrite a file. Either remove the os.path.exists(dst) check because it cannot reliably detect if the target exists at the time you try to remove it, or employ the following algorithm instead:
Create a hardlink from src to dst. If a file named dst exists, the operation will fail and you can bail out. If src does not exist, the operation will fail, too.
Remove src.
The following code implements the two-step algorithm outlined above in Go.
import "os"
func renameAndCheck(src, dst string) error {
err := os.Link(src, dst)
if err != nil {
return err
}
return os.Remove(src)
}
You can check for which reason the call to os.Link() failed:
If the error satisfies os.IsNotExist(), the call failed because src did not exist at the time os.Link() was called
If the error satisfies os.IsExist(), the call failed because dst exists at the time os.Link() is called
If the error satisfies os.IsPermission(), the call failed because you don't have sufficient permissions to create a hard link
As far as I know, other reasons (like the file system not supporting the creation of hard links or src and dst being on different file systems) cannot be tested portably.
The translation if your Python code to Go is:
if _, err := os.Stat(src); err != nil {
// The source does not exist or some other error accessing the source
log.Fatal("source:", err)
}
if _, err := os.Stat(dst); !os.IsNotExists(dst) {
// The destination exists or some other error accessing the destination
log.Fatal("dest:", err)
}
if err := os.Rename(src, dst); err != nil {
log.Fatal(err)
}
The three function call sequence is not safe (I am referring to both the original Python version and my replication of it here). The source can be removed or the destination can be created after the checks, but before the rename.
The safe way to move a file is OS dependent. On Windows, you can just call os.Rename(). On Windows, this function will fail if the destination exists or the source does not. On Posix systems, you should link and remove as described in another answer.
Oops, there was one thing I forgot when I made this answer, and it's something that I'm both not quite sure on myself and that I can't seem to find information for on MSDN and Google and the Stack Overflow search.
There are a number of places in the Windows API where you use a negative number, or a number too large to fit in a signed integer; for instance, CW_USEDEFAULT, INVALID_HANDLE_VALUE, GWLP_USERDATA, and so on. In the world of C, everything is all fine and dandy: the language's integer promotion rules come to the rescue.
But in Go, I have to pass all my arguments to functions as uintptr (which is equivalent to C's uintptr_t). The return value from the function is also returned this way, and then I will need to compare. Go doesn't allow integer promotion, and it doesn't allow you to convert a signed constant expression into an unsigned one at compile-time.
Right now, I have a bit of a jerry-rig set up for handling these constants in my UI library. (Here's an example of what this solution looks like in action.) However, I'm not quite satisfied with this solution; it feels to me like it's assuming things about the ABI, and I want to be absolutely sure of what I'm doing.
So my question is: how are signed values handled when passing them to Windows API functions and how are they handled when returning?
All my constants are autogenerated (example output). The autogenerator uses a C ffi, which I'd rather not use for the main project since I can call the DLLs directly (this also makes cross-compilation easier at least for the rest of the year). If I could somehow leverage that, for instance by making everything into a C-side variable of the form
uintptr_t x_CONST_NAME = (uintptr_t) (CONST_NAME);
that would be helpful. But I can't do that without this answer.
Thanks!
Update
Someone on IRC put it differently (reformatted to avoid horizontal scrolling):
[19:13] <FraGag> basically, you're asking whether an int with a value of -1
will be returned as 0x00000000FFFFFFFF or as 0xFFFFFFFFFFFFFFFF
if an int is 4 bytes and an uintptr is 8 bytes
Basically this, but specifically for Windows API interop, for parameters passed in, and regardless of uintptr size.
#twotwotwo's comments to my question pointed me in the right direction. If Stack Overflow allowed marking comments as answers and having multiple answers marked, I'd do that.
tl;dr version: what I have now is correct after all.
I wrote a program (below) that simply dumped all the constants from package syscall and looked for constants that were negative, but not == -1 (as that would just be ^0). The standard file handles (STD_ERROR_HANDLE, STD_INPUT_HANDLE, and STD_OUTPUT_HANDLE) are (-12, -10, and -11, respectively). The code in package syscall passes these constants as the sole argument of getStdHandle(h int), which produces the required file handle for package os. getStdHandle() passes this int to an autogenerated function GetStdHandle(stdhandle int) that wraps a call to the GetStdHandle() system call. GetStdHandle() takes the int and merely converts it to uintptr for passing into syscall.Syscall(). Though no explanation is given in the autogenerator's source (mksyscall_windows.go), if this didn't work, neither would fmt.Println() =P
All of the above is identical on both windows/386 and windows/amd64; the only thing in a processor-specific file is GetStdHandle(), but the relevant code is identical.
My negConst() function is already doing the same thing, just more directly. As such, I can safely assume that it is correct.
Thanks!
// 4 june 2014
// based on code from 24 may 2014
package main
import (
"fmt"
"os"
"strings"
"go/token"
"go/ast"
"go/parser"
"code.google.com/p/go.tools/go/types"
_ "code.google.com/p/go.tools/go/gcimporter"
)
var arch string
func getPackage(path string) (typespkg *types.Package, pkginfo types.Info) {
var pkg *ast.Package
fileset := token.NewFileSet() // parser.ParseDir() actually writes to this; not sure why it doesn't return one instead
filter := func(i os.FileInfo) bool {
if strings.Contains(i.Name(), "_windows") &&
strings.Contains(i.Name(), "_" + arch) &&
strings.HasSuffix(i.Name(), ".go") {
return true
}
if i.Name() == "race.go" || // skip these
i.Name() == "flock.go" {
return false
}
return strings.HasSuffix(i.Name(), "_windows.go") ||
(!strings.Contains(i.Name(), "_"))
}
pkgs, err := parser.ParseDir(fileset, path, filter, parser.AllErrors)
if err != nil {
panic(err)
}
for k, _ := range pkgs { // get the sole key
if pkgs[k].Name == "syscall" {
pkg = pkgs[k]
break
}
}
if pkg == nil {
panic("package syscall not found")
}
// we can't pass pkg.Files directly to types.Check() because the former is a map and the latter is a slice
ff := make([]*ast.File, 0, len(pkg.Files))
for _, v := range pkg.Files {
ff = append(ff, v)
}
// if we don't make() each map, package types won't fill the structure
pkginfo.Defs = make(map[*ast.Ident]types.Object)
pkginfo.Scopes = make(map[ast.Node]*types.Scope)
typespkg, err = new(types.Config).Check(path, fileset, ff, &pkginfo)
if err != nil {
panic(err)
}
return typespkg, pkginfo
}
func main() {
pkgpath := "/home/pietro/go/src/pkg/syscall"
arch = os.Args[1]
pkg, _ := getPackage(pkgpath)
scope := pkg.Scope()
for _, name := range scope.Names() {
obj := scope.Lookup(name)
if obj == nil {
panic(fmt.Errorf("nil object %q from scope %v", name, scope))
}
if !obj.Exported() { // exported names only
continue
}
if _, ok := obj.(*types.Const); ok {
fmt.Printf("egrep -rh '#define[ ]+%s' ~/winshare/Include/ 2>/dev/null\n", obj.Name())
}
// otherwise skip
}
}
Go's standard library does not have a function solely intended to check if a file exists or not (like Python's os.path.exists). What is the idiomatic way to do it?
To check if a file doesn't exist, equivalent to Python's if not os.path.exists(filename):
if _, err := os.Stat("/path/to/whatever"); errors.Is(err, os.ErrNotExist) {
// path/to/whatever does not exist
}
To check if a file exists, equivalent to Python's if os.path.exists(filename):
Edited: per recent comments
if _, err := os.Stat("/path/to/whatever"); err == nil {
// path/to/whatever exists
} else if errors.Is(err, os.ErrNotExist) {
// path/to/whatever does *not* exist
} else {
// Schrodinger: file may or may not exist. See err for details.
// Therefore, do *NOT* use !os.IsNotExist(err) to test for file existence
}
Answer by Caleb Spare posted in gonuts mailing list.
[...] It's not actually needed very often and [...] using os.Stat is
easy enough for the cases where it is required.
[...] For instance: if you are going to open the file, there's no reason to check whether it exists first. The file could disappear in between checking and opening, and anyway you'll need to check the os.Open error regardless. So you simply call os.IsNotExist(err) after you try
to open the file, and deal with its non-existence there (if that requires special handling).
[...] You don't need to check for the paths existing at all (and you shouldn't).
os.MkdirAll works whether or not the paths already exist. (Also you need to check the error from that call.)
Instead of using os.Create, you should use os.OpenFile(path, os.O_RDWR|os.O_CREATE|os.O_EXCL, 0666) . That way you'll get an error if the file already exists. Also this doesn't have a race condition with something else making the file, unlike your version which checks for existence beforehand.
Taken from: https://groups.google.com/forum/#!msg/golang-nuts/Ayx-BMNdMFo/4rL8FFHr8v4J
The first thing to consider is that it is rare that you would only want to check whether or not a file exists. In most situations, you're trying to do something with the file if it exists. In Go, any time you try to perform some operation on a file that doesn't exist, the result should be a specific error (os.ErrNotExist) and the best thing to do is check whether the return err value (e.g. when calling a function like os.OpenFile(...)) is os.ErrNotExist.
The recommended way to do this used to be:
file, err := os.OpenFile(...)
if os.IsNotExist(err) {
// handle the case where the file doesn't exist
}
However, since the addition of errors.Is in Go 1.13 (released in late 2019), the new recommendation is to use errors.Is:
file, err := os.OpenFile(...)
if errors.Is(err, os.ErrNotExist) {
// handle the case where the file doesn't exist
}
It's usually best to avoid using os.Stat to check for the existence of a file before you attempt to do something with it, because it will always be possible for the file to be renamed, deleted, etc. in the window of time before you do something with it.
However, if you're OK with this caveat and you really, truly just want to check whether a file exists without then proceeding to do something useful with it (as a contrived example, let's say that you're writing a pointless CLI tool that tells you whether or not a file exists and then exits ¯\_(ツ)_/¯), then the recommended way to do it would be:
if _, err := os.Stat(filename); errors.Is(err, os.ErrNotExist) {
// file does not exist
} else {
// file exists
}
You should use the os.Stat() and os.IsNotExist() functions as in the following example:
func Exists(name string) (bool, error) {
_, err := os.Stat(name)
if err == nil {
return true, nil
}
if errors.Is(err, os.ErrNotExist) {
return false, nil
}
return false, err
}
edit1: fixed issue of returning true when under some circumstances.
edit2: switched to using errors.Is() from os.IsNotExist(), which many say is a best-practice and here
What other answers missed, is that the path given to the function could actually be a directory. Following function makes sure, that the path is really a file.
func fileExists(filename string) bool {
info, err := os.Stat(filename)
if os.IsNotExist(err) {
return false
}
return !info.IsDir()
}
Another thing to point out: This code could still lead to a race condition, where another thread or process deletes or creates the specified file, while the fileExists function is running.
If you're worried about this, use a lock in your threads, serialize the access to this function or use an inter-process semaphore if multiple applications are involved. If other applications are involved, outside of your control, you're out of luck, I guess.
The example by user11617 is incorrect; it will report that the file exists even in cases where it does not, but there was an error of some other sort.
The signature should be Exists(string) (bool, error). And then, as it happens, the call sites are no better.
The code he wrote would better as:
func Exists(name string) bool {
_, err := os.Stat(name)
return !os.IsNotExist(err)
}
But I suggest this instead:
func Exists(name string) (bool, error) {
_, err := os.Stat(name)
if os.IsNotExist(err) {
return false, nil
}
return err != nil, err
}
_, err := os.Stat(file)
if err == nil {
log.Printf("file %s exists", file)
} else if os.IsNotExist(err) {
log.Printf("file %s not exists", file)
} else {
log.Printf("file %s stat error: %v", file, err)
}
basicly
package main
import (
"fmt"
"os"
)
func fileExists(path string) bool {
_, err := os.Stat(path)
return !os.IsNotExist(err)
}
func main() {
var file string = "foo.txt"
exist := fileExists(file)
if exist {
fmt.Println("file exist")
} else {
fmt.Println("file not exists")
}
}
run example
other way
with os.Open
package main
import (
"fmt"
"os"
)
func fileExists(path string) bool {
_, err := os.Open(path) // For read access.
return err == nil
}
func main() {
fmt.Println(fileExists("d4d.txt"))
}
run it
Best way to check if file exists:
if _, err := os.Stat("/path/to/file"); err == nil || os.IsExist(err) {
// your code here if file exists
}
The function example:
func file_is_exists(f string) bool {
_, err := os.Stat(f)
if os.IsNotExist(err) {
return false
}
return err == nil
}
Let's look at few aspects first, both the function provided by os package of golang are not utilities but error checkers, what do I mean by that is they are just a wrapper to handle errors on cross platform.
So basically if os.Stat if this function doesn't give any error that means the file is existing if it does you need to check what kind of error it is, here comes the use of these two function os.IsNotExist and os.IsExist.
This can be understood as the Stat of the file throwing error because it doesn't exists or is it throwing error because it exist and there is some problem with it.
The parameter that these functions take is of type error, although you might be able to pass nil to it but it wouldn't make sense.
This also points to the fact that IsExist is not same as !IsNotExist, they are way two different things.
So now if you want to know if a given file exist in go, I would prefer the best way is:
if _, err := os.Stat(path/to/file); !os.IsNotExist(err){
//TODO
}
As mentioned in other answers, it is possible to construct the required behaviour / errors from using different flags with os.OpenFile. In fact, os.Create is just a sensible-defaults shorthand for doing so:
// Create creates or truncates the named file. If the file already exists,
// it is truncated. If the file does not exist, it is created with mode 0666
// (before umask). If successful, methods on the returned File can
// be used for I/O; the associated file descriptor has mode O_RDWR.
// If there is an error, it will be of type *PathError.
func Create(name string) (*File, error) {
return OpenFile(name, O_RDWR|O_CREATE|O_TRUNC, 0666)
}
You should combine these flags yourself to get the behaviour you are interested in:
// Flags to OpenFile wrapping those of the underlying system. Not all
// flags may be implemented on a given system.
const (
// Exactly one of O_RDONLY, O_WRONLY, or O_RDWR must be specified.
O_RDONLY int = syscall.O_RDONLY // open the file read-only.
O_WRONLY int = syscall.O_WRONLY // open the file write-only.
O_RDWR int = syscall.O_RDWR // open the file read-write.
// The remaining values may be or'ed in to control behavior.
O_APPEND int = syscall.O_APPEND // append data to the file when writing.
O_CREATE int = syscall.O_CREAT // create a new file if none exists.
O_EXCL int = syscall.O_EXCL // used with O_CREATE, file must not exist.
O_SYNC int = syscall.O_SYNC // open for synchronous I/O.
O_TRUNC int = syscall.O_TRUNC // truncate regular writable file when opened.
)
Depending on what you pick, you will get different errors.
Below is an example which will either truncate an existing file, or fail when a file exists.
openOpts := os.O_RDWR|os.O_CREATE
if truncateWhenExists {
openOpts |= os.O_TRUNC // file will be truncated
} else {
openOpts |= os.O_EXCL // file must not exist
}
f, err := os.OpenFile(filePath, openOpts, 0644)
// ... do stuff
This is how I check if a file exists in Go 1.16
package main
import (
"errors"
"fmt"
"io/fs"
"os"
)
func main () {
if _, err:= os.Stat("/path/to/file"); errors.Is(err, fs.ErrNotExist){
fmt.Print(err.Error())
} else {
fmt.Print("file exists")
}
}
Here is my take on a file exists method. It also checks that the file is not a directory and in case of an error, returns it as well.
// FileExists checks if a file exists (and it is not a directory).
func FileExists(filePath string) (bool, error) {
info, err := os.Stat(filePath)
if err == nil {
return !info.IsDir(), nil
}
if errors.Is(err, os.ErrNotExist) {
return false, nil
}
return false, err
}