How to calculate sha256 file checksum in Go - file

I need utility for Windows that calculates sha256 file checksum so that when I download
fedora I can verify checksum from here: https://fedoraproject.org/static/checksums/Fedora-18-i386-CHECKSUM
Microsoft utility from http://support.microsoft.com/kb/889768 does only md5 and sha1.
I don't want to use other downloadable tools that are not signed and not available from https or from sources that I don't know about, because it does not make any sense to download unsigned code over unencrypted connection or from untrusted source to verify signature of another code to trust it.
Luckily google provides possibility to use https for all downloads so I can download Go over secure connection and start from there.
Here is simple code that does that for a small file, but it's not very good for big files because it's not streaming.
package main
import (
"io/ioutil"
"crypto/sha256"
"os"
"log"
"encoding/hex"
)
func main() {
hasher := sha256.New()
s, err := ioutil.ReadFile(os.Args[1])
hasher.Write(s)
if err != nil {
log.Fatal(err)
}
os.Stdout.WriteString(hex.EncodeToString(hasher.Sum(nil)))
}
How to make it to use streams so that it works on any file size.

The crypto/sha256 godoc actually has a snippet that shows how to do that (it's basically the same code as James):
package main
import (
"crypto/sha256"
"fmt"
"io"
"log"
"os"
)
func main() {
f, err := os.Open("file.txt")
if err != nil {
log.Fatal(err)
}
defer f.Close()
h := sha256.New()
if _, err := io.Copy(h, f); err != nil {
log.Fatal(err)
}
fmt.Printf("%x", h.Sum(nil))
}

The SHA256 hasher implements the io.Writer interface, so one option would be to use the io.Copy() function to copy the data from an appropriate io.Reader in blocks. Something like this should do:
f, err := os.Open(os.Args[1])
if err != nil {
log.Fatal(err)
}
defer f.Close()
if _, err := io.Copy(hasher, f); err != nil {
log.Fatal(err)
}

Full example of md5sum:
func md5sum(filePath string) (string, error) {
file, err := os.Open(filePath)
if err != nil {
return "", err
}
defer file.Close()
hash := md5.New()
if _, err := io.Copy(hash, file); err != nil {
return "", err
}
return hex.EncodeToString(hash.Sum(nil)), nil
}
EncodeToString does not omits leading 0 bytes, so fmt.Println(hex.EncodeToString([]byte{0x00, 0x00, 0xA, 0xB, 0xC})) gives
00000a0b0c

Related

I get an error that I don't know for what reason. a file disappears when I run the program

I'm coding in Go, and I created a file handler and a program that prints the value of that file.
However, the file that should be created with file.Filename is deleted when I run it.
I don't know what the reason is, even if I try to debug, the answer doesn't come out, and even if I google it, I don't get the answer.
(64bit windows 10 (WSL2))
package main
import (
"fmt"
"io"
"io/ioutil"
"os"
"github.com/labstack/echo"
)
func checkErr(err error) {
if err != nil {
panic(err)
}
}
func readFile(filename string) string {
data, err := ioutil.ReadFile(filename)
checkErr(err)
return string(data)
}
func main() {
e := echo.New()
e.POST("/file", func(c echo.Context) error {
file, err := c.FormFile("file")
checkErr(err)
src, err := file.Open()
checkErr(err)
defer src.Close()
dst, err := os.Create(file.Filename)
checkErr(err)
defer dst.Close()
_, err = io.Copy(dst, src)
checkErr(err)
data := readFile(file.Filename)
fmt.Println(data)
return c.String(200, "sd")
})
e.Logger.Fatal(e.Start(":5000"))
}
I'm guessing that your file exists, but the code that you wrote is reading the file before the changes are "flushed to disk".
Right here:
defer dst.Close()
_, err = io.Copy(dst, src)
Should Close() or Sync() your writer as soon as possible, otherwise you may read before the write is finished. And since your readFile() function isn't re-using the file, you might as well just close (not Sync()) it immediately, not deferred
Try this:
_, err = io.Copy(dst, src)
dst.Close()
if err != nil {
}
There could be an error while copying, but we still want to Close() the file (if there wasn't an error during the os.Create, os.Open, or os.OpenFile...

How to filter out first few bad characters of Gerrit JSON api response

In querying Gerrit, they intentionally put a )]}' at the beginning of their api response, see: https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html. I am trying to remove it so the JSON is valid, but I'm unsure of the best way to do this in Go
this is my current program to query gerrit and pull out the changeID and the status from its json :
package main
import (
"encoding/json"
"flag"
"fmt"
"io/ioutil"
"log"
"net/http"
)
type gerritData struct {
ChangeID string `json:"change_id"`
Status string `json:"status"`
}
func gerritQuery(gerrit string) (gerritData, error) {
username := "redacted"
password := "redacted"
client := &http.Client{}
req, err := http.NewRequest("GET", "https://gerrit.company.com/a/changes/?q="+gerrit, nil)
req.SetBasicAuth(username, password)
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
respBody, err := ioutil.ReadAll(resp.Body)
// Trying to cut it out manually.
respBody = respBody[:len(respBody)-4]
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
var gerritResponse gerritData
if err := json.NewDecoder(resp.Body).Decode(&gerritResponse); err != nil {
panic(err.Error())
}
return gerritResponse, nil
}
func main() {
gerritFlag := flag.String("gerrit", "foo", "The Gerrit you want to query")
flag.Parse()
gerritResponse, _ := gerritQuery(*gerritFlag)
fmt.Println(gerritResponse)
}
Go is still complaining with panic: invalid character ')' looking for beginning of value. I'm still new to the language so any advice would be great.
The code in the question trims four bytes from the end of the response.
Trim the bytes from the beginning of the response:
respBoby = respBody[4:]

How to remove the last N bytes from a file

I want to delete thelast N bytes from file in Go,
Actually, this is already implemented is the os.Truncate() function. But this function takes the new size. So to use this, you have to first get the size of the file. For that, you may use os.Stat().
Wrapping it into a function:
func truncateFile(name string, bytesToRemove int64) error {
fi, err := os.Stat(name)
if err != nil {
return err
}
return os.Truncate(name, fi.Size()-bytesToRemove)
}
Using it to remove the last 5000 bytes:
if err := truncateFile("C:\\Test.zip", 5000); err != nil {
fmt.Println("Error:", err)
}
Another alternative is to use the File.Truncate() method for that. If we have an os.File, we may also use File.Stat() to get its size.
This is how it would look like:
func truncateFile(name string, bytesToRemove int64) error {
f, err := os.OpenFile(name, os.O_RDWR, 0644)
if err != nil {
return err
}
defer f.Close()
fi, err := f.Stat()
if err != nil {
return err
}
return f.Truncate(fi.Size() - bytesToRemove)
}
Using it is the same. This may be preferable if we're working on a file (we have it opened) and we have to truncate it. But in that case you'd want to pass os.File instead of its name to truncateFile().
Note: if you try to remove more bytes than the file currently has, truncateFile() will return an error.

How to connect to IBM Cloud object storage (Cleversafe) from Golang using the AWS S3 sdk

I'm trying to connect to IBM Cloud object storage (Cleversafe) using the following code and I get following error
"bad response: MissingRegion: could not find region configuration"
package main
import (
"fmt"
"os"
"bytes"
"net/http"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awsutil"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/aws/session"
)
func main() {
fmt.Printf("Starting ...")
aws_access_key_id := "some id"
aws_secret_access_key := "some key"
token := ""
creds := credentials.NewStaticCredentials(aws_access_key_id, aws_secret_access_key, token)
fmt.Printf("creds: ", creds)
_, err := creds.Get()
if err != nil {
fmt.Printf("bad credentials: %s", err)
}
//EndpointResolver endpoints.Resolver
cfg := aws.NewConfig().WithEndpoint("dal.ibmselect.objstor.com").WithCredentials(creds)
fmt.Printf("cfg: %+v \n", cfg)
//cfg := aws.NewConfig().WithEndpointResolver()
//cfg := aws.NewConfig().WithCredentials(creds)
svc := s3.New(session.New(), cfg)
fmt.Printf("svc: %+v \n", svc)
file, err := os.Open("./test.jpg")
if err != nil {
fmt.Printf("err opening file: %s", err)
}
defer file.Close()
fileInfo, _ := file.Stat()
size := fileInfo.Size()
buffer := make([]byte, size) // read file content to buffer
file.Read(buffer)
fileBytes := bytes.NewReader(buffer)
fileType := http.DetectContentType(buffer)
path := file.Name()
params := &s3.PutObjectInput{
Bucket: aws.String("cosv1-jlnab"),
Key: aws.String(path),
Body: fileBytes,
ContentLength: aws.Int64(size),
ContentType: aws.String(fileType),
}
resp, err := svc.PutObject(params)
if err != nil {
fmt.Printf("bad response: %s \n", err)
}
fmt.Printf("response %s", awsutil.StringValue(resp))
}
Thanks
Following is a working sample code:
package main
import (
"fmt"
"os"
"bytes"
"net/http"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awsutil"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/aws/session"
)
func main() {
fmt.Printf("Starting ...")
aws_access_key_id := "Access key from Bluemix credentials"
aws_secret_access_key := "Secret key from Bluemix credentials"
token := ""
creds := credentials.NewStaticCredentials(aws_access_key_id, aws_secret_access_key, token)
fmt.Printf("creds: ", creds)
_, err := creds.Get()
if err != nil {
fmt.Printf("bad credentials: %s", err)
}
//EndpointResolver endpoints.Resolver
cfg := aws.NewConfig().WithRegion("cleversafe").WithEndpoint("s3-api.us-geo.objectstorage.softlayer.net").WithCredentials(creds)
fmt.Printf("cfg: %+v \n", cfg)
svc := s3.New(session.New(), cfg)
fmt.Printf("svc: %+v \n", svc)
file, err := os.Open("./test.jpg")
if err != nil {
fmt.Printf("err opening file: %s", err)
}
defer file.Close()
fileInfo, _ := file.Stat()
size := fileInfo.Size()
buffer := make([]byte, size) // read file content to buffer
file.Read(buffer)
fileBytes := bytes.NewReader(buffer)
fileType := http.DetectContentType(buffer)
path := file.Name()
params := &s3.PutObjectInput{
Bucket: aws.String("test-ophir"),
Key: aws.String(path),
Body: fileBytes,
ContentLength: aws.Int64(size),
ContentType: aws.String(fileType),
}
resp, err := svc.PutObject(params)
if err != nil {
fmt.Printf("bad response: %s \n", err)
}
fmt.Printf("response %s", awsutil.StringValue(resp))
}
Essentially this happens because IBM COS and AWS S3 handle the concept of a 'region' somewhat differently. This error is being thrown because the AWS Go SDK doesn't know what value to provide for the 'region' variable used to calculate the authorization header. In this context, IBM COS doesn't care what value, as long is it isn't null. The easiest way is to provide an environment variable via the command line: export AWS_REGION=foo where foo can be any string, although it could also be set as a configuration variable at client creation within your application.
I'd recommend using something that makes sense, like us or us-south as IBM COS will probably enforce this at some point down the road by linking the concept of 'region' to the AWS LocationConstraint variable, which is currently used by IBM COS to define the storage class of a bucket, although this distinction will be made explicit at some point in the future.

How to write the output of this statement into a file in Golang

I'm trying to write the output of the statement below into a text file but I can't seem to find out if there is a printf function that writes directly to a text file. For example if the code below produces the results [5 1 2 4 0 3] I would want to read this into a text file for storage and persistence. Any ideas please?
The code I want to goto the text file:
//choose random number for recipe
r := rand.New(rand.NewSource(time.Now().UnixNano()))
i := r.Perm(5)
fmt.Printf("%v\n", i)
fmt.Printf("%d\n", i[0])
fmt.Printf("%d\n", i[1])
You can use fmt.Fprintf together with an io.Writer, which would represent a handle to your file.
Here is a simple example:
func check(err error) {
if err != nil {
panic(err)
}
}
func main() {
f, err := os.Create("/tmp/yourfile")
check(err)
defer f.Close()
w := bufio.NewWriter(f)
//choose random number for recipe
r := rand.New(rand.NewSource(time.Now().UnixNano()))
i := r.Perm(5)
_, err = fmt.Fprintf(w, "%v\n", i)
check(err)
_, err = fmt.Fprintf(w, "%d\n", i[0])
check(err)
_, err = fmt.Fprintf(w, "%d\n", i[1])
check(err)
w.Flush()
}
More ways of writing to file in Go are shown here.
Note that I have used panic() here just for the sake of brevity, in the real life scenario you should handle errors appropriately (which in most cases means something other than exiting the program, what panic() does).
This example will write the values into the output.txt file.
package main
import (
"bufio"
"fmt"
"math/rand"
"os"
"time"
)
func main() {
file, err := os.OpenFile("output.txt", os.O_WRONLY|os.O_CREATE, 0666)
if err != nil {
fmt.Println("File does not exists or cannot be created")
os.Exit(1)
}
defer file.Close()
w := bufio.NewWriter(file)
r := rand.New(rand.NewSource(time.Now().UnixNano()))
i := r.Perm(5)
fmt.Fprintf(w, "%v\n", i)
w.Flush()
}
Use os package to create file and then pass it to Fprintf
file, fileErr := os.Create("file")
if fileErr != nil {
fmt.Println(fileErr)
return
}
fmt.Fprintf(file, "%v\n", i)
This should write to file.

Resources