I've been dealing with an issue where I have to put an io.Reader instance as a parameter to a function provided as an end-point by an api. The task I need to do is to upload local folder to company's cloud storage.
func (s *server) uploadFileToPutIo(w http.ResponseWriter, r *http.Request) {
tokenSource := oauth2.StaticTokenSource(&oauth2.Token{AccessToken: token})
oauthClient := oauth2.NewClient(context.TODO(), tokenSource)
client := putio.NewClient(oauthClient)
var testIO io.Reader // ?
upload, err := client.Files.Upload(context.TODO(), testIO, "test", 0)
if err != nil {
log.Fatal(err)
}
fmt.Println(upload.File)
sendResponse(w, []byte("successful"), http.StatusOK)
}
When I make a request to this end-point /upload under POST method. I get the following error.
2021/12/01 18:28:47 http: panic serving 127.0.0.1:61057: runtime error: invalid memory address or nil pointer dereference
goroutine 8 [running]:
net/http.(*conn).serve.func1(0xc000108d20)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:1804 +0x153
panic(0x1390ae0, 0x164fdd0)
/usr/local/Cellar/go/1.16.6/libexec/src/runtime/panic.go:971 +0x499
io.copyBuffer(0x1462700, 0xc000026360, 0x0, 0x0, 0xc000170000, 0x8000, 0x8000, 0x0, 0x0, 0x13d5e01)
/usr/local/Cellar/go/1.16.6/libexec/src/io/io.go:423 +0x10b
io.Copy(...)
/usr/local/Cellar/go/1.16.6/libexec/src/io/io.go:382
github.com/putdotio/go-putio/putio.(*FilesService).Upload(0xc000010088, 0x1468390, 0xc00001c088, 0x0, 0x0, 0x13ef46f, 0x6, 0x0, 0x170d108, 0x90, ...)
/Users/barisertas/go/pkg/mod/github.com/putdotio/go-putio/putio#v0.0.0-20200123120452-16d982cac2b8/files.go:235 +0x187
main.(*server).uploadFileToPutIo(0xc000010028, 0x1467d60, 0xc00014a2a0, 0xc000154500)
/Users/barisertas/workspace/mini-project/api/handler.go:79 +0xe5
net/http.HandlerFunc.ServeHTTP(0xc000012db0, 0x1467d60, 0xc00014a2a0, 0xc000154500)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2049 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000144000, 0x1467d60, 0xc00014a2a0, 0xc000154300)
/Users/barisertas/go/pkg/mod/github.com/gorilla/mux#v1.8.0/mux.go:210 +0xd3
net/http.serverHandler.ServeHTTP(0xc00014a000, 0x1467d60, 0xc00014a2a0, 0xc000154300)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2867 +0xa3
net/http.(*conn).serve(0xc000108d20, 0x1468400, 0xc00005e300)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:1932 +0x8cd
created by net/http.(*Server).Serve
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2993 +0x39b
2021/12/01 18:28:47 http: panic serving 127.0.0.1:61059: runtime error: invalid memory address or nil pointer dereference
goroutine 11 [running]:
net/http.(*conn).serve.func1(0xc000108dc0)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:1804 +0x153
panic(0x1390ae0, 0x164fdd0)
/usr/local/Cellar/go/1.16.6/libexec/src/runtime/panic.go:971 +0x499
io.copyBuffer(0x1462700, 0xc000026400, 0x0, 0x0, 0xc000198000, 0x8000, 0x8000, 0x0, 0x0, 0x13d5e01)
/usr/local/Cellar/go/1.16.6/libexec/src/io/io.go:423 +0x10b
io.Copy(...)
/usr/local/Cellar/go/1.16.6/libexec/src/io/io.go:382
github.com/putdotio/go-putio/putio.(*FilesService).Upload(0xc0000100c8, 0x1468390, 0xc00001c088, 0x0, 0x0, 0x13ef46f, 0x6, 0x0, 0x170d108, 0x90, ...)
/Users/barisertas/go/pkg/mod/github.com/putdotio/go-putio/putio#v0.0.0-20200123120452-16d982cac2b8/files.go:235 +0x187
main.(*server).uploadFileToPutIo(0xc000010028, 0x1467d60, 0xc00014a380, 0xc000154800)
/Users/barisertas/workspace/mini-project/api/handler.go:79 +0xe5
net/http.HandlerFunc.ServeHTTP(0xc000012db0, 0x1467d60, 0xc00014a380, 0xc000154800)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2049 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000144000, 0x1467d60, 0xc00014a380, 0xc000154600)
/Users/barisertas/go/pkg/mod/github.com/gorilla/mux#v1.8.0/mux.go:210 +0xd3
net/http.serverHandler.ServeHTTP(0xc00014a000, 0x1467d60, 0xc00014a380, 0xc000154600)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2867 +0xa3
net/http.(*conn).serve(0xc000108dc0, 0x1468400, 0xc00005e580)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:1932 +0x8cd
created by net/http.(*Server).Serve
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2993 +0x39b
And this is the documentation of the function I am trying to use:
// Upload reads from given io.Reader and uploads the file contents to Put.io
// servers under directory given by parent. If parent is negative, user's
// preferred folder is used.
//
// If the uploaded file is a torrent file, Put.io will interpret it as a
// transfer and Transfer field will be present to represent the status of the
// tranfer. Likewise, if the uploaded file is a regular file, Transfer field
// would be nil and the uploaded file will be represented by the File field.
//
// This method reads the file contents into the memory, so it should be used for
// <150MB files.
func (f *FilesService) Upload(ctx context.Context, r io.Reader, filename string, parent int64) (Upload, error) {
if filename == "" {
return Upload{}, fmt.Errorf("filename cannot be empty")
}
var buf bytes.Buffer
mw := multipart.NewWriter(&buf)
// negative parent means use user's preferred download folder.
if parent >= 0 {
err := mw.WriteField("parent_id", itoa(parent))
if err != nil {
return Upload{}, err
}
}
formfile, err := mw.CreateFormFile("file", filename)
if err != nil {
return Upload{}, err
}
_, err = io.Copy(formfile, r)
if err != nil {
return Upload{}, err
}
err = mw.Close()
if err != nil {
return Upload{}, err
}
req, err := f.client.NewRequest(ctx, "POST", "/v2/files/upload", &buf)
if err != nil {
return Upload{}, err
}
req.Header.Set("Content-Type", mw.FormDataContentType())
var response struct {
Upload
}
_, err = f.client.Do(req, &response)
if err != nil {
return Upload{}, err
}
return response.Upload, nil
}
I am confused about putting the io.Reader instance to a function. How can I put the io.Reader function properly under this case. Just an instance after creating as follows:
var testIO io.Reader or should I do some extra operations?
runtime error: invalid memory address or nil pointer dereference
As you surely know, this is because you've declared an io.Reader but you haven't set its value, so it is still equal to the default value of an interface, which is nil.
var testIO io.Reader // ?
The point of passing the io.Reader to Upload is to provide the data to be uploaded. By passing an io.Reader, an arbitrary source of data can provide an abitrary quantity of bytes, unrestricted by memory availability (unlike []byte, which would require holding all data in memory before uploading). io.Reader is commonly used for providing data for this kind of "streaming" operation.
Upload reads from given io.Reader and uploads the file contents
That io.Reader should be the source of the data to upload.
io.Reader could be a file from os.Open().
But it could be anything that satisfies io.Reader - for example, it could also be a bytes.Buffer.
It could even be something more esoteric, like the result of an GetObject API call against the popular S3 service from AWS, which also returns an io.ReadCloser which satisfies io.Reader.
io.Reader is a great example of how Go interfaces allow independent libraries to be connected to each other. the SDK you're using doesn't care what io.Reader it's passed; it is enough that the value satisfies io.Reader, a requirement enforced at compile time. You can pass it anything that satisfies io.Reader, and the interface type guarantees that Upload() will be able to handle it properly.
Upload takes an io.Reader. If you want to pass it something like an *os.File from os.Open or an io.ReadCloser from say S3 GetObject, that works because *os.File and io.ReadCloser satisfy io.Reader. But since Upload takes io.Reader, you can be confident that it will call nothing but the functions defined in io.Reader. That means you'll have to do any closing yourself, after Upload is called.
Make sure to take the time to understand how io.Reader leaves the input to this function open-ended while also being specific about the interface it expects. This is one of the most important concepts in Go.
This:
var testIO io.Reader
is equivalent to this:
testIO := io.Reader(nil)
so this is why you are getting a panic of a nil-pointer reference:
2021/12/01 18:28:47 http: panic serving 127.0.0.1:61059: runtime error: invalid memory address or nil pointer dereference
goroutine 11 [running]:
An io.Reader is an interface which allows one to pass generic values, provided they implement the interface (i.e. implement the Read method).
Since you are uploading a file, your byte stream should come from an OS file. os.File implements the correct Read method - so is compatible io.Reader.
So try:
f, err := os.Open(uploadFilePath)
if err != nil { /* ... */ }
upload, err := client.Files.Upload(context.TODO(), f, "test", 0)
When you define a variable, it is initial value is the zero value for that type. io.Reader is an interface and the zero value for it is nil. Thus the nil pointer dereference error. Simply initialize the io.Reader before passing it to Upload:
file, err := os.Open("path/to/file")
// if err != nil { ... }
upload, err := client.Files.Upload(context.TODO(), file, "test", 0)
Related
In order to reuse CGO pointers (type C.uintptr_t) between multiple applications, I tried to use go rpc to pass the initialized pointer, but the program reported an error: rpc: gob error encoding body: gob: type not registered for interface: main._Ctype_ulong. I think there might be some issues with pointer types.
1. init func
func initApp(configPath *C.char) C.uintptr_t
2. App1, daemon process, call the init func, and pass the pointer to another by go rpc
var globalSDKPtr C.ulong
type HelloService struct{}
func (p *HelloService) Hello(request string, reply *C.ulong) error {
*reply = globalSDKPtr
return nil
}
func startRPS() {
rpc.RegisterName("HelloService", new(HelloService))
listener, err := net.Listen("tcp", ":1234")
if err != nil {
log.Fatal("ListenTCP error:", err)
}
conn, err := listener.Accept()
if err != nil {
log.Fatal("Accept error:", err)
}
rpc.ServeConn(conn)
}
3. App2, recevie the pointer reuse it.
client, err := rpc.Dial("tcp", "localhost:1234")
if err != nil {
log.Fatal("dialing:", err)
}
var reply C.ulong
err = client.Call("HelloService.Hello", "hello", &reply)
if err != nil {
log.Fatal(err)
}
res := C.query(reply)
I guess the reason for the problem is that my thinking is wrong. The way to reuse cgo pointers may not be the way of go rpc, but shared memory, but in any case, passing cgo-related things is always confusing. . Can anyone help me out.
I've got trouble overwriting a files content with zeros. The problem is that the very last byte of the original file remains, even when I exceed its size by 100 bytes. Someone got an idea what I'm missing?
func (h PostKey) ServeHTTP(w http.ResponseWriter, r *http.Request) {
f, err := os.Create("received.dat")
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
return
}
defer f.Close()
_, err = io.Copy(f, r.Body)
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
return
}
// Retrieve filesize
size, _ := f.Seek(0, 1)
zeroFilled := make([]byte, size + 100)
n, err := f.WriteAt(zeroFilled, 0)
if err != nil {
return
}
fmt.Printf("Size: %d\n", size) // prints 13
fmt.Printf("Bytes written: %d\n", n) // prints 113
}
The problem may occurred because the data is written into a same file (shared resource) inside an http handler, and the handler itself may be executed concurrently. You need to lock access to the file during data serialization (overwriting process). Quick solution will be:
import (
"sync"
//... other packages
)
var muFile sync.Mutex
func (h PostKey) ServeHTTP(w http.ResponseWriter, r *http.Request) {
muFile.Lock()
defer muFile.Unlock()
f, err := os.Create("received.dat")
//other statements
//...
}
If your server load is low, the above solution will be fine. But if your server needs to handle a lot of requests concurrently, you need to use different approach (although the rule is the same, lock access to any shared resource).
I was writing to the file and trying to overwrite it in the same context, and so parts of the first write operation were still in memory and not yet written to the disk. By using f.Sync() to flush everything after copying the bodys content I was able to fix the issue.
I try to serialize a structured data to file. I looked through some examples and made such construction:
func (order Order) Serialize(folder string) {
b := bytes.Buffer{}
e := gob.NewEncoder(&b)
err := e.Encode(order)
if err != nil { panic(err) }
os.MkdirAll(folder, 0777)
file, err := os.Create(folder + order.Id)
if err != nil { panic(err) }
defer file.Close()
writer := bufio.NewWriter(file)
n, err := writer.Write(b.Bytes())
fmt.Println(n)
if err != nil {
panic(err)
}
}
Serialize is a method serializing its object to file called by it's id property. I looked through debugger - byte buffer contains data before writing. I mean object is fully initialized. Even n variable representing quantity of written bytes is more than a thousand - the file shouldn't be empty at all. The file is created but it is totally empty. What's wrong?
bufio.Writer (as the package name hints) uses a buffer to cache writes. If you ever use it, you must call Writer.Flush() when you're done writing to it to ensure the buffered data gets written to the underlying io.Writer.
Also note that you can directly write to an os.File, no need to create a buffered writer "around" it. (*os.File implements io.Writer).
Also note that you can create the gob.Encoder directly directed to the os.File, so even the bytes.Buffer is unnecessary.
Also os.MkdirAll() may fail, check its return value.
Also it's better to "concatenate" parts of a file path using filepath.Join() which takes care of extra / missing slashes at the end of folder names.
And last, it would be better to signal the failure of Serialize(), e.g. with an error return value, so the caller party has the chance to examine if the operation succeeded, and act accordingly.
So Order.Serialize() should look like this:
func (order Order) Serialize(folder string) error {
if err := os.MkdirAll(folder, 0777); err != nil {
return err
}
file, err := os.Create(filepath.Join(folder, order.Id))
if err != nil {
return err
}
defer file.Close()
if err := gob.NewEncoder(file).Encode(order); err != nil {
return err
}
return nil
}
I have code like this:
package main
import (
"database/sql"
"flag"
"fmt"
"log"
"net/http"
"os"
_ "github.com/go-sql-driver/mysql"
)
// Default constant for configuration
const DefaultHTTPAddr = ":8080"
const DefaultDSN = "root:root#tcp(127.0.0.1:3306)/librarian"
// Parameters
var (
httpAddr string
dsn string
db *sql.DB
)
// init initializes this package.
func init() {
flag.StringVar(&httpAddr, "addr", DefaultHTTPAddr, "Set the HTTP bind address")
flag.StringVar(&dsn, "dsn", DefaultDSN, "Set the Data Source Name")
flag.Usage = func() {
fmt.Fprintf(os.Stderr, "Usage: %s [options]\n", os.Args[0])
flag.PrintDefaults()
}
}
type Book struct {
id int
title string
author string
}
func main() {
flag.Parse()
var err error
db, err := sql.Open("mysql", DefaultDSN)
if err != nil {
log.Fatal(err)
}
if err = db.Ping(); err != nil {
log.Fatal(err)
}
// handler
http.HandleFunc("/", homepage)
http.HandleFunc("/books", booksIndex)
log.Println("httpd started successfully")
http.ListenAndServe(httpAddr, nil)
}
func booksIndex(w http.ResponseWriter, r *http.Request) {
if r.Method != "GET" {
http.Error(w, http.StatusText(405), 405)
return
}
rows, err := db.Query("SELECT * FROM books")
if err != nil {
log.Fatal(err)
}
defer rows.Close()
books := make([]*Book, 0)
for rows.Next() {
bk := new(Book)
err = rows.Scan(&bk.id, &bk.title, &bk.author)
if err != nil {
log.Fatal(err)
}
books = append(books, bk)
}
if err = rows.Err(); err != nil {
log.Fatal(err)
}
for _, bk := range books {
fmt.Fprintf(w, "%v, %v, %v\n", bk.id, bk.title, bk.author)
}
}
func homepage(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Welcome!")
}
Everytime I try to access /books it always panic.
$ curl localhost:8080/books
curl: (52) Empty reply from server
Like this one:
2016/05/31 11:56:38 http: panic serving 127.0.0.1:51711: runtime error: invalid memory address or nil pointer dereference
goroutine 6 [running]:
net/http.(*conn).serve.func1(0xc820074100)
/usr/local/opt/go/libexec/src/net/http/server.go:1389 +0xc1
panic(0x357b40, 0xc82000a150)
/usr/local/opt/go/libexec/src/runtime/panic.go:443 +0x4e9
database/sql.(*DB).conn(0x0, 0xc820010b01, 0xc8200de000, 0x0, 0x0)
/usr/local/opt/go/libexec/src/database/sql/sql.go:778 +0xac9
database/sql.(*DB).query(0x0, 0x4022c0, 0x13, 0x0, 0x0, 0x0, 0x3c4601, 0x6, 0x0, 0x0)
/usr/local/opt/go/libexec/src/database/sql/sql.go:1073 +0x46
database/sql.(*DB).Query(0x0, 0x4022c0, 0x13, 0x0, 0x0, 0x0, 0xc820016280, 0x0, 0x0)
/usr/local/opt/go/libexec/src/database/sql/sql.go:1061 +0xa3
main.booksIndex(0x6979c0, 0xc82006da00, 0xc8200e2000)
/Users/rahmatawaludin/gocode/src/github.com/rahmatawaludin/librarian/main.go:68 +0xd9
net/http.HandlerFunc.ServeHTTP(0x4666f8, 0x6979c0, 0xc82006da00, 0xc8200e2000)
/usr/local/opt/go/libexec/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc820010ba0, 0x6979c0, 0xc82006da00, 0xc8200e2000)
/usr/local/opt/go/libexec/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc820074080, 0x6979c0, 0xc82006da00, 0xc8200e2000)
/usr/local/opt/go/libexec/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc820074100)
/usr/local/opt/go/libexec/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
/usr/local/opt/go/libexec/src/net/http/server.go:2137 +0x44e
I thought I could access db from booksIndex as I set in to be global variable. When I move the db initialization to booksIndex the errors doesn't show up.
What parts is wrong in my code?
Also, I'm new in Golang. If you have any suggestion on how to organize my code, please tell me. Thanks.. :)
Your db variable in function is shadowing global variable.
When you do this:
db,err:=
It assigns it to a new local variable db.
This is because it's not from same block. According to the standard:
Unlike regular variable declarations, a short variable declaration may redeclare variables provided they were originally declared earlier
in the same block (or the parameter lists if the block is the function
body) with the same type, and at least one of the non-blank variables
is new. As a consequence, redeclaration can only appear in a
multi-variable short declaration. Redeclaration does not introduce a
new variable; it just assigns a new value to the original.
So global variable remains nil pointer. And when it is access, you get nil pointer dereference
Change it to d and then assign it later to db. Or which is more correct(as given in comment by rahmat):=
d , err =
I would recommend you to have separate files for handler, models etc. And look into https://github.com/mattermost/platform to how to organise the code.
I am trying to use a file instead of a DB to get a prototype up and running. I have a program that (1) reads existing content from the file to a map, (2) takes JSON POSTs that add content to the map, (3) on exit, writes to the file.
First, the file is not being created. Then I created an empty file. It is not being written to.
I am trying to read the file, determine if there is existing content. If there is not existing content, create a blank map. If there is existing content, unmarshal it into a new map.
func writeDB() {
eventDBJSON, err := json.Marshal(eventDB)
if err != nil {
panic(err)
}
err2 := ioutil.WriteFile("/Users/sarah/go/dat.txt", eventDBJSON, 0777)
if err2 != nil {
panic(err2)
}
}
func main() {
dat, err := ioutil.ReadFile("/Users/sarah/go/dat.txt")
if err != nil {
panic(err)
}
if dat == nil {
eventDB = DB{
events: map[string]event{},
}
} else {
if err2 := json.Unmarshal(dat, &eventDB); err2 != nil {
panic(err2)
}
}
router := httprouter.New()
router.POST("/join", JoinEvent)
router.POST("/create", CreateEvent)
log.Fatal(http.ListenAndServe(":8080", router))
defer writeDB()
}
There is no way for the server to ever reach defer writeDB().
http.ListenAndServe blocks, and if it did return anything, you log.Fatal that, which exits your app at that point.
You can't intercept all ways an app can exit, getting SIGKILL, machine loss of power, etc.
I'm assuming you really just want to write some code, bounce the server, repeat
If that's the case, then Ctrl-C is good enough.
If you want to write your file on Ctrl-C, look at the signal package.
Also, defer on the last line of a function really has no purpose as defer basically means "do this last".
you can use (*os.File).Stat() to get a file's FileInfo which contain its size
file, err := os.Open( filepath )
if err != nil {
// handle error
}
fi, err := file.Stat()
if err != nil {
// handle error
}
s := fi.Size()