Idea. Have tcp server and client. On client writing info about file and file contents to to tcp connection. On server reading that data, creating files and want to return errors if any back to client.
On client using fmt.Fprintf(conn, "test.txt\n") for file info and io.Copy(conn, f), where f is file contents
On server using ReadString('\n') for info about file and
w := io.MultiWriter(writers...)
_, err = io.Copy(w, r)
if err != nil {
fmt.Println([]error{err})
}
And all goes well at 1st, but then I ran into problem on server side with io.Copy going into infinite loop trying to read data from client even when there's none. It doesn't understand that it's EOF or anything.
Tried using conn.Read and similar in loop on server side instead of IO.Copy - and got infinite loop again.
w := io.MultiWriter(writers...)
for {
input := make([]byte, 1024*4)
n, err := conn.Read(input)
if n == 0 || err != nil {
log.Error(err, "Read error:")
break
}
_, err = w.Write(input[0:n])
if err != nil {
log.Error(err, "Failed to copy data on server")
m = m + fmt.Sprintf("failed to copy data on server: %s\n", err)
}
}
Read some articles and ran into explanation that it's "normal" behaviour for such blocks during server-client exchanges with readers-writers. One of the suggestions was to use timeout for connection reading(but won't work for me, cuz files can be huge and hard to figure out how long is enough)
Another option is sending some STOP-WORD from client to server to break the loop
But I wander if there's some other way(may be I'm missing out some instruments in go) to achieve desired result without infinite loops, time-outs and stop-words.
Related
In my case I have a logic which should upload large chunked files for example if I have a file which size is 10mb , I need to send PUT request with 1mb file chunks , 10 times, but the mgo (mgo.v2) is not allow to open file for writing
func UploadFileChunk(rw http.ResponseWriter,rq *http.Request) {
fileid:= mux.Vars(rq)["fileid"]
rq.ParseMultipartForm(10000)
formFile:=rq.MultipartForm.File["file"]
content,err:= formFile[0].Open()
defer content.Close()
if err != nil {
http.Error(rw,err.Error(),http.StatusInternalServerError)
return
}
file,err:= db.GridFS("fs").OpenId(bson.ObjectIdHex(fileid))
if err != nil {
http.Error(rw,err.Error(),http.StatusInternalServerError)
return
}
data,err := ioutil.ReadAll(content)
n,_:= file.Write(data)
file.Close()
// Write a log type message
fmt.Printf("%d bytes written to the Mongodb instance\n", n)
}
So I want to every time write a new chunk but 1) The mgo not allows to open file for writing 2) I don't know is this way a good?
I've got trouble overwriting a files content with zeros. The problem is that the very last byte of the original file remains, even when I exceed its size by 100 bytes. Someone got an idea what I'm missing?
func (h PostKey) ServeHTTP(w http.ResponseWriter, r *http.Request) {
f, err := os.Create("received.dat")
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
return
}
defer f.Close()
_, err = io.Copy(f, r.Body)
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
return
}
// Retrieve filesize
size, _ := f.Seek(0, 1)
zeroFilled := make([]byte, size + 100)
n, err := f.WriteAt(zeroFilled, 0)
if err != nil {
return
}
fmt.Printf("Size: %d\n", size) // prints 13
fmt.Printf("Bytes written: %d\n", n) // prints 113
}
The problem may occurred because the data is written into a same file (shared resource) inside an http handler, and the handler itself may be executed concurrently. You need to lock access to the file during data serialization (overwriting process). Quick solution will be:
import (
"sync"
//... other packages
)
var muFile sync.Mutex
func (h PostKey) ServeHTTP(w http.ResponseWriter, r *http.Request) {
muFile.Lock()
defer muFile.Unlock()
f, err := os.Create("received.dat")
//other statements
//...
}
If your server load is low, the above solution will be fine. But if your server needs to handle a lot of requests concurrently, you need to use different approach (although the rule is the same, lock access to any shared resource).
I was writing to the file and trying to overwrite it in the same context, and so parts of the first write operation were still in memory and not yet written to the disk. By using f.Sync() to flush everything after copying the bodys content I was able to fix the issue.
I am trying to read a file from client and then send it to server.
It goes like this, you input send <fileName> in the client program, then <fileName> will be sent to server. The server read 2 things from the client via TCP connection, first the command send <fileName> and second the content of the file.
However, sometimes my program will randomly include the file content in the <fileName> string. For example, say I have a text file called xyz.txt, the content of which is "Hellow world". The server sometimes receive send xyz.txtHellow world. Sometimes it doesn't and it works just fine.
I think that it is the problem of synchronization or not flushing reader/writer buffer. But I am not quite sure.
Thanks in advance!
Client code:
func sendFileToServer(fileName string, connection net.Conn) {
fileBuffer := make([]byte, BUFFER_SIZE)
var err error
file, err := os.Open(fileName) // For read access.
lock := make(chan int)
w := bufio.NewWriter(connection)
go func(){
w.Write([]byte("send " + fileName))
w.Flush()
lock <- 1
}()
<-lock
// make a read buffer
r := bufio.NewReader(file)
//read file until there is an error
for err == nil || err != io.EOF {
//read a chunk
n, err := r.Read(fileBuffer)
if err != nil && err != io.EOF {
panic(err)
}
if n == 0 {
break
}
// write a chunk
if _, err := w.Write(fileBuffer[:n]); err != nil {
panic(err)
}
}
file.Close()
connection.Close()
fmt.Println("Finished sending.")
}
Server code: (connectionHandler is a goroutine that is invoked for every TCP connection request from client)
func connectionHandler(connection net.Conn, bufferChan chan []byte, stringChan chan string) {
buffer := make([]byte, 1024)
_, error := connection.Read(buffer)
if error != nil {
fmt.Println("There is an error reading from connection", error.Error())
stringChan<-"failed"
return
}
fmt.Println("command recieved: " + string(buffer))
if("-1"==strings.Trim(string(buffer), "\x00")){
stringChan<-"failed"
return
}
arrayOfCommands := strings.Split(string(buffer)," ")
arrayOfCommands[1] = strings.Replace(arrayOfCommands[1],"\n","",-1)
fileName := strings.Trim(arrayOfCommands[1], "\x00")
if arrayOfCommands[0] == "get" {
fmt.Println("Sending a file " + arrayOfCommands[1])
sendFileToClient(fileName, connection, bufferChan, stringChan)
} else if arrayOfCommands[0] == "send" {
fmt.Println("Getting a file " + arrayOfCommands[1])
getFileFromClient(fileName, connection, bufferChan, stringChan)
} else {
_, error = connection.Write([]byte("bad command"))
}
fmt.Println("connectionHandler finished")
}
func getFileFromClient(fileName string, connection net.Conn,bufferChan chan []byte, stringChan chan string) { //put the file in memory
stringChan<-"send"
fileBuffer := make([]byte, BUFFER_SIZE)
var err error
r := bufio.NewReader(connection)
for err == nil || err != io.EOF {
//read a chunk
n, err := r.Read(fileBuffer)
if err != nil && err != io.EOF {
panic(err)
}
if n == 0 {
break
}
bufferChan<-fileBuffer[:n]
stringChan<-fileName
}
connection.Close()
return
}
TCP is a stream protocol. It doesn't have messages. The network is (within some limits we don't need to concern us about) free to send your data one byte at a time or everything at once. And even if you get lucky and the network sends your data in packets like you want them there's nothing that prevents the receive side from concatenating the packets into one buffer.
In other words: there is nothing that will make each Read call return as many bytes as you wrote with some specific Write calls. You sometimes get lucky, sometimes, as you noticed, you don't get lucky. If there are no errors, all the reads you do from the stream will return all the bytes you wrote, that's the only guarantee you have.
You need to define a proper protocol.
This is not related to Go. Every programming language will behave this way.
I'm trying to understand how I can read content of the file, calculate its hash and return its bytes in one Go. So far, I'm doing this in two steps, e.g.
// calculate file checksum
hasher := sha256.New()
f, err := os.Open(fname)
if err != nil {
msg := fmt.Sprintf("Unable to open file %s, %v", fname, err)
panic(msg)
}
defer f.Close()
b, err := io.Copy(hasher, f)
if err != nil {
panic(err)
}
cksum := hex.EncodeToString(hasher.Sum(nil))
// read again (!!!) to get data as bytes array
data, err := ioutil.ReadFile(fname)
Obviously it is not the most efficient way to do this, since read happens twice, once in copy to pass to hasher and another in ioutil to read file and return list of bytes. I'm struggling to understand how I can combine these steps together and do in one go, read data once, calculate any hash and return it along with list of bytes to another layer.
If you want to read a file, without creating a copy of the entire file in memory, and at the same time calculate its hash, you can do so with a TeeReader:
hasher := sha256.New()
f, err := os.Open(fname)
data := io.TeeReader(f, hasher)
// Now read from data as usual, which is still a stream.
What happens here is that any bytes that are read from data (which is a Reader just like the file object f is) will be pushed to hasheras well.
Note, however, that hasher will produce the correct hash only once you have read the entire file through data, and not until then. So if you need the hash before you decide whether or not you want to read the file, you are left with the options of either doing it in two passes (for example like you are now), or to always read the file but discard the result if the hash check failed.
If you do read the file in two passes, you could of course buffer the entire file data in a byte buffer in memory. However, the operating system will typically cache the file you just read in RAM anyway (if possible), so the performance benefit of doing a buffered two-pass solution yourself rather than just doing two passes over the file is probably negligible.
You can write bytes directly to the hasher. For example:
package main
import (
"crypto/sha256"
"encoding/hex"
"io/ioutil"
)
func main() {
hasher := sha256.New()
data, err := ioutil.ReadFile("foo.txt")
if err != nil {
panic(err)
}
hasher.Write(data)
cksum := hex.EncodeToString(hasher.Sum(nil))
println(cksum)
}
As the Hash interface embeds io.Writer. This allows you to read the bytes from the file once, write them into the hasher then also return them out.
Do data, err := ioutil.ReadFile(fname) first. You'll have your slice of bytes. Then create your hasher, and do hasher.Write(data).
If you plan on hashing files you shouldn't read the whole file into memory because... there are large files that don't fit into RAM. Yes, in practice you very rarely will run into such out-of-memory issues but you can easily prevent them. The Hash interface is an io.Writer. Usually, the Hash packages have a New function that return a Hash. This allows you to read the file in blocks and continuously feed it to the Write method of the Hash you have. You may also use methods like io.Copy to do this:
h := sha256.New()
data := &bytes.Buffer{}
data.Write([]byte("hi there"))
data.Write([]byte("folks"))
io.Copy(h, data)
fmt.Printf("%x", h.Sum(nil))
io.Copy uses a bufer of 32KiB internally so using it requires around 32KiB of memory max.
I am trying to write a program in go which has two parts. One part is the client who tries to upload multiple pictures to the other part the server.
The server side should do the following:
Get the number of files which will be send
Loop for every file
Get filename
Get the file and save it
Go to 3
So far the server side is doing the following:
func getFileFromClient(connection net.Conn) {
var numberOfPics int
var err error
var receivedBytes int64
var fileName string
r := bufio.NewReader(connection)
strNumberOfPics, err := r.ReadString('\n')
if err != nil {
fmt.Printf("Error reading: %s\n", err)
return
}
fmt.Printf("Read: %s\n", strNumberOfPics)
strNumberOfPics = strings.Trim(strNumberOfPics, "\n")
numberOfPics, err = strconv.Atoi(strNumberOfPics)
if err != nil {
fmt.Printf("Error Atoi: %s\n", err)
panic("Atoi")
}
fmt.Printf("Receiving %d pics:\n", numberOfPics)
for i := 0; i < numberOfPics; i++ {
// Getting the file name:
fileName, err = r.ReadString('\n')
if err != nil {
fmt.Printf("Error receiving: %s\n", err)
}
fmt.Printf("Filename: %s\n", fileName)
fileName = strings.Trim(fileName, "\n")
f, err := os.Create(fileName)
defer f.Close()
if err != nil {
fmt.Println("Error creating file")
}
receivedBytes, err = io.Copy(f, connection)
if err != nil {
panic("Transmission error")
}
fmt.Printf("Transmission finished. Received: %d \n", receivedBytes)
}
}
io.Copy is working for just one file and nothing additional (because it does not empty the queue I think). I do not want to reconnect every time for every file if I do not have too. But I am not sure what I actually can do about that.
Has anyone any suggestions of an existing package or method which could help? Or example code? Or am I just plain wrong and it is a bad idea to even try this with go?
I think it might be enough if the server is able to flush the connection buffer after every read so no additional info is read and/or copied.
Really looking forward for help, thanks in advance
EDIT: Updated Code still not working. I think it might be the bufio.reader
func getFileFromClient(connection net.Conn) {
var numberOfPics int
var err error
var receivedBytes int64
var fileName string
r := bufio.NewReader(connection)
strNumberOfPics, err := r.ReadString('\n')
if err != nil {
fmt.Printf("Error reading: %s\n", err)
return
}
strNumberOfPics = strings.Trim(strNumberOfPics, "\n")
numberOfPics, err = strconv.Atoi(strNumberOfPics)
if err != nil {
fmt.Printf("Error Atoi: %s\n", err)
panic("Atoi")
}
fmt.Printf("Receiving %d pics:\n", numberOfPics)
for i := 0; i < numberOfPics; i++ {
// Getting the file name:
fileName, err = r.ReadString('\n')
if err != nil {
fmt.Printf("Error receiving: %s\n", err)
}
fileName = strings.Trim(fileName, "\n")
fmt.Printf("Filename: %s\n", fileName)
f, err := os.Create(fileName)
defer f.Close()
if err != nil {
fmt.Println("Error creating file")
}
// Get the file size
strFileSize, err := r.ReadString('\n')
if err != nil {
fmt.Printf("Read size error %s\n", err)
panic("Read size")
}
strFileSize = strings.Trim(strFileSize, "\n")
fileSize, err := strconv.Atoi(strFileSize)
if err != nil {
fmt.Printf("Error size Atoi: %s\n", err)
panic("size Atoi")
}
fmt.Printf("Size of pic: %d\n", fileSize)
receivedBytes, err = io.CopyN(f, connection, int64(fileSize))
if err != nil {
fmt.Printf("Transmission error: %s\n", err)
panic("Transmission error")
}
fmt.Printf("Transmission finished. Received: %d \n", receivedBytes)
}
}
EDIT 2: I did not get this solution to work. I am pretty sure it is because I used bufio. I did however get it to work by transmitting a single zip file with io.copy. Another solution which worked was to transmit a zip file by using http. If you are stuck trying something similar and need help feel free to send me a message. Thanks to all of you for your help
Keeping your implementation so far, the thing you're missing is that io.Copy() reads from source until it finds an EOF, so it will read all the remaining images in one go.
Also, the client must send, for each image, its size in bytes (you could do that after sending the name).
In the server, just read the size and then use io.CopyN() to read that exact number of bytes.
EDIT: as a matter of fact, you could also do things like you were doing and send images in parallel instead of serially, that would mean you open a new connection for each file transfer and then read all of the file withouth needing to send the amount of images or their size.
In case you want an alternative, a good option would be using good 'ol HTTP and multipart requests. There's the built-in module mime/multipart that allows you to do file transfers over HTTP. Of course, that would mean you'd have to rewrite your program.
My suggestion is to zip all the images you want to transfer and then send them as a single multipart POST request. In that way you have a standard way of knowing all your Acceptance criteria.
You can easily zip multiple files using https://golang.org/pkg/archive/zip/