I'm working on GRPC stream, In server side, I receive the multiple byte inside the for loop, I want to merge in a single array of byte (I tried append method but not use), Here I have attached my sample code. Any one guide me.
Example code
func (s *ServerGRPC) Upload(stream pb.GuploadService_UploadServer) (err error) {
for {
resp, err := stream.Recv()
if err != nil {
if err == io.EOF {
goto END
}
err = errors.Wrapf(err,
"failed unexpectadely while reading chunks from stream")
return err
}
for _, result := range resp.Content {
fmt.Println("result ====>>>", result)
//Actual Output
//result ====>>> 136
//result ====>>> 84
//result ====>>> 232
//result ====>>> 12
//Expectation
//result ===> [136 84 232 12]
}
}
s.logger.Info().Msg("upload received")
END:
err = stream.SendAndClose(&pb.UploadStatus{
Message: "Upload received with success",
Code: pb.UploadStatusCode_Ok,
})
if err != nil {
err = errors.Wrapf(err,
"failed to send status code")
return
}
return
}
You are appending to Sample but printing req.Content
There is nothing wrong with merging slices. if just have to print Sample to see merged result.
func (s *ServerGRPC) Upload(stream pb.GuploadService_UploadServer) (err error) {
var respBytes []byte
for {
resp, err := stream.Recv()
if err != nil {
if err == io.EOF {
goto END // you can use break here
}
err = errors.Wrapf(err,
"failed unexpectadely while reading chunks from stream")
return err
}
for _, result := range resp.Content {
fmt.Println("result ====>>>", result)
respBytes = append(respBytes, result)
}
}
s.logger.Info().Msg("upload received")
// print respBytes here
END:
err = stream.SendAndClose(&pb.UploadStatus{
Message: "Upload received with success",
Code: pb.UploadStatusCode_Ok,
})
if err != nil {
err = errors.Wrapf(err,
"failed to send status code")
return
}
return
}
Related
As you can see I am calling the "coll.CountDocuments" functions multiples times. What I want is to write the code without calling the "coll.CountDocuments" function multiple times by aggregating all the filters into a single query.
func NoOfDocumentsInfo(DB string, col string, filters ...bson.D) ([]int64, error) {
if nil == dbInstance {
if nil == GetDBInstance() {
logger.Error("Not connecting to DB")
err := errors.New("DB connection error")
return nil, err
}
}
logger.Debugf("%s %s", DB, col)
coll := dbInstance.Database(DB).Collection(col)
counts := make([]int64, len(filters))
for i, filter := range filters {
count, err := coll.CountDocuments(context.TODO(), filter)
if err != nil {
logger.Fatal(err)
return nil, err
}
counts[i] = count
}
return counts, nil
}
I have tried to used aggragation pipeline but "cur" and "result" is giving null output.
`func NoOfDocumentsInfo(DB string, col string, filters ...bson.D) ([]int64, error) {
if dbInstance == nil {
if GetDBInstance() == nil {
logger.Error("Not connecting to DB")
err := errors.New("DB connection error")
return nil, err
}
}
logger.Debugf("%s %s", DB, col)
coll := dbInstance.Database(DB).Collection(col)
pipeline := make([]bson.M, 0, len(filters)+2)
pipeline = append(pipeline, bson.M{"$match": bson.M{"$or": filters}})
pipeline = append(pipeline, bson.M{"$group": bson.M{"_id": nil, "count": bson.M{"$sum": 1}}})
pipeline = append(pipeline, bson.M{"$group": bson.M{"_id": nil, "count": bson.M{"$first": "$count"}}})
var result struct {
Count int64 `bson:"count"`
}
cur, err := coll.Aggregate(context.TODO(), pipeline)
if err != nil {
logger.Fatal(err)
return nil, err
}
logger.Debugf("cur: %+v", cur)
err = cur.Decode(&result)
logger.Debugf("result: %+v, err: %v", result, err)
if err != nil {
logger.Fatal(err)
return nil, err
}
return []int64{result.Count}, nil
}`
A much simpler approach would be the one that I'm going to share here. Let's start with the code:
package main
import (
"context"
"fmt"
"time"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
var (
dbInstance *mongo.Client
ctx context.Context
cancel context.CancelFunc
)
func NoOfDocumentsInfo(client *mongo.Client, DB string, col string, filters bson.A) (int64, error) {
coll := client.Database(DB).Collection(col)
myFilters := bson.D{
bson.E{
Key: "$and",
Value: filters,
},
}
counts, err := coll.CountDocuments(ctx, myFilters)
if err != nil {
panic(err)
}
return counts, nil
}
func main() {
ctx, cancel = context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
// set MongoDB connection
clientOptions := options.Client().ApplyURI("mongodb://root:root#localhost:27017")
mongoClient, err := mongo.Connect(ctx, clientOptions)
if err != nil {
panic(err)
}
defer mongoClient.Disconnect(ctx)
// query with filters
numDocs, err := NoOfDocumentsInfo(mongoClient, "demodb", "myCollection", bson.A{
bson.D{bson.E{Key: "Name", Value: bson.D{bson.E{Key: "$eq", Value: "John Doe"}}}},
bson.D{bson.E{Key: "Song", Value: bson.D{bson.E{Key: "$eq", Value: "White Roses"}}}},
})
if err != nil {
panic(err)
}
fmt.Println("num docs:", numDocs)
}
Let's see the relevant changes applied to the code:
Expect a parameter called filters of type bson.A which is the type for the array in the MongoDB environment.
Build the myFilters variable which is of type bson.D (slice) with the following single item (bson.E) in this way:
The Key is the logical operator
The Value is the array passed into the function
Build the array to pass to the function with all of the needed filters (e.g. two equal conditions: one on the Name key and the other on the Song).
Finally, I also did some improvements on how you've opened the MongoDB connection and how you've released the allocated resources.
Let me know if this solves your issue, thanks!
So again im trying to get this data but it is returning an error of
data.Body undefined (type []byte has no field or method Body)
on line 16 and 23 of this code. so when its decoding the json
If anyone could help me,
here is my code
func SkyblockActiveAuctions() (structs.SkyblockActiveAuctions, error) {
var auctions structs.SkyblockActiveAuctions
startTime := time.Now()
statusCode, data, err := fasthttp.Get(nil, "https://api.hypixel.net/skyblock/auctions")
if err != nil {
return auctions, err
}
fmt.Println(statusCode)
var totalPages = auctions.TotalAuctions
for i := 0; i < totalPages; i++ {
statusCode, data1, err := fasthttp.Get(nil, "https://api.hypixel.net/skyblock/auctions")
if err != nil {
return auctions, err
}
fmt.Println(statusCode)
json.NewDecoder(data1.Body).Decode(&auctions)
fmt.Println(auctions.LastUpdated)
}
endTime := time.Now()
var timeTook = endTime.Sub(startTime).Milliseconds()
fmt.Println(data)
json.NewDecoder(data.Body).Decode(&auctions)
fmt.Println(auctions.LastUpdated)
fmt.Println(timeTook)
return auctions, err
}
json.NewDecoder(data.Body).Decode(&auctions)
data.Body undefined (type []byte has no field or method Body)
data is already the body of the response.
json.NewDecoder expects an io.Reader but since fasthttp has already read the data into []byte, it would be more appropriate to use json.Unmarshal:
err := json.Unmarshal(data, &auctions)
if err != nil {
return nil, err
}
Don't forget to handle errors from json.Unmarshal (or, from json.Decoder.Decode for that matter). acutions won't hold the expected data if the Json failed to parse, so you should handle that possiblity.
Am trying to encode the videos with Golang ffmpeg. Am not able to get the video file, it shows this error
invalid character '-' in numeric literal
Postman:
This is my code:
package main
import (
"encoding/json"
"fmt"
"net/http"
"github.com/gorilla/mux"
"github.com/xfrr/goffmpeg/transcoder"
)
type Encode struct {
Video string `json:"video"`
}
func encodeFfmpeg(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
fmt.Println("Encodeing Started")
var encode Encode
video := json.NewDecoder(r.Body).Decode(encode)
file, err := os.Open(video)
if err != nil {
log.Fatalln(err)
}
defer file.Close()
buf, err := ioutil.ReadAll(file)
if err != nil {
log.Fatalln(err)
}
cmd := exec.Command("ffmpeg",
"-i", "pipe:0", // take stdin as input
"-c:a", "libmp3lame", // use mp3 lame codec
"-f", "avi",
"pipe:1",
)
resultBuffer := bytes.NewBuffer(make([]byte, 5*1024*1024)) // pre allocate 5MiB buffer
cmd.Stderr = os.Stderr // bind log stream to stderr
cmd.Stdout = resultBuffer // stdout result will be written here
stdin, err := cmd.StdinPipe() // Open stdin pipe
if err != nil {
log.Fatalln(err)
}
err = cmd.Start() // Start a process on another goroutine
if err != nil {
log.Fatalln(err)
}
_, err = stdin.Write(buf) // pump audio data to stdin pipe
if err != nil {
log.Fatalln(err)
}
err = stdin.Close() // close the stdin, or ffmpeg will wait forever
if err != nil {
log.Fatalln(err)
}
err = cmd.Wait() // wait until ffmpeg finish
if err != nil {
log.Fatalln(err)
}
outputFile, err := os.Create(encodeFile) // create new file
if err != nil {
log.Fatalln(err)
}
defer outputFile.Close()
_, err = outputFile.Write(resultBuffer.Bytes()) // write result buffer to file
if err != nil {
log.Fatalln(err)
}
json.NewEncoder(w).Encode(map[string]interface{}{
"status": "success",
"statusCode": 200,
"data": "Successfully Encoded file",
})
}
func main() {
router := mux.NewRouter()
router.HandleFunc("/encode", encodeFfmpeg).Methods("POST")
//router.HandleFunc("/rtsp", rtsp).Methods("POST")
// config port
fmt.Printf("Starting server at 8080 \n")
http.ListenAndServe(":8080", router)
}
video.mp4 file not showing in request body. Please help me to solve this issue
json.NewDecoder(r.Body).Decode(encode) returns error not video and another thing is that you send your data in form so you can access file from form like:
file, header, err := r.FormFile("video")
On app engine I have a large number of entities of a particular kind.
I want to run a function on each entity (e.g. edit the entity or copy it)
I would do this in a taskqueue but a taskqueue is limited to 10 minutes runtime and each function call is prone to many kinds of errors. What is the best way to do this?
Here's my solution although I'm hoping someone out there has a better solution. I also wonder if this is prone to fork bombs e.g. if the task runs twice, it will set off two chains of iteration.. ! I'm only using it to iterate a few hundred thousand entities, although the operation on each entity is expensive.
First I create a taskqueue for running each individual function call on an entity one at a time:
queue:
- name: entity-iter
rate: 100/s
max_concurrent_requests: 1
retry_parameters:
task_retry_limit: 3
task_age_limit: 30m
min_backoff_seconds: 200
and then I have an iterate entity method which, given the kind, will call your delay func on each entity with the key.
package sysadmin
import (
"google.golang.org/appengine/datastore"
"golang.org/x/net/context"
"google.golang.org/appengine/log"
"google.golang.org/appengine/delay"
"google.golang.org/appengine/taskqueue"
)
func ForEachEntity(kind string, f *delay.Function) *delay.Function {
var callWithNextKey *delay.Function // func(c context.Context, depth int, cursorString string) error
callWithNextKey = delay.Func("something", func(c context.Context, depth int, cursorString string) error {
q := datastore.NewQuery(kind).KeysOnly()
if cursorString != "" {
if curs, err := datastore.DecodeCursor(cursorString); err != nil {
log.Errorf(c, "error decoding cursor %v", err)
return err
} else {
q = q.Start(curs)
}
}
it := q.Run(c)
if key, err := it.Next(nil); err != nil {
if err == datastore.Done {
log.Infof(c, "Done %v", err)
return nil
}
log.Errorf(c, "datastore error %v", err)
return err
} else {
curs, _ := it.Cursor()
if t, err := f.Task(key); err != nil {
return err
} else if _, err = taskqueue.Add(c, t, "entity-iter"); err != nil {
log.Errorf(c, "error %v", err)
return err
}
if depth - 1 > 0 {
if err := callWithNextKey.Call(c, depth - 1, curs.String()); err != nil {
log.Errorf(c, "error2 %v", err)
return err
}
}
}
return nil
})
return callWithNextKey
}
example usage:
var DoCopyCourse = delay.Func("something2", CopyCourse)
var DoCopyCourses = ForEachEntity("Course", DoCopyCourse)
func CopyCourses(c context.Context) {
//sharedmodels.MakeMockCourses(c)
DoCopyCourses.Call(c, 9999999, "")
}
I'm trying to send a multipart/form with both a file and an access token,
it works fine with the dev server, but the exact same post to AppEngine deployment result in a different received token string (I can see that its length is a longer. 938 chars when its supposed to be 902).
I'm actually executing the exact same POST request:
curl -X POST --form "token=<ACCESS_TOKEN>" --form "file=#myfile.jpg" http://upload_url
the upload response handler:
c := appengine.NewContext(r)
blobs, values, err := blobstore.ParseUpload(r)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
files := blobs["file"]
if len(files) == 0 {
fmt.Fprintln(w, "No file uploaded")
return
}
token := values.Get("token")
EDIT: I tried to simply create an endpoint for posting the token and printing its length, which returns the correct length.. what am I doing wrong ?
func t(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "%d", len(r.FormValue("token")))
}
EDIT2: when I print the received token from the AppEngine deployment I get something like:
eyJhbGciOiJSUzI1NiIsImtpZCI6ImZjZmQ4NGYxZGZhN2NiODUyMTg4MDFkNDRjNzYwNDFmMzB=
lMzg2OGIifQ.eyJpc3MiOiJhY2NvdW50cy5nb29nbGUuY29tIiwiYXVkIjoiMjEwMTAyMTk5NDI=
4LmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tIiwidG9rZW5faGFzaCI6IklQMmduQjFsZGMwTE=
VEdVg5LWlZa2ciLCJhdF9oYXNoIjoiSVAyZ25CMWxkYzBMRUR1WDktaVlrZyIsImlkIjoiMTA5O=
.
.
it has line breaks... for some reason the dev server doesn't behave like that and doesn't split the lines.
how can I get the original string or stop this behavior ?
How about using multiplart.Reader?
c := appengine.NewContext(r)
if r.Method != "POST" {
http.Error(w, "invalid request", 400)
return
}
ct := r.Header.Get("Content-Type")
if strings.SplitN(ct, ";", 2)[0] != "multipart/form-data" {
http.Error(w, "invalid request", 40400)
return
}
_, params, err := mime.ParseMediaType(ct)
if err != nil {
http.Error(w, "invalid request", 400)
return
}
boundary, ok := params["boundary"]
if !ok {
http.Error(w, "invalid request", 400)
return
}
reader := multipart.NewReader(r.Body, boundary)
var data []byte
for {
part, err := reader.NextPart()
if part == nil || err != nil {
break
}
if part.FormName() != "file" {
continue
}
v := part.Header.Get("Content-Disposition")
if v == "" {
continue
}
d, _, err := mime.ParseMediaType(v)
if err != nil {
continue
}
if d != "form-data" {
continue
}
data, _ = ioutil.ReadAll(part)
// do something using data
}