Dynamically insert multiple documents using mgo golang mongodb - database

How can I insert an array of documents into MongoDB with mgo library using only a single DB call as in db.collection.insert()?
I have the following Transaction structure:
type Transaction struct {
Brand string `json:"brand"`
Name string `json:"name"`
Plu string `json:"plu"`
Price string `json:"price"`
}
From a POST request I will recieve an array of these structures. I want to insert them into MongoDB as individual documents but using a single DB call as explained in db.collection.insert()
I tried using c.Insert of mgo
The following is the code snippet:
func insertTransaction(c *gin.Context) {
var transactions []Transaction
err := c.BindJSON(&transactions)
if err != nil {
c.AbortWithStatusJSON(http.StatusBadRequest, map[string]string{"error":"invalid JSON"})
return
}
err = InsertTransactons(transactions)
if err != nil {
c.AbortWithStatusJSON(http.StatusInternalServerError, &map[string](interface{}){
"status": "error",
"code": "500",
"message": "Internal server error",
"error": err,
})
return
}
c.JSON(http.StatusCreated, &map[string](interface{}){
"status": "success",
"code": "0",
"message": "created",
})
}
func InsertTransactons(u []Transaction) error {
s := GetSession()
defer s.Close()
c := s.DB(DB).C(TransactionColl)
err := c.Insert(u...)
if err != nil {
return err
}
return nil
}
But as I compile and run the code, I get the following error:
go/database.go:34:17: cannot use u (type *[]Transaction) as type
[]interface {} in argument to c.Insert

You cannot pass []*Transaction as []interface{}. You need to convert each Transaction to inferface{} to change its memory layout.
var ui []interface{}
for _, t := range u{
ui = append(ui, t)
}
Pass ui to c.Insert instead

Create slice of interface for document structs by appending and then inserting data using Bulk insert which takes variable arguments.
type Bulk struct {
// contains filtered or unexported fields
}
func (b *Bulk) Insert(docs ...interface{})
For inserting documents in Bulk
const INSERT_COUNT int = 10000
type User struct {
Id bson.ObjectId `bson:"_id,omitempty" json:"_id"`
Email string `bson:"email" json:"email"`
}
func (self *User) Init() {
self.Id = bson.NewObjectId()
}
Call Bulk() function on collection returned from db connection. Bulk() function returns pointer to *Bulk.
bulk := dbs.Clone().DB("").C("users").Bulk()
bulk.Insert(users...)
Assign it to variable which will be used to call Insert() method using Bulk pointer receiver.
func main(){
// Database
dbs, err := mgo.Dial("mongodb://localhost/")
if err != nil {
panic(err)
}
// Collections
uc := dbs.Clone().DB("").C("users")
defer dbs.Clone().DB("").Session.Close()
for n := 0; n < b.N; n++ {
count := INSERT_COUNT
users := make([]interface{}, count)
for i := 0; i < count; i++ {
loop_user := User{}
loop_user.Init()
loop_user.Email = fmt.Sprintf("report-%d#example.com", i)
users[i] = loop_user
}
bulk := uc.Bulk()
bulk.Unordered()
bulk.Insert(users...)
_, bulkErr := bulk.Run()
if bulkErr != nil {
panic(err)
}
}
}

Related

MongoDB aggregation query not matching the structure that being defined

I have a requirement in my project where I have to perform a DB operation for getting a particular type of a total number of users. What I am doing is that, filtered all the queries in a slice and passing that Silce to my DB function.
This is the code snippet from where I am calling the DB function
{
filters = []bson.D{
{{Key: "Mykey", Value: myvalue}},
{{Key: "Mykey", Value: myvalue}},
{{Key: "Mykey", Value: myvalue}},
{{Key: "Mykey", Value: myvalue}},
counts, err := dbmain.NoOfDocumentsInfo(MyDBName, myCollectionName, filters...)
}
Below is my called function
func NoOfDocumentsInfo(DB string, col string, filters ...bson.D) ([]int64, error) {
if nil == dbInstance {
if nil == GetDBInstance() {
logger.Error("Not connecting to DB")
err := errors.New("DB connection error")
return nil, err
}
}
logger.Debugf("%s %s", DB, col)
coll := dbInstance.Database(DB).Collection(col)
counts := make([]int64, len(filters))
for i, filter := range filters {
count, err := coll.CountDocuments(context.TODO(), filter)
if err != nil {
logger.Fatal(err)
return nil, err
}
counts[i] = count
}
return counts, nil
}
As you can see I am calling the "coll.CountDocuments" functions multiple times. What I want is to write the code without calling the "coll.CountDocuments" function multiple times by aggregating all the filters into a single query.
I have tried to use the aggregation pipeline but my "cur" and "result" is giving null output. If you run the code you will be able to see it.
func NoOfDocumentsInfo(DB string, col string, filters ...bson.D) ([]int64, error) {
if dbInstance == nil {
if GetDBInstance() == nil {
logger.Error("Not connecting to DB")
err := errors.New("DB connection error")
return nil, err
}
}
logger.Debugf("%s %s", DB, col)
coll := dbInstance.Database(DB).Collection(col)
pipeline := make([]bson.M, 0, len(filters)+2)
pipeline = append(pipeline, bson.M{"$match": bson.M{"$or": filters}})
pipeline = append(pipeline, bson.M{"$group": bson.M{"_id": nil, "count": bson.M{"$sum": 1}}})
pipeline = append(pipeline, bson.M{"$group": bson.M{"_id": nil, "count": bson.M{"$first": "$count"}}})
var result struct {
Count int64 `bson:"count"`
}
cur, err := coll.Aggregate(context.TODO(), pipeline)
if err != nil {
logger.Fatal(err)
return nil, err
}
logger.Debugf("cur: %+v", cur)
err = cur.Decode(&result)
logger.Debugf("result: %+v, err: %v", result, err)
if err != nil {
logger.Fatal(err)
return nil, err
}
return []int64{result.Count}, nil
}
You have to add a field for each filter in $group, you may use $cond to conditionally increment the given counter. But this may very well end up not using indices, and thus being even slower than the separate, original count queries. Also note that using $or may also result in skipping indices. Also note that in $cond you may have to transform filters (e.g. add $ to field names).
You'd better launch concurrent count queries (using go) for each filter, and if they are indexed, they will complete fast. This is how it could look like:
func docCounts(db string, col string, filters ...bson.D) ([]int64, error) {
// ... obtain collection
coll := dbInstance.Database(db).Collection(col)
counts := make([]int64, len(filters))
errs := make([]error, len(filters))
wg := &sync.WaitGroup{}
wg.Add(len(filters))
for i := range filters {
go func(i int) {
defer wg.Done()
counts[i], errs[i] = coll.CountDocuments(context.TODO(), filters[i])
}(i)
}
wg.Wait()
// Produce some kind of error if any of the queries failed.
var err error
for _, e := range errs {
if e != nil {
err = fmt.Errorf("at least one query failed: %w", e)
break
}
}
// Note: starting with Go 1.20, you could simply write:
// err = errors.Join(errs)
return counts, err
}

Alternative to 'coll.CountDocuments' function on Mongodb in golang. (Aggregation Pipeline)

As you can see I am calling the "coll.CountDocuments" functions multiples times. What I want is to write the code without calling the "coll.CountDocuments" function multiple times by aggregating all the filters into a single query.
func NoOfDocumentsInfo(DB string, col string, filters ...bson.D) ([]int64, error) {
if nil == dbInstance {
if nil == GetDBInstance() {
logger.Error("Not connecting to DB")
err := errors.New("DB connection error")
return nil, err
}
}
logger.Debugf("%s %s", DB, col)
coll := dbInstance.Database(DB).Collection(col)
counts := make([]int64, len(filters))
for i, filter := range filters {
count, err := coll.CountDocuments(context.TODO(), filter)
if err != nil {
logger.Fatal(err)
return nil, err
}
counts[i] = count
}
return counts, nil
}
I have tried to used aggragation pipeline but "cur" and "result" is giving null output.
`func NoOfDocumentsInfo(DB string, col string, filters ...bson.D) ([]int64, error) {
if dbInstance == nil {
if GetDBInstance() == nil {
logger.Error("Not connecting to DB")
err := errors.New("DB connection error")
return nil, err
}
}
logger.Debugf("%s %s", DB, col)
coll := dbInstance.Database(DB).Collection(col)
pipeline := make([]bson.M, 0, len(filters)+2)
pipeline = append(pipeline, bson.M{"$match": bson.M{"$or": filters}})
pipeline = append(pipeline, bson.M{"$group": bson.M{"_id": nil, "count": bson.M{"$sum": 1}}})
pipeline = append(pipeline, bson.M{"$group": bson.M{"_id": nil, "count": bson.M{"$first": "$count"}}})
var result struct {
Count int64 `bson:"count"`
}
cur, err := coll.Aggregate(context.TODO(), pipeline)
if err != nil {
logger.Fatal(err)
return nil, err
}
logger.Debugf("cur: %+v", cur)
err = cur.Decode(&result)
logger.Debugf("result: %+v, err: %v", result, err)
if err != nil {
logger.Fatal(err)
return nil, err
}
return []int64{result.Count}, nil
}`
A much simpler approach would be the one that I'm going to share here. Let's start with the code:
package main
import (
"context"
"fmt"
"time"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
var (
dbInstance *mongo.Client
ctx context.Context
cancel context.CancelFunc
)
func NoOfDocumentsInfo(client *mongo.Client, DB string, col string, filters bson.A) (int64, error) {
coll := client.Database(DB).Collection(col)
myFilters := bson.D{
bson.E{
Key: "$and",
Value: filters,
},
}
counts, err := coll.CountDocuments(ctx, myFilters)
if err != nil {
panic(err)
}
return counts, nil
}
func main() {
ctx, cancel = context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
// set MongoDB connection
clientOptions := options.Client().ApplyURI("mongodb://root:root#localhost:27017")
mongoClient, err := mongo.Connect(ctx, clientOptions)
if err != nil {
panic(err)
}
defer mongoClient.Disconnect(ctx)
// query with filters
numDocs, err := NoOfDocumentsInfo(mongoClient, "demodb", "myCollection", bson.A{
bson.D{bson.E{Key: "Name", Value: bson.D{bson.E{Key: "$eq", Value: "John Doe"}}}},
bson.D{bson.E{Key: "Song", Value: bson.D{bson.E{Key: "$eq", Value: "White Roses"}}}},
})
if err != nil {
panic(err)
}
fmt.Println("num docs:", numDocs)
}
Let's see the relevant changes applied to the code:
Expect a parameter called filters of type bson.A which is the type for the array in the MongoDB environment.
Build the myFilters variable which is of type bson.D (slice) with the following single item (bson.E) in this way:
The Key is the logical operator
The Value is the array passed into the function
Build the array to pass to the function with all of the needed filters (e.g. two equal conditions: one on the Name key and the other on the Song).
Finally, I also did some improvements on how you've opened the MongoDB connection and how you've released the allocated resources.
Let me know if this solves your issue, thanks!

how do i enforce the proper data types on gofiber golang from POST data from frontend?

I am receiving the following POST request json data from react frontend
{
"field_one": "first",
"field_two": "second",
"field_three": "3.00"
}
but i want golang to convert it to this before processing the request
{
"field_one": "first",
"field_two": "second",
"field_three": 3.00
}
I want to convert the field_three from string to float64, but i am unable to have golang accept the string and process the proper data type
here is my golang function processing the POST request data
func PostCreate(c *fiber.Ctx) error {
type PostCreateData struct {
fieldOne string `json:"field_one" form:"field_one" validate:"required"`
fieldTwo string `json:"field_two" form:"field_two" validate:"required"`
fieldThree float64 `json:"field_three" form:"field_three" validate:"required"`
}
data := PostCreateCreateData{}
if err := c.BodyParser(&data); err != nil {
return err
}
validate := validator.New()
if err := validate.Struct(data); err != nil {
return err
}
postCreate := models.PostCreate{
fieldOne: data.fieldOne,
fieldTwo: data.fieldTwo,
fieldThree: float64(data.fieldThree),
}
database.DB.Create(&postCreate)
return c.JSON(postCreate)
}
Currently the request is not getting processed because the wrong data type for field_three which is supposed to be float64 but frontend is sending everything as string
What steps am i missing here?
fixed by updating the function to this
func PostCreate(c *fiber.Ctx) error {
type PostCreateData struct {
fieldOne string `json:"field_one" form:"field_one" validate:"required"`
fieldTwo string `json:"field_two" form:"field_two" validate:"required"`
fieldThree float64 `json:"field_three,string" form:"field_three" validate:"required"`
}
data := PostCreateCreateData{}
if err := c.BodyParser(&data); err != nil {
return err
}
validate := validator.New()
if err := validate.Struct(data); err != nil {
return err
}
postCreate := models.PostCreate{
fieldOne: data.fieldOne,
fieldTwo: data.fieldTwo,
fieldThree: float64(data.fieldThree),
}
database.DB.Create(&postCreate)
return c.JSON(postCreate)
}
so updated the struct to expect string in the body
fieldThree float64 `json:"field_three,string" form:"field_three" validate:"required"`
and works like a charm!!!

restfull api on model with foreign-key

TLDR; How to create REST api on model having Foreign key (or db relationship in general) in buffalo framework?
I am absolute beginner in go and I am trying to write an RESTFul service using buffalo framework following following example given on it official website. I am able to create RESTful api on models which have no database relationship. But I am stuck when I faced a situation where model has a foreign key. I am not able to find any documentation or reference on web. I concept on Go is also weak, you can also educate me on those thins.
Models: (ref: https://gobuffalo.io/en/docs/db/relations#example
type Composer struct {
ID uuid.UUID `json:"id" db:"id"`
Name string `json:"name" db:"name"`
Description string `json:"description" db:"description"`
CreatedAt time.Time `json:"created_at" db:"created_at"`
UpdatedAt time.Time `json:"updated_at" db:"updated_at"`
}
type Track struct {
ID uuid.UUID `json:"id" db:"id"`
Title string `json:"title" db:"title"`
Description string `json:"description" db:"description"`
Composer Composer `has_one:"composer" fk_id:"id"`
CreatedAt time.Time `json:"created_at" db:"created_at"`
UpdatedAt time.Time `json:"updated_at" db:"updated_at"`
}
Resources: (ref: https://gobuffalo.io/en/docs/resources)
type TrackResource struct {
buffalo.Resource
}
func (v TrackResource) List(c buffalo.Context) error {
tx, ok := c.Value("tx").(*pop.Connection)
if !ok {
return errors.WithStack(errors.New("no transaction found"))
}
pieces := &models.Tracks{}
q := tx.PaginateFromParams(c.Params())
if err := q.All(pieces); err != nil {
return errors.WithStack(err)
}
c.Set("pagination", q.Paginator)
return c.Render(200, r.JSON(pieces))
}
func (v TrackResource) Show(c buffalo.Context) error {
tx, ok := c.Value("tx").(*pop.Connection)
if !ok {
return errors.WithStack(errors.New("no transaction found"))
}
piece := &models.Track{}
if err := tx.Find(piece, c.Param("track_id")); err != nil {
return c.Render(404, r.JSON(err))
}
return c.Render(200, r.JSON(piece))
}
func (v TrackResource) Create(c buffalo.Context) error {
piece := &models.Track{}
if err := c.Bind(piece); err != nil {
return errors.WithStack(err)
}
tx, ok := c.Value("tx").(*pop.Connection)
if !ok {
return errors.WithStack(errors.New("no transaction found"))
}
verrs, err := piece.Create(tx)
if err != nil {
return errors.WithStack(err)
}
if verrs.HasAny() {
return c.Render(422, r.JSON(verrs))
}
return c.Render(201, r.Auto(c, piece))
}
func (v TrackResource) Update(c buffalo.Context) error {
tx, ok := c.Value("tx").(*pop.Connection)
if !ok {
return errors.WithStack(errors.New("no transaction found"))
}
piece := &models.Track{}
if err := tx.Find(piece, c.Param("track_id")); err != nil {
return c.Error(404, err)
}
if err := c.Bind(piece); err != nil {
return errors.WithStack(err)
}
verrs, err := piece.Update(tx)
if err != nil {
return errors.WithStack(err)
}
if verrs.HasAny() {
return c.Render(422, r.JSON(verrs))
}
return c.Render(200, r.JSON(piece))
}
func (v TrackResource) Destroy(c buffalo.Context) error {
tx, ok := c.Value("tx").(*pop.Connection)
if !ok {
return errors.WithStack(errors.New("no transaction found"))
}
piece := &models.Track{}
if err := tx.Find(piece, c.Param("track_id")); err != nil {
return c.Error(404, err)
}
if err := tx.Destroy(piece); err != nil {
return errors.WithStack(err)
}
return c.Render(200, r.JSON(piece))
}
When I am trying to create a track. I am getting error:
json: cannot unmarshal string into Go struct field Track.Composer of type models.Composer
gitlab.com/****/****/actions.TrackResource.Create
Please help.
You're trying to bind a Track containing a Composer with its ID, but Composer is defined as a struct.
To make it work, you need to implement the Unmarshaler interface and define how to convert this ID into the Composer struct you want.
You should add a ComposerID field to your Track struct. Then you can use Eager or Load, and pop will load the associated Composer object for you. This is shown on the page you've linked to for your Models reference.

Get Object by specific field

I'm actually learning go, following some tutorial as this one to build an Resftul API app.
First time Using Go and mongoDB, I don't understand well, how to get specific key in certain document of my collection.
Actually I have this object model :
type Numobject struct {
ID bson.ObjectId `bson:"_id" json:"id"`
Text string `bson:"text" json:"text"`
Number int `bson:"number" json:"number"`
Found bool `bson:"found" json:"found"`
Type string `bson:"type" json:"type"`
}
And I can have a specific object by ID with this function :
// Find Object by ID
func (m *NumObjectDAO) FindByNumber(id string) (Numobject, error) {
var numObject Numobject
err := db.C(COLLECTION).FindId(bson.ObjectIdHex(id)).One(&numObject)
return numObject, err
}
I call my method in main.go as follow
package main
import (
"encoding/json"
"log"
"net/http"
"github.com/gorilla/mux"
"gopkg.in/mgo.v2/bson"
. "github.com/sandaleRaclette/03-coddingChallenge/config"
. "github.com/sandaleRaclette/03-coddingChallenge/dao"
. "github.com/sandaleRaclette/03-coddingChallenge/models"
)
// Save the configuration of mongodatabase (localhost and which db use) in Config array
var config = Config{}
var dao = NumObjectDAO{}
// GET list of all objects
func AllObjectsEndPoint(w http.ResponseWriter, r *http.Request) {
movies, err := dao.FindAll()
if err != nil {
respondWithError(w, http.StatusInternalServerError, err.Error())
return
}
respondWithJson(w, http.StatusOK, movies)
}
// GET an Object by its ID
func FindObjectEndpoint(w http.ResponseWriter, r *http.Request) {
params := mux.Vars(r)
numObject, err := dao.FindByNumber(params["number"])
if err != nil {
respondWithError(w, http.StatusBadRequest, "Invalid Object ID")
return
}
respondWithJson(w, http.StatusOK, numObject)
}
// Respond Methods
func respondWithError(w http.ResponseWriter, code int, msg string) {
respondWithJson(w, code, map[string]string{"error": msg})
}
func respondWithJson(w http.ResponseWriter, code int, payload interface{}) {
response, _ := json.Marshal(payload)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(code)
w.Write(response)
}
// Parse the configuration file 'config.toml', and establish a connection to DB
func init() {
config.Read()
dao.Server = config.Server
dao.Database = config.Database
dao.Connect()
}
// Define HTTP request routes and define the differents endpoints
func main() {
r := mux.NewRouter()
r.HandleFunc("/api/v1/trivia", AllObjectsEndPoint).Methods("GET")
r.HandleFunc("/api/v1/trivia/{number}", FindObjectEndpoint).Methods("GET")
if err := http.ListenAndServe(":3000", r); err != nil {
log.Fatal(err)
}
}
There is other methods than getting an object by ID? How can I get an object with a specific key as Number or Type following my model ?
I want to obtain something like this when I GET "/api/v1/trivia/45000000" :
{
"id": "5aa554c89d63b0d3580449a5",
"text": "45000000 is the number of turkeys Americans eat at Thanksgiving annually.",
"number": 45000000,
"found": true,
"type": "trivia"
}
I'm actually looking here for some answers but I have some difficulties with query... If someone can give me beginner explanation...
Use Find for general queries:
err := db.c(COLLECTION).Find(bson.D{{"Number", 10}}).One(&numobjecct)

Resources