How to reuse MongoDB connection in Go - database

I would like to connect my server that was written in Go with a MongoDB but I'm not sure how to do it in an efficient way. A couple of examples I found implemented it like shown below.
libs/mongodb/client.go
package mongodb
import (
"context"
"log"
"project/keys"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
func GetClient() *mongo.Database {
client, err := mongo.Connect(
context.Background(),
options.Client().ApplyURI(keys.GetKeys().MONGO_URI),
)
if err != nil {
log.Fatal(err)
}
return client.Database(keys.GetKeys().MONGO_DB_NAME)
}
services/user/findOne.go
package userservices
import (
"context"
"log"
"project/libs/mongodb"
"project/models"
"go.mongodb.org/mongo-driver/bson"
)
func FindOne(filter bson.M) (models.User, error) {
var user models.User
collection := mongodb.GetClient().Collection("users")
result := collection.FindOne(context.TODO(), filter)
if result.Err() != nil {
return user, result.Err()
}
if err := result.Decode(&user); err != nil {
log.Println("Failed to decode user with error:", err)
return user, err
}
return user, nil
}
The GetClient function returns a database instance that is then used throughout the app. This seems to work, but I'm wondering if this really is best practice as it seems to create a new connection every time a new client is requested as shown in the second code snippet or is that assumption incorrect? I also thought about converting GetClient to a singleton, that always returns the same database instance but how would a lost connection be handled in that case? Thank you

I do it this way. Do it once at the service start and then pass the MongoDatastore object around to orchestrator, service layers and repository layers. I am using the "github.com/mongodb/mongo-go-driver/mongo" driver for mongo. I think it internally monitors and recycles idle connections. Hence, we don't have to bother about broken connections as long as reference to the mongo.Client object is not lost.
const CONNECTED = "Successfully connected to database: %v"
type MongoDatastore struct {
db *mongo.Database
Session *mongo.Client
logger *logrus.Logger
}
func NewDatastore(config config.GeneralConfig, logger *logrus.Logger) *MongoDatastore {
var mongoDataStore *MongoDatastore
db, session := connect(config, logger)
if db != nil && session != nil {
// log statements here as well
mongoDataStore = new(MongoDatastore)
mongoDataStore.db = db
mongoDataStore.logger = logger
mongoDataStore.Session = session
return mongoDataStore
}
logger.Fatalf("Failed to connect to database: %v", config.DatabaseName)
return nil
}
func connect(generalConfig config.GeneralConfig, logger *logrus.Logger) (a *mongo.Database, b *mongo.Client) {
var connectOnce sync.Once
var db *mongo.Database
var session *mongo.Client
connectOnce.Do(func() {
db, session = connectToMongo(generalConfig, logger)
})
return db, session
}
func connectToMongo(generalConfig config.GeneralConfig, logger *logrus.Logger) (a *mongo.Database, b *mongo.Client) {
var err error
session, err := mongo.NewClient(generalConfig.DatabaseHost)
if err != nil {
logger.Fatal(err)
}
session.Connect(context.TODO())
if err != nil {
logger.Fatal(err)
}
var DB = session.Database(generalConfig.DatabaseName)
logger.Info(CONNECTED, generalConfig.DatabaseName)
return DB, session
}
You may now create your repository as below:-
type TestRepository interface{
Find(ctx context.Context, filters interface{}) []Document, error
}
type testRepository struct {
store *datastore.MongoDatastore
}
func (r *testRepository) Find(ctx context.Context , filters interface{}) []Document, error{
cur, err := r.store.GetCollection("some_collection_name").Find(ctx, filters)
if err != nil {
return nil, err
}
defer cur.Close(ctx)
var result = make([]models.Document, 0)
for cur.Next(ctx) {
var currDoc models.Document
err := cur.Decode(&currDoc)
if err != nil {
//log here
continue
}
result = append(result, currDoc)
}
return result, err
}

I solved it doing this
var CNX = Connection()
func Connection() *mongo.Client {
// Set client options
clientOptions := options.Client().ApplyURI("mongodb://localhost:27017")
// Connect to MongoDB
client, err := mongo.Connect(context.TODO(), clientOptions)
if err != nil {
log.Fatal(err)
}
// Check the connection
err = client.Ping(context.TODO(), nil)
if err != nil {
log.Fatal(err)
}
fmt.Println("Connected to MongoDB!")
return client
}
//calll connection
func main() {
collection := db.CNX.Database("tasks").Collection("task")
}
output "Connected to MongoDB!"

Related

Using transaction and simple DB connection in the same struct because of this pattern what can I do?

I found a great example of transactions between Repositories using a Clean Architecture approach.
This guy is using Gorm.
Gorm has the same type for a DB connection and a transaction, example:
var db *gorm.DB
var tx *gorm.DB
I'm a fan of go-pg. But here the types are different (maybe it's even better as is), example:
var db *pg.DB
var tx *pg.Tx
And of course the error is: Cannot use 'tx' (type *Tx) as type *pg.DB
A small reproduction:
package main
import (
"github.com/go-pg/pg/v10"
)
type Player struct {
ID int
Name string
}
type PlayerRepo struct {
db *pg.DB
teamRepo *TeamRepo
}
type TeamRepo struct {
db *pg.DB
}
func NewPlayerRepo(db *pg.DB) *PlayerRepo {
return &PlayerRepo{
db: db,
teamRepo: NewTeamRepo(db),
}
}
func NewTeamRepo(db *pg.DB) *TeamRepo {
return &TeamRepo{db: db}
}
func (r *PlayerRepo) Find(id int) (*Player, error) {
var player Player
err := r.db.Model(&player).Where("id = ?", id).Select()
if err != nil {
return nil, err
}
return &player, nil
}
func (r *PlayerRepo) All() ([]*Player, error) {
// Long code
return nil, nil
}
func (r *PlayerRepo) Insert() (*Player, error) {
// Long code
return nil, nil
}
func (r *PlayerRepo) Update() (*Player, error) {
// Long code
return nil, nil
}
func (r *PlayerRepo) Delete() (*Player, error) {
// Long code
return nil, nil
}
func (r *PlayerRepo) WithTransaction(txFunc func(*PlayerRepo) error) (err error) {
tx, _ := r.db.Begin()
manager := NewPlayerRepo(tx) // <<<--- here the problem! tx is not good here, it's `pg.Tx` not `pg.DB`
err = txFunc(manager)
return
}
What can I do to fix this?
Thanks in advance. ❤️
You can define an interface that is already, implicitly implemented by both:
type DB interface {
Begin() (*Tx, error)
Close() error
Context() context.Context
CopyFrom(r io.Reader, query interface{}, params ...interface{}) (res Result, err error)
CopyTo(w io.Writer, query interface{}, params ...interface{}) (res Result, err error)
Exec(query interface{}, params ...interface{}) (Result, error)
ExecContext(c context.Context, query interface{}, params ...interface{}) (Result, error)
ExecOne(query interface{}, params ...interface{}) (Result, error)
ExecOneContext(c context.Context, query interface{}, params ...interface{}) (Result, error)
Formatter() orm.QueryFormatter
Model(model ...interface{}) *orm.Query
ModelContext(c context.Context, model ...interface{}) *orm.Query
Prepare(q string) (*Stmt, error)
Query(model interface{}, query interface{}, params ...interface{}) (Result, error)
QueryContext(c context.Context, model interface{}, query interface{}, params ...interface{}) (Result, error)
QueryOne(model interface{}, query interface{}, params ...interface{}) (Result, error)
QueryOneContext(c context.Context, model interface{}, query interface{}, params ...interface{}) (Result, error)
RunInTransaction(ctx context.Context, fn func(*Tx) error) error
}
NOTE: I only know that the method names match, I didn't bother checking if the signatures also do, if they don't, you'll need to edit the interface accordingly.
You can add a simple "compiler check":
var _ DB = (*pg.DB)(nil)
var _ DB = (*pg.Tx)(nil)
And then you can change the type of the PlayerRepo.db field from *pg.DB to your new DB interface.
type PlayerRepo struct {
db DB
teamRepo *TeamRepo
}
type TeamRepo struct {
db DB
}
func NewPlayerRepo(db DB) *PlayerRepo {
return &PlayerRepo{
db: db,
teamRepo: NewTeamRepo(db),
}
}
func NewTeamRepo(db DB) *TeamRepo {
return &TeamRepo{db: db}
}
func (r *PlayerRepo) WithTransaction(txFunc func(*PlayerRepo) error) (err error) {
tx, err := r.db.Begin()
if err != nil {
return err
}
defer func() {
// rollback if err; commit if no err
}()
manager := NewPlayerRepo(tx)
err = txFunc(manager)
return
}
If your repo types need to be able to invoke some of the methods that are not common to both pg.DB and pg.Tx, and therefore not defined by the new DB interface, then, one approach would be to retain the original types for such use, for example:
type PlayerRepo struct {
db DB
pg *pg.DB
teamRepo *TeamRepo
}
type TeamRepo struct {
db DB
pg *pg.DB
}
func NewPlayerRepo(db DB, pg *pg.DB) *PlayerRepo {
return &PlayerRepo{
db: db,
pg: pg,
teamRepo: NewTeamRepo(db, pg),
}
}
func NewTeamRepo(db DB, pg *pg.DB) *TeamRepo {
return &TeamRepo{db: db, pg: pg}
}
func (r *PlayerRepo) WithTransaction(txFunc func(*PlayerRepo) error) (err error) {
tx, err := r.db.Begin()
if err != nil {
return err
}
defer func() {
// rollback if err; commit if no err
}()
manager := NewPlayerRepo(tx, r.pg)
err = txFunc(manager)
return
}
Note that if you decide to use orm.DB, which is reasonable, but it is missing some of the methods that you need and that are already implemented by both pg.DB and pg.Tx, then you could embed orm.DB into your custom interface and add only those methods that are missing.
type DB interface {
Begin() (*Tx, error)
orm.DB
}

How to stop golang gRPC server with a function?

Looking at the helloworld example from grpc-go, how can I add a StopGrpcServer() function to stop gRPC server?
I'm trying to make a shared C library file of gRPC server which can be used to start and stop the gRPC server by invoking the functions via ffi.
In my tests I am able to start the gRPC server with StartGrpcServer() but I'm feeling lost how to implement StopGrpcServer().
package main
import C
import (
"context"
"log"
"net"
"google.golang.org/grpc"
pb "google.golang.org/grpc/examples/helloworld/helloworld"
)
const (
port = ":50051"
)
// server is used to implement helloworld.GreeterServer.
type server struct {
pb.UnimplementedGreeterServer
}
// SayHello implements helloworld.GreeterServer
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
log.Printf("Received: %v", in.GetName())
return &pb.HelloReply{Message: "Hello " + in.GetName()}, nil
}
//export StopGrpcServer
func (s *server) StopGrpcServer() {
s.Stop()
}
//export StartGrpcServer
func StartGrpcServer() {
lis, err := net.Listen("tcp", port)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
pb.RegisterGreeterServer(s, &server{})
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
func main() {}
Would really appreciate some help on it.
Thanks,
Satinder
Thanks for the hint #Marc. I am able to solve the problem by declaring *grpc.Server variable globally outside StartGrpcServer(), and then using it in StopGrpcServer().
Here's the working code:
package main
import C
import (
"context"
"log"
"net"
"google.golang.org/grpc"
pb "google.golang.org/grpc/examples/helloworld/helloworld"
)
var srv *grpc.Server
const (
port = ":50051"
)
// server is used to implement helloworld.GreeterServer.
type server struct {
pb.UnimplementedGreeterServer
}
// SayHello implements helloworld.GreeterServer
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
log.Printf("Received: %v", in.GetName())
return &pb.HelloReply{Message: "Hello " + in.GetName()}, nil
}
//export StopGrpcServer
func StopGrpcServer() {
srv.Stop()
}
//export StartGrpcServer
func StartGrpcServer() {
lis, err := net.Listen("tcp", port)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
srv = grpc.NewServer()
pb.RegisterGreeterServer(srv, &server{})
if err := srv.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
func main() {}
// Compile with command:
// go build -o greeter.so -buildmode=c-shared server.go

How I can safely check if file exists in S3 bucket using go in lambda?

I am working on a service for my project that is used to synchronize Lambdas works in AWS. The idea is to write a TrackerFile module that will store structures on S3. Each time I use the tracker, I will check if there is a file with the name assigned to the called tracker.
I have no idea but how to safely check if a file with a given name exists on S3. can you show a sample piece of code that would be able to return (bool, err) where bool is True if the file exists?
Make sure you have the following permissions:
"s3:GetObject",
"s3:ListBucket",
Regarding Safety, AWS just introduced strong consistency for S3
s3svc = s3.New(sess)
func keyExists(bucket string, key string) (bool, error) {
_, err := s3svc.HeadObject(&s3.HeadObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
})
if err != nil {
if aerr, ok := err.(awserr.Error); ok {
switch aerr.Code() {
case "NotFound": // s3.ErrCodeNoSuchKey does not work, aws is missing this error code so we hardwire a string
return false, nil
default:
return false, err
}
}
return false, err
}
return true, nil
}
If you use the AWS SDK for Go V2:
import (
...
awshttp "github.com/aws/aws-sdk-go-v2/aws/transport/http"
...
)
func uploaded(ctx context.Context, client *s3.Client, bucket string, key string) (bool, error) {
_, err := client.HeadObject(ctx, &s3.HeadObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
})
if err != nil {
var responseError *awshttp.ResponseError
if errors.As(err, &responseError) && responseError.ResponseError.HTTPStatusCode() == http.StatusNotFound {
return false, nil
}
return false, err
}
return true, nil
}
Handling Errors in the AWS SDK for Go V2
You can try this code to get object metadata:
sess, err := session.NewSession(&aws.Config{
Region: aws.String("your-region"),
})
if err != nil {
//handle error here
}
svc := s3.New(session.Must(sess, err))
output, err := svc.HeadObject(&s3.HeadObjectInput{
Bucket: aws.String("bucket_name"),
Key: aws.String("object_key"),
})
if err != nil {
//handle error here
}
fmt.Println(output.LastModified) //do something with metadata
Just refer to AWS API S3 HeadObject method: AWS API documentation.
It will return status code 404 in case of a missing object.
A HEAD request has the same options as a GET action on an object. The
response is identical to the GET response except that there is no
response body. Because of this, if the HEAD request generates an
error, it returns a generic 404 Not Found or 403 Forbidden code.
import (
"context"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/service/s3"
"log"
)
func IsObjectExistsInS3Bucket(bucketName string, objectKey string) bool{
_, err := executeHeadObjectMethodOnS3Api(bucketName, objectKey)
return ! handleErrorOnS3HeadObjectAction(bucketName, objectKey, err)
}
func executeHeadObjectMethodOnS3Api(bucketName string, objectKey string) (output *s3.HeadObjectOutput, err error){
s3Client := getS3Client()
return s3Client.HeadObject(context.TODO(), &s3.HeadObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
})
}
func handleErrorOnS3HeadObjectAction(bucketName, objectKey string, err error) (hit bool){
logUnableToHeadObjectFromS3Bucket(bucketName, objectKey, err)
return err != nil
}
func logUnableToHeadObjectFromS3Bucket(bucketName string, objectKey string, err error) {
if err != nil {
log.Printf("Unable to head object {%s} from S3 bucket {%s}: { %v }", objectKey, bucketName, err)
}
}

iterate over a large number of entities on appengine with go

On app engine I have a large number of entities of a particular kind.
I want to run a function on each entity (e.g. edit the entity or copy it)
I would do this in a taskqueue but a taskqueue is limited to 10 minutes runtime and each function call is prone to many kinds of errors. What is the best way to do this?
Here's my solution although I'm hoping someone out there has a better solution. I also wonder if this is prone to fork bombs e.g. if the task runs twice, it will set off two chains of iteration.. ! I'm only using it to iterate a few hundred thousand entities, although the operation on each entity is expensive.
First I create a taskqueue for running each individual function call on an entity one at a time:
queue:
- name: entity-iter
rate: 100/s
max_concurrent_requests: 1
retry_parameters:
task_retry_limit: 3
task_age_limit: 30m
min_backoff_seconds: 200
and then I have an iterate entity method which, given the kind, will call your delay func on each entity with the key.
package sysadmin
import (
"google.golang.org/appengine/datastore"
"golang.org/x/net/context"
"google.golang.org/appengine/log"
"google.golang.org/appengine/delay"
"google.golang.org/appengine/taskqueue"
)
func ForEachEntity(kind string, f *delay.Function) *delay.Function {
var callWithNextKey *delay.Function // func(c context.Context, depth int, cursorString string) error
callWithNextKey = delay.Func("something", func(c context.Context, depth int, cursorString string) error {
q := datastore.NewQuery(kind).KeysOnly()
if cursorString != "" {
if curs, err := datastore.DecodeCursor(cursorString); err != nil {
log.Errorf(c, "error decoding cursor %v", err)
return err
} else {
q = q.Start(curs)
}
}
it := q.Run(c)
if key, err := it.Next(nil); err != nil {
if err == datastore.Done {
log.Infof(c, "Done %v", err)
return nil
}
log.Errorf(c, "datastore error %v", err)
return err
} else {
curs, _ := it.Cursor()
if t, err := f.Task(key); err != nil {
return err
} else if _, err = taskqueue.Add(c, t, "entity-iter"); err != nil {
log.Errorf(c, "error %v", err)
return err
}
if depth - 1 > 0 {
if err := callWithNextKey.Call(c, depth - 1, curs.String()); err != nil {
log.Errorf(c, "error2 %v", err)
return err
}
}
}
return nil
})
return callWithNextKey
}
example usage:
var DoCopyCourse = delay.Func("something2", CopyCourse)
var DoCopyCourses = ForEachEntity("Course", DoCopyCourse)
func CopyCourses(c context.Context) {
//sharedmodels.MakeMockCourses(c)
DoCopyCourses.Call(c, 9999999, "")
}

Golang connection to angularjs

How do I connect a backend server in go lang to angularjs? I know angularjs communicates with go via $http or $resource services but what part of the go code links communicates with angular once all the data structs have been made? Would this be the encoded/marshalled json or do we create some kind of route...
I am a newbie looking to start this project after studying angular and go but this is the part i don't understand - what is the end point from go that angulars $resource or $http service links with?
Create a http endpoint that reads json string, unmarshalls it, does some logic with it and writes a json string to the response. E.g:
func main() {
http.HandleFunc("/api/", apiHandler)
logInfo(fmt.Sprintf("Starting server on port %d", serverPort))
err := http.ListenAndServe(fmt.Sprintf(":%d", serverPort), nil)
if err != nil {
logError(err)
}
}
func apiHandler(w http.ResponseWriter, r *http.Request) {
//ensure its a post
method := r.Method
if method != "POST" {
fail(w, "Invalid http method")
return
}
requestData, err := ioutil.ReadAll(r.Body)
if err != nil {
fail(w, err.Error())
return
}
respData, err := doBusinessLogicWithData(requestData)
if err != nil {
fail(w, err.Error())
return
}
respJSONBytes, err := json.Marshal(respData)
if err != nil {
fail(w, err.Error())
return
}
fmt.Fprintln(w, string(respJSONBytes))
}
func fail(w http.ResponseWriter, message string) {
result := &struct {
Success bool
Message string
}{
Success: false,
Message: message,
}
resultJSONBytes, err := json.Marshal(result)
if err != nil {
logError(err)
fmt.Fprintln(w, "Unable to generate result")
return
}
fmt.Fprintln(w, string(resultJSONBytes))
}

Resources