I've implemented dao.go file with the following realization:
type DbClient struct {
db *gorm.DB
}
GetDBClient() initializes connection with database and returns (*DbClient, error)
func (db *DbClient) Close() {
db.db.Close()
}
Different CRUD methods of DbClient
And the main.go file that serves all the handlers consumes it like this:
var dbClient *DbClient
func main() {
db, err := GetDBClient()
if err != nil {
panic(err)
}
dbClient = db
defer dbClient.Close()
...
}
So all the handlers of the main.go use global dbClient.
Is this architecture thread safe and does it provide atomicity of operations with database?
This design should be good.
sql.DB handles concurrent access and implements pooling. gorm inherits these features from it.
One change I would made though. Global variables are hard to manage.
You can inject db into your code that uses it.
// to be removed
// var dbClient *DbClient
func main() {
db, err := GetDBClient()
if err != nil {
panic(err)
}
defer db.Close()
CodeThatUsesDB(db)
...
}
Related
A common pattern in monolithic application design is to delegate business logic to a dedicated service, passing in an open transaction as, for example, a javax.persistence.EntityTransaction instance in Java, or an sql.Transaction in Go.
Go example:
// business.go
type BusinessLogicService interface {
DoSomething(tx *sql.Transaction)
}
type businessLogicService struct {
}
func (s *BusinessLogicService) DoSomething(tx *sql.Transaction) {
tx.ExecuteContext(.....)
}
func NewBusinessLogicService() {
return &businessLogicService{}
}
// server.go
ctx := context.Background()
tx, err := db.BeginTx(ctx)
if err != nil {
log.Fatal(err)
}
bls := business.NewBusinessLogicService()
bls.DoSomething(tx)
tx.Commit()
Could the same effect be achieved in an architecture where each of these components are implemented in a different language/runtime? In such an application, Postgres is responsible for doing the 'bookkeeping' in relation to the DB transaction. It seems to me that it should be possible to pass a similar 'handle' for the transaction to another process to read its state and append operations.
For example the equivalent business logic is provided as a gRPC service with the following definition:
message TransactionInfo {
string transaction_id = 1;
}
message DoSomethingRequest {
TransactionInfo transaction_info = 1;
}
message DoSomethingResponse {
}
service BusinessLogicService {
rpc DoSomething(DoSomethingRequest) returns (DoSomethingResponse)
}
The server process BEGINs the transaction and passes a reference to this BusinessLogicService.
ctx := context.Background()
tx, err := db.BeginTx(ctx)
if err != nil {
log.Fatal(err)
}
conn, err := grpc.Dial(*serverAddr, opts...)
if err != nil {
...
}
defer conn.Close()
bls := pb.NewBusinessLogicClient()
/// SOMEHOW PASS THE TX OBJECT TO THE REMOTE SERVICE
txObj := &pb.TransactionInfo{....???????????.....}
result := bls.DoSomething(txObj)
tx.Commit()
Is this possible with Postgres or another DBMS?
Is there a way I can set gorm to time out after a configurable period of time when running a long query? I am using mssql. I have looked through the documentation and haven't discovered a way yet.
This code seems to work for me and is pretty clean. Just use transactions I guess.
//Gorm query below
query = query.Where(whereClause)
//Set up Timeout
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var x sql.TxOptions
db := query.BeginTx(ctx, &x)
defer db.Commit()
// Execute the query
if err := db.Find(*results).Error; err != nil {
return err
}
By using WithContext from *gorm.DB you can pass a Timeout Context to Gorm:
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var products []Product
db.WithContext(ctx).Find(&products)
gorm at the moment seems is not implementing any query that accepts as a parameter a context.Context, as e.g. QueryRowContext is doing.
You can create a workaround and use a Context to make "expirable" your query.
type QueryResponse struct {
MyResult *MyResult
Error error
}
func queryHelper(ctx context.Context) <- chan *QueryResponse {
chResult := make(chan *QueryResponse, 1)
go func() {
//your query here
//...
//blah blah check stuff do whatever you want
//err is an error that comes from the query code
if err != nil {
chResult <- &QueryResponse{nil, err}
return
}
chResult <- &QueryResponse{queryResponse, nil}
} ()
return chResult
}
func MyQueryFunction(ctx context.Context) (*MyResult, error) {
select {
case <-ctx.Done():
return nil, fmt.Errorf("context timeout, query out of time")
case res := <-queryHelper(ctx):
return res.MyResult, res.Error
}
}
And then in your upper function, whatever is it you can create a context and pass it to the MyQueryFunction. If the query exceeds the time you have set an error is raised and you should (must) check it.
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
defer cancel()
res, err := MyQueryFunction(ctx)
if err != nil {
fmt.Printf("err %v", err)
} else {
fmt.Printf("res %v", res)
}
Of course, it is an example, is not considering a lot of use cases and a proper implementation inside the framework must be preferred.
I want to create a database driven application using Golang. I am trying to do it TDD way.
When I try to test methods that make Sql queries, What all are the packages available ?
I don't want to connect to the default database that I use for development. I can write code to take up another test database while running a test, but is there any go library that already does it.
Is there any library that does db tests without connecting to database at all ?
What is the standard way to do database test with golang ?
I had a similar question not long ago when refactoring some of my own tests, and there's a couple of ways you can do it:
a) Provide an exported type and an Open or Connect function that returns it - e.g.
type DB struct {
db *sql.DB
}
// Using http://jmoiron.github.io/sqlx/ for this example, but
// it has the same interface as database/sql
func Open(opts *Options) (*DB, error) {
db, err := sqlx.Connect(opts.Driver, fmt.Sprintf("host=%s user=%s dbname=%s sslmode=%s", opts.Host, opts.User, opts.Name, opts.SSL))
if err != nil {
return nil, err
}
return &DB{db}, nil
}
... and then each of your tests, write setup & teardown functions that return an instance of *DB that you define your database functions on (as methods - i.e. func (db *DB) GetUser(user *User) (bool, error)):
// Setup the test environment.
func setup() (*DB, error) {
err := withTestDB()
if err != nil {
return nil, err
}
// testOptions is a global in this case, but you could easily
// create one per-test
db, err := Open(testOptions)
if err != nil {
return nil, err
}
// Loads our test schema
db.MustLoad()
return db, nil
}
// Create our test database.
func withTestDB() error {
db, err := open()
if err != nil {
return err
}
defer db.Close()
_, err = db.Exec(fmt.Sprintf("CREATE DATABASE %s;", testOptions.Name))
if err != nil {
return err
}
return nil
}
Note that this is somewhat "integration" testing, but I strongly prefer to test against a "real" database since mocking the interface won't help you catch issues with your queries/query syntax.
b) The alternative, although less extensible on the application side, is to have a global db *sql.DB variable that you initialise in init() within your tests—since tests have no guaranteed order you'll need to use init()—and then run your tests from there. i.e.
var db *sql.DB
func init() {
var err error
// Note the = and *not* the assignment - we don't want to shadow our global
db, err = sqlx.Connect(...)
if err != nil {
...
}
err := db.loadTestSchema
// etc.
}
func TestGetUser(t *testing.T) {
user := User{}
exists, err := db.GetUser(user)
...
}
You can find some practical examples in drone.io's GitHub repo, and I'd also recommend this article on structuring Go applications (especially the DB stuff).
I use a global variable to store the data source (or connection string) of current database and set to different value in test function. Since there is only one database I need to operate so I choose the easiest way.
TLDR: What's the best way of unit-testing a function that uses a database as its datasource but that the datasource does not have to be a database (that is, the function isn't aware of where the data is coming from. e.g. it can be an object, etc.)?
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
I'm trying to find out what's the best practice for unit-testing functions that use a database as its datasource - which is passed through its parameter - but that the datasource does not have to be a database (e.g. it can be an object, etc.). The reason I add the clause, "that the datasource does not have to be a database," is because I don't want to use a database during unit-tests. So, how do I write a function that is database-agnostic?
One possible approach is to provide an "env" parameter that holds the datasource, like so:
type Env struct {
DataSource interface{}
}
func FunctionToTest(env Env) {
switch et := (env.DataSource).(type) {
case UserDatasource:
userSource := (env.DataSource).(UserDatasource)
user := userSource.getUser()
default:
// Throw error
}
}
func FunctionToTest2(env Env) {
switch et := (env.DataSource).(type) {
case CredentialsDatasource:
credentialSource := (env.DataSource).(CredentialsDatasource)
password := credentialSource.getPassword()
default:
// Throw error
}
}
The issue with this is that it seems "hacky" and it feels like there exists a better solution. The reason I want to learn how to do this is so that I can write unit-tests that mock the database.
I appreciate any and all input.
Thanks in advance!
You would want to use an interface here. A common pattern used to abstract away where the data comes from is the repository pattern. Here is an article where you can read about the repository pattern and other tips for developing well-written Go applications.
package user
type User struct {
ID int64
Name string
}
type Repository interface {
Find(id int64) (*User, error)
Store(u *User) error
}
func ChangeUserName(id int64, name string, r Repository) error {
u, err := r.Find(id)
if err != nil {
return err
}
u.Name = name
err = r.Store(u)
return err
}
Now you are able to pass in any struct with the methods matching your user.Repository interface. For example:
package mysql
type DB struct {
*sql.DB
}
func New(db *sql.DB) *DB {
return DB{db}
}
func (d *DB) Find(id int64) (*user.User, error) {
// mysql stuff to find user
}
func (d *DB) Store(u *user.User) error {
// mysql stuff to store user
}
then ...
package main
func main() {
// ...
// conn := code to open *sql.DB connection
db := mysql.New(conn)
err := user.ChangeUserName(1, 'bob', db)
// ...
}
You can now also use a mock to test your function.
package mocks
type UserRepository struct {
Users []*user.User
ShouldError bool
}
func (r *UserRepository) Find(id int64) (*user.User, error) {
if r.ShouldError {
return nil, errors.New("")
}
for _, u := range r.Users {
if u.ID == id {
return u
}
}
return errors.New("user not found")
}
func (r *UserRepository) Store(u *user.User) error {
if r.ShouldError {
return errors.New("")
}
r.Users = append(r.Users, u)
return nil
}
then to test...
func Test_ChangeUserName_Stores_Changed_User(t *testing.T) {
u := &user.User{ID: 1, Name: 'Bob'}
r := mocks.UserRepository{Users: []*user.User{u}}
err := user.ChangeUserName(1, 'Fred', r)
// ...
}
I'm a beginner in Golang and can't understand some concepts in this language. I like it very much, but every examples on the web are very simple and not explain the correct way of developing.
I want to configure db connection with MySQL. I create a package dbconfig with file dbconfig.go and package dastructure with interfaces files and another package entity with entities files.
This is the structure:
[
main.go:
import (
y "github.com/danyalov/shebeke/dbconfig"
"github.com/danyalov/shebeke/routes"
_ "github.com/go-sql-driver/mysql"
"github.com/labstack/gommon/log"
)
func main() {
db, err := y.InitDB("mysql", "root:root#tcp(localhost:3306)/dbtest?parseTime=true")
if err != nil {
log.Fatal(err)
}
e := routes.NewConnection(db)
e.Logger.Fatal(e.Start(":9898"))
}
routes.go:
import (
"github.com/danyalov/shebeke/datastructure"
y "github.com/danyalov/shebeke/dbconfig"
"github.com/labstack/echo"
"github.com/labstack/echo/middleware"
)
func NewConnection(db *y.DB) *echo.Echo {
e := echo.New()
env := Env{db}
e.Use(middleware.Logger())
e.Use(middleware.Recover())
e.GET("/contracts", env.GetContracts)
e.GET("/contract/:id", env.GetContractByID)
return e
}
type Env struct {
contract datastructure.Contract
}
services.go:
import (
"github.com/labstack/echo"
"log"
"net/http"
"strconv"
)
func (env *Env) GetContracts(c echo.Context) error {
contracts, err := env.contract.GetContracts()
if err != nil {
log.Fatal(err)
}
return c.JSON(http.StatusOK, &contracts)
}
dbconfig.go:
import (
"database/sql"
"fmt"
"github.com/labstack/gommon/log"
)
type DB struct {
*sql.DB
}
//InitDB initialize mysql database
func InitDB(driver, path string) (*DB, error) {
db, err := sql.Open(driver, path)
if err != nil {
log.Fatal(err)
}
err = db.Ping()
if err != nil {
log.Fatal(err)
} else {
fmt.Println("Connected to DB")
}
return &DB{db}, err
}
datastructure/contract.go:
import y "github.com/danyalov/shebeke/datastructure/entity"
type Contract interface {
GetContracts() (y.Contracts, error)
GetContractByID(id int) (y.Contract, error)
}
datastructure/entity/contract.go:
import (
"github.com/labstack/gommon/log"
"time"
)
type Contract struct {
ID int `json:"id"`
State string `json:"state"`
StartDate time.Time `json:"start_date"`
FinishDate time.Time `json:"finish_date"`
}
type Contracts []Contract
func (db *DB) GetContracts() (c Contracts, err error) {
rows, err := db.Query("select * from contract")
if err != nil {
log.Fatal(err)
}
contract := Contract{}
for rows.Next() {
err = rows.Scan(&contract.ID, &contract.State, &contract.StartDate, &contract.FinishDate)
c = append(c, contract)
}
return c, err
}
Why can't I import DB type from dbconfig package into entity package as a method receiver? I get Unresolved type 'DB' error.
This is my working copy(Git) of this project, I put dbconfig.go inside the entity, but I don't like it, I think it's incorrect place for dbconfig file.
What is the correct file structure for configuring db in Go? Maybe you have your own examples in Git or some tutorial?
You can only define methods on a type defined in that same package. Your DB type, in this case, is defined within your dbconfig package, so your entity package can't define methods on it.
In this case, your options are to make GetContracts a function instead of a method and hand it the *dbconfig.DB as an argument, or to invert the dependency by importing your entity package in dbconfig and write GetContracts there (as a method or function, works either way). This second one may actually be the better option, because, from a design perspective, it breaks abstraction to have packages other than your database package creating SQL query strings.
The straightforward way to do this is to wrap the type with a type in your own package:
type MyRouterGroup struct {
*gin.RouterGroup
}
func InitRouter() *gin.Engine {
r := gin.Default()
v1Group := &MyRouterGroup{r.Group("v1")}
v1Group.MyReceiverMethod()
}