i've been trying to wrap my head around unit testing, dependency injection, tdd and all that stuff and i've been stuck on testing functions that make database calls, for example.
Let's say you have a PostgresStore struct that takes in a Database interface, which has a Query() method.
type PostgresStore struct {
db Database
}
type Database interface {
Query(query string, args ...interface{}) (*sql.Rows, error)
}
And your PostgresStore has a GetPatients method, which calls database query.
func (p *PostgresStore) GetPatients() ([]Patient, error) {
rows, err := p.db.Query("SELECT id, name, age, insurance FROM patients")
if err != nil {
return nil, err
}
defer rows.Close()
items := []Patient{}
for rows.Next() {
var i Patient
if err := rows.Scan(
&i.ID,
&i.Name,
&i.Surname,
&i.Age,
&i.InsuranceCompany,
); err != nil {
return nil, err
}
items = append(items, i)
}
if err := rows.Close(); err != nil {
return nil, err
}
if err := rows.Err(); err != nil {
return nil, err
}
return items, nil
}
In the real implementation, you would just pass a *sql.DB as Database argument, but how would you guys write a unit test with a fake database struct?
let me try to clarify some of your doubts. First of all, I'm gonna share a working example to better understand what's going on. Then, I'm gonna mention all of the relevant aspects.
repo/db.go
package repo
import "database/sql"
type Patient struct {
ID int
Name string
Surname string
Age int
InsuranceCompany string
}
type PostgresStore struct {
// rely on the generic DB provided by the "sql" package
db *sql.DB
}
func (p *PostgresStore) GetPatient(id int) ([]Patient, error) {
rows, err := p.db.Query("SELECT id, name, age, insurance FROM patients")
if err != nil {
return nil, err
}
defer rows.Close()
items := []Patient{}
for rows.Next() {
var i Patient
if err := rows.Scan(
&i.ID,
&i.Name,
&i.Surname,
&i.Age,
&i.InsuranceCompany,
); err != nil {
return nil, err
}
items = append(items, i)
}
if err := rows.Close(); err != nil {
return nil, err
}
if err := rows.Err(); err != nil {
return nil, err
}
return items, nil
}
Here, the only relevant change is how you define the PostgresStore struct. As the db field, you should rely on the generic DB provided by the database/sql package of the Go Standard Library. Thanks to this, it's trivial to swap its implementation with a fake one, as we're gonna see later.
Please note that in the GetPatient method you're accepting an id parameter but you're not using it. Your query is more suitable to a method like GetAllPatients or something like that. Be sure to fix it accordingly.
repo/db_test.go
package repo
import (
"testing"
"github.com/DATA-DOG/go-sqlmock"
"github.com/stretchr/testify/assert"
)
func TestGetPatient(t *testing.T) {
// 1. set up fake db and mock
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("err not expected: %v", err)
}
// 2. configure the mock. What we expect (query or command)? The outcome (error vs no error).
rows := sqlmock.NewRows([]string{"id", "name", "surname", "age", "insurance"}).AddRow(1, "john", "doe", 23, "insurance-test")
mock.ExpectQuery("SELECT id, name, age, insurance FROM patients").WillReturnRows(rows)
// 3. instantiate the PostgresStore with the fake db
sut := &PostgresStore{
db: db,
}
// 4. invoke the action we've to test
got, err := sut.GetPatient(1)
// 5. assert the result
assert.Nil(t, err)
assert.Contains(t, got, Patient{1, "john", "doe", 23, "insurance-test"})
}
Here, there are a lot to cover. First, you can check the comments within the code that give you a better idea of each step. In the code, we're relying on the package github.com/DATA-DOG/go-sqlmock that allows us to easily mock a database client.
Obviously, the purpose of this code is to give a general idea on how to implement your needs. It can be written in a better way but it can be a good starting point for writing tests in this scenario.
Let me know if this helps, thanks!
Related
I want to create a database driven application using Golang. I am trying to do it TDD way.
When I try to test methods that make Sql queries, What all are the packages available ?
I don't want to connect to the default database that I use for development. I can write code to take up another test database while running a test, but is there any go library that already does it.
Is there any library that does db tests without connecting to database at all ?
What is the standard way to do database test with golang ?
I had a similar question not long ago when refactoring some of my own tests, and there's a couple of ways you can do it:
a) Provide an exported type and an Open or Connect function that returns it - e.g.
type DB struct {
db *sql.DB
}
// Using http://jmoiron.github.io/sqlx/ for this example, but
// it has the same interface as database/sql
func Open(opts *Options) (*DB, error) {
db, err := sqlx.Connect(opts.Driver, fmt.Sprintf("host=%s user=%s dbname=%s sslmode=%s", opts.Host, opts.User, opts.Name, opts.SSL))
if err != nil {
return nil, err
}
return &DB{db}, nil
}
... and then each of your tests, write setup & teardown functions that return an instance of *DB that you define your database functions on (as methods - i.e. func (db *DB) GetUser(user *User) (bool, error)):
// Setup the test environment.
func setup() (*DB, error) {
err := withTestDB()
if err != nil {
return nil, err
}
// testOptions is a global in this case, but you could easily
// create one per-test
db, err := Open(testOptions)
if err != nil {
return nil, err
}
// Loads our test schema
db.MustLoad()
return db, nil
}
// Create our test database.
func withTestDB() error {
db, err := open()
if err != nil {
return err
}
defer db.Close()
_, err = db.Exec(fmt.Sprintf("CREATE DATABASE %s;", testOptions.Name))
if err != nil {
return err
}
return nil
}
Note that this is somewhat "integration" testing, but I strongly prefer to test against a "real" database since mocking the interface won't help you catch issues with your queries/query syntax.
b) The alternative, although less extensible on the application side, is to have a global db *sql.DB variable that you initialise in init() within your tests—since tests have no guaranteed order you'll need to use init()—and then run your tests from there. i.e.
var db *sql.DB
func init() {
var err error
// Note the = and *not* the assignment - we don't want to shadow our global
db, err = sqlx.Connect(...)
if err != nil {
...
}
err := db.loadTestSchema
// etc.
}
func TestGetUser(t *testing.T) {
user := User{}
exists, err := db.GetUser(user)
...
}
You can find some practical examples in drone.io's GitHub repo, and I'd also recommend this article on structuring Go applications (especially the DB stuff).
I use a global variable to store the data source (or connection string) of current database and set to different value in test function. Since there is only one database I need to operate so I choose the easiest way.
I am working on a multi tenant application, I need to query a particular user from a KIND and From Particular Namespace.
I am able to get the values from default Namespace.the package i am using here is "google.golang.org/appengine/datastore"
q := datastore.NewQuery(ENTITYNAME).Filter("Name =", ed.Expense.Name)
var expenses []ExpenseEntiry
return q.GetAll(ed.Ctx, &expenses)
The namespace value is not part of the query (it's not a property of the query). The namespace comes from the context which you pass when executing the query, e.g. to Query.GetAll().
If you have a context (you do as you pass it to q.GetAll()), you can create a derivative context with a given namespace using the appengine.Namespace() function.
For example:
ctx2, err := appengine.Namespace(ed.Ctx, "mynamespace")
// check err
And use this new context to pass to Query.GetAll():
return q.GetAll(ctx2, &expenses)
It is rare that you need to create a new context with a different namespace, ed.Ctx should already be a context with the right namespace. So when / where you create ed.Ctx, you should already apply the namespace there, so you can avoid "accidental" exposure of data of other tenants (which is a major security issue).
If you are using the old lib: google.golang.org/appengine/datastore, then you need to create the context with the namespace:
ctx2, err := appengine.Namespace(ed.Ctx, "mynamespace")
if err != nil {
return err
}
But you WANT to be using the latest lib: cloud.google.com/go/datastore. The Namespace can be set directly on the Query object. This is new. You must then run the query using datastoreClient.Run(ctx, query).
func deleteTestNamespace(ctx context.Context, namespaces string) error {
dsClient, err := datastore.NewClient(ctx, log, datastore.Config{...})
err := dsClient.DeleteMulti(ctx, keys[i:i+chunk])
if err != nil {
return err
}
var keys []*datastore.Key
for _, kind := range envKinds {
// Get all keys
query := datastore.NewQuery(kind).KeysOnly().Namespace(namespace)
it := dsClient.Run(ctx, query)
for {
var key datastore.Key
_, err := it.Next(&key)
if err == iterator.Done {
break
}
if err != nil {
return err
}
keys = append(keys, &key)
}
// Delete all records in chunks of 500 or less
for i := 0; i < len(keys); i += 500 {
chunk := min(len(keys)-i, 500)
err := dsClient.DeleteMulti(ctx, keys[i:i+chunk])
if err != nil {
return err
}
}
}
return nil
}
func min(num1 int, num2 int) int {
if num1 < num2 {
return num1
}
return num2
}
TLDR: What's the best way of unit-testing a function that uses a database as its datasource but that the datasource does not have to be a database (that is, the function isn't aware of where the data is coming from. e.g. it can be an object, etc.)?
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
I'm trying to find out what's the best practice for unit-testing functions that use a database as its datasource - which is passed through its parameter - but that the datasource does not have to be a database (e.g. it can be an object, etc.). The reason I add the clause, "that the datasource does not have to be a database," is because I don't want to use a database during unit-tests. So, how do I write a function that is database-agnostic?
One possible approach is to provide an "env" parameter that holds the datasource, like so:
type Env struct {
DataSource interface{}
}
func FunctionToTest(env Env) {
switch et := (env.DataSource).(type) {
case UserDatasource:
userSource := (env.DataSource).(UserDatasource)
user := userSource.getUser()
default:
// Throw error
}
}
func FunctionToTest2(env Env) {
switch et := (env.DataSource).(type) {
case CredentialsDatasource:
credentialSource := (env.DataSource).(CredentialsDatasource)
password := credentialSource.getPassword()
default:
// Throw error
}
}
The issue with this is that it seems "hacky" and it feels like there exists a better solution. The reason I want to learn how to do this is so that I can write unit-tests that mock the database.
I appreciate any and all input.
Thanks in advance!
You would want to use an interface here. A common pattern used to abstract away where the data comes from is the repository pattern. Here is an article where you can read about the repository pattern and other tips for developing well-written Go applications.
package user
type User struct {
ID int64
Name string
}
type Repository interface {
Find(id int64) (*User, error)
Store(u *User) error
}
func ChangeUserName(id int64, name string, r Repository) error {
u, err := r.Find(id)
if err != nil {
return err
}
u.Name = name
err = r.Store(u)
return err
}
Now you are able to pass in any struct with the methods matching your user.Repository interface. For example:
package mysql
type DB struct {
*sql.DB
}
func New(db *sql.DB) *DB {
return DB{db}
}
func (d *DB) Find(id int64) (*user.User, error) {
// mysql stuff to find user
}
func (d *DB) Store(u *user.User) error {
// mysql stuff to store user
}
then ...
package main
func main() {
// ...
// conn := code to open *sql.DB connection
db := mysql.New(conn)
err := user.ChangeUserName(1, 'bob', db)
// ...
}
You can now also use a mock to test your function.
package mocks
type UserRepository struct {
Users []*user.User
ShouldError bool
}
func (r *UserRepository) Find(id int64) (*user.User, error) {
if r.ShouldError {
return nil, errors.New("")
}
for _, u := range r.Users {
if u.ID == id {
return u
}
}
return errors.New("user not found")
}
func (r *UserRepository) Store(u *user.User) error {
if r.ShouldError {
return errors.New("")
}
r.Users = append(r.Users, u)
return nil
}
then to test...
func Test_ChangeUserName_Stores_Changed_User(t *testing.T) {
u := &user.User{ID: 1, Name: 'Bob'}
r := mocks.UserRepository{Users: []*user.User{u}}
err := user.ChangeUserName(1, 'Fred', r)
// ...
}
I'm a beginner in Golang and can't understand some concepts in this language. I like it very much, but every examples on the web are very simple and not explain the correct way of developing.
I want to configure db connection with MySQL. I create a package dbconfig with file dbconfig.go and package dastructure with interfaces files and another package entity with entities files.
This is the structure:
[
main.go:
import (
y "github.com/danyalov/shebeke/dbconfig"
"github.com/danyalov/shebeke/routes"
_ "github.com/go-sql-driver/mysql"
"github.com/labstack/gommon/log"
)
func main() {
db, err := y.InitDB("mysql", "root:root#tcp(localhost:3306)/dbtest?parseTime=true")
if err != nil {
log.Fatal(err)
}
e := routes.NewConnection(db)
e.Logger.Fatal(e.Start(":9898"))
}
routes.go:
import (
"github.com/danyalov/shebeke/datastructure"
y "github.com/danyalov/shebeke/dbconfig"
"github.com/labstack/echo"
"github.com/labstack/echo/middleware"
)
func NewConnection(db *y.DB) *echo.Echo {
e := echo.New()
env := Env{db}
e.Use(middleware.Logger())
e.Use(middleware.Recover())
e.GET("/contracts", env.GetContracts)
e.GET("/contract/:id", env.GetContractByID)
return e
}
type Env struct {
contract datastructure.Contract
}
services.go:
import (
"github.com/labstack/echo"
"log"
"net/http"
"strconv"
)
func (env *Env) GetContracts(c echo.Context) error {
contracts, err := env.contract.GetContracts()
if err != nil {
log.Fatal(err)
}
return c.JSON(http.StatusOK, &contracts)
}
dbconfig.go:
import (
"database/sql"
"fmt"
"github.com/labstack/gommon/log"
)
type DB struct {
*sql.DB
}
//InitDB initialize mysql database
func InitDB(driver, path string) (*DB, error) {
db, err := sql.Open(driver, path)
if err != nil {
log.Fatal(err)
}
err = db.Ping()
if err != nil {
log.Fatal(err)
} else {
fmt.Println("Connected to DB")
}
return &DB{db}, err
}
datastructure/contract.go:
import y "github.com/danyalov/shebeke/datastructure/entity"
type Contract interface {
GetContracts() (y.Contracts, error)
GetContractByID(id int) (y.Contract, error)
}
datastructure/entity/contract.go:
import (
"github.com/labstack/gommon/log"
"time"
)
type Contract struct {
ID int `json:"id"`
State string `json:"state"`
StartDate time.Time `json:"start_date"`
FinishDate time.Time `json:"finish_date"`
}
type Contracts []Contract
func (db *DB) GetContracts() (c Contracts, err error) {
rows, err := db.Query("select * from contract")
if err != nil {
log.Fatal(err)
}
contract := Contract{}
for rows.Next() {
err = rows.Scan(&contract.ID, &contract.State, &contract.StartDate, &contract.FinishDate)
c = append(c, contract)
}
return c, err
}
Why can't I import DB type from dbconfig package into entity package as a method receiver? I get Unresolved type 'DB' error.
This is my working copy(Git) of this project, I put dbconfig.go inside the entity, but I don't like it, I think it's incorrect place for dbconfig file.
What is the correct file structure for configuring db in Go? Maybe you have your own examples in Git or some tutorial?
You can only define methods on a type defined in that same package. Your DB type, in this case, is defined within your dbconfig package, so your entity package can't define methods on it.
In this case, your options are to make GetContracts a function instead of a method and hand it the *dbconfig.DB as an argument, or to invert the dependency by importing your entity package in dbconfig and write GetContracts there (as a method or function, works either way). This second one may actually be the better option, because, from a design perspective, it breaks abstraction to have packages other than your database package creating SQL query strings.
The straightforward way to do this is to wrap the type with a type in your own package:
type MyRouterGroup struct {
*gin.RouterGroup
}
func InitRouter() *gin.Engine {
r := gin.Default()
v1Group := &MyRouterGroup{r.Group("v1")}
v1Group.MyReceiverMethod()
}
I'm working on an appengine app using the datastore. I'm attempting to gob
encode an interface and store it into the datastore. But when I try to load from
the datastore, I get the error:
gob: name not registered for interface: "main27155.strand"
The peculiar thing is that the load() method starts working after having
called the save() method. It no longer returns an error, and everything saved
in the datastore is loaded as expected. But when I restart the intance, the
load() method stops working again.
The load and save methods I mention refer to the methods defined by the
datastore.PropertyLoadSaver interface
From the looks of it, it seems like a problem with registering the
type/interfaces with gob, but I have exactly the same gob.Register() calls in
both the load() and save() methods.
I even tried removing the gob.Register() calls from both load and save methods
and adding it to init(). The exact same behavior was observed.
How can I load my datastore on a cold start?
type bio struct {¬
Id string¬
Hp int¬
godie chan bool //should be buffered¬
dam chan int¬
Genetics dna¬
}¬
type dna interface {
decode() mRNA
Get(int) trait
Set(int, trait)
Duplicate() dna
Len() int
}
type trait interface {
mutate() trait
}
// implements dna{}
type strand []trait
// implements trait{}
type tdecoration string
type color struct {
None bool // If true, colors are not shown in theme
Bg bool // If true, color is a background color
R int // 0-255
G int
B int
}
.
func start(w http.ResponseWriter, r *http.Request) error {
c := appengine.NewContext(r)
var bs []bio
if _, err := datastore.NewQuery("bio").GetAll(c, &bs); err != nil {
log.Println("bs is len: ", len(bs))
return err
}
...
return nil
}
func stop(w http.ResponseWriter, r *http.Request) error {
c := appengine.NewContext(r)
log.Println("Saving top 20 colors")
var k []*datastore.Key
var bs []*bio
stat := getStats()
for i, b := range stat.Leaderboard {
k = append(k, datastore.NewKey(c, "bio", b.Id, 0, nil))
bv := b
bs = append(bs, &bv)
// At most 20 bios survive across reboots
if i > 178 {
break
}
}
// Assemble slice of keys for deletion
dk, err := datastore.NewQuery("bio").KeysOnly().GetAll(c, nil)
if err != nil {
return errors.New(fmt.Sprintf("Query error: %s", err.Error()))
}
fn := func(c appengine.Context) error {
// Delete all old entries
err := datastore.DeleteMulti(c, dk)
if err != nil {
return errors.New(fmt.Sprintf("Delete error: %s", err.Error()))
}
// save the elite in the datastore
_, err = datastore.PutMulti(c, k, bs)
if err != nil {
return err
}
return nil
}
return datastore.RunInTransaction(c, fn, &datastore.TransactionOptions{XG: true})
}
// satisfy datastore PropertyLoadSaver interface ===============================
func (b *bio) Load(c <-chan datastore.Property) error {
gob.Register(&color{})
gob.Register(new(tdecoration))
var str strand
gob.Register(str)
tmp := struct {
Id string
Hp int
Gengob []byte
}{}
if err := datastore.LoadStruct(&tmp, c); err != nil {
return err
}
b.Id = tmp.Id
b.Hp = tmp.Hp
return gob.NewDecoder(strings.NewReader(string(tmp.Gengob))).Decode(&(b.Genetics))
}
func (b *bio) Save(c chan<- datastore.Property) error {
defer close(c)
gob.Register(&color{})
gob.Register(new(tdecoration))
var str strand
gob.Register(str)
var buf bytes.Buffer
gen := b.Genetics
if err := gob.NewEncoder(&buf).Encode(&gen); err != nil {
log.Println(err)
return err
}
dp := []datastore.Property{
{Name: "Id", Value: b.Id},
{Name: "Hp", Value: int64(b.Hp)},
{Name: "Gengob", Value: buf.Bytes(), NoIndex: true},
}
for _, p := range dp {
c <- p
}
return nil
}
Additional info: This behavior was not present before I stuffed the datastore
calls in stop() into datastore.RunInTransaction()
Register all types an in init() functions using RegisterName(). Delete all existing data from the store and you should be good to go.
App Engine generates a mangled name for the main package every time the application is built. The name generated by Register() includes this mangled package name. Any gobs encoded with the mangled name will only be readable using the same build of the app. If you cause the application to be rebuilt by modifying the code, then the app will not be able to decode gobs stored previously.