I'm developing a website builder and store each website data in separate databases.
My question is how to handle multiple database connection properly and efficiently.
All databases and code are in same server
I have create my own way to connect to multiple connection database.
first I create the base file for postgre :
type PostgreHost struct {
Driver string
Database string
Username string
Ssl string
Password string
}
type PostgreSystem interface {
Init()
Connect() (*sqlx.DB, error)
}
var logger *log.Logger
func (p PostgreHost) Init() {
logger = log.New(os.Stderr,
"Postgre",
log.Ldate|log.Ltime|log.Lshortfile)
}
func (p *PostgreHost) Connect() (*sqlx.DB, error) {
connection := fmt.Sprintf("user=%v password= %v dbname=%v sslmode=%v", p.Username, p.Password, p.Database, p.Ssl)
db, err := sqlx.Connect(
p.Driver,
connection)
if err != nil {
logger.Fatal(err)
return nil, err
}
return db, nil
}
func GetPostgreDb(postgre PostgreSystem) (*sqlx.DB, error) {
return postgre.Connect()
}
and then call it to create list of the connection like this :
//we create different types of databse connection here
func SystemConnection() map[string]interface{} {
listConnection := make(map[string]interface{})
var err error
// create redis connection
redisConn := RedisHost{
Address: "localhost:6379",
Password: "",
DB: 0,
}
redisConnection, err := redisConn.Connect()
if err != nil {
panic(err)
}
// create postgre connection
postgreConn := PostgreHost{
Driver: "postgres",
Database: "postgres",
Username: "postgres",
Ssl: "disable",
Password: "root",
}
// you can create your another connection here :
postgreConn2 := PostgreHost{
Driver: "postgres",
Database: "postgres",
Username: "postgres",
Ssl: "disable",
Password: "root",
}
postgreConnection, err := GetPostgreDb(&postgreConn)
if err != nil {
panic(err)
}
postgreConnection2, err := GetPostgreDb(&postgreConn2)
if err != nil {
panic(err)
}
listConnection["redis"] = redisConnection
listConnection["postgre"] = postgreConnection
listConnection["postgre2"] = postgreConnection2
return listConnection
}
and finally call all the connection from the map :
//getting list of all the connection.
listConnection := database.SystemConnection()
//getting redis connection convert it from interface to *redisClient.
redisConn := listConnection["redis"].(*redis.Client)
// get postgre connection.
postgreConn := listConnection["postgre"].(*sqlx.DB)
postgreConn2 := listConnection["postgre2"].(*sqlx.DB)
you can get all the source code from here. it still on progress but hopefully you can get the idea. Hope it helps.
Have a context struct to represent a site
This context struct holds a DB connection
When a user logged in, also set which site is active for him / her (I assume that a user account can have more than one sites).
When the user accesses the server, check the cookie for the site
Carefully use mutex to get the context from global list / map (map would be map[string]Context) based on the site name obtained from the cookie.
If the context doesn't exist, instantiate one. Internally, it creates a connection to the DB to the appropriate table for the site and registers itself to the global list (use mutex).
In each context instance, have a timer for certain minutes which resets when accessed. When it timed out (i.e., when it's not accessed for certain minutes), or the last user who owns the site logged out, it will remove itself from global list (again, use mutex) and drop its DB connection.
Related
Im new to the Golang language and im coming from Nodejs, where it was quite simple to access database and manipulate db within a http request handler.Now i want to do the same in Golang and i cant access the db variable from the handlers.
Lets say i want to get users on a get request from a postgres db.
func getHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "get req made")
rows, error := db.Query("SELECT id, name, age FROM new_table LIMIT $1", 4)
if error != nil {
panic(error)
}
defer rows.Close()
for rows.Next() {
var id int
var name string
var age int
error = rows.Scan(&id, &name, &age)
if error != nil {
panic(error)
}
fmt.Println(id, name, age)
}
error = rows.Err()
if error != nil {
panic(error)
}
}```
And i get error: undeclared name: db, if i use this code inside the main function where the db connection is located, its working fine.
How can i use the db variable outside the scope where its declared?
Probably your db variable is created in your main func when you try to connect to the database. The problem is, that db variable will have a scope within your main func only. So to work, you need to declare it globally at package level.
So in your main.go declare a variable outside of your main func, then use it everywhere.
package main
var db *DB
func main() {
var err error
db, err = sql.Connect(...)
if err != nil {
log.Fatal(err)
}
defer db.Close()
//Start HTTP server
}
But if you use global variables, you must check whenever it supports multi threaded access. Your db connection will work fine, but you have to read some tutorial about variable scopes and mutexes in go.
If db is from another package make sure it has public access. You will need to start with a capital letter e.g. Db
This would be a basic database function in a separate file using gorm.io
package boot
import (
"fmt"
"log"
"os"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
var DB *gorm.DB
func ConnectDB() {
var err error
dsn := fmt.Sprintf("host=%s user=%s password=%s dbname=%s port=%s sslmode=disable",
os.Getenv("PG_HOST"),
os.Getenv("PG_USER"),
os.Getenv("PG_PASSWORD"),
os.Getenv("PG_DBNAME"),
os.Getenv("PG_PORT"),
)
DB, err = gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
log.Println(err)
panic("Failled to connect to Database. ")
}
}
So I am trying to use https://github.com/mongodb/mongo-go-driver to connect to a mongo database in golang.
Here is my connection handler:
var DB *mongo.Database
func CreateConnectionHandler()(*mongo.Database, error){
fmt.Println("inside createConnection in database package")
godotenv.Load()
fmt.Println("in CreateConnectionHandler and SERVER_CONFIG: ")
fmt.Println(os.Getenv("SERVER_CONFIG"))
uri:=""
if os.Getenv("SERVER_CONFIG")=="kubernetes"{
fmt.Println("inside kubernetes db config")
uri = "mongodb://patientplatypus:SUPERSECRETPASSDOOT#
mongo-release-mongodb.default.svc.cluster.local:27017/
platypusNEST?authMechanism=SCRAM-SHA-1"
}else if os.Getenv("SERVER_CONFIG")=="compose"{
fmt.Println("inside compose db config")
uri = "mongodb://datastore:27017"
}
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
client, err := mongo.Connect(ctx, uri)
if err != nil {
return nil, fmt.Errorf("mongo client couldn't connect: %v", err)
}
DB := client.Database("platypusNEST")
return DB, nil
}
And the error I am getting:
api | database/connection.go:29:30: cannot use uri (type
string) as type *options.ClientOptions in argument to mongo.Connect
So I tried replacing uri with the connection string like this:
client, err := mongo.Connect(ctx, "mongodb://datastore:27017")
But I still got the error.
Compare this with what is in the documentation:
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
client, err := mongo.Connect(ctx, "mongodb://localhost:27017")
And it is exactly the same! I'm really not sure why there is this error. Any ideas?
To those who come searching - the docs are out of date as of this posting, but their latest push here: https://github.com/mongodb/mongo-go-driver/commit/32946b1f8b9412a6a94e68ff789575327bb257cf has them doing this with the connect:
client, err := mongo.NewClient(options.Client().ApplyURI(uri))
You will also now need to import the options package. Happy hacking.
EDIT: thanks vcanales for finding this - you're a gentleman and a scholar.
In addition to the accepted answer - this snippet below may be improved by using an environment variable to pass in the Mongodb URL.
package main
import (
"context" //use import withs " char
"fmt"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"go.mongodb.org/mongo-driver/mongo/readpref"
)
func ConnectMongo() {
var (
client *mongo.Client
mongoURL = "mongodb://localhost:27017"
)
// Initialize a new mongo client with options
client, err := mongo.NewClient(options.Client().ApplyURI(mongoURL))
// Connect the mongo client to the MongoDB server
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
err = client.Connect(ctx)
// Ping MongoDB
ctx, _ = context.WithTimeout(context.Background(), 10*time.Second)
if err = client.Ping(ctx, readpref.Primary()); err != nil {
fmt.Println("could not ping to mongo db service: %v\n", err)
return
}
fmt.Println("connected to nosql database:", mongoURL)
}
func main() {
ConnectMongo()
}
More information on options and readpref respectively:
https://docs.mongodb.com/manual/reference/method/cursor.readPref/index.html
https://docs.mongodb.com/manual/core/read-preference/
I have a simple question, I been playing with dev_appserver testing my Go app locally.
I saved some keys:
client, err = datastore.NewClient(ctx, utils.ProjectID)
key := datastore.NewKey(ctx, KindSpace, "", id, nil)
if _, err := client.Put(ctx, key, value); err != nil {
return err
}
However, When connecting to: http://localhost:8000/datastore my datastore is empty.
I then tried to retrieve the keys from inside my code using:
client, err = datastore.NewClient(ctx, utils.ProjectID)
entity := new(Space)
key := datastore.NewKey(ctx, KindSpace, "", id, nil)
if err := client.Get(ctx, key, entity); err != nil {
return nil, err
}
and indeed it returned the keys I saved. Do you have any idea what do I miss? Does it have something to do with the database namespace maybe?
Is there anyway to read the local data base from the terminal?
Google's AppEngine service provides an eventually consistent database for storage of application data in production. For testing, Google provides a similar database that emulates the consistency characteristics of the production database. Testing requirements may vary so Google supplies command line parameters to their test server, dev_appserver.py, that modify the consistency characteristics as needed for testing.
I am using the goapp tools to run our automated test environment for my AppEngine development. goapp test is responsible for running our automated server API tests. goapp test does not appear to have a way of setting the datastore's consistency level via the command line parameters, unlike dev_appserver.py, even though goapp test launches dev_appserver.py at some point during the testing process.
Is there a way to pass command line parameters to dev_appserver.py from goapp test? If not, is there an alternative method for setting the consistency from the command line?
I presume you are using the aetest package.
If this is the case you should set the StronglyConsistentDatastore member of the aetest.Options struct accordingly.
Here is an example:
hello.go
package hello
import (
"fmt"
"net/http"
"time"
"appengine"
datastore "appengine/datastore"
)
type Employee struct {
FirstName string
LastName string
HireDate time.Time
}
func init() {
http.HandleFunc("/", handler)
}
func handler(w http.ResponseWriter, r *http.Request) {
ctx := appengine.NewContext(r)
q := datastore.NewQuery("Employee").Filter("FirstName =", "Antonio")
var people []Employee
if _, err := q.GetAll(ctx, &people); err != nil {
fmt.Fprintf(w, "Error %v", err)
}
fmt.Fprintf(w, "Matches %v", len(people))
}
hello_test.go
package hello
import (
"time"
"testing"
"net/http/httptest"
"appengine"
"appengine/aetest"
datastore "appengine/datastore"
)
const consistency_strong = true; // set to false and the test will fail
func TestMyHandler(t *testing.T) {
options := &aetest.Options{StronglyConsistentDatastore: consistency_strong}
inst, err := aetest.NewInstance(options)
if err != nil {
t.Fatalf("Failed to create instance: %v", err)
}
defer inst.Close()
req, err := inst.NewRequest("GET", "/", nil)
if err != nil {
t.Fatalf("Failed to create req1: %v", err)
}
ctx := appengine.NewContext(req)
employee := &Employee{
FirstName: "Antonio",
LastName: "Salieri",
HireDate: time.Now(),
}
key := datastore.NewIncompleteKey(ctx, "Employee", nil)
_, err = datastore.Put(ctx, key, employee)
if err != nil {
t.Fatalf("Error setting test data: %v", err)
}
w := httptest.NewRecorder()
handler(w, req)
if w.Body.String() != "Matches 1" {
t.Fatalf("Expected 1 record got %v", w.Body)
}
}
As you have mentioned the go tools merely wrap dev_appserver.py. This means that if you are not using aetest you can always run your app with a chosen consistency policy like so:
/usr/local/go_appengine/dev_appserver.py --datastore_consistency_policy consistent .
I am having issues with urlfetch's timeouts on Google App Engine in Go. The app does not appear to want to take a longer timeout than about 5 seconds (it ignores a longer timeout and times out after its own time).
My code is:
var TimeoutDuration time.Duration = time.Second*30
func Call(c appengine.Context, address string, allowInvalidServerCertificate bool, method string, id interface{}, params []interface{})(map[string]interface{}, error){
data, err := json.Marshal(map[string]interface{}{
"method": method,
"id": id,
"params": params,
})
if err != nil {
return nil, err
}
req, err:=http.NewRequest("POST", address, strings.NewReader(string(data)))
if err!=nil{
return nil, err
}
tr := &urlfetch.Transport{Context: c, Deadline: TimeoutDuration, AllowInvalidServerCertificate: allowInvalidServerCertificate}
resp, err:=tr.RoundTrip(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
result := make(map[string]interface{})
err = json.Unmarshal(body, &result)
if err != nil {
return nil, err
}
return result, nil
}
No matter what I try to set TimeoutDuration to, the app times out after about 5 seconds. How prevent it from doing that? Did I make some error in my code?
You need to pass the time duration like this (otherwise it will default to the 5 sec timeout):
tr := &urlfetch.Transport{Context: c, Deadline: time.Duration(30) * time.Second}
Update Jan 2 2016:
With the new GAE golang packages (google.golang.org/appengine/*), this has changed. urlfetch no longer receives a deadline time duration in the transport.
You should now set the timeout via the new context package. For example, this is how you would set a 1 minute deadline:
func someFunc(ctx context.Context) {
ctx_with_deadline, _ := context.WithTimeout(ctx, 1*time.Minute)
client := &http.Client{
Transport: &oauth2.Transport{
Base: &urlfetch.Transport{Context: ctx_with_deadline},
},
}
Try the code below:
// createClient is urlfetch.Client with Deadline
func createClient(context appengine.Context, t time.Duration) *http.Client {
return &http.Client{
Transport: &urlfetch.Transport{
Context: context,
Deadline: t,
},
}
}
Here is how to use it.
// urlfetch
client := createClient(c, time.Second*60)
Courtesy #gosharplite
Looking at the source code of Go's appengine:
http://code.google.com/p/appengine-go/source/browse/appengine/urlfetch/urlfetch.go
and the protobuffer generated code:
http://code.google.com/p/appengine-go/source/browse/appengine_internal/urlfetch/urlfetch_service.pb.go
Looks like there should not be a problem with Duration itself.
My guess is that the whole application inside appengine timeouts after 5 seconds.
for me, this worked:
ctx_with_deadline, _ := context.WithTimeout(ctx, 15*time.Second)
client := urlfetch.Client(ctx_with_deadline)
This is had now changed with the recent updates to the library . Now the Duration of timeout/delay have to carried by the context , urlfetch.transport no more has the Deadline field in it . context.WithTimeout or context.WithDeadline is the method to use , here is the link https://godoc.org/golang.org/x/net/context#WithTimeout