How to access database connection in http handler in Golang? - database

Im new to the Golang language and im coming from Nodejs, where it was quite simple to access database and manipulate db within a http request handler.Now i want to do the same in Golang and i cant access the db variable from the handlers.
Lets say i want to get users on a get request from a postgres db.
func getHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "get req made")
rows, error := db.Query("SELECT id, name, age FROM new_table LIMIT $1", 4)
if error != nil {
panic(error)
}
defer rows.Close()
for rows.Next() {
var id int
var name string
var age int
error = rows.Scan(&id, &name, &age)
if error != nil {
panic(error)
}
fmt.Println(id, name, age)
}
error = rows.Err()
if error != nil {
panic(error)
}
}```
And i get error: undeclared name: db, if i use this code inside the main function where the db connection is located, its working fine.
How can i use the db variable outside the scope where its declared?

Probably your db variable is created in your main func when you try to connect to the database. The problem is, that db variable will have a scope within your main func only. So to work, you need to declare it globally at package level.
So in your main.go declare a variable outside of your main func, then use it everywhere.
package main
var db *DB
func main() {
var err error
db, err = sql.Connect(...)
if err != nil {
log.Fatal(err)
}
defer db.Close()
//Start HTTP server
}
But if you use global variables, you must check whenever it supports multi threaded access. Your db connection will work fine, but you have to read some tutorial about variable scopes and mutexes in go.

If db is from another package make sure it has public access. You will need to start with a capital letter e.g. Db
This would be a basic database function in a separate file using gorm.io
package boot
import (
"fmt"
"log"
"os"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
var DB *gorm.DB
func ConnectDB() {
var err error
dsn := fmt.Sprintf("host=%s user=%s password=%s dbname=%s port=%s sslmode=disable",
os.Getenv("PG_HOST"),
os.Getenv("PG_USER"),
os.Getenv("PG_PASSWORD"),
os.Getenv("PG_DBNAME"),
os.Getenv("PG_PORT"),
)
DB, err = gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
log.Println(err)
panic("Failled to connect to Database. ")
}
}

Related

Mongo-Go-Driver Failing to Connect

So I am trying to use https://github.com/mongodb/mongo-go-driver to connect to a mongo database in golang.
Here is my connection handler:
var DB *mongo.Database
func CreateConnectionHandler()(*mongo.Database, error){
fmt.Println("inside createConnection in database package")
godotenv.Load()
fmt.Println("in CreateConnectionHandler and SERVER_CONFIG: ")
fmt.Println(os.Getenv("SERVER_CONFIG"))
uri:=""
if os.Getenv("SERVER_CONFIG")=="kubernetes"{
fmt.Println("inside kubernetes db config")
uri = "mongodb://patientplatypus:SUPERSECRETPASSDOOT#
mongo-release-mongodb.default.svc.cluster.local:27017/
platypusNEST?authMechanism=SCRAM-SHA-1"
}else if os.Getenv("SERVER_CONFIG")=="compose"{
fmt.Println("inside compose db config")
uri = "mongodb://datastore:27017"
}
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
client, err := mongo.Connect(ctx, uri)
if err != nil {
return nil, fmt.Errorf("mongo client couldn't connect: %v", err)
}
DB := client.Database("platypusNEST")
return DB, nil
}
And the error I am getting:
api | database/connection.go:29:30: cannot use uri (type
string) as type *options.ClientOptions in argument to mongo.Connect
So I tried replacing uri with the connection string like this:
client, err := mongo.Connect(ctx, "mongodb://datastore:27017")
But I still got the error.
Compare this with what is in the documentation:
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
client, err := mongo.Connect(ctx, "mongodb://localhost:27017")
And it is exactly the same! I'm really not sure why there is this error. Any ideas?
To those who come searching - the docs are out of date as of this posting, but their latest push here: https://github.com/mongodb/mongo-go-driver/commit/32946b1f8b9412a6a94e68ff789575327bb257cf has them doing this with the connect:
client, err := mongo.NewClient(options.Client().ApplyURI(uri))
You will also now need to import the options package. Happy hacking.
EDIT: thanks vcanales for finding this - you're a gentleman and a scholar.
In addition to the accepted answer - this snippet below may be improved by using an environment variable to pass in the Mongodb URL.
package main
import (
"context" //use import withs " char
"fmt"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"go.mongodb.org/mongo-driver/mongo/readpref"
)
func ConnectMongo() {
var (
client *mongo.Client
mongoURL = "mongodb://localhost:27017"
)
// Initialize a new mongo client with options
client, err := mongo.NewClient(options.Client().ApplyURI(mongoURL))
// Connect the mongo client to the MongoDB server
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
err = client.Connect(ctx)
// Ping MongoDB
ctx, _ = context.WithTimeout(context.Background(), 10*time.Second)
if err = client.Ping(ctx, readpref.Primary()); err != nil {
fmt.Println("could not ping to mongo db service: %v\n", err)
return
}
fmt.Println("connected to nosql database:", mongoURL)
}
func main() {
ConnectMongo()
}
More information on options and readpref respectively:
https://docs.mongodb.com/manual/reference/method/cursor.readPref/index.html
https://docs.mongodb.com/manual/core/read-preference/

Best way to connect to multiple database in go postgress

I'm developing a website builder and store each website data in separate databases.
My question is how to handle multiple database connection properly and efficiently.
All databases and code are in same server
I have create my own way to connect to multiple connection database.
first I create the base file for postgre :
type PostgreHost struct {
Driver string
Database string
Username string
Ssl string
Password string
}
type PostgreSystem interface {
Init()
Connect() (*sqlx.DB, error)
}
var logger *log.Logger
func (p PostgreHost) Init() {
logger = log.New(os.Stderr,
"Postgre",
log.Ldate|log.Ltime|log.Lshortfile)
}
func (p *PostgreHost) Connect() (*sqlx.DB, error) {
connection := fmt.Sprintf("user=%v password= %v dbname=%v sslmode=%v", p.Username, p.Password, p.Database, p.Ssl)
db, err := sqlx.Connect(
p.Driver,
connection)
if err != nil {
logger.Fatal(err)
return nil, err
}
return db, nil
}
func GetPostgreDb(postgre PostgreSystem) (*sqlx.DB, error) {
return postgre.Connect()
}
and then call it to create list of the connection like this :
//we create different types of databse connection here
func SystemConnection() map[string]interface{} {
listConnection := make(map[string]interface{})
var err error
// create redis connection
redisConn := RedisHost{
Address: "localhost:6379",
Password: "",
DB: 0,
}
redisConnection, err := redisConn.Connect()
if err != nil {
panic(err)
}
// create postgre connection
postgreConn := PostgreHost{
Driver: "postgres",
Database: "postgres",
Username: "postgres",
Ssl: "disable",
Password: "root",
}
// you can create your another connection here :
postgreConn2 := PostgreHost{
Driver: "postgres",
Database: "postgres",
Username: "postgres",
Ssl: "disable",
Password: "root",
}
postgreConnection, err := GetPostgreDb(&postgreConn)
if err != nil {
panic(err)
}
postgreConnection2, err := GetPostgreDb(&postgreConn2)
if err != nil {
panic(err)
}
listConnection["redis"] = redisConnection
listConnection["postgre"] = postgreConnection
listConnection["postgre2"] = postgreConnection2
return listConnection
}
and finally call all the connection from the map :
//getting list of all the connection.
listConnection := database.SystemConnection()
//getting redis connection convert it from interface to *redisClient.
redisConn := listConnection["redis"].(*redis.Client)
// get postgre connection.
postgreConn := listConnection["postgre"].(*sqlx.DB)
postgreConn2 := listConnection["postgre2"].(*sqlx.DB)
you can get all the source code from here. it still on progress but hopefully you can get the idea. Hope it helps.
Have a context struct to represent a site
This context struct holds a DB connection
When a user logged in, also set which site is active for him / her (I assume that a user account can have more than one sites).
When the user accesses the server, check the cookie for the site
Carefully use mutex to get the context from global list / map (map would be map[string]Context) based on the site name obtained from the cookie.
If the context doesn't exist, instantiate one. Internally, it creates a connection to the DB to the appropriate table for the site and registers itself to the global list (use mutex).
In each context instance, have a timer for certain minutes which resets when accessed. When it timed out (i.e., when it's not accessed for certain minutes), or the last user who owns the site logged out, it will remove itself from global list (again, use mutex) and drop its DB connection.

Cannot change consistency using "goapp test"

Google's AppEngine service provides an eventually consistent database for storage of application data in production. For testing, Google provides a similar database that emulates the consistency characteristics of the production database. Testing requirements may vary so Google supplies command line parameters to their test server, dev_appserver.py, that modify the consistency characteristics as needed for testing.
I am using the goapp tools to run our automated test environment for my AppEngine development. goapp test is responsible for running our automated server API tests. goapp test does not appear to have a way of setting the datastore's consistency level via the command line parameters, unlike dev_appserver.py, even though goapp test launches dev_appserver.py at some point during the testing process.
Is there a way to pass command line parameters to dev_appserver.py from goapp test? If not, is there an alternative method for setting the consistency from the command line?
I presume you are using the aetest package.
If this is the case you should set the StronglyConsistentDatastore member of the aetest.Options struct accordingly.
Here is an example:
hello.go
package hello
import (
"fmt"
"net/http"
"time"
"appengine"
datastore "appengine/datastore"
)
type Employee struct {
FirstName string
LastName string
HireDate time.Time
}
func init() {
http.HandleFunc("/", handler)
}
func handler(w http.ResponseWriter, r *http.Request) {
ctx := appengine.NewContext(r)
q := datastore.NewQuery("Employee").Filter("FirstName =", "Antonio")
var people []Employee
if _, err := q.GetAll(ctx, &people); err != nil {
fmt.Fprintf(w, "Error %v", err)
}
fmt.Fprintf(w, "Matches %v", len(people))
}
hello_test.go
package hello
import (
"time"
"testing"
"net/http/httptest"
"appengine"
"appengine/aetest"
datastore "appengine/datastore"
)
const consistency_strong = true; // set to false and the test will fail
func TestMyHandler(t *testing.T) {
options := &aetest.Options{StronglyConsistentDatastore: consistency_strong}
inst, err := aetest.NewInstance(options)
if err != nil {
t.Fatalf("Failed to create instance: %v", err)
}
defer inst.Close()
req, err := inst.NewRequest("GET", "/", nil)
if err != nil {
t.Fatalf("Failed to create req1: %v", err)
}
ctx := appengine.NewContext(req)
employee := &Employee{
FirstName: "Antonio",
LastName: "Salieri",
HireDate: time.Now(),
}
key := datastore.NewIncompleteKey(ctx, "Employee", nil)
_, err = datastore.Put(ctx, key, employee)
if err != nil {
t.Fatalf("Error setting test data: %v", err)
}
w := httptest.NewRecorder()
handler(w, req)
if w.Body.String() != "Matches 1" {
t.Fatalf("Expected 1 record got %v", w.Body)
}
}
As you have mentioned the go tools merely wrap dev_appserver.py. This means that if you are not using aetest you can always run your app with a chosen consistency policy like so:
/usr/local/go_appengine/dev_appserver.py --datastore_consistency_policy consistent .

Trouble with Queries/Datastore in Google App Engine - Go API

I'm playing with Google App Engine using the Go APIs and despite everything I've tried I can't get the Queries to return data that is in the datastore. I can see that Put() works as expected, and the entities are listable/accessible from inspection of the Admin Console available from dev_appserver.py
The struct I'm storing is defined this way:
type TweetData struct {
Id int64 `datastore:",noindex" json:"id"`
Text string `datastore:",noindex" json:"text"`
Screen_name string `json:"screen_name"`
}
And my calls to query it are such:
func getDatastoreTweets(c appengine.Context, twitterUser string) []*TweetData {
q := datastore.NewQuery("TweetData").Filter("Screen_name =", twitterUser).Order("-Id").Limit(10)
var oldTweets []*TweetData
if _, err := q.GetAll(c, &oldTweets); err != nil {
fmt.Printf("Getall had non-nil error! %v\n", err)
}
return oldTweets
}
For this query, err is never non-nil, but always returns 0 results, even when the Console tells me there are many. I've tried it without the Filter call as well, after the guestbook example the SDK provides, to no avail.
In case it's an issue with keys (unlikely if I understand correctly, because I'm querying on a property), the call to Put is as follows:
// tweetData passed in via parameter...
key := datastore.NewIncompleteKey(c, "TweetData", nil)
_, err := datastore.Put(c, key, &tweetData)
Any help would be appreciated, thanks! ^_^
The query ask for ordering by Id desc, while the index field is un-indexed, you should either:
rewrite the TweetData annotation to index Id field:
Id int64 `json:"id"`
Remove the Order clause of your query:
q := datastore.NewQuery("TweetData").Filter("Screen_name =", twitterUser).Limit(10)

Golang GAE - intID in struct for mustache

Here is an Example of the app. The essential code is in: golang-code/handler/handler.go (After the subject should appear an ID!)
Im trying to build a little blog system in Golang on Google Appengine and use Mustache as template engine.
So, i have a struct:
type Blogposts struct {
PostTitle string
PostPreview string
Content string
Creator string
Date time.Time
}
The data is passed to GAE via
datastore.Put(c, datastore.NewIncompleteKey(c, "Blogposts", nil), &blogposts)
So, GAE assigns automatically a intID (int64).
Now I tried to get the latest blogposts
// Get the latest blogposts
c := appengine.NewContext(r)
q := datastore.NewQuery("Blogposts").Order("-Date").Limit(10)
var blogposts []Blogposts
_, err := q.GetAll(c, &blogposts)
Until there all things works fine, but when I try to access intID (or stringID, whatever) I dont have access to this :-(
<h3>{{{PostTitle}}}</h3>
(PostTitle works, intID not, i've tried thousand of things, nothing worked :-( )
Anyone an idea? This would be great!
Edit:
I use mustache.
http://mustache.github.com/
In the code I use:
x["Blogposts"] = blogposts
data := mustache.RenderFile("templates/about.mustache", x)
sendData(w, data) // Equivalent to fmt.Fprintf
And then the data can be accessed in the .mustache template with {{{Content}}} or {{{PostTitle}}} etc.
As hyperslug pointed out, the id field of an entity is on the key, not the struct it gets read into.
Another way around this is to add an id field to your struct and tell datastore to ignore it, eg:
type Blogposts struct {
PostTitle string
PostPreview string
Content string
Creator string
Date time.Time
Id int64 `datastore:"-"`
}
You can then populate the Id field manually after a call to GetAll() like so
keys, err := q.GetAll(c, &blogposts)
if err != nil {
// handle the error
return
}
for i, key := range keys {
blogposts[i].Id = key.IntID()
}
This has the benefit of not introducing an extra type.
intID is an internal property of a Key not the struct, and is accessible through a getter:
id := key.IntID()
GetAll returns []*Key, which you're not using:
_, err := q.GetAll(c, &blogposts)
One way to get around this is to create a viewmodel struct that has both your post and key info (untested, but this is the gist of it):
//... handler code ...
keys, err := q.GetAll(c, &blogposts)
if err != nil {
http.Error(w, "Problem fetching posts.", http.StatusInternalServerError)
return
}
models := make([]BlogPostVM, len(blogposts))
for i := 0; i < len(blogposts); i++ {
models[i].Id = keys[i].IntID()
models[i].Title = blogposts[i].Title
models[i].Content = blogposts[i].Content
}
//... render with mustache ...
}
type BlogPostVM struct {
Id int
Title string
Content string
}
I know this question is a couple years old, but the following article was very helpful to me in this regard: Golang basics with Google Datastore.
In the article, the author provides a nice example of how you can run a query that gets an entity by its ID...
func GetCategory(c appengine.Context, id int64) (*Category, error) {
var category Category
category.Id = id
k := category.key(c)
err := datastore.Get(c, k, &category)
if err != nil {
return nil, err
}
category.Id = k.IntID()
return &category, nil
}
...as well as getting a list/collection of entities with their associated ID:
func GetCategories(c appengine.Context) ([]Category, error) {
q := datastore.NewQuery("Category").Order("Name")
var categories []Category
keys, err := q.GetAll(c, &categories)
if err != nil {
return nil, err
}
// you'll see this a lot because instances
// do not have this by default
for i := 0; i < len(categories); i++ {
categories[i].Id = keys[i].IntID()
}
return categories, nil
}
The snippet above is very close to the helpful answer by #koz.
AFAICS, the Blogposts struct has no field intID, but it has a field PostTitle. I guess that could be the reason why the former doesn't and the later does get rendered, though I've never used Mustache...

Resources