I find two ways to insert data into SQLServer with GORM.
GORM.DB.Exec("insert into [tableA] (value1,value2) VALUES (?,?)",v1,v2). It works.
GROM.DB.Create(&myDataStruct).Error. This reports Error and message is "LastInsertId is not supported. Please use the OUTPUT clause or add select ID = convert(bigint, SCOPE_IDENTITY()) to the end of your query."
I understand what's the means of this instruction, but I don't know how to code.
Thanks for any help.
following my code
db := mssql.GetMssqlDB()
defer db.Close()
newData := mssql.People{
Name: "Tom",
Age: 12,
}
err := db.Create(&newData).Error
if err != nil {
fmt.Println()
}
and data struct
type People struct {
ID int64 `gorm:"primary_key;column:id"`
Name string `gorm:"column:name"`
Age int `gorm:"column:age"`
}
func (p People) TableName() string {
return "dbo.people"
}
This is a bug and have been fixed since https://github.com/go-gorm/gorm/pull/2690.
If you get the same error, try update your gorm and it should work.
Related
I make a HTTP call and unmarshal a createdTimestamp field from the larger json object:
CreatedTimestamp string `json:"createdTimestamp"`
An Example of what I receive from the HTTP call for the createdTimestamp would be: "2021-07-19T18:51:23".
It won't automatically convert it to a time.Time so the only type it would really accept is a string which was working until the type changed in Postgresql to timestamp with timezone < looks like this according to Postgresql db console >(yyyy-MM-dd HH:mm:ss.ffffff). where as before it was without timezone.
I am able to implement a custom unmarshal method for the time.Time object and even format it so I can output something like 2021-07-19 18:51:23.+00 but this won't be accepted by postgres.
The exact error I see when trying to insert this (via GoLang, postgres driver) is: pq: invalid input syntax for type timestamp with time zone: ""
This error comes after I try to execute an insert using db.Exec -> db being of type *sql.DB.
I'm doing this in Go , any help would be greatly appreciated!
Code to Unmarshal Json
type CustomTime struct {
time.Time
}
func (ct *CustomTime) UnmarshalJSON(b []byte) (err error) {
timeToInsertIntoDb := strings.Trim(string(b), "\"")
if timeToInsertIntoDb == "null" {
ct.Time = time.Time{}
return
}
timeToInsertIntoDb = timeToInsertIntoDb + "Z"
ct.Time, err = time.Parse(time.RFC3339, timeToInsertIntoDb)
return
}
With formatting, it depends on the output but I do receive the output of whatever the format is. So if I do, CustomTime.Time.Format(2006-02-01 15:04:05.-07)
I will get the output of 2021-07-19 18:51:23.+00
Though at this point, i'm not even sure about the exact format needed for Timestamp with Timezone, there isn't too much documentation on this for the Golang Postgres driver.
If there is any more information needed, please ask. I'm trying my best to organize this question.
Edit 1
As suggested, I tried to append on a 'Z' to the timestamp given from the http call. After doing a parse with time.RFC3339, I am given a time of 2021-07-19T18:51:23Z - this still failed (gave the same syntax error stated above). Tried it a few different ways with this parsing. With it formatted the way I stated above, and with it formatted with it's .String() method which would give 2021-07-20 18:51:23 +0000 UTC. Both failed with pq: invalid input syntax for type timestamp with time zone: ""
Code Changes during Unmarshal:
func (ct *CustomTime) UnmarshalJSON(b []byte) (err error) {
timeToInsertIntoDb := strings.Trim(string(b), "\"")
if timeToInsertIntoDb == "null" {
ct.Time = time.Time{}
return
}
timeToInsertIntoDb = timeToInsertIntoDb + "Z"
ct.Time, err = time.Parse(time.RFC3339, timeToInsertIntoDb)
return
}
Edit 2
Another thing to mention would be that I am using the "database/sql" package with the "github.com/lib/pq" as the Postgres driver for DB connection.
Which is why the error is from pq. Just wanted to clarify cause I know others are using gorm. I can write to other tables, it's just this table having the timestamp with timezone Postgres Column I guess.
I am making the call with db.Exec("INSERT INTO db (created_timestamp) VALUES ($1)", obj.createdTimestamp.Time
I've tried passing it along as a string (it's what I did before when it was working) but now this is where i'm at since people say it's better to pass a time.Time variable.
If you check the timeFormat ctlayout provided matches closely with the timeformat standard RFC3339
const ctLayout = "2006-01-02T15:04:05"
const RFC3339 = "2006-01-02T15:04:05Z07:00"
From this blogpost.
In RFC 3339, we can also know the time-zone from the format. It
displayed in the “Z” syntax. “Z” means UTC+0. “Z” stands for Zulu
timezone which is the same with GMT or UTC
(https://stackoverflow.com/a/9706777/4075313). So if we put Z on the
DateTime, it’s mean its timezone is UTC+0.
If we change the incoming time and append 'Z' at the end the time parser should be satisfied.
Here is a refactored code. You can find working code on playground.
timeToInsertInDB := "2006-01-02T15:04:05" + "Z"
testTime, err := time.Parse(time.RFC3339, timeToInsertInDB)
if err != nil {
fmt.Println(err)
}
Note :- More detailed example of timezones
2019-10-12T07:20:50.52Z (UTC+0)
2019-10-12T07:20:50.52+00:00 (UTC+0)
2019-10-12T14:20:50.52+07:00 (UTC+7)
2019-10-12T03:20:50.52-04:00 (UTC-4)
The postgres driver expects time.Time. If the parsing is succesfull, driver should be able to insert into the database as well as unmarshall the response in the Model structs. Have a look at this example on playground.
edit 1
As per the OP's edit, I changed the drivers used I tried this with database/sql package and github.com/lib/pq but could not reproduce the issue. The insert and select worked completely fine. playground
As mentioned by Brits, this is most likely cause is a logic error in your code (but cannot confirm this as you have not shared the code).
Your question is difficult to answer because it does not currently contain sufficient information to allow me to replicate the issue.
As mentioned in the comments you can, and should, pass a time.Time when inserting timestamps into the database. Doing this means that the library takes care of passing the timestamp to the server in an appropriate format (and your code is likely to continue working if you move to, for example, SQL Server).
If you are passing a time.Time then the source of that time becomes irrelevant. This means your question does not really need to mention the conversion from JSON at all - just make sure you double check that this conversion is successful and you are passing the time.Time to Postgres.
The code below demonstrates this (and completes successfully for me with Postgres 13.1) - this code converts from JSON using the technique suggested by Shailesh Suryawanshi:
package main
import (
"database/sql"
"fmt"
_ "github.com/lib/pq"
"time"
)
func main() {
db, err := sql.Open("postgres", "DSN goes here")
if err != nil {
panic(err)
}
defer db.Close()
// Create the table (would not normally do this here but this means the demo is self contained)
_, err = db.Exec("create temporary table db(created_timestamp timestamp with time zone)")
if err != nil {
panic(err)
}
jsonTS := `2021-07-19T18:51:23`
ts, err := time.Parse(time.RFC3339, jsonTS+"Z")
if err != nil {
panic(err)
}
_, err = db.Exec("INSERT INTO db (created_timestamp) VALUES ($1)", ts)
if err != nil {
panic(err)
}
// Test retrieving the value
rows, err := db.Query("select * from db")
if err != nil {
panic(err)
}
defer rows.Close()
for rows.Next() {
var t time.Time
err = rows.Scan(&t)
if err != nil {
panic(err)
}
fmt.Println(t)
}
fmt.Println("Complete")
}
I can replicate your issue by running db.Exec("INSERT INTO db (created_timestamp) VALUES ($1)", ""); the full error is:
panic: pq: invalid input syntax for type timestamp with time zone: ""
It may be possible that your issue is due to something in the Postgres configuration; running a modified version of the above code against your server will enable you to confirm if that is the case. However, based on the above, I believe the most likely cause is a logic error in your code (but cannot confirm this as you have not shared the code).
Note: the lib/pq readme states: "We recommend using pgx which is actively maintained."
I am writing an AWS lambda to query 10 different tables from RDS(SQL Server) using Golang SDK. What I have learned so far is we have to create a similar struct for the table to query it. But as I want to query 10 tables, So I don't want to create the struct for every table, even the table schema may get changed someday.
Lately, I want to create a CSV file per table as the backup with the queried data and upload it to S3. So is it possible to directly import the CSV file into a lambda, so that I can directly upload it to S3?
You can see my current code below
func executeQuery(dbconnection *sql.DB) {
println("\n\n----------Executing Query ----------")
query := "select TOP 5 City,State,Country from IMBookingApp.dbo.Address"
rows, err := dbconnection.Query(query)
if err != nil {
fmt.Println("Error:")
log.Fatal(err)
}
println("rows", rows)
defer rows.Close()
count := 0
for rows.Next() {
var City, State, Country string
rows.Columns
err := rows.Scan(&City, &State, &Country)
if err != nil {
fmt.Println("Error reading rows: " + err.Error())
}
fmt.Printf("City: %s, State: %s, Country: %s\n", City, State, Country)
count++
}
}
This code can only work for the Address table, and not for other tables
I have also tried it with GORM
package main
import (
"fmt"
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/mssql"
)
type Currency struct {
CurrencyId int `gorm:"column:CurrencyId;"`
Code string `gorm:"column:Code;"`
Description string `gorm:"column:Description;"`
}
func main() {
db, err := gorm.Open("mssql", "sqlserver://***")
db.SingularTable(true)
gorm.DefaultTableNameHandler = func(dbVeiculosGorm *gorm.DB, defaultTableName string) string {
return "IMBookingApp.dbo.Currency"
}
fmt.Println("HasTable-Currency:", db.HasTable("ClientUser"))
var currency Currency
db.Debug().Find(¤cy)
fmt.Println("Currency:", currency)
fmt.Println("Error", err)
defer db.Close()
}
With both the approaches I couldn't find any way to make the code generic for multiple tables. I would appreciate it if anyone can give me some suggestions or if you can point to some resources.
I did not test this code but is should give you an idea how to fetch Rows into strings array.
defer rows.Close()
columns, err := rows.Columns()
if err != nil {
panic(err)
}
for rows.Next() {
receiver := make([]*string, len(columns))
err := rows.Scan(receiver)
if err != nil {
fmt.Println("Error reading rows: " + err.Error())
}
}
GO internally converts many types into strings - https://github.com/golang/go/blob/master/src/database/sql/convert.go#L219
If data is cannot be converted you have 2 options:
Easy - update your SQL query to return strings or string compatible data
Complicated. Use slice of interface{} instead of slice of *string and fill it in with default values of correct type based on rows.ColumnTypes(). Later you will have to convert real values into strings to save into csv.
Below code worked for me -
conn, _ := getConnection() // Get database connection
rows, err := conn.Query(query)
if err != nil {
fmt.Println("Error:")
log.Fatal(err)
}
defer rows.Close()
columns, err := rows.Columns()
if err != nil {
panic(err)
}
for rows.Next() {
receiver := make([]string, len(columns))
is := make([]interface{}, len(receiver))
for i := range is {
is[i] = &receiver[i]
// each is[i] will be of type interface{} - compatible with Scan()
// using the underlying concrete `*string` values from `receiver`
}
err := rows.Scan(is...)
if err != nil {
fmt.Println("Error reading rows: " + err.Error())
}
fmt.Println("receiver", receiver)
Reference:- sql: expected 3 destination arguments in Scan, not 1 in Golang
I have the following query:
SELECT ... ,
grade as [grade],
grade as [grade2]
FROM dbo.[qc_runs] r
JOIN ...
WHERE ...
I send it to MS SQL Server 2014 from my Go code and want to get back data (I am using github.com/denisenkom/go-mssqldb driver). However, I can read first grade value (type nvarchar(max)), but second one arrives empty! These are the same table fields, just duplicated. If I delete first grade value from query and leave just one, it will still arrive empty! The column is described as following:
[grade] [nvarchar](max) NULL,
SQL Management Studio executes this query just fine, both grade values are not empty, but Go code doesn't!
UPDATE #1
Go code:
evaluations, err := db.Query("EXEC qa.dbo.sp_get_evaluation_list ?", uid)
if err != nil {
if err == sql.ErrNoRows {
return list, nil
}
return list, err
}
// Get column names
colNames, err := evaluations.Columns()
if err != nil {
log.Logf("Failed to get columns: %v", err)
return list, err
}
// Result is your slice string.
readCols := make([]interface{}, len(colNames))
// Read data
for evaluations.Next() {
writeCols := make([]string, len(colNames))
for i := range writeCols {
readCols[i] = &writeCols[i]
}
evaluations.Scan(readCols...)
for i := range writeCols {
fmt.Println(colNames[i], ": ", writeCols[i])
}
...
}
Output:
...
grade : <some text from DB>
grade2 :
I'm not a go programmer. But you need to name those fields differently. How could your code possible discern between the first and second [grade]?
SELECT ... ,
grade as [grade],
grade as [grade2]
FROM dbo.[qc_runs] r
JOIN ...
WHERE ...
Given that you're getting an empty value on the second read, I suspect that the go driver (and the data reader you're using from it) is one-way only.
Problem solved, evaluations.Scan(readCols...) returned error err sql: Scan error on column index 3: unsupported Scan, storing driver.Value type <nil> into type *string, and left the rest of array blank
I've seen two ways of people executing queries using Golang builtin database/sql queries. One of them is using fmt.Sprintf:
func (db *DB) CreateUserTable() (sql.Result, error) {
statement := "CREATE TABLE %s (%s, %s, %s, %s, %s)"
v := []interface{}{"User", "ID int PRIMARY KEY NOT NULL", "Name varchar(100) UNIQUE", "Email varchar(100) UNIQUE", "Address varchar(100) ", "Username varchar(100) UNIQUE"}
return db.Exec(fmt.Sprintf(statement, v...))
}
and the other one is using prepared statement:
func (db *DB) CreateUserTable() (sql.Result, error) {
statement, err := db.Prepare("INSERT INTO User(tbl1,tbl2,tbl3) VALUES(?,?,?)")
if err != nil {
log.Fatal(err)
}
return statement.Exec("value1", "value2", "value3")
}
The first gives benefit by enabling you to dynamically set the table name, column name, and the values. But the second one only for values. What's the difference? Which one should I use?
Never build SQL from strings that come from outside your system.
Always use the ? syntax.
If you must set SQL parts like table names, prepare multiple, complete SQL statements that contain ? for the values. Select the SQL to execute, maybe based on user input, but never build SQL from user input.
It is cleaner to use prepared statements so that whenever a requirement changes you can easily modify the statements. Also to prevent SQL injections.
Prepared statements is much better than concatenating strings, for all
the usual reasons (avoiding SQL injection attacks, for example).
In MySQL, the parameter placeholder is ?, and in PostgreSQL it is $N,
where N is a number. SQLite accepts either of these.
One more thing is Prepared statements can be used for repetitive approach, can be executed multiple times and can be destroyed.
stmt, err := db.Prepare("select id, name from users where id = ?")
if err != nil {
log.Fatal(err)
}
defer stmt.Close() // closing the statement
rows, err := stmt.Query(1)
And you are using interfaces
func (db *DB) CreateUserTable() (sql.Result, error) {
statement := "CREATE TABLE %s (%s, %s, %s, %s, %s)"
v := []interface{}{"User", "ID int PRIMARY KEY NOT NULL", "Name varchar(100) UNIQUE", "Email varchar(100) UNIQUE", "Address varchar(100) ", "Username varchar(100) UNIQUE"}
return db.Exec(fmt.Sprintf(statement, v...))
}
which can take any type of parameter under the hood which can be vulnerable
For more detailed information Go for this Link
In an application I have a globally scoped
var db *sql.DB
that is later called with
slcstrSource, slcint64Timestamp, slcstrContent, err := DB_functions.GetContent(db)
if err != nil {
fmt.Println("Error: " + err.Error())
}
GetContent is this:
func GetContent(db *sql.DB) ([]string, []int64, []string, error) {
var slcstrContent []string
var slcint64Timestamp []int64
var slcstrSource []string
// Run the query
rows, err := db.Query("SELECT source, timestamp, content FROM MyDatabase.MyTable")
if err != nil {
return slcstrSource, slcint64Timestamp, slcstrContent, err
}
defer rows.Close()
for rows.Next() {
// Holding variables for the content in the columns
var source, content string
var timestamp int64
// Get the results of the query
err := rows.Scan(&source, ×tamp, &content)
if err != nil {
return slcstrSource, slcint64Timestamp, slcstrContent, err
}
// Append them into the slices that will eventually be returned to the caller
slcstrSource = append(slcstrSource, source)
slcstrContent = append(slcstrContent, content)
slcint64Timestamp = append(slcint64Timestamp, timestamp)
}
return slcstrSource, slcint64Timestamp, slcstrContent, nil
}
When I run the application and these sections of code are hit, I get an:
Error: mssql: Invalid object name 'MyDatabase.MyTable'.
When I db.Ping() the database, it seems to work. From what I've narrowed down the error is happening right at the query, but I can't find what's wrong. I checked the database and there is a database called MyDatabase with a table called MyTable and the table has information in those three columns...
Is there something I'm missing before making the query, or in making the query?
I checked the database and there is a database called MyDatabase with
a table called MyTable and the table has information in those three
columns...
It seems like the driver is working just like it should. In order to query a table in SQL Server you should use [Database].[Schema].[TableName]. If you have not defined a particular schema name for your table then it will be created under the dbo schema by default.
In saying that you don't really need to specify the database name in your query. You rather define that on the connection string. I'm not sure how you have defined your connection details but have a look at the below and adapt accordingly to your needs.
var (
debug = flag.Bool("debug", false, "enable debugging")
password = flag.String("password", "mypwd", "the database password")
port *int = flag.Int("port", 1433, "the database port")
server = flag.String("server", "MyServer", "the database server")
user = flag.String("user", "MyUser", "the database user")
connStr = fmt.Sprintf("server=%s;Initial Catalog=MySchema;userid=%s;password=%s;port=%d", *server, *user, *password, *port)
)
func main() {
db, err := sql.Open("mssql", connStr)
}
Then you can query your table like this:
rows, err := db.Query("SELECT source, timestamp, content FROM MySchema.MyTable")