Spring Boot 2.3.4
org.springframework.boot:spring-boot-starter-web
org.springframework.boot:spring-boot-starter-data-mongodb
MongoDB 4.2.8
I have the following class:
#TypeAlias("RefreshToken")
#Document("refresh_token")
#CompoundIndex(name = "organization_user", def = "{'organizationId' : 1, 'userId': 1}")
data class RefreshToken(
#Id
val token: String,
val organizationId: UUID,
val userId: UUID,
val expiry: Date
)
A unique index/constraint on the token field is not created. I'm getting duplicates for some reason.
The indices being created are these:
Related
Let suppose a Postgres user table contains a property of type NestedMutableJson.
first_name character varying (120)
last_name character varying (120
country_info NestedMutableJson
...
from sqlalchemy_json import NestedMutableJson
country_info = db.Column(NestedMutableJson, nullable=True)
country_info = {"name": "UK", "code": "11"}
How to query the user table based on a country_info key.
POSTGRES Query
SELECT * FROM user WHERE country_info ->> 'name' = 'UK'
Does any SqlAlchemy way give the same query result?
I tried several ways, example:
Way 1:
User.query.filter(User.country_info['name'].astext == 'UK').all()
Error:
Operator 'getitem' is not supported on this expression
Way 2:
User.query.filter(User.country_info.op('->>')('name') == 'UK').all()
Issue:
Always getting an empty response
I'm wondering if the issue caused by the column definition db.Column(NestedMutableJson, nullable=True)
I'm avoiding using db.session.execute("SELECT * FROM user WHERE country_info ->> 'name' = 'UK'").fetchall(). looking for something else
simply user the text which allows you to write plain text filter inside orm process, it works like get function for a dict, and it can handle None fields also.
User.query.filter(text("COALESCE(user.country_info ->> 'name', '') = 'UK'")).all()
note that the name of the table should be the real name in the database
Take a look at the JSON datatype documentation.
You should be able to do use this filter clause
select(CountryInfo).filter(CountryInfo.country_info['name'].astext == "UK")
You can use .op('->>') to use the PostgreSQL operator ->> in the following way:
from sqlalchemy import Column, Integer, create_engine, String, select
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import Session
from sqlalchemy.dialects.postgresql import JSON
from sqlalchemy_json import NestedMutableJson
dburl = 'postgresql://...'
Base = declarative_base()
class CountryInfo(Base):
__tablename__ = 'country_info'
id = Column(Integer, unique=True, nullable = False, primary_key=True)
name = Column(String)
country_info = Column(NestedMutableJson, nullable=True)
def __repr__(self):
return f'CountryInfo({self.name!r}, {self.country_info!r})'
engine = create_engine(dburl, future=True, echo=True)
Base.metadata.drop_all(engine)
Base.metadata.create_all(engine)
with Session(engine) as session:
test = CountryInfo(name='test', country_info={"name": "UK", "code": "11"})
test2 = CountryInfo(name='test2', country_info={"name": "NL", "code": "12"})
test3 = CountryInfo(name='test3', country_info={"name": "UK", "code": "13"})
session.add(test)
session.add(test2)
session.add(test3)
session.commit()
stmt = select(CountryInfo).filter(CountryInfo.country_info.op('->>')('name') == "UK")
query = session.execute(stmt).all()
for row in query:
print(row)
This results in the following SQL:
SELECT country_info.id, country_info.name, country_info.country_info
FROM country_info
WHERE (country_info.country_info ->> %(country_info_1)s) = %(param_1)s
with {'country_info_1': 'name', 'param_1': 'UK'}
Which results in:
(CountryInfo('test', {'name': 'UK', 'code': '11'}),)
(CountryInfo('test3', {'name': 'UK', 'code': '13'}),)
I don't get it. I'm trying to start a brand new table in MS SQL Server 2012 with the following:
In SQL Server:
TABLE [dbo].[Inventory](
[Index_No] [bigint] IDENTITY(1,1) NOT NULL,
[Part_No] [varchar(150)] NOT NULL,
[Shelf] [int] NOT NULL,
[Bin] [int] NOT NULL,
PRIMARY KEY CLUSTERED
(
[Index_No] ASC
)
UNIQUE NONCLUSTERED
(
[Part_No] ASC
)
GO
NOTE: This is a BRAND NEW TABLE! There is no data in it at all
Next, this is the Database.py file:
import pymssql
from sqlalchemy import create_engine, Table, MetaData, select, Column, Integer, Float, String, text,
func, desc, and_, or_, Date, insert
from marshmallow_sqlalchemy import SQLAlchemyAutoSchema
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
USERNAME = "name"
PSSWD = "none_of_your_business"
SERVERNAME = "MYSERVER"
INSTANCENAME = "\SQLSERVER2012"
DB = "Inventory"
engine = create_engine(f"mssql+pymssql://{USERNAME}:{PSSWD}#{SERVERNAME}{INSTANCENAME}/{DB}")
class Inventory(Base):
__tablename__ = "Inventory"
Index_No = Column('Index_No', Integer, primary_key=True, autoincrement=True)
Part_No = Column("Part_No", String, unique=True)
Shelf = Column("Shelf", Integer)
Bin = Column("Bin", Integer)
def __repr__(self):
return f'Drawing(Index_No={self.Index_No!r},Part_No={self.Part_No!r}, Shelf={self.Shelf!r}, ' \
f'Bin={self.Bin!r})'
class InventorySchema(SQLAlchemyAutoSchema):
class Meta:
model = Inventory
load_instance = True
It's also to note that I'm using SQLAlchemy 1.4.3, if that helps out.
and in the main.py
import Database as db
db.Base.metadata.create_all(db.engine)
data_list = [{Part_No:123A, Shelf:1, Bin:5},
{Part_No:456B, Shelf:1, Bin:7},
{Part_No:789C, Shelf:2, Bin:1}]
with db.Session(db.engine, future=True) as session:
try:
session.add_all(data_list) #<--- FAILS HERE AND THROWS AN EXCEPTION
session.commit()
except Exception as e:
session.rollback()
print(f"Error! {e!r}")
raise
finally:
session.close()
Now what I've googled on this "Class 'builtins.dict' is not mapped", most of the solutions brings me to marshmallow-sqlalchemy package which I've tried, but I'm still getting the same error. So I've tried moving the Base.metadata.create_all(engine) from the Database.py into the main.py. I also tried implementing a init function in the Inventory class, and also calling the Super().init, which doesn't work
So what's going on?? Why is it failing and is there a better solution to this problem?
Try creating Inventory objects:
data_list = [
Inventory(Part_No='123A', Shelf=1, Bin=5),
Inventory(Part_No='456B', Shelf=1, Bin=7),
Inventory(Part_No='789C', Shelf=2, Bin=1)
]
I am having trouble to get username and password using slick in scala like basically similar to something like
var query = "SELECT * FROM \"default\".users as A " + "WHERE " + " A.username LIKE \'" + email + "\' " + "AND" + " A.password LIKE \'" + password + "\' ";
Here is my case class for schema
case class User(
id: Long = 0L,
username: String,
password: String,
author_id:Long,
created_on: DateTime,
updated_by: Long,
updated_on:Option[DateTime]
)
class UserTable(tag:Tag) extends Table[User](tag,"user"){
override def * = (id,username,password,author_id,created_on,updated_by,updated_on.?) <> (User.tupled,User.unapply)
def id = column[Long]("id",O.PrimaryKey,O.AutoInc)
def username = column[String]("username")
def password = column[String]("password")
def created_on = column[DateTime]("created_on")
def updated_on = column[DateTime]("dateupdated")
def author_id = column[Long]("author_id")
def updated_by = column[Long]("updated_by")
}
lazy val UserTable = TableQuery[UserTable]
Below is my Query
val users = Main.UserTable.filter(_.username == email)
.filter(_.password == password).take(1)
filter(_.username == email)
You probably meant === instead of ==.
That is part of the Slick DSL to build query expressions (and they could not call it == because Scala's equality comparator cannot be replaced).
Warning: Most operators mimic the plain Scala equivalents, but you have to use === instead of == for comparing two values for equality and =!= instead of != for inequality. This is necessary because these operators are already defined (with unsuitable types and semantics) on the base type Any, so they cannot be replaced by extension methods.
I have created a table below in SQL using the following:
CREATE TABLE [dbo].[Validation](
[RuleId] [int] IDENTITY(1,1) NOT NULL,
[AppId] [varchar](255) NOT NULL,
[Date] [date] NOT NULL,
[RuleName] [varchar](255) NOT NULL,
[Value] [nvarchar](4000) NOT NULL
)
NOTE the identity key (RuleId)
When inserting values into the table as below in SQL it works:
Note: Not inserting the Primary Key as is will autofill if table is empty and increment
INSERT INTO dbo.Validation VALUES ('TestApp','2020-05-15','MemoryUsageAnomaly','2300MB')
However when creating a temp table on databricks and executing the same query below running this query on PySpark as below:
%python
driver = <Driver>
url = "jdbc:sqlserver:<URL>"
database = "<db>"
table = "dbo.Validation"
user = "<user>"
password = "<pass>"
#import the data
remote_table = spark.read.format("jdbc")\
.option("driver", driver)\
.option("url", url)\
.option("database", database)\
.option("dbtable", table)\
.option("user", user)\
.option("password", password)\
.load()
remote_table.createOrReplaceTempView("YOUR_TEMP_VIEW_NAMES")
sqlcontext.sql("INSERT INTO YOUR_TEMP_VIEW_NAMES VALUES ('TestApp','2020-05-15','MemoryUsageAnomaly','2300MB')")
I get the error below:
AnalysisException: 'unknown requires that the data to be inserted have the same number of columns as the target table: target table has 5 column(s) but the inserted data has 4 column(s), including 0 partition column(s) having constant value(s).;'
Why does it work on SQL but not when passing the query through databricks? How can I insert through pyspark without getting this error?
The most straightforward solution here is use JDBC from a Scala cell. EG
%scala
import java.util.Properties
import java.sql.DriverManager
val jdbcUsername = dbutils.secrets.get(scope = "kv", key = "sqluser")
val jdbcPassword = dbutils.secrets.get(scope = "kv", key = "sqlpassword")
val driverClass = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
// Create the JDBC URL without passing in the user and password parameters.
val jdbcUrl = s"jdbc:sqlserver://xxxx.database.windows.net:1433;database=AdventureWorks;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;"
// Create a Properties() object to hold the parameters.
val connectionProperties = new Properties()
connectionProperties.put("user", s"${jdbcUsername}")
connectionProperties.put("password", s"${jdbcPassword}")
connectionProperties.setProperty("Driver", driverClass)
val connection = DriverManager.getConnection(jdbcUrl, jdbcUsername, jdbcPassword)
val stmt = connection.createStatement()
val sql = "INSERT INTO dbo.Validation VALUES ('TestApp','2020-05-15','MemoryUsageAnomaly','2300MB')"
stmt.execute(sql)
connection.close()
You could use pyodbc too, but the SQL Server ODBC drivers aren't installed by default, and the JDBC drivers are.
A Spark solution would be to create a view in SQL Server and insert against that. eg
create view Validation2 as
select AppId,Date,RuleName,Value
from Validation
then
tableName = "Validation2"
df = spark.read.jdbc(url=jdbcUrl, table=tableName, properties=connectionProperties)
df.createOrReplaceTempView(tableName)
sqlContext.sql("INSERT INTO Validation2 VALUES ('TestApp','2020-05-15','MemoryUsageAnomaly','2300MB')")
If you want to encapsulate the Scala and call it from another language (like Python), you can use a scala package cell.
eg
%scala
package example
import java.util.Properties
import java.sql.DriverManager
object JDBCFacade
{
def runStatement(url : String, sql : String, userName : String, password: String): Unit =
{
val connection = DriverManager.getConnection(url, userName, password)
val stmt = connection.createStatement()
try
{
stmt.execute(sql)
}
finally
{
connection.close()
}
}
}
and then you can call it like this:
jdbcUsername = dbutils.secrets.get(scope = "kv", key = "sqluser")
jdbcPassword = dbutils.secrets.get(scope = "kv", key = "sqlpassword")
jdbcUrl = "jdbc:sqlserver://xxxx.database.windows.net:1433;database=AdventureWorks;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;"
sql = "select 1 a into #foo from sys.objects"
sc._jvm.example.JDBCFacade.runStatement(jdbcUrl,sql, jdbcUsername, jdbcPassword)
I have the json field like blow which I want to store in database
{
id: 1
name: "test entity 1"
description: "a test entity for some guy's blog"
status: "passed"
web_url: "http://localhost:3000"
jobs: [{
id: "1"
name: "test1"
status: "passed"
},
{
id: "2"
name: "test2"
status: "passed"
},
{
id: "3"
name: "test3"
status: "failed"
}]
}
I proceed with one way like for creating table uses:
CREATE TABLE test3 (id INT PRIMARY KEY, name VARCHAR, description VARCHAR, status VARCHAR, web_url VARCHAR, jobs JSON[]);
and for Inserting data uses:
sqlStatement := `
INSERT INTO jobs (id, name, description, status, web_url, jobs)
VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT (id) DO UPDATE
SET status = $4
RETURNING id`
id := 0
err = database.Db.QueryRow(sqlStatement, y[i].ID, y[i].Name, y[i].Description, y[i].Status, y[i].WebURL, jobsdata).Scan(&id)
if err != nil {
panic(err)
}
But won't work, need help!!
Getting errors:
panic: sql: converting argument $6 type: unsupported type handler.Jobs, a slice of struct
What i want:
postgres=# SELECT * FROM test3;
id | name | description | status | web_url | jobs
------+------------------------------------------+--------+---------+----------------------------------------------------------+----------------------------------------------------------
1 | test entity 1 | a test entity for some guy's blog | passed | https://localhost:3000 | {id: "1",name: "test1", status: "passed"},{id: "2",name: "test2", status: "passed"},{id: "3",name: "test3", status: "failed"}
As the error indicates, you're trying to bind the sixth value from an unsupported data type, handler.Jobs. You haven't told us what this type is, but from the error, it's clear that it does not implement the driver.Valuer interface, so it won't work, because it has no way of knowing how to represent that value to the database.
You'll need to implement that interface, by adding a Value() method to the handler.Jobs type, or use a different data type.
sqlx has a type JSONText in github.com/jmoiron/sqlx/types that will do what you need.
Example