I'm struggling to create tables that belong to a schema in a SQL Server database, and ensuring that primary/foreign keys work correctly.
I'm looking for some examples of code to illustrate how this is done
The ingredients needed for this are __table_args__ and the use of the schema prefix on the ForeignKey
DBSession = sessionmaker(bind=engine)
session = DBSession()
from sqlalchemy import Column, Integer, String
from sqlalchemy.orm import relationship
Base = declarative_base()
class Table1(Base):
__tablename__ = 'table1'
__table_args__ = {"schema": 'my_schema'}
id = Column(Integer,primary_key = True)
col1 = Column(String(150))
col2 = Column(String(100))
reviews = relationship("Table2", cascade = "delete")
class Table2(Base):
__tablename__ = 'table2'
__table_args__ = {"schema": 'my_schema'}
id = Column(Integer,primary_key = True)
key = Column(Integer)
col2 = Column(String(100))
key = Column(Integer, ForeignKey("my_schema.table1.id"), index=True)
premise = relationship("Table1")
Base.metadata.create_all(bind=engine)
Related
I have this query:
SELECT * FROM "transaction"
WHERE type = {transaction_type} AND
from_member_id IN (SELECT member_id FROM dao_member WHERE org_id = "{org}") AND
to_member_id IN (SELECT member_id FROM dao_member WHERE org_id = "{org}")
Which I have written in peewee like this:
members = ORG_Member.select(ORG_Member.member).where(ORG_Member.org_id == org)
transactions = Transaction.select().where(
Transaction.type == transaction_type).where(
Transaction.from_member.in_(members)).where(
Transaction.to_member.in_(members)).sql()
The SQL shown is then;
SELECT * FROM "transaction" AS "t1"
WHERE ((("t1"."type" = ?)
AND ("t1"."from_member_id" IN
(SELECT "t2"."member_id" FROM "org_member" AS "t2" WHERE ("t2"."org_id" = ?) LIMIT ?)))
AND ("t1"."to_member_id" IN
(SELECT "t2"."member_id" FROM "org_member" AS "t2" WHERE ("t2"."org_id" = ?) LIMIT ?))),
[1, 'lala', 1, 'baba', 1])
Notice the limit! Where did this limit come from? I've tried setting limit explicitly to None, no luck.
I don't observe that. Probably you have omitted some important details.
from peewee import *
db = SqliteDatabase(':memory:')
class Member(Model):
org_id = IntegerField(default=0)
class Meta:
database = db
class Transaction(Model):
from_member = ForeignKeyField(Member, backref='from_tx')
to_member = ForeignKeyField(Member, backref='to_tx')
type = TextField()
class Meta:
database = db
members = Member.select().where(Member.org_id == 0)
txns = (Transaction.select()
.where(Transaction.type == 'tx-type')
.where(Transaction.from_member.in_(members))
.where(Transaction.to_member.in_(members)))
print(txns.sql())
This results in:
SELECT "t1"."id", ...
FROM "transaction" AS "t1"
WHERE "t1"."type" = ? AND
"t1"."from_member_id" IN (SELECT "t2"."id"
FROM "member" AS "t2"
WHERE ( "t2"."org_id" = ? )) AND
"t1"."to_member_id" IN (SELECT "t2"."id"
FROM "member" AS "t2"
WHERE ( "t2"."org_id" = ? ))
I don't get it. I'm trying to start a brand new table in MS SQL Server 2012 with the following:
In SQL Server:
TABLE [dbo].[Inventory](
[Index_No] [bigint] IDENTITY(1,1) NOT NULL,
[Part_No] [varchar(150)] NOT NULL,
[Shelf] [int] NOT NULL,
[Bin] [int] NOT NULL,
PRIMARY KEY CLUSTERED
(
[Index_No] ASC
)
UNIQUE NONCLUSTERED
(
[Part_No] ASC
)
GO
NOTE: This is a BRAND NEW TABLE! There is no data in it at all
Next, this is the Database.py file:
import pymssql
from sqlalchemy import create_engine, Table, MetaData, select, Column, Integer, Float, String, text,
func, desc, and_, or_, Date, insert
from marshmallow_sqlalchemy import SQLAlchemyAutoSchema
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
USERNAME = "name"
PSSWD = "none_of_your_business"
SERVERNAME = "MYSERVER"
INSTANCENAME = "\SQLSERVER2012"
DB = "Inventory"
engine = create_engine(f"mssql+pymssql://{USERNAME}:{PSSWD}#{SERVERNAME}{INSTANCENAME}/{DB}")
class Inventory(Base):
__tablename__ = "Inventory"
Index_No = Column('Index_No', Integer, primary_key=True, autoincrement=True)
Part_No = Column("Part_No", String, unique=True)
Shelf = Column("Shelf", Integer)
Bin = Column("Bin", Integer)
def __repr__(self):
return f'Drawing(Index_No={self.Index_No!r},Part_No={self.Part_No!r}, Shelf={self.Shelf!r}, ' \
f'Bin={self.Bin!r})'
class InventorySchema(SQLAlchemyAutoSchema):
class Meta:
model = Inventory
load_instance = True
It's also to note that I'm using SQLAlchemy 1.4.3, if that helps out.
and in the main.py
import Database as db
db.Base.metadata.create_all(db.engine)
data_list = [{Part_No:123A, Shelf:1, Bin:5},
{Part_No:456B, Shelf:1, Bin:7},
{Part_No:789C, Shelf:2, Bin:1}]
with db.Session(db.engine, future=True) as session:
try:
session.add_all(data_list) #<--- FAILS HERE AND THROWS AN EXCEPTION
session.commit()
except Exception as e:
session.rollback()
print(f"Error! {e!r}")
raise
finally:
session.close()
Now what I've googled on this "Class 'builtins.dict' is not mapped", most of the solutions brings me to marshmallow-sqlalchemy package which I've tried, but I'm still getting the same error. So I've tried moving the Base.metadata.create_all(engine) from the Database.py into the main.py. I also tried implementing a init function in the Inventory class, and also calling the Super().init, which doesn't work
So what's going on?? Why is it failing and is there a better solution to this problem?
Try creating Inventory objects:
data_list = [
Inventory(Part_No='123A', Shelf=1, Bin=5),
Inventory(Part_No='456B', Shelf=1, Bin=7),
Inventory(Part_No='789C', Shelf=2, Bin=1)
]
I have created a table below in SQL using the following:
CREATE TABLE [dbo].[Validation](
[RuleId] [int] IDENTITY(1,1) NOT NULL,
[AppId] [varchar](255) NOT NULL,
[Date] [date] NOT NULL,
[RuleName] [varchar](255) NOT NULL,
[Value] [nvarchar](4000) NOT NULL
)
NOTE the identity key (RuleId)
When inserting values into the table as below in SQL it works:
Note: Not inserting the Primary Key as is will autofill if table is empty and increment
INSERT INTO dbo.Validation VALUES ('TestApp','2020-05-15','MemoryUsageAnomaly','2300MB')
However when creating a temp table on databricks and executing the same query below running this query on PySpark as below:
%python
driver = <Driver>
url = "jdbc:sqlserver:<URL>"
database = "<db>"
table = "dbo.Validation"
user = "<user>"
password = "<pass>"
#import the data
remote_table = spark.read.format("jdbc")\
.option("driver", driver)\
.option("url", url)\
.option("database", database)\
.option("dbtable", table)\
.option("user", user)\
.option("password", password)\
.load()
remote_table.createOrReplaceTempView("YOUR_TEMP_VIEW_NAMES")
sqlcontext.sql("INSERT INTO YOUR_TEMP_VIEW_NAMES VALUES ('TestApp','2020-05-15','MemoryUsageAnomaly','2300MB')")
I get the error below:
AnalysisException: 'unknown requires that the data to be inserted have the same number of columns as the target table: target table has 5 column(s) but the inserted data has 4 column(s), including 0 partition column(s) having constant value(s).;'
Why does it work on SQL but not when passing the query through databricks? How can I insert through pyspark without getting this error?
The most straightforward solution here is use JDBC from a Scala cell. EG
%scala
import java.util.Properties
import java.sql.DriverManager
val jdbcUsername = dbutils.secrets.get(scope = "kv", key = "sqluser")
val jdbcPassword = dbutils.secrets.get(scope = "kv", key = "sqlpassword")
val driverClass = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
// Create the JDBC URL without passing in the user and password parameters.
val jdbcUrl = s"jdbc:sqlserver://xxxx.database.windows.net:1433;database=AdventureWorks;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;"
// Create a Properties() object to hold the parameters.
val connectionProperties = new Properties()
connectionProperties.put("user", s"${jdbcUsername}")
connectionProperties.put("password", s"${jdbcPassword}")
connectionProperties.setProperty("Driver", driverClass)
val connection = DriverManager.getConnection(jdbcUrl, jdbcUsername, jdbcPassword)
val stmt = connection.createStatement()
val sql = "INSERT INTO dbo.Validation VALUES ('TestApp','2020-05-15','MemoryUsageAnomaly','2300MB')"
stmt.execute(sql)
connection.close()
You could use pyodbc too, but the SQL Server ODBC drivers aren't installed by default, and the JDBC drivers are.
A Spark solution would be to create a view in SQL Server and insert against that. eg
create view Validation2 as
select AppId,Date,RuleName,Value
from Validation
then
tableName = "Validation2"
df = spark.read.jdbc(url=jdbcUrl, table=tableName, properties=connectionProperties)
df.createOrReplaceTempView(tableName)
sqlContext.sql("INSERT INTO Validation2 VALUES ('TestApp','2020-05-15','MemoryUsageAnomaly','2300MB')")
If you want to encapsulate the Scala and call it from another language (like Python), you can use a scala package cell.
eg
%scala
package example
import java.util.Properties
import java.sql.DriverManager
object JDBCFacade
{
def runStatement(url : String, sql : String, userName : String, password: String): Unit =
{
val connection = DriverManager.getConnection(url, userName, password)
val stmt = connection.createStatement()
try
{
stmt.execute(sql)
}
finally
{
connection.close()
}
}
}
and then you can call it like this:
jdbcUsername = dbutils.secrets.get(scope = "kv", key = "sqluser")
jdbcPassword = dbutils.secrets.get(scope = "kv", key = "sqlpassword")
jdbcUrl = "jdbc:sqlserver://xxxx.database.windows.net:1433;database=AdventureWorks;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;"
sql = "select 1 a into #foo from sys.objects"
sc._jvm.example.JDBCFacade.runStatement(jdbcUrl,sql, jdbcUsername, jdbcPassword)
I have a big table with a binary column for picture. I need to show contents of this table in a view and make it searchable. I have tried only selecting a subset of columns that are needed in this. However, the generated SQL always has all the columns of the table in the generated query.
public IQueryable<ApplicantDTO> GetApplicantQueryable()
{
return DataContext.Applicants
.Include(a => a.Nationality)
.Select(x => new ApplicantDTO
{
Id = x.Id,
BirthDate = x.BirthDate,
Gender = x.Gender,
FirstName = x.FirstName,
LastName = x.LastName,
OtherName = x.OtherName,
MobileNo = x.MobileNo,
Age = x.Age,
Nationality = x.Nationality.National,
Admitted = x.admitted,
Completed = x.Completed
})
.Where(a => a.Admitted == false && a.Completed == true)
.OrderBy(a => a.LastName)
.AsNoTracking();
}
But instead of just specifying the above rows, the generated SQL from profiler is
SELECT
[a].[Id], [a].[BirthDate], [a].[BirthPlace], [a].[CompleteDate],
[a].[Completed], [a].[ContentType], [a].[CreateDate],
[a].[Denomination], [a].[Disability], [a].[Email],
[a].[FirstName], [a].[Gender], [a].[HomeTown], [a].[LastName],
[a].[MarryStatus], [a].[MatureApp], [a].[MobileNo], [a].[NationalityID],
[a].[OtherName], [a].[Passport], [a].[Pin], [a].[PostalAddress],
[a].[Region], [a].[Religion], [a].[ResAddress], [a].[SerialNo],
[a].[Title], [a].[VoucherID], [a].[admitted], [a.Nationality].[National]
FROM
[Applicants] AS [a]
INNER JOIN
[Nationality] AS [a.Nationality] ON [a].[NationalityID] = [a.Nationality].[Id]
WHERE
([a].[admitted] = 0)
AND ([a].[Completed] = 1)
ORDER BY
[a].[LastName]
With all the underlying columns all included in the query.
I tried putting it in an anonymous type before casting it to the ApplicantDTO but still the same effect.
What's wrong?
I need to change some primary keys from non-clustered to clustered but I can't drop the constraint because it is referenced from other foreign keys.
How can I find the tables that reference a primary key in the parent table as part of a foreign relation without looping through all tables in the DB? I need to disable the constraints on those, change the PK and re-enable.
Update:
I do not want to use plain SQL to do this but SMO only.
Marc, I know about ForeignKeys by I need something like:
table.PrimaryKey.ForeignKeys (i.e. which tables are referencing my table's primary key)
I just want to avoid looping through all the tables in the database and check the ForeignKeys property on each and every one of them to see if any of them reference my table.(not scalable)
Ok I think I found it.
table.Columns[0].EnumForeignKeys()
or directly
table.EnumForeignKeys()
I was expecting a property instead of a function. I am pretty sure behind the scenes it does what cmsjr suggested.
Using SMO, you could do this:
using Microsoft.SqlServer.Management.Smo;
Server localServer = new Server("your server name");
Database dasecoDB = localServer.Databases["your database name"];
Table table = dasecoDB.Tables["your table name"];
foreach(ForeignKey fk in table.ForeignKeys)
{
Console.WriteLine("Foreign key {0} references table {1} and key {2}", fk.Name, fk.ReferencedTable, fk.ReferencedKey);
}
Marc
This query should work, and could be executed using Database.ExecuteWithResults
Select fk.Table_Name from
INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS C
INNER JOIN
INFORMATION_SCHEMA.TABLE_CONSTRAINTS FK
ON C.CONSTRAINT_NAME = FK.CONSTRAINT_NAME
INNER JOIN
INFORMATION_SCHEMA.TABLE_CONSTRAINTS PK
ON C.UNIQUE_CONSTRAINT_NAME = PK.CONSTRAINT_NAME
where PK.Table_Name = 'SomeTable'
e.g.
SqlConnection sqlConnection =
new SqlConnection(#"Integrated Security=SSPI; Data Source=SomeInstance");
Server server = new Server(serverConnection);
Database db = server.Databases["somedatabase"];
DataSet ds = db.ExecuteWithResults(thesqlabove);
You could use the INFORMATION_SCHEMA Views.
INFORMATION_SCHEMA.TABLE_CONSTRAINTS will give you the names of the primary keys on that table.
SELECT CONSTRAINT_NAME FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS WHERE TABLE_NAME = #TableName
Given the primary key names you can get the referential constraints that use those keys from INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS
And then the table names by querying INFORMATION_SCHEMA.CONSTRAINT_TABLE_USAGE
Not SMO as such, but given the above you should be able to put together a query that will list the constraints you need to disable.
It doesn't work for me.
Consider the following relations:
Table1 --> master table;
Table2 --> slave table;
Table2.Table1_ID is a foreign key of Table1.ID
Table1.EnumForeignKeys() return null.
Instead I tried with success the DependencyWalker object. The following code list all the tables which dipend from a given collection of tables.
DependencyWalker w = new DependencyWalker(db.Parent);
DependencyTree tree = w.DiscoverDependencies(urns,false);
DependencyCollection depends = w.WalkDependencies(tree);
foreach (DependencyCollectionNode dcn in depends)
{
if (dcn.Urn.Type == "Table")
{
dcn.Urn.GetNameForType("Table");
Console.WriteLine(dcn.Urn.GetNameForType("Table"));
}
}
where "urns" is a collection of table.Urn.
You will have to travel through dependency tree.
Following is the script which use the SMO to generate Create table and insert script.
**
**ServerConnection conn = new ServerConnection( GetConnection() );
Server server = new Server( conn );
Database db = server.Databases[ mDestinationDatabase ];
// Create database script
StringBuilder dbScript = new StringBuilder();
ScriptingOptions dbCreateOptions = new ScriptingOptions();
dbCreateOptions.DriAll = true;
dbCreateOptions.NoCollation = true;
StringCollection coll = db.Script( dbCreateOptions );
foreach( string str in coll )
{
dbScript.Append( str );
dbScript.Append( Environment.NewLine );
}
sqlInsertCommands = dbScript.ToString();
// Create dependency tree
DependencyWalker w = new DependencyWalker(db.Parent);
UrnCollection urnCollection = new UrnCollection();
DataTable table = db.EnumObjects( DatabaseObjectTypes.Table );
string tableName = string.Empty;
foreach( DataRow row in table.Rows )
{
urnCollection.Add( new Urn( ( string )row[ "Urn" ] ) );
}
DependencyTree tree = w.DiscoverDependencies( urnCollection, true );
DependencyCollection depends = w.WalkDependencies(tree);
// walk through the dependency tree and for each table generate create and insert scripts
foreach (DependencyCollectionNode dcn in depends)
{
if (dcn.Urn.Type == "Table")
{
tableName = dcn.Urn.GetNameForType( "Table" );
DataTable dataTableWithData = GetTableWithData( tableName);
ArrayList columnList = new ArrayList();
foreach(DataColumn dataColumn in dataTableWithData.Columns)
{
columnList.Add( dataColumn.ColumnName );
}
sqlInsertCommands = sqlInsertCommands + Environment.NewLine + Environment.NewLine
+ GetCreateTableScript(tableName )
+ Environment.NewLine + Environment.NewLine
+ BuildInsertSQL( columnList, dataTableWithData, tableName );
}
}**
**