I'm trying to get the names of the columns from a table in an Azure SQL database using a PyPika SQL query, but keep running into trouble. Here's the code I'm using to generate the query:
def dbView(table):
infoSchema = ppk.Table("INFORMATION_SCHEMA.COLUMNS")
return ppk.MSSQLQuery.from_(infoSchema).select(infoSchema.COLUMN_NAME).where(infoSchema.TABLE_NAME == table)
I created another function that uses the PyODBC library to get the SQL from the query, execute it against the database, and return all the rows:
def getData(query: ppk.Query):
'''
Execute a query against the Azure db and return
every row in the results list.
'''
print("QUERY: ", query.get_sql())
conn = getConnection()
with conn.cursor() as cursor:
cursor.execute(query.get_sql())
return cursor.fetchall()
I know the getData() function works because when I pass it a simple select query, everything works correctly. However, when I try to use the query generated by pypika above, I get the following error:
pyodbc.ProgrammingError: ('42S02', "[42S02] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Invalid object name 'INFORMATION_SCHEMA.COLUMNS'. (208) (SQLExecDirectW)")
To make sure this wasn't just some kind of permissions error, I wrote the following query by hand and executed it using the getData() function and it worked just fine:
SELECT COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = 'Validation'
I also printed out the query that pypika generated to the console. The only difference appears to be the addition of some double quotes:
SELECT "COLUMN_NAME" FROM "INFORMATION_SCHEMA.COLUMNS" WHERE "TABLE_NAME"='Validation'
What am I doing wrong? For some reason, this error appears to be limited to specifically the information schema table, because I have used similar queries several other times in my code without issue. I know I can just use the query I wrote by hand, but the point of using PyPika was to make all my SQL queries more readable and reusable - it'd be nice to understand why it doesn't work in this very specific situation.
Thanks!
It apparently has an API to schema-qualify tables.
from pypika import Table, Query, Schema
views = Schema('views')
q = Query.from_(views.customers).select(customers.id, customers.phone)
https://pypika.readthedocs.io/en/latest/2_tutorial.html#tables-columns-schemas-and-databases
I have a very simple EF query below that never returns. The call to Any() just hangs.
if (!_context.SalesRoundRobinAssignments.Any()) return salesPeopleIds.First();
When I look at the sql profiler I don't see that query being executed however I do see the following.
SQL: BatchStarting select serverproperty('EngineEdition')
SQL: BatchCompleted select serverproperty('EngineEdition')
I've also tried adding this before the query but nothing gets written to the output window aside from the same serverproperty query shown above.
_context.Database.Log = s => System.Diagnostics.Debug.WriteLine(s);
Any ideas what else I can look at to figure out what's going on?
Did you check the connection string from your web.config/app.config that your Entity Framework Context uses to connect with the database? The query can be executing in another database and that's why you don't see nothing with the SQL profile.
Try to debug and see in execution time, which connection string your application is passing to string connectionString parameter:
public YourDBContext(string connectionString): base(connectionString)
{
}
I have a large set of pre-existing sql select statements.
From a stored procedure on [Server_A], I would like to execute each of these statements on multiple different SQL Servers & Databases (the list is stored in a local table on [Server_A] , and return the results into a table on [Server_A].
However, I do not want to have to use fully qualified table names in my sql statements. I want to execute "select * from users", not "select * from ServerName.DatabaseName.SchemaName.Users"
I've investigated using Openrowset, but I am unable to find any examples where both the Server name and DatabaseName can be specified as an attribute of the connection, rather than physically embedded within the actual SQL statement.
Is Openrowset capable of this? Is there an alternate way of doing this (from within a stored procedure, as opposed to resorting to Powershell or some other very different approach?)
The inevitable "Why do I want to do this?"
You can do it (specify the server and database in the connection
attributes and then use entirely generic sql across all databases) in
virtually every other language that accesses SQL Server.
Changing all my pre-existing complex SQL to be fully qualified is a
huge PITA (besides, you simply shouldn't have to do this)
This can be done quite easily via SQLCLR. If the result set is to be dynamic then it needs to be a Stored Procedure instead of a TVF.
Assuming you are doing a Stored Procedure, you would just:
Pass in #ServerName, #DatabaseName, #SQL
Create a SqlConnection with a Connection String of: String.Concat("Server=", ServerName.Value, "; Database=", DatabaseName.Value, "; Trusted_Connection=yes; Enlist=false;") or use ConnectionStringBuilder
Create a SqlCommand for that SqlConnection and using SQL.Value.
Enable Impersonation via SqlContext.WindowsIdentity.Impersonate();
_Connection.Open();
undo Impersonation -- was only needed to establish the connection
_Reader = Command.ExecuteReader();
SqlContext.Pipe.Send(_Reader);
Dispose of Reader, Command, Connection, and ImpersonationContext in finally clause
This approach is less of a security issue than enabling Ad Hoc Distributed Query access as it is more insulated and controllable. It also does not allow for a SQL Server login to get elevated permissions since a SQL Server login will get an error when the code executes the Impersonate() method.
Also, this approach allows for multiple result sets to be returned, something that OPENROWSET doesn't allow for:
Although the query might return multiple result sets, OPENROWSET returns only the first one.
UPDATE
Modified pseudo-code based on comments on this answer:
Pass in #QueryID
Create a SqlConnection (_MetaDataConnection) with a Connection String of: Context Connection = true;
Query _MetaDataConnection to get ServerName, DatabaseName, and Query based on QueryID.Value via SqlDataReader
Create another SqlConnection (_QueryConnection) with a Connection String of: String.Concat("Server=", _Reader["ServerName"].Value, "; Database=", _Reader["DatabaseName"].Value, "; Trusted_Connection=yes; Enlist=false;") or use ConnectionStringBuilder
Create a SqlCommand (_QueryCommand) for _QueryConnection using _Reader["SQL"].Value.
Using _MetaDataConnection, query to get parameter names and values based on QueryID.Value
Cycle through SqlDataReader to create SqlParameters and add to _QueryCommand
_MetaDataConnection.Close();
Enable Impersonation via SqlContext.WindowsIdentity.Impersonate();
_QueryConnection.Open();
undo Impersonation -- was only needed to establish the connection
_Reader = _QueryCommand.ExecuteReader();
SqlContext.Pipe.Send(_Reader);
Dispose of Readers, Commands, Connections, and ImpersonationContext in finally clause
If you want to execute a sql statement on every database in a instance you can use (the unsupported, unofficial, but widely used) exec sp_MSforeachdb like this:
EXEC sp_Msforeachdb 'use [?]; select * from users'
This will be the equivalent of going through every database through a
use db...
go
select * from users
This is an interesting problem because I googled for many, many hours, and found several people trying to do exactly the same thing as asked in the question.
Most common responses:
Why would you want to do that?
You can not do that, you must fully qualify your objects names
Luckily, I stumbled upon the answer, and it is brutally simple. I think part of the problem is, there are so many variations of it with different providers & connection strings, and there are so many things that could go wrong, and when one does, the error message is often not terribly enlightening.
Regardless, here's how you do it:
If you are using static SQL:
select * from OPENROWSET('SQLNCLI','Server=ServerName[\InstanceName];Database=AdventureWorks2012;Trusted_Connection=yes','select top 10 * from HumanResources.Department')
If you are using Dynamic SQL - since OPENROWSET does not accept variables as arguments, you can use an approach like this (just as a contrived example):
declare #sql nvarchar(4000) = N'select * from OPENROWSET(''SQLNCLI'',''Server=Server=ServerName[\InstanceName];Database=AdventureWorks2012;Trusted_Connection=yes'',''#zzz'')'
set #sql = replace(#sql,'#zzz','select top 10 * from HumanResources.Department')
EXEC sp_executesql #sql
Noteworthy: In case you think it would be nice to wrap this syntax up in a nice Table Valued function that accepts #ServerName, #DatabaseName, #SQL - you cannot, as TVF's resultset columns must be determinate at compile time.
Relevant reading:
http://blogs.technet.com/b/wardpond/archive/2005/08/01/the-openrowset-trick-accessing-stored-procedure-output-in-a-select-statement.aspx
http://blogs.technet.com/b/wardpond/archive/2009/03/20/database-programming-the-openrowset-trick-revisited.aspx
Conclusion:
OPENROWSET is the only way that you can 100% avoid at least some full-qualification of object names; even with EXEC AT you still have to prefix objects with the database name.
Extra tip: The prevalent opinion seems to be that OPENROWSET shouldn't be used "because it is a security risk" (without any details on the risk). My understanding is that the risk is only if you are using SQL Server Authentication, further details here:
https://technet.microsoft.com/en-us/library/ms187873%28v=sql.90%29.aspx?f=255&MSPPError=-2147217396
When connecting to another data source, SQL Server impersonates the login appropriately for Windows authenticated logins; however, SQL Server cannot impersonate SQL Server authenticated logins. Therefore, for SQL Server authenticated logins, SQL Server can access another data source, such as files, nonrelational data sources like Active Directory, by using the security context of the Windows account under which the SQL Server service is running. Doing this can potentially give such logins access to another data source for which they do not have permissions, but the account under which the SQL Server service is running does have permissions. This possibility should be considered when you are using SQL Server authenticated logins.
I have a program that was written in VB6 and we are trying to convert to VB.Net. When calling stored procedures in Microsoft SQL 2012 and in Oracle using this .Open() method, Microsoft SQL executes fine while Oracle does not. Here is a snippet of code:
Private moRs As ADORecordSetHelper = New ADORecordSetHelper("")
Private moCmd As DbCommand 'where moCmd has the procedure call and parameters
moRs.Open(moCmd)
Here is the definition of .Open()
Public Sub Open(command As System.Data.Common.DbCommand)
Member of UpgradeHelpers.DB.ADO.ADORecordSetHelper
This all works for Microsoft SQL just not with Oracle. I found execute moCmd.ExecuteNonQuery() will work execute the stored procedure, but I need to get the recordset in certain instances as a result.
Is there a way to use moRs.Open(moCmd) for both Microsoft and Oracle? If not, how could I accomplish the same task of getting back the recordsets?
UPDATE: I have figured out that the .Open() method uses Execute Reader which is working fine for the Microsoft SQL that strictly returns a recordset. The Oracle stored procedure is inserting rows and returning values from the procedure. Doing an Execute NonQuery on the same object (moCmd) executes just fine, but all I am able to get from this method is the number of rows affected and not the other information that I need. Thoughts on why the Execture Reader doesn't work? My search says that it should work for all types of SQL; insert, delete, update, etc.
I've been trying to retrieve constraint data for all tables using an SQL query over JDBC.
My test database has only 3 tables.
If I execute the query interactively using MS SQL Server Management Studio, I get all the results that I expect (ie. 3 rows - there's a primary key in each of 3 tables).
if I use the JDBC method to specifically retrieve primary keys (as below) then I also correctly get 3 results:
ResultSet rs = dbmd.getPrimaryKeys(jdbcCatalog, jdbcSchema, jdbcTableName);
If I use the exact same SQL statement (that I used interactively and got 3 results back) as a query over JDBC (using executeQuery() shown below) then I only get 1 result instead of the expected 3.
String query =
"select PK.CONSTRAINT_NAME, PK.TABLE_SCHEMA, PK.TABLE_NAME " +
"from information_schema.TABLE_CONSTRAINTS PK";
ResultSet rs = null;
try {
Statement stmt = con.createStatement();
rs = stmt.executeQuery(query);
}catch (Exception exception) {
// Exception handler code
}
while (rs.next()){
// Only executes once.
}
I would be very grateful if someone could explain why the SQL query over JDBC is performing differently to the exact same SQL query performed interactively. Could it be a security/ownership issue? (although the JDBC call getPrimaryKeys() doesn't suffer this)
Thanks.
I don't see where you're setting your database context, but I suspect that that's the issue. As a test, you can change your statement to "select db_name()" and see what it returns. If it's not the database that you think that you should be in, that's your issue.