I'm using SQLAlchemy UNION, ORDER_BY is not working, it throwing this
error
Can't resolve label reference for ORDER BY/ GROUP BY/ DISTINCT etc. Textual SQL expression 'LastName' should be explictly declared as text('LastName')
query1 = db.query(Employee.LastName).filter(Employee.Age == 30)
query2 = db.query(Employee.LastName).filter(Employee.Salary > 25000)
result = query1.union(query2).order_by('LastName').all()
Then I tried the below line too, but it didn't work, its throwing this error
(pyodbc.ProgrammingError)('42S22',"[42S22][Microsoft][ODBC Driver 17 for SQL Server][SQL Server] invalid column name 'LastName'. [SQL Server] Statement(s) could not be prepared. (8180)
result = query1.union(query2).order_by(text('LastName').all()
Related
Background/Problem
According to Microsoft docs, the syntax for a CONTAINSTABLE query uses a CONTAINS search condition which as far as I can tell, must be entirely enclosed in single quotes. The problem occurs when I try to parameterize the query in pyodbc. Any form of quotation that is required for specifying the condition part of the query seems to cause pyodbc/the odbc driver to ignore the parameter markers, resulting in the infamous "The SQL contains x parameter markers, but >x parameters were supplied' error.
What I'm (Trying) to do
I get user search input that I break into a list of words and build a dynamic SQL CONTAINS search condition that puts in parameter markers instead of the actual words that is finally inserted into the rest of the SQL statement string in my cursor.execute() statement.
Ex: User searches for "Backflow prevention device"
Used this answer as the general way to solve the problem of putting multiple words in a fulltext CONTAINS/CONTAINSTABLE query. My dynamic CONTAINSTABLE condition is generated as ? AND ? AND ? which is then inserted into my query (which actually contains this and another fulltext query combined with UNION that uses the NEAR keyword between words instead of AND) via .format().
My Code
def parameterize(search_string, kw):
expanded = search_string.split(maxsplit=14)
sql = ""
for word in expanded:
sql = "{sql}{cond_keyword} ? ".format(sql=sql,
cond_keyword=kw if sql and len(expanded) > 1 else "",
word=word
)
sql = "{sql}".format(sql=sql.lstrip().rstrip())
qargs = tuple(expanded)
return sql, qargs
def ka_search(search_str):
and_cond = parameterize(search_str, "AND")
near_cond = parameterize(search_str, "NEAR")
near_params = near_cond[0]
and_params = and_cond[0]
# build a final tuple of all parameters in order of the individual sub-queries that are union-ed together
args1 = list(and_cond[1])
args2 = [search_str]
args3 = list(near_cond[1])
args = args1 + args2 + args3 + args2
try:
cursor = db_connect()
cursor.execute("""
SELECT Title, CONCAT(RTRIM(LEFT(Bodytext,150)), '...') as BodyText, Keywords, RecId,
rank, CONCAT('/kb/', RecId) AS Link
FROM KnowledgeArticle AS FT_TBL INNER JOIN
CONTAINSTABLE(KnowledgeArticle, (Title, keywords), {and_params}) AS KEY_TBL
ON FT_TBL.RecID = KEY_TBL.[KEY]
WHERE VisibleToCustomerPortal = 'true' AND
Status = 'Published'
UNION
SELECT Title, CONCAT(RTRIM(LEFT(Bodytext,150)), '...') as BodyText, Keywords, RecId,
rank, CONCAT('/kb/', RecId) AS Link
FROM KnowledgeArticle AS FT_TBL INNER JOIN
FREETEXTTABLE(KnowledgeArticle, Title, ?) AS KEY_TBL
ON FT_TBL.RecID = KEY_TBL.[KEY]
WHERE VisibleToCustomerPortal = 'true' AND
Status = 'Published'
UNION
SELECT Title, CONCAT(RTRIM(LEFT(Bodytext,150)), '...') as BodyText, Keywords, RecId,
rank, CONCAT('/kb/', RecId) AS Link
FROM KnowledgeArticle AS FT_TBL INNER JOIN
CONTAINSTABLE(KnowledgeArticle, (Title, keywords), {near_params}) AS KEY_TBL
ON FT_TBL.RecID = KEY_TBL.[KEY]
WHERE VisibleToCustomerPortal = 'true' AND
Status = 'Published'
SELECT Title, CONCAT(RTRIM(LEFT(Bodytext,150)), '...') as BodyText, Keywords, RecId,
rank, CONCAT('/kb/', RecId) AS Link
FROM KnowledgeArticle AS FT_TBL INNER JOIN
FREETEXTTABLE(KnowledgeArticle, Bodytext, ?) AS KEY_TBL
ON FT_TBL.RecID = KEY_TBL.[KEY]
WHERE VisibleToCustomerPortal = 'true' AND
Status = 'Published'
ORDER BY rank DESC
""".format(and_params=and_params, near_params=near_params), args)
The above code manages to pass the search terms into the parameters and the correct number of parameter markers is recognized, but I hit a SQL error when the statement executes:
"('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near the keyword 'AND'. (156) (SQLExecDirectW); [42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near 'NEAR'. (102); [42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Statement(s) could not be prepared. (8180)")"
The query does work as expected when plugging in the search string in the FREETEXTTABLE sub-queries and the '"[word] AND [word] AND ...' and '"[word] NEAR [word] NEAR ...' conditions directly and running them in SSMS.
It seems like I have mutually exclusive options when trying to make this work as a dynamic T-SQL parameterized query using pyodbc. I attempted to find an answer on here but none of the parameterized query/pyodbc/full text T-SQL questions seem to address my problem. Thank you in advance for the help!
Additional Info
SQL Server 2016
Python 3.6
Running on Windows
I am running into the error when running a python script pushing data from PostgreSQL to MSSQL.
2020/09/17 09:55:26:mssql:ERROR:Something went very wrong inserting data into mssql - bailing
Traceback (most recent call last):
File "./my_file.py", line 40, in importToMSSQLReal
dataToInsert)
pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near the keyword 'key'. (156) (SQLExecute)")
Here is the line of code:
query = """SELECT vi.asset_id ,vi.vulnerability_id,a.last_assessed_for_vulnerabilities as "last_detected",vi.status,vi.proof::varchar(10485760),vi.key::varchar(10485760),vi.service,vi.port,vi.protocol
FROM public.fact_asset_vulnerability_instance vi
INNER JOIN public.dim_asset a
ON vi.asset_id = a.asset_id"""
Insert
def importToMSSQLReal(mssql,mssqlstring,dataToInsert):
mssql = pyodbc.connect(mssqlstring)
log = logging.getLogger('mssql')
curs = mssql.cursor()
curs.fast_executemany = True
try:
curs.executemany(
"INSERT INTO {table} (asset_id,vulnerability_id,last_detected,status,proof,key) VALUES (?,?,?,?,?,?)".format(table=mscfg['table']),
dataToInsert)
mssql.commit()
except:
print("bad")
log.exception("Something went very wrong interting data into mssql - bailing")
exit(2)
curs.close()
mssql.close()
del mssql
Key is a reserved keyword in SQL Server. You have to make sure it is between brackets:
SELECT 1 as [key]
I'm trying to make a simple select on my table mapped with sqlalchemy but I can't get it to match the exact table name.
As I noticed, the output of this:
class Users(base):
__tablename__ = "users"
[...]
was
[Microsoft][SQL Server Native Client 11.0][SQL Server]Invalid object name 'users'
And I tried to fix the error by explicitly writing the database I'm refering to
class Users(base):
__tablename__ = "[homework-3-cc-database].users"
[...]
giving the output:
[Microsoft][SQL Server Native Client 11.0][SQL Server]Incorrect syntax near ']'.
I noticed that it inserted some square brackets, unwanted. Here is their SQL statement:
[SQL: SELECT TOP 1 [[homework-3-cc-database].users].user_id AS [[homework-3-cc-database].users_user_id]
FROM [[homework-3-cc-database].users]]
This statement, ran from DataGrip works just fine:
SELECT TOP 1 [homework-3-cc-schema].users.user_id
FROM [homework-3-cc-schema].users
Do you have any suggestions regarding how should I fix this?
I had to specify the schema name in the following way:
__table_args__ = {"schema": "homework-3-cc-schema"}
I am trying to run the following code
DB::table('gbb_intimation_claim_max_serial')->where('id', 1)->increment('intimation_max_serial');
$intimation_max_serial = DB::table('gbb_intimation_claim_max_serial')->where('id', 1)->value('intimation_max_serial');
DB::table('claims')->where('id', $claim->id)->update(['intimation_number' => $intimation_max_serial]);
But weirdly, I am getting this error on laravel
SQLSTATE[01003]: [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Warning: Null value is eliminated by an aggregate or other SET operation. (SQL: update [claims] set [intimation_number] = 1763 where [id] = 7172)
I have tried
SET ANSI_WARNINGS OFF;
But this didn't help
You do not normally get that error on an update statement, I suspect it is a carryover from your first query where it looks like your are returning the max claim serial, if you look at your data you will find one of the claim serial values is null and when you perform an aggregate function on a range of values that include a null that warning is thrown.
All i need is a simple query with GROUP BY CASE like
SELECT
COUNT([Personnel].[PERSONNEL_ID]) AS value,
CASE
WHEN ([Personnel].[T_FIRSTNAME] = 'John') THEN 'John'
ELSE 'somethingelse'
END AS somelabel
FROM [Personnel]
GROUP BY
CASE
WHEN ([Personnel].[T_FIRSTNAME] = 'John') THEN 'John'
ELSE 'somethingelse'
END
in Sqlalchemy my query looks like this:
case_groups = case(
[(HrPersonnel.T_FIRSTNAME=='John', 'John')],
else_='somethingelse'
)
volunteers = session_mssql.query(
func.count(HrPersonnel.S_HR_PERSONNEL_ID).label('value'),
case_groups.label('somelabel'))\
.group_by(case_groups)\
.all()
mssql server executes the query without any problem (engine.connect().execute(text_query))
but Sqlalchemy ORM query gives me an error.
File
"c:\projects\sa_test\eggs\sqlalchemy-0.9.2-py2.7.egg\sqlalchemy\engine\default.py",
line 425, in do_execute cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (ProgrammingError) ('42000', "[42000]
[Microsoft][ODBC SQL Server Driver][SQL Server] Column
'Personnel.T_FIRSTNAME' is invalid in the select list because it is
not contained in either an aggregate function or the GROUP BY clause.
(8120) (SQLExecDirectW); [42000] [Microsoft][ODBC SQL Server
Driver][SQL Server]Statement(s) could not be prepared. (8180)")
u'SELECT count([Personnel].[PERSONNEL_ID]) AS value, CASE WHEN
([Personnel].[T_FIRSTNAME] = ?) THEN ? ELSE ? END AS somelabel \nFROM
[Personnel] GROUP BY CASE WHEN ([Personnel].[T_FIRSTNAME] = ?) THEN ?
ELSE ? END' ('John', 'John', 'somethingelse', 'John', 'John',
'somethingelse')
Why 'Personnel.T_FIRSTNAME' is invalid in the select list?
any suggestions are welcome.
Thanks,
I don't have an answer per se (although future readers may want to check this thread on the SQLAlchemy Google Group), but I've had a similar issue recently and found a workaround.
Instead of executing the query directly, convert it into a string and execute that. For example:
session.execute(
str(query.selectable.compile(compile_kwargs={'literal_binds': True}))
).fetchall()