I am trying to append records from a dataframe in R to an established SQL data table using the odbc::dbWriteTable() function. This is a function I use for many workflows to append records to various database tables.
Specifically:
odbc::dbWriteTable(connection, DBI::SQL(glue("{database}.{schema}.{table}")), value = dataframe, append = TRUE)
The dataframe and the target SQL table share the same column names and variable types.
However, when I attempt to run the function and append the data records, I receive the following error:
Error in result_insert_dataframe(rs#ptr, values, batch_rows) :
nanodbc/nanodbc.cpp:####: ######: [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Invalid column name 'row_names'. [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Statement(s) could not be prepared.
The dataframe contains no row names. Why is the column name "row_names" being generated, and is there a way to ensure this column name is not generated? Many thanks in advance for any suggestions!
For anyone having similar issues, the answer was very simple. I just needed to add
row.names=FALSE
to the import function:
odbc::dbWriteTable(connection, DBI::SQL(glue("{database}.{schema}.{table}")), value = dataframe, append = TRUE, row.names=FALSE)
Related
Firstly, I have seen many answers which is specific to the Invalid Object Name error working with SQL Server, but None of them seem to solve my problem. I don't have much idea on SQL Server dialect, but here is my current setup required on the project.
SQL Server 2017
SQLAlchemy (pyodbc+mssql)
Python 3.9
I'm trying to insert a database row, using the SQLAlchemy ORM, but it fails to resolve the schema and table, giving me the error of type
(pyodbc.ProgrammingError) ('42S02', "[42S02] [Microsoft][ODBC Driver
17 for SQL Server][SQL Server]Invalid object name 'agent.dbo.tasks'.
(208) (SQLExecDirectW); [42S02] [Microsoft][ODBC Driver 17 for SQL
Server][SQL Server]Statement(s) could not be prepared. (8180)")
I'm creating a session with the following code.
engine = create_engine(connection_string, pool_size=10, max_overflow=5,
pool_use_lifo=False, pool_pre_ping=True, echo=True, echo_pool=True)
db_session_manager = sessionmaker()
db_session_manager.configure(bind=engine)
session = db_session_manager()
I have a task object defined like
class Task(BaseModel):
__tablename__ = "tasks"
__table_args__ = {"schema": "agent.dbo"}
# .. field defs
I'm trying to fill the object fields and then do the insert like the usual
task = Task()
task.tid = 1
...
session.add(task)
session.commit()
But this fails, with the error mentioned before. I tried to execute a direct query like
session.execute("SELECT * FROM agent.dbo.tasks")
And it returned a result set.
The connection string is a URL object, which prints like
mssql+pyodbc://task-user:******#CRD4E0050L/agent?driver=ODBC+Driver+17+for+SQL+Server
I tried to use the SQL Server Management Studio to insert manually and check, there it shown me a sql dialect with [] as field separators
like
INSERT INTO [dbo].[tasks]
([tid]..
, but SQLAlchemy on echo did not show that, instead it used the one I see in MySQL like
INSERT INTO agent.dbo.tasks (onbase_handle_id,..
What is that I'm doing wrong ? I thought SQLAlchemy if configured with a supported dialect, should work fine (I use it on MySQL quite well). Am I missing any configuration ? Any help appreciated.
I configure a Heterogenous Service from Oracle to access SQL Server using the ODBC Drive from Microsoft
It works, but some query in specific table return the right message for example:
ORA-28500: connection from ORACLE to a non-Oracle system returned this message:
[Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Attempt to access a column 'UtilizaMetrica_DescontoComerci'. {42S22,NativeErr = 207}[Microsoft][ODBC Driver 11 for SQL Server][SQL Server]
The right column 'UtilizaMetrica_DescontoComercial' has 32 characters, but truncate in the return message to 30 characters
It seems that OHS has a limitation on the length of a column name (30 characters).
Workaround is to shorten the name to an acceptable length by defining a shorter alias for that column or using a view to do the same thing.
I am writing a dataframe in R to Microsoft SQL server table and seeing below error. I have required permissions to insert records to this table, dbo.temp2 but it is failing with this error. The number of columns in Dataframe and Table are matching.
dbWriteTable2(connDBI,"dbo.temp2",test.df1,append=TRUE,overwrite=FALSE,row.names=FALSE,add.id=TRUE)
Error: <SQL> 'SELECT id FROM dbo.temp2 ORDER BY id DESC LIMIT 1'
nanodbc/nanodbc.cpp:1587: 42000: [Microsoft][ODBC SQL Server Driver][SQL Server]Incorrect syntax near 'LIMIT'
Also tried with function dbWriteTable from DBI package and it is failing with Create table error even though am using append=TRUE option.
I could able to resolve this issue with some online research and use of RMySQL package.
Below is the line of R code which fixed this and now am able to write the Dataframe into SQL Database table.
RMySQL::dbWriteTable(connDBI,name = "temp2"
,value=test.df1,append=TRUE,row.names=FALSE,overwrite=FALSE)
I'm working on a Shiny app for updating entries maintained in a SQLServer2008 remote DB. I've been connecting to the DB using RODBC and I'm attempting to use parameterized queries through RODBCext to support mass updates of information.
I'm able to get the parameterized queries to work from my Windows 7, RStudio running R 3.2.3, but for some reasons when I try to run the same code from the linux machine running the same version of R and connecting with the same version of the driver, I get the following error:
Error in sqlExecute(Connection, data = dat) :
42000 402 [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]The data types char and text are incompatible in the equal to operator.
42000 8180 [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Statement(s) could not be prepared.
[RODBCext] Error: SQLExecute failed
In addition: Warning messages:
1: In sqlExecute(Connection, data = dat) :
42000 402 [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]The data types char and text are incompatible in the equal to operator.
2: In sqlExecute(Connection, data = dat) :
42000 8180 [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Statement(s) could not be prepared.
Here is the simple example code that works properly on my windows machine, but not on the linux machine (I removed the connection string information):
library(RODBCext)
Connection <- odbcDriverConnect(paste('Driver=ODBC Driver 13 for SQL Server',
'Server=<Server IP>', 'Port=<Port>', 'Database=<Database>', 'UID = <UserID>',
'PWD=<Password>', sep = ';'))
dat <- data.frame(Node_ID = "999", NodeGUID = "AF213171-201B-489B-B648-F7D289B735B1")
query <- "UPDATE dbo.Nodes SET Node_ID = ? WHERE NodeGUID = ?"
sqlPrepare(Connection, query)
sqlExecute(Connection, data = dat)
In this example, the dataframe is created with the columns as factors. I've tried explicitly casting the columns as characters first, as this seemed to work for the users having trouble with dates, but that still results in the same SQL error. I've also tried casting the Node_ID as numeric to match the SQL table, and I get the same error. The columns in the Nodes table in SQL are defined as:
NodeGUID (PK, char(36), not null)
Node_ID (int, null)
I've tried combining the sqlPrepare and sqlExecute calls by supplying the query argument for sqlExecute, and from what I understand that's a trivial difference and it results in the same error.
I suspect there must be a difference in the drivers and how they implement whatever SQL calls sqlExecute() makes. I also suspect sqlExecute() must handle the data types, as my results don't change regardless of the column types.
Thank you for any help you can provide!
Thanks to everyone who took a look at my question.
One of the SQL Server folks at my job was able to solve the issue. They suggested explicitly casting the arguments in the SQL query written for sqlExecute(). Here's the code that works, note I know the GUID will always be 36 characters and I'm confident the rest of the arguments I use this query for will be less than 1000 when converted to strings:
my_query <- "UPDATE dbo.Nodes SET Node_ID = CAST(? As varchar(1000)) WHERE NodeGUID = CAST(? As varchar(36))"
sqlExecute(Connection, data = dat, query = my_query)
I'm guessing the driver for Windows is somehow handling the casting from text to varchar, but the linux driver does not.
I hope this helps others working with RODBCext. Thanks to Mateusz Zoltak and the team of contributors for a great package!
I have a package in my ETL that loops through 4 different databases, each with a copy of the same table. I found out that one of the four tables has one extra column named MTFvalue_Permit thus giving me the below error. My fact and staging tables, however, do include this column.
I was hoping ssis would just insert a null value if the column didn't exist. How can I add a null value into my Data Warehouse tables for the other three tables in which this column doesn't exist?
Error:
[Source DB IBS [1]] Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80040E14.
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80040E14 Description: "Statement(s) could not be prepared.".
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80040E14 Description: "Invalid column name 'MTFvalue_Permit'.".
Edit: I'm thinking about using a case statement like this for the query but I'm getting a different error.
SELECT
[blahblahblah] ,
[blahblahblah] ,
[blahblahblah] ,
[blahblahblah] ,
[blahblahblah] ,
CASE WHEN COL_LENGTH('wmManifests', 'MTFvalue_Permit') IS NOT NULL
THEN [MTFvalue_Permit]
ELSE NULL
END AS 'MTFvalue_Permit'
FROM dbo.wmManifests
Invalid column name 'MTFvalue_Permit'.
I don't think that you can do it using expressions.
You can do this workaround.
Add a Script task that list columns from the table
Build the Select query dynamically:
If the column is not found pass NULL instead of the column name and give an alias (same as column name)
save this query string into a SSIS variable and use it as a source
You can also create a stored procedure that generate the query string. Execute it from a sql task and store the query string into ssis variable
Perhaps I am not understanding your question, but it seams like you are approaching this problem backwards. You cannot insert data into columns that do not exist. Is there a reason this column exists in only one out of four versions of this table? And does this column have a default value that you can leverage?
I would continue inserting to the set of columns that are common among all tables and allow the tables with columns that do not exist in every one of the tables in the set to use the default value for that column.