I tried to append my pandas dataframe to an existing data table in sql server like below. All my column names in the data are absolutely identical to the database table.
df.to_sql(table_name,engine,schema_name,index=False,method='multi',if_exists='append',chunksize=100)
But it failed and I got error like below:
IntegrityError: ('23000', "[23000] [Microsoft][ODBC Driver 17 for SQL Server]
[SQL Server]Cannot insert explicit value for identity column in table 'table_name'
when IDENTITY_INSERT is set to OFF. (544) (SQLParamData)")
I have non clue what that means and what I should do to make it work. It looks like the issue is IDENTITY_INSERT is set to OFF?. Appreciate if anyone can help me understand why and what potentially I can do. Thanks.
In Layman's terms, the data frame consists of primary key values and this insert is not allowed in the database as the INDENTITY_INSERT is set to OFF. This means that the primary key will be generated by the database itself. Another point is that probably the primary keys are repeating in the dataframe and the database and you cannot add duplicate primary keys in the table.
You have two options:
First: Check in the database, which column is your primary key column or identity column, once identified remove that column from your dataframe and then try to save it to the database.
SECOND: Turn on the INDENTITY INSERT SET IDENTITY_INSERT Table1 ON and try again.
If your dataframe doesn't consists of unique primary keys, you might still get another error.
If you get error after trying both of the option, kindly update your question with the table schema and the dataframe value using df.head(5)
I have 2 major problems while Migrating my data from a database to another.
Sql query insert into FlexSchoolDemo.dbo.BursaryFeeMaster([FeeName]) select [FeeType] from FlexUni.dbo.Bursary_FeeMaster
Error Cannot insert the value NULL into column 'IsSchoolFee', table 'FlexSchoolDemo.dbo.BursaryFeeMaster'; column does not allow nulls. INSERT fails.
While Migrating this data, the destination table contains some columns with bool attribute one of which is IsSchoolFee and now the error says cannot insert the Value NULL into the column IsSchoolFee. How do i migrate this successfully without having to change the bool parameters?
Sql Query insert into FlexSchoolDemo.dbo.StApplicationsOlevel([ApplicationNo],[JambregNo],[SubjectId],[GradeId]) select [JambregNo],[JambregNo],[SubjectID],[GradeID] from FlexUni.dbo.Student_Applications_Olevel
Error The INSERT statement conflicted with the FOREIGN KEY constraint "FK_StApplicationsOlevel_AdmSubjectGrades_GradeId". The conflict occurred in database "FlexSchoolDemo", table "dbo.AdmSubjectGrades", column 'GradeId'.
While Migrating this data, SubjectId and GradeId are both foreign keys in their different tables but throws error. How can i also achieve a successful migration too?
When I tried to insert data into a table in SQL Server, which has an ID column specification as following, I'm getting the error,
Cannot insert NULL in RowID column
even though I've written insert trigger for specifying the value of RowID column.
mapping.Id(x => x.Id, "RowID").GeneratedBy.TriggerIdentity();
You are using a FOR trigger; this is a different way of specifying an AFTER trigger. And you need to switch this to an "INSTEAD OF" trigger.Please refer to this webpage.
The reason it is failing now is because SQL Server validation is occurring prior to your trigger executing and performing it's primary key creation.
My webApp works with 2 DataBase servers Informix and DB2 (v9.5 run on localhost), when i work with informix DB i can insert null into a primary key (informix DB handle it and accepts nulls and auto increment the column (Serial8)) but when i switched to use DB2 it doesn't work and this error arrises (DB2 SQL Error: SQLCODE=-407, SQLSTATE=23502, SQLERRMC=TBSPACEID=2, TABLEID=280, COLNO=0, DRIVER=3.50.152 , sqlCode = -407) it looks like DB2 doesn't allow nulls for primary keys (BigInt) ,so how to enforce DB2 to allow nulls for primary key? in a word i want DB2 to allow me insert nulls for this column and auto increment the values in it each time i make an insert
here's the script to create the table:
CREATE TABLE corr.CORRESPONDENCE (
this is the specified col---->CORR_ID BIGINT NOT NULL,
CORR_NAME VARCHAR(255) NOT NULL,
CORR_NO VARCHAR(255),
CREATE_DATE_TIME TIMESTAMP NOT NULL,
DELIVERY_DATE_TIME DATE,
NO_OF_ATTACH INTEGER,
SITE_ID VARCHAR(20),
DELIVERY_ID VARCHAR(20),
CREATE_USER BIGINT NOT NULL,
SECURITY_ID BIGINT,
DELIVERY_BY VARCHAR(20),
WORKFLOW_ID BIGINT
)
DATA CAPTURE NONE ;
ALTER TABLE corr.correspondence ADD CONSTRAINT u143_159 PRIMARY KEY
(corr_id) ;
Which version of Informix? What is the schema of the table? What is the INSERT statement? Which API are you using to access Informix? Which platform is the client code running on? Which platform is the database server running on?
I'm not convinced that you can insert nulls into a SERIAL-like column in Informix. Do you have a primary key constraint on your table, or just a SERIAL8 column that has no NOT NULL and no PRIMARY KEY constraint on it? You cannot insert NULL directly into a SERIAL8 column (nor, by inference, into a SERIAL or BIGSERIAL column).
Demonstration (using a development version of Informix 11.70.FC6 on RHEL 5 Linux x86/64; client is ESQL/C based, and both client and server are on the same machine):
SQL[1910]: begin;
SQL[1911]: create table t1(s8 serial8 not null, v1 char(8) not null);
SQL[1912]: insert into t1(s8, v1) values(null, "works?");
SQL -391: Cannot insert a null into column (t1.s8).
SQLSTATE: 23000 at /dev/stdin:6
SQL[1913]: rollback;
SQL[1914]: begin;
SQL[1915]: create table t1(s8 serial8 primary key, v1 char(8) not null);
SQL[1916]: insert into t1(s8, v1) values(null, "works?");
SQL -703: Primary key on table (t1) has a field with a null key value.
SQLSTATE: IX000 at <<temp>>:2
SQL[1917]: rollback;
SQL[1918]: create table t1(s8 serial8, v1 char(8) not null);
SQL[1919]: insert into t1(s8, v1) values(null, "works?");
SQL -391: Cannot insert a null into column (t1.s8).
SQLSTATE: 23000 at <<temp>>:2
SQL[1920]: drop table t1;
SQL[1921]:
And bother, I forgot to restart the transaction after 1917!
This is behaving exactly as it should; you should not be allowed to insert a NULL into a SERIAL8 (or SERIAL, or BIGSERIAL) column. You can insert zeroes into those columns and a new value will be allocated automatically. But you cannot, and should not be able to, insert a NULL into the column.
DB2 is likewise correct in rejecting attempts to insert NULL into any of the columns in a primary key. It simply is not something you should be allowed to do.
Answering comments
Frank Computer commented:
Strange, I was under the impression that when loading data into a table with a SERIAL column, if I did not supply a value for the SERIAL column, it would convert the NULL to a zero during the insert, as if the loaded data contained a zero?
Also, with ISQL Perform, when I insert a new row into a table containing a SERIAL column, I don't supply a value for the SERIAL column, yet Perform displays a zero (0) and after hitting Escape, it converts it to the next highest INT value!
My immediate response was:
LOAD is done by a complex sub-routine in the client that munges the data, and it could/would deal with NULL for SERIAL columns.
With ISQL, Perform explicitly enforces 0 during data entry and reports back on the inserted value; again, the client-side code is preventing the error.
This is why it is important to know what is in use to demonstrate problems or features. Now I've got to make a LOAD and NULL demo using DB-Access...I don't think my SQLCMD program fixes up NULL for SERIAL columns during LOAD (or, if it does, I made that hack a long, long time ago).
Testing DB-Access (from IDS 11.70.FC2 on Mac OS X 10.7.4, this time), with:
xx.unl
|data for row 1|1|
|data for row 2|2|
xx.sql
BEGIN;
CREATE TABLE load_null(s8 SERIAL8 PRIMARY KEY, v32 VARCHAR(32) NOT NULL, i INTEGER NOT NULL);
LOAD FROM "xx.unl" INSERT INTO load_null;
ROLLBACK;
DB-Access Output
$ dbaccess stores xx
Database selected.
Started transaction.
Table created.
703: Primary key on table (load_null) has a field with a null key value.
847: Error in load file line 1.
Error in line 3
Near character position 41
Transaction rolled back.
Database closed.
$
This does not lend support to the 'DB-Access maps NULL for a SERIAL8 column into a zero' hypothesis. This is SERIAL8 rather than plain SERIAL, but when I changed SERIAL8 into SERIAL, I got the same error; ditto with BIGSERIAL. I don't have ISQL as opposed to DB-Access on the Mac (laziness; I did the port a while ago, but didn't install it as it was not official, and it is not GA), and it is possible that there's a difference between the two LOAD commands, but relatively unlikely.
Testing SQLCMD on the same SQL and data (unload) files, I get the same error message.
I am more than ever unconvinced by the claim that it is possible to insert NULL values into a primary key column with Informix.
More comments and explanations
Although I know LOAD is not an Informix SQL native statement, I assumed it was added to the SE (Standard Engine) and OL (OnLine) engines?
No; the LOAD statement is handled by code in client programs: DB-Access, ISQL, I4GL, DB-Load, DB-Import. In each case, the statement is recognized and parsed by the client, converted into a suitable INSERT statement that is prepared, then the client reads and parses the data file, and sends the data to the server one row at a time (logically; actually, there's an INSERT cursor involved which gives you batch operation on insertions).
Or does LOAD statement actually call the DBLOAD.EXE utility in SE/OL or onload.exe?
No: the LOAD statement does not involve DB-Load, nor does it involve ON-Load.
Is the source for SQLCMD available? If so, can I dump dbaccess and replace it with a stripped down version of SQLCMD?
Yes. It is available from the IIUG (International Informix User Group) Software Archive. The version available there (87.02) is close to current (I'm using 87.06, but I'm not quite ready to release that to the rest of the world, and it'll be 88.00 when it is released). I don't support it on Windows, simply because I find Windows too hostile a development environment. It has, on occasion, been made to work on Windows, though. My last attempt stopped when I found Microsoft promulgating the 'Safe C Library' routines, but the routines they provide are not the same as the ones in the standard TR 27341. I gave up again at that point.
I just confirmed that my ole SE-4.10 clunker accepts NULL values for SERIAL columns when inserting a load file with LOAD.
OK. You couldn't specify PRIMARY KEY constraints in that version (those arrived with 5.00, I'm almost certain), but you could create unique indexes on SERIAL columns, etc. To the extent that it is a bug, it has presumably been fixed. It might or might not be fixed in SE 7.26; I'd expect it to be, but haven't demonstrated that it is. It is fixed in 11.70; my tests above demonstrate that.
You can't insert a null value into a primary key DB2. Instead, you need to modify your insert query to insert the new key, or just not include it in your Insert statement and have the database handle it automatically...
It would help if we knew the insert query (or at least part of it). We could offer better guidance on how to correct it. However, taking a guess at the source of your issue:
Assuming the table looks like this:
ID INTEGER NOT NULL GENERATED DEFAULT (START WITH 1, INCREMENT BY 1)
SomeOtherField VARCHAR(50)
Your statement should just be:
Insert into MyTable (SomeOtherField) Values ('somevalue')
instead of
Insert into MyTable (ID, SomeOtherField) Values (null, 'somevalue')
or
Insert into Values (null, 'somevalue')
A similar question and more info can be found here: http://www.dbforums.com/db2/669352-autoincrement-fields.html
With Informix SERIAL columns, you can insert a zero (0) and it will automatically convert it to the next highest available integer value. You can also insert a specific integer value as long as it hasn't already been used, since SERIAL columns have a unique constraint.
Your question is ambiguous. Primary keys can also be non-SERIAL datatypes and accept a NULL value. If this is the case, I suggest you create a surrogate key (usually an autoincrement column) in order to uniquely identify each row.
What is your primary key used for and what is the reason for inserting NULL into your primary key?.. Is it because at the moment you're inserting the row, the value is unknown, but later on it will be updated with a known value?.. NULL's as primary keys tend to make things not work properly, especially when joining to foreign keys in a child table. If your primary key doesn't have a unique constraint, that means you could have several rows with duplicate NULL values as their primary key.. not a good idea in any DB, DB2 included!
You should try the following SQL statement to solve your problem:
insert into table(primarykey,c1, ...) values(null,v1,...).
Try insert into table(c1,...) values(v1,...)
In my application, using SQL Server 2005, I'm having two tables, let's call them Table A and Table B; a foreign key constraint is defined on Table B, referencing the primary-key column in Table A, which is an auto-generated integer ID. I'm running the following simple transaction:
Start transaction
Insert a row to table A
Retrieve the last-generated ID ("SELECT ##IDENTITY ... ")
Insert data to table B, using this ID
Commit
It all works well, until I'm trying to create merge replication (continuous) with another SQL Server 2005. Both publisher and subscriber now fail this transaction when trying to insert data to table B, because of foreign key constraint failure:
The INSERT statement conflicted with the FOREIGN KEY constraint "FK_TableB_TableA". The conflict occurred in database "MyDB", table "TableA", column 'ID'.
I was not able to make it work by committing after inserting data to table A. However, after removing the merge replication, everything worked. The database code is written in C++, using ADO.
Is the replication interfering with the transaction anyhow? Any other possible explanation?
Are the Primary Key values on Table A at both server nodes discrete from one another (In other words are you using Identity Range management at each node)?
Also, has the Foreign Key constraint been configured with the Not For Replication property?
I would assume that because your Foreign Key constraint has already been enforced locally at the Publisher, that you do not need to re-check it when merging with the Subscriber.
Looks like the issue is related to the scope of the ##IDENTITY function. When I used LAST_IDENT('TableB') instead, things seem to work.
As described in MSDN:
IDENT_CURRENT returns the last identity value generated for a specific table in any session and any scope.
##IDENTITY returns the last identity value generated for any table in the current session, across all scopes.
SCOPE_IDENTITY returns the last identity value generated for any table in the current session and the current scope.