I have a CSV file which contains '\N' in some cells under a column for which the header is defined as int in SQL Server.
I am using pyodbc to update the SQL server data each day from the CSV supplied. The problem is whenever I have '\N' in the CSV file then the SQL server update results in an error and I have to delete the rows with '\N' to have them updated.
Is there any way I can update '\N' in int type column in SQL Server?
Below is the code
with open (Result_File, 'r') as h:
reader = csv.reader(h)
columns = next(reader)
query = "INSERT into dbo.test({0}) values ({1})"
query = query.format(','.join(columns), ','.join('?' * len(columns)))
for data_line in reader:
cur.execute(query,data_line)
Use a list comprehension to iterate through the lists returned from the csv.reader object, replacing elements with \n value with the default value you specified (0).
...
for data_line in reader:
# substitute 0 if the column value is '\n'
cur.execute(query, [0 if value == '\n' else value for value in data_line])
Note that is workaround is highly specific, and would be difficult to maintain if other cases of value substitution crop up. If possible, I'd fix the process that creates the input file so your processing code can be generalized and more readable.
Related
I try to search but didn't found answer to relative simple thing. I have a CSV, that doesn't have all the column as in my database table, as well as it miss the auto increment, primary key in CSV too.
All I did is I read CSV into the DataSet, and then run a traditional SQLBulkCopy code to read the first table of dataset to database table. But it give me following error:
The given ColumnMapping does not match up with any column in the source or destination.
My code for bulkcopy is
using (SqlBulkCopy blkcopy = new SqlBulkCopy(DBUtility.ConnectionString))
{
blkcopy.EnableStreaming = true;
blkcopy.DestinationTableName = "Project_" + this.ProjectID.ToString() + "_Data";
blkcopy.BatchSize = 100;
foreach (DataColumn c in ds.Tables[0].Columns)
{
blkcopy.ColumnMappings.Add(c.ColumnName, c.ColumnName);
}
blkcopy.WriteToServer(ds.Tables[0]);
blkcopy.Close();
}
I add Mapping to test, but it doesn't make difference to remove mapping part. If we remove mapping that it try to match column in order and since column are different in count they end up mismatch datatype and lesser column values etc. Oh yes the column names from CSV does match that from Table, and are in same case.
EDIT: I change the mapping code to compare the column name from live DB. For this I simply run a SQL Select query to fetch 1 record from database table and then do following
foreach (DataColumn c in ds.Tables[0].Columns)
{
if (LiveDT.Columns.Contains(c.ColumnName))
{
blkcopy.ColumnMappings.Add(c.ColumnName, c.ColumnName);
}
else
{
log.WriteLine(c.ColumnName + " doesn't exists in final table");
}
}
I would dump the results of CSV into a staging SQL table...and then do a simple insert from staging table to production table.
also do a simple Import of CSV into SQL Table, maybe there are some empty/invalid columns within CSV file.
I once had this problem and the cause was a difference in the case of the column names. One of the columns was "Id", but in the DB it was "id".
I have pyODBC installed for Python 3.2 and I am attempting to update a SQL Server 2008 R2 database that I created as a test.
I have no problem retrieving data and that has always worked.
However when the program performs a cursor.execute("sql") to insert or delete a row then it does not work - no error, nothing. The response is as if I am successful updating the database but no changes are reflected.
The code below essentially is creating a dictionary (I have plans for this later) and just doing a quick build of a sql insert statement (which works as I testing the entry I wrote to the log)
I have 11 rows in my table, Killer, which is not being affected at all, even after a commit.
I know this is something dumb but I can't see it.
Here is the code:
cnxn = pyodbc.connect('DRIVER={SQL Server Native Client 10.0};SERVER=PHX-500222;DATABASE=RoughRide;UID=sa;PWD=slayer')
cursor = cnxn.cursor()
# loop through dictionary and create insert entries
logging.debug("using test data to build sql")
for row in data_dictionary:
entry = data_dictionary[row]
inf = entry['Information']
dt = entry['TheDateTime']
stat = entry['TheStatus']
flg = entry['Flagg']
# create sql and set right back into row
data_dictionary[row] = "INSERT INTO Killer(Information, TheDateTime, TheStatus, Flagg) VALUES ('%s', '%s', '%s', %d)" % (inf, dt, stat, flg)
# insert some rows
logging.debug("inserting test data")
for row in data_dictionary.values():
cursor.execute(row)
# delete a row
rowsdeleted = cursor.execute("DELETE FROM Killer WHERE Id > 1").rowcount
logging.debug("deleted: " + str(rowsdeleted))
cnxn.commit
Assuming this isn't a typo in the post, looks like you're just missing parentheses for the Connection.commit() method:
...
# delete a row
rowsdeleted = cursor.execute("DELETE FROM Killer WHERE Id > 1").rowcount
logging.debug("deleted: " + str(rowsdeleted))
cnxn.commit()
I have a customer setup that uses stored procedures to return data from its SQL Server database.
Those stored procedures are all built the same way - they take a bunch of input parameters, and they return:
the first is just a single row, single column with the result code (type INT) - 0 for success, some other values otherwise; if the value is 0, then there's a second result set that contains the actual data
the second result set can be anything - any number of columns and rows
I'm trying to create a "generic" way to interface with this system, and my attempt is this:
create a class that takes stored procedure name and input parameters
a return type that contains both an ErrorCode INT property as well as a DataTable results property
However, I'm having trouble getting this to work with ADO.NET and SQL Server 2008 R2.
My code boils down to this:
public MyResultType CallService(string procedureName, MyParameters parameters)
{
MyResultType result = new MyResultType { ErrorCode = 0 };
using (SqlConnection conn = new SqlConnection(_connectionString))
using (SqlCommand cmd = new SqlCommand(procedureName, conn))
{
cmd.CommandType = CommandType.StoredProcedure;
SetupParameters(cmd, parameters);
// open connection, execute the stored procedure
conn.Open();
using (SqlDataReader rdr = cmd.ExecuteReader())
{
// get the first result set - the status code, one row, one column of type INT
while (rdr.Read())
{
result.ErrorCode = rdr.GetInt32(0);
}
// if we got a "0" (success) -> go to the next result set and load it into the table
if(result.ErrorCode == 0 && rdr.NextResult())
{
result.ResultTable = new DataTable();
result.ResultTable.Load(rdr);
int colCount = result.ResultTable.Columns.Count;
int rowCount = result.ResultTable.Rows.Count;
}
rdr.Close();
}
conn.Close();
}
return result;
}
My issue is: the call to the stored procedure works just fine, the error code = 0 is picked up just fine, the data table is created and the number of columns is the expected value - but there are NO ROWS loaded...
I've been trying everything I can think of to get those rows into the DataTable - no luck. And of course - if I execute this very same stored procedure in SQL Server Management Studio, everything works just fine, I get my ErrorCode=0 and my result set of 18 columns and 5 rows - no problem....
What am I missing? Can anyone spot the problem in my code? Why aren't the rows of the second result set loaded (but the columns are being detected, it seems)?
The code as published works fine - there was a difference between how I call it from C# and in SQL Server Management Studio : NULL handling.
I had some input parameters set to int and thus provided a 0 value to the stored procedure, while it really expected to get a NULL if the value isn't defined......
Stupid rookie mistake..... SORRY GUYS! And thanks for your inputs!
The DataTable.Load method implicitly calls the NextResult method, so combining that with your explicit call, it is advancing to a resultset that isn't there. You should remove your own call to NextResult if you want to use DataTable.Load or loop through and fill a datatable yourself.
I am using a function I read about here to merge a series of recordsets that are generated by a stored procedure that is called in a loop. I have used this function before in other stored procedure cases and it has never produced this problem.
The research I did online points to basically one of two reasons:
1) I am trying to update or insert a date/time value that is not formatted correctly into a SQL Server 2005 table
2) I am trying to insert a, for example, CHAR(60) string into a CHAR(50) field.
Reason one is not applicable here since I am not using dates (in datetime format at least).
Reason two seemed to be the most likely issue. So I did a series of Response.Write() to spit out the objField.Name, objField.Type, objField.ActualSize, and if it was a numeric field the objField.Precision and objField.NumericScale.
Let us say that the stored SQL procedure is called twice as I am querying for values that are occurring in the same time frame but 2 different states. The loop I have in the ASP page does a For Each on the state in the state list and calls the stored procedure for each of the elements in the state list. I then call the MergeRecordSets function so it combines the 2 results into one. The general rule is that the data types and sizes of the columns in each resultset must be the same. With my Response.Write() checks of each of the columns returned in the 2 data sets I have found that they are identical. Doing my checks I also found that it breaks on the first column that is a NUMERIC column. The previous columns it was OK with were all CHAR or VARCHAR.
Is there any other reason why this error would come up?
The following is how I am calling the record merger function. The oQueryResult is going to be the final output (the combined records). objSingleRS is the result set returned by the stored procedure.
If oQueryResult Is Nothing Then
Set oQueryResult = objSingleRS
Else
Set oQueryResult = MergeRecordSets(Array(oQueryResult, objSingleRS))
End If
Here is the merge function. The line in which the code breaks is marked below.
Function MergeRecordSets(arrRecordsets)
Dim x, y, objCurrentRS
Dim objMergedRecordSet, objField, blnExists
Set objMergedRecordSet = Server.CreateObject("ADODB.Recordset")
For x=0 To UBound(arrRecordsets)
Set objCurrentRS = arrRecordsets(x)
For Each objField In objCurrentRS.Fields
blnExists = False
For y=0 To objMergedRecordSet.Fields.Count-1
If LCase(objMergedRecordSet.Fields(y).Name) = Lcase(objField.Name) Then
blnExists = True : Exit For
End If
Next
If Not(blnExists) Then
objMergedRecordSet.Fields.Append objField.Name, objField.Type, objField.DefinedSize
'objMergedRecordSet.Fields(objMergedRecordset.Fields.Count-1).Attributes = 32 'adFldIsNullable
End If
Next
Next
objMergedRecordSet.Open
For x=0 To UBound(arrRecordsets)
Set objCurrentRS = arrRecordsets(x)
Do Until objCurrentRS.EOF
objMergedRecordSet.AddNew
For Each objField In objCurrentRS.Fields
If Not(IsNull(objField.Value)) Then
'Response.Write(objField.Name & "<br>")
'Response.Write(objField.Type & "<br>")
objMergedRecordSet.Fields(objField.Name).Value = objField.Value 'Here is where it throws the Error.
End If
Next
objCurrentRS.MoveNext
Loop
Next
objMergedRecordSet.MoveFirst
Set MergeRecordSets = objMergedRecordSet
End Function
Here is the full error message returned:
Microsoft Cursor Engine error '80040e21'
Multiple-step operation generated errors. Check each status value.
/includes/funcs/Common.asp, line 4109
You mentioned that you have numeric columns, but you never set the Precision and NumericScale properties when you create the new Field in objMergedRecordSet. You need to set these properties for adNumeric and adDecimal fields.
objMergedRecordSet.Fields.Append objField.Name, objField.Type, objField.DefinedSize
With objMergedRecordSet.Fields(objField.Name)
.Precision = objField.Precision
.NumericScale = objField.NumericScale
End With
Also make sure you are not trying to put a NULL into a column that will not accept a NULL value. There is also the possibility of a type mismatch to cause this error so make sure you are passing a numeric value.
- Freddo
I have a SQLite database in one system, I need to extract the data stored in SQLite to Oracle database. How do I do this?
Oracle provides product called the Oracle Database Mobile Server (previously called Oracle Database Lite) which allows you to synchronize between a SQLite and an Oracle database. It provides scalable bi-directional sync, schema mapping, security, etc. The Mobile Server supports both synchronous and asynchronous data sync. If this is more than a one-time export and you need to keep your SQLite and Oracle Databases in sync, this is a great tool!
Disclaimer: I'm one of the Product Managers for Oracle Database Mobile Server, so I'm a bit biased. However, the Mobile Server really is a great tool to use for keeping your SQLite (or Berkeley DB) and Oracle Databases in sync.
You'll have to convert the SQLite to a text file (not certain of the format) and then use Oracle to load the database from text (source is http://www.orafaq.com/wiki/SQLite). You can use the .dump command from the SQLite interactive shell to dump to a text file (see the docs for syntax).
SQL Loader is a utility that will read a delimited text file and import it into an oracle database. You will need to map out how each column from your flat file out of sqlite matches to the corresponding one in the Oracle database. Here is a good FAQ that should help you get started.
If you are a developer, you could develop an application to perform the sync. You would do
SELECT name FROM sqlite_master WHERE type='table'
to get the table names, then you could re-create them in Oracle (you can do DROP TABLE tablename in Oracle first, to avoid a conflict, assuming SQLite will be authoritative) with CREATE TABLE commands. Getting the columns for each one takes
SELECT sql FROM sqlite_master WHERE type='table' and name='MyTable'
And then you have to parse the result:
string columnNames = sql.replace(/^[^\(]+\(([^\)]+)\)/g, '$1').replace(/ [^,]+/g, '').split(',');
string[] columnArray = columnNames.Split(',');
foreach (string s in columnArray)
{
// Add column to table using:
// ALTER TABLE MyTable ADD COLUMN s NVARCHAR(250)
}
A StringBuilder can be used to collect the table name with its columns to create your INSERT command. To add the values, it would just be a matter of doing SELECT * FROM MyTable for each of the tables during your loop through the table names you got back from the initial query. You would iterate the columns of the rows of the datatable you were returned and add the values to the StringBuilder:
INSERT INTO MyTable ( + columnA, columnB, etc. + ) VALUES ( datarow[0], datarow[1], etc. + ).
Not exactly like that, though - you fill in the data by appending the column name and its data as you run through the loops. You can get the column names by appending s in that foreach loop, above. Each column value is then set using a foreach loop that gives you each object obj in drData.ItemArray. If all you have are string fields, it's easy, you just add obj.ToString() to your StringBuilder for each column value in your query like I have below. Then you run the query after collecting all of the column values for each row. You use a new StringBuilder for each row - it needs to get reset to INSERT INTO MyTable ( + columnA, columnB, etc. + ) VALUES ( prior to each new row, so the new column values can be appended.
If you have mixed datatypes (i.e. DATE, BLOB, etc.), you'll need to determine the column types along the way, store it in a list or array, then use a counter to determine the index of that list/array slot and get the type, so you know how to translate your object into something Oracle can use - whether that means simply adding to_date() to the result, with formatting, for a date (since SQLite stores these as date strings with the format yyyy-MM-dd HH:mm:ss), or adding it to an OracleParameter for a BLOB and sending that along to a RunOracleCommand function. (I did not go into this, below.)
Putting all of this together yields this:
string[] columnArray = null;
DataTable dtTableNames = GetSQLiteTable("SELECT name FROM sqlite_master WHERE type='table'");
if (dtTableNames != null && dtTableNames.Rows != null)
{
if (dtTableNames.Rows.Count > 0)
{
// We have tables
foreach (DataRow dr in dtTableNames.Rows)
{
// Do everything about this table here
StringBuilder sb = new StringBuilder();
sb.Append("INSERT INTO " + tableName + " ("); // we will collect column names here
string tableName = dr["NAME"] != null ? dr["NAME"].ToString() : String.Empty;
if (!String.IsNullOrEmpty(tableName))
{
RunOracleCommand("DROP TABLE " + tableName);
RunOracleCommand("CREATE TABLE " + tableName);
}
DataTable dtColumnNames = GetSQLiteTable("SELECT sql FROM sqlite_master WHERE type='table' AND name='"+tableName+"'");
if (dtColumnNames != null && dtColumnNames.Rows != null)
{
if (dtColumnNames.Rows.Count > 0)
{
// We have columns
foreach (DataRow drCol in dtTableNames.Rows)
{
string sql = drCol["SQL"] != null ? drCol["SQL"].ToString() : String.Empty;
if (!String.IsNullOrEmpty(sql))
{
string columnNames = sql.replace(/^[^\(]+\(([^\)]+)\)/g, '$1').replace(/ [^,]+/g, '').split(',');
columnArray = columnNames.Split(',');
foreach (string s in columnArray)
{
// Add column to table using:
RunOracleCommand("ALTER TABLE " + tableName + " ADD COLUMN " + s + " NVARCHAR(250)"); // can hard-code like this or use logic to determine the datatype/column width
sb.Append("'" + s + "',");
}
sb.TrimEnd(",");
sb.Append(") VALUES (");
}
}
}
}
// Get SQLite Table data for insertion to Oracle
DataTable dtTableData = GetSQLiteTable("SELECT * FROM " + tableName);
if (dtTableData != null && dtTableData.Rows != null)
{
if (dtTableData.Rows.Count > 0)
{
// We have data
foreach (DataRow drData in dtTableData.Rows)
{
StringBuilder sbRow = sb; // resets to baseline for each row
foreach (object obj in drData.ItemArray)
{
// This is simplistic and assumes you have string data for an NVARCHAR field
sbRow.Append("'" + obj.ToString() + "',");
}
sbRow.TrimEnd(",");
sbRow.Append(")");
RunOracleCommand(sbRow.ToString());
}
}
}
}
}
}
All of this assumes you have a RunOracleCommand() void function that can take a SQL command and run it against an Oracle DB, and a GetSQLiteTable() function that can return a DataTable from your SQLite DB by passing it a SQL command.
Note that this code is untested, as I wrote it directly in this post, but it is based heavily on code I wrote to sync Oracle into SQLite, which has been tested and works.