I am writing a small utility to copy a database from a proprietary ODBC database into a SQL Server database. Everything is working great, except when I use SQLBulkCopy to copy over the data. It works in most cases, but not when it's trying to copy a TIME field into a SQL DATETIME. I get this error:
The given value of type TimeSpan from the data source cannot be converted to type datetime of the specified target column.
When I create the schema in SQL Server I have to make the DATE and TIME fields DATETIME types in SQL Server, so there is no way around this. I wanted to use SQLBulkCopy so I didn't have to manually read through every record in every table and wrap logic around the special cases. Before I go down that road, is there another way I can do this? I have no control at all on the source database schema or content.
I assume you are dealing with pre-SQL Server 2008. SQL Server 2008 has DATE and TIME data types.
I think you would have to use a DataTable which matched the SQL Server schema and load this from your source reader, appropriately changing any TIME to a DATETIME by adding date information (e.g. 1/1/1900). Then use WriteToServer(DataTable). You might want to do it in batches, since you may use a bunch of memory reading it all into a DataTable.
Any particular reason you can't use SSIS?
Related
I have a project that needs to work on both Oracle and SQL Server. It currently runs on Oracle, and my job is to make queries compatible for Oracle and SQL Server.
The Oracle database has a table with a column called study_date which has DATE type. So in Java code, I create parameter like this:
MapSqlParameterSource parameters = new MapSqlParameterSource();
Calendar calendar = new GregorianCalendar(TimeZone.getTimeZone("America/New_York"));
calendar.setTimeInMillis(event.getTime()); //returns long value of time
parameters.addValue("study_date", calendar);
...
...
return parameters;
This works and insert a row with given time. However, if I run the same code with SQL Server, it doesn't work. I found there are many date types in SQL Server: https://learn.microsoft.com/en-us/sql/t-sql/data-types/date-and-time-types?view=sql-server-ver15, but none of the types work with Calendar object in java.
I was able to insert a row by creating Timestamp using Calendar object I created. It works, but that means I have to write two different sets of code for Oracle and SQL Server. I can, but I am looking for the best way to handle this.
I have an SQLite3 database. I also have an SQL Server database with the same structure. I need to export the data from SQLite and insert it into the SQL Server database.
The export from SQLite and the modification of the generated export needs to be 100% scripted. Inserting into the SQL Server database will be done manually through SQL Server Management Studio.
I have a mostly good dump of the database through this answer here. I can modify most of the script as needed with sed.
The one thing I'm stuck on right now is that the SQLite database stores timestamps as number of seconds since UNIX epoch. The equivalent column in SQL Server is DATETIME. As far as I know, inserting an integer into a DateTime won't work.
Is there a way to specify that certain fields be converted a certain way upon dumping from SQLite? Meaning, specify that the integer fields be dumped as proper DateTime strings that SQL Server will understand?
Or, is there something I can run on the Linux command line that will somehow find these Integer timestamps and convert them?
EDIT: Anything that runs in a Bash script on Ubuntu is acceptable.
Three basic solutions: (1) modify the data before the dump; (2) manipulate the file after the dump, or (3) modify the data on import. Which you choose will depend on how much freedom you have to modify schemas.
If you wish to do it in SQLite, I'd suggest adding text columns with the dates stored as needed for import to SQL Server, then ignore or remove the original columns on dump. The SQLite doc page for datetime() may help, as might answers to this question.
Or, you can write a function in SQL Server that handles the import. Perhaps set it on an insert trigger.
Otherwise, a script that manipulates your dump file would work too. It sounds like you have a good handle on how to do this.
I am trying to copy data from views on a trusted SQL Server 2012 to tables on a local instance of SQL Server on a scheduled transfer. What would be the best practice for this situation?
Here are the options I have come up with so far:
Write an executable program in C# or VB to delete existing local table, query the data from remote database and then write results to tables in the local database. The executable would run on a scheduled task.
Use BCP to copy data to a file and then upload into local table.
Use SSIS
Note: The connection between local and remote SQL Server is very slow.
Since the transfers are scheduled, so I suppose you want this data to be up-to-date.
My recommendation would be to use SSIS and schedule it using SQL Agent. If you wrote a C# program, I think the best outcome you will gain is a program imitating SSIS. Moreover, SSIS will be a very easy to amend the workflow anytime.
Either way, to make such program/package up-to-date, you will have to answer an important question: Is the source table updatable or is it like a log (inserts only)?
This question is so important because it will determine how you will fetch the new updates from the source table. For example, if the table represents logs, you will most probably use the Primary Key to detect new records, if not, you might want to seek a column representing update date/time. If you have the authority to alter the source table, you might want to add timestamp column which represent the row version (timestamp differs than datetime)
For building an SSIS package, it will mainly contain the following components:
Execute SQL Task to get the maximum value from source table.
Execute SQL Task to get the last value where it should start from at the destination table. You can get this value either by selecting the maximum value from the destination table or if the table is pretty large you can store that value in another table (configuration table for example).
Data Flow which moves the data from source table starting after the value fetched in step 2 to the value fetched in step 1.
Execute SQL Task for updating the new maximum value back to the configuration table if you chose this technique.
BCP can be used to export the data compress and transfer over network which can be then imported into local instance of SQL.
Also with BCP data exports can be contained with smaller batches of data for easier management of data.
https://msdn.microsoft.com/en-us/library/ms191232.aspx
https://technet.microsoft.com/en-us/library/ms190923(v=sql.105).aspx
When using SSIS in SQL Server 2005 to convert a FoxPro database to a SQL Server database, if the given FoxPro database has a date type, SSIS assumes it is an integer type. The only way to convert it to a dateTime type is to manually select this type. However, that is not practical to do for over 100 tables.
Thus, I have been using a workaround in which I use DTS on SQL Server 2000 which converts it to a smallDateTime, then make a backup, then a restore into SQL Server 2005.
This workaround is starting to be a little annoying.
So, my question is: Is there anyway to setup SSIS so that whenever it encounters a date type to automatically assume it should be converted to a dateTime in SQL Server and apply that rule across the board?
Update
To be specific, if I use the import/export wizard in SSIS, I get the following error:
Column information for the source and the destination data could not be retrieved, or the data types of source columns were not mapped correctly to those available on the destination provider.
Followed by a list of a given table's date columns.
If I manually set each one to a dateTime, it imports fine.
But I do not wish to do this for a hundred tables.
You could make a small FoxPro program that will loop through your list of tables and write out a SQL INSERT INTO statement for each record to a .sql file which you could then open from or paste into SQL Management Studio and execute. You could then control the date formats that will work with SQL Server's date type fields.
Something similar could be done in c#.
I'm trying to copy data from MySQL to SQL Server 2008.
My SSIS is generating error for time (DBTime) column in MySQL database. (cannot convert dbtime to dbtime2)
What datatype can i use in SQL server for time? I tired nvarchar, varchar and also tried data conversion task but i get same error.
You could use time or datetime.
EDIT:
Now that I see the type of data that MySQL uses for time what you probably want to do is to put the data into an nvarchar on the SQL Server side, and in SSIS you can invoke
TIME_FORMAT(timecol, '%H:%i:%S')
The SSIS tool lets you do specifications of how to manipulate individual columns before inserting into the other database using scripting.