I have a possibly peculiar data migration problem:
I have an existing and populated SQLite3 database.
I do receive a new schema for a (hopefully compatible) database.
Result should be a new database, built according to new schema, containing as much as possible of the old database content.
Given limitations in both SQLite3 ALTER statement and our workflow it is safe to assume:
normal case will be new columns are added to end of table.
added columns (fields) will either have a default or can be left NULL.
rarely some table will be added.
very rarely some table or column may be dropped.
no table/column renaming will happen.
no column reshuffling will happen.
NOTE: if the new schema is not compatible with the old one (i.e.: any of the above assumptions does not hold true) it's accepted to fail badly.
I tried this script (old database is data.sql3 and new schema is data.schema):
mkdir tmp
cd tmp
#compute old DB schema
sqlite3 ../data.sql3 .schema >old_s
#purge new schema for any initialization...
grep -v ^INSERT ../data.schema >data.schema
#... create a dew, empty DB...
sqlite3 new.sql3 <data.schema
#... and compute a standard schema
#(this is done to avoid typing differences)
sqlite3 new.sql3 .schema >new_s
#iff the schemas are different
if ! diff -q old_s new_s
then
#save old DB
mv ../data.sql3 .
#dump contents
sqlite3 data.sql3 .dump >old_d
#expunge all statements needed to recreate DB/Tables
#new_d contains only INSERT statements
grep -v -f old_s old_d >new_d
#add old DB content to new DB
sqlite3 new.sql3 <new_d
#move new DB in place
mv new.sql3 ../data.sql3
fi
cd ..
This works to detect changes, but fails to repopulate the new database because .dump does not contain column names and thus insertion fails (missing value).
What I'm looking for is some way to force sqlite3 DB .dump to output INSERT statements containing all field names (normally it relies on position) or, it that's not possible, some way to tell sqlite3 DB <new_d to consider any undefined field as null or default (without failing).
Any other way to achieve the same result (without requiring knowledge of what, exactly, has been modified) would be equally welcome.
To be able to insert/import dumps with less columns into a table you can provide default values for the new, appended columns, or simply enable them to be set to NULL. The constraint clause is the same for CREATE TABLE and ALTER TABLE:
http://www.sqlite.org/syntax/column-constraint.html
-- newColumn is set to a default value if not provided with INSERT
alter table myTable
add column newColumn INTEGER NOT NULL default 0;
-- newColumn may be NULL, which is the default if not provided with INSERT
alter table myTable
add column newColumn INTEGER;
-- It is also valid to combine NULL and DEFAULT constraints
alter table myTable
add column newColumn INTEGER default 0;
Note that in order for the INSERT statement to work with the new columns it must provide the column names.
Related
I have a data set and want to import it into my database with the condition:
In case there is a record that cannot be imported, it can be extracted into a log
Although existing records can not be imported but still allow import of records that can be imported (other records) and continue to process
Currently I use the BCP utility to import data into the table from the csv file with:
bcp table_name IN C:\Users\09204086121\Desktop\data.csv -T -c -o C:\Users\09204086121\Desktop\logOut.log -e C:\Users\09204086121\Desktop\errOut.log
It just satisfies my condition 1 above.
I need that when the record has error (duplicate primary key,...), write to log (1) and continue to insert into the table the other normal records (2).
I came up with the idea that combining trigger with bcp, after creating a trigger and adding the parameter -h "FIRE_TRIGGERS" to the bcp statement, the insert will ignore records that have the same key but it won't write to the log.
This is my trigger.
ALTER TRIGGER [PKGORDERCOMMON].[T_ImportData] ON [PKGORDERCOMMON].[IF_R_BUNRUI1]
INSTEAD OF INSERT
AS
BEGIN
--Insert non duplicate records
INSERT INTO [IF_R_BUNRUI1]
(
SYSTEM_KB,
BUNRUI1_CD,
BUNRUI1_KANJI_NA,
BUNRUI1_KANA_NA,
CREATE_TS
)
SELECT SYSTEM_KB,
BUNRUI1_CD,
BUNRUI1_KANJI_NA,
BUNRUI1_KANA_NA,
CREATE_TS
FROM inserted i
WHERE NOT EXISTS
(
SELECT *
FROM [IF_R_BUNRUI1] c
WHERE c.BUNRUI1_CD = i.BUNRUI1_CD
AND c.SYSTEM_KB = i.SYSTEM_KB
);
END;
Is there anyone who can help me.
BCP is not meant for what you are asking it to do (separate good and bad records). For instance, bcp -e option has a limit to how many records it will show. Im not sure if this limit is tied to the "max errors" option, but regardless there is a limit.
Your best option is to load all the records and address bad data in t-sql.
Load all records in such a way to ignore conversion errors. Either:
load entire line from file into a single, large varchar column. Parse out columns and qc data as needed.
or
load all columns from source file into generic varchar columns with large enough size to accomodate your source data.
Either way, when done, use t-sql to inspect your data and split among good/bad records.
I recently had a horrible blunder.
While attempting to fix an issue we were having with our Exact Synergy system I was attempting to replace the data in two columns for one account with NULL, instead I replaced those two columns in ALL accounts with NULL. Completely restoring from a backup is not an option so now I am left trying to figure out how to replace the missing data.
I have made a full restore of a recent backup for this database to a test database and have confirmed that the data I need is there. I am trying to figure out how to properly write a query that will replace the data in the two columns.
Since this is a backup of the same database, the tables and columns are all identically named.
The databases are Synergy and Synergy_TESTDB
The owner of the tables is dbo
The table is called Addresses
The columns are called textfield1 and textfield2
What I would like to do is take the data in textfield1 and textfield2 from the backup database and use it to populate the empty, or NULL, columns in the live database.
I am extremely new to SQL, and would appreciate any help.
This is obviously untested. I take no responsibility for you using this code.
That said I'd like to try and help you.
The main point is the 3 part database.table naming: I'm assuming you restored backup to same server. I'm also assuming you have a primary key on the table? And that Synergy_TESTDB is the restored database:
update target
set target.textfield1 = source.textfield1
from Synergy.dbo.Addresses target
join Synergy_TESTDB.dbo.Addresses source on target.PrimaryKeyCol = source.PrimaryKeyCol
where target.textfield1 IS NULL
update target
set target.textfield2 = source.textfield2
from Synergy.dbo.Addresses target
join Synergy_TESTDB.dbo.Addresses source on target.PrimaryKeyCol = source.PrimaryKeyCol
where target.textfield2 IS NULL
(Sure it could be done in a single update, but I'm trying to keep it simple.)
I strongly suggest you try in another test database first.
A good habit to get in to is to use a pattern like this:
BEGIN TRANSACTION
-- Perform updates
-- Examine the results: select * from dbo.Blah ...
-- If results are wrong, we just rollback anyway
ROLLBACK
-- If results are what you want, uncomment the COMMIT and comment out the ROLLBACK
-- COMMIT TRANS
I'm using BCP because I want to export some columns from a table:
bcp "SELECT csusUsageDate, csusType, csusTrack1, csusTrack2, csusTrack3, csusDateReaded, csusLoggedIn FROM [DbJamaica].[dbo].[CS_Usage]" queryout "C:\temp\CS_Usage.txt" /U.. /P.. -c -T
Here I exclude my primary key, now I want to import this txt and I want that my primary key will be auto generate:
bcp DbJamaica.dbo.CS_Usage out "C:\temp\CS_Usage.txt" /U.. /P.. -c -T
I have always format not valid why? I also used format file but I have the same error.
This is my question: how to exclude identify column or some columns?
You can't, as far as I know. Bcp basically bulk inserts blindly into a table, if the columns don't match, you get an error. What you can do, though, is create a staging table like:
SELECT TOP 0 csusUsageDate, csusType, csusTrack1, csusTrack2, csusTrack3, csusDateReaded, csusLoggedIn
INTO [DbJamaica].[dbo].[CS_Usage_TEMP]
FROM [DbJamaica].[dbo].[CS_Usage]
Then you can use bcp on your staging table:
bcp DbJamaica.dbo.CS_Usage_TEMP out "C:\temp\CS_Usage.txt" /U.. /P.. -c -T
Next you can insert the data from the staging to the actual table, where you set the PK column as IDENTITY:
INSERT INTO DbJamaica.dbo.CS_Usage (csusUsageDate, csusType, csusTrack1, csusTrack2, csusTrack3, csusDateReaded, csusLoggedIn)
SELECT csusUsageDate, csusType, csusTrack1, csusTrack2, csusTrack3, csusDateReaded, csusLoggedIn
FROM DbJamaica.dbo.CS_Usage_TEMP
And finally cleanup:
DROP TABLE DbJamaica.dbo.CS_Usage_TEMP
Your source file should match the target table structure. Which means, the number of columns in the table should match the number of column in your csv/txt (source file).
In your case, even if your PK column is the identity column you must have the column in the source file. SQL server will take care of identifying the column as identity and it will ignore the values you put there. So you can either have any value or not, your bcp will work.
It's a different use case if you want to retain the identity values. Refer to -E argument in the documentation (bcp utility)
I have a table. For example:
CREATE TABLE temp AS SELECT * FROM range(20)
How i can create in sqlite a new database with one command with the table temp?
Since your in sqlite, in the command line just type:
sqlite3 datbase_name.db
This will create a new database called database_name.db, now to create your table to this database just type:
sqlite3 database_name.db "CREATE TABLE .......;"
Type your create table statement inside the quotes.
For more info, please take a look at their documentation.
I think that would depend on the DBMS but this may be what you're looking for.
COPY Command Syntax
"COPY {FROM database | TO database | FROM database TO database} {APPEND|CREATE|INSERT|REPLACE} destination_table [(column, column, column, ...)] USING query"
http://docs.oracle.com/cd/B19306_01/server.102/b14357/apb.htm
I actually don't know how I've done it : but I got a nameless table into my postgres DB. Needless to say that such a table is problematic, it doesn't get erased, neither could you change it in any way.
Here is a picture of how it looks on the pgAdmin explorer :
As for its declaration, it goes like this :
CREATE TABLE
(
_id_ integer NOT NULL DEFAULT nextval('bu_b__id__seq'::regclass),
pt_utilisateur integer,
pt_date date,
lon double precision,
lat double precision,
CONSTRAINT PRIMARY KEY (_id_)
)
So, one simple question : how can I delete this table (since trying to delete it comes to using its name... which doesn't exist !) ?
Regards.
If this is NOT about a table which name is not displayable due to encoding / font issues you can do this:
Determine the table OID from the system catalogues.
Set a new table name in the system catalogues to a temporary name. Use the OID as an identifer
Re-rename the table to something useful (or just drop it).
This of course assumes, that you have superuser access to the database.
The first step - determine the OID from the catalogue:
SELECT oid, relname FROM pg_class WHERE length(relname) <= 1;
If you don't get any result the table has a name - it is just not displayable. In this case just skip the WHERE part and look with your eyes.
The second step - change the system catalogue:
UPDATE pg_class SET relname = 'foo' WHERE oid = /*OID from step 1*/;
In any case - don't stop here, you must proceed.
The third step - cleanup
PostgreSQL stores the table name in more than one place (e.g. pg_type). Some tools might depend on this. To clean this up you must either drop the table (and hence the unrenamed stuff) OR you must rename the table again using the official tools.
ALTER TABLE foo RENAME TO /*somthing real*/;
or
DROP TABLE foo;
The usual warnings: You are munching in the internals of PostgreSQL - don't do silly things and do it on your own risk :-)
I don't use Postgresql but I guess you can delete that table by following steps:
Take the sql dump
Delete the database
Remove that nameless table from the dump and restore sql dump