I have a database where each table has two columns that indicate when a table row has been created (gen_date) or modified (mod_date).
I would like to export this complete database (structure) but would like to include only the data modified after a given date.
At the moment I use mysqldump to export the data:
mysqldump --user=username --password=password mydatabase
This exports all data and the table definitions.
Question: Is it possible to export only data with mod_date > some given date?
I was thinking about "cloning" the database and then delete all outdated data and use mysqldump to export the remaining data.
Example of a table:
CREATE TABLE `brand` (
`id` int(11) UNSIGNED NOT NULL,
`brand` tinytext CHARACTER SET utf8 COLLATE utf8_unicode_ci,
`gen_date` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`mod_date` datetime NOT NULL DEFAULT '0000-00-00 00:00:00'
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
Try something like this
mysqldump --user=username --password=password mydatabase mytable --no_create_info --where mod_date >= '2016-03-01' > mytable.sql
Related
I have a text file with 6 columns and 200 million rows and none of them is unique. I'd like to import them into a table in SQL Server and want to define an Identity column as primary key.
Therefore I created the following table first:
CREATE TABLE dbo.Inventory
(
ProductID NUMERIC(18,3) NOT NULL,
RegionID NUMERIC(18,3) NULL,
ShopCode INT NULL,
QTY FLOAT NULL,
OLAPDate VARCHAR(6) NULL,
R Float NULL,
ID BIGINT NOT NULL PRIMARY KEY IDENTITY(1,1)
)
Then I use the below command for importing the text file into the table:
bcp ETLDB.dbo.Inventory in D:\SCM\R.txt -T -b 10000 -t "," -c -e D:\SCM\Errors.txt
and I got these errors:
I am not sure if the errors are because of the identity id column which is in my table design and not in my original text file or not. Because when I delete the identity id key from the table, the bcp works fine. But I want the bcp defines the identity id in the process of importing my file into table.
The sample text file:
Any help would be appreciated.
Create a view that looks like what you want to load into and load into that
CREATE VIEW dbo.Inventory_Stage
AS SELECT
ProductID,
RegionID,
ShopCode,
QTY,
OLAPDate,
R Float
FROM Inventory
Now load into Inventory_Stage instead of Inventory
also, use -F to start loading at the second row, because the first row has column names
bcp ETLDB.dbo.Inventory_Stage in -F 1 D:\SCM\R.txt -T -b 10000 -t "," -c -e D:\SCM\Errors.txt
Also, seriously consider if you want to use float. For your sample data I recommend NUMERIC(19,6)
There is a workaround I tried for a similar case.
Step 1:
Create a Table with the columns available to your CSV/TXT file.
Step 2:
Push the data using the BCP script.
bcp dbo.<tablename> in <file location in local folder> -S <server_name> -d <database_name> -U <username> -P <password> -b 20000 -q -c -t"<column delimiter>"
Step 3:
Once the data is available on your destination table you can now alter the table with the below SQL command:
ALTER TABLE <Table Name>
ADD <Identity Column> BIGINT IDENTITY(1,1)
Adding Few SQL Statement to help you understand Update-Insert Script for Incremental Load.
CREATE TABLE Employees
(
ID INT IDENTITY(1,1),
Name VARCHAR(100),
Salary INT,
InsertDate DATETIME,
UpdateDate DATETIME
)
INSERT INTO Employees
VALUES
('Kristeen',1420,NULL,NULL)
,('Ashley',2006,NULL,NULL)
,('Julia',2210,NULL,NULL)
,('Maria',3000,NULL,NULL)
CREATE PROCEDURE dbo.InsertOrUpdateEmployee
#Name VARCHAR(100),
#Salary INT
AS BEGIN
CREATE TABLE #tmpData
(
Name VARCHAR(50),
Salary INT
)
INSERT INTO #tmpData(Name,Salary)
VALUES(
#Name,
#Salary
)
UPDATE A
SET A.Name = B.Name,
A.Salary = B.Salary,
A.updatedate = GETDATE(),
A.IsNewRecord = 0
FROM Employees A
JOIN #tmpData B
ON A.Name = B.Name
AND A.Salary = B.Salary
INSERT INTO Employees
(
Name,
Salary,
InsertDate,
IsNewRecord
)
SELECT
S.Name,
S.Salary,
GETDATE(),
1
FROM #tmpData S
LEFT JOIN Employees D
ON S.Name = D.Name
AND S.Salary = D.Salary
WHERE D.Name IS NULL
AND D.Salary IS NULL
DROP TABLE #tmpData
END
EXEC InsertOrUpdateEmployee 'Gaurav',4500000
You need to modify a bit with the code above as the above code is to insert the data through SP parameter, but in your case, you might need to use the Source Table in place of a temporary table and in the end you can truncate the source table after moving the complete data into the Destination table.
The issue is that you are trying not passing last column, which is an INT column.
"-E Specifies that identity value or values in the imported data file
are to be used for the identity column. If -E is not given, the
identity values for this column in the data file being imported are
ignored."
You have three options...
Add an INT column to the source data as the first row and have it incremented like an IDENTITY would be incremented and continue to pass the -E option. Doing this will allow the data from the source to be used as the IDENTITY column.
Add a random INT to the last column of your source data, say 1 for every row, then do not pass in the -E. According to the documentation, when -E is not provided it will ignore the values for the identity column and start at the currently seeded value and auto-increment.
Leverage a format file to specify which columns from your data file go into which columns in our SQL table.
How to specify the format file
How to construct a format file
Updated Answer
When you don't have option to modify the source data, then please remove the identity columns and perform as below:
- Remove Identity Column from the table
- Do your Import
- After successful of import, please add the identity column as below:
Alter Table Names
Add Id_new BigInt Identity(1, 1)
Go
As Marc_s mentioned here
Don't BULK INSERT into your real tables directly.
I would always
insert into a staging table dbo.Employee_Staging (without the IDENTITY column) from the CSV file
possibly edit / clean up / manipulate your imported data
and then copy the data across to the real table with a T-SQL statement like:
INSERT INTO dbo.Employee(Name, Address)
SELECT Name, Address
FROM dbo.Employee_Staging
Useally if I want to add default value , I use this way
ALTER TABLE tab1 ADD coll datetime DEFAULT '11-01-15' NOT NULL
however I want to add default value as the getdate().
so If use the below I got the below error
ALTER TABLE tab1 ADD coll datetime DEFAULT GETDATE() NOT NULL
error: ALTER TABLE 'tab1' failed. Default cannot be an expression when adding non-NULL column 'tab1'. Use a default value instead.
what I want is the default value is the getdate(). how to do that when adding a new datetime column ?
I solved my question by adding the column to null
alter table tab1 add col1 datetime default getdate() null
then I modify it to not null
Use the below for Sybase :
ALTER TABLE table_name
ADD effective_date Datetime DEFAULT "3/16/2009 09:15" NOT NULL
ALTER TABLE table_name
REPLACE effective_date DEFAULT getdate()
How do include a function like getdate() in an insert statement for Hibernate query?
Lets say one of the properties of a mapped class is a datetime column how do put specify getdate() in order for the .save() method call to use the server timestamp?
Simple answer is don't include it in the insert statement and let SQL do it.
In the SQL table designer where you have the date column you are inserting into, set the Default Value to GETDATE() in the column properties and SQL Server will handle it for you.
Reference and sample: SQL Server GETDATE() Function
Table create script will look like this:
CREATE TABLE Orders
(
OrderId int NOT NULL PRIMARY KEY,
ProductName varchar(50) NOT NULL,
OrderDate datetime NOT NULL DEFAULT GETDATE()
)
I am converting some of my MySQL statements to DB2 database, but I faced a problem on the following query
CREATE TABLE RFX_EVENT_MAPPING (
EVENT_TYPE varchar(4) NOT NULL,
EVENT_DESC varchar(50) NOT NULL,
EVENT_CLASS varchar(50) default NULL,
OWNER varchar(6) default NULL,
LAST_UPDATE_TIME timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
LAST_UPDATE_USER varchar(20) NOT NULL
);
As you can see there is
LAST_UPDATE_TIME timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP
Which is not working so how can I achieve the same functionality with db2?
In DB2 9.7 for Linux, UNIX, Windows, IBM added the concept of a row change timestamp.
create table rcttest (
c1 int,
c2 char(10),
insert_ts timestamp not null with default current timestamp,
change_ts timestamp not null generated always for each row
on update as row change timestamp
);
I've got a table that collects forms submitted from our website, but for some reason, when they created the table, they didn't put a timestamp in the table. I want it to enter the exact date and time that the record was entered.
I know it's in there somewhere, but I can't seem to find how to set the default value (like in Access, you use getNow() or Now()) but I don't know where to put it.
For modifying an existing column in an existing table:
ALTER TABLE YourTable ADD CONSTRAINT DF_YourTable DEFAULT GETDATE() FOR YourColumn
This can also be done through the SSMS GUI.
Put your table in design view (Right click on table in object explorer->Design)
Add a column to the table (or click on the column you want to update if it already exists)
In Column Properties, enter (getdate()) in Default Value or
Binding field as pictured below
In that table in SQL Server, specify the default value of that column to be CURRENT_TIMESTAMP.
The datatype of that column may be datetime or datetime2.
e.g.
Create Table Student
(
Name varchar(50),
DateOfAddmission datetime default CURRENT_TIMESTAMP
);
While the marked answer is correct with:
ALTER TABLE YourTable ADD CONSTRAINT DF_YourTable DEFAULT GETDATE() FOR YourColumn
You should always be aware of timezones when adding default datetime values in to a column.
Say for example, this datetime value is designed to indicate when a member joined a website and you want it to be displayed back to the user, GETDATE() will give you the server time so could show discrepancies if the user is in a different locale to the server.
If you expect to deal with international users, it is better in some cases to use GETUTCDATE(), which:
Returns the current database system timestamp as a datetime value. The database time zone offset is not included. This value represents the current UTC time (Coordinated Universal Time). This value is derived from the operating system of the computer on which the instance of SQL Server is running.
ALTER TABLE YourTable ADD CONSTRAINT DF_YourTable DEFAULT GETUTCDATE() FOR YourColumn
When retrieving the values, the front end application/website should transform this value from UTC time to the locale/culture of the user requesting it.
Disallow Nulls on the column and set a default on the column of getdate()
/*Deal with any existing NULLs*/
UPDATE YourTable SET created_date=GETDATE() /*Or some sentinel value
'19000101' maybe?*/
WHERE created_date IS NULL
/*Disallow NULLs*/
ALTER TABLE YourTable ALTER COLUMN created_date DATE NOT NULL
/*Add default constraint*/
ALTER TABLE YourTable ADD CONSTRAINT
DF_YourTable_created_date DEFAULT GETDATE() FOR created_date
The syntax for this when creating a new table is:
CREATE TABLE MyTable
(
MYTableID INT IDENTITY(1,1),
CreateDate DATETIME NOT NULL CONSTRAINT DF_MyTable_CreateDate_GETDATE DEFAULT GETDATE()
)
This works for me...
ALTER TABLE [accounts]
ADD [user_registered] DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ;
This also works:
CREATE TABLE Example(
...
created datetime default GETDATE()
);
Or:
ALTER TABLE EXAMPLE ADD created datetime default GETDATE();
This worked for me. I am using SQL Developer with Oracle DB:
ALTER TABLE YOUR_TABLE
ADD Date_Created TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL;
Let's say you create a database table for a registration system.
IF OBJECT_ID('dbo.registration_demo', 'U') IS NOT NULL
DROP TABLE dbo.registration_demo;
CREATE TABLE dbo.registration_demo (
id INT IDENTITY PRIMARY KEY,
name NVARCHAR(8)
);
Now a couple people register.
INSERT INTO dbo.registration_demo (name) VALUES
('John'),('Jane'),('Jeff');
Then you realize you need a timestamp for when they registered.
If this app is limited to a geographically localized region, then you can use the local server time with GETDATE(). Otherwise you should heed Tanner's consideration for the global audience with GETUTCDATE() for the default value.
Add the column with a default value in one statement like this answer.
ALTER TABLE dbo.registration_demo
ADD time_registered DATETIME DEFAULT GETUTCDATE();
Let's get another registrant and see what the data looks like.
INSERT INTO dbo.registration_demo (name) VALUES
('Julia');
SELECT * FROM dbo.registration_demo;
id name time_registered
1 John NULL
2 Jane NULL
3 Jeff NULL
4 Julia 2016-06-21 14:32:57.767
To make it simpler to follow, I will summarize the above answers:
Let`s say the table is called Customer
it has 4 columns/less or more...
you want to add a new column to the table where every time when there is insert... then that column keeps a record of the time the event happened.
Solution:
add a new column, let`s say timepurchase is the new column, to the table with data type datetime.
Then run the following alter:
ALTER TABLE Customer ADD CONSTRAINT DF_Customer DEFAULT GETDATE() FOR timePurchase
In SQLPlus while creating a table it is be like as
SQL> create table Test
( Test_ID number not null,
Test_Date date default sysdate not null );
SQL> insert into Test(id) values (1);
Test_ID Test_Date
1 08-MAR-19