I am using Dbeaver 22.1.4 on Windows 10 Home Single 64bit. My RAM is 8 Gb. I want to insert 16 millions data from one server to another using dblink (All servers are Linux Ubuntu, running Postgresql 12). The query looks like this ( I split it to 5000 first for testing) :
INSERT INTO table_server1 ([some 115 columns])
SELECT *
FROM dblink('myconn',$MARK$
SELECT [some 115 columns]
FROM public.table_server2 limit 5000
$MARK$) AS t1 (
id varchar, col1 varchar, col2 varchar, col3 integer, ... , col115 varchar);
It only inserts 1000 data which takes 1-2 seconds. It says "Updated rows : 1000" on the result window. There is no error as such.
What happen ? How can I insert all data ? I have edit the config file by modifying the max memory to 2 GB : -Xmx2048m
do you insist on using Dbeaver and/or dblink? If not, and you can connect to terminal on either postgres server, you can do this very fast (no splitting needed) and easily without "middle man" (your machine), directly server-to-server:
psql -d sourcedb -c "\copy (SELECT [some 115 columns] FROM public.table_server2) TO STDOUT" | psql -d targetdb -c "\copy table_server1 FROM STDIN"
Of course you need to specify host, user/password for both sides psql
Related
Using BCP (Bulk Copy Program) / bulk insert; can we send data from server1 to another server2 database?
SERVER_1 SERVER_2
| |
DATBASE_1 DATABASE_2
| |
TABLE_1 (5 COLUMNS)
|_____________________________> TABLE_1 (ID, NAME)
|_____________________________> TABLE_2 (AGE, GENDER)
|_____________________________> TABLE_3 (ADDR)
Run BCP once to output all 5 columns of Table_1, then run BCP 3 times to load the 3 different tables on Server_2, using 3 different format files to pick which columns to load into each table. See the BCP Utility documentation for more info, and there's a lot of info on using format files too.
I'm using bcp to insert a CSV file in my SQL table, but i'm having weird results.
All the environment is on Linux, I create the CSV file with IFS=\n in Bash. BCP is running from Bash too like this:
bcp Table1 in "./file.csv" -S server_name -U login_id-P password -d databse_name -c -t"," -r"0x0A"
The file.csv is about 50k rows. BCP tells me he copied 25K rows. No warnings, no errors. I have tried on a smaller sample with only 2 rows and here is the result:
file.csv data:
A.csv, A, B
C.csv, C, D
My table Table1:
CREATE TABLE Table1 (
ID INT NOT NULL IDENTITY(1,1) PRIMARY KEY,
a VARCHAR(8),
b VARCHAR(8),
c VARCHAR(8)
FOREIGN KEY (a) REFERENCES Table2(a)
);
Table2 is a very basic table just like Table1
What I can see in Microsoft SQL Server Manager Studio when i do SELECT * FROM Table1
ID | a | b | c |
1 A BC C.csv E
I export with bcp like this:
bcp Table1 out"./file_export.csv" -S server_name -U login_id-P password -d databse_name -c -t"," -r"0x0A"
What I get in file_export.csv
1, A, B
C, C.csv, D, E
I've check on NotePad++ any strange End Of File, they are all EF (50k Rows) each values are separated by comas.
UPDATE:
Changing the options -t and -r does not change anything. So my CSV is well formatted. I think the issue the fact that I'm trying to import x columns in a tables of x+1 fields (The auto-incremented ID).
So i'm using the simplest command for bcp
bcp Table1 in "./file.csv" -S server_name -U login_id-P password -d databse_name -c
Here is what i get in my SELECT * FROM Table1;
ID a b c
1 A BC.csv C,D
Here is what I expected:
1,A.csv,A,B
2,C.csv,C,D
Why does my first field is "eaten" and replace by 1 (ID)? Which makes everything swift to the right.
I have an extremely large database I need to send to the developer, the table has over 120million rows. the developer says he only needs about 10,000 or so rows so I was going to use the sqlcmd -S -d -Q "select top 10000 * from table" -s "," -o "C:\temp\filename.csv"
I decided rather than truncate immediately I would script out the table, rename and test bulk inserting, I tried using
bulk insert tablename from 'c:\temp\filename.csv'
with (
fieldterminator = ',',
rowterminator = '\n'
)
this ends in "Bulk load data conversion error (truncation) for row 1..." error. I also tried in import/export wizard and it fails for the same problem (truncation). increasing the size of the field lengths, solves the problem but I really am having problems understanding why I need to do this. Its the same data from the same table, it should bulk insert right back in?!?
also the problem is happening on every column in the table and by varying lengths, so there is no column with the same number of chars I have to add. all the columns are varchar data type. could the sqlcmd be inserting some kind of corruption in the file? I have tried to look for a problem, I also tried rtrim(ltrim(columname)) to make sure there is no whitespace but I'm not sure this is how it works. I'm using sql server 2012 if this helps.
thanks
You should look into BCP Queryout and BULK INSERT options. Use NATIVE format if you're going from SQL to SQL.
(BCP is command-line):
bcp "select top(10000) * from table" queryout "OUTPUTFILENAME.DAT" -S serverInstanceName -d databaseName -T -n
The Bulk Insert command is SQL (not command line):
bulk insert table from 'path\and\OUTPUTFILENAME.DAT' with (keepidentity,datafiletype = 'native');
(If the table doesn't have an identity, you can eliminate keepidentity,
I'm running BCP against a table of 112m rows to select approx 1.6m
The table definition has a 25 x nvarchar(10) , 20 x INT columns and 2 Bit columns. It also has 1 non persisted compute column, which is nvarch(14) casting 2 int columns into a string. It has a clustered index on the computed column and one other index on an INT column (which is used in the where clause)
bcp "select * from db.dbo.table where src = 1002" queryout F:/path/1002_1638762.dat -n -U ausername -P *********
judging by the performance - 20 minutes - it looks as tho the query is running a scan, This conclusion is also supported by the fact that when the output file reaches its end size, the read continues for sometime.
Does bcp use indexes? I would be expecting this to run much faster using an index seek and key lookup?
Can anyone recommend any performance improvements?
I haven't been able to answer the question "does BCP queryout use indexes" - ostensibly it does seem to, but whether it creating optimal execution plans I didn't find an answer to.
However, I did find a performance enhancement with a change to my approach:
select the rows into a new table
bcp out the new table
drop the new table
Completes in less than 2 minutes.
I have two database, which are "OLD_DB" and "NEW_DB". I am trying to move data from OLD_DB to NEW_DB. I use script like below :
INSERT INTO [NEW_DB].[dbo].[Table1](Field1,Field2,Field3) SELECT Field1,Field2,Field3 FROM [Linkserver].[OLD_DB].[dbo].[Table1] WHERE Field4 = 1234
[Linkserver]: I use linked server because OLD_DB is in different server.
I added script above in SQL SERVER Agent, so it will trigger itself on 5:00PM. At 5:00PM, my users were still working with OLD_DB. After they(users) left, I came to check my NEW_DB by using script like below :
SELECT Field1,Field2,Field3 FROM [New_DB].[dbo].[Table1]
there are only 291 records. And I try to use SELECT statement in OLD DB
SELECT Field1,Field2,Field3 FROM [Linkserver].[OLD_DB].[dbo].[Table1] WHERE Field4 = 1234
There are 431 records. then it turn into question, why does number of record in OLD_DB and NEW_DB are different ?
NOTE : I use SQL SERVER 2012 for both server.