Here are the details:
The database has to be archived such that records older than 6 months can be copied to a new database and deleted from the main(production) database. The complexity here will be to copy all rows in all tables which have reference each other. After that, these copied rows from some of the tables (which are really huge and whose data is no more needed) will be deleted.
The postgres database is an Amazon RDS instance.
What is the best way to achieve this?
I was thinking either a Springboot application
OR
Have postgresql.conf invoke a shell script which invokes a sql batch.
For the second approach, I am not sure how to edit a amazon RDS postgresql.conf file and where to specify the shell script. Where would be the sql batch written? This is a little new to me, appreciate any pointers.
It will be much faster if you do everything server side instead of using a Springboot application. The problem is not dump/restore which you could easily do with pg_dump utility or psql -d dbname -t -A -F";" -c "SELECT * FROM yourdata WHERE cutdate<=current_timestamp-interval '6 months'" > output.csv
But you have to guarantee that everything that is exported is loaded into the second database and that you do not delete anything that has not been exported.
I would first SELECT a subset of primary keys into a temporary table. Then use server side COPY command to export the preselected keys (and all its dependencies)
COPY (SELECT d.* FROM yourdata d INNER JOIN temporal t WHERE d.pk=t.pk) To '/tmp/yourdata.csv' WITH CSV DELIMITER ',';
After all the export files have been generated
DELETE FROM yourdata WHERE pk IN (SELECT pk FROM temporal)
Then on the backup database do
COPY yourdata(column1,column2,column3) FROM '/tmp/yourdata.csv' DELIMITER ',' CSV
You can write a script that invokes all that commands on server side using psql command line tool and last move the imported files into a permanent location just in case something went wrong and you need to process them again.
See Save PL/pgSQL output from PostgreSQL to a CSV file and How to import CSV file data into a PostgreSQL table?
Related
My customers runs an very old (seems to me) Sybase 12.5.2 database. I want/need to export all tables from a database to multiple (for each table) flat (text) files. I have access to ISQL command line prompt with the admin user. I havent worked ever with an Sybase database before.
Sybase Adaptive Server Enterprise (ASE) allows multiple databases to be hosted. You don't specify whether only one of the databases in the database server needs to be exported or if all of them do.
For each database, the following query will list the names of the tables
select name from sysobjects where type = 'U'
Sybase ASE also comes with a tool called "bcp" which stands for "Bulk Copy". It is an easy way of creating a flat file of a table's contents.
bcp database.schema.table out file_name -c -U username -S server_name
It has more options that may be of interest, especially around field and row terminators. Documentation for the most relevant version (12.5.1) can be found here:
http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.dc30191_1251/html/utility/BABGCCIC.htm
i have been using BCP commands to export data from sybase environments.bcp is a command line utility which you can use it to export data from multiple types of databases
below is a very example and you can try it for
bcp Table Name out OUTPUT FILE PATH\FILENAME.dat -S SERVER NAME -U USERNAME -P PASSWORD -F Format -r row_terminator -e error output file path and name
You can create a batch file with such commands and do multiple exports on one hit.
If you have access to any ETL tool you can exporting the data using the same as well.
First forgive me for my English. It is a little bad. Second forgive my ignorance, i'm newiest in postgres
I'm having trouble when I try to up a backup database on another database. I need to dump the database just to get one table, but I only have the files that was in /var/lib/pgsql/data/base/
Here what I try:
I create a database named "test" with OID 227763 so I put the files of the old database to this new database with another OID. I fix the folder and files permissions, but when I log into "test" and run select * from pg_tables; the tables does not appears to me. And when I try to create the table on PhpPgAdmin, I got
ERROR: relation already exists
I'm trying to do this because I need to know which of this files is the table that i want. I will log into database and run SELECT oid,* from pg_class; to get the OID.
I found the old OID database in /var/lib/pgsql/data/global/pg_database
If anyone can help me, I thank you.
There are many ways to backup and restore an entire database or a single table. It sounds like you need to be using pgDump instead of working on individual files. A file level copy is likely to corrupt your database if not in backup mode and if not copying the entire thing + archive logs.
If you MUST copy it by files, make sure the database is shut down for maximum safety.
For me, if I had one table to backup, I'd use pg_dump
pg_dump -U {user-name} {source_db} -f {dumpfilename.sql}
you can use the -t flag to list a single table if you like.
I use a Wordpress plugin called 'Shopp'. It stores product images in the database rather than the filesystem as standard, I didn't think anything of this until now.
I have to move server, and so I made a backup, but restoring the backup is proving a horrible task. I need to restore one table called wp_shopp_assets which is 18MB.
Any advice is hugely appreciated.
Thanks,
Henry.
For large operations like this it is better to go to command line. phpMyAdmin gets tricky when lots of data is involved because there are all sorts of timeouts in PHP that can trip it up.
If you can SSH into both servers, then you can do a sequence like the following:
Log in to server1 (your current server) and dump the table to a file using "mysqldump" --- mysqldump --add-drop-table -uSQLUSER -pPASSWORD -h
SQLSERVERDOMAIN DBNAME TABLENAME > BACKUPFILE
Do a secure copy of that file from server1 to server2 using "scp" ---
scp BACKUPFILE USER#SERVER2DOMAIN:FOLDERNAME
Log out of server 1
Log into server 2 (your new server) and import that file into the new DB using "mysql" --- mysql -uSQLUSER -pPASSWORD DBNAME < BACKUPFILE
You will need to replace the UPPERCASE text with your own info. Just ask in the comments if you don't know where to find any of these.
It is worthwhile getting to know some of these command line tricks if you will be doing this sort of admin from time to time.
try HeidiSQL http://www.heidisql.com/
connect to your server and choose the database
go to menu "import > Load sql file" or simply paste the sql file into the sql tab
execute sql (F9)
HeidiSQL is an easy-to-use interface
and a "working-horse" for
web-developers using the popular
MySQL-Database. It allows you to
manage and browse your databases and
tables from an intuitive Windows®
interface.
EDIT: Just to clarify. This is a desktop application, you will connect to your database server remotely. You won't be limited to php script max runtime, or upload size limit.
use bigdupm.
create a folder on your server witch is not easy to guess like "BigDump_D09ssS" or w.e
Download the http://www.ozerov.de/bigdump.php importer file and add them to that directory after reading the instructions and filling out your config information.
FTP The .SQL File to that folder along side the bigdump script and go to your browser and navigate to that folder.
Selecting the file you uploaded will start importing the SQL is split chunks and would be a much faster method!
Or if this is an issue i reccomend the other comment about SSH And mysql -u -p -n -f method!
Even though this is an old post I would like to add that it is recommended to not use database-storage for images when you have more than like 10 product(image)s.
Instead of exporting and importing such a huge file it would be better to transfer the Shopp installation to file-storage for images before transferring.
You can use this free plug-in to help you. Always backup your files and database before performing this action.
What I do is open the file in a code editor, copy and paste into a SQL window within phpmyadmin. Sounds silly, but I swear by it via large files.
I was thinking about using bcp command to solve the user authentication, but does a bcp command capable to import to a table in my database? By the way, I am using SQL Server 2000 environment.
Here's the code I have got so far:
SET #Command = 'bcp "SELECT vwTest.* from [myserver\sql].test.dbo.vwTest" queryout dbo.Test -C ACP -c -r \n -t ";" -S myserver\sql -Umyuser -Puser1'
EXEC master.dbo.xp_cmdshell #Command
Based on the comparison of BCP, BULK INSERT, OPENROWSET (infer Linked Server) here:
...the bcp utility runs out-of-process. To move data across process memory spaces, bcp must use inter-process data marshaling. Inter-process data marshaling is the process of converting parameters of a method call into a stream of bytes. This can add significant load to the processor. However, because bcp [both] parses the data and [converts the] data into [the] native storage format in the client process, they can offload parsing and data conversion from the SQL Server process.
...bcp possibly isn't the most efficient means of transferring data. You might be better off to:
Create a linked server instance to the other database
Use INSERT statements, so that the tables are populated based on records from the database exposed in the linked server instance.
Besides potentially being more efficient, you only need to setup the linked server instance once versus running BCP to create output scripts every time you want to move data.
Mind that the linked server instance is based on a user on the other database, so permissions to the other database are based on that users' permissions.
SURE !!
Use this command (adopt it for your needs) on your source machine:
bcp database.dbo.viewname out c:\temp\viewname.bcp
and then import the data back into your destination system using:
bcp newdatabase.dbo.importtable in c:\temp\viewname.bcp
-c -S(servername) -U(username) -P(password)
That should grab the contents of your "viewname" from the source server, put it in a temporary file, and insert that file back into the new database on the new server.
Typically, you would load those data rows into a new, temporary staging table, and form there, use T-SQL or other means to insert that data into your actual tables.
Check out the MSDN documentation on bcp in SQL Server 2000 for details on all those switches and their meanings.
A while back I needed to parse a bunch of Serve-U FTP log files and store them in a database so people could report on them. I ended up developing a small C# app to do the following:
Look for all files in a dir that have not been loaded into the db (there is a table of previously loaded files).
Open a file and load all the lines into a list.
Loop through that list and use RegEx to identify the kind of row (CONNECT, LOGIN, DISCONNECT, UPLOAD, DOWNLOAD, etc), parse it into a specific kind of object corresponding to the kind of row and add that obj to another List.
Loop through each of the different object lists and write each one to the associated database table.
Record that the file was successfully imported.
Wash, rinse, repeat.
It's ugly but it got the job done for the deadline we had.
The problem is that I'm in a DBA role and I'm not happy with running a compiled app as the solution to this problem. I'd prefer something more open and more DBA-oriented.
I could rewrite this in PowerShell but I'd prefer to develop an SSIS package. I couldn't find a good way to split input based on RegEx within SSIS the first time around and I wasn't familiar enough with SSIS. I'm digging into SSIS more now but still not finding what I need.
Does anybody have any suggestions about how I might approach a rewrite in SSIS?
I have to do something similar with Exchange logs. I have yet to find an easier solution utilizing an all SSIS solution. Having said that, here is what I do:
First I use logparser from Microsoft and the bulk copy functionality of sql2005
I copy the log files to a directory that I can work with them in.
I created a sql file that will parse the logs. It looks similar to this:
SELECT TO_Timestamp(REPLACE_STR(STRCAT(STRCAT(date,' '), time),' GMT',''),'yyyy-M-d h:m:s') as DateTime, [client-ip], [Client-hostname], [Partner-name], [Server-hostname], [server-IP], [Recipient-Address], [Event-ID], [MSGID], [Priority], [Recipient-Report-Status], [total-bytes], [Number-Recipients], TO_Timestamp(REPLACE_STR([Origination-time], ' GMT',''),'yyyy-M-d h:m:s') as [Origination Time], Encryption, [service-Version], [Linked-MSGID], [Message-Subject], [Sender-Address] INTO '%outfile%' FROM '%infile%' WHERE [Event-ID] IN (1027;1028)
I then run the previous sql with logparser:
logparser.exe file:c:\exchange\info\name_of_file_goes_here.sql?infile=c:\exchange\info\logs\*.log+outfile=c:\exchange\info\logs\name_of_file_goes_here.bcp -i:W3C -o:TSV
Which outputs a bcp file.
Then I bulk copy that bcp file into a premade database table in SQL server with this command:
bcp databasename.dbo.table in c:\exchange\info\logs\name_of_file_goes_here.bcp -c -t"\t" -T -F 2 -S server\instance -U userid -P password
Then I run queries against the table. If you can figure out how to automate this with SSIS, I'd be glad to hear what you did.