i received many .dat.gz files from a pg_dump at a postgres database.
i would like to convert it to a .sqli file to import it in a software like dbeaver.
Is there a way to do this? Unfortunately I can't find the answer by googling.
thanks
Take the below as suggestions and test before using in production.
At a guess is was some form of pg_dump -Fd ... per pg_dump.
If that is the case and all the files are there you can either do:
pg_restore -f dump.sql /dat_dir
which will create a plain version of the dump or:
pg_restore -d some_db -C /dat_dir
which will restore directly. In this case using -d to connect and then -C to create the database that was dumped.
For more information see pg_restore
Related
I am in the process of doing export and import with postgres database.
I had used the following command to take the backup of postgres db
C:\dirs> pg_dump -U postgres -p 15432 -W -F t cgate-next-demo > .\dbexport_10th_February_2022.tar
Password:*****
I have unzipped dbexport_10th_February_2022.tar file and proceeded with database import. As a initial step, I had dropped the database.
#drop database if exists "cgate-next-demo";
And I had recreated the empty database.
#create database "cgate-next-demo";
In order to do this, I have logged in to psql once,
C:\dirs> psql -U postgres -p 15432
Password for user postgres:*****
postgres=#
For database import I have used the following command.
C:\dirs> psql -U postgres -p 15432 -d cgate-next-demo <restore.sql
While I do that I have got the following error. I took this excerpt from console logs.
ERROR: could not open file "$$PATH$$/6052.dat" for reading: No such
file or directory HINT: COPY FROM instructs the PostgreSQL server
process to read a file. You may want a client-side facility such as
psql's \copy.
Can someone guide on what would've caused this issue.
You are doing this in the wrong fashion. Rather than unpacking the archive, pass it as argument to pg_restore. That will do everything for you.
I was trying to restore an SEC form preloaded database from Arelle.org using postgres. Below is the link:
http://arelle.org/documentation/xbrl-database/
It's the one towards the bottom of the page where it says "Preloaded Database".
I was able to download the file, but unable to gunzipped it at first. So, I copied the file and renamed it with .gz extension instead of .gzip. Then, I was able to gunzip it, but noot sure if that affects the file.
After that I tried the following command on postgress to restore the database in the database that I created:
psql -U username -d mydb -f secfile.pg (no luck)
I also tried:
pg_restore -C -d mydb secfile.pg (also no luck)
I am not sure if it's because I copied and renamed the file. But, I'd really appreciate it if anyone could help.
I'm looking to copy a production PostgreSQL database to a development server. What's the quickest, easiest way to go about doing this?
You don't need to create an intermediate file. You can do
pg_dump -C -h localhost -U localuser dbname | psql -h remotehost -U remoteuser dbname
or
pg_dump -C -h remotehost -U remoteuser dbname | psql -h localhost -U localuser dbname
using psql or pg_dump to connect to a remote host.
With a big database or a slow connection, dumping a file and transfering the file compressed may be faster.
As Kornel said there is no need to dump to a intermediate file, if you want to work compressed you can use a compressed tunnel
pg_dump -C dbname | bzip2 | ssh remoteuser#remotehost "bunzip2 | psql dbname"
or
pg_dump -C dbname | ssh -C remoteuser#remotehost "psql dbname"
but this solution also requires to get a session in both ends.
Note: pg_dump is for backing up and psql is for restoring. So, the first command in this answer is to copy from local to remote and the second one is from remote to local. More -> https://www.postgresql.org/docs/9.6/app-pgdump.html
pg_dump the_db_name > the_backup.sql
Then copy the backup to your development server, restore with:
psql the_new_dev_db < the_backup.sql
Use pg_dump, and later psql or pg_restore - depending whether you choose -Fp or -Fc options to pg_dump.
Example of usage:
ssh production
pg_dump -C -Fp -f dump.sql -U postgres some_database_name
scp dump.sql development:
rm dump.sql
ssh development
psql -U postgres -f dump.sql
If you are looking to migrate between versions (eg you updated postgres and have 9.1 running on localhost:5432 and 9.3 running on localhost:5434) you can run:
pg_dumpall -p 5432 -U myuser91 | psql -U myuser94 -d postgres -p 5434
Check out the migration docs.
pg_basebackup seems to be the better way of doing this now, especially for large databases.
You can copy a database from a server with the same or older major version. Or more precisely:
pg_basebackup works with servers of the same or an older major version, down to 9.1. However, WAL streaming mode (-X stream) only works with server version 9.3 and later, and tar format mode (--format=tar) of the current version only works with server version 9.5 or later.
For that you need on the source server:
listen_addresses = '*' to be able to connect from the target server. Make sure port 5432 is open for that matter.
At least 1 available replication connection: max_wal_senders = 1 (-X fetch), 2 for -X stream (the default in case of PostgreSQL 12), or more.
wal_level = replica or higher to be able to set max_wal_senders > 0.
host replication postgres DST_IP/32 trust in pg_hba.conf. This grants access to the pg cluster to anyone from the DST_IP machine. You might want to resort to a more secure option.
Changes 1, 2, 3 require server restart, change 4 requires reload.
On the target server:
# systemctl stop postgresql#VERSION-NAME
postgres$ pg_basebackup -h SRC_IP -U postgres -D VERSION/NAME --progress
# systemctl start postgresql#VERSION-NAME
Accepted answer is correct, but if you want to avoid entering the password interactively, you can use this:
PGPASSWORD={{export_db_password}} pg_dump --create -h {{export_db_host}} -U {{export_db_user}} {{export_db_name}} | PGPASSWORD={{import_db_password}} psql -h {{import_db_host}} -U {{import_db_user}} {{import_db_name}}
Run this command with database name, you want to backup, to take dump of DB.
pg_dump -U {user-name} {source_db} -f {dumpfilename.sql}
eg. pg_dump -U postgres mydbname -f mydbnamedump.sql
Now scp this dump file to remote machine where you want to copy DB.
eg. scp mydbnamedump.sql user01#remotemachineip:~/some/folder/
On remote machine run following command in ~/some/folder to restore the DB.
psql -U {user-name} -d {desintation_db}-f {dumpfilename.sql}
eg. psql -U postgres -d mynewdb -f mydbnamedump.sql
Dump your database : pg_dump database_name_name > backup.sql
Import your database back: psql db_name < backup.sql
I struggled quite a lot and eventually the method that allowed me to make it work with Rails 4 was:
on your old server
sudo su - postgres
pg_dump -c --inserts old_db_name > dump.sql
I had to use the postgres linux user to create the dump. also i had to use -c to force the creation of the database on the new server. --inserts tells it to use the INSERT() syntax which otherwise would not work for me :(
then, on the new server, simpy:
sudo su - postgres
psql new_database_name < dump.sql
to transfer the dump.sql file between server I simply used the "cat" to print the content and than "nano" to recreate it copypasting the content.
Also, the ROLE i was using on the two database was different so i had to find-replace all the owner name in the dump.
Let me share a Linux shell script to copy your table data from one server to another PostgreSQL server.
Reference taken from this blog:
Linux Bash Shell Script for data migration between PostgreSQL Servers:
#!/bin/bash
psql \
-X \
-U user_name \
-h host_name1 \
-d database_name \
-c "\\copy tbl_Students to stdout" \
| \
psql \
-X \
-U user_name \
-h host_name2 \
-d database_name \
-c "\\copy tbl_Students from stdin"
I am just migrating the data; please create a blank table at your destination/second database server.
This is a utility script. Further, you can modify the script for generic use something like by adding parameters for host_name, database_name, table_name and others
Here is an example using pg_basebackup
I chose to go this route because it backs up the entire database cluster (users, databases, etc.).
I'm posting this as a solution on here because it details every step I had to take, feel free to add recommendations or improvements after reading other answers on here and doing some more research.
For Postgres 12 and Ubuntu 18.04 I had to do these actions:
On the server that is currently running the database:
Update pg_hba.conf, for me located at /etc/postgresql/12/main/pg_hba.conf
Add the following line (substitute 192.168.0.100 with the IP address of the server you want to copy the database to).
host replication postgres 192.168.0.100/32 trust
Update postgresql.conf, for me located at /etc/postgresql/12/main/postgresql.conf. Add the following line:
listen_addresses = '*'
Restart postgres:
sudo service postgresql restart
On the host you want to copy the database cluster to:
sudo service postgresql stop
sudo su root
rm -rf /var/lib/postgresql/12/main/*
exit
sudo -u postgres pg_basebackup -h 192.168.0.101 -U postgres -D /var/lib/postgresql/12/main/
sudo service postgresql start
Big picture - stop the service, delete everything in the data directory (mine is in /var/lib/postgreql/12). The permissions on this directory are drwx------ with user and group postgres. I could only do this as root, not even with sudo -u postgres. I'm unsure why. Ensure you are doing this on the new server you want to copy the database to! You are deleting the entire database cluster.
Make sure to change the IP address from 192.168.0.101 to the IP address you are copying the database from. Copy the data from the original server with pg_basebackup. Start the service.
Update pg_hba.conf and postgresql.conf to match the original server configuration - before you made any changes adding the replication line and the listen_addresses line (in my care I had to add the ability to log-in locally via md5 to pg_hba.conf).
Note there are considerations for max_wal_senders and wal_level that can be found in the documentation. I did not have to do anything with this.
If you are more comfortable with a GUI, you can use the pgAdmin software.
Connect to your source and destination servers
Right-click on the source db > backup
Right-click on the destination server > create > database. Use the same properties as the source db (you can see the properties of the source db by right-click > properties)
Right-click on the created db > restore.
I'm working on PostgreSQL 8.4 and I'd like to do backup and restore (from Ubuntu 11.10 to Ubuntu 12.4)
I want to include all partitions, clusters, roles and stuff.
My commands:
Back up:
dumb_all > filename
Compress:
zip -f mybackup
Uncompress and restore:
sudo gunzip -c /home/ubuntu/Desktop/backupFile.zip | psql -U postgres
The issue is in the restore process, I got an error
invalid command \.
ERROR: syntax error at or near "2"
LINE 1: 2 2 1
^
invalid command \.
ERROR: syntax error at or near "1"
LINE 1: ...
^
out of memory
Plus, the tables with partitions did not restored. also some tables restored without any data!
Please help!
EDIT
I used pgAdmin to do the back up, using the "backup server" option.
If you did used zip to compress the output, then you should use unzip do uncompress it, not gunzip, they use different formats/algorithms.
I'd suggest you to use gzip and gunzip only. For instance, if you generated a backup named mybackup.sql, you can gzip it with:
gzip mybackup.sql
It will generate a file named mybackup.sql.gz. Then, to restore, you can use:
gunzip -c mybackup.sql.gz | psql -U postgres
Also, I'd suggest you to avoid using pgAdmin to do the dump. Not that it can't do, it is just that you can't automatize it, you can easily use pg_dumpall the same way:
pg_dumpall -U postgres -f mybackup.sql
You can either dump and compress without intermediate files using pipe:
pg_dumpall -U postgres | gzip -c > mybackup.sql.gz
BTW, I'd really suggest you avoiding pg_dumpall and use pg_dump with custom format for each database, as with that you already get the result compressed and easier to use latter. But pg_dumpall is ok for small databases.
We are switching hosts and the old one provided a SQL dump of the PostgreSQL database of our site.
Now, I'm trying to set this up on a local WAMP server to test this.
The only problem is that I don't have an idea how to import this database in the PostgreSQL 9 that I have set up.
I tried pgAdmin III but I can't seem to find an 'import' function. So I just opened the SQL editor and pasted the contents of the dump there and executed it, it creates the tables but it keeps giving me errors when it tries to put the data in it.
ERROR: syntax error at or near "t"
LINE 474: t 2011-05-24 16:45:01.768633 2011-05-24 16:45:01.768633 view...
The lines:
COPY tb_abilities (active, creation, modtime, id, lang, title, description) FROM stdin;
t 2011-05-24 16:45:01.768633 2011-05-24 16:45:01.768633 view nl ...
I've also tried to do this with the command prompt but I can't find the command that I need.
If I do
psql mydatabase < C:/database/db-backup.sql;
I get the error
ERROR: syntax error at or near "psql"
LINE 1: psql mydatabase < C:/database/db-backu...
^
What's the best way to import the database?
psql databasename < data_base_dump
That's the command you are looking for.
Beware: databasename must be created before importing.
Have a look at the PostgreSQL Docs Chapter 23. Backup and Restore.
Here is the command you are looking for.
psql -h hostname -d databasename -U username -f file.sql
I believe that you want to run in psql:
\i C:/database/db-backup.sql
That worked for me:
sudo -u postgres psql db_name < 'file_path'
I'm not sure if this works for the OP's situation, but I found that running the following command in the interactive console was the most flexible solution for me:
\i 'path/to/file.sql'
Just make sure you're already connected to the correct database. This command executes all of the SQL commands in the specified file.
Works pretty well, in command line, all arguments are required, -W is for password
psql -h localhost -U user -W -d database_name -f path/to/file.sql
Just for funsies, if your dump is compressed you can do something like
gunzip -c filename.gz | psql dbname
As Jacob mentioned, the PostgreSQL docs describe all this quite well.
make sure the database you want to import to is created, then you can import the dump with
sudo -u postgres -i psql testdatabase < db-structure.sql
If you want to overwrite the whole database, first drop the database
# be sure you drop the right database !!!
#sudo -u postgres -i psql -c "drop database testdatabase;"
and then recreate it with
sudo -u postgres -i psql -c "create database testdatabase;"
Follow the steps:
Go to the psql shell
\c db_name
\i path_of_dump [eg:-C:/db_name.pgsql]
I tried many different solutions for restoring my postgres backup. I ran into permission denied problems on MacOS, no solutions seemed to work.
Here's how I got it to work:
Postgres comes with Pgadmin4. If you use macOS you can press CMD+SPACE and type pgadmin4 to run it. This will open up a browser tab in chrome.
If you run into errors getting pgadmin4 to work, try killall pgAdmin4 in your terminal, then try again.
Steps to getting pgadmin4 + backup/restore
1. Create the backup
Do this by rightclicking the database -> "backup"
2. Give the file a name.
Like test12345. Click backup. This creates a binary file dump, it's not in a .sql format
3. See where it downloaded
There should be a popup at the bottomright of your screen. Click the "more details" page to see where your backup downloaded to
4. Find the location of downloaded file
In this case, it's /users/vincenttang
5. Restore the backup from pgadmin
Assuming you did steps 1 to 4 correctly, you'll have a restore binary file. There might come a time your coworker wants to use your restore file on their local machine. Have said person go to pgadmin and restore
Do this by rightclicking the database -> "restore"
6. Select file finder
Make sure to select the file location manually, DO NOT drag and drop a file onto the uploader fields in pgadmin. Because you will run into error permissions. Instead, find the file you just created:
7. Find said file
You might have to change the filter at bottomright to "All files". Find the file thereafter, from step 4. Now hit the bottomright "Select" button to confirm
8. Restore said file
You'll see this page again, with the location of the file selected. Go ahead and restore it
9. Success
If all is good, the bottom right should popup an indicator showing a successful restore. You can navigate over to your tables to see if the data has been restored propery on each table.
10. If it wasn't successful:
Should step 9 fail, try deleting your old public schema on your database. Go to "Query Tool"
Execute this code block:
DROP SCHEMA public CASCADE; CREATE SCHEMA public;
Now try steps 5 to 9 again, it should work out
Summary
This is how I had to backup/restore my backup on Postgres, when I had error permission issues and could not log in as a superuser. Or set credentials for read/write using chmod for folders. This workflow works for a binary file dump default of "Custom" from pgadmin. I assume .sql is the same way, but I have not yet tested that
I use:
cat /home/path/to/dump/file | psql -h localhost -U <user_name> -d <db_name>
Hope this will help someone.
If you are using a file with .dump extension use:
pg_restore -h hostname -d dbname -U username filename.dump
I noticed that many examples are overcomplicated for localhost where just postgres user without password exist in many cases:
psql -d db_name -f dump.sql
You can do it in pgadmin3. Drop the schema(s) that your dump contains. Then right-click on the database and choose Restore. Then you can browse for the dump file.
I used this
psql -d dbName -U username -f /home/sample.sql
Postgresql12
from sql file:
pg_restore -d database < file.sql
from custom format file:
pg_restore -Fc database < file.dump
I had more than 100MB data, therefore I could not restore database using Pgadmin4.
I used simply postgres client, and write below command.
postgres#khan:/$ pg_restore -d database_name /home/khan/Downloads/dump.sql
It worked fine and took few seconds.You can see below link for more information.
https://www.postgresql.org/docs/8.1/app-pgrestore.html