Postgres copy Heroku Production DB to local development DB - database

I have a heroku database, d76mj7ltuqs.
I then have a local database, test_development.
The schema is the same on both of these databases - I want to pull all of the data from my production database and overwrite my local database, so that local is an exact replica of production at the time of pull.
How can I do that in Postgres?

Use heroku's "pg:pull":
You'll need to clear your local DB:
rake db:drop
Then collect some information from Heroku:
heroku pg:pull DATABASE_URL test_development
This will connect to the heroku DB, and copy it to the local database.
See Heroku's documentation on pg:pull for more details.

This command should do the work:
heroku pg:pull DATABASE_URL database-name --app heroku-app-name

clean your local database:
rake db:schema:load
dump your heroku database:
heroku pg:backups:capture -r <**your production git repo name**>
heroku pg:backups:download -r <**your production git repo name**>
load data in your local database
pg_restore --verbose --clean --no-acl --no-owner -h localhost -d <**test database name**> latest.dump

this is how i do it, be sure to gzip it as your database grows. also don't export the ACL as you likely don't have the same postgres user on heroku and local accounts. replace with your specific details.
pg_dump -h ec2-##-##-##-##.compute-1.amazonaws.com -p <port> -Fc --no-acl --no-owner -o -U <username> <databasename> | gzip > dumpfile.gz
#<Prompt for Password>
gunzip -c dumpfile.gz | pg_restore --verbose --clean --no-acl --no-owner -d test_development -U <local_username>

Use your terminal to make a local pg_dump and then either psql or pg_restore it into your local database.
Similar method can be found here.

If this is a Rails app, you can use the following script to overwrite your local database with the latest dump you've generated on Heroku. If you uncomment the line with heroku pg:backups capture, the script will generate a new snapshot on Heroku before downloading it to your machine.
Note that you shouldn't have to edit the script, since it reads all the configuration from your database.yml file.
#!/usr/bin/env ruby
require_relative '../config/environment'
# Uncomment the line below if you want to generate a new snapshot of the
# Heroku production database before downloading it to the local machine
# `heroku pg:backups capture`
database_dump_file_pathname = Tempfile.new('latest.dump').path
`heroku pg:backups:download --output #{database_dump_file_pathname}`
# Get database config fom database.yml file
database_config = YAML::load_file(Rails.root.join('config', 'database.yml'))
database_name = database_config['development']['database']
database_username = database_config['development']['username']
database_password = database_config['development']['password']
# Overwrite local database with dump
cmd_line_arguments = [
'--verbose',
'--clean',
'--no-acl',
'--no-owner',
'--host localhost',
"-U #{database_username}",
"-d #{database_name}",
database_dump_file_pathname
].join(' ')
`PGPASSWORD=#{database_password} pg_restore #{cmd_line_arguments}`
See the Heroku docs on downloading DB backups for details.

Related

DB reset after deploy to meteor servers

I re-deployed my app to meteor by using 'meteor deploy '
and my Database was reset.
Any clue why this happened or how I can avoid it in the future ?
When a meteor app is deployed, your data saved in local mongo would not deployed to the server. So you could use mongodump and mongorestore to solve it:(docs)
Now first dump your database somewhere
mongodump --host localhost:3001
Get your mongodb`s credentials by running (in your app dir):
meteor mongo myapp.meteor.com --url
This will give you database url in the form:
mongodb://username:password#host:port/databasename
With these info you could fill them into mongorestore (docs) and restore your local database over
mongorestore -u username -p password -h host:port -d databasename ~/desktop/location_of_your_mongodb_dump
All of your data would get transferred in this way. I wish it could help.

PostgreSQL: duplication by one command [duplicate]

I'm looking to copy a production PostgreSQL database to a development server. What's the quickest, easiest way to go about doing this?
You don't need to create an intermediate file. You can do
pg_dump -C -h localhost -U localuser dbname | psql -h remotehost -U remoteuser dbname
or
pg_dump -C -h remotehost -U remoteuser dbname | psql -h localhost -U localuser dbname
using psql or pg_dump to connect to a remote host.
With a big database or a slow connection, dumping a file and transfering the file compressed may be faster.
As Kornel said there is no need to dump to a intermediate file, if you want to work compressed you can use a compressed tunnel
pg_dump -C dbname | bzip2 | ssh remoteuser#remotehost "bunzip2 | psql dbname"
or
pg_dump -C dbname | ssh -C remoteuser#remotehost "psql dbname"
but this solution also requires to get a session in both ends.
Note: pg_dump is for backing up and psql is for restoring. So, the first command in this answer is to copy from local to remote and the second one is from remote to local. More -> https://www.postgresql.org/docs/9.6/app-pgdump.html
pg_dump the_db_name > the_backup.sql
Then copy the backup to your development server, restore with:
psql the_new_dev_db < the_backup.sql
Use pg_dump, and later psql or pg_restore - depending whether you choose -Fp or -Fc options to pg_dump.
Example of usage:
ssh production
pg_dump -C -Fp -f dump.sql -U postgres some_database_name
scp dump.sql development:
rm dump.sql
ssh development
psql -U postgres -f dump.sql
If you are looking to migrate between versions (eg you updated postgres and have 9.1 running on localhost:5432 and 9.3 running on localhost:5434) you can run:
pg_dumpall -p 5432 -U myuser91 | psql -U myuser94 -d postgres -p 5434
Check out the migration docs.
pg_basebackup seems to be the better way of doing this now, especially for large databases.
You can copy a database from a server with the same or older major version. Or more precisely:
pg_basebackup works with servers of the same or an older major version, down to 9.1. However, WAL streaming mode (-X stream) only works with server version 9.3 and later, and tar format mode (--format=tar) of the current version only works with server version 9.5 or later.
For that you need on the source server:
listen_addresses = '*' to be able to connect from the target server. Make sure port 5432 is open for that matter.
At least 1 available replication connection: max_wal_senders = 1 (-X fetch), 2 for -X stream (the default in case of PostgreSQL 12), or more.
wal_level = replica or higher to be able to set max_wal_senders > 0.
host replication postgres DST_IP/32 trust in pg_hba.conf. This grants access to the pg cluster to anyone from the DST_IP machine. You might want to resort to a more secure option.
Changes 1, 2, 3 require server restart, change 4 requires reload.
On the target server:
# systemctl stop postgresql#VERSION-NAME
postgres$ pg_basebackup -h SRC_IP -U postgres -D VERSION/NAME --progress
# systemctl start postgresql#VERSION-NAME
Accepted answer is correct, but if you want to avoid entering the password interactively, you can use this:
PGPASSWORD={{export_db_password}} pg_dump --create -h {{export_db_host}} -U {{export_db_user}} {{export_db_name}} | PGPASSWORD={{import_db_password}} psql -h {{import_db_host}} -U {{import_db_user}} {{import_db_name}}
Run this command with database name, you want to backup, to take dump of DB.
pg_dump -U {user-name} {source_db} -f {dumpfilename.sql}
eg. pg_dump -U postgres mydbname -f mydbnamedump.sql
Now scp this dump file to remote machine where you want to copy DB.
eg. scp mydbnamedump.sql user01#remotemachineip:~/some/folder/
On remote machine run following command in ~/some/folder to restore the DB.
psql -U {user-name} -d {desintation_db}-f {dumpfilename.sql}
eg. psql -U postgres -d mynewdb -f mydbnamedump.sql
Dump your database : pg_dump database_name_name > backup.sql
Import your database back: psql db_name < backup.sql
I struggled quite a lot and eventually the method that allowed me to make it work with Rails 4 was:
on your old server
sudo su - postgres
pg_dump -c --inserts old_db_name > dump.sql
I had to use the postgres linux user to create the dump. also i had to use -c to force the creation of the database on the new server. --inserts tells it to use the INSERT() syntax which otherwise would not work for me :(
then, on the new server, simpy:
sudo su - postgres
psql new_database_name < dump.sql
to transfer the dump.sql file between server I simply used the "cat" to print the content and than "nano" to recreate it copypasting the content.
Also, the ROLE i was using on the two database was different so i had to find-replace all the owner name in the dump.
Let me share a Linux shell script to copy your table data from one server to another PostgreSQL server.
Reference taken from this blog:
Linux Bash Shell Script for data migration between PostgreSQL Servers:
#!/bin/bash
psql \
-X \
-U user_name \
-h host_name1 \
-d database_name \
-c "\\copy tbl_Students to stdout" \
| \
psql \
-X \
-U user_name \
-h host_name2 \
-d database_name \
-c "\\copy tbl_Students from stdin"
I am just migrating the data; please create a blank table at your destination/second database server.
This is a utility script. Further, you can modify the script for generic use something like by adding parameters for host_name, database_name, table_name and others
Here is an example using pg_basebackup
I chose to go this route because it backs up the entire database cluster (users, databases, etc.).
I'm posting this as a solution on here because it details every step I had to take, feel free to add recommendations or improvements after reading other answers on here and doing some more research.
For Postgres 12 and Ubuntu 18.04 I had to do these actions:
On the server that is currently running the database:
Update pg_hba.conf, for me located at /etc/postgresql/12/main/pg_hba.conf
Add the following line (substitute 192.168.0.100 with the IP address of the server you want to copy the database to).
host replication postgres 192.168.0.100/32 trust
Update postgresql.conf, for me located at /etc/postgresql/12/main/postgresql.conf. Add the following line:
listen_addresses = '*'
Restart postgres:
sudo service postgresql restart
On the host you want to copy the database cluster to:
sudo service postgresql stop
sudo su root
rm -rf /var/lib/postgresql/12/main/*
exit
sudo -u postgres pg_basebackup -h 192.168.0.101 -U postgres -D /var/lib/postgresql/12/main/
sudo service postgresql start
Big picture - stop the service, delete everything in the data directory (mine is in /var/lib/postgreql/12). The permissions on this directory are drwx------ with user and group postgres. I could only do this as root, not even with sudo -u postgres. I'm unsure why. Ensure you are doing this on the new server you want to copy the database to! You are deleting the entire database cluster.
Make sure to change the IP address from 192.168.0.101 to the IP address you are copying the database from. Copy the data from the original server with pg_basebackup. Start the service.
Update pg_hba.conf and postgresql.conf to match the original server configuration - before you made any changes adding the replication line and the listen_addresses line (in my care I had to add the ability to log-in locally via md5 to pg_hba.conf).
Note there are considerations for max_wal_senders and wal_level that can be found in the documentation. I did not have to do anything with this.
If you are more comfortable with a GUI, you can use the pgAdmin software.
Connect to your source and destination servers
Right-click on the source db > backup
Right-click on the destination server > create > database. Use the same properties as the source db (you can see the properties of the source db by right-click > properties)
Right-click on the created db > restore.

How can I import a .sql file into my Heroku postgres database?

I have a backup sql file from another database that I want to import into Heroku's postgres database. How do you do that?
This is how you do it:
heroku pg:psql --app YOUR_APP_NAME_HERE < updates.sql
And if you want to restore your production into staging (assuming both are heroku postgres DBs):
heroku pgbackups:restore YOUR_STAGING_DATABASE_NAME `heroku pgbackups:url --app YOUR_PRODUCTION_APP_NAME` --app YOUR_STAGING_APP_NAME --confirm YOUR_STAGING_APP_NAME
Make sure to preserve the special single quotes around heroku pgbackups:url --app YOUR_PRODUCTION_APP_NAME.
HEROKU TOOLBELT UPDATE
Heroku has recently updated their toolbelt so the old commands are no longer valid (see this link for more info). Below is the new version of the restore command.
heroku pg:backups restore \
`heroku pg:backups public-url -a YOUR_PRODUCTION_APP_NAME` \
YOUR_STAGING_DATABASE_NAME \
--app YOUR_STAGING_APP_NAME \
--confirm YOUR_STAGING_APP_NAME
Making backup file:
pg_dump -U USERNAME DATABASE --no-owner --no-acl -f backup.sql
Restoring from sql file to heroku :
heroku pg:psql --app APPNAME < backup.sql
(Bonus) Deleting all tables from heroku app database (example):
heroku pg:reset --app APPNAME HEROKU_POSTGRESQL_ROSE
get DATABASE_URL from posgresql heroku panel (psql line)
Load the SQL into a local Postgres instance and make sure it's correct. Then dump the data using the directions here: https://devcenter.heroku.com/articles/heroku-postgres-import-export
Finally, upload the dump to a public web server (like S3) and restore to Heroku like this:
heroku pgbackups:restore DATABASE 'https://s3.amazonaws.com/me/items/3H0q/mydb.dump'
Django local db import on Heroku on windows
create backup
pg_dump -U postgres -d hawkishfinance > C:\Users\Fauzan\Projects\hawkishfinance.sql
dump on server
heroku pg:psql --app hawkishfinance < hawkishfinance.sql
You must add run in this command. It will work successfully!
heroku pg:psql run --app YOUR_APP_NAME_HERE < updates.sql

How can I attach a database to an app in Heroku?

I'm using Heroku's Postgres addon, and I created a new production database from the Heroku Postgres addon page.
I Didn't add it directly to my App using the Resources page of my App.
Now I want to attach this database to my App so it'll be recognized by the heroku pg command.
I'm able to use the database btw after setting the DATABASE_URL config var of my app to point to it, but heroku pg command doesn't recognize it yet.
Additional info: The previous database was Shared, and the new one is a Production.
Heroku add-ons may now be attached across applications and multiple times on a single app.
heroku addons:attach ADDON_NAME -a APP_NAME
Source: https://devcenter.heroku.com/changelog-items/646
To know the name of your addon, do:
heroku addons
Source: https://devcenter.heroku.com/articles/managing-add-ons
Did you add the database using the app-independent https://postgres.heroku.com/ site? Or did you just create a postgresql database in your Heroku control panel?
If you created your database on https://postgres.heroku.com/, you will not see the database via your heroku pg:info command. What you can do to add your database to your application, however, would be to:
Log into https://postgres.heroku.com/.
Click on the database you want to attach to your application.
Under 'Connection Settings', click the configuration button at the top right.
Then click the 'URL' option.
Copy your database URL, this should be something like "postgres://blah:blah#ec2-23-23-122-88.compute-1.amazonaws.com:5432/omg".
In your application, on the command line, run heroku config:set DATABASE_URL=postgres://blah:blah#ec2-23-23-122-88.compute-1.amazonaws.com:5432/omg
What we did there was assign your database to the DATABASE_URL environment variable in your application. This is the variable that's used by default when you provision databases locally to your application, so theoretically, assigning this value should work just fine for you.
To get your database that you created at https://postgres.heroku.com/ attached to your actual heroku app that you are working on you can't use any of the pg backup commands and as far as I can tell there is no supported Heroku way of attaching a database to a heroku app.
You can however create a backup of your database using pg_dump and then use pg_restore to populate your new database that is attached to your app:
pg_dump -i -h hostname -p 5432 -U username -F c -b -v -f "backup-filename" database_name
Once that is complete you can populate your new database with:
pg_restore -i -h new_hostname -p 5432 -U new_username -d new_database_name -v "same_backup_filename"
Even if you are upgrading from the "basic plan" to a the "crane plan" you still have to do a backup and restore, but since the db's are already attached to your app you have the advantage of using the heroku backup commands.

Copying PostgreSQL database to another server

I'm looking to copy a production PostgreSQL database to a development server. What's the quickest, easiest way to go about doing this?
You don't need to create an intermediate file. You can do
pg_dump -C -h localhost -U localuser dbname | psql -h remotehost -U remoteuser dbname
or
pg_dump -C -h remotehost -U remoteuser dbname | psql -h localhost -U localuser dbname
using psql or pg_dump to connect to a remote host.
With a big database or a slow connection, dumping a file and transfering the file compressed may be faster.
As Kornel said there is no need to dump to a intermediate file, if you want to work compressed you can use a compressed tunnel
pg_dump -C dbname | bzip2 | ssh remoteuser#remotehost "bunzip2 | psql dbname"
or
pg_dump -C dbname | ssh -C remoteuser#remotehost "psql dbname"
but this solution also requires to get a session in both ends.
Note: pg_dump is for backing up and psql is for restoring. So, the first command in this answer is to copy from local to remote and the second one is from remote to local. More -> https://www.postgresql.org/docs/9.6/app-pgdump.html
pg_dump the_db_name > the_backup.sql
Then copy the backup to your development server, restore with:
psql the_new_dev_db < the_backup.sql
Use pg_dump, and later psql or pg_restore - depending whether you choose -Fp or -Fc options to pg_dump.
Example of usage:
ssh production
pg_dump -C -Fp -f dump.sql -U postgres some_database_name
scp dump.sql development:
rm dump.sql
ssh development
psql -U postgres -f dump.sql
If you are looking to migrate between versions (eg you updated postgres and have 9.1 running on localhost:5432 and 9.3 running on localhost:5434) you can run:
pg_dumpall -p 5432 -U myuser91 | psql -U myuser94 -d postgres -p 5434
Check out the migration docs.
pg_basebackup seems to be the better way of doing this now, especially for large databases.
You can copy a database from a server with the same or older major version. Or more precisely:
pg_basebackup works with servers of the same or an older major version, down to 9.1. However, WAL streaming mode (-X stream) only works with server version 9.3 and later, and tar format mode (--format=tar) of the current version only works with server version 9.5 or later.
For that you need on the source server:
listen_addresses = '*' to be able to connect from the target server. Make sure port 5432 is open for that matter.
At least 1 available replication connection: max_wal_senders = 1 (-X fetch), 2 for -X stream (the default in case of PostgreSQL 12), or more.
wal_level = replica or higher to be able to set max_wal_senders > 0.
host replication postgres DST_IP/32 trust in pg_hba.conf. This grants access to the pg cluster to anyone from the DST_IP machine. You might want to resort to a more secure option.
Changes 1, 2, 3 require server restart, change 4 requires reload.
On the target server:
# systemctl stop postgresql#VERSION-NAME
postgres$ pg_basebackup -h SRC_IP -U postgres -D VERSION/NAME --progress
# systemctl start postgresql#VERSION-NAME
Accepted answer is correct, but if you want to avoid entering the password interactively, you can use this:
PGPASSWORD={{export_db_password}} pg_dump --create -h {{export_db_host}} -U {{export_db_user}} {{export_db_name}} | PGPASSWORD={{import_db_password}} psql -h {{import_db_host}} -U {{import_db_user}} {{import_db_name}}
Run this command with database name, you want to backup, to take dump of DB.
pg_dump -U {user-name} {source_db} -f {dumpfilename.sql}
eg. pg_dump -U postgres mydbname -f mydbnamedump.sql
Now scp this dump file to remote machine where you want to copy DB.
eg. scp mydbnamedump.sql user01#remotemachineip:~/some/folder/
On remote machine run following command in ~/some/folder to restore the DB.
psql -U {user-name} -d {desintation_db}-f {dumpfilename.sql}
eg. psql -U postgres -d mynewdb -f mydbnamedump.sql
Dump your database : pg_dump database_name_name > backup.sql
Import your database back: psql db_name < backup.sql
I struggled quite a lot and eventually the method that allowed me to make it work with Rails 4 was:
on your old server
sudo su - postgres
pg_dump -c --inserts old_db_name > dump.sql
I had to use the postgres linux user to create the dump. also i had to use -c to force the creation of the database on the new server. --inserts tells it to use the INSERT() syntax which otherwise would not work for me :(
then, on the new server, simpy:
sudo su - postgres
psql new_database_name < dump.sql
to transfer the dump.sql file between server I simply used the "cat" to print the content and than "nano" to recreate it copypasting the content.
Also, the ROLE i was using on the two database was different so i had to find-replace all the owner name in the dump.
Let me share a Linux shell script to copy your table data from one server to another PostgreSQL server.
Reference taken from this blog:
Linux Bash Shell Script for data migration between PostgreSQL Servers:
#!/bin/bash
psql \
-X \
-U user_name \
-h host_name1 \
-d database_name \
-c "\\copy tbl_Students to stdout" \
| \
psql \
-X \
-U user_name \
-h host_name2 \
-d database_name \
-c "\\copy tbl_Students from stdin"
I am just migrating the data; please create a blank table at your destination/second database server.
This is a utility script. Further, you can modify the script for generic use something like by adding parameters for host_name, database_name, table_name and others
Here is an example using pg_basebackup
I chose to go this route because it backs up the entire database cluster (users, databases, etc.).
I'm posting this as a solution on here because it details every step I had to take, feel free to add recommendations or improvements after reading other answers on here and doing some more research.
For Postgres 12 and Ubuntu 18.04 I had to do these actions:
On the server that is currently running the database:
Update pg_hba.conf, for me located at /etc/postgresql/12/main/pg_hba.conf
Add the following line (substitute 192.168.0.100 with the IP address of the server you want to copy the database to).
host replication postgres 192.168.0.100/32 trust
Update postgresql.conf, for me located at /etc/postgresql/12/main/postgresql.conf. Add the following line:
listen_addresses = '*'
Restart postgres:
sudo service postgresql restart
On the host you want to copy the database cluster to:
sudo service postgresql stop
sudo su root
rm -rf /var/lib/postgresql/12/main/*
exit
sudo -u postgres pg_basebackup -h 192.168.0.101 -U postgres -D /var/lib/postgresql/12/main/
sudo service postgresql start
Big picture - stop the service, delete everything in the data directory (mine is in /var/lib/postgreql/12). The permissions on this directory are drwx------ with user and group postgres. I could only do this as root, not even with sudo -u postgres. I'm unsure why. Ensure you are doing this on the new server you want to copy the database to! You are deleting the entire database cluster.
Make sure to change the IP address from 192.168.0.101 to the IP address you are copying the database from. Copy the data from the original server with pg_basebackup. Start the service.
Update pg_hba.conf and postgresql.conf to match the original server configuration - before you made any changes adding the replication line and the listen_addresses line (in my care I had to add the ability to log-in locally via md5 to pg_hba.conf).
Note there are considerations for max_wal_senders and wal_level that can be found in the documentation. I did not have to do anything with this.
If you are more comfortable with a GUI, you can use the pgAdmin software.
Connect to your source and destination servers
Right-click on the source db > backup
Right-click on the destination server > create > database. Use the same properties as the source db (you can see the properties of the source db by right-click > properties)
Right-click on the created db > restore.

Resources