My EC2 database server failed, preventing SSH or other access (not sure why ... grrr AWS ... that's another story).
I was able to make a snapshot of the EBS root volume. I can not boot a new instance from this volume (I'm guessing the boot partition is corrupt). However, I can attach and mount the volume on a new instance.
Now I need to get the PostgreSQL 8.4 on the new machine (Ubuntu 10.04) to load the data from the mounted volume. Is this possible? I've tried:
pg_ctl start -D /<mount_dir>/etc/postgresql/8.4/main/
But no joy ... PostgreSQL just starts with empty tables.
Is /etc/postgresql/8.4/main/ the correct location for PostgreSQL data files?
Is there a way to recover the data from the mounted volume in a way that PostgreSQL can read again?
(You should really specify your distro and version, etc, with this sort of system admin question.)
Running Pg via pg_ctl as shown above should work, assuming the original database was from Pg 8.4 and so are the binaries you're trying to use to start it. Perhaps you forgot to stop the instance of PostgreSQL automatically started by the distro? Or connected on the wrong port, so you got the distro's default instance instead of your DB on another port (or different unix socket path, for unix sockets)?
Personally I wouldn't do what you're doing anyway. First, before I did anything else, I'd make a full backup of the entire data directory because you clearly don't have good backups, otherwise you wouldn't be worrying about this. Take them now, because if you break something while restoring you're going to hate yourself. As demonstrated by this fault, trusting Amazon's storage (snapshot or otherwise) probably isn't good enough.
Once you've done that: The easiest way to restore your DB will be to, on a new instance you know you don't have any important data on that has the same major version (eg "8.4" or "9.0") of postgresql as your original instance did installed:
/etc/init.d/postgresql-8.4 stop
datadir=/var/lib/postgresql/8.4/main
rm -rf "$datadir"
cp -aR /<mount_dir>/etc/postgresql/8.4/main/ "$datadir"
chown -R postgres:postgres "$datadir"
/etc/init.d/postgresql-8.4 start
In other words: take a copy, fix the permissions, start the DB.
You might need to edit /etc/postgresql/8.4/main/postgresql.conf and/or /etc/postgresql/8.4/main/pg_hba.conf because any edits you made to the originals aren't there anymore; they're on your corrupted root FS. The postgresql.conf and pg_hba.conf in the datadir are just symlinks to the ones in etc under Debian - something I understand the rationale behind, but don't love.
Once you get it running, do an immediate pg_dumpall and/or just a pg_dump of your important DB, then copy it somewhere safe.
Related
There are plenty of resources on how to dump Postgres/Mariadb/MySQL/etc. databases from a volume/container; my question is if I need to do so before backing them up. More explicitly, is it safe to stop my MariaDB container, copy the contents of the volume to another folder, and back that up directly? Are there consequences I should be aware of?
My current export code:
mkdir -p $HOME/backup/mariadb_backup
docker run --rm -v mariadb_volume:/data -v $HOME/backup:/backup ubuntu cp -aruT /data /backup/mariadb_backup
I then run borg on the backup folder.
It is safe to back up the files of a stopped database.
People usually don't want to shut down a database that's providing some service, so they come up with methods how not to do that.
One is run a dump operation that exports the contents of a database while it is serving other requests.
Another is a filesystem snapshot. That is atomically take a snapshot of the files underlying the database so that all files retain their content from a single point in time and then back that up.
The only thing you should not do is back up the files of a running database one by one. You will get an inconsistent copy if you do that.
Basically when creating new database, PostgreSQL make directory in "data/base" with OID.
Now I have a directory like this from my previous database, can i import this directory as a real database in pgAdmin?
I have a folder from my old database called 16384.I past it in my data/base folder but pgAdmin does not recognize it as a database.
I want to import it in my Pgadmin.
Is there any way to do this?
Thanks a lot.
no. you can't do it with pgadmin.
try starting postgres with your data_directory pg_ctl -D DIR_WITH_BASE_CATALOG. If you lucky to star it, you can try taking a full backup - if success, the directory is usable, if not, you would recommend asking experts to help you extract as much as possible. If you decide to use zero_damaged_pages or other advanced features you can irreversibly destroy data (if there still is what to destroy)
anyway, before you start, I'd recommend copying the whole data_directory to somewhere else, and trying to start cluster from copy, not original...
I was trying to upgrade my postgres installation from 9.5.7 to 9.6.5
My postgres database production instance has several databases and consumed ~700 GB space till now.
pg_upgrade needs 2 different dir for old and new datadir.
pg_upgrade -b oldbindir -B newbindir -d olddatadir -D newdatadir
it needs a new directory to do the pg_upgrade where as I was able to run the above command in my local/stage database as my database size was small in comparison to prod and I observed the following in my local
sudo du -sh /var/lib/pgsql/data-9.5
64G /var/lib/pgsql/data-9.5
sudo du -sh /var/lib/pgsql/data-9.6
60G /var/lib/pgsql/data-9.6
and I was having sufficient free data space to do the interim pg_upgrade process in my local/stage and I did it successfully there.
While in production I have only ~300 GB free space.
However after the successful upgrade we will delete the /var/lib/pgsql/data-9.5 dir.
Is there any way to do the in-place data upgrade so that It will not need the same amount of extra space for interim pg_upgrade process ?
Run pg_upgrade
/usr/lib/postgresql/9.6/bin/pg_upgrade
-b /usr/lib/postgresql/9.5/bin/
-B /usr/lib/postgresql/9.6/bin/
-d /var/lib/pgsql/data-9.5/
-D /var/lib/pgsql/data/
--link --check
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for reg* system OID user data types ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for roles starting with 'pg_' ok
Checking for presence of required libraries ok
Checking database user is the install user ok
Checking for prepared transactions ok
Clusters are compatible
Always run the pg_upgrade binary of the new server, not the old one. pg_upgrade requires the specification of the old and new cluster's data and executable (bin) directories. You can also specify user and port values, and whether you want the data linked instead of copied (the default).
If you use link mode, the upgrade will be much faster (no file copying) and use less disk space, but you will not be able to access your old cluster once you start the new cluster after the upgrade. Link mode also requires that the old and new cluster data directories be in the same file system. (Tablespaces and pg_xlog can be on different file systems.) See pg_upgrade --help for a full list of options.
Thanks to the postgres community comprehensive documentation which helped me a lot to find the solution after all.
I am pretty new with postgresql and pgAdmin and I made a beginner error.
I changed the hard drive of my computer and unfortunately (and stupidly I really admit) I have not make a backup of my database in pgAdmin3 before. Is there anyway I can manage to transfer it back from my old hard drive to the new one?
I tried to copy the file "Data" of pgAdmin from the old hard drive to the new one but it did not succeed. Is there any hidden file that would retain information of my database still on the old hard drive?
If I re-install the old hard drive, would I be able to access the database back?
if anyone have any idea I would be really glad,
thanks
Do as "Richard Huxton" suggested, then
On Windows you need may need to set PG as a Service
Run the following on the command line (as adminstrator):
pg_ctl.exe register -N postgres -D "C:\Program Files\PostgreSQL\9.6\data"
Start the service
answer from:
Register and run PostgreSQL 9.0 as Windows Service
Three steps are required:
Copy everything in the data folder (base, global, ... pg_xlog, ...)
Make sure the permissions are right
Make sure you have the same version of the PostgreSQL binaries installed.
Then just point the configuration at it (data_directory) and start it up. If there are any problems, check the logs for details - it'll probably be permissions.
Sometimes it's convenient to do all this in a virtual machine if you have that sort of stuff set up already.
I have learned a lot about setting up vagrant with chef and I am hitting a wall since I am new to ruby - vagrant - chef and I am not the biggest developer. Being mostly front end but trying to set up a better environment to develop in.
I have searched and found great answers but left with one final question.
I have this code creating the database but I can not figure out where to place the database to import from...
# import an sql dump from your app_root/data/dump.sql to the my_database database
execute "import" do
command "mysql -u root -p\"#{node['mysql']['server_root_password']}\" my_database < /chef/vagrant_db/database-name.mysql"
action :run
end
So I need to know where the path should start from, the top level home directory, from the top level folder where I run vagrant up? Where it is currently and a few other tried places is not working.
Any ideas would be great. I have search google so much so that I am almost ready to give up.
Thanks
Tim
I would recommend using Chef::Config[:file_cache_path] for this. Let's say you want to get that SQL file from a remote web server:
db = File.join(Chef::Config[:file_cache_path], 'database.mysql')
remote_file db do
source 'http://my.web.server/db.mysql
action :create_if_missing
notifies :run, 'execute[import]', :immediately
end
execute "import" do
command "mysql -u root -p\"#{node['mysql']['server_root_password']}\" my_database < #{db}"
action :nothing
end
This will:
Add idempotency - meaning it won't try to import the database on each run
Leverage Chef's file-cache-path, which is persisted and guaranteed to be writable on supported Chef systems
Extensible (you could easily change remote_file to cookbook_file or some custom resource to fetch the database)
Now, getting the file from Vagrant is a different story. By default, Vagrant mounts the directory where the Vagrantfile is located at on the host (local laptop) at /vagrant on the VM (guest machine). You can mount additional locations (called "shared folders") anywhere on your local laptop.
Bonus
If you are running the database on your local machine, you can actually share the socket over a shared folder with Vagrant :). Then you don't even need MySQL on your VM - it will use the one running on your host laptop.
Sources:
I write a lot of Chef :)