I have a python process that forks a psql command. The command connects to the remote database and COPY's into a local psql which reads the data. Here's the command vastly simplified and broken into several lines for readability.:
psql --quiet -X --quiet
--command='\pset tuples_only'
--command='\copy (TABLE QUERY) to program
\'psql "--command=copy TABLE from stdin with (format binary)"
postgresql://apolloqa:Apollo123#LOCALHOST:PORT/LOCAL_DB\'
with (format binary)' postgresql://REMOTE_USER:REMOTE_PASS#REMOTE_HOST:PORT/REMOTE_DB
(The quotes may be wrong after editing it from my logs, but it works on other machines, and it's not germane to my question.) The logs show:
Dec 20 03:06:51 LOCAL_HOST copying_server.sh[2788715]: PSQL STDOUT:
Dec 20 03:06:51 LOCAL_HOST copying_server.sh[2788715]: COPY 1310
Dec 20 03:06:51 LOCAL_HOST copying_server.sh[2788715]: COPY 29285588
Dec 20 03:06:51 LOCAL_HOST copying_server.sh[2788715]:
Dec 20 03:06:51 LOCAL_HOST copying_server.sh[2788715]: PSQL STDERR:
Dec 20 03:06:51 LOCAL_HOST copying_server.sh[2788715]: ERROR: could not write to file "base/pgsql_tmp/pgsql_tmp122309.3": No space left on device
Dec 20 03:06:51 LOCAL_HOST db_copy_server.sh[2788715]: ERROR: COPY file signature not recognized
The psql breaks because of no disk space. But, which host ran out of disk space? Local or remote host? When I look at my local host, it has plenty of space. So I doubt it's a local problem, but I cannot go onto the remote machine to see how much space it has.
I want to make sure it's not on my local host, before going to the admins for the remote host and remote database.
Thanks for the help.
Related
I’m trying to install a oracle database on my mac machine, but my container status is unhealthy. after i run this command:
docker run -d -it --name store/oracle/database-enterprise:12.2.0.1
and check the status by docker ps
it shows that it's unhealthy.
and when i run docker exec -it bash -c "source /home/oracle/.bashrc; sqlplus /nolog"
I get sqlplus command not found.
this is the logs:
Oracle Database 12.2.0.1 Setup
Mon Feb 17 13:46:41 UTC 2020
Check parameters ......
log file is : /home/oracle/setup/log/paramChk.log
paramChk.sh is done at 0 sec
untar DB bits ......
log file is : /home/oracle/setup/log/untarDB.log
untarDB.sh is done at 38 sec
config DB ......
log file is : /home/oracle/setup/log/configDB.log
mkdir: cannot create directory '/u02/app': No space left on device
mkdir: cannot create directory '/u03/app': No space left on device
mkdir: cannot create directory '/u04/app': No space left on device
mkdir: cannot create directory '/u01/app/oracle/product/12.2.0/dbhome_1/admin': No space left on device
ln: failed to create symbolic link '/u02/app/oracle/audit': No such file or directory
ln: failed to create symbolic link '/u02/app/oracle/oradata': No such file or directory
ln: failed to create symbolic link '/u03/app/oracle/fast_recovery_area': No such file or directory
ln: failed to create symbolic link '/u04/app/oracle/redo': No such file or directory
/home/oracle/setup/configDBora.sh: line 137: orapwd: command not found
/home/oracle/setup/configDBora.sh: line 171: /u01/app/oracle/product/12.2.0/dbhome_1/admin/ORCLCDB/sqlnet.ora: No such file or directory
/home/oracle/setup/configDBora.sh: line 172: /u01/app/oracle/product/12.2.0/dbhome_1/admin/ORCLCDB/sqlnet.ora: No such file or directory
/home/oracle/setup/configDBora.sh: line 173: /u01/app/oracle/product/12.2.0/dbhome_1/admin/ORCLCDB/sqlnet.ora: No such file or directory
/home/oracle/setup/configDBora.sh: line 184: /u01/app/oracle/product/12.2.0/dbhome_1/admin/ORCLCDB/listener.ora: No such file or directory
/home/oracle/setup/configDBora.sh: line 185: /u01/app/oracle/product/12.2.0/dbhome_1/admin/ORCLCDB/listener.ora: No such file or directory
/home/oracle/setup/configDBora.sh: line 186: /u01/app/oracle/product/12.2.0/dbhome_1/admin/ORCLCDB/listener.ora: No such file or directory
/home/oracle/setup/configDBora.sh: line 197: /u01/app/oracle/product/12.2.0/dbhome_1/admin/ORCLCDB/tnsnames.ora: No such file or directory
/home/oracle/setup/configDBora.sh: line 206: /u01/app/oracle/product/12.2.0/dbhome_1/admin/ORCLCDB/tnsnames.ora: No such file or directory
/home/oracle/setup/configDBora.sh: line 209: lsnrctl: command not found
Mon Feb 17 13:47:19 UTC 2020
Start Docker DB configuration
Call configDBora.sh to configure database
Configure DB as oracle user
Setup Database directories ...
/home/oracle/setup/configDBora.sh: line 123: sqlplus: command not found
update password
create pdb : ORCLPDB1
/home/oracle/setup/configDBora.sh: line 141: sqlplus: command not found
Reset Database parameters
/home/oracle/setup/configDBora.sh: line 155: sqlplus: command not found
DONE!
Remove password info
Docker DB configuration is complete !
configDB.sh is done at 38 sec
Done ! The database is ready for use .
# ===========================================================================
# == Add below entries to your tnsnames.ora to access this database server ==
# ====================== from external host =================================
ORCLCDB=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<ip-address>)(PORT=<port>))
(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=ORCLCDB.localdomain)))
ORCLPDB1=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<ip-address>)(PORT=<port>))
(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=ORCLPDB1.localdomain)))
#
#ip-address : IP address of the host where the container is running.
#port : Host Port that is mapped to the port 1521 of the container.
# The mapped port can be obtained from running "docker port <container-id>"
JIT: pid 5408 requesting stop
2017-08-08T17:41:41.417556+00:00
ARCH: Archival disabled due to shutdown: 1090
Shutting down archive processes
Archiving is disabled
2017-08-08T17:41:41.436309+00:00
Stopping background process VKTM
2017-08-08T17:41:47.017665+00:00
Instance shutdown complete (OS id: 5408)
Mon Feb 17 14:05:05 UTC 2020
shutdown database
/home/oracle/setup/shutDB.sh: line 53: sqlplus: command not found
stop listener
/home/oracle/setup/shutDB.sh: line 61: lsnrctl: command not found
Start up Oracle Database
Mon Feb 17 14:05:07 UTC 2020
start database
/home/oracle/setup/startupDB.sh: line 53: sqlplus: command not found
start listener
/home/oracle/setup/startupDB.sh: line 61: lsnrctl: command not found
The database is ready for use
Prune your volumes and then make sure your mount volume is under the docker desktop (OSX)> Docker Dashboard> Resources> File sharing
More about mounting
Starting an Oracle database server instance: Example
docker run --name TEST-DB -p 9999:1521 -p 9991:5500 -v /Users/yourusername/data/oracle:/opt/oracle/oradata store/oracle/database-enterprise:12.2.0.1
The default user credentials from dockerhub:
user:sys password:Oradoc_db1.
I found a lot of cases, where people try to open a file from outside the docker container. But mine is clearly inside.
The whole message I am getting is:
$ /opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P "mypw" -Q
"RESTORE FILELISTONLY FROM DISK = '/var/opt/mssql/backup/MyDB.bak'"
Msg 3201, Level 16, State 2, Server 1c7bf85afdaf, Line 1
Cannot open backup device '/var/opt/mssql/backup/MyDB.bak'. Operating system error 5(Access is denied.).
Msg 3013, Level 16, State 1, Server 1c7bf85afdaf, Line 1
RESTORE FILELIST is terminating abnormally.
Same happens when I try to actually restore the database. Any idea what is going wrong?
I have the feeling it might be some permission things where need to change permissions for the file
$ ls -la var/opt/mssql/backup/MyDB.bak
-rw-r----- 1 501 dialout 3395584 Jan 16 02:12 var/opt/mssql/backup/MyDB.bak
The solution was to change the permission of the file.
I needed to do:
docker exec -it -u root MicrosoftSQLServer "bash"
Then change the user (whoami returns mssql when I run the docker container not as root)
chown mssql /var/opt/mssql/backup/TestDB.bak
And then it works.
The answer of Juliano works for me
EDIT-2
I found out that the database doesn't even start after making the file location change.
This is with the default file location:
$pg_isready
/var/run/postgresql:5432 - accepting connections
$pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
9.5 main 5432 online postgres /var/lib/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log
pg_lsclusters output is green.
After the file location has changed on postgresql.conf:
$pg_isready
/var/run/postgresql:5432 - no response
$pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
9.5 main 5432 down root /mnt/Data/postgresdb/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log
Here the output is red.
Following this post here, I tried to start the cluster manually:
$pg_ctlcluster 9.5 main start
Warning: the cluster will not be running as a systemd service. Consider using systemctl:
sudo systemctl start postgresql#9.5-main
Error: You must run this program as the cluster owner (root) or root
I tried the same command with sudo:
Error: Config owner (postgres:124) and data owner (root:0) do not match, and config owner is not root
Which again makes me think the problem might lie with permissions of the directory. The directory is owned by root whose ownership I am unable to change.
EDIT-1
I've been working on this and I'd like to distill this post further to give more specifics. This is my current situation:
I installed postgres: sudo apt-get install postgresql and postgresql-contrib
I used sudo -U postgres psql to get into the postgres shell (I'm not sure if this is what I need to do)
show data_directory returns: /var/lib/postgresql/9.5/main
The data directory is located in Ubuntu ext4 formatted hard drive. I also have a 1 TB NTFS formatted hard disk mounted on /mnt/Data (which is mounted automatically on boot). What I tried:
Stop the postgres service: sudo systemctl stop postgresql
Create a new directory /mnt/Data/postgresdb and copy contents of the previous main to this which gives me a full path of /mnt/Data/postgresdb/postgresql/9.5/main using: sudo rsync -av /var/lib/postgresql/ /mnt/Data/postgresdb/postgresql/
Edit /etc/postgresql/9.5/main/postgresql.conf to change data_directory from the path mentioned above to /mnt/Data/postgresdb/postgresql/9.5/main
Start the postgres service: sudo systemctl start postgresl
Run sudo -U postgres psql but get the error that was mentioned in the original post.
These are the permissions on the respective main directories:
ls -l /var/lib/postgresql/9.5/
total 4.0K drwx------ 19 postgres postgres 4.0K Jan 16 12:40 main
ls -l /mnt/Data/postgresdb/postgresql/9.5/
total 4.0K drwxrwxrwx 1 root root 4.0K Jan 16 12:13 main
From the looks of it, the default directory is owned by "postgres" and the new directory is owned by root. However, when I try to change ownership to postgres: chown -R postgres main, it doesn't output any error, but the ownership doesn't change. I'm curious whether this is because this drive is NTFS formatted and is mounted.
Here is my /etc/fstab:
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda5 during installation
UUID=3f5a9875-89a3-4ce5-b778-9d7aaf148ed6 / ext4 errors=remount-ro 0 1
# swap was on /dev/sda6 during installation
UUID=85c3f4d4-e450-435b-8dd6-cf1b2cbd8fc2 none swap sw 0 0
/dev/disk/by-label/Data /mnt/Data auto nosuid,nodev,nofail,x-gvfs-show 0 0
Any ideas on how I can go about fixing this?
ORIGINAL POST
Recently, I installed Postgresql for storing some data for my research. The dataset came with instructions on how to setup the data on a Postgresql database (if interested, more info on that here and here). I installed Postgresql and set up a "role" and used the script that was provided for loading the database. It worked but I underestimated the size of the dataset and the script quit saying there was no more space.
I have two drives on my computer a 250G SSD drive with Windows and Ubuntu installed (125G each). And a 1TB HDD NTFS formatted where I store my data. So I thought moving the database to a folder on the other drive would be helpful. I purged all the data and the database to start afresh and followed the instructions here to move the database directory. However, after moving the directory, when I try to connect using psql I get the following error:
~ psql -U username -d postgres 14:48:33
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
How can I fix this? I am running 64-bit Ubuntu 16.04 with Postgresql-9.5. As mentioned earlier, I moved the DB directory a NTFS formatted filesystem (not sure if that cause any problems).
Thanks.
As mentioned in the comments the NTFS was the problem. I ended up resizing my bigger hard drive with 100GB formatted as ext4 and was able to launch postgres with the new data directory without any problems.
I have finally managed to get around managing to manually set "PostgreSQL Binary Path" in preferences (why should the user have to do this when it wasn't necessary in pgAdmin3?).
Though to be honest, I have no idea what is meant with the second path field called "EDB Advanced Server Binary Path". But I get no more error message concerning that so I assume it is o.k. to leave that empty.
Now I am stuck on the next step.
When trying to backup the database I have to enter a filename. No matter what I do here, I keep getting this error:
[Errno 2] No such file or directory
Well, it can not be that I have to create a file before it is created by pgAdmin, can it?
What could I be doing wrong?
Alex
update: answering to #n33rma here are the print screens:
I solved the problem through the command line.
If we use a previous format, it has the same response as you had:
$ pg_restore -d <DBNAME> --username=postgres g=<FILENAME>
pg_restore: [archiver] could not open input file "g=<FILENAME>": No such file or directory
If we change the last param, it works properly:
$ pg_restore -d <DBNAME> --username=postgres <FILENAME>
I suppose the GUI-command changed in v4, but the console pg_restore is not.
I just updated to pgAdmin v4.1 and this solved the issue :-)
Thanks all for helping!
You need to give absolute path with the file name to sql file like below.
Windows
C:/test/test.sql
Linux
/u01/test.sql
Enter only a filename, no path, because pgadmin puts the backup in its own directory, specifically, in ~/.pgadmin/storage/(user).
I suspect that path for "PostgreSQL Binary Path" is not set properly in pgAdmin4.
pgAdmin4 is Web application and pgAdmin3 is Desktop application, So pgAdmin4 needs to know path of utilities like pg_dump & pg_restore binaries so that it can execute them.
In pgAdmin4 Goto: File > Preferences > Paths > Binary paths > PostgreSQL Binary Path
If you are using Windows than provide path like,
C:\Program Files\PostgreSQL\9.6\bin
If you are using Linux than provide path like,
/opt/PostgreSQL/9.5/bin
user#mint:/opt/PostgreSQL/9.5/bin$ lsh pg_du* pg_res*
-rwxr-xr-x 1 xxx xxx 150K Sep 2 2015 pg_restore
-rwxr-xr-x 1 xxx xxx 49K Sep 2 2015 pg_resetxlog
-rwxr-xr-x 1 xxx xxx 83K Sep 2 2015 pg_dumpall
-rwxr-xr-x 1 xxx xxx 364K Sep 2 2015 pg_dump
On MacOS 10.11 and using postgres.app I resolved it this way:
I'm trying to test a small PostgreSQL setup, so I cobbled together a quick local install. However, when I'm trying to create my personal db with createdb, it chokes on errors like this (notably, it starts with base/16384 the first time, and increments each time I run it). Anyone know what's going on here, or if there's some trivial config I missed that would cause this? Thanks, and this is somewhat time-critical, so please respond if you do know anything. Thanks!
UPDATES:
I'm running this on a CentOS 5 server, apologies that I don't have too many further details (it's a shared account on that server). uname -a has the following output:
Linux {OMITTED} 2.6.18-194.11.4.el5 #1 SMP Tue Sep 21 05:04:09 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
I installed PostgreSQL from source from:
http://wwwmaster.postgresql.org/download/mirrors-ftp/source/v9.0.1/postgresql-9.0.1.tar.bz2
built in my home directory and installed to prefix=$HOME/local/pgsql.
Here's a terminal readout for me attempting to create my user's db on a fresh data setup:
[htung#{OMITTED}:~]$ killall postgres
LOG: autovacuum launcher shutting down
LOG: received smart shutdown request
LOG: shutting down
LOG: database system is shut down
[htung#{OMITTED}:~]$ rm -r tmp
mk[1]+ Done ../local/pgsql/bin/postgres -D $HOME/tmp (wd: ~/tmp)
(wd now: ~)
[htung#{OMITTED}:~]$ mkdir tmp
[htung#{OMITTED}:~]$ local/pgsql/bin/initdb -D $HOME/tmp
The files belonging to this database system will be owned by user "htung".
This user must also own the server process.
The database cluster will be initialized with locale en_US.UTF-8.
The default database encoding has accordingly been set to UTF8.
The default text search configuration will be set to "english".
fixing permissions on existing directory /afs/{OMITTED}/htung/tmp ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 32MB
creating configuration files ... ok
creating template1 database in /afs/{OMITTED}/htung/tmp/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the -A option the
next time you run initdb.
Success. You can now start the database server using:
local/pgsql/bin/postgres -D /afs/{OMITTED}/htung/tmp
or
local/pgsql/bin/pg_ctl -D /afs/{OMITTED}/htung/tmp -l logfile start
[htung#{OMITTED}:~]$ local/pgsql/bin/postgres -D $HOME/tmp
LOG: database system was shut down at 2010-11-15 13:47:25 PST
LOG: autovacuum launcher started
LOG: database system is ready to accept connections
[1]+ Stopped local/pgsql/bin/postgres -D $HOME/tmp
[htung#{OMITTED}:~]$ bg
[1]+ local/pgsql/bin/postgres -D $HOME/tmp &
[htung#{OMITTED}:~]$ local/pgsql/bin/createdb
ERROR: could not fsync file "base/16384": Invalid argument
STATEMENT: CREATE DATABASE htung;
createdb: database creation failed: ERROR: could not fsync file "base/16384": Invalid argument
[htung#{OMITTED}:~]$
I would guess that you're possibly running into the SE linux system here. I'd recommend to either turn off SELinux and see if that works, or to install from RPMs available from the postgresql website.