How to timestamp data copied into a postgresql database - database

I have a shell script pulling data from a server into a postgresql table.
df -g | awk 'BEGIN{OFS=","}NR>1{$1=$1; print}' > /data/metric.csv
psql -h localhost -d metrics -U postgres -c "copy tablename from STDIN with delimiter as ',';" < /data/metric.csv
Displays as:
filesystem | gb_blocks | free | %used | iused | %iused | mounted_on
/dev/hd2 | 16.75 | 12.60 | 25% | 79098 | 3% | /usr
/dev/hd9var | 8.00 | 6.00 | 25% | 11965 | 1% | /var
/dev/hd3 | 36.75 | 18.83 | 49% | 5614 | 1% | /tmp
/dev/hd1 | 3.25 | 3.11 | 5% | 674 | 1% | /home
/dev/hd11admin | 0.25 | 0.25 | 1% | 16 | 1% | /admin
/proc | - | - | - | - | - | /proc
I'm working with Postgresql on an Ubuntu OS and pulling the info from an AIX server. I'd like to add a column with a timestamp for every time new data is added to the table because right now it just all blends together. I've tried to add another column for timestamp and give it a timestamp value but the timestamp isn't in the csv file and I'm not sure how to add it either. I appreciate an help I can get to solve this.

Create table and add date column with Default value like current_Date/now() ).
CREATE TABLE IF NOT EXISTS metrics
(
filesystem text ,
gb_blocks text ,
free text ,
per_used text ,
iused text ,
per_iused text ,
mounted_on text ,
load_dttm timestamp without time zone DEFAULT now()
);
mention columnswith table while loading data as below command
psql -h localhost -d metrics -U postgres -c "copy metrics(filesystem,gb_blocks,free,per_used,iused,per_iused,mounted_on) from STDIN with delimiter as ',';" < /data/metric.csv

Related

FATAL: database "Employees" does not exist

Here is the console input-output, where I'm trying to access DB but psql cannot find it. I have tried changing capitalization, but result is the same
(base) username#MacBook-Pro-Ruslan ~ % psql -U username Employees
psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: database "Employees" does not exist
But, I have this DB in my Psql app, here the screenshoot of it
UPD: I just checked in command line all the databases
postgres-# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
----------------+----------------+----------+---------+-------+-----------------------------------
postgres | ruslanpilipyuk | UTF8 | C | C |
ruslanpilipyuk | ruslanpilipyuk | UTF8 | C | C |
template0 | ruslanpilipyuk | UTF8 | C | C | =c/ruslanpilipyuk +
| | | | | ruslanpilipyuk=CTc/ruslanpilipyuk
template1 | ruslanpilipyuk | UTF8 | C | C | =c/ruslanpilipyuk +
| | | | | ruslanpilipyuk=CTc/ruslanpilipyuk
(4 rows)
And it looks like that my psql app has not connected or transferred db between console. How could I get access to this db in app? That I already have. Db that are shown in list in console - are the Dbs that I created manually in console

Can't connect to postgres on ubuntu

I have a ubuntu 20 on dreamcompute (which is cloud computing).
I create a user and a database. Here is the list of database and users (for some reason, I can't see database under a matt username).
I went into:
nano /etc/postgresql/13/main/postgresql.conf &
nano /etc/postgresql/13/main/pg_hba.conf and did the whole '*' and '0.0.0.0/0'
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+---------+---------+-----------------------
postgres | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
strapi | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =Tc/postgres +
| | | | | postgres=CTc/postgres+
| | | | | hossein=CTc/postgres
template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
as you can see you can't see Superuser and database strapi under the matt username.
List of roles
Role name | Attributes | Member of
-----------+------------------------------------------------------------+-----------
matt | | {}
postgres | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
ubuntu | | {}
I'm using my dream compute's ip address as my host and use my database and user and password but get error message: connection attempt time out.
Could someone please give me a pointer on why this is happening? I have been working onthis for 2 weeks now and I can't get it to work.
Error message is connection time out. That usually means that the port is blocked by a firewall. Check your cloud provider firewall settings and iptablesin your Linux box in case you have installed it.
If there was problem with permissions, the error message would be something else.
as ex4 mentioned above I needed to reachout to the company that I was renting my cloud computer from but I still could not connect to the database.
The way I went around it is that you can ssh into your database and then connect to your database as a localhost since you are ssh into your cloud computer.
In DBeaver you have a ssh tab and you can connect and then you got back to your postgres tab and fill the localhost, user, database name, and user password area and simply click connect.
Sadly this took weeks to come to this :/

Full execution history of a stored procedure

I am wondering how to get the execution_time for all executions for a specific stored procedure. (using Microsoft SQL Server 2016)
I know that via dm_exec_procedure_stats I get information about the last_execution_time and execution_count but I am interested in the execution_time of every execution (of one stored procedure) which got counted for the execution_count.
What I get is something like this (as an example):
| name | database_id | execution_count | last_execution_time |
------------------------------------------------------------------
| sp_name1 | db_id1 | 23 | 11.09.2019 hh:mm:ss |
| sp_name2 | db_id1 | 12 | 09.09.2019 hh:mm:ss |
| sp_name3 | db_id2 | 3456 | 11.09.2017 hh:mm:ss |
So basically I want a query to get a table which has in one column the name of the procedure and in the other the execution times of this procedure such that the number of rows should equal the execution_count from the procedure.
What I want is something like this:
| name | database_id | execution_time |
------------------------------------------------
| sp_name1 | db_id1 | 11.09.2019 hh:mm:ss |
| sp_name1 | db_id1 | dd.mm.yyyy hh:mm:ss |
| sp_name1 | db_id1 | dd.mm.yyyy hh:mm:ss |
| ... | ... | ... |
which should have 23 rows.
To get the execution history of a stored procedure you can use :
Profiler :
SQL Server >Tools>SQL Server Profiler
Navigate to File > New Trace.
Trace Properties > click on the Events Selection tab > select the SP:Completed counter in the Stored Procedures grouping of counters >click on the General Tab to save the results (table / file).>Columns Filters: you can capture the database that you are working on> Finally Run

Database not created

I installed postgresql and tried to create new database but couldn't succeed in creating database:
Server [localhost]:
Database [postgres]:
Port [5433]:
Username [postgres]:
psql (9.2.17)
WARNING: Console code page (850) differs from Windows code page (1252)
8-bit characters might not work correctly. See psql reference
page "Notes for Windows users" for details.
Type "help" for help.
postgres=# createdb gps_heatmap
postgres-# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+---------------------+---------------------+-----------------------
database4 | postgres | UTF8 | English_Canada.1252 | English_Canada.1252 |
postgres | postgres | UTF8 | English_Canada.1252 | English_Canada.1252 |
template0 | postgres | UTF8 | English_Canada.1252 | English_Canada.1252 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | English_Canada.1252 | English_Canada.1252 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
postgres-#
Database is not created . I couldnt solve this problem. Help needed.
You are missing ; use
createdb gps_heatmap;

Robot Framework data-driven automation testing: Can data derived from a database be used as a data source for a test template?

I am familiar with using template keywords in data-driven Robot Framework testing and know that external sources of data such as text files and csv files can be used to provide test data. However, the organisation I work for wants to use data held in a database as a source for test case data. Does anybody know if this is possible? I have searched Stack Exchange, Stack Overflow and other resources but cannot find an answer or any examples.
Here is an example of the data-driven approach I am familiar just to give you an idea of where we are now.
*** Settings ***
Library Selenium2Library
Library AFRCLibrary
| Test Template | Suspend Region
*** Variables ***
*** Test Cases ***
| Pillar 1 BPS 2019 Suspend Region | Pillar 1 | 2019 | BPS | BPS Region 1 | Pillar 1 BPS 2019 Suspend Region Comments |
| Pillar 2 FGS 2018 Suspend Region | Pillar 2 | 2018 | FGS | FGS Region 1 | Pillar 2 FGS 2018 Suspend Region Comments |
*** Keywords ***
| Suspend Region
| | [Arguments] | ${pillar} | ${year} | ${scheme} | ${region} | ${comments} |
| | Futures Open Application | http://ags125p01:8080/operationalsite/login | ff |
| | FuturesPublicsiteWM | root | gtn | http://ags125p01:8080/operationalsite/futures/maintain_budget |
| | Select Pillar | ${pillar} | ${year} |
| | Select Scheme | ${scheme} |
| | View |
| | Suspend And Confirm | ${region} | ${comments} |
| | Futures Close Application |
| |
Unfortunately, the use of test templates more-or-less require that the data is hard-coded in the test case. However, the test template is not much more than a wrapper around a for loop. You could do something like this:
| | ${database_rows}= | Run sql query
| | ... | Select * from the_database where ...
| |
| | :FOR | ${row} | IN | #{database_rows}
| | | Suspend Region | #{row}
Of course, this requires that you write the "Run sql query" keyword or an equivalent to fetch the data.
The downside of this is that all of the permutations are considered a single test case with multiple keywords, versus multiple test cases with a single keyword.
If you want to have one test case per row in a database, you could write a script that does the query, generates a test suite file using the results of the query, and then runs pybot on the generated file.

Resources