NPGSQL SELECT FROM dblink issue - database

I have created a Blazor application with .Net6 framework. This application is generating multiple threads to transfer data from multiple PostgreSQL databases (tenant) to a single PostgreSQL database (group) using dblink.
DBLink group-tenant connections "reporting_cn_salesinvoicing{x}" have been created at the moment of the database creation.
For example if the group has three tenants, it also has three dblink servers:
reporting_lk_salesinvoicing1
reporting_lk_salesinvoicing2
reporting_lk_salesinvoicing3
And also has three dblink connections:
reporting_cn_salesinvoicing1
reporting_cn_salesinvoicing2
reporting_cn_salesinvoicing3
DBLink connections are being checked every time before a command is executed using the following command:
SELECT dblink_connect('reporting_cn_salesinvoicing{0}', 'reporting_lk_salesinvoicing{0}')
Each thread is operating a foreach loop for every table transfer with the following command:
INSERT INTO public."sales_document_line" ("tenantx_id", "id", "sales_header_id", "description", "quantity", "amount") SELECT 1, "id", "sales_header_id", "description", "quantity", "amount" FROM dblink('reporting_cn_salesinvoicing{x}', 'SELECT {x}, "id", "sales_header_id", "description", "quantity", "amount" FROM public."sales_document_line" ') AS rt(id uuid, sales_header_id uuid, description text, quantity numeric, amount numeric) ON CONFLICT ("tenantx_id", "id") DO UPDATE SET "sales_header_id" = excluded."sales_header_id", "description" = excluded."description", "quantity" = excluded."quantity", "amount" = excluded."amount"
If the above query is executed in parallel for different tenant databases usually it throws the following error:
On Postgres11
2F003: password is required
DETAIL: Non-superusers must provide a password in the connection string.
On Postgres14
08001: could not establish connection
DETAIL: missing "=" after "reporting_cn_salesinvoicing{x}" in connection info string
While some other time it exposés:
'An exception has been raised that is likely due to a transient failure.'
This error seems misleading to me because if there is only one thread is working.
I have tried to set lot of different connection parameters or changes the threading population without success.
UPDATE
After lot of hours, seems that even with parallel connections if the credentials are specified on the dblink instead of named connnetion it works:
INSERT INTO public."sales_document_line" ("tenantx_id", "id", "sales_header_id", "description", "quantity", "amount") SELECT {x}, "id", "sales_header_id", "description", "quantity", "amount" FROM dblink('dbname=salesinvoicing-qa{x} port=5432 host=xxxxxxxxxxx user=postgres password=xxxxxxxxxxxx', 'SELECT "id", "sales_header_id", "description", "quantity", "amount" FROM public."sales_document_line" ') AS rt(id uuid, sales_header_id uuid, description text, quantity numeric, amount numeric) ON CONFLICT("tenantx_id", "id") DO UPDATE SET "sales_header_id" = excluded."sales_header_id", "description" = excluded."description", "quantity" = excluded."quantity", "amount" = excluded."amount";
Any ideas?

Related

Is it possible to parse json by using select statement in Netezza?

I have json data in one of the column of my table and I would like to parse json data by using select statement in Netezza. I am not able to figure it out.
Can you all help me to solve this problem?
Let's say I have TableA and this table has column Customer_detail. data from customer_detail field lookss like this
'{"Customer":[{"id":"1","name":"mike","address":"NYC"}]}'
Now I would like to query id from customer object of customer_detail column.
Thanks in advance.
From NPS 11.1.0.0 onwards, you can parse and use json datatype itself in NPS.
Here's an example
SYSTEM.ADMIN(ADMIN)=> create table jtest(c1 jsonb);
CREATE TABLE
SYSTEM.ADMIN(ADMIN)=> insert into jtest values('{"name": "Joe Smith", "age": 28, "sports": ["football", "volleyball", "soccer"]}');
INSERT 0 1
SYSTEM.ADMIN(ADMIN)=> insert into jtest values('{"name": "Jane Smith", "age": 38, "sports": ["volleyball", "soccer"]}');
INSERT 0 1
SYSTEM.ADMIN(ADMIN)=> select * from jtest;
C1
----------------------------------------------------------------------------------
{"age": 28, "name": "Joe Smith", "sports": ["football", "volleyball", "soccer"]}
{"age": 38, "name": "Jane Smith", "sports": ["volleyball", "soccer"]}
(2 rows)
SYSTEM.ADMIN(ADMIN)=> select c1 -> 'name' from jtest where c1 -> 'age' > 20::jsonb ;
?COLUMN?
--------------
"Joe Smith"
"Jane Smith"
(2 rows)
You can refer to https://www.ibm.com/support/knowledgecenter/SSTNZ3/com.ibm.ips.doc/postgresql/dbuser/r_dbuser_functions_expressions.html for more details as well.
Looking at the comment you put above, something like
select customer_detail::json -> 'Customer' -> 0 -> 'id' as id,
customer_detail::json -> 'Customer' -> 0 -> 'name' as name
from ...
This will parse the text to json during every execution. A more performant would be to convert customer_detail to jsonb datatype
If the NPS version is below 11.1.x then the json handling needs to be done (a) externally as in using sql to get the json data and then processing it outside the database or (b) using UDF - creating a UDF that supports json parsing
E.g -
Using the programming language of choice, process the json external to SQL
import nzpy # install using "python3 -m pip install nzpy"
import os
import json
# assume NZ_USER, NZ_PASSWORD, NZ_DATABASE and NZ_HOST are set
con = nzpy.connect(user=os.environ["NZ_USER"],
password=os.environ["NZ_PASSWORD"], host=os.environ["NZ_HOST"],
database=os.environ["NZ_DATABASE"], port=5480)
with con.cursor() as cur:
cur.execute('select customer_detail from ...')
for customer_detail in cur.fetch_all():
c = json.loads(customer_detail)
print((c['Customer'][0]['name'], c['Customer'][0]['id']))
Or create a UDF that parses json and use that in the SQL query
If none of those are options, and the json is always well formatted (ie. no new lines, only one key called "id" and one key called "name", etc) then a regex may be a way around, though its not recommended since its not a real json parser
select regexp_extract(customer_detail,
'"id"[[:space:]]*:[[:space:]]*"([^"]+)"', 1, 1) as id,
regexp_extract(customer_detail,
'"name"[[:space:]]*:[[:space:]]*"([^"]+)"', 1, 1) as name
....

Snowflake dynamic masking masks underlying table: the derivative tables are not masked and views become empty?

I have a raw table which has a variant column of json data.
There are some normal views (not materialised view) and tables are created using the json events from the raw table.
After applied a masking policy using UDF on the variant column of the raw table when the role is bi_analyst, there are two issues I found:
The tables derived from the underlying table are not masked with bi_analyst role;
The views derived using the underlying table become empty with bi_analyst role;
Does anyone know why this is happened? dose this dynamic masking feature not support views on underlying table?
What I would like to do is masking the underlying data and all the tables and views coming from it are also masked with the specified role.
It is easy to deal with those tables, since I can just apply the masking policy on them as well.
However, I have no idea about the views. How can I still access the views with the role, which should able to see the data but not the sensitive columns?
The UDF is:
-- JavaScript UDF to mask pii data --
use role ACCOUNTADMIN;
CREATE OR REPLACE FUNCTION full_address_masking(V variant)
RETURNS variant
LANGUAGE JAVASCRIPT
AS
$$
if ("detail" in V) {
if ("latitude" in V.detail) {
V.detail.latitude = "******";
}
if ("longitude" in V.detail) {
V.detail.longitude = "******";
}
if ("customerAddress" in V.detail) {
V.detail.customerAddress = "******";
}
}
return V;
$$;
The Masking policy is:
-- Create a masking policy using JavaScript UDF --
create or replace masking policy json_address_mask as (val variant) returns variant ->
CASE
WHEN current_role() IN ('ACCOUNTADMIN') THEN val
WHEN current_role() IN ('BI_ANALYST') THEN full_address_masking(val)
ELSE full_address_masking(val)
END;
The sql command to set masking policy on raw data is:
-- Set masking policy --
use role ACCOUNTADMIN;
alter table DB.PUBLIC.RAW_DATA
modify column EVERYTHING
set masking policy json_address_mask;
The masking policy is applied on the variant column EVERYTHING, which data structure looks like:
{
"detail": {
"customAddress": "******",
"id": 1,
"latitude": "******",
"longitude": "******"
},
"source": "AAA"
}
A derivative table is:
create or replace table DB.SCHEMA_A.TABLE_A
as
select * from DB.PUBLIC.RAW_DATA
where everything:source='AAA';
grant select on table DB.schema_A.table_A to role bi_analyst;
A view is:
create or replace view DB.SCHEMA_A.VIEW_A as (
select
everything:account::string as account,
everything:detail:latitude::float as detail_latitude,
everything:detail:longitude::float as detail_longitude,
from
DB.PUBLIC.RAW_DATA
where
everything:source::string = 'AAA'
grant select on view DB.SCHEMA_A.VIEW_A to role bi_analyst;
The result is that RAW_DATA is masked, TABLE_A is not masked at all, VIEW_A gets 0 rows returned when querying data with BI_ANALYST role.
#1 - When you create a table from a table that has masked data, you're going to get the data that the role creating the new table has access to in the masked table. So, in your example, TABLE_A has unmasked data, because it was created by a role that has access to it. The masking policy does not automatically get applied to the new table.
#2 - As for #2, I believe your only issue is that the JSON in your example isn't correctly formed, which is why you are getting NULL values. When I fixed this json to the following, it works fine using the same function and masking policy that you've posted:
{
"detail":{
"latitude": 132034034.00,
"longitude": 12393438583732,
"id": 1,
"customAddress" : "XXX Road, XXX city, UK"
},
"source": "AAA"
}
Masked Result:
{
"detail": {
"customAddress": "XXX Road, XXX city, UK",
"id": 1,
"latitude": "******",
"longitude": "******"
},
"source": "AAA"
}
The issue of the tables not been masked is explained well by #Mike in his answer. The solution can be just to create the derivative tables using a role which is restricted by the masking policy.
The issue of the views is about the type of masked value "******", which is a string type, while the actual type of fields latitude and longitude are float.
When creating the view, I still cast the latitude and longitude fields to float type:
create or replace view DB.SCHEMA_A.VIEW_A as (
select
everything:account::string as account,
everything:detail:latitude::float as detail_latitude,
everything:detail:longitude::float as detail_longitude,
from
DB.PUBLIC.RAW_DATA
where
everything:source::string = 'AAA'
There is a hidden error of casting "******" to float but snowflake still go ahead and create the view. But when I query the data with the BI_ANALYST role, it returns 0 row.
So the workaround is casting those fields to variant type:
create or replace view DB.SCHEMA_A.VIEW_A as (
select
everything:account::string as account,
everything:detail:latitude::variant as detail_latitude,
everything:detail:longitude::variant as detail_longitude,
from
DB.PUBLIC.RAW_DATA
where
everything:source::string = 'AAA'
Which is not perfect because it completely changed the definition of the view, none of roles can get the actual float/number type of the data, even including accountadmin

Use of functions in where clause in feathers

The problem is that I need to include a function in the where clause of the SQL generated by the feathers
If the where clause is assigned only the function the SQL is generated correct, but I of cause missing the status part
options.where=fn
SELECT
"id", "feature_name", "status", "priority", "label", "st_asgeojson" FROM "gis34_registration"."geojson_tasks" AS "geojson_tasks"
WHERE
ST_Intersects(geom, ST_transform(ST_MakeEnvelope(12.370044675467057, 55.73287419556607, 12.385791865781385, 55.7422305387, 4326), 25832))
This is the sql that I need feathers to generate
options.where.status='Registreret'
options.where.fn=fn
SELECT
"id", "feature_name", "status", "priority", "label", "st_asgeojson" FROM "gis34_registration"."geojson_tasks" AS "geojson_tasks"
WHERE
status = 'Registreret' AND
fn = ST_Intersects(geom, ST_transform(ST_MakeEnvelope(12.370044675467057, 55.73287419556607, 12.385791865781385, 55.7422305387, 4326), 25832))
This is the sql that I need feathers to generate
SELECT
"id", "feature_name", "status", "priority", "label", "st_asgeojson" FROM "gis34_registration"."geojson_tasks" AS "geojson_tasks"
WHERE
status = 'Registreret' AND
ST_Intersects(geom, ST_transform(ST_MakeEnvelope(12.370044675467057, 55.73287419556607, 12.385791865781385, 55.7422305387, 4326), 25832))
How do i get the feathers to generate SQL with a function and attributes together?
The answer is to use the predefined porperty $and. The code below generates the required SQL
options.where.status='Registreret'
options.where.$and=fn

How to import documents that have arrays with the Cosmos DB Data Migration Tool

I'm trying to import documents from a SQL Server database. Each document will have a list of products that a customer has bought, for example:
{
"name": "John Smith"
"products": [
{
"name": "Pencil Sharpener"
"description": "Something, this and that."
},
{
"name": "Pencil case"
"description": "A case for pencils."
}
]
}
In the SQL Server database, the customer and products are stored in separate tables with a one-to-many relationship between the customer and products:
Customer
Id INT
Name VARCHAR
Product
Id INT
CustomerId INT (FK)
Name VARCHAR
Description VARCHAR
I've checked through the documentation , but can't see any mention of how to write the SQL query to map the one-to-many relationships to a single document.
I think there may be a way to do it as on the Target Information step (and when selecting DocumentDB - Bulk import (single partition collections)) there's the option to provide a bulk insert stored procedure. Maybe the products can be assigned to the document's products array from within there. I'm just not sure how to go about doing it as I'm new to Cosmos DB.
I hope that's clear enough and thanks in advance for your help!
It seems that you’d like to return products info formatted as json when you import data from SQL Server using the Azure Cosmos DB: DocumentDB API Data Migration tool. Based on your customer and products table structure and your requirement, I do the following test, which works fine on my side. You can refer to it.
Import data from SQL Server to JSON file
Query
select distinct c.Name, (SELECT p.Name as [name], p.[Description] as [description] from [dbo].[Product] p where c.Id = p.CustomerId for JSON path) as products
from [dbo].[Customer] c
JSON output
[
{
"Name": "Jack",
"products": null
},
{
"Name": "John Smith",
"products": "[{\"name\":\"Pencil Sharpener\",\"description\":\"Something, this and that\"},{\"name\":\"Pencil case\",\"description\":\"A case for pencils.\"}]"
}
]
Parsing the products
On the 'Target Information' step, you'll need to use your own version of BulkTransformationInsert.js. On line 32 is a 'transformDocument' function where you can edit the document. The following will parse the products and then assign them back to document before returning;
function transformDocument(doc) {
if (doc["products"]) {
let products = doc["products"];
let productsArr = JSON.parse(products);
doc["products"] = productsArr;
}
return doc;
}

How to Update Doc in Cloudant no sql-db

I am following Link to integrate cloudant no sql-db.
There are methods given create DB, Find all, count, search, update. Now I want to update one key value in one of my DB doc file. how Can i achieve that. Document shows like
updateDoc (name, doc)
Arguments:
name - database name
docID - document to update
but when i pass my database name and doc ID its throwing database already created can not create db. But i wanted to updated doc. So can anyone help me out.
Below is one of the doc of may table 'employee_table' for reference -
{
"_id": "0b6459f8d368db408140ddc09bb30d19",
"_rev": "1-6fe6413eef59d0b9c5ab5344dc642bb1",
"Reporting_Manager": "sdasd",
"Designation": "asdasd",
"Access_Level": 2,
"Employee_ID": 123123,
"Employee_Name": "Suhas",
"Project_Name": "asdasd",
"Password": "asda",
"Location": "asdasd",
"Project_Manager": "asdas"
}
So I want to update some values from above doc file of my table 'employee_table'. So what parameters I have to pass to update.
first of all , there is no concept named table in no sql world.
second, to update document first you need to get document based on any input field of document. you can also use Employee_ID or some other document field. then use database.get_query_result
db_name = 'Employee'
database = client.create_database(db_name,throw_on_exists=False)
EmployeeIDValue = '123123'
#here throw_on_exists=False meaning dont throw error if DB already present
def updateDoc (database, EmployeeIDValue):
results = database.get_query_result(
selector= { 'Employee_ID': {'$eq': EmployeeIDValue} }, )
for result in results:
my_doc_id = result["_id"]
my_doc = database[my_doc_id] #===> this give you your document.
'''Now you can do update'''
my_doc['Employee_Name'] = 'XYZ'
my_doc.save() ====> this command updates current document

Resources