export a relational Snowflake table as valid JSON or XML - snowflake-cloud-data-platform

Question

I have a table in Snowflake and want to export it as JSON or XML but am struggling to get all of the records from my table into one valid output. I'm not very familiar with JSON formats but have made some progress.
Here is a sample of my table:
USER_ID
PRODUCT_EXTERNAL_ID
PRODUCT_PRICE
"1770607c104641caaf5fde0e433b76a6"
["817720022911","817720022782"]
[39.99,39.99]
"27d4c6a6559e48629dc558c94080c5ca"
["882709630449" ]
[43.65 ]
Here is the query I am using to get the results below:
create or replace temp table json_output as
with cte as
(
select
to_json(
object_construct('order_details',
array_agg(
object_construct('user_id', user_id,
'PRODUCT_EXTERNAL_ID',PRODUCT_EXTERNAL_ID,
'PRODUCT_PRICE',PRODUCT_PRICE
)
)
)
)as output
from one_row_per_order
group by user_id
)
select *
from cte;
When querying the above table, this is the format of the JSON (both records combined):
{
"order_details": [{
"PRODUCT_EXTERNAL_ID": ["817720022911", "817720022782"],
"PRODUCT_PRICE": [ 39.99, 39.99 ],
"USER_ID": "1770607c104641caaf5fde0e433b76a6",
}]
} {
"PRODUCT_EXTERNAL_ID": [ "882709630449" ],
"PRODUCT_PRICE": [ 43.65 ],
"USER_ID": "27d4c6a6559e48629dc558c94080c5ca"
}]
}
However, from my understanding the above isn't valid JSON - see the where the first object ends and the next begins (the first close-square-bracket is in the wrong place). Instead, this is my desired output format (valid JSON syntax) but I can't achieve it:
{
"order_details": [{
"PRODUCT_EXTERNAL_ID": ["12345", "26810"],
"PRODUCT_PRICE": [17.99, 10.99],
"USER_ID": "1770607c104641caaf5fde0e433b76a6"
},
{
"PRODUCT_EXTERNAL_ID": ["4578", "9876", "8888"],
"PRODUCT_PRICE": [4.99, 12.50],
"USER_ID": "27d4c6a6559e48629dc558c94080c5ca"
}]
}
I have tried array_agg but reach the maximum size limit imposed by LOB objects in Snowflake (I actually have 70k records my table with many more columns than shown in the example).I also tried TO_XML but ran into the same issue where each row is treated as an object where I want all rows combined into one object.
Have you any suggestions as to how to format the JSON please? As mentioned, I would accept valid XML too.

Answer 1

Original Table -
select * from test_json;
+---------------------+---------------------+---------------+
| USER_ID | PRODUCT_EXTERNAL_ID | PRODUCT_PRICE |
|---------------------+---------------------+---------------|
| 1770607c90283414343 | [ | [ |
| | "817823487432", | 39.99, |
| | "817982433987" | 39.99 |
| | ] | ] |
| 27d4c90283414343 | [ | [ |
| | "882487432" | 43.65 |
| | ] | ] |
+---------------------+---------------------+---------------+
Query to return valid JSON, as converted from values in table; however, we still have limits for size as imposed in snowflake for various functions such as ARRAY_AGG -
select object_construct('order_details',array_agg(json_1)) as json_val
from
(
select '1' as id,
OBJECT_CONSTRUCT('PRODUCT_EXTERNAL_ID',
PRODUCT_EXTERNAL_ID,
'PRODUCT_PRICE',
PRODUCT_PRICE,
'USER_ID',user_id) as json_1
from test_json)
group by id;
+----------------------------------------+
| JSON_VAL |
|----------------------------------------|
| { |
| "order_details": [ |
| { |
| "PRODUCT_EXTERNAL_ID": [ |
| "817823487432", |
| "817982433987" |
| ], |
| "PRODUCT_PRICE": [ |
| 39.99, |
| 39.99 |
| ], |
| "USER_ID": "1770607c90283414343" |
| }, |
| { |
| "PRODUCT_EXTERNAL_ID": [ |
| "882487432" |
| ], |
| "PRODUCT_PRICE": [ |
| 43.65 |
| ], |
| "USER_ID": "27d4c90283414343" |
| } |
| ] |
| } |
+----------------------------------------+
Another approach (which though does not meet 'combine all' requirement), is load all records as JSON in JSON format stage file and then we can use that as needed.
As per below test case - for 70K record the input bytes will be around 14MB.
create stage test_int_stage file_format=(type=JSON);
+-------------------------------------------------+
| status |
|-------------------------------------------------|
| Stage area TEST_INT_STAGE successfully created. |
+-------------------------------------------------+
copy into #TEST_INT_STAGE from (select
(OBJECT_CONSTRUCT('PRODUCT_EXTERNAL_ID',
PRODUCT_EXTERNAL_ID,
'PRODUCT_PRICE',
PRODUCT_PRICE,
'USER_ID',user_id)) as json_col
from test_json);
+---------------+-------------+--------------+
| rows_unloaded | input_bytes | output_bytes |
|---------------+-------------+--------------|
| 2 | 209 | 143 |
+---------------+-------------+--------------+
list #test_int_stage;
+-----------------------------------+------+
| name | size |
|-----------------------------------+------+
| test_int_stage/data_0_0_0.json.gz | 144 |
+-----------------------------------+------+