Within my Postgres database, I have a column (type=JSONB) that contains an API response that represents someone response to a Survey Monkey survey. Within this JSON object, there is an array called pages that contains at least 1 item, and something more than 1 item. Then within pages, there is another array called questions. This array also contains at least 1 item, and more often more than 1 item.
I am trying to write a Postgres query to extract the responses from all the questions seen within pages and questions. The final format should be 1 record per question. If the survey has 5 questions, that would create 5 rows. If the survey has 100 questions, across 5 pages, that would mean 100 rows.
+────────────+────────────────────────────────────────────+──────────────────────+
| id | heading | simple_text |
+────────────+────────────────────────────────────────────+──────────────────────+
| 775249149 | I am confident that I am the best. | Somewhat Agree |
| 775249153 | I currently have the skills to be amazing | Somewhat Agree |
| 775249166 | How long until I win the lottery? | 6 to 12 months more |
+────────────+────────────────────────────────────────────+──────────────────────+
I've been trying to use the function json_array_elements, but I'm experiencing some errors when trying to pass in an array-of-arrays.
{
"id": "12345",
"href": "https://api.surveymonkey.com",
"pages": [
{
"id": "142250690",
"questions": [
{
"id": "775249149",
"family": "matrix",
"answers": [
{
"row_id": "5133514018",
"choice_id": "5133514023",
"simple_text": "Somewhat Agree",
"choice_metadata": {
"weight": "5"
}
}
],
"heading": "I am confident that I am the best.",
"subtype": "rating"
},
{
"id": "775249153",
"family": "matrix",
"answers": [
{
"row_id": "5133514112",
"choice_id": "5133514117",
"simple_text": "Somewhat Agree",
"choice_metadata": {
"weight": "5"
}
}
],
"heading": "I currently have the skills to be amazing",
"subtype": "rating"
},
{
"id": "775249166",
"family": "matrix",
"answers": [
{
"row_id": "5133514278",
"choice_id": "5133514280",
"simple_text": "6 to 12 months more",
"choice_metadata": {
"weight": "2"
}
}
],
"heading": "How long until I win the lottery?",
"subtype": "rating"
}
]
}
],
"survey_id": "123456789",
"total_time": 29,
"date_created": "2022-07-25T23:08:36+00:00",
"recipient_id": "",
"date_modified": "2022-07-25T23:09:06+00:00",
"email_address": "",
"collection_mode": "default",
"response_status": "completed",
}
Try to use json_array_elements() twice:
http://sqlfiddle.com/#!17/9eecb/94455
select v->'id', v->'heading', v->'subtype', v->'answers'->0->'simple_text' from (
select json_array_elements(value->'questions')::jsonb as v from json_array_elements('{
"id": "12345",
"href": "https://api.surveymonkey.com",
"pages": [
{
"id": "142250690",
"questions": [
{
"id": "775249149",
"family": "matrix",
"answers": [
{
"row_id": "5133514018",
"choice_id": "5133514023",
"simple_text": "Somewhat Agree",
"choice_metadata": {
"weight": "5"
}
}
],
"heading": "I am confident that I am the best.",
"subtype": "rating"
},
{
"id": "775249153",
"family": "matrix",
"answers": [
{
"row_id": "5133514112",
"choice_id": "5133514117",
"simple_text": "Somewhat Agree",
"choice_metadata": {
"weight": "5"
}
}
],
"heading": "I currently have the skills to be amazing",
"subtype": "rating"
},
{
"id": "775249166",
"family": "matrix",
"answers": [
{
"row_id": "5133514278",
"choice_id": "5133514280",
"simple_text": "6 to 12 months more",
"choice_metadata": {
"weight": "2"
}
}
],
"heading": "How long until I win the lottery?",
"subtype": "rating"
}
]
},
{
"id": "142250690",
"questions": [
{
"id": "775249199",
"family": "matrix",
"answers": [
{
"row_id": "5133514278",
"choice_id": "5133514280",
"simple_text": "6 to 12 months more",
"choice_metadata": {
"weight": "2"
}
}
],
"heading": "qweqweqweqwqqqq?",
"subtype": "rating"
}
]
}
],
"survey_id": "123456789",
"total_time": 29,
"date_created": "2022-07-25T23:08:36+00:00",
"recipient_id": "",
"date_modified": "2022-07-25T23:09:06+00:00",
"email_address": "",
"collection_mode": "default",
"response_status": "completed"
}'::jsonb::json->'pages')
) t;
UPDATE
If you need to process data from table contained described json rows, you need something like this:
http://sqlfiddle.com/#!17/16b65/1
I'm trying to fetch data from API whose response is nested json, I used collection reference to capture all the results but the results have arrays in which by default only first value is captured. I checked the documentation also (https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-schema-and-type-mapping ) and there it has mentioned only single array is supported for such operation.
Please let me know how i can capture all the nested array json values.
I tried using the Map complex values to string option also but it's not working.
Please check the image link for the json result and ADF mapping done ADF Mapping JSON Response
{
"story_count": 2325,
"next": "?page=2",
"previous": null,
"results": [
{
"id": 20111247056317,
"title": "ANZ exec bonuses cut, salaries hiked",
"summary": "[Source: AFR Online] ANZ chief executive Shayne Elliott and his executive team have had short-term bonuses cut by 50 per cent or more after profits slumped 42 per cent. However, they have almost all had sizeable increases in fixed pay that should stop them from harbouring any ill feelings towards their Melbourne-based employer. ",
"source_url": "https://global.factiva.com/en/du/article.asp?accountid=9ERN000600&accessionno=AFNROL0020201109egb9000p1",
"source_name": "Factiva",
"pub_date": "2020-11-09T06:52:18Z",
"image_url": "",
"attachments": [],
"Triggers": [
{
"id": 15174,
"name": "Cost Reduction",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 15178,
"name": "Tax Risk",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 24194,
"name": "Rewards & benefits",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Firms": [
{
"id": 17984,
"name": "AUSTRALIA AND NEW ZEALAND BANKING GROUP LIMITED",
"logo": "//112233.contify.com/client_data/custom_tag/logo/eg1lrjgo_400x400-17984.jpg"
}
],
"Duns Number": [
{
"id": 18498,
"name": "753682830",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Channels": [
{
"id": 17,
"name": "News and Other Websites",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Content Types": [
{
"id": 3,
"name": "News Articles",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Sources": [
{
"id": 68636,
"name": "Factiva",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"previews": [],
"duplicates": [],
"duplicate_count": 0
},
{
"id": 20102741904140,
"title": "HSBC to cut up to 300 jobs in UK commercial banking unit",
"summary": "[Source: FinTech Futures News] HSBC has launched a restructuring of its commercial banking business in Britain. A source familiar with the matter tells Reuters that the plan will result in around 300 job losses. The cuts are part of a wider restructuring announced in February. ",
"source_url": "https://global.factiva.com/en/du/article.asp?accountid=9ERN000600&accessionno=FTEFN00020201027egar0002w",
"source_name": "Factiva",
"pub_date": "2020-10-27T00:00:00Z",
"image_url": "",
"attachments": [],
"Triggers": [
{
"id": 15174,
"name": "Cost Reduction",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 24196,
"name": "Restructuring",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Firms": [
{
"id": 17967,
"name": "HSBC Holdings PLC",
"logo": "//112233.contify.com/client_data/custom_tag/logo/nbzwhpqp_400x400-17967.png"
},
{
"id": 35194,
"name": "HSBC Holdings",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Duns Number": [
{
"id": 18655,
"name": "288451024",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Channels": [
{
"id": 17,
"name": "News and Other Websites",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Content Types": [
{
"id": 3,
"name": "News Articles",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Sources": [
{
"id": 68636,
"name": "Factiva",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"previews": [],
"duplicates": [],
"duplicate_count": 0
},
{
"id": 20102641537340,
"title": "AT&T cuts more than 8,700 workers in third quarter amid COVID-19",
"summary": "[Source: Dallas Business Journal Online] AT&T is making some deeper cuts with its employee base. The Dallas telecommunications and media company shrunk the number of employees by 8,720 in the third quarter to less than 235,000, according to information on its investor relations website. ",
"source_url": "https://global.factiva.com/en/du/article.asp?accountid=9ERN000600&accessionno=DALBJO0020201026egap00001",
"source_name": "Factiva",
"pub_date": "2020-10-25T00:00:00Z",
"image_url": "",
"attachments": [],
"Triggers": [
{
"id": 15174,
"name": "Cost Reduction",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 23058,
"name": "Covid-19",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Firms": [
{
"id": 17963,
"name": "AT&T Inc.",
"logo": "//112233.contify.com/client_data/custom_tag/logo/a7jsrall_400x400-17963.png"
}
],
"Duns Number": [
{
"id": 18497,
"name": "108024050",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Channels": [
{
"id": 17,
"name": "News and Other Websites",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Content Types": [
{
"id": 3,
"name": "News Articles",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Sources": [
{
"id": 68636,
"name": "Factiva",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"previews": [],
"duplicates": [
{
"id": 20102641627014,
"title": "AT&T cuts more than 8,700 workers in third quarter amid Covid-19",
"summary": "[Source: Louisville Business First Online] AT&T is making some deeper cuts with its employee base. The Dallas telecommunications and media company shrunk the number of employees by 8,720 in the third quarter to less than 235,000, according to information on its investor relations website. ",
"source_url": "https://global.factiva.com/en/du/article.asp?accountid=9ERN000600&accessionno=BSFLVO0020201026egaq00001",
"source_name": "Factiva"
},
{
"id": 20102741677548,
"title": "AT&T cuts more than 8,700 workers in third quarter amid Covid-19",
"summary": "[Source: Orlando Business Journal] AT&T is making some deeper cuts with its employee base. The Dallas-based telecommunications and media company (NYSE: T) shrunk the number of its employees by 8,720 in the third quarter to less than 235,000, according to information on its investor relations website. ",
"source_url": "https://global.factiva.com/en/du/article.asp?accountid=9ERN000600&accessionno=ORBJ000020201026egaq0005m",
"source_name": "Factiva"
}
],
"duplicate_count": 2
},
{
"id": 20102641660233,
"title": "Lufthansa considers cutting 30,000 jobs due to fall in air traffic",
"summary": "[Source: French Collection] German airline Lufthansa is facing a fall in air traffic due to the Covid-19 pandemic and consequently up to 30,000 jobs are threatened, its management said. Lufthansa is determined to keep at least 100,000 of the 130,000 positions in total at the group, Carsten Spohr, CEO of the airline, explained. ",
"source_url": "https://global.factiva.com/en/du/article.asp?accountid=9ERN000600&accessionno=FRECOL0020201026egaq000jh",
"source_name": "Factiva",
"pub_date": "2020-10-26T00:00:00Z",
"image_url": "",
"attachments": [],
"Triggers": [
{
"id": 15174,
"name": "Cost Reduction",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Firms": [
{
"id": 18140,
"name": "Deutsche Lufthansa AG",
"logo": "//112233.contify.com/client_data/custom_tag/logo/-99kuzpm_400x400-18140.jpg"
},
{
"id": 35588,
"name": "Lufthansa Group",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Duns Number": [
{
"id": 18584,
"name": "315000893",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Channels": [
{
"id": 17,
"name": "News and Other Websites",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Content Types": [
{
"id": 3,
"name": "News Articles",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Sources": [
{
"id": 68636,
"name": "Factiva",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"previews": [],
"duplicates": [],
"duplicate_count": 0
},
{
"id": 20102541378844,
"title": "ExxonMobil chief warns US, Canada jobs cuts coming soon",
"summary": "[Source: Agence France Presse] Global oil giant ExxonMobil will release details soon of expected job cuts in the United States and Canada, the company's chief Darren Woods said in a letter to employees. \"As difficult as this is, I hope you understand it is critically important for the future of our company,\" Woods said in a statement Wednesday. ",
"source_url": "https://global.factiva.com/en/du/article.asp?accountid=9ERN000600&accessionno=AFPR000020201022egam00dqh",
"source_name": "Factiva",
"pub_date": "2020-10-22T13:04:58Z",
"image_url": "",
"attachments": [],
"Triggers": [
{
"id": 15174,
"name": "Cost Reduction",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 15178,
"name": "Tax Risk",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Firms": [
{
"id": 24236,
"name": "Exxon Mobil Corporation",
"logo": "//112233.contify.com/client_data/custom_tag/logo/ytocyynn_400x400-24236.jpg"
}
],
"Duns Number": [
{
"id": 24342,
"name": "1213214",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 34007,
"name": "001213214",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Channels": [
{
"id": 17,
"name": "News and Other Websites",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Content Types": [
{
"id": 3,
"name": "News Articles",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Sources": [
{
"id": 68636,
"name": "Factiva",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"previews": [],
"duplicates": [],
"duplicate_count": 0
},
{
"id": 20102541377022,
"title": "Cathay Pacific to lay off 5,900 employees, close low-cost subsidiary",
"summary": "[Source: Kazinform International News Agency] HONG. KONG. KAZINFORM Hong Kong s flag carrier Cathay Pacific announced Wednesday it would lay off 5,900 employees and close low-cost subsidiary Cathay Dragon following corporate restructuring in response to the coronavirus s effect on aviation.«The restructuring will enable the company to secure its future, so it can protect as many jobs as possible, whilst meeting its responsibilities to the Hong Kong aviation hub and its customers,» the carrier said in a statement, EFE-EPA reports. ",
"source_url": "https://global.factiva.com/en/du/article.asp?accountid=9ERN000600&accessionno=KAZNAE0020201021egal000b7",
"source_name": "Factiva",
"pub_date": "2020-10-21T12:13:00Z",
"image_url": "",
"attachments": [],
"Triggers": [
{
"id": 15174,
"name": "Cost Reduction",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 15178,
"name": "Tax Risk",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 23058,
"name": "Covid-19",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 24196,
"name": "Restructuring",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Firms": [
{
"id": 35127,
"name": "Cathay Pacific Airways Ltd.",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Duns Number": [
{
"id": 33981,
"name": "686093188",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Channels": [
{
"id": 17,
"name": "News and Other Websites",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Content Types": [
{
"id": 3,
"name": "News Articles",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Sources": [
{
"id": 68636,
"name": "Factiva",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"previews": [],
"duplicates": [],
"duplicate_count": 0
}
]
}
I would recommend you consider calling a Stored Procedure activity instead of a copy activity. Then parse and store the JSON inside the stored procedure. I have written up this approach in the following answer. You can ignore the For loop if it’s a single REST API call. I’m working under the assumption that your destination database is SQL Server or Azure SQL Database, but correct me if that’s incorrect.
How can I extract only the objects id and name from this JSON via jq?
The output should be look like the format below. This is just an example for the required output, the real one that I need is to catch the whole values id and name like in the JSON source.
This is the required output:
{
"name": "Auto Body Styles",
"id": "1.1"}
{
"name": "Convertible",
"id": "1.1.2"
}
This is the JSON source file:
{
"name": "Automotive",
"id": "1",
"categories": [
{
"name": "Auto Body Styles",
"id": "1.1",
"categories": [
{
"name": "Commercial Trucks",
"id": "1.1.1"
},
{
"name": "Convertible",
"id": "1.1.2"
},
{
"name": "Coupe",
"id": "1.1.3"
},
{
"name": "Crossover",
"id": "1.1.4"
},
{
"name": "Hatchback",
"id": "1.1.5"
},
{
"name": "Microcar",
"id": "1.1.6"
},
{
"name": "Minivan",
"id": "1.1.7"
},
{
"name": "Off-Road Vehicles",
"id": "1.1.8"
},
{
"name": "Pickup Trucks",
"id": "1.1.9"
},
{
"name": "Sedan",
"id": "1.1.10"
},
{
"name": "Station Wagon",
"id": "1.1.11"
},
{
"name": "SUV",
"id": "1.1.12"
},
{
"name": "Van",
"id": "1.1.13"
}
]
},
{
"name": "Auto Buying and Selling",
"id": "1.2"
},
{
"name": "Auto Insurance",
"id": "1.3"
},
{
"name": "Auto Parts",
"id": "1.4"
},
{
"name": "Auto Recalls",
"id": "1.5"
},
{
"name": "Auto Repair",
"id": "1.6"
},
{
"name": "Auto Safety",
"id": "1.7"
},
{
"name": "Auto Shows",
"id": "1.8"
},
{
"name": "Auto Technology",
"id": "1.9",
"categories": [
{
"name": "Auto Infotainment Technologies",
"id": "1.9.1"
},
{
"name": "Auto Navigation Systems",
"id": "1.9.2"
},
{
"name": "Auto Safety Technologies",
"id": "1.9.3"
}
]
},
{
"name": "Auto Type",
"id": "1.10",
"categories": [
{
"name": "Budget Cars",
"id": "1.10.1"
},
{
"name": "Certified Pre-Owned Cars",
"id": "1.10.2"
},
{
"name": "Classic Cars",
"id": "1.10.3"
},
{
"name": "Concept Cars",
"id": "1.10.4"
},
{
"name": "Driverless Cars",
"id": "1.10.5"
},
{
"name": "Green Vehicles",
"id": "1.10.6"
}
]
} ] }
i think what you want is Recursive Descent: ..
cat car.json | jq -r '.. | [.name?, .id?] | select(length>0) | #tsv'
to produce something like in your example,
cat car.json | jq -r '.. | {name:.name?, id:.id?}'
H have json document with array. As I can't add to beginning of array with push or addtoset I need to sort array. Example
{
"Component": [
{
"Id": "PDP-1",
"Links": {"Link": [
{
"Text": "Western Division",
"Url": "/1x7-en70ai/last-minute-holidays-western-division",
"Title": "Last minute holidays Western Division"
},
{
"Text": "Browse Regions ",
"Url": "/1x7-en6uly-10ts/last-minute-holidays-gambia/regions",
"Title": "Last minute holidays Gambia",
"Style": "BrowseForMore"
},
{
"Text": "City of Banjul",
"Url": "/1x6-en6vq7/holidays-city-of-banjul",
"Title": "City of Banjul Holidays"
},
{
"Text": "Western Division",
"Url": "/1x6-en70ai/holidays-western-division",
"Title": "Western Division Holidays"
}
]},
"Title": "Regions",
"Type": "PDP"
},
{
"Id": "PDP-2",
"Links": {"Link": [
{
"Text": "Bijilo",
"Url": "/1x7-enbmy6/last-minute-holidays-bijilo",
"Title": "Last minute holidays Bijilo"
},
{
"Text": "Browse Cities ",
"Url": "/1x7-en6uly-10tt/last-minute-holidays-gambia/cities",
"Title": "Last minute holidays Gambia",
"Style": "BrowseForMore"
},
{
"Text": "Banjul Beach",
"Url": "/1x6-enakgm/holidays-banjul-beach",
"Title": "Banjul Beach Holidays"
},
{
"Text": "Bijilo",
"Url": "/1x6-enbmy6/holidays-bijilo",
"Title": "Bijilo Holidays"
},
{
"Text": "Brufut Heights",
"Url": "/1x6-encok8/holidays-brufut-heights",
"Title": "Brufut Heights Holidays"
},
{
"Text": "Kololi",
"Url": "/1x6-enpnle/holidays-kololi",
"Title": "Kololi Holidays"
},
{
"Text": "Kotu",
"Url": "/1x6-enq067/holidays-kotu",
"Title": "Kotu Holidays"
}
]},
"Title": "Cities",
"Type": "PDP"
}
],
"Id": "118431",
"Template": {
"PageTemplate": {
"Code": "2B2",
"text": "041 - TEMP2 - COP_CONCOU_{LAST MINUTE}"
},
"Category": {
"Code": "1X7",
"Type": "Product",
"text": "Last minute holidays"
},
"GeoObject": {
"Code": "EN6ULY",
"text": "Gambia, The"
},
"GeoObjectType": {
"Code": "1A",
"text": "Political"
},
"GeoObjectSubType": {
"Code": "10TR",
"text": "Country"
}
},
"Type": "Content",
"Url": "/1x7-en6uly/last-minute-holidays-gambia",
"_id": {"$oid": "528492d4c90fa9fcd0436929"}
}
I want to sort this by Style in Links.Link 'BrowseForMore'. Any idea how to do it? I thought I could add dummy array with push which could then sort it the way I want. Any help appreciated
You appear to want to update the array and keep the sort order with your Links.Link.Style value at the front of the list. In which case use the $sort modifier with update.
db.collection.update(
{ _id: id },
{ $push: { "Links.Link: {$each: [doc], $sort { Style: -1 }}} }
)
The $each operator is required even if there is only one document, but can take many.
if you are trying to use $addToSet to maintain unique documents the official MongoDB line is that sets are considered to be unordered and hence the modifiers are not available here.