Copy Rest API Nested JSON response data to SQL - arrays

Question

I'm trying to fetch data from API whose response is nested json, I used collection reference to capture all the results but the results have arrays in which by default only first value is captured. I checked the documentation also (https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-schema-and-type-mapping ) and there it has mentioned only single array is supported for such operation.
Please let me know how i can capture all the nested array json values.
I tried using the Map complex values to string option also but it's not working.
Please check the image link for the json result and ADF mapping done ADF Mapping JSON Response
{
"story_count": 2325,
"next": "?page=2",
"previous": null,
"results": [
{
"id": 20111247056317,
"title": "ANZ exec bonuses cut, salaries hiked",
"summary": "[Source: AFR Online] ANZ chief executive Shayne Elliott and his executive team have had short-term bonuses cut by 50 per cent or more after profits slumped 42 per cent. However, they have almost all had sizeable increases in fixed pay that should stop them from harbouring any ill feelings towards their Melbourne-based employer. ",
"source_url": "https://global.factiva.com/en/du/article.asp?accountid=9ERN000600&accessionno=AFNROL0020201109egb9000p1",
"source_name": "Factiva",
"pub_date": "2020-11-09T06:52:18Z",
"image_url": "",
"attachments": [],
"Triggers": [
{
"id": 15174,
"name": "Cost Reduction",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 15178,
"name": "Tax Risk",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 24194,
"name": "Rewards & benefits",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Firms": [
{
"id": 17984,
"name": "AUSTRALIA AND NEW ZEALAND BANKING GROUP LIMITED",
"logo": "//112233.contify.com/client_data/custom_tag/logo/eg1lrjgo_400x400-17984.jpg"
}
],
"Duns Number": [
{
"id": 18498,
"name": "753682830",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Channels": [
{
"id": 17,
"name": "News and Other Websites",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Content Types": [
{
"id": 3,
"name": "News Articles",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Sources": [
{
"id": 68636,
"name": "Factiva",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"previews": [],
"duplicates": [],
"duplicate_count": 0
},
{
"id": 20102741904140,
"title": "HSBC to cut up to 300 jobs in UK commercial banking unit",
"summary": "[Source: FinTech Futures News] HSBC has launched a restructuring of its commercial banking business in Britain. A source familiar with the matter tells Reuters that the plan will result in around 300 job losses. The cuts are part of a wider restructuring announced in February. ",
"source_url": "https://global.factiva.com/en/du/article.asp?accountid=9ERN000600&accessionno=FTEFN00020201027egar0002w",
"source_name": "Factiva",
"pub_date": "2020-10-27T00:00:00Z",
"image_url": "",
"attachments": [],
"Triggers": [
{
"id": 15174,
"name": "Cost Reduction",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 24196,
"name": "Restructuring",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Firms": [
{
"id": 17967,
"name": "HSBC Holdings PLC",
"logo": "//112233.contify.com/client_data/custom_tag/logo/nbzwhpqp_400x400-17967.png"
},
{
"id": 35194,
"name": "HSBC Holdings",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Duns Number": [
{
"id": 18655,
"name": "288451024",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Channels": [
{
"id": 17,
"name": "News and Other Websites",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Content Types": [
{
"id": 3,
"name": "News Articles",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Sources": [
{
"id": 68636,
"name": "Factiva",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"previews": [],
"duplicates": [],
"duplicate_count": 0
},
{
"id": 20102641537340,
"title": "AT&T cuts more than 8,700 workers in third quarter amid COVID-19",
"summary": "[Source: Dallas Business Journal Online] AT&T is making some deeper cuts with its employee base. The Dallas telecommunications and media company shrunk the number of employees by 8,720 in the third quarter to less than 235,000, according to information on its investor relations website. ",
"source_url": "https://global.factiva.com/en/du/article.asp?accountid=9ERN000600&accessionno=DALBJO0020201026egap00001",
"source_name": "Factiva",
"pub_date": "2020-10-25T00:00:00Z",
"image_url": "",
"attachments": [],
"Triggers": [
{
"id": 15174,
"name": "Cost Reduction",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 23058,
"name": "Covid-19",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Firms": [
{
"id": 17963,
"name": "AT&T Inc.",
"logo": "//112233.contify.com/client_data/custom_tag/logo/a7jsrall_400x400-17963.png"
}
],
"Duns Number": [
{
"id": 18497,
"name": "108024050",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Channels": [
{
"id": 17,
"name": "News and Other Websites",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Content Types": [
{
"id": 3,
"name": "News Articles",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Sources": [
{
"id": 68636,
"name": "Factiva",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"previews": [],
"duplicates": [
{
"id": 20102641627014,
"title": "AT&T cuts more than 8,700 workers in third quarter amid Covid-19",
"summary": "[Source: Louisville Business First Online] AT&T is making some deeper cuts with its employee base. The Dallas telecommunications and media company shrunk the number of employees by 8,720 in the third quarter to less than 235,000, according to information on its investor relations website. ",
"source_url": "https://global.factiva.com/en/du/article.asp?accountid=9ERN000600&accessionno=BSFLVO0020201026egaq00001",
"source_name": "Factiva"
},
{
"id": 20102741677548,
"title": "AT&T cuts more than 8,700 workers in third quarter amid Covid-19",
"summary": "[Source: Orlando Business Journal] AT&T is making some deeper cuts with its employee base. The Dallas-based telecommunications and media company (NYSE: T) shrunk the number of its employees by 8,720 in the third quarter to less than 235,000, according to information on its investor relations website. ",
"source_url": "https://global.factiva.com/en/du/article.asp?accountid=9ERN000600&accessionno=ORBJ000020201026egaq0005m",
"source_name": "Factiva"
}
],
"duplicate_count": 2
},
{
"id": 20102641660233,
"title": "Lufthansa considers cutting 30,000 jobs due to fall in air traffic",
"summary": "[Source: French Collection] German airline Lufthansa is facing a fall in air traffic due to the Covid-19 pandemic and consequently up to 30,000 jobs are threatened, its management said. Lufthansa is determined to keep at least 100,000 of the 130,000 positions in total at the group, Carsten Spohr, CEO of the airline, explained. ",
"source_url": "https://global.factiva.com/en/du/article.asp?accountid=9ERN000600&accessionno=FRECOL0020201026egaq000jh",
"source_name": "Factiva",
"pub_date": "2020-10-26T00:00:00Z",
"image_url": "",
"attachments": [],
"Triggers": [
{
"id": 15174,
"name": "Cost Reduction",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Firms": [
{
"id": 18140,
"name": "Deutsche Lufthansa AG",
"logo": "//112233.contify.com/client_data/custom_tag/logo/-99kuzpm_400x400-18140.jpg"
},
{
"id": 35588,
"name": "Lufthansa Group",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Duns Number": [
{
"id": 18584,
"name": "315000893",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Channels": [
{
"id": 17,
"name": "News and Other Websites",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Content Types": [
{
"id": 3,
"name": "News Articles",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Sources": [
{
"id": 68636,
"name": "Factiva",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"previews": [],
"duplicates": [],
"duplicate_count": 0
},
{
"id": 20102541378844,
"title": "ExxonMobil chief warns US, Canada jobs cuts coming soon",
"summary": "[Source: Agence France Presse] Global oil giant ExxonMobil will release details soon of expected job cuts in the United States and Canada, the company's chief Darren Woods said in a letter to employees. \"As difficult as this is, I hope you understand it is critically important for the future of our company,\" Woods said in a statement Wednesday. ",
"source_url": "https://global.factiva.com/en/du/article.asp?accountid=9ERN000600&accessionno=AFPR000020201022egam00dqh",
"source_name": "Factiva",
"pub_date": "2020-10-22T13:04:58Z",
"image_url": "",
"attachments": [],
"Triggers": [
{
"id": 15174,
"name": "Cost Reduction",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 15178,
"name": "Tax Risk",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Firms": [
{
"id": 24236,
"name": "Exxon Mobil Corporation",
"logo": "//112233.contify.com/client_data/custom_tag/logo/ytocyynn_400x400-24236.jpg"
}
],
"Duns Number": [
{
"id": 24342,
"name": "1213214",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 34007,
"name": "001213214",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Channels": [
{
"id": 17,
"name": "News and Other Websites",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Content Types": [
{
"id": 3,
"name": "News Articles",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Sources": [
{
"id": 68636,
"name": "Factiva",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"previews": [],
"duplicates": [],
"duplicate_count": 0
},
{
"id": 20102541377022,
"title": "Cathay Pacific to lay off 5,900 employees, close low-cost subsidiary",
"summary": "[Source: Kazinform International News Agency] HONG. KONG. KAZINFORM Hong Kong s flag carrier Cathay Pacific announced Wednesday it would lay off 5,900 employees and close low-cost subsidiary Cathay Dragon following corporate restructuring in response to the coronavirus s effect on aviation.«The restructuring will enable the company to secure its future, so it can protect as many jobs as possible, whilst meeting its responsibilities to the Hong Kong aviation hub and its customers,» the carrier said in a statement, EFE-EPA reports. ",
"source_url": "https://global.factiva.com/en/du/article.asp?accountid=9ERN000600&accessionno=KAZNAE0020201021egal000b7",
"source_name": "Factiva",
"pub_date": "2020-10-21T12:13:00Z",
"image_url": "",
"attachments": [],
"Triggers": [
{
"id": 15174,
"name": "Cost Reduction",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 15178,
"name": "Tax Risk",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 23058,
"name": "Covid-19",
"logo": "//112233.contify.com/images/tags-ico.png"
},
{
"id": 24196,
"name": "Restructuring",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Firms": [
{
"id": 35127,
"name": "Cathay Pacific Airways Ltd.",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Duns Number": [
{
"id": 33981,
"name": "686093188",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Channels": [
{
"id": 17,
"name": "News and Other Websites",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Content Types": [
{
"id": 3,
"name": "News Articles",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"Sources": [
{
"id": 68636,
"name": "Factiva",
"logo": "//112233.contify.com/images/tags-ico.png"
}
],
"previews": [],
"duplicates": [],
"duplicate_count": 0
}
]
}

Answer 1

I would recommend you consider calling a Stored Procedure activity instead of a copy activity. Then parse and store the JSON inside the stored procedure. I have written up this approach in the following answer. You can ignore the For loop if it’s a single REST API call. I’m working under the assumption that your destination database is SQL Server or Azure SQL Database, but correct me if that’s incorrect.