PowerShell - Import CSV with pivotted data - sql-server

Question

I managed to create a PowerShell script to download and load into a SQL Server database some files that have pivotted data: each day of the series is a column with the values of that day.
The problem is that it takes a long time to execute, about 15 seconds to load each file that has an average of 250 lines. I think the problem is when I try to get the values of the fields by its "index", because I didn't find a clever way of doing that.
Is there a better way of inserting this kind of CSV data that can change each day into the database using the PowerShell?
I don't know much of PowerShell scripting, but I managed to create the below scripts:
The files I downloaded from here:
https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_time_series
Script to process the files:
Clear-Host
$date = (Get-Date).Date.AddDays(-1).ToString('MM-dd-yyyy')
$path = 'C:\Dataset\'
$items = (Get-ChildItem -Path ($path + "time_series*.csv") | Select FullName) #| Select -First 1
$sql_instance_name = '.'
$db_name = 'COVID-19'
foreach ($item in $items)
{
$dt_start = Get-Date
$schema = "stg"
$table = (Split-Path -Path $item.FullName -Leaf).Split('.')[0]
Write-Host "File:"(Split-Path -Path $item.FullName -Leaf) -ForegroundColor Yellow
Write-Host "Schema: $schema" -ForegroundColor Yellow
Write-Host "Table: [$table]" -ForegroundColor Yellow
$header = (Get-Content $item.FullName | Select -First 1).replace(",", "|,|")
$i = 0; $new_header = #();
foreach ($column in $header.Replace('|', '').split(','))
{
$new_header += "Column_$i"
$i++
}
$drop_table = "if (object_id('stg.[$table]')) is not null drop table $schema.[$table];"
Invoke-Sqlcmd -Database $db_name -Query $drop_table -ServerInstance $sql_instance_name
$create_table = ("if (object_id('stg.[$table]')) is null
create table $schema.[$table] (" +
" id int identity constraint [pk_$table] primary key," +
" [" + $header + "] varchar(500),`n`tload_date datetime`n);").Replace('|,|', "] varchar(500), [")
Invoke-Sqlcmd -Database $db_name -Query $create_table -ServerInstance $sql_instance_name
$csv = Import-Csv -Path $item.FullName -Header $new_header | Select -Skip 1
$insert = $null
foreach ($row in $csv)
{
$query = "insert into stg.[" + (Split-Path -Path $item.FullName -Leaf).Split('.')[0] + "] values ("
foreach ($column in $new_header)
{
<# Perhaps this part slows down the process by the means of
getting the value for the column (I couldn't find a way to
simply reference the column by index like $csv.column[$i]
till the last one)
#>
$value = ($row | Select $column)
$query += "nullif('" + ($value | % { $_.$(( $value | gm | ? { $_.membertype -eq "noteproperty"} )[0].name) }).Replace("'", "''") + "',''),"
}
$query += " current_timestamp);"
$insert = $query
#Write-Host $row.Column_1
Invoke-Sqlcmd -Database $db_name -Query $insert -ServerInstance $sql_instance_name
}
Write-Host "Lines:"$csv.count -ForegroundColor Yellow
#Start-Sleep -Seconds 2
$dt_end = Get-Date
Write-Host "Elapsed time:"(New-TimeSpan -Start $dt_start -End $dt_end).TotalSeconds -ForegroundColor Red
Write-Host ("-" * 50)
}
Execution results:
File: time_series_covid19_confirmed_global.csv
Schema: stg
Table: [time_series_covid19_confirmed_global]
Lines: 264
Elapsed time: 14,3725288
--------------------------------------------------
File: time_series_covid19_deaths_global.csv
Schema: stg
Table: [time_series_covid19_deaths_global]
Lines: 264
Elapsed time: 14,1963788
--------------------------------------------------
File: time_series_covid19_recovered_global.csv
Schema: stg
Table: [time_series_covid19_recovered_global]
Lines: 250
Elapsed time: 13,5150064
--------------------------------------------------

Answer 1

If 2016+, you can download the JSON data.
If <2016, I believe there is an XML option as well
The download takes a couple of seconds, and the parsing of the JSON takes less than 4 seconds (depending on your hardware).
Example
exec master..xp_cmdshell 'powershell.exe Invoke-WebRequest "https://opendata.ecdc.europa.eu/covid19/casedistribution/json/" -OutFile "c:\working\covid.json"',no_output
Declare #json varchar(max);
Select #json = BulkColumn FROM OPENROWSET(BULK 'c:\working\covid.json', SINGLE_BLOB) x;
;with cte as (
Select [DataDate] = try_convert(date,DateRep,105)
,[CountryCd]= try_convert(varchar(50),countryterritoryCode)
,[Country] = try_convert(varchar(150),countriesAndTerritories)
,[Cases] = try_convert(int ,cases)
,[Deaths] = try_convert(int ,deaths)
,[Pop] = try_convert(int ,[popData2018])
,rtc = sum(try_convert(int ,cases)) over (partition by countryterritoryCode order by try_convert(date,DateRep,105))
,rtd = sum(try_convert(int ,deaths)) over (partition by countryterritoryCode order by try_convert(date,DateRep,105))
From (
Select Idx= B.[Key]
,C.*
From OpenJSON(#json ) A
Cross Apply OpenJson(A.value) B
Cross Apply OpenJson(B.value) C
) src
Pivot (max(value) for [Key] in ( [dateRep],[cases],[deaths],[countriesAndTerritories],[geoId],[countryterritoryCode],[popData2018] ) ) pvt
)
Select DataDate
,CountryCd
,Country
,Cases = format(Cases,'#,###')
,Deaths= format(Deaths,'#,###')
,Pop = format(Pop/1000000.0,'#,##0.0')+'MM'
,RTC = format(RTC,'#,###')
,RTD = format(RTD,'#,###')
,Mort = format((rtd*100.0) / nullif(rtc,0),'##0.00')+'%'
,PctPop= format((cases*100.0) / nullif(Pop,0),'##0.0000')+'%'
From cte
Where DataDate='2020-04-11'
Order By try_convert(money,RTC) desc
Returns