How to dynamically change server url withing http connection manager SSIS? - sql-server

Question

I am fairly new to SSIS, Visual studio. Thought that might be good to mention in the beginning.
What I wanted to achieve was to download a certain xls file from http://www.ads-slo.org/statistika/ website and store it in a certain folder on my computer. I have achieved that, but the problem is that I know how to do it one file at a time. I did it by opening new connection, going to http connection and in the manager typing the server url: which in my case if lets say we start with January 2016 was this:http://www.ads-slo.org/media/xls/2016/Januar-2016.xls. After doing so I've constructed a script task or more or less copied it from a website that downloads the file given a certain url based on the connection manager.
My problem is that I would like to download all of the files on this site, so starting with January 2007 and ending with January 2016 with a single package and by not changing my connection manager server url settings 100 times.
Is there any way you might help me. I would be forever grateful.
Thank you in advance.
Kind regards, Domen

Answer 1

Here is one very simple example (it can be improved - see comments after code block) of changing a connection string dynamically by using a Script Task. You can also dynamically change connection strings using expressions and the Connection Manager's expressions property. However, since you are using a Script Task to handle the downloads, I have demonstrated it using one.
As you haven't tagged the Script language (VB or C#) you are using, I have written a rough draft in VB.
I have added comments, but stackoverflow syntax highlighting interprets it strangely; apologies.
Public Sub Main()
' Get the HTTP Connection
Dim connObj As Object = Dts.Connections("HTTP Connection Manager").AcquireConnection(Nothing)
Dim connection As New HttpClientConnection(connObj)
' Static list of month names for dynamic connection string (obviously add as many as needed)
Dim monthNames As String() = New String() {"Januar", "February", "March"}
' Nested loop - for each year and month, try to download the Excel file
For Y As Integer = 2007 To 2016 Step 1
For M As Integer = 0 To monthNames.Length - 1 Step 1
' Set the assumed name of the remote file
Dim remoteFileName As String = monthNames(M) + "-" + Y.ToString() + ".xls"
' Change the connection string a.k.a dynamic connection string
connection.ServerURL = "http://www.ads-slo.org/media/xls/" + Y.ToString() + "/" + remoteFileName
' Set where to download the file to
Dim localFileName As String = "C:\Temp\" + remoteFileName
Try
connection.DownloadFile(localFileName, True)
Dim buffer As Byte() = connection.DownloadData()
Dim data As String = Encoding.ASCII.GetString(buffer)
Catch E As Exception
' File may not exist on remote server, delete the blank copy it attempted to create
File.Delete(localFileName)
End Try
Next
Next
Dts.TaskResult = DTSExecResult.Success
End Sub
How can this be improved?
One potential improvement is to parse the remote server for the folders and directory contents (to save having static lists of month names, hardcoded start and end years and building file names) using a HttpWebRequest.
However, there might be an issue with the remote server permissions in allowing such requests to be made so you would have to investigate further and speak with the server administrator.
Testing the above code, it successfully downloaded the Januar-2015 and Januar-2016 Excel files from the website.