I want to import the latest csv file into a table using SSIS? I currently have a step that gets the last file in a folder:
Report_201209030655.csv
Report_201209030655.csv
Report_201209030655.csv
Based on created time I want steps to import data of the latest csv to a table.
refer this solution:
[http://blog.sqlauthority.com/2011/05/12/sql-server-import-csv-file-into-database-table-using-ssis/][1]
then use script task to populate the file name and pass that variable as file name for source component.
Getting latest file code:
public void Main()
{
string[] files = System.IO.Directory.GetFiles(#"C:\SSIS\Files");
DataTable NewList=new DataTable();
DataColumn col = new DataColumn("FileName");
NewList.Columns.Add(col);
System.IO.FileInfo finf;
foreach (string f in files)
{
finf = new System.IO.FileInfo(f);
if (finf.LastWriteTime > DateTime.Now.AddHours(-24))
{
NewList.Rows.Add(f);
}
}
Dts.Variables["User::FileNameArray"].Value = NewList;
Dts.TaskResult = (int)ScriptResults.Success;
}
Related
I have an application that trying to extract all data in different table with 1 Database. First, I stored all the query in a .txt file to retrieve the table name and stored it in List.
[Here's my .txt file]
string script = File.ReadAllText(#"D:\Schooldb\School.txt");
List<string> strings = new List<string>();
strings.Add(script);
using (SqlConnection connection = new SqlConnection(constring))
{
foreach (string x in strings)
{
using (SqlCommand cmd = new SqlCommand(x, connection))
{
using (SqlDataAdapter adapter = new SqlDataAdapter())
{
cmd.Connection = connection;
adapter.SelectCommand = cmd;
using (DataTable dt = new DataTable())
{
adapter.Fill(dt);
string txt = string.Empty;
foreach (DataColumn column in dt.Columns)
{
//Add the Header row for Text file.
txt += column.ColumnName + "\t\t";
}
//Add new line after Column Name.
txt += "\r\n";
foreach (DataRow row in dt.Rows)
{
foreach (DataColumn column in dt.Columns)
{
//Add the Data rows.
txt += row[column.ColumnName].ToString() + "***";
}
//Add new line.
txt += "\r\n";
}
int y = 0;
StreamWriter file = new StreamWriter($#"D:\SchoolOutput\{x}_{DateTime.Now.ToString("yyyyMMdd")}.txt");
file.WriteLine(txt.ToString());
file.Close();
y++;
}
}
}
}
Expected:
teachers_datetoday
students_datetoday
subjects_datetoday
But reality my output is just
datetoday txt
Can someone tell me, where part did I go wrong?
Thanks in advance!
There are other approaches for extracting data directly using SSMS.
In this case, your code reads the entire text as a single string, and the for loop runs only once.
Instead of reading the entire file as a string, you can have each line as one command and read the commands like the following.
foreach (string line in System.IO.File.ReadLines(#"D:\Schooldb\School.txt"))
{
//Each line contains one command
//Write your logic here
}
Hi I have an S3 bucket containing gzip files. Within each zip there is a single TSV file I want to move to a new folder or bucket (dont really mind which). The S3 bucket will be added to with new zip file each hour so this script needs to be something I can schedule or trigger. Happy to use CLI, Lambda or any other method! Pointers, links, help very much appreciated.
Ok so the fudge way to do this is with local processing:
Connect to S3
AmazonS3Config config = new AmazonS3Config();
config.ServiceURL = "https://s3-eu-west-1.amazonaws.com";
AmazonS3Client s3Client = new AmazonS3Client(
S3AccessKey,
S3SecretKey,
config
);
Copy Down the Files you want to process
S3DirectoryInfo dir = new S3DirectoryInfo(s3Client, bucketname, "jpbodenukproduction");
dir.CopyToLocal(#"C:\S3Local");
Decompress the Gzip (containing the tar, containing the multiple files):
string directorypath = #"C:\S3Local";
DirectoryInfo directoryselected = new DirectoryInfo(directorypath);
foreach (FileInfo FileToDecompress in directoryselected.GetFiles("*.gz"))
{
Decompress(FileToDecompress);
}
public static void Decompress(FileInfo fileToDecompress)
{
using (FileStream originalFileStream = fileToDecompress.OpenRead())
{
string currentFileName = fileToDecompress.FullName;
string newFileName = currentFileName.Remove(currentFileName.Length - fileToDecompress.Extension.Length);
using (FileStream decompressedFileStream = File.Create(newFileName))
{
using (GZipStream decompressionStream = new GZipStream(originalFileStream, CompressionMode.Decompress))
{
decompressionStream.CopyTo(decompressedFileStream);
Console.WriteLine("Decompressed: {0}", fileToDecompress.Name);
}
}
}
}
Now deal with the tar file (using ICSharpCode.SharpZipLib):
foreach (FileInfo TarFile in directoryselected.GetFiles("*.tar"))
{
var stream = File.OpenRead(TarFile.FullName);
var tarArchive = ICSharpCode.SharpZipLib.Tar.TarArchive.CreateInputTarArchive(stream);
tb1.Text = "Processing:" + TarFile.Name;
try
{
tarArchive.ExtractContents(#"C:\S3Local\Trash\");
}
catch (Exception ziperror)
{
tb1.Text = "Delay Error in TarUnzip:" + ziperror;
Thread.Sleep(10000);
}
finally
{
tarArchive.Close();
stream.Close();
}
Finally do what you want with the unzipped files, I simply extracted the single file I need, recompressed and moved back up to S3.
My plan is to next convert into Lambda and get this running on a schedule.
I have a file with 2 sheets in it. It has the header information in it. I tried foreach loop container to load the data
Error i got is:
[SSIS.Pipeline] Error: "Excel Source" failed validation and returned validation status "VS_NEEDSNEWMETADATA".
I tried removing the header row manually from the sheets and ran foreach loop container and it worked perfectly fine.
But in my requirement i will be getting the header row followed by blank row in each sheet.
How do i do in this case.
I believe we need to use script task to to eliminate header and followed null row from the file and read the rest of the records.
My problem is i am bad at c# code logic.
Your help is much appreciated.
Thank you,
swathi
The following Script Task will delete the top 2 rows from every worksheet in the file (you'll need to create the variable 'ExcelFilePath' in SSIS and pass that in to the task, along with 'System::TaskName'):
public void Main()
{
MainTask();
GC.Collect();
GC.WaitForPendingFinalizers();
}
private void MainTask()
{
xl.Application xlApp = null;
xl.Workbook excelFile = null;
string excelFilePath = Dts.Variables["User::ExcelFilePath"].Value.ToString();
string thisTask = Dts.Variables["System::TaskName"].Value.ToString();
try
{
xlApp = new xl.Application();
excelFile = xlApp.Workbooks.Open(excelFilePath);
xlApp.DisplayAlerts = false;
foreach (xl.Worksheet ws in excelFile.Worksheets)
{
ws.Rows["1:2"].EntireRow.Delete();
}
xlApp.DisplayAlerts = true;
excelFile.Save();
excelFile.Close();
xlApp.Quit();
Dts.TaskResult = (int)ScriptResults.Success;
}
catch (Exception ex)
{
Dts.Events.FireError(0, thisTask, ex.Message, String.Empty, 0);
if (excelFile != null) excelFile.Close(SaveChanges:false);
if (xlApp != null) xlApp.Quit();
}
}
You will need to add references to 'COM' > 'Microsoft Excel [version number] Object Library' (whichever version you have) and '.NET' > 'Microsoft.CSharp'. You'll then need to declare using xl = Microsoft.Office.Interop.Excel; in your 'Namespaces' region.
I got a package that runs through a folder and it's sub folders to get client data. The agreement has changed and now the client will post his data in different folder name every time. I was wondering if I can do a foreach loop on the main folder and exclude specific folders like archive .
I don't have knowledge in writing scripts so I was wondering if SSIS can do that without the script.
Using Execute Script Task
Get List of - filtered - files using an Execute Script Task before entering Loop and loop over then using ForEach Loop container (Ado enumerator)
You have to a a SSIS variable (ex: User::FilesList) with type System.Object (Scope: Package)
Add an Execute Script Task before the for each Loop container and add User::FilesList as a ReadWrite Variable
In the Script Write The following Code:
Imports System.Linq
Imports System.IO
Imports System.Collections.Generic
Public Sub Main()
Dim Directory as String = "C\Temp"
Dim strSubDirectory as String = Directory & "\New Folder"
Dim lstFiles As New List(Of String)
lstFiles.AddRange(Directory.GetFiles(Directory, "*.*", SearchOption.TopDirectoryOnly).Where(Function(x) Not x.Contains(strSubDirectory)).ToList)
Dts.Variables.Item("FilesList").Value = lstFiles
Dts.TaskResult = ScriptResults.Success
End Sub
In the For each Loop Container Choose the Enumertaion Type as From variable Enumerator and choose FilesList variable as a source
ScreenShots
Using Expression Task
For more details you can refer to my answer in the following link (it is a similar case) WildCards in SSIS Collection {not include} name xlsx
you may have a more control, if you use Script task
Here is the sample code which I have used in one of SSIS:
// Fetch Exclude Directory list from Table
List<excludeDir> excludeDir = new List<excludeDir>();
SqlConnection conn = new SqlConnection(#"Data Source=.\SQLEXPRESS;AttachDbFilename=C:\testDB.mdf;Integrated Security=True;User Instance=True");
SqlCommand cmd = new SqlCommand("select DirList from excludeDir", conn);
SqlDataReader dr;
try
{
conn.Open();
dr = cmd.ExecuteReader();
while (dr.Read())
{
excludeDir.Add(new excludeDir()
{
Dir = dr.GetInt32(dr.GetOrdinal("DirList")),
});
}
dr.Close();
}
catch (Exception exp)
{
throw;
}
finally
{
conn.Close();
}
// compare against Sub directory list and process
string[] dirs = Directory.GetDirectories(#"C:\My Sample Path\");
string[] fileExclude = excludeDir ;
foreach (string path in dirs)
{
FileInfo f = new FileInfo(item2);
listBox1.Items.Add(f.Name);
for (int i = 0; i < fileExclude.Length; i++)
{
-- Console.WriteLine(fileArray[i]);
IF dirs [i] == fileExclude [i]
{
//Set Flags accordingly and execute
}
}
}
You can't do this in the foreach loop properties, but what you can do is start the tasks inside the loop with a script task that checks to see if the folder name is a value that you want to exclude, and if it is, do nothing but loop to the next folder.
I would achieve this (without a Script Task) by setting the Disable property on the Tasks within the For Each Loop Container using an Expression, e.g.
FINDSTRING ( #[User::Each_File_Path] , "archive" , 1 ) > 0
I am using SMS caster to send sms.It has an option to Import csv files.
Now I want to dynamically create csv file of CellNo column of Person table from Visual Studio 2010 connected SQL Server 2008.So that I click on a button and it creates a csv file which I can then access from my software SMSCaster to send sms.
The solutions available are either manual-based or if some query is provided it requires Microsoft OLEDB.....so is there any simple query to convert queryresult into .csv file?
Try this :
Namespace : System.IO;
var _lines = new List<string>();
for(int _i=0;i<gridview1.rows.count;_i++)
{
string[] _mobileNos = gridView1.rows[_i].cells[mobilecolumn index in gridview].text;
var header = string.Join(",", _mobileNos);
_lines.Add(header);
}
File.WriteAllLines("FileName.csv",_lines);
Here is the solution that worked:
public void gridtoCSVFILE()
{
string ing;
List<string> lines = new List<string>();
for (int i = 0; i < gvStudCellNo.Rows.Count; i++)
{
ing = gvStudCellNo.Rows[i].Cells[0].Value.ToString();
lines.Add(ing);
File.WriteAllLines("StudentsCellNo.csv", lines);
}
}
//it will create csv file in your bin folder...also it automatically replaces each new file with the old one