Join SQL query Results and Get-ChildItem Results - sql-server

Background: I have a directory with a number of files that are imported to SQL server.
Task: Creating a PowerShell script which will pick up files within this directory and use the filenames as in the SQL query.
Ultimate objective: To display SQL results besides the filenames but the resultset being displayed should also show files having no entries in SQL server. Something like RIGHT JOIN in SQL server queries.
Powershell Code
$files = Get-ChildItem -Recurse -Force $filePath -ErrorAction SilentlyContinue | Where-Object { ($_.PSIsContainer -eq $false) } | Select-Object Name
$Server = "Loadv1"
$DB = "LoadDB"
$dbResults = #()
ForEach ($file in $files)
{
$fileName = $file.name
write-host $fileName
if($fileName.Length -gt 1)
{
$Query = "
SELECT FileName,CurrentStatus
FROM LogStatus
WHERE FileName LIKE '$fileName%'
"
# Write-host $Query
}
$dbResults += Invoke-Sqlcmd -ServerInstance $Server -Database $DB -Query $Query
}
$dispResults = $dbResults,$file
$dispResults | Format-Table -autosize
Work done so far: I have been able to fetch the file names using Get-ChildItem and loop them to get the query results. However, the result I am currently getting does not show the files that don't have corresponding entry in SQL server table
Current Result
OperationalLanding20150622061502.dat
OperationalLandingAudit20150622061502.dat
OperativeThird_Party_System20150616090701.dat
FileName CurrentStatus
OperationalLandingAudit20150622061502.dat SSIS Package Complete
OperativeThird_Party_System20150616090701.dat SSIS Package Complete
Expected Result
OperationalLanding20150622061502.dat
OperationalLandingAudit20150622061502.dat
OperativeThird_Party_System20150616090701.dat
FileName CurrentStatus
OperationalLanding20150622061502.dat NULL
OperationalLandingAudit20150622061502.dat SSIS Package Complete
OperativeThird_Party_System20150616090701.dat SSIS Package Complete
Hoping I was able to explain my requirement above.

OK so if the SQL query does not have results then NULL is returned and, in essence, nothing is added to the $dbResults array. Instead lets append the results to a custom object. I don't know what PowerShell version you have so I needed to do something that I know should work. I also don't use the SQL cmdlets much so I had to guess for some of this.
$files = Get-ChildItem -Recurse -Force $filePath -ErrorAction SilentlyContinue |
Where-Object {$_.PSIsContainer -eq $false -and $_.Length -gt 1} |
Select-Object -ExpandProperty Name
$Server = "Loadv1"
$DB = "LoadDB"
$files | ForEach-Object{
write-host $_
$Query = "
SELECT FileName,CurrentStatus
FROM LogStatus
WHERE FileName LIKE '$_%'
"
$Results = Invoke-Sqlcmd -ServerInstance $Server -Database $DB -Query $Query
$props = #{Name = $_}
If($Results){
$props.CurrentStatus = $Results.CurrentStatus
} Else {
$props.CurrentStatus = "Null"
}
New-Object -TypeName PSCustomObject -Property $props
} | Format-Table -autosize
What this does is create a custom object that contains the results of the sql query (Which I did not change for reasons stated above). If there are no results returned we use the string "null" as a filler.
I cleaned up how you generated the $files variable by making is a simple string array with -Expand and moved the length condition there as well.
You should now have all the expected results. I say should since I am assuming what the return object looks like.

$Query = "
SELECT isNull(A.FileName, b.FileName) FileName,ISNULL(A.CurrentStatus,B.CurrentStatus) CurrentStatus
FROM LogStatus A
Right JOIN (SELECT '$filename' FileName,NULL CurrentStatus) B
ON a.Filename like '$filename%'
"
This should pad out the filenames for you. A little tough to prototype since it's in powershell but I might be able to come up with a sql fiddle to prove it.
EDIT
Answer edited, with sql fiddle:
http://sqlfiddle.com/#!3/12b43/9
Obviously, since you're in a cursor, we can only prove one query at a time.

Related

How to compare data from SQL to the one in a directory using PowerShell 5.1?

The two columns in the database MAFN and Version which when concatenated with a dot "." have the (almost the)same format as the folder names in the directory.
The data in col. MAFN is 123and Version is 1
The folders are named like:
001234.001,
000789.011
etc.
I was able to remove the leading zeros from the directory folder names.
I was also able to concatenate the data from the two columns.
I want to compare concatenated items from the query to the (new)folder names in the directory and if they match I want to do something.
The question is how do I compare the two, the concatenated data from SQL and the folder names.
I have tried storing the data from SQL in a variable but it doesn't work. Let's say the data is stored is $sqlData then Write-Host $sqlData gives the following output
System.Data.DataRow System.Data.DataRow System.Data.DataRow System.Data.DataRow System.Data.DataRow System.Data.DataRow System.Data.DataRow System.Data.DataRow System.Data.DataRow Syste m.Data.DataRow
The code is:
#SQL connection and concat columns
Invoke-Sqlcmd -ServerInstance "Sidney9.sidney.nii" -Database "NML_Sidney" -Query "SELECT TOP (10) MAFN,Version,(Convert(varchar(50),MAFN)+'.'+Convert(varchar(50),Version))
FROM [NML_Sidney].[dbo].[vNML_MAFN_CNCP_ByPartResource]"
#Split, trim, and join folder names to match concated data from SQL
$aidLibPath = "C:\Users\userName\Desktop\CNC_Transfer_Test_Folders\AidLib_Test\*"
Get-Item -Path $aidLibPath | ForEach-Object{
$splitFileName = $_.Name.Split('.')
$trimSplitFileName = $splitFileName.trimstart("0")-join(".")
write-host $trimSplitFileName
}
Basically, how do I store the data in a variable and loop over them to compare them?
I am using MSSQL 2005 and PowerShell 5.1 with SQL Server and SQLPS modules loaded.
Any help is appreciated.
String concatenation in TSQL is not required if you have a nice scripting language like PowerShell available.
How I would do this given the sql returns the values desired:
#SQL connection and concat columns
$dbResult = Invoke-Sqlcmd -ServerInstance "Sidney9.sidney.nii" -Database "NML_Sidney" -Query "SELECT TOP (10) MAFN,Version
FROM [NML_Sidney].[dbo].[vNML_MAFN_CNCP_ByPartResource]"
$dbFolders = $dbResult | ForEach-Object { "$( $_.MAFN ).$( $_.Version )" }
#Split, trim, and join folder names to match concated data from SQL
$aidLibPath = 'C:\Users\userName\Desktop\CNC_Transfer_Test_Folders\AidLib_Test'
$folders = ( Get-ChildItem -Path $aidLibPath -Directory ).Name | Where-Object {
$_.IndexOf( '.' ) -gt 0 } | ForEach-Object {
$parts = $_.Name -split '.'
"$( $parts[0].TrimStart('0') ).$( $parts[1].TrimStart('0') )"
}
# compare the lists to see what the intersect is
$result = Compare-Object -ReferenceObject $dbFolders -DifferenceObject $folders -IncludeEqual -ExcludeDifferent
if ( $result ) {
$result.InputObject
}

How may I export the output of SQL query in Excel fetched through PowerShell?

I am using this power-shell script to fetch the versions of all SQL Servers in a list.
How may I export the result columns (only query output not error messages) into excel and send to email after the script is run?
Can someone help me add the required script please?
Import-Module SQLPS -DisableNameChecking
$ServerInstences = Get-Content "D:\DBA\All_Server_monitoring\ServerList.txt"
$SQLQuery = #"
Select ##Servername 'Server Name' ,##version 'Version'
"#
$DBName = "master"
$ServerInstences |
ForEach-Object {
$ServerObject = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Server -ArgumentList $_
Invoke-Sqlcmd -ServerInstance $_ -Database $DBName -Query $SQLQuery
}
The easiest way to export data to a csv file is by using Export-CSV which takes an input object (or object array) and a path and can fill out the csv file from that object/array. For you, it would look like this:
$results = Invoke-Sqlcmd -ServerInstance $_ -Database $DBName -Query $SQLQuery
New-Item -Path "MYPATH.csv"
Export-CSV -Path "MYPATH.csv" -InputObject $results -Append
CSV files are versatile and can be opened with the most lightweight text editors. They also can be easily emailed and opened with Excel.

Getting an error when importing Excel file to SQL server database

I want to import an Excel file into my SQL server DB. Here is my code below:
$dir = "\\server\files\"
$latest = Get-ChildItem -Path $dir |
Where-Object {$_.name -like "*Data Action source *"} |
Sort-Object LastWriteTime -Descending |
Select-Object -First 1
Write-Output "The latest file is: $latest"
Write-SqlTableData -ServerInstance "instance1" -DatabaseName "db1" -SchemaName dbo -TableName Table1 -InputData $latest -Force
Write-Output "The latest file is $latest. Let's start the import"
The code picks the latest file nicely but when it comes to inserting into a database its failing with the error below:
Write-SqlTableData : A mapping between .Net type 'System.IO.DirectoryInfo' and SQL type for column 'Directory' was not found. Consider removing the column with that type and repeat the operation
From the error: The variable $latest is of the type System.IO.DirectoryInfo, which is not expected by the Write-SqlTableData command's -InputData parameter.
Run this line in the console for the documentation:
Get-Help Write-SqlTableData -Full
There, you will see this entry for -InputData
-InputData <PSObject>
Specifies the data to write to the database.
Typical input data is a System.Data.DataTable, but you can specify System.Data.DataSet or
System.Data.DateRow objects.
Required? true
Position? named
Default value none
Accept pipeline input? true (ByValue)
Accept wildcard characters? false
So the solution is to convert the csv into a format that this command can understand. The first step is to actually import the data into PowerShell as the moment all you have is file information (path, size etc)
Below, I have cast the CSV as psobject data type. This may still give you an error. For converting to System.Data.Datable which will be accepted, check out this link
dir = "\\server\files\"
$latest = Get-ChildItem -Path $dir |
Where-Object {$_.name -like "*Data Action source *"} |
Sort-Object LastWriteTime -Descending |
Select-Object -First 1
Write-Output "The latest file is: $latest"
# Import CSV to get the data
$csvData = Import-csv $latest.FullName
# Convert to psobject. This may still be rejected; I can't test. If it is, convert to DataTable.
$thingToImport = [psobject]$csvData
Write-SqlTableData -ServerInstance "instance1" -DatabaseName "db1" -SchemaName dbo -TableName Table1 -InputData $thingToImport -Force
Write-Output "The latest file is $latest. Let's start the import"

How to load specific packages from a server with Powershell

I have a PowerShell script that I am writing to extract all the jobs of a specific server like
$sqlserver = "Servername"
[System.Reflection.Assembly]::LoadWithPartialName('Microsoft.SqlServer.Smo') | Out-Null
$srv = New-Object ('Microsoft.SqlServer.Management.Smo.Server') $sqlserver
$jobs = $srv.JobServer.Jobs
ForEach ($job in $jobs)
{
$jobname =$Servernaam.replace("\","_") + '_'+ $job.Name.replace(" ","_").replace("\","_").replace("[","_").replace("]","_").replace(".","_").replace(":","_") + ".sql"
$job.Script() | Out-File C:\Users\Desktop\Jobs_from_Server\Orgineel\$jobname
if ($jobs -like '*.dtsx*')
}
The code I got now gets all the jobs from the server and store them in separate files.
The problem is that I only want to get the jobs of the Micrososft SQL Server that contains .dtsx in the string of #command
I tried for example
ForEach ($job in $jobs)
{
$jobname =$Servernaam.replace("\","_") + '_'+ $job.Name.replace(" ","_").replace("\","_").replace("[","_").replace("]","_").replace(".","_").replace(":","_") + ".sql"
$job.Script() | Out-File C:\Users\Desktop\Jobs_from_Server\Orgineel\$jobname
if ($jobs -like '*.dtsx*')
I also have tried - Contain and set the code in the foreachloop like
ForEach ($job in $jobs |if ($jobs -like '*.dtsx*'))
Your code has a number of typos and other errors in which objects you're using where. Try this as a starter. It assumes that the first step in a job that uses SSIS is an SSIS step. Modify as needed.
The key here is checking the subsystem of the job step(s) to detect if the step is run with the SSIS subsystem.
$sqlserver = "servername"
[System.Reflection.Assembly]::LoadWithPartialName('Microsoft.SqlServer.Smo') | Out-Null
$srv = New-Object ('Microsoft.SqlServer.Management.Smo.Server') $sqlserver
$jobs = $srv.JobServer.Jobs
ForEach ($job in $jobs)
{
if ($Job.JobSteps[0].Subsystem -eq "ssis") {
# Do SSIS stuff
write-output "Found an SSIS job $($job.name)"
}
}

How to scan SQL Server error log (and ignore some errors) with PowerShell?

I need a PowerShell script (2.0 compatible) to scan SQL Server 2008 R2 and later error logs. I need to have a list of phrases to search for, and a list of phrases to exclude.
param ([String]$instanceName=$(throw "Instance name was not supplied"))
[System.Reflection.Assembly]::LoadWithPartialName('Microsoft.SqlServer.SMO')|Out-Null;
$sqlServer = new-object ("Microsoft.SqlServer.Management.Smo.Server") $instanceName;
$r = $sqlServer.ReadErrorLog();
$find = "Error:","Failed";
$exclude = "Error: 0x2098";
# need to do something with $r here and involve $find and $exclude
So, for example I want to find all lines in the ERRORLOG that contain Error: and Failed, but exclude the ones in the $exclude array. Any ideas?
You can find a lot of info on this here:
Use PowerShell to Parse SQL Server 2012 Error Logs
But one way to do what you are asking is by filtering the results when setting $r
$r = $sqlServer.ReadErrorLog() | ? { $_.Text -match 'error' -OR $_.text -match 'Failed' -and $_text -notmatch "Error: 0x2098"}
You can iterate through each list and update $r accordingly, something like below should get you started.
Foreach($ExcludeText in $exclude){
$r = $r | ? {$_.text -notmatch $ExcludeText}
}
Foreach($IncludeText in $find){
$r = $r | ?{$_.text -match $IncludeText}
}
The above article has details on the undocumented TSQL version.
http://www.mssqltips.com/sqlservertip/1476/reading-the-sql-server-log-files-using-tsql/
I think you are missing some parameters. The xp_readerrorlog extended procedure takes the following parameters.
A - Number of error log file
B - 1 = SQL Server log, 2 - SQL Agent log
C - Search string 1
D - Search string 2
Here is a powershell solution like you asked.
I think the order of the and/or is important??
$srv = new-Object Microsoft.SqlServer.Management.Smo.Server("(local)")
$d = $srv.ReadErrorLog(0)
foreach ($r in $d.Rows)
{
if ( ($r['Text'] -match 'Error:' -and $r['Text'] -notmatch 'Error: 0x2098')
-or $r['Text'] -match 'Failed:' )
{
Write-Host "============================================"
Foreach ($c in $d.Columns)
{ Write-Host $c.ColumnName "=" $r[$c]}
}
}

Resources