I have the following powershell script which reads in a list of servers, and runs SQL command on these servers. This data is then exported to csv and to excel format
I would like to be able to add the targeted server name from my server list as the first column so columns would look like this (server name added to front)
Server Name | Name | CollectionSet ID | Collection Mode | Retention Period | Schedule
This is the current script I have:
Param
(
[string]$fServers = 'W:\Theo\Scripts\mdw_servers.csv'
)
$query = "SELECT a.name AS 'DC Name',
collection_set_id AS 'Collection_set ID',
CASE collection_mode
WHEN 1 THEN 'non-cached'
WHEN 0 THEN 'cached'
END AS 'Collection Type' ,
days_until_expiration AS 'Retention Period' ,
b.name AS 'Schedule Name'
FROM msdb.dbo.syscollector_collection_sets a ,
msdb.dbo.sysschedules b
WHERE a.schedule_uid = b.schedule_uid
AND is_running = 1;"
$csvFilePath = "W:\Theo\Scripts\queryresults.csv"
$excelFilePath = "W:\Theo\Scripts\queryresults.xls"
# Run Query against multiple servers, combine results
$allServers = Get-Content -Path $fServers
foreach ($Server in $allServers) {
write-host "Executing query against server: " $Server
$results += Invoke-Sqlcmd -Query $query -ServerInstance $Server;
}
# Output to CSV
write-host "Saving Query Results in CSV format..."
$results | export-csv $csvFilePath -NoTypeInformation
# Convert CSV file to Excel
write-host "Converting CSV output to Excel..."
$excel = New-Object -ComObject excel.application
$excel.visible = $False
$excel.displayalerts=$False
$workbook = $excel.Workbooks.Open($csvFilePath)
$workSheet = $workbook.worksheets.Item(1)
$resize = $workSheet.UsedRange
$resize.EntireColumn.AutoFit() | Out-Null
$xlExcel8 = 56
$workbook.SaveAs($excelFilePath,$xlExcel8)
$workbook.Close()
$excel.quit()
$excel = $null
write-host "Results are saved in Excel file: " $excelFilePath
Any input is appreciated!
have you tried
SELECT ##SERVERNAME AS 'Server Name'
https://msdn.microsoft.com/en-us/library/ms187944.aspx
Related
Looking for some help with ingesting data retrieved from an API endpoint and inserting it into SQL Server. This endpoint returns json.
I'm trimming the script to make this more readable. The script gets the data I'm requesting, the issues is the insert into the SQL table.
# Pass the `Authorization` header value with the payload we calculate from the id + secret
$Time = Invoke-RestMethod -Uri $TimeCardURL -Method Get -Header #{ Authorization = "Bearer ${AuthorizationValue}" } -Body $Body -Certificate $Cert -ContentType 'application/json'
$Time.teamTimeCards | Select-Object associateOID,timeCards
#SQL authentication and insert $Result data
$serverName = "sql-server"
$databaseName = "database"
$tableName = "table"
$Connection = New-Object System.Data.SQLClient.SQLConnection
$Connection.ConnectionString = "server='$serverName';database='$databaseName';trusted_connection=true;"
$Connection.Open()
$Command = New-Object System.Data.SQLClient.SQLCommand
$Command.Connection = $Connection
foreach($ID in $associateOID){
$insertquery="
INSERT INTO $tableName
([associateOID],[timeCards])
VALUES
('$associateOID','$timeCards')"
$Command.CommandText = $insertquery
$Command.ExecuteNonQuery()
}
$Connection.Close();
It's trying to load the entire object $associateOID into the column, which is resulting in a "String or binary data would be truncated in table 'database.table', column 'associateOID'. Truncated value: '1 2 3 4 5 6 7 8'. The statement has been terminated."
I'm guessing there is an issue with this section of the script
$Time.teamTimeCards | Select-Object associateOID,timeCards
I rewrote your code to use another source of data, but you probably get the idea:
$output = Get-Process | Select-Object Handles, Id, ProcessName
echo($output)
foreach($value in $output){
$id = $value.Id
$name = $value.ProcessName
$insertquery="
INSERT INTO $tableName
([ID],[Name])
VALUES
('$id','$name')"
echo($insertquery)
}
I have a file with multiple expressions like "$REGX('CareMedic.2_0','CustomerInformation','Customer Information')". The file can be a xml file, text file or any other type. If the file contains 9 of those expressions, I'm trying to pull all nine and send the values to a database.
I've tried my code as below:
$input_path = ‘C:\Users\Administrator\Desktop\test2.xml’
$SQLServer = "WIN-17V7QT0IJVK"
$SQLDBName = "Test"
$uid ="WIN-17V7QT0IJVK\Administrator"
$pwd = "letmebackinplease"
$SqlQuery = "SELECT * from product_schema;"
$ConnectionString = "Server = $SQLServer; Database = $SQLDBName; Integrated Security = True;"
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection $ConnectionString
$SqlConnection.open()
if($SqlConnection.state -eq "Open"){
Write-Host "Test connection successful"
}
$regex = '()\(.*?\)'
$output = select-string -Path $input_path -Pattern $regex -AllMatches | % { $.Matches } | % { $.Value } |
ForEach-Object {
($_ -split "\(|\)")[1]
}
foreach ($line in $output){
$line = $line -replace "\(",""
$line = $line -replace "\)",""
$line = $line -replace "\'",""
$col1,$col2,$col3 = $line -split ","
[PSCustomObject]#{
col1 = $col1
col2 = $col2
col3 = $col3
} | select col1,col2,col3
$insert_query = "INSERT INTO [$SQLDBName].[dbo].[product_schema]
([version]
,[field]
,[value])
VALUES
($col1, $col2, $col3);"
$execute_query = New-Object System.Data.SqlClient.SqlCommand
$execute_query.connection = $SQLConnection
$execute_query.commandtext = $insert_query
$execute_query.ExecuteNonQuery()
}
$SqlConnection.close()
If the file has two of the below:
('Medic.2_0','AgeInformation','Age Information')
('Medic.2_0','TransactionID','Transaction ID')
My actual output should be:
'Medic.2_0' stored in Version Column
'AgeInformation' stored in the Field Column
'Age Information' stored in the value column
'Medic.2_0' stored in Version Column
'TransactionID' stored in the Field Column
'Transaction ID' stored in the value column
I have to take each of the values and store it in a column in a temp table setup on MySQL server like below:
**Version** **Field** **Value**
Medic.2_0 AgeInformation Age Information
Medic.2_0 TransactionID Transaction ID
Error Encountered:
Exception calling "ExecuteNonQuery" with "0" argument(s): "Incorrect syntax near '.2'."
At C:\Users\Administrator\Desktop\test.ps1:47 char:10
+ $execute_query.ExecuteNonQuery()
+ ~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : SqlException
Can someone please recommend how shall I change my code to solve this?
In answer to your original question before editing: Assuming your output looks like this and is saved in a variable named $output
('Medic.2_0','AgeInformation','Age Information')
('Medic.2_0','TransactionID','Transaction ID')
Try this:
foreach ($line in $output){
$line = $line -replace "\(",""
$line = $line -replace "\)",""
$line = $line -replace "\'",""
$col1,$col2,$col3 = $line -split ","
[PSCustomObject]#{
col1 = $col1
col2 = $col2
col3 = $col3
} | select col1,col2,col3 | export-csv d:\test.csv -append -NoTypeInformation
}
We are looping through the $output line by line removing the brackets and the single quotes, splitting the remaining text on the comma, then assigning each of the three entries into the relevant variables. Once they are in variables we can then easily create a PSObject and use it to select our requirements for our export-csv
Try to add this code:
$info=#() #for store your values
foreach($item in $output){
$z=$item.split(',') #for split to 3 strings
$info+=[PSCustomObject]#{ #create custom object which have named columns and store our values
Version = $z[0]
Field = $z[1]
Value = $z[2]
}
}
Write-Output $info #variable that store all columns
Then you must run foreach loop to each object in $info .
you can run it like this:
foreach($data in $info){
$data.Version #to access Version field
$data.Field #to access Field field
$data.Value #to access Value field
.......your SQL query......
}
I have created a PowerShell function that bulk copies data from a .csv file (first row is the header), and inserts the data in to a SQL Server database table.
See my code:
function BulkCsvImport($sqlserver, $database, $table, $csvfile, $csvdelimiter, $firstrowcolumnnames) {
Write-Host "Bulk Import Started."
$elapsed = [System.Diagnostics.Stopwatch]::StartNew()
[void][Reflection.Assembly]::LoadWithPartialName("System.Data")
[void][Reflection.Assembly]::LoadWithPartialName("System.Data.SqlClient")
# 50k worked fastest and kept memory usage to a minimum
$batchsize = 50000
# Build the sqlbulkcopy connection, and set the timeout to infinite
$connectionstring = "Data Source=$sqlserver;Integrated Security=true;Initial Catalog=$database;"
# Wipe the bulk insert table first
Invoke-Sqlcmd -Query "TRUNCATE TABLE $table" -ServerInstance $sqlserver -Database $database
$bulkcopy = New-Object Data.SqlClient.SqlBulkCopy($connectionstring, [System.Data.SqlClient.SqlBulkCopyOptions]::TableLock)
$bulkcopy.DestinationTableName = $table
$bulkcopy.bulkcopyTimeout = 0
$bulkcopy.batchsize = $batchsize
# Create the datatable, and autogenerate the columns.
$datatable = New-Object System.Data.DataTable
# Open the text file from disk
$reader = New-Object System.IO.StreamReader($csvfile)
$columns = (Get-Content $csvfile -First 1).Split($csvdelimiter)
if ($firstrowcolumnnames -eq $true) { $null = $reader.readLine() }
foreach ($column in $columns) {
$null = $datatable.Columns.Add()
}
# Read in the data, line by line
while (($line = $reader.ReadLine()) -ne $null) {
$null = $datatable.Rows.Add($line.Split($csvdelimiter))
$i++;
if (($i % $batchsize) -eq 0) {
$bulkcopy.WriteToServer($datatable)
Write-Host "$i rows have been inserted in $($elapsed.Elapsed.ToString())."
$datatable.Clear()
}
}
# Add in all the remaining rows since the last clear
if($datatable.Rows.Count -gt 0) {
$bulkcopy.WriteToServer($datatable)
$datatable.Clear()
}
# Clean Up
$reader.Close();
$reader.Dispose()
$bulkcopy.Close();
$bulkcopy.Dispose()
$datatable.Dispose()
Write-Host "Bulk Import Completed. $i rows have been inserted into the database."
# Write-Host "Total Elapsed Time: $($elapsed.Elapsed.ToString())"
# Sometimes the Garbage Collector takes too long to clear the huge datatable.
$i = 0
[System.GC]::Collect()
}
I am looking to modify the above though so that the column names in the .csv file match up with the column names in the SQL Server database table. They should be identical. At the moment the data is being imported in to the incorrect database columns.
Could I get some assistance as what I need to do to modify the above function to achieve this?
I would use existing open source solution:
Import-DbaCsv - dbatools.io
Import-DbaCsv.ps1
Efficiently imports very large (and small) CSV files into SQL Server.
Import-DbaCsv takes advantage of .NET's super fast SqlBulkCopy class to import CSV files into SQL Server.
Parameters:
-ColumnMap
By default, the bulk copy tries to automap columns. When it doesn't
work as desired, this parameter will help.
PS C:\> $columns = #{
>> Text = 'FirstName'
>> Number = 'PhoneNumber'
>> }
PS C:\> Import-DbaCsv -Path c:\temp\supersmall.csv
-SqlInstance sql2016 -Database tempdb -ColumnMap $columns
-BatchSize 50000 -Table table_name -Truncate
The CSV column 'Text' is inserted into SQL column 'FirstName' and CSV column Number is inserted into the SQL Column 'PhoneNumber'. All other columns are ignored and therefore null or default values.
I'm trying to execute an SQL query with few select statements, that returns multiple tables as a result. The problem is that I can't find a way to read and use the tables separately.
Expected results:
Actual results: (it is printed row by row)
Purpose: I've made a script that creates an empty excel file with multiple sheets and each of the sheets will be used to contain each resultset of the query.
The only thing left is to put the needed text into the sheets. Here is my code for that part only:
$ConnectionString = "Data Source=...;Initial Catalog=...;User Id=...;Password=..."
$DBServerName = $ConnectionString.split('=')[1].split(';')[0]
$DBName = $ConnectionString.split('=')[2].split(';')[0]
$DBUser = $ConnectionString.split('=')[3].split(';')[0]
$DBPassword = $ConnectionString.split('=')[4].split(';')[0]
$CurrentFilePath = "C:\SQLqueryWithManyResultsets.sql"
$query = Get-Content -literalPath $CurrentFilePath | Out-String #getting the query string from file
$resultTables = Invoke-Sqlcmd -Query $query -ServerInstance $DBServerName -Database $DBName -DisableVariables -Password $DBPassword -Username $DBUser -ErrorAction Stop
foreach ($result in $resultTables) {
$result | Format-Table #where the magic happens
}
I've made a lot of research, but I cannot find a proper way to store and read the tables the way i need.
Try this:
Clear-Host;
$objConnection = New-Object System.Data.SqlClient.SqlConnection;
$objConnection.ConnectionString = "...";
$ObjCmd = New-Object System.Data.SqlClient.SqlCommand;
$ObjCmd.CommandText = "...";
$ObjCmd.Connection = $objConnection;
$ObjCmd.CommandTimeout = 0;
$objAdapter = New-Object System.Data.SqlClient.SqlDataAdapter;
$objAdapter.SelectCommand = $ObjCmd;
$objDataSet = New-Object System.Data.DataSet;
$objAdapter.Fill($objDataSet) | Out-Null;
for ($i=0; $i -lt $objDataSet.Tables.Count; $i++) {
Write-Host ($objDataSet.Tables[$i] | Format-Table | Out-String);
}
$query = $null;
$objDataSet = $null;
$objConnection.Close();
$objConnection = $null;
I am using the Scripter class to give me a script for the data out of an existing database. I want to script a dataset that can be inserted into a production database. We are doing this to test if an installation of our Software is correct.
Unfortunately the dataset has to be removed later without any entries left behind so that it does not interfere with the data of our customers. So what I need are INSERT and DELTE statements. These are maintained manually at the moment which is too much of a burden.
Very well so I just went and executed the Scripter twice (once for INSERT, once for DELETE)
Problem is that when setting ScriptDrops to true then the output is in the form
DELETE FROM [dbo].[TableName]
What I would like is something of the form:
DELETE FROM [dbo].[TableName] WHERE ID = 'GUID'
Technically this would be possible since there are Primary Keys on all the tables.
The Scripter class must also in some form know of that things since it also gets the order of the DELETE-statements (dependencies) correct via foreign keys.
Any help on this would be appreciated.
Following are the 2 PowerShell-scripts I am using to export the data:
ScriptRepositoryData.ps1
$scriptPath = $MyInvocation.MyCommand.Path
$scriptDirectory = Split-Path $scriptPath -Parent
. $scriptDirectory\DatabaseScripting.ps1
$filepath='c:\data.sql'
$database='ECMS_Repository'
$tablesToExclude = #(
"SomeUnwantedTable"
)
$tablesListFromDatabase = GetTableList $database
$tablesArray = #()
$tablesListFromDatabase |% {
if (-not $tablesToExclude.Contains($_.Name.ToString()))
{
$tablesArray += $_.Name
}
}
ScriptInsert $database $tablesArray $filepath
DatabaseScripting.ps1
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.SMO") | out-null
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.SMOExtended") | out-null
Function GetTableList ($database)
{
Invoke-SqlCmd -Database $database -query "SELECT * FROM sys.tables"
}
Function ScriptInsert ($database, $tables, $destination)
{
try {
$serverMO = new-object ("Microsoft.SqlServer.Management.Smo.Server") "localhost"
if ($serverMO.Version -eq $null) {Throw "Can't find the instance localhost"}
$urnsToScript = New-Object Microsoft.SqlServer.Management.Smo.UrnCollection
$databaseMO = $serverMO.Databases.Item("ECMS_Repository")
if ($databaseMO.Name -ne $database) {Throw "Can't find the database $database"}
$tables |% {
$tableListMO = $databaseMO.Tables.Item($_, "dbo")
$tableListMO |% {
$urnsToScript.Add($_.Urn)
}
}
$scripter = new-object ('Microsoft.SqlServer.Management.Smo.Scripter') $serverMO
$scripter.Options.ScriptSchema = $False;
$scripter.Options.ScriptData = $true;
$scripter.Options.ScriptDrops = $true;
$scripter.Options.ScriptAlter = $true;
$scripter.Options.NoCommandTerminator = $true;
$scripter.Options.Filename = $destination;
$scripter.Options.ToFileOnly = $true
$scripter.Options.Encoding = [System.Text.Encoding]::UTF8
$scripter.EnumScript($urnsToScript)
Write-Host -ForegroundColor Green "Done"
}
catch {
Write-Host
Write-Host -ForegroundColor Red "Error occured"
Write-Host
Write-Host $_.Exception.ToString()
Write-Host
}
}
Unfortunately I did not find a way to do this using the Sql Management Objects.
Anyhow I now use the output of the Scripter and select the IDs of each table. I then use the IDs to change every line that looks like
DELETE FROM [dbo].[tableName]
to this:
DELETE FROM [dbo].[tableName] WHERE ID IN ('guid1', 'guid2')
Here is how I did it:
$content = Get-Content $destination
Clear-Content $destination
$content |% {
$line = $_
$table = $line.Replace("DELETE FROM [dbo].[","").Replace("]","")
$query = "SELECT ID, ClassID FROM" + $_
$idsAsQueryResult = Invoke-SqlCmd -Database $database -query $query
$ids = $idsAsQueryResult | Select-Object -Expand ID
if ($ids -ne $null) {
$joinedIDs = [string]::Join("','",$ids)
$newLine = $line + " WHERE ID IN ('" + $joinedIDs + "')"
Add-Content $destination $newLine
}
}
Where $destination is the script that has been generated with the Scripter class and $database is a string containing the database name.
I had to select a second column (ClassID which is there on all tables due to our OR mapper re-store) because of some weird error in Select-Object which I do not fully understand.
This of course only works because all tables have primary keys and all primary keys are named ID and are not combined primary keys or something.
You could of course achieve the same thing for other more complicated database schemas by extracting primary key information via SQL management objects.