Powershell - Ingesting API json results into SQL Server table - sql-server

Looking for some help with ingesting data retrieved from an API endpoint and inserting it into SQL Server. This endpoint returns json.
I'm trimming the script to make this more readable. The script gets the data I'm requesting, the issues is the insert into the SQL table.
# Pass the `Authorization` header value with the payload we calculate from the id + secret
$Time = Invoke-RestMethod -Uri $TimeCardURL -Method Get -Header #{ Authorization = "Bearer ${AuthorizationValue}" } -Body $Body -Certificate $Cert -ContentType 'application/json'
$Time.teamTimeCards | Select-Object associateOID,timeCards
#SQL authentication and insert $Result data
$serverName = "sql-server"
$databaseName = "database"
$tableName = "table"
$Connection = New-Object System.Data.SQLClient.SQLConnection
$Connection.ConnectionString = "server='$serverName';database='$databaseName';trusted_connection=true;"
$Connection.Open()
$Command = New-Object System.Data.SQLClient.SQLCommand
$Command.Connection = $Connection
foreach($ID in $associateOID){
$insertquery="
INSERT INTO $tableName
([associateOID],[timeCards])
VALUES
('$associateOID','$timeCards')"
$Command.CommandText = $insertquery
$Command.ExecuteNonQuery()
}
$Connection.Close();
It's trying to load the entire object $associateOID into the column, which is resulting in a "String or binary data would be truncated in table 'database.table', column 'associateOID'. Truncated value: '1 2 3 4 5 6 7 8'. The statement has been terminated."
I'm guessing there is an issue with this section of the script
$Time.teamTimeCards | Select-Object associateOID,timeCards

I rewrote your code to use another source of data, but you probably get the idea:
$output = Get-Process | Select-Object Handles, Id, ProcessName
echo($output)
foreach($value in $output){
$id = $value.Id
$name = $value.ProcessName
$insertquery="
INSERT INTO $tableName
([ID],[Name])
VALUES
('$id','$name')"
echo($insertquery)
}

Related

How to pass csv file as query param in api call using powershell

I have developed two simple PS scripts that work fine separately. Script1 connects to a DB, run a sql query and save the output (only one column that is a list of the project names) as csv file. Script2 connects to an endpoint using API calls and prints the details of a projects. I use script1's output as script2's input manually. I have tried a couple of different ways to automate this process but I haven't been able to get it to work. Does anyone know how can I pass the csv file as query param in api call?
Here is what I have so far:
This is Script1:
#SQL Connection variables
$Server = "my server"
$DBName = "db name"
$credential = Import-CliXml -Path "C:\Test\MyCredential.xml"
$User = $Credential.UserName
$PW = $credential.GetNetworkCredential().Password
$Connection = New-Object System.Data.SqlClient.SqlConnection
$Connection.ConnectionString = "Server = $Server; Database = $DBName; User ID = $User; Password = $PW;"
$Connection.Open()
#$Connection.State
$SqlQuery = "select from table example"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.Connection = $Connection
$SqlCmd.CommandText = $SqlQuery
$CxSqlCmd.CommandTimeout = 0
#Creating sql adapter
$SqlAdapter = New-Object System.Data.sqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
#Creating Dataset
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$DataSet.Tables[0] | export-csv -Path $OuputFile -NoTypeInformation
The output for script 1 is 11223344, So I use this project name as my input or query param in my second script.
And this is Script2:
$credential = Import-CliXml -Path "C:\Test\MyCredential2.xml"
$credential = Import-CliXml -Path "C:\Test\MyCredential2.xml"
$APIKEY = $credential.GetNetworkCredential().Password
$token = "APIKEY " + "$APIKEY"
$Params = #{
uri = 'https:myendpoint/search?name=11223344'
Headers = #{'Authorization' = "API KEY $token"}
Method ='GET'
ContentType = 'application/json'
}
$Response = Invoke-RestMethod #Params
I really appreciate it if someone can help me with this.

Convert JSON array to single JSON object in Powershell

I am trying to convert JSON array into single JSON Object.
Below is the output screenshot
I want the output as
I am using the below powershell script
$getdb_Conn=Invoke-Sqlcmd -ServerInstance $ServerName -Database 'master' -Username $UserName -Password $Password -Query $getdb
$deadlockDB=$getdb_Conn.Database_Name
$deadlockSP=$getdb_Conn.SP_Name
$deadlockTable_Name=$getdb_Conn.Table_Name
$deadlockTIMESTAMP=$getdb_Conn.TIMESTAMP
$Obj = [PSCustomObject]#{
Database = $deadlockDB
SP = $deadlockSP
Table = $deadlockTable_Name
TIMESTAMP = $deadlockTIMESTAMP
}
Write-Output ( $obj | ConvertTo-Json)
Please someone help me, on how to the get required output. I do not want the JSON in an array.
The issue is you have one single non-array object that contains properties, which all contain arrays of values. To achieve the desired results, you need an array of objects that each contain one set of property values. You can choose to do this at the source when you are building $Obj or build a new set of objects to be sent to ConvertTo-Json.
For building a new set of objects using your current $Obj:
for ($i = 0; $i -lt $obj.TIMESTAMP.Count; $i++) {
[pscustomobject]#{
Database = $obj.Database[$i]
SP = $obj.SP[$i]
Table = $obj.Table[$i]
TIMESTAMP = $obj.TIMESTAMP[$i]
} | ConvertTo-Json
}
Building the objects from the source data:
$getdb_Conn=Invoke-Sqlcmd -ServerInstance $ServerName -Database 'master' -Username $UserName -Password $Password -Query $getdb
foreach ($conn in $getdb_Conn) {
[PSCustomObject]#{
Database = $conn.Database_Name
SP = $conn.SP_Name
Table = $conn.Table_Name
TIMESTAMP = $conn.TIMESTAMP
} | ConvertTo-Json
}

Looping through pscustomobject dataset and add to a SQL DB

I have a variable, $CiscoCMDB, which holds data for 2968 records like this (some fields are all filled, some are not like below):
SearchCode: D12345678911
Name1: 1212
Category: Office Phone
AssetTag:
Status: Stock
SerialNumber: FCH1549BBBB
Model: CISCO IP PHONE 7945G
MacAddress:
Location: SF
OwnerOrganization:
OwnerPerson:
I also have a SQL DB created, CiscoCMDB, with the same name for the columns that are present for the rows. I would like All records to be written to the SQL table. I've tried to use the Write-Datatable cmdlet, but it gave me errors. along with other CMDlets. I figured, if I am using SQL SELECT statements in my code, I should just use SQL INSERTs in my code. Only problem is, I'm not sure how to go about this other than knowing I will need a foreach loop to loop through.
Can someone help me on this as I'm getting confused and frustrated. I'm not exactly sure how to go about looping through correctly.
Pull data from another SQL DB
Organize data furthur (picking the columns I want)
Manipulate data to strip ^C from searchcode (all our search codes are prefaced with C)
Manipulate data to strip ^DN from Name1 field (again all Name1s are prefaced with DN)
Fill in my data
$CMDBCiscoQuery = #()
foreach ($row in $table) {
$hash = [ordered]#{
'SearchCode' = $row.SearchCode
'Category' = $row.Category
'Status' = $row.Status
'Name1' = $row.Name1
'SerialNumber' = $row.SerialNumber
'Model' = $row.Model
'MacAddress' = $row.MacAddress
'Location' = $row.Location
'OwnerOrg' = $row.OwnerOrganization
'Owner' = $row.OwnerPerson
}
$obj = New-Object -TypeName PSObject -Property $hash
$CMDBCiscoQuery+= $obj
$CMDBCisco = $CMDBCiscoQuery |
Select #{n='SearchCode';e={$_.Searchcode -replace "^C"}},
#{n='Name1';e={$_.DN -replace "^DN"}}, Category, AssetTag,
Status, SerialNumber, Model, MacAddress, Location,
OwnerOrganization, OwnerPerson
#Filling CMDB table
Write-Verbose "Filling CMDBCisco Table"
$ConnectionTimeout = 30
$ServerInstance = "myserver"
$Database = "Audits"
$conn = New-Object System.Data.SqlClient.SQLConnection
$ConnectionString = "Server={0};Database={1};Connect Timeout={2};Integrated Security = True;" -f $ServerInstance,$Database,$ConnectionTimeout
$conn.ConnectionString = $ConnectionString
$conn.Open()
$cmd = New-Object System.Data.SqlClient.SqlCommand($conn);
$conn.Close()

Adding column to SQL query on multiple instances from powershell

I have the following powershell script which reads in a list of servers, and runs SQL command on these servers. This data is then exported to csv and to excel format
I would like to be able to add the targeted server name from my server list as the first column so columns would look like this (server name added to front)
Server Name | Name | CollectionSet ID | Collection Mode | Retention Period | Schedule
This is the current script I have:
Param
(
[string]$fServers = 'W:\Theo\Scripts\mdw_servers.csv'
)
$query = "SELECT a.name AS 'DC Name',
collection_set_id AS 'Collection_set ID',
CASE collection_mode
WHEN 1 THEN 'non-cached'
WHEN 0 THEN 'cached'
END AS 'Collection Type' ,
days_until_expiration AS 'Retention Period' ,
b.name AS 'Schedule Name'
FROM msdb.dbo.syscollector_collection_sets a ,
msdb.dbo.sysschedules b
WHERE a.schedule_uid = b.schedule_uid
AND is_running = 1;"
$csvFilePath = "W:\Theo\Scripts\queryresults.csv"
$excelFilePath = "W:\Theo\Scripts\queryresults.xls"
# Run Query against multiple servers, combine results
$allServers = Get-Content -Path $fServers
foreach ($Server in $allServers) {
write-host "Executing query against server: " $Server
$results += Invoke-Sqlcmd -Query $query -ServerInstance $Server;
}
# Output to CSV
write-host "Saving Query Results in CSV format..."
$results | export-csv $csvFilePath -NoTypeInformation
# Convert CSV file to Excel
write-host "Converting CSV output to Excel..."
$excel = New-Object -ComObject excel.application
$excel.visible = $False
$excel.displayalerts=$False
$workbook = $excel.Workbooks.Open($csvFilePath)
$workSheet = $workbook.worksheets.Item(1)
$resize = $workSheet.UsedRange
$resize.EntireColumn.AutoFit() | Out-Null
$xlExcel8 = 56
$workbook.SaveAs($excelFilePath,$xlExcel8)
$workbook.Close()
$excel.quit()
$excel = $null
write-host "Results are saved in Excel file: " $excelFilePath
Any input is appreciated!
have you tried
SELECT ##SERVERNAME AS 'Server Name'
https://msdn.microsoft.com/en-us/library/ms187944.aspx

Import-CSV Where -notmatch list of array values

I am trying to import a csv file where it doesn't import any values that are listed in an array declared above the import line. The array is made up of certain values that are pulled out of a database and I wan't to import all rows in the csv file that the txnID column values do not match the values in the array however I am having trouble trying to loop through my array.
I am new to using powershell and maybe I am not even implementing the array correctly but I haven't been able to find anything about import-csv Filename |Where column -notmatch $array
$Database = 'Database'
$Server = "Server"
$SqlQuery = 'SELECT DISTINCT WebOrderNumber FROM tbOrders
WHERE WebOrderNumber IS NOT NULL AND Len(WebOrderNumber)>8'
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Data Source=Datasource;Initial Catalog=Database;User ID=ID;Password=Pass;Integrated Security=False;"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $SqlQuery
$SqlCmd.Connection = $SqlConnection
$SqlConnection.Open()
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$Reader = $SqlCmd.ExecuteReader()
while ($Reader.Read()) {
#write-Output($Reader.GetValue($0))
$Key = $Reader.GetValue($0)
$table += $Key
}
foreach ($Row in $table){
write-output($Row)
$CSVFile = (import-csv "C:\Users\Office-Admin\Documents\Complete Sales Orders.csv") |where {$_.txnID -ne $Row} | select txnID, FirstName, LastName, Cust_Name, mc_Shipping, Payment_Gross, address_street, Address_Zip, quantity, item_name, item_number, payer_email, address_city, address_state, address_country, address_name, Shipping_Method, mc_gross
}
$CSVFile | export-csv "C:\Users\Office-Admin\Documents\Sales Order Import List.csv" -notypeinformation
remove-item variable:table
#Send SMTP Message
$SqlConnection.Close()
I've updated my code slightly however the problem still persists. I'm realizing that I believe with the code now, everytime I loop through and import, the previous condition in the where is forgotten so the only value that is not imported in the end is the last $Row value but I need all of the values in $table to be excluded and I don't know how I can do this.
Something list this should work. The main problem is you are over writing your csv every loop.
$table = import-csv file1.csv | % {$_ID} #gets array of just the ID values
$CSVFile = Import-csv file2.csv | where{$table -notcontains $_.ID} | export-csv output.csv -notypeinformation
To show you how this works I created to files as an example:
File 1: CSV with IDs:
ID,Stuff
123,alittlestuff
234,morestuff
345,evenmore
456,alotmore
567,somemore
678,notsomuch
789,tonesofstuff
File 2: csv with ID and stuff:
ID,stuff
123,hello
ghf,world
234,test
lkj,this
after running the code the only rows that get output are:
ID,Stuff
ghf,world
lkj,this
So I think to fit it into your code use this:
$filter = $table | %{$_.txnID}
$CSVFile = (import-csv "C:\Users\Office-Admin\Documents\Complete Sales Orders.csv") | where{$filter -notcontains $_.txnID} || export-csv "C:\Users\Office-Admin\Documents\Sales Order Import List.csv" -notypeinformation

Resources