I have CSV file
My PowerShell Script attempts to store SourceIP, DestinationIP, and Traffic in multidimensional array
$source = #((Import-Csv D:\Script\my.csv).SourceIP)
$dest = #((Import-Csv D:\Script\my.csv).DestinationIP)
$t = #((Import-Csv D:\Script\my.csv).Traffic)
$multi = #($source),#($dest),#($t)
When I try to read from first element of $multi, I expect to get a list of SourceIP
foreach ($q in $multi){
write-host $q[0]
write-host `n
}
But instead, I get SourceIP, DestinationIP, Traffic, i.e.
10.153.128.110
10.251.68.80
3.66 GB
And if I try
foreach ($q in $multi){
write-host $q[0][0][0]
write-host `n
}
I get
1
1
3
How to troubleshoot?
UPDATE
Ultimate goal is to
Count total traffic
Count traffic if SourceIP or Destination IP fits into certain pattern, i.e. 10.251.22.x
Get percentage
UPDATE II
I am able to get code to import CSV and tally total bandwidth only, but I also need bandwidth from SourceIP and DestinationIP with certain pattern.
$t = #((Import-Csv D:\Script\my.csv).Traffic)
foreach ($k in $t){
write-host $k
}
foreach ($i in $t){
$j += ,#($i.split(" "))
}
foreach ($m in $j){
switch ($m[1]){
GB {
$m[0] = [int]($m[0]) * 1000
$m[1] = 'MB'
}
MB {}
KB {
$m[0] = [int]($m[0]) / 1000
$m[1] = 'MB'
}
}
$total_bandwidth += $m[0]
}
write-host Total bandwidth is ("{0:N2}" -f $total_bandwidth) MB
You should not split array of object to multiple parallel arrays of properties. It is much easy to operate when objects are whole.
$Scale=#{
B=1e00
KB=1e03
MB=1e06
GB=1e09
TB=1e12
}
$TrafficBytes={
$a=-split$_.Traffic
[double]$a[0]*$Scale[$a[1]]
}
Import-Csv D:\Script\my.csv|
ForEach-Object $TrafficBytes|
Measure-Object -Sum #total traffic
Import-Csv D:\Script\my.csv|
Where-Object {$_.DestinationIP-like'10.*'}| #condition
ForEach-Object $TrafficBytes|
Measure-Object -Sum #traffic by condition
PetSerAl has a good idea for the conversion, but here is a way to do this that requires iterating the CSV only once and will give your percentages.
$filter = "10.251.22.*"
$Scale=#{
B=1e00
KB=1e03
MB=1e06
GB=1e09
TB=1e12
}
$myCsv = Import-Csv D:\Script\my.csv | Select-Object *, #{ Name = "TrafficBytes"; Expression = { $a = -split $_.Traffic; [double] $a[0] * $Scale[$a[1]] } }
$trafficFiltered = $myCsv | Group-Object { $_.SourceIP -like $filter -or $_.DestinationIP -like $filter } | Select-Object #{ Name = "IPFilter"; Expression = { if ($_.Name -eq $true) { $filter } else { "Other" } } }, #{ Name = "TrafficBytes"; Expression = { ($_.Group | Measure-Object -Sum "TrafficBytes").Sum } }
$trafficTotal = $myCsv | Measure-Object -Sum TrafficBytes
$trafficReport = Select-Object IPFilter, TrafficBytes, #{ Name = "Percent"; Expression = { "{0:P}" -f $_.TrafficBytes / $trafficTotal.Sum * 100.0 } }
$trafficReport
Related
Maybe the header is wrong but i dont know how to explain.
I have 4 csv files with aprox 15000 rows in each looking like this
number,"surname","forename","emailAddress","taxIdentifier"
100238963,"Smith","John","john.smith#gmail.com","xxxxxxxxxxxx"
Im reading in 9999 of the rows and creating a json file we use on a site to check every person, we then get a respond back for most of the users, and that respons is "number"
Then i need to find all them persons in the first array.
I have done it like this today, but it take to much time to check every person like this, is there any better way of doing this?
This is the code for getting the persons from the file and create json file:
$Files = Get-ChildItem -Path "$Folders\\*" -Include *.csv -Force
foreach ($File in $Files){
$fname = $file
$fname = (Split-Path $File.name -leaf).ToString().Replace(".csv", "")
$Savefile = $fname+ "_Cleaned.csv"
$users = Import-Csv $File
$body = "{`"requestId`": `"144x25`",`"items`": ["
$batchSize = 9999
$batchNum = 0
$row = 0
while ($row -lt $users.Count) {
$test = $users[$row..($row + $batchSize - 1)]
foreach ($user in $test) {
$nr = $user.number
$tax = $user.taxIdentifier
$body += "{`"itemId`": `"$nr`",`"subjectId`": `"$tax`"},"
}
And then this is the code to deal with the respons:
$Result = #()
foreach ($1 in $response.allowedItemIds)
{
foreach ($2 in $Users){
If ($2.number -like $1)
{
$Result += [pscustomobject]#{
number = $2.number
Surname = $2.surname
Forename = $2.forename
Email = $2.emailaddress
Taxidendifier = $2.taxIdentifier
}
}
}
}
$Result | Export-Csv -path "$folders\$savefile" -NoTypeInformation -Append
$row += $batchSize
$batchNum++
Hope someone has any ideas
Cheers
I think you can just do this:
# read the original data file
$originalCsv = #"
number,"surname","forename","emailAddress","taxIdentifier"
1000,"Smith","Mel","mel.smith#example.org","xxxxxxxxxxxx"
3000,"Wilde","Kim","kim.wilde#example.org","xxxxxxxxxxxx"
2000,"Jones","Gryff Rhys","gryff.jones#example.org","xxxxxxxxxxxx"
"#
$originalData = $originalCsv | ConvertFrom-Csv
# get a response from the api
$responseJson = #"
{
"requestId": "144x25",
"responseId": "2efb8b47-d693-46ac-96b1-a31288567cf3",
"allowedItemIds": [ 1000, 2000 ]
}
"#
$responseData = $responseJson | ConvertFrom-Json
# filter original data for matches to the response
$matches = $originalData | where-object { $_.number -in $responseData.allowedItemIds }
# number surname forename emailAddress taxIdentifier
# ------ ------- -------- ------------ -------------
# 1000 Smith Mel mel.smith#example.org xxxxxxxxxxxx
# 2000 Jones Gryff Rhys gryff.jones#example.org xxxxxxxxxxxx
# write the data out
$matches | Export-Csv -Path ".\myfile.csv" -NoTypeInformation -Append
I don't know if that will perform better than your example, but it should do as it's not got a nested loop that runs original row count * response row count times.
I have file with 80k lines. I want to read each line then look for group name which is between number and backup type ( incr,full or manual). then add group in hash table to corresponding server name. I want use hashtable which has array for group names. this is piece of requirement in my big script.
Input file
rspedw03.corpads.local 3085876532 JC_UNIX_FS01_INCR_DD2 JC_FS_DD2 UNIX_FS01_INCR_DD2 incr 02/23/2022 03/29/2022 03/29/2022 disk 1645592426 backup 3013 MB JCBDD2301P.CORPADS.LOCAL
rsuedw01.corpads.local 1020344 JC_DB_DB2 JC_DB Clone_DR full 02/23/2022 04/04/2022 04/04/2022 disk 1645592431 Clone_DR 997 KB MNBDD3302P.corpads.local
rsuedw01.corpads.local 1020344 JC_DB_DB2 full 02/23/2022 04/04/2022 03/30/2022 disk 1645592431 997 KB JCBDD1300P.corpads.local
rsuedw03.corpads.local 12608 MN_UNIX_NP_7_Days MN_DB Clone_DR full 02/23/2022 04/21/2022 04/21/2022 disk 1645592432 Clone_DR 13 KB JCBDD1300P.corpads.local
'# -split '\r?\n'
output should look like
rspedw03.corpads.local JC_UNIX_FS01_INCR_DD2 JC_FS_DD2 UNIX_FS01_INCR_DD2 MN_UNIX_NP_7_Days MN_DB
rsuedw01.corpads.local JC_DB_DB2 JC_DB Clone_DR
i got so far ,
$out = Get-Content C:\scripts\test1.txt
$ht = #{}
$arr = #()
foreach ( $line in $out)
{
$mn = $line -csplit "incr|full|manual"
$md = $mn[0].split(" ")
}
but some line has one group other might have 4 group how do i capture that ?
Here is my complete code ,
$out=Get-Content C:\anil\scripts\test2.txt
$ht = #{}
$arr = #()
$today = Get-Date
foreach ( $line in $out){
$arr=$line.Split(" ")
if ( $arr[0] -ne "nwsppl300p.corpads.local"){
$mn=$line -csplit "incr|full|manual"
$md=$mn[1] -split "\s{1,}"
if ($line -match '.*( backup |Clone_DR ).*') {$btype=$md[9]} else {$btype=$md[8]}
$clientHostName,$null,$backupPlans = -split $mn[0]
$date =$mn[1].split(" ")[2]
$newdate=[Datetime]::ParseExact($date, 'MM/dd/yyyy', $null)
$diff = New-TimeSpan -Start $today -end $newdate
#### look for one year only ########
if ( $diff.Days -lt 400 ) {
if ( $arr[12] -ne "Clone_DR") {
if ($arr[0] -notin $ht.keys){
$ht[$arr[0]] = #{}
if ($btype -match "DB2") {
$ht[$arr[0]]['Db2size'] = $arr[1]
$ht[$arr[0]]['groups'] = #($backupPlans)
}
if ($btype -match "RMAN") {
$ht[$arr[0]]['RMANsize'] = $arr[1]
$ht[$arr[0]]['groups'] = #($backupPlans)
}
if ($btype -notmatch "RMAN" -and $btype -notmatch "DB2" ){
$ht[$arr[0]]['Filesize'] = $arr[1]
$ht[$arr[0]]['groups'] = #($backupPlans)
}
} else {
if ($btype -match "DB2" -and $arr[1] -gt $ht[$arr[0]]['Db2size'] ) {
$ht[$arr[0]]['Db2size'] = $arr[1]
if ($backupplans -notin $ht[$arr[0]]['groups']) { $ht[$arr[0]]['groups'] += #($backupPlans)}
}
if ($btype -match "RMAN" -and $arr[1] -gt $ht[$arr[0]]['RMANsize']) {
$ht[$arr[0]]['RMANsize'] = $arr[1]
if ($backupplans -notin $ht[$arr[0]]['groups']) { $ht[$arr[0]]['groups'] += #($backupPlans)}
}
if ($btype -notmatch "RMAN" -and $btype -notmatch "DB2" -and $arr[1] -gt $ht[$arr[0]]['Filesize']){
$ht[$arr[0]]['Filesize'] = $arr[1]
if ($backupplans -notin $ht[$arr[0]]['groups']) { $ht[$arr[0]]['groups'] += #($backupPlans)}
}
}
} ###clone_dr
} ###less than 400
} ### chcking for networker server
} #### looping thru file
write-host "=================================In MB ==============================================="
write-host "===ServerName==============OverAllsize======DB2size===========RMANsize========FileSize"
write-host "======================================================================================"
$ht.GetEnumerator()| ForEach-Object {
$total = $_.value.Db2size/1024/1024 + $_.value.RMANsize/1024/1024 + $_.value.Filesize/1024/1024
"{0,-25} {1:n2} {2:n2} {3:n2} {4:n2} {5,-25}" -f $_.name,$total,$($_.value.Db2size/1024/1024),$($_.value.RMANsize/1024/1024),$($_.value.Filesize/1024/1024),$_.value.groups}
Once you've got the part before the type (eg. 'rsuedw01.corpads.local 1020344 JC_DB_DB2 ') split into individual words ('rsuedw01.corpads.local', '1020344', 'JC_DB_DB2'), then you know that the first string is going to be the hostname, the second string is going to be the ignored, and anything else is necessarily a label you want to collect.
Then you just need to ensure that an entry with an associated array exists:
$backupLogLines = #(Get-Content C:\scripts\test1.txt) -match "incr|full|manual"
$planTable = #{}
foreach ($line in $backupLogLines)
{
$metadataParts = $line -csplit "incr|full|manual"
# first string is host name,
# second string is discarded,
# ... and the rest are backup plans
$clientHostName,$null,$backupPlans = -split $metadataParts[0]
if(-not $planTable.Contains($clientHostName)){
# Create new array containing the backup plans for the current
$planTable[$clientHostName] = #($backupPlans)
}
else {
$planTable[$clientHostName] += $backupPlans
}
}
To write the results to a file:
$planTable.GetEnumerator() |ForEach-Object {
$_.Name,$(#($_.Value |Sort-Object -Unique) -join ' ') -join ' '
} |Set-Content path\to\output.txt
In a high-availability environment (two DCs, Primary and Standby) with 50 clusters. Each cluster has 5 to 6 nodes in it. I want to ensure all the nodes in every cluster is "Online" (State) and running in "Primary" (OwnerNode). Any node that shows otherwise has to be noted.
I'm using arrays to store the required information in a foreach loop. The problem is, it is taking too long to compile. I want to complete it sooner.
$Clusternodes= * some 50 elements *
$Standbynodes= * some 50 elements *
foreach ($cluster in $Clusternodes) {
$NotOnline += Get-ClusterGroup -Cluster $Cluster |
where {$_.State -ne "Online"} |
Select-Object Name,OwnerNode,State
foreach ($node in $Standbynodes) {
$Standbys += Get-ClusterGroup -Cluster $Cluster |
where {$_.OwnerNode -eq "$node"} |
Select-Object Name,OwnerNode,State
}
}
Edit:
Get-clustergroup -Cluster $Cluster returns 5 to 6 entries in every cluster. The output contains three columns Name, OwnerNode, State.
I'm storing every entry in an array based on its state (whether it is running or not) and owner node (whether in primary or secondary DC). Hence, I want four arrays. While the former is easy, the latter isn't. As that itself is another two arrays with 50 odd elements in each. Hence I used hashtables for it as mentioned below. However, when I tried the below code it always returns an empty array.
$Clusternodes = * some 50 elements *
$Standbynodes = * some 50 elements *
$Primarynodes = * some 50 elements *
$pr = #{}
$sb = #{}
$Standbynodes | ForEach-Object { $sb[$_] = $true }
$Primarynodes | ForEach-Object { $pr[$_] = $true }
$RunninginPrimary = #()
$NotRunninginPrimary = #()
$RunninginStandby = #()
$NotRunninginStandby = #()
foreach ($cluster in $Clusternodes) {
$c = Get-ClusterGroup -Cluster $Cluster
$NotRunninginStandby += $c | Where-Object {
($_.State -ne "Online") -and ($sb.ContainsKey($_.OwnerNode))
} | Select-Object Name,OwnerNode,State
$NotRunninginPrimary += $c | Where-Object {
($_.State -ne "Online") -and ($pr.ContainsKey($_.OwnerNode))
} | Select-Object Name,OwnerNode,State
$RunninginStandby += $c | Where-Object {
($_.State -eq "Online") -and ($sb.ContainsKey($_.OwnerNode))
} | Select-Object Name,OwnerNode,State
$RunninginPrimary += $c | Where-Object {
($_.State -eq "Online") -and ($pr.ContainsKey($_.OwnerNode))
} | Select-Object Name,OwnerNode,State
}
You query each cluster multiple times. To speed up your code query each cluster just once, store the result in a variable, and use that variable in the rest of the loop. You may also want to replace the nested loop that iterates over $Standbynodes with a hashtable lookup.
$Clusternodes = ...
$Standbynodes = ...
$sb = #{}
$Standbynodes | ForEach-Object { $sb[$_] = $true }
$NotOnline = #()
$Standbys = #()
foreach ($cluster in $Clusternodes) {
$c = Get-ClusterGroup -Cluster $Cluster
$NotOnline += $c | Where-Object { $_.State -ne "Online" } |
Select-Object Name,OwnerNode,State
$Standbys += $c | Where-Object { $sb.ContainsKey($_.OwnerNode) } |
Select-Object Name,OwnerNode,State
}
Try using workflow.
workflow clusterCheck {
$Clusternodes= * some 50 elements *
$Standbynodes= * some 50 elements *
foreach -parallel ($cluster in $Clusternodes) {
$NotOnline += Get-ClusterGroup -Cluster $Cluster |
where {$_.State -ne "Online"} |
Select-Object Name,OwnerNode,State
foreach -parallel ($node in $Standbynodes) {
$Standbys += Get-ClusterGroup -Cluster $Cluster |
where {$_.OwnerNode -eq "$node"} |
Select-Object Name,OwnerNode,State
}
}
}
Source:- Scripting guy
Below is only an example, I have seen a lot of script to breakdown a .CSV file in smaller files but struggling with this.
How can we with PowerShell, find the header indicated by ALPH take each subsequent line, stop when it reaches ALPT (inclusive) and put this text into another file.
The operation will need to run through the whole file and the number of ALPD or ALPC lines will vary.
ALPH can be considered as a header while the information contained is needed as some field value can be different. The only constant are ALPH and ALPT.
ALPH;8102014
ALPC;PK
ALPD;50
ALPD;40
ALPT;5
ALPH;15102014
ALPC;PK
ALPD;50
ALPD;50
ALPD;70
ALPD;70
ALPD;71
ALPD;72
ALPD;40
ALPT;6
ALPH;15102014
ALPC;PK
ALPD;50
ALPD;50
ALPD;40
ALPT;6
If I understood your question correctly, something like this should work:
$csv = 'C:\path\to\your.csv'
$pattern = 'ALPH[\s\S]*?ALPT.*'
$cnt = 0
[IO.File]::ReadAllText($csv) | Select-String $pattern -AllMatches |
select -Expand Matches | select -Expand Groups |
% {
$cnt++
$outfile = Join-Path (Split-Path $csv -Parent) "split${cnt}.csv"
[IO.File]::WriteAllText($outfile, $_.Value)
}
Here is a way using switch. Your original file is in C:\temp\ALPH.CSV here is the way I imagine to find the begin an the end.
$n = 1
switch -File 'C:\temp\ALPH.CSV' -Regex
{
'^ALPH.*' {
Write-Host "Begin $n"
}
'^ALPT.*' {
Write-Host "End $n"
$n++
}
}
Now saving lines to a var and exporting files :
$n = 1
$csvTmp = #()
switch -File 'C:\temp\ALPH.CSV' -Regex
{
'^ALPH.*' {
Write-Host "Begin $n"
$csvTmp += $_
}
'^ALPT.*' {
Write-Host "End $n"
$csvTmp += $_
$csvTmp | Set-Content "c:\temp\file$n.csv"
$csvTmp = #()
$n++
}
default {
$csvTmp += $_
}
}
I am trying to parse robocopy log files to get file size, path, and date modified. I am getting the information via regex with no issues. However, for some reason, I am getting an array with a single element, and that element contains 3 hashes. My terminology might be off; I am still learning about hashes. What I want is a regular array with multple elements.
Output that I am getting:
FileSize FilePath DateTime
-------- -------- --------
{23040, 36864, 27136, 24064...} {\\server1\folder\Test File R... {2006/03/15 21:08:01, 2010/12...
As you can see, there is only one row, but that row contains multiple items. I want multiple rows.
Here is my code:
[regex]$Match_Regex = "^.{13}\s\d{4}/\d{2}/\d{2}\s\d{2}:\d{2}:\d{2}\s.*$"
[regex]$Replace_Regex = "^\s*([\d\.]*\s{0,1}\w{0,1})\s(\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2})\s(.*)$"
$MainContent = New-Object System.Collections.Generic.List[PSCustomObject]
Get-Content $Path\$InFile -ReadCount $Batch | ForEach-Object {
$FileSize = $_ -match $Match_Regex -replace $Replace_Regex,('$1').Trim()
$DateTime = $_ -match $Match_Regex -replace $Replace_Regex,('$2').Trim()
$FilePath = $_ -match $Match_Regex -replace $Replace_Regex,('$3').Trim()
$Props = #{
FileSize = $FileSize;
DateTime = $DateTime;
FilePath = $FilePath
}
$Obj = [PSCustomObject]$Props
$MainContent.Add($Obj)
}
$MainContent | % {
$_
}
What am I doing wrong? I am just not getting it. Thanks.
Note: This needs to be as fast as possible because I have to process millions of lines, which is why I am trying System.Collections.Generic.List.
I think the problem is that for what you're doing you actually need two foreach-object loops. Using Get-Content with -Readcount is going to give you an array of arrays. Use the -Match in the first Foreach-Object to filter out the records that match in each array. That's going to give you an array of the matched records. Then you need to foreach through that array to create one object for each record:
[regex]$Match_Regex = "^.{13}\s\d{4}/\d{2}/\d{2}\s\d{2}:\d{2}:\d{2}\s.*$"
[regex]$Replace_Regex = "^\s*([\d\.]*\s{0,1}\w{0,1})\s(\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2})\s(.*)$"
$MainContent =
Get-Content $Path\$InFile -ReadCount $Batch |
ForEach-Object {
$_ -match $Match_Regex |
ForEach-Object {
$FileSize = $_ -replace $Replace_Regex,('$1').Trim()
$DateTime = $_ -replace $Replace_Regex,('$2').Trim()
$FilePath = $_ -replace $Replace_Regex,('$3').Trim()
[PSCustomObject]#{
FileSize = $FileSize
DateTime = $DateTime
FilePath = $FilePath
}
}
}
You don't really need to use the collection as an accumulator, just output PSCustomObjects, and let them accumulate in the result variable.