My Research:
Okay, I have looked at dozens upon dozens of examples for Write-Progress and observed the following. Most of the time used in conjuction with a loop, usually "for" or "foreach". Most examples don't do anything but count to 100. Other more useful examples will perform a certain command, such as copying files. No examples envelope entire scripts.
My Setup:
I have several scripts (thousands of lines of code) which call each other at certain times. One single script controls or calls all the other scripts. The script runs for about 15 minutes, during that time I would like to provide status using Write-Progress.
My Question:
How can I use Write-Progress to provide status while ALL my scripts are executing? Basically I want to "wrap" Write-Progress around all my scripts or whatever is the best method for providing status for multiple called scripts from a single script.
Best Example:
The best use of this I've seen thus far is when using the Update-Help CmdLet in PowerShell V3. But since I cannot see the source-code for the Update-Help CmdLet, this doesn't help me.
Try this out. First file is master.p1:
$parentId = 1
$childId = 2
Write-Progress -Id $parentId -Activity "Running master script" -Status "Step 1 of 3" -PercentComplete 0
.\slave.ps1 $parentId $childId
Write-Progress -Id $parentId -Activity "Running master script" -Status "Step 2 of 3" -PercentComplete 33.3
.\slave.ps1 $parentId $childId
Write-Progress -Id $parentId -Activity "Running master script" -Status "Step 3 of 3" -PercentComplete 66.3
.\slave.ps1 $parentId $childId
Second file is slave.ps1:
param([int32]$ProgressParentId, [int32]$progressId)
for($i = 0; $i -le 100; $i += 10)
{
Write-Progress -Id $progressId -ParentId $parentId `
-Activity "Running slave script" `
-Status "Processing $i" `
-CurrentOperation "CurrentOp $i" -PercentComplete $i
Start-Sleep -Milliseconds 500
}
Put those two files in the same dir and from PowerShell (or ISE) execute master.ps1. I have used this approach before to report progress of multiple phases across multiple scripts. The key is to pass the ParentId of the top level progress to the child scripts so they can report progress in that same context. If you provide a unique Id for each, they can get their own separate progress bar. Or just the same Id everywhere to update a single progress bar.
Related
In my PowerShell script, I will query the database using the select statement and it will run every one hour. It will took a quite longer period for completing all the process each time the scheduler executed, so when the next scheduler executed, it might query the old data where its still processing by the previous scheduler. May I know whether any method to prevent this? From what I read (this link) there is a method to achieve it.
Even I changed my query+PowerShell into the following, I found it still not able to achieve the things that I want
$queryAuth2="select * from [StagingDB].[dbo].[UnixAccResetPassword] WITH (HOLDLOCK,READPAST)"
#write-host "The command is : "$queryAuth
$command2=$connection.CreateCommand()
$command2.CommandText=$queryAuth2
$reader2 = $command2.ExecuteReader()
#write-host "The reader is : "$reader
#start-sleep -seconds 180
while ($reader2.read())
{
# write-host "In reader 1"
for ($i = 0; $i -lt $reader2.FieldCount; $i++)
{
# write-host "In reader 2"
$pfno=$reader2.GetValue($i)
write-host "pfno :"$pfno
}
}
The thing I want to achieve is when the select statement query the 5 records(item1,item2,item3,item4,item5) from table, it will locked until those records finished processed and remove away from table.
I have a PowerShell script that pulls in 1.4+ million rows of data and saves it to a HUGE CSV file that then gets imported into an SQL server. I thought there might be a way to have PowerShell insert the data into the SQL server directly but I am not sure how.
One of my concerns is that I don't want to buffer up the AD result into memory and then write them. I'd rather write them in batches of 1000 or something so memory consumption stays down. Get 1000 records, save to SQL server, and repeat...
I see articles about how to get PowerShell to write to an SQL server but they all seem to either do ALL data at one time or one record at a time -- both of which seem inefficient to me.
This is the PowerShell script I have to query AD.
# the attributes we want to load
$ATTRIBUTES_TO_GET = "name,distinguishedName"
# split into an array
$attributes = $ATTRIBUTES_TO_GET.split(",")
# create a select string to be used when we want to dump the information
$selectAttributes = $attributes | ForEach-Object {#{n="AD $_";e=$ExecutionContext.InvokeCommand.NewScriptBlock("`$_.$($_.toLower())")}}
# get a directory searcher to search the GC
[System.DirectoryServices.DirectoryEntry] $objRoot = New-Object System.DirectoryServices.DirectoryEntry("GC://dc=company,dc=com")
[System.DirectoryServices.DirectorySearcher] $objSearcher = New-Object System.DirectoryServices.DirectorySearcher($objRoot)
# set properties
$objSearcher.SearchScope = "Subtree"
$objSearcher.ReferralChasing = "All"
# need to set page size otherwise AD won't return everything
$objSearcher.PageSize = 1000
# load the data we want
$objSearcher.PropertiesToLoad.AddRange($attributes)
# set the filter
$objSearcher.Filter = "(&(objectClass=group)(|(name=a*)(name=b*)))"
# get the data and export to csv
$objSearcher.FindAll() | select -expandproperty properties | select $selectAttributes | export-csv -notypeinformation -force "out.csv"
I use Out-DataTable to convert my object array into a DataTable object type, then use Write-DataTable to bulk insert that into a database (Write-DataTable uses SqlBulkCopy to do this).
Caveats/gotchas for this (SqlBulkCopy can be a nuisance to troubleshoot):
Make sure your properties are the correct type (string for varchar/nvarchar, int for any integer values, dateTime can be string as long as the format is correct and SQL can parse it)
Make sure you properties are in order and line up with the table you're inserting to, including any fields that auto fill (incrementing ID key, RunDt, etc).
Out-DataTable: https://gallery.technet.microsoft.com/scriptcenter/4208a159-a52e-4b99-83d4-8048468d29dd
Write-DataTable: https://gallery.technet.microsoft.com/scriptcenter/2fdeaf8d-b164-411c-9483-99413d6053ae
Usage
If I were to continue on your example and skip the CSV, this is how I would do it... replace the last two lines with the code below (assuming that your object properties line up with the table perfectly, your SQL server name is sql-server-1, database name is org, and table name is employees):
try {
Write-DataTable -ServerInstance sql-server-1 -Database org -TableName employees -Data $($objSearcher.FindAll() | Select-Object -expandproperty properties | Select-Object $selectAttributes | Out-DataTable -ErrorAction Stop) -ErrorAction Stop
}
catch {
$_
}
Looking at your code it looks like you come from .NET or some language based on .NET. Have you heard of the cmdlets Get-ADUser / Get-ADGroup? This would simplify things tremendously for you.
As far as the SQL connection goes PowerShell doesn't have any native support for it. Microsoft has made cmdlets for it though! You just have to have SQL Server Installed in order to get them.... Which is kinda a bummer since SQL is so heavy and not everyone wants to install it. It is still possible using .NET, it's just not very quick or pretty. I won't be giving advice on the cmdlets here, you can Google that. As far as .NET, I would start by reading some of the documentation on the System.Data.SqlClient namespace as well as some historical questions on the subject.
Finally, as you said it would be a good idea to try and avoid overloading your RAM. The big thing here is trying to keep your entire script down to one single AD query. This way you avoid the troubling scenario of data changing between one query and the next. I think the best way of doing this would be to save your results straight to a file. Once you have that you could use SqlBulkCopy to insert into the table straight from your file. The downside to this is that it doesn't allow for multiple AD Properties. At least I don't think SqlBulkCopy will allow for this?
Get-ADUser "SomeParamsHere" | Out-File ADOutput.txt
If you have to have multiple AD properties and still want to keep the RAM usage to a minimum...well I toyed around with a script that would work but makes a few calls that would read from the entire file, which defeats the whole purpose. Your best option might be to save each property to a separate file then do your whole write DB thing. Example:
New-Item Name.txt
New-Item DistinguishedName.txt
Get-ADUser "SomeParamsHere" -Properties "Name,DistinguishedName" | Foreach {
Add-Content -Path "Name.txt" -Value "$_.Name"
Add-Content -PassThru "DistinguishedName.txt" -Value "$_.DistinguishedName"
}
Store results in your last line of code in a variable instead of exporting it to csv.
Then create group's of size you want't.
Using Out-DataTable and Write-DataTable write to SQL - links in nferrell's answer.
$res = $objSearcher.FindAll() | select -expandproperty properties | select
$selectAttributes
$counter = [pscustomobject] #{ Value = 0 }
#create groups with 1000 entries each
$groups = $res | Group-Object -Property { [math]::Floor($counter.Value++ / 1000) }
foreach ($group in $groups){
#convert to data table
$dt = $group.group | Out-DataTable
$dt | Write-DataTable -Database DB -ServerInstance SERVER -TableName TABLE
}
`
You're making this unneccesarily complicated.
If I read your code correctly, you want all groups starting with 'a' or 'b'.
# the attributes we want to export
$attributes = 'name', 'distinguishedName'
Import-Module ActiveDirectory
Get-ADGroup -Filter {(name -like "a*") -or (name -like "b*")} -SearchBase 'dc=company,dc=com' |
select $attributes | Export-Csv -NoTypeInformation -Force "out.csv"
Instead of using Export-Csv at the end, just pipe the output to the command which creates the SQL rows. By piping objects (instead of assigning them to a variable) you give PowerShell the ability to handle them efficiently (it will start processing objects as they come in, not buffer everything).
Unfortunately I can't help you with the SQL part.
I have a PowerShell script I am using to automate creating volumes on a SAN. The script prompts the user to enter the pertinent information: number of volumes, volume size and naming pattern. Next it should run a command on the array to create the volumes using the information provided. Here is where I am stuck. Right now the script asks the user to specify the number of volumes to create. For example let's say the user enters the number 4. I a stuck on how to take 4 and make it an array 1,2,3,4 or 7 to 1,2,3,4,5,6,7. Maybe there is an easier way to do this via counting or some other function. I am new to scripting, so any help is greatly appreciated.
# Define the number of volumes to create
$num_vols = read-host "Please enter the number of volumes"
# Define naming pattern
$vol_name = read-host "Please enter the volume naming pattern"
What I wanted to do was create a function that would basically run something like:
# Run command to create volumes on array
foreach ($i in $num_vols){
& "command to execute" $vol_name + $i
}
Thanks for the suggestions. I am now using:
# Run command to create volumes on array
foreach ($i in 1..$num_vols){
& "command to execute" $vol_name"."$i
}
Works perfectly! I can now create X volumes named VolName.1, VolName.2 and so on, where X is the number of volumes entered by the user.
You can use the range operator (..):
foreach ($i in 1..$num_vols) { ... }
Or the range operator like this:
1..$num_vols | ForEach-Object { ... }
Or you can use a traditional for loop:
for ($i = 0; $i -lt $num_vols; $i++) { ... }
Is it possible to have the arguments in a daily scheduled job increment each day? I have an array of values and I want to run one job per value, spread out so that the jobs occur once a day.
This is my code so far:
$dailyTrigger = New-JobTrigger -Daily -At "11:00 AM"
$option = New-ScheduledJobOption -StartIfOnBattery -StartIfIdle -WakeToRun -IdleTimeout "10:00:00"
Register-ScheduledJob -Name $JobName -FilePath $ScriptToRun -Trigger $dailyTrigger -ScheduledJobOption $option
I was planning on using Register-ScheduledJob's -ArgumentList parameter, but from what I've seen there would be no way to pass a different value to each daily instance of the job. Is there some way I could store which element of the array is next so that the jobs could reach it?
One possible way to persist a counter between job runs is to store the value in the registry:
Set-ItemProperty -Path 'HKCU:\foo' -Name 'Job1' -Value ($counter + 1)
and read it on the next run:
$counter = (Get-ItemProperty -Path 'HKCU:\foo').Job1
TL;DR: Is there a way for a PowerShell script calling Microsoft.AnalysisServices functions to process multiple cube structures concurrently?
I have a Microsoft SSAS cube that needs several measure groups processed before the rest of the cube is processed later in the job plan. I have created a PowerShell script that enumerates the measure groups to process and calls measureGroup.Process('ProcessFull') from the Microsoft.AnalysisServices namespace. This works to process the measure group, dimension, et.al.
However, processing the measure groups in this manner doesn't allows SQL Server 2014 to parallelize the processing. A cube that takes on average 2 hours to fully process was running for 7 hours before we killed it.
Is there a way in PowerShell to batch the processes so that they are sent at the same time to the cube? If this could be done, it would allow the server to do concurrent processing instead of one object at a time.
I have taken a look through the documentation on the MSDN as well as consulted Google, but was unable to find an answer. Thanks!
You should check this method in PowerShell: http://ssas-info.com/analysis-services-scripts/1238-powershell-script-to-process-all-dimensions-and-cubes-in-one-db-limiting-workload
Have a look at powershell jobs:
Background Jobs
Here's a quick example that you could adapt to run your measuregroup processing:
$cmd = {
param($a)
Write-Host $a
}
$list = #("a","b","c")
$list | ForEach-Object {
Start-Job -ScriptBlock $cmd -ArgumentList $_
}
It's quite simple, you define your script block in the $cmd variable, this is where you would put your logic around processing the measure group. The $list variable could contain a list of the measure groups to process.
You then start a job for each item in the list, which executes the code in the script block, passing through the item in the list as a parameter. In this example it simply prints out the parameter that you passed in. You can of course pass in as many parameters as you like.
To get the results, you can use the Get-Job cmdlet to check the status of the jobs and Receive-Job to get the output. Remove-Job can then be used to clear finished jobs from the queue.
The following command run after the code above will get the results of all the jobs (in this case just the a,b,c that we passed in and then will remove it from the queue:
Get-Job | ForEach-Object { Receive-Job $_.Id; Remove-Job $_.Id }