Zipping older files with powershell - file

I have the below Powershell script which tries to archive the logs.
1st step is to move all the files that contain Spotfire.Dxp...* string the ProjectLogsDir directory.
2nd step is identify the Spotfire.Dxp...* files in RotatedLogsDir directory that are older than 60 days and put them in a zip file together in a ArchivedLogsDir directory
3rd step is to delete zipped file older than 120 days.
here 2nd step function isn't working with error
E:\TIBCO\logsArchival\rotatedDir\C0005749_2014-05-09-12-53-47_Spotfire.Dxp.Web.231.log E:\TIBCO\logsArchival\rotatedDir\C0005749_2014-05-09-12-53-47_Spotfire.Dxp.Web.232.log: WARNING: The filename, directory name, or volume label syntax is incorrect.
$sysname=$env:COMPUTERNAME
$Date = Get-Date
$Now = Get-Date -format yyyy-MM-dd-HH-mm-ss
$host_date=$sysname +"_"+ $Now
$RotateDays = "60"
$ArchiveDays="120"
$ProjectLogsDir = "E:\TIBCO\*\6.0.0\Logfiles"
$RotatedLogsDir = "E:\TIBCO\logsArchival\rotatedDir"
$ArchivedLogsDir= "E:\TIBCO\logsArchival\archiveDir"
$psLogsDir= "E:\TIBCO\logsArchival\shLogsDir"
$LastRotate = $Date.AddDays(-$RotateDays)
$LastArchive = $Date.AddDays(-$ArchiveDays)
$RenameLogFiles = Get-Childitem $ProjectLogsDir -include Spotfire.Dxp.*.*.* -exclude spotfire.Dxp.web.KeepAlive.* -Recurse
$RenameLogFiles
$RenameLogFiles | Where-Object {!$_.PSIsContainer} | Rename-Item -NewName { $host_date +"_"+ $_.Name.Replace(' ','_') };
$RotatedLogFiles = Get-Childitem $ProjectLogsDir -include *_Spotfire.Dxp.*.*.* -Recurse
$RotatedLogFiles
$RotatedLogFiles | move-item -destination "$RotatedLogsDir"
$ZippedLogFiles = Get-Childitem $RotatedLogsDir -include *_Spotfire.Dxp.*.*.* -Recurse | Where {$_.LastWriteTime -le "$LastRotate"}
$ZippedLogFiles
function create-7zip([String] $aDirectory, [String] $aZipfile) {
[string]$pathToZipExe = "C:\Program Files\7-zip\7z.exe";
[Array]$arguments = "a", "-tzip", "$aZipfile", "$aDirectory";
& $pathToZipExe $arguments;
}
create-7zip "$ZippedLogFiles" "$ArchivedLogsDir\$host_date.zip"
$ZippedLogFiles | Remove-Item -Force
$DeleteZippedFiles = Get-Childitem $ArchivedLogsDir\*.zip -Recurse | Where {$_.LastWriteTime -le "$LastArchive"}
$DeleteZippedFiles
$DeleteZippedFiles | Remove-Item -Force
$DeletePsFiles = Get-Childitem $psLogsDir\*.log -Recurse | Where {$_.LastWriteTime -le "$LastRotate"}
$DeletePsFiles
$DeletePsFiles | Remove-Item -Force
Please provide the help here to get the files zipped.

The issue is that you are calling your 7-Zip function incorrectly. Look at what the function takes:
function create-7zip([String] $aDirectory, [String] $aZipfile) {
It has 2 strings as parameters. Then look at what you are calling it with:
create-7zip "$ZippedLogFiles" "$ArchivedLogsDir\$host_date.zip"
"$ZippedLogFiles" was defined just before this by running $ZippedLogFiles = Get-Childitem $RotatedLogsDir with a few parameters to filter the results. So that right there is an array of FileInfo objects... not a string. That's the issue, is that you are not calling the function correctly.
What you really want to include is "$RotatedLogsDir\*_Spotfire.Dxp*.*" so try calling it with that instead of "$ZippedLogFiles".
Edit: Comment moved to answer for zipping only files over 60 days.
You can use built in .Net calls to archive things and not have to use 7-Zip. Even easier in my opinion would be to install the PowerShell Community Extensions and use their Write-Zip cmdlet like this:
$ZippedLogFiles = Get-Childitem $RotatedLogsDir -include *_Spotfire.Dxp.*.*.* -Recurse | Where {$_.LastWriteTime -le "$LastRotate"}
$ZippedLogFiles
$ZippedLogFiles | Write-Zip "$ArchivedLogsDir\$host_date.zip"

Related

Powershell read file for filename and path

I have the following script which runs on .zip files in a directory which have a whole directory structure with many files. These files then have 7zip run on them to extract them and then .eml is added to the extracted file.
& "c:\program files\7-zip\7z.exe" x c:\TestZip -r -oC:\TestRestore 2> c:\TestLog\ziplog.txt
& "c:\program files\7-zip\7z.exe" x c:\TestRestore -r -aos -oc:\TestExtract 2> c:\TestLog\sevenzip.txt
gci -path "c:\TestExtract" -file | rename-item -newname {$PSItem.name + ".eml"}
My problem is that out of these files sometimes the final extraction cannot be done by 7zip as it does not see it as an archive. I have found that these particular files if I just put .eml on them they are accessible as emails. So when the archive fails to extract I write the output to the sevenzip.txt file.
What I need help with is how do I read this file to get the filenames and place them in a directory so I can add the .eml extension.
An example of the output in the sevenzip.txt file is as follows
ERROR: c:\TestRestore\0\0\54\3925ccb78f80d28b7569b6759554d.0_4011
Can not open the file as archive
ERROR: c:\TestRestore\0\0\5b\6fa7acb219dec5d9e55d4eadd6eb1.0_3958
Can not open the file as archive
Any help would be greatly appreciated on how to do this.
Sorry for all the comments but I am working on this
$SourceFile = 'c:\testlog\sevenzip.txt'
$DestinationFile = 'c:\testlog\testlogextractnew.txt'
$Pattern = 'c:\\TestRestore\\' (Get-Content $SourceFile) |
% {if ($_ -match $Pattern){$_}} |
Set-Content $DestinationFile (Get-Content $DestinationFile).replace('ERROR: ', '') |
Set-Content $DestinationFile (Get-Content$DestinationFile).replace('7z.exe : ', '') |
Set-Content $DestinationFile
If there are no other errors, then you can pick those filenames out of the file with a regex pattern match:
$filesToFix = Select-String -Pattern "ERROR: (.*)" -LiteralPath 'c:\testlog\sevenzip.txt' |
ForEach-Object {$_.Matches.Groups[1].Value}
And probably rename them with
$filesToFix | ForEach-Object {
Rename-Item -LiteralPath $_ -NewName {"$_.eml"}
}
If there might be other errors and you only want these, you'll need more work on the matching:
$fileContent = Get-Content -LiteralPath 'c:\testlog\sevenzip.txt' -Raw
$filesToFix = [regex]::Matches($fileContent,
"ERROR: (.*)\nCan not open the file as archive",
'Multiline') |
ForEach-Object {$_.Groups[1].Value}
This is what I ended going with it is in the comments because I didn't know how to format but have worked it out now so hopefully this makes it easier to understand in case it is useful.
This section creates a list of all the files that failed to extract with their directory location. For use in next section.
$SourceFile = 'c:\testlog\sevenzip.txt' $DestinationFile = c:\testlog\testlogextractnew.txt'
$Pattern = 'c:\\TestRestore\\'
(Get-Content $SourceFile) | % {if ($_ -match $Pattern){$_}} | Set-Content $DestinationFile
(Get-Content $DestinationFile).replace('ERROR: ', '') | Set-Content $DestinationFile
(Get-Content $DestinationFile).replace('7z.exe : ', '') | Set-Content $DestinationFile
This is just some result checking
Get-Content C:\testlog\testlogextractnew.txt | Measure-Object –Line > c:\testlog\numberoffilesthatfailed.txt
This copies all files listed in the txt file to another directory
c:\testlog\testlogextractnew.txt | copy-item -destination "c:\TestCopy"
This renames the current extensions to .eml
gci -path "C:\TestCopy" -file | rename-item -newname { [io.path]::changeextension($_.name, "eml")}

Copy-Item with exclude files and folders in powershell

I have a question. I want to copy content of one directory into another and made some progress. But, at the moment I'm stuck because if I can define array of files to exclude I have a problem with array of folders. So the arrays looks like:
$excludeFiles = #('app.config', 'file.exe')
$excludeFolders = #('Logs', 'Reports', 'Backup', 'Output')
It works when there is only one item in $excludeFolders array, but if I add multiple items, it copies all folders without excluding them.
Script I have:
Get-ChildItem -Path $binSolutionFolder -Recurse -Exclude $excludeFiles |
where { $excludeFolders -eq $null -or $_.FullName.Replace($binSolutionFolder, "") -notmatch $excludeFolders } |
Copy-Item -Destination {
if ($_.PSIsContainer) {
Join-Path $deployFolderDir $_.Parent.FullName.Substring($binSolutionFolder.length -1)
}
else {
Join-Path $deployFolderDir $_.FullName.Substring($binSolutionFolder.length -1)
}
} -Force -Exclude $excludeFiles
Where $binSolutionFolder is source and $deployFolderDir is target.
Files work fine, but with folders I've run out of ideas.
-notmatch uses a regex-pattern and not a collection. To match against a collection of words you could use -notin $excludedfolders, but if the path includes 2+ level of folders or just a simple \ then the test would fail.
I would use -notmatch, but first create a regex-pattern that checks all folders at ones. Ex:
$excludeFiles = #('app.config', 'file.exe')
$excludeFolders = #('Logs', 'Reports', 'Backup', 'Output','Desktop')
$excludeFoldersRegex = $excludeFolders -join '|'
Get-ChildItem -Path $binSolutionFolder -Recurse -Exclude $excludeFiles |
where { $_.FullName.Replace($binSolutionFolder, "") -notmatch $excludeFoldersRegex } |
.....

Comparing files within folders and moving files not found in one folder to a different location

I am working on folders that contain many types and sizes of files within them. What I want to do is move files that are not contained in one folder into a new folder. I have embedded a picture link that helps illustrate what I am aiming to do.
I would like test123.pdf to be moved to a new location because it's not contained within the other folder. Below I have some code that simply compares the contents of each folder and outputs which file is out of place. I have been researching some things online, but have come up empty. Can anyone help me proceed?
Disclaimer: I know the path is wrong, but I can't show it for security reasons.
$Folder1 = Get-ChildItem -Recurse -path "Enter Path here"
$Folder2 = Get-ChildItem -Recurse -path "Enter the Path here"
Compare-Object -ReferenceObject $Folder1 -DifferenceObject $Folder2
It sounds like you want to compare the contents of two folders, ID the file(s) that are not present in both folders, and then move them to a third folder. To accomplish this, you can define your 2 paths in variables, compare the contents of the folders, grab the full path names of the different items, and then move them to the new destination.
$path1 = yourfirstpath
$path2 = yoursecondpathforcomparing
$path3 = yourdestinationpath
diff (ls $path1 -recurse) (ls $path2 -recurse) | ForEach {$_.InputObject.FullName} | Move-Item -Destination $path3
diff = Compare-Object , ls = Get-ChildItem
This will do the work, any file that is not in both Folder1 & Folder2 will be moved to Folder 3
$Folder1 = 'C:\folder1'
$Folder2 = 'C:\folder2'
$Folder3 = 'C:\folder3'
foreach ($file in Get-ChildItem -Recurse $Folder1)
{
if (-not(Test-Path "$Folder2\$file"))
{
Move-Item $file.FullName $Folder3
}
}
This will compare by file name, and maintain the directory structure the extra files were found in, and handle the recursion properly. The output of the code demonstrates the directory structure I tested with.
# be explict here to ensure everything is lowercase
# so when this is cut and pasted, it does not break
# when you enter the real path
$d1 = "d:\test\one".ToLower()
$d2 = "d:\test\two".ToLower()
$f1 = gci -Recurse $d1 -File | % {$_.FullName.ToLower()}
$f2 = gci -Recurse $d2 -File | % {$_.FullName.ToLower()}
"f1"
$f1
"`nf2"
$f2
$same = #()
$extra = #()
foreach ($f in $f2)
{
$f2tof1path = $f.Replace($d2, $d1)
if ($f1.Contains($f2tof1path) -eq $false)
{
$extra += $f
}
else {
$same += $f
}
}
"`nSame"
$same
"`nExtra"
$extra
"`nMOVE"
$folder3 = "d:\test\three"
foreach ($f in $extra)
{
# move files somewhere, create dir if not exists
$dest = $f.Replace($d2,$folder3)
$destdir = $(Split-Path -Parent -Path $dest)
if (!(Test-Path $destdir))
{
# remove quotes to do new-item, keep them to show what it will do
"New-Item -ItemType Directory -Force -Path $destdir"
}
# remove quotes to do move-item, keep them to show what it will do
"Move-Item $f $dest"
}
Output
f1
d:\test\one\1.txt
d:\test\one\2.txt
d:\test\one\sub\1.txt
d:\test\one\sub\2.txt
f2
d:\test\two\1.txt
d:\test\two\2.txt
d:\test\two\3.txt
d:\test\two\sub\1.txt
d:\test\two\sub\2.txt
d:\test\two\sub\3.txt
Same
d:\test\two\1.txt
d:\test\two\2.txt
d:\test\two\sub\1.txt
d:\test\two\sub\2.txt
Extra
d:\test\two\3.txt
d:\test\two\sub\3.txt
MOVE
New-Item -ItemType Directory -Force -Path d:\test\three
Move-Item d:\test\two\3.txt d:\test\three\3.txt
New-Item -ItemType Directory -Force -Path d:\test\three\sub
Move-Item d:\test\two\sub\3.txt d:\test\three\sub\3.txt
If this does not solve your problem, it should get you 99% there. Play around with the code, have fun, and good luck.

powershell getting the remote directory name and filename to put in destination filename

I have this bit of powershell script but i can't get the $DirectoryName to behave as expected.
1,2,3 |
foreach {
$count = $_;
$x = gci -Path \\myserver-web$count\d$\IISLogs\ -include *.log -recurse
$x | Copy-Item -Destination D:\ServerLogsAndBackups\IIS\w$count\$_.DirectoryName_$_.Name -whatIf
}
When I run this though I get
What if: Performing operation "Copy File" on Target "Item: \myserver-web1\d$\IISLogs\W3SVC1165836668\ex101224.log Destination: D:\ServerLogsAndBackups\IIS\w1\1.DirectoryName_1.Name".
What I want it to be is
W3SVC1165836668_ex101206.log
where my directory structure is like:
\\myserver-web1\d$\IISLogs\W3SVC1165836668
\\myserver-web1\d$\IISLogs\W3SVC1165837451
\\myserver-web1\d$\IISLogs\W3SVC1165836966
\\myserver-web1\d$\IISLogs\W3SVC1165812365
with files called ex101206.log in each folder
Cheers
You need to evaluate the $_.Directoryname_$_.Name part. Like so,
$x | Copy-Item -Destination $(D:\ServerLogsAndBackups\IIS\w$count\$_.DirectoryName_$_.Name) -whatIf
1,2,3 |
foreach {
$count = $_;
gci -Path \\myserver-web$count\d$\IISLogs\ -include *.log -recurse | % { $dirName = $_.directoryname.Substring($_.directoryname.LastIndexOf("\")+1); $logname = $_.Name; $_ | Copy-Item -Destination $("D:\ServerLogsAndBackups\IIS\w"+$count+"\"+$dirname+"_"+$logName) -whatif }
}

How to delete empty subfolders with PowerShell?

I have a share that is a "junk drawer" for end-users. They are able to create folders and subfolders as they see fit. I need to implement a script to delete files created more than 31 days old.
I have that started with Powershell. I need to follow up the file deletion script by deleting subfolders that are now empty. Because of the nesting of subfolders, I need to avoid deleting a subfolder that is empty of files, but has a subfolder below it that contains a file.
For example:
FILE3a is 10 days old. FILE3 is 45 days old.
I want to clean up the structure removing files older than 30 days, and delete empty subfolders.
C:\Junk\subfolder1a\subfolder2a\FILE3a
C:\Junk\subfolder1a\subfolder2a\subfolder3a
C:\Junk\subfolder1a\subfolder2B\FILE3b
Desired result:
Delete: FILE3b, subfolder2B & subfolder3a.
Leave: subfolder1a, subfolder2a, and FILE3a.
I can recursively clean up the files. How do I clean up the subfolders without deleting subfolder1a? (The "Junk" folder will always remain.)
I would do this in two passes - deleting the old files first and then the empty dirs:
Get-ChildItem -recurse | Where {!$_.PSIsContainer -and `
$_.LastWriteTime -lt (get-date).AddDays(-31)} | Remove-Item -whatif
Get-ChildItem -recurse | Where {$_.PSIsContainer -and `
#(Get-ChildItem -Lit $_.Fullname -r | Where {!$_.PSIsContainer}).Length -eq 0} |
Remove-Item -recurse -whatif
This type of operation demos the power of nested pipelines in PowerShell which the second set of commands demonstrates. It uses a nested pipeline to recursively determine if any directory has zero files under it.
In the spirit of the first answer, here is the shortest way to delete the empty directories:
ls -recurse | where {!#(ls -force $_.fullname)} | rm -whatif
The -force flag is needed for the cases when the directories have hidden folders, like .svn
This will sort subdirectories before parent directories working around the empty nested directory problem.
dir -Directory -Recurse |
%{ $_.FullName} |
sort -Descending |
where { !#(ls -force $_) } |
rm -WhatIf
Adding on to the last one:
while (Get-ChildItem $StartingPoint -recurse | where {!#(Get-ChildItem -force $_.fullname)} | Test-Path) {
Get-ChildItem $StartingPoint -recurse | where {!#(Get-ChildItem -force $_.fullname)} | Remove-Item
}
This will make it complete where it will continue searching to remove any empty folders under the $StartingPoint
I needed some enterprise-friendly features. Here is my take.
I started with code from other answers, then added a JSON file with original folder list (including file count per folder). Removed the empty directories and log those.
https://gist.github.com/yzorg/e92c5eb60e97b1d6381b
param (
[switch]$Clear
)
# if you want to reload a previous file list
#$stat = ConvertFrom-Json (gc dir-cleanup-filecount-by-directory.json -join "`n")
if ($Clear) {
$stat = #()
} elseif ($stat.Count -ne 0 -and (-not "$($stat[0].DirPath)".StartsWith($PWD.ProviderPath))) {
Write-Warning "Path changed, clearing cached file list."
Read-Host -Prompt 'Press -Enter-'
$stat = #()
}
$lineCount = 0
if ($stat.Count -eq 0) {
$stat = gci -Recurse -Directory | %{ # -Exclude 'Visual Studio 2013' # test in 'Documents' folder
if (++$lineCount % 100 -eq 0) { Write-Warning "file count $lineCount" }
New-Object psobject -Property #{
DirPath=$_.FullName;
DirPathLength=$_.FullName.Length;
FileCount=($_ | gci -Force -File).Count;
DirCount=($_ | gci -Force -Directory).Count
}
}
$stat | ConvertTo-Json | Out-File dir-cleanup-filecount-by-directory.json -Verbose
}
$delelteListTxt = 'dir-cleanup-emptydirs-{0}-{1}.txt' -f ((date -f s) -replace '[-:]','' -replace 'T','_'),$env:USERNAME
$stat |
? FileCount -eq 0 |
sort -property #{Expression="DirPathLength";Descending=$true}, #{Expression="DirPath";Descending=$false} |
select -ExpandProperty DirPath | #-First 10 |
?{ #(gci $_ -Force).Count -eq 0 } | %{
Remove-Item $_ -Verbose # -WhatIf # uncomment to see the first pass of folders to be cleaned**
$_ | Out-File -Append -Encoding utf8 $delelteListTxt
sleep 0.1
}
# ** - The list you'll see from -WhatIf isn't a complete list because parent folders
# might also qualify after the first level is cleaned. The -WhatIf list will
# show correct breath, which is what I want to see before running the command.
To remove files older than 30 days:
get-childitem -recurse |
? {$_.GetType() -match "FileInfo"} |
?{ $_.LastWriteTime -lt [datetime]::now.adddays(-30) } |
rm -whatif
(Just remove the -whatif to actually perform.)
Follow up with:
get-childitem -recurse |
? {$_.GetType() -match "DirectoryInfo"} |
?{ $_.GetFiles().Count -eq 0 -and $_.GetDirectories().Count -eq 0 } |
rm -whatif
This worked for me.
$limit = (Get-Date).AddDays(-15)
$path = "C:\Some\Path"
Delete files older than the $limit:
Get-ChildItem -Path $path -Recurse -Force | Where-Object { !$_.PSIsContainer -and $_.CreationTime -lt $limit } | Remove-Item -Force
Delete any empty directories left behind after deleting the old files:
Get-ChildItem -Path $path -Recurse -Force | Where-Object { $_.PSIsContainer -and (Get-ChildItem -Path $_.FullName -Recurse -Force | Where-Object { !$_.PSIsContainer }) -eq $null } | Remove-Item -Force -Recurse

Resources