Powershell Duplicate file Cleanup for Plex Camera Uploads

Some time ago, my wife had her phone stolen and we had not setup any sort of backup for the pictures, so a good number of photos and videos of our kids were lost that day. We now both use Google devices, so we have automatic backup to Drive for free, but I didn’t want to rely only on that.

Enter Plex Camera Upload

We use Plex a lot in our house – one of the major features we use is the automatic upload of photos to a library – this library is shared with parents and in-laws and it also provides a simple way to backup photos or videos taken by our phones. Initially I had some issues with the feature since I was storing files on an SMB share mounted to Linux – and when that happened, I found I needed to restart the upload to get things working.

Since then, things have just worked. Even switching to new devices, once the Plex app is installed and configured for upload, it just works. Until it doesn’t. A recent update broke the upload feature.

It has since been fixed, but in the meantime, I went back and disabled\reset the camera upload capability to try to get it working. Needless to say, I have some duplicate files in my Camera Uploads directory. I think there is some validation in the Plex app for previously updated files, but not much. I had racked up about 4000+ duplicate files and these are not small files, so there was several GB of space savings to be reclaimed. If only there were an automated way to do this…

Powershell to the Rescue

The following script will do the following:

  • Prompt the user for a path to the Camera Uploads directory
  • Parse all of the image and video files in the directory and group them by file size where each group has more than 1 file
  • Generate a SHA1 hash of each of the files to guarantee that they are actually duplicates
  • Delete hyphenated or malformed date files

A few notes:

  • I found -1, -2 and -3 duplicate files, there could potentially be more in your case, but the script would need to be modified to accommodate for this
  • Nothing is deleted unless you pass the -delete parameter to the script and confirm
  • The directory selection window likes to pop up behind the ISE window…making it look like the script is hung – check the log file for activity in your MyDocs
<#
.SYNOPSIS
Script to remove duplicate photos from the Plex Mobile Uploads directory

.DESCRIPTION
This script will enumerate all files in the Mobile Uploads directory, then find files that are the same size. A SHA1 hash is generated for each of these files to verify that they
are the same image prior to being deleted. Files that have incorrect date formats (1970-01-01) or are hyphenated take precedence to be deleted, unless all duplicate files are formatted
incorrectly, then the first file found to be duplicate will be deleted.

.NOTES
Script only looks for -1, -2 or -3 JPG files or -1 and -2 MP4 files
More file types can be added to the Get-ChileItem -Include section if needed

.PARAMETER delete
If the 'delete' parameter is used, the script will delete files once hashed duplicates have been found

#>
param(
    [parameter(Mandatory=$false)]
    [switch]$delete=$false
    )

$logFile = "$([Environment]::GetFolderPath("mydocuments"))\DupeDeleteLog.txt"

# Write to a log file in the current users MyDocs directory
Function Write-Log
{
    Param([string]$logStr)
    Add-Content $logFile -Value $logStr
}

# User select location to find duplicate files
Function Get-FolderName 
{ 
    Add-Type -AssemblyName System.Windows.Forms 
    $FolderBrowser = New-Object System.Windows.Forms.FolderBrowserDialog 
    [void]$FolderBrowser.ShowDialog() 
    $FolderBrowser.SelectedPath 
} 

Write-Log "$(Get-Date)"
Write-Log "Waiting for input from user to select working directory..."
$mypath = Get-FolderName 

# Initial log file notes
If($delete){Write-Log "WARNING: File deletion will occur!"}
Write-Log "Checking for duplicates in $($mypath)"
Write-Log "Finding same size files and hashing them. This will take some time...."

$dupeHash = foreach ($i in (Get-ChildItem -path $mypath -Recurse -Include "*.jpg","*.jpeg","*.mp4" | ? {( ! $_.ISPScontainer)} | Group Length | ? {$_.Count -gt 1} | Select -Expand Group | Select FullName, Length)){Get-FileHash -Path $i.Fullname -Algorithm SHA1}
$dupeHashGrouped = $dupeHash | Group Hash | ? {$_.Count -gt 1}

# Set confirm, if $delete param is not set, no delete will occur
If(!($delete)){$delConfirm = "Y"}
If($delete)
{
    Write-Log "Waiting for input from user to confirm delete..."
    $delConfirm = Read-Host "WARNING: This script will now DELETE FILES. Press 'Y' to confirm you want to DELETE, or any other key to cancel"
}

If (($delConfirm -eq 'y') -or ($delConfirm -eq 'Y'))
{
    foreach ($dhGroup in $dupeHashGrouped)
    {
        Write-Log "Current file hash: $($dhGroup.Group[0].Hash)"
        Write-Log "Matching files: $($dhGroup.Count)"
        $goodFile = $false
        $i = 1
        foreach($matchFile in $dhGroup.Group.Path)
        {
            If($i -eq $dhGroup.Count -and (!($goodFile)))
            {
                #Last file, no good found...keeping
                Write-Log "Last file in group. Keeping: $($matchFile)"
            }
            ElseIf($matchFile -like "*1970*")
            {
                Write-Log "File in group contains invalid date 1970: $($matchFile) - will be deleted."
                If($delete){Remove-Item -Path $matchFile -Confirm:$false}
            }
            #modify if more duplicated hyphens are found
            ElseIf($matchFile -like "*-1.jpg" -or $matchFile -like "*-2.jpg" -or $matchFile -like "*-3.jpg" -or $matchFile -like "*-1.mp4" -or $matchFile -like "*-2.mp4")
            {
                Write-Log "File in group is hyphenated: $($matchFile) - will be deleted."
                If($delete){Remove-Item -Path $matchFile -Confirm:$false}
            }
            Else
            {
                If($goodFile)
                {
                    #Already found valid file in group
                    Write-Log "Deleting this file: $($matchFile) already have good file."
                    If($delete){Remove-Item -Path $matchFile -Confirm:$false}
                }
                Else
                {
                    #Don't have good file in group; keep
                    Write-Log "Keeping: $($matchFile)"
                    $goodFile = $true
                }
            }
            $i++
        }
        Write-Log " "
    }
}
Else
{
    Write-Log "Deletion cancelled by user"
}

The script will also be posted in my PowerShell library here.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.