I see it's been brought up a few times earlier, but I wanted to chime in and mention that I also have the issue where media files aren't being deleted from the cache, even if they haven't been watched for weeks. Something appears to cause the accessed time to update at 2:30 AM every night. I'm not sure what it is, I don't have any user scripts or Plex scheduled tasks that should be running at that time.
The mode I use is "Media Server: Only", so I think a solution that would work 100% of the time for that setting would be to store the access times from the media player playback itself rather than going off of the file system access times.
In any case, it's a really useful plugin. Thanks a lot for making it, I appreciate it!