It has been a while. Took a lot of testing and rebuilding but I think I have pinpointed the issue now.
I applied the tunables from powertop and that seems to be the issue, especially the 'Enable SATA link power management', when those are applied it takes a couple of days and/or disk sleep cycles to 'fail' the disk.