Today at 16:23, a scheduled disk scan in Sonarr reported that over 850 episodes of TV shows had disappeared from my system without warning.
There are no log entries at that time that would indicate anything had gone wrong.
Feb 25 16:05:50 Global-Dynamics sshd[24685]: Accepted publickey for root from 192.168.0.2 port 51256 ssh2: ED25519 SHA256:5b6Ebu2l2lANTclMchJAJk4XviWfjlgDJt7ngCuXszc Feb 25 16:05:50 Global-Dynamics sshd[24685]: Received disconnect from 192.168.0.2 port 51256:11: disconnected by user Feb 25 16:05:50 Global-Dynamics sshd[24685]: Disconnected from user root 192.168.0.2 port 51256 Feb 25 17:00:01 Global-Dynamics root: Parity Check / rebuild in progress. Not running mover Feb 25 17:00:09 Global-Dynamics sshd[21305]: Accepted publickey for root from 192.168.0.2 port 59744 ssh2: ED25519 SHA256:5b6Ebu2l2lANTclMchJAJk4XviWfjlgDJt7ngCuXszc Feb 25 17:00:09 Global-Dynamics sshd[21305]: Received disconnect from 192.168.0.2 port 59744:11: disconnected by user Feb 25 17:00:09 Global-Dynamics sshd[21305]: Disconnected from user root 192.168.0.2 port 59744
One of the missing files is an episode that I watched just last night, and the overall used space on my array has not changed at all. That would seem to indicate to me that nothing was actually deleted. I paused the parity check and rebooted the system to see if there was a software glitch that would bring them all back. When it came back up, the array would not autostart due to "changed configuration". The only thing I did was disconnect two drives that were not assigned to the array for reasons detailed below. I manually started the array, but the docker service was also disabled and all of its settings were reverted to defaults. I reset those to where they should be and all appears well now.
The good news is that the missing files did reappear. I obviously can't confirm all of them but enough have that it's likely they all did. It seems that all of the missing files were on disks 6 & 7 which mysteriously dropped out but without reporting any errors to the OS. From this, I'm able to deduce that the files all went missing before 14:50, because the Windows 10 ISO was missing at that time while I was trying to spin up a new VM (at that time, I just thought I had never redownloaded Win10 after a user error led to wiping the isos share last year).
What seems to be the reason:
Earlier this week I obtained a new drive to replace the parity drive and make the old parity drive a data drive. As part of this process I also removed the drives formerly disk 6 & 7 from the array. The approximate timeline of my steps is as follows:
Sat Feb 17
- Installed the new drive and started preclear on it as a stress test.
Mon Feb 19
- Stopped the preclear and moved the new drive into the parity slot of the array, moved the old parity drive to disk10 slot.
Tue Feb 20
- Parity Sync finished, began transferring data from disks 6 & 7 to disk 10
Wed Feb 21
- Noticed anomalies in the logs related to the new parity disk, see
Sat Feb 24
- Realized I had erred in how I was transferring data and deleted most of disk10 data
- Used the new config option to remove the parity disk from the array and restart it
- Copied the data from disks 6 & 7 to disk 10 using the proper method
- After finishing this transfer, used the new config option again to
- Add the parity disk back to the array
- Change disk10 to disk1
- Remove disk6 & disk7
- Reorder other disks in descending order of capacity
Sun Feb 25
- Physically removed former disk7 from the machine and replaced it with a failing NTFS disk belonging to my brother to attempt data recovery for him
- Wiped former disk6 and reformatted it as NTFS
- Mounted both disks
- Discovered the disk failure was extensive and began spinning up a Windows VM to use Windows only tools to attempt further steps
I took a break from working at this point, and it was later today that I noticed the missing files. While the system was rebooting I removed the former disk6 and the new failed disk, as I suspected they might be somehow connected to the issue. Based on the missing files all being on the new disks 6 & 7, this does seem to be the case although it's unclear if removing them was necessary.
It seems clear that the step during which I mounted the old disk6 and a new disk in the place of the old disk7 caused something very bad to happen. Luckily, it doesn't appear that I've suffered any permanent data loss. However, the root cause of this problem needs to be isolated and corrected.
Recommended Comments
There are no comments to display.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.