Hi all, looking for some assistance if possible. Installed a new 16TB drive to the Array a couple days ago. Formatted, waited for the clear and thenadded to the array and started adding media to it once it was online.  Today, it's suddenly unmountable.  Drive doesn't show any errors, it just kinda disappeared. 

 

Attaching logs if anyone knows how to dig in.  Thank you!

tower900-diagnostics-20241214-2148.zip

7 hours ago, scythe944 said:

Formatted, waited for the clear

Are you sure? Format is only after the clear.

 

Post the output from:

 

fdisk -l /dev/sde

 

Sorry, yes I think I did the clear first, then format / parity check for 20 hours.  After that, I was finally seeing some files get placed on the new drive.

Thanks, it shows this as the output:
Disk /dev/sde: 14.55 TiB, 16000900661248 bytes, 31251759104 sectors
Disk model: ST16000NT001-3LV
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

 

Edited December 15, 20241 yr by scythe944

Running an extended SMART test on the drive now. I suppose I lost whatever was put on there over the last day or so, but not a huge deal. I'm not sure if I should pull the drive and put it in my system to run a HDD test or just format it again and add it back to the array. I guess there could be something wrong with the drive, but it's brand new straight from Seagate's store as of last Friday.  Not seeing any errors yet, just unmountable.

The drive doesn't have a partition, suggesting it was never formatted, but if you think it was, you can try unassigning the disk and starting the array, if there was one and for some reason it was removed, parity should emulate it, note that doing this will degrade the array.

I saw it add two or three movies to this disk.  I was watching something on Plex on Friday night, and after I was done watching the movie, I went back to the main screen of Plex and saw that the movies that I recently added were gone. That's when I went to my computer and saw that the disk was unmountable.

 

I don't want to degrade the array, I'd rather just get whatever is wrong with this disk fixed.  The extended drive test finished at some point over night last night.   No errors were found.  Should I just re-format this disk and try to see if it works again?  I'm not sure why it would have had partitions at some point and lose them out of no where?

After the Extended self-test was completed, I decided to try to restart the server to see if it magically fixed itself.  It did not, so I started the array in Maintenance mode and did a File System Check on the disk.  The output is below.  Bunch of CRC errors.  It does seem that it found some of the folders that were created previously on the drive.  I'm thinking I should shut down the system, bring the disk to my computer, do a test with SeaTools and see if there are any issues with the disk.  Or maybe find a drive stress test tool before trying to add it back. Maybe the drive is faulty already.

Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Metadata CRC error detected at 0x43e204, xfs_agfl block 0x3/0x200 agfl has bad CRC for ag 0 bad agbno 0 in agfl, agno 0 bad agbno 0 in agfl, agno 0 bad agbno 0 in agfl, agno 0 bad agbno 0 in agfl, agno 0 Superblock has bad magic number 0x0. Not an XFS filesystem? Superblock has bad magic number 0x0. Not an XFS filesystem? Metadata CRC error detected at 0x43e660, xfs_agf block 0x2ffffffd1/0x200Metadata CRC error detected at 0x43e660, xfs_agf block 0x27fffffd9/0x200 agf has bad CRC for ag 5 agf has bad CRC for ag 6 Metadata CRC error detected at 0x469f20, xfs_agi block 0x27fffffda/0x200 agi has bad CRC for ag 5 bad on-disk superblock 5 - bad magic number primary/secondary superblock 5 conflict - AG superblock geometry info conflicts with filesystem geometry bad log/data device sector size fields in superblock 5 Superblock has bad magic number 0x0. Not an XFS filesystem? would fix incompat feature mismatch in AG 5 super, 0xb != 0x0 would fix ro compat feature mismatch in AG 5 super, 0xd != 0x0 Metadata CRC error detected at 0x469f20, xfs_agi block 0x2ffffffd2/0x200 Superblock has bad magic number 0x0. Not an XFS filesystem?bad magic # 0x0 for agf 5 agi has bad CRC for ag 6 bad version # 0 for agf 5 bad sequence # 0 for agf 5 bad length 0 for agf 5, should be 268435455 bad on-disk superblock 6 - bad magic number Metadata CRC error detected at 0x43e660, xfs_agf block 0x1ffffffe1/0x200Metadata CRC error detected at 0x469f20, xfs_agi block 0x7ffffffa/0x200 agi has bad CRC for ag 1 bad on-disk superblock 1 - bad magic number primary/secondary superblock 1 conflict - AG superblock geometry info conflicts with filesystem geometry Metadata CRC error detected at 0x47494d, xfs_refcountbt block 0x28/0x1000primary/secondary superblock 6 conflict - AG superblock geometry info conflicts with filesystem geometry agf has bad CRC for ag 4 bad uuid 00000000-0000-0000-0000-000000000000 for agf 5 bad magic # 0x0 for agi 5 bad version # 0 for agi 5 Metadata CRC error detected at 0x469f20, xfs_agi block 0x1ffffffe2/0x200bad log/data device sector size fields in superblock 6 would fix incompat feature mismatch in AG 6 super, 0xb != 0x0 bad sequence # 0 for agi 5 bad length # 0 for agi 5, should be 268435455 bad log/data device sector size fields in superblock 1 would fix ro compat feature mismatch in AG 6 super, 0xd != 0x0 bad magic # 0x0 for agf 6 bad version # 0 for agf 6 bad uuid 00000000-0000-0000-0000-000000000000 for agi 5 would reset bad sb for ag 5 btree block 0/5 is suspect, error -74 bad magic # 0 in refcount btree block 0/5 bad sequence # 0 for agf 6 bad length 0 for agf 6, should be 268435455 would reset bad agf for ag 5 would reset bad agi for ag 5 bad refcountbt block count 0, saw 1 would fix incompat feature mismatch in AG 1 super, 0xb != 0x0 would fix ro compat feature mismatch in AG 1 super, 0xd != 0x0 bad magic # 0x0 for agi 1 bad version # 0 for agi 1 bad sequence # 0 for agi 1 bad length # 0 for agi 1, should be 268435455 bad uuid 00000000-0000-0000-0000-000000000000 for agf 6 bad magic # 0x0 for agi 6 bad uncorrected agheader 5, skipping ag... bad uuid 00000000-0000-0000-0000-000000000000 for agi 1 would reset bad sb for ag 1 would reset bad agi for ag 1 bad uncorrected agheader 1, skipping ag... agi has bad CRC for ag 4 bad on-disk superblock 4 - bad magic number primary/secondary superblock 4 conflict - AG superblock geometry info conflicts with filesystem geometry bad log/data device sector size fields in superblock 4 would fix incompat feature mismatch in AG 4 super, 0xb != 0x0 would fix ro compat feature mismatch in AG 4 super, 0xd != 0x0 bad magic # 0x0 for agf 4 bad version # 0 for agi 6 bad sequence # 0 for agi 6 bad length # 0 for agi 6, should be 268435455 bad version # 0 for agf 4 bad sequence # 0 for agf 4 bad length 0 for agf 4, should be 268435455 bad uuid 00000000-0000-0000-0000-000000000000 for agf 4 bad magic # 0x0 for agi 4 bad version # 0 for agi 4 bad sequence # 0 for agi 4 bad length # 0 for agi 4, should be 268435455 bad uuid 00000000-0000-0000-0000-000000000000 for agi 4 would reset bad sb for ag 4 would reset bad agf for ag 4 would reset bad agi for ag 4 bad uncorrected agheader 4, skipping ag... bad uuid 00000000-0000-0000-0000-000000000000 for agi 6 would reset bad sb for ag 6 would reset bad agf for ag 6 would reset bad agi for ag 6 bad uncorrected agheader 6, skipping ag... Superblock has bad magic number 0x0. Not an XFS filesystem? Metadata CRC error detected at 0x43e660, xfs_agf block 0x17fffffe9/0x200 agf has bad CRC for ag 3 Metadata CRC error detected at 0x469f20, xfs_agi block 0x17fffffea/0x200 agi has bad CRC for ag 3 bad on-disk superblock 3 - bad magic number primary/secondary superblock 3 conflict - AG superblock geometry info conflicts with filesystem geometry bad log/data device sector size fields in superblock 3 would fix incompat feature mismatch in AG 3 super, 0xb != 0x0 would fix ro compat feature mismatch in AG 3 super, 0xd != 0x0 bad magic # 0x0 for agf 3 Superblock has bad magic number 0x0. Not an XFS filesystem? bad version # 0 for agf 3 bad sequence # 0 for agf 3 bad length 0 for agf 3, should be 268435455 bad uuid 00000000-0000-0000-0000-000000000000 for agf 3 bad magic # 0x0 for agi 3 bad version # 0 for agi 3 bad sequence # 0 for agi 3 Metadata CRC error detected at 0x43e660, xfs_agf block 0xfffffff1/0x200 agf has bad CRC for ag 2 bad length # 0 for agi 3, should be 268435455 bad uuid 00000000-0000-0000-0000-000000000000 for agi 3 would reset bad sb for ag 3 would reset bad agf for ag 3 would reset bad agi for ag 3 bad uncorrected agheader 3, skipping ag... Metadata CRC error detected at 0x469f20, xfs_agi block 0xfffffff2/0x200 agi has bad CRC for ag 2 bad on-disk superblock 2 - bad magic number primary/secondary superblock 2 conflict - AG superblock geometry info conflicts with filesystem geometry bad log/data device sector size fields in superblock 2 would fix incompat feature mismatch in AG 2 super, 0xb != 0x0 would fix ro compat feature mismatch in AG 2 super, 0xd != 0x0 bad magic # 0x0 for agf 2 bad version # 0 for agf 2 bad sequence # 0 for agf 2 bad length 0 for agf 2, should be 268435455 bad uuid 00000000-0000-0000-0000-000000000000 for agf 2 bad magic # 0x0 for agi 2 bad version # 0 for agi 2 bad sequence # 0 for agi 2 bad length # 0 for agi 2, should be 268435455 bad uuid 00000000-0000-0000-0000-000000000000 for agi 2 would reset bad sb for ag 2 would reset bad agf for ag 2 would reset bad agi for ag 2 bad uncorrected agheader 2, skipping ag... sb_icount 960, counted 576 sb_ifree 899, counted 531 sb_fdblocks 3850655829, counted 2245055160 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 Metadata corruption detected at 0x438a03, xfs_inode block 0xa0/0x4000 bad CRC for inode 160 bad magic number 0x0 on inode 160 bad version number 0x0 on inode 160 bad next_unlinked 0x0 on inode 160 inode identifier 0 mismatch on inode 160 bad CRC for inode 161 bad magic number 0x0 on inode 161 bad version number 0x0 on inode 161 bad next_unlinked 0x0 on inode 161 inode identifier 0 mismatch on inode 161 bad CRC for inode 162 bad magic number 0x0 on inode 162 bad version number 0x0 on inode 162 bad next_unlinked 0x0 on inode 162 inode identifier 0 mismatch on inode 162 bad CRC for inode 163 bad magic number 0x0 on inode 163 bad version number 0x0 on inode 163 bad next_unlinked 0x0 on inode 163 inode identifier 0 mismatch on inode 163 bad CRC for inode 164 bad magic number 0x0 on inode 164 bad version number 0x0 on inode 164 bad next_unlinked 0x0 on inode 164 inode identifier 0 mismatch on inode 164 bad CRC for inode 165 bad magic number 0x0 on inode 165 bad version number 0x0 on inode 165 bad next_unlinked 0x0 on inode 165 inode identifier 0 mismatch on inode 165 bad CRC for inode 166 bad magic number 0x0 on inode 166 bad version number 0x0 on inode 166 bad next_unlinked 0x0 on inode 166 inode identifier 0 mismatch on inode 166 bad CRC for inode 167 bad magic number 0x0 on inode 167 bad version number 0x0 on inode 167 bad next_unlinked 0x0 on inode 167 inode identifier 0 mismatch on inode 167 bad CRC for inode 168 bad magic number 0x0 on inode 168 bad version number 0x0 on inode 168 bad next_unlinked 0x0 on inode 168 inode identifier 0 mismatch on inode 168 bad CRC for inode 169 bad magic number 0x0 on inode 169 bad version number 0x0 on inode 169 bad next_unlinked 0x0 on inode 169 inode identifier 0 mismatch on inode 169 bad CRC for inode 170 bad magic number 0x0 on inode 170 bad version number 0x0 on inode 170 bad next_unlinked 0x0 on inode 170 inode identifier 0 mismatch on inode 170 bad CRC for inode 171 bad magic number 0x0 on inode 171 bad version number 0x0 on inode 171 bad next_unlinked 0x0 on inode 171 inode identifier 0 mismatch on inode 171 bad CRC for inode 172 bad magic number 0x0 on inode 172 bad version number 0x0 on inode 172 bad next_unlinked 0x0 on inode 172 inode identifier 0 mismatch on inode 172 bad CRC for inode 173 bad magic number 0x0 on inode 173 bad version number 0x0 on inode 173 bad next_unlinked 0x0 on inode 173 inode identifier 0 mismatch on inode 173 bad CRC for inode 174 bad magic number 0x0 on inode 174 bad version number 0x0 on inode 174 bad next_unlinked 0x0 on inode 174 inode identifier 0 mismatch on inode 174 bad CRC for inode 175 bad magic number 0x0 on inode 175 bad version number 0x0 on inode 175 bad next_unlinked 0x0 on inode 175 inode identifier 0 mismatch on inode 175 bad CRC for inode 176 bad magic number 0x0 on inode 176 bad version number 0x0 on inode 176 bad next_unlinked 0x0 on inode 176 inode identifier 0 mismatch on inode 176 bad CRC for inode 177 bad magic number 0x0 on inode 177 bad version number 0x0 on inode 177 bad next_unlinked 0x0 on inode 177 inode identifier 0 mismatch on inode 177 bad CRC for inode 178 bad magic number 0x0 on inode 178 bad version number 0x0 on inode 178 bad next_unlinked 0x0 on inode 178 inode identifier 0 mismatch on inode 178 bad CRC for inode 179 bad magic number 0x0 on inode 179 bad version number 0x0 on inode 179 bad next_unlinked 0x0 on inode 179 inode identifier 0 mismatch on inode 179 bad CRC for inode 180 bad magic number 0x0 on inode 180 bad version number 0x0 on inode 180 bad next_unlinked 0x0 on inode 180 inode identifier 0 mismatch on inode 180 bad CRC for inode 181 bad magic number 0x0 on inode 181 bad version number 0x0 on inode 181 bad next_unlinked 0x0 on inode 181 inode identifier 0 mismatch on inode 181 bad CRC for inode 182 bad magic number 0x0 on inode 182 bad version number 0x0 on inode 182 bad next_unlinked 0x0 on inode 182 inode identifier 0 mismatch on inode 182 bad CRC for inode 183 bad magic number 0x0 on inode 183 bad version number 0x0 on inode 183 bad next_unlinked 0x0 on inode 183 inode identifier 0 mismatch on inode 183 bad CRC for inode 184 bad magic number 0x0 on inode 184 bad version number 0x0 on inode 184 bad next_unlinked 0x0 on inode 184 inode identifier 0 mismatch on inode 184 bad CRC for inode 185 bad magic number 0x0 on inode 185 bad version number 0x0 on inode 185 bad next_unlinked 0x0 on inode 185 inode identifier 0 mismatch on inode 185 bad CRC for inode 186 bad magic number 0x0 on inode 186 bad version number 0x0 on inode 186 bad next_unlinked 0x0 on inode 186 inode identifier 0 mismatch on inode 186 bad CRC for inode 187 bad magic number 0x0 on inode 187 bad version number 0x0 on inode 187 bad next_unlinked 0x0 on inode 187 inode identifier 0 mismatch on inode 187 bad CRC for inode 188 bad magic number 0x0 on inode 188 bad version number 0x0 on inode 188 bad next_unlinked 0x0 on inode 188 inode identifier 0 mismatch on inode 188 bad CRC for inode 189 bad magic number 0x0 on inode 189 bad version number 0x0 on inode 189 bad next_unlinked 0x0 on inode 189 inode identifier 0 mismatch on inode 189 bad CRC for inode 190 bad magic number 0x0 on inode 190 bad version number 0x0 on inode 190 bad next_unlinked 0x0 on inode 190 inode identifier 0 mismatch on inode 190 bad CRC for inode 191 bad magic number 0x0 on inode 191 bad version number 0x0 on inode 191 bad next_unlinked 0x0 on inode 191 inode identifier 0 mismatch on inode 191 imap claims inode 160 is present, but inode cluster is sparse, would correct imap imap claims inode 161 is present, but inode cluster is sparse, would correct imap imap claims inode 162 is present, but inode cluster is sparse, would correct imap imap claims inode 163 is present, but inode cluster is sparse, would correct imap imap claims inode 164 is present, but inode cluster is sparse, would correct imap imap claims inode 165 is present, but inode cluster is sparse, would correct imap imap claims inode 166 is present, but inode cluster is sparse, would correct imap imap claims inode 167 is present, but inode cluster is sparse, would correct imap imap claims inode 168 is present, but inode cluster is sparse, would correct imap imap claims inode 169 is present, but inode cluster is sparse, would correct imap imap claims inode 170 is present, but inode cluster is sparse, would correct imap imap claims inode 171 is present, but inode cluster is sparse, would correct imap imap claims inode 172 is present, but inode cluster is sparse, would correct imap imap claims inode 173 is present, but inode cluster is sparse, would correct imap imap claims inode 174 is present, but inode cluster is sparse, would correct imap imap claims inode 175 is present, but inode cluster is sparse, would correct imap imap claims inode 176 is present, but inode cluster is sparse, would correct imap imap claims inode 177 is present, but inode cluster is sparse, would correct imap imap claims inode 178 is present, but inode cluster is sparse, would correct imap imap claims inode 179 is present, but inode cluster is sparse, would correct imap imap claims inode 180 is present, but inode cluster is sparse, would correct imap imap claims inode 181 is present, but inode cluster is sparse, would correct imap imap claims inode 182 is present, but inode cluster is sparse, would correct imap imap claims inode 183 is present, but inode cluster is sparse, would correct imap imap claims inode 184 is present, but inode cluster is sparse, would correct imap imap claims inode 185 is present, but inode cluster is sparse, would correct imap imap claims inode 186 is present, but inode cluster is sparse, would correct imap imap claims inode 187 is present, but inode cluster is sparse, would correct imap imap claims inode 188 is present, but inode cluster is sparse, would correct imap imap claims inode 189 is present, but inode cluster is sparse, would correct imap imap claims inode 190 is present, but inode cluster is sparse, would correct imap imap claims inode 191 is present, but inode cluster is sparse, would correct imap - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 - agno = 14 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 entry "Downloads" in shortform directory 128 references non-existent inode 6442451072 - agno = 10 - agno = 6 - agno = 5 - agno = 13 - agno = 14 - agno = 7 - agno = 3 - agno = 1 - agno = 9 - agno = 11 - agno = 12 would have junked entry "Downloads" in directory inode 128 would have corrected i8 count in directory 128 from 1 to 0 - agno = 4 entry "Movies" in shortform directory 131 references non-existent inode 2147483776 would have junked entry "Movies" in directory inode 131 entry ".." at block 0 offset 80 in directory inode 15036559552 references non-existent inode 12884902080 entry "TV Shows" in shortform directory 131 references non-existent inode 2147483777 would have junked entry "TV Shows" in directory inode 131 entry "Music" in shortform directory 131 references non-existent inode 8589934721 would have junked entry "Music" in directory inode 131 - agno = 8 would have corrected i8 count in directory 131 from 1 to 0 entry ".." at block 0 offset 80 in directory inode 17179869376 references non-existent inode 12884902080 would rebuild corrupt refcount btrees. No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... entry "Downloads" in shortform directory 128 references non-existent inode 6442451072 would junk entry would fix i8count in inode 128 entry "Movies" in shortform directory 131 references non-existent inode 2147483776 would junk entry entry "TV Shows" in shortform directory 131 references non-existent inode 2147483777 would junk entry entry "Music" in shortform directory 131 references non-existent inode 8589934721 would junk entry would fix i8count in inode 131 entry ".." in directory inode 15036559552 points to non-existent inode 12884902080, would junk entry bad hash table for directory inode 15036559552 (no data entry): would rebuild would rebuild directory inode 15036559552 entry ".." in directory inode 17179869376 points to non-existent inode 12884902080, would junk entry bad hash table for directory inode 17179869376 (no data entry): would rebuild would rebuild directory inode 17179869376 - traversal finished ... - moving disconnected inodes to lost+found ... disconnected dir inode 132, would move to lost+found disconnected dir inode 15036559552, would move to lost+found disconnected dir inode 17179869376, would move to lost+found disconnected dir inode 19327353024, would move to lost+found disconnected dir inode 23622320320, would move to lost+found disconnected dir inode 25769803968, would move to lost+found disconnected dir inode 27917287616, would move to lost+found disconnected dir inode 30064771264, would move to lost+found Phase 7 - verify link counts... would have reset inode 128 nlinks from 4 to 3 would have reset inode 131 nlinks from 5 to 2 No modify flag set, skipping filesystem flush and exiting.

If you can get the data again, I would just reformat the disk and retest, copy some data and do a couple of reboots to see if no more issues.

Thanks, I will give it a try. 

Reformatted, ran the filesystem test (output below), will try to copy a file and reboot a few times. Weird but not trusting this drive at this moment. I tried to put the drive on my SATA > USB converter but it wasn't providing enough power for some reason. Didn't feel like actually installing in my system to run a seatools test, but if it randomly breaks again, I'm going to have to so that I can get this thing RMA'd. 

Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 - agno = 14 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 7 - agno = 1 - agno = 11 - agno = 5 - agno = 14 - agno = 8 - agno = 10 - agno = 3 - agno = 12 - agno = 13 - agno = 6 - agno = 4 - agno = 9 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting.