FCP - Unable to write to disk1

[extrobe]

FCP has just given me a warning that it's unable to write to disk 1, and that "Drive mounted read-only or completely full"

But I'm struggling to find an issue anywhere

 

Plenty of space on both the disk (45%) and the wider array

SMART data all looks healthy - and UR reports the disk as 'Normal Operation, Device is Active'

Using Krusader, I was able to browse the content of the drive, and (via the relevant share) was able to delete files from it OK. I also created a simple txt file directly onto the disk, which I was able to open and read from windows share as well.

 

So I'm unsure what the issue is that it's trying to tell me about. I've re-ran several times, and still finds it.

 

My contents of /mnt/ all looks ok...

- cache

- disk1

- ...

- disk16

- disks

- user

- user0

 

Only other notable event is that I've added a new disk (disk 17), which is currently in clearing stage - and that this possibly occurred following the 6.6.1 upgrade.

(Ran the upgrade, rebooted, stopped array to add new disk, restarted array, spotted FCP message)

 

Is there anything else I should be checking? I've attached diagnostics in case it's of any use.

demeter-diagnostics-20181003-2355.zip

Edited October 3, 2018 by extrobe

[JustinAiken]

I've been getting "unable to write to..." errors since upgrading to 6.6.1, from my Mac to a my cache share.  

 

I've tried running `newperms /mnt/cache/<folder that complains>` then ejecting and remounting the SMB share, same error...

 

[JorgeB]

OP: You need to check filesystem on disk1

https://wiki.unraid.net/Check_Disk_Filesystems#Drives_formatted_with_XFS

or

https://wiki.unraid.net/Check_Disk_Filesystems#Checking_and_fixing_drives_in_the_webGui

 

 

 

Edited October 3, 2018 by johnnie.black

[extrobe]

Thanks @johnnie.black - I'll give that a go. Will stopping the array cancel the 'clear' I have running?

And if so, and I appreciate this is probably impossible to answer, is this something I should cancel the clear for and do asap, or is it safe to leave it a day or so until everything is caught up?

[JorgeB]

4 minutes ago, extrobe said:

Will stopping the array cancel the 'clear' I have running?

Yes

 

4 minutes ago, extrobe said:

And if so, and I appreciate this is probably impossible to answer, is this something I should cancel the clear for and do asap, or is it safe to leave it a day or so until everything is caught up?

As long as you don't need to write anything to disk1 it *should* be OK to leave like that until the clear is done.

[extrobe]

Thanks - it's gone midnight here, so at least I can fairly safely leave it until the morning to pick up again - not sure I was up for an all-nighter  ! Most of what is on that disk is cold storage stuff, so not likely to be accessed further over night.

[extrobe]

Ran the checker - and got a lot of warnings, such as 

Quote

entry "Season 03" in shortform directory 4420105934 references non-existent inode 562661054 would have junked entry "Season 03" in directory inode 4420105934

and the message

Quote

Inode allocation btrees are too corrupted, skipping phases 6 and 7

 

No idea what most of this means, but doesn't sound good.

 

Having read up on others having similar issues, it doesn't sound like an xfs_repair is the way to go, so I'll remove the disk from the array, start the array (and let parity take over), re-format the disk and add it again as a blank disk

[JorgeB]

xfs_repair should fix the filesystem and recover most data, though some data loss might occur if there's a lot of corruption, but if you can re-format the disk and restore the data from backups it's also a good way to go.

[extrobe]

Getting a bit lost with this now!

 

I tried re-formatting and re-building, but same errors

I tried a repair - same errors

So tried re-formatting again, but converting to something other than xfs before converting back to xfs - same errors

 

Figured it might be a disk issue, so bought a brand new disk. Re built it. Same darn errors. 

So began a unbalanced session to move everything on it to other disks. About half way through, the disk became unmountable. Restarted - still unmountable, although the disk and array shows as 'active' (not emulated), but can't get access to anything.

 

So stopped the array, took the disk out the array and restarted the array. Disk is now emulated, but the emulated disk is also not mountable.

 

Not really sure where I should be taking this next - I do have a spare drive (well, the original one which turns out to be fine, as ran extended tests overnight on my windows PC) - but suspect it'll just rebuild this corrupt data.

 

At the point unbalanced was running, the data itself all seemed fine - I was randomly picking various media files stored on the disk, and everything was playing fine - but now don't appear to have a way to access / rebuild any of that data - I have 2x parity, but if it's emulating a corrupt unmountable drive, I"m not sure if that counts for anything.

 

The disk is question I've currently got sat on my desk. Might try firing up a linux VM on windows to see if I can pull anything from it

[itimpi]

The commonest cause of disk errors is the cabling (power or SATA) so have you checked those?

[extrobe]

Yes - I have 20 bays in total - and I've had the disk in several of those bays, which covers different sata connections to the PSU, and different raid controllers.

I think it's must have been some sort of corrupted filesystem, but that corruption appeared to be being replicated whenever the disk was rebuilt - and about half way through actually shifting the data to different disks, it 'gave up'.

I way eventually able to mount the drive in my Ubunutu VM and recover about 200GB - reckon there was closer to 1TB of data, so unless I can figure out how to recover it from parity ,looks like I've lost it. The important stuff is backed up, so will just be media.

 

Just got to figure out how to get out of this 'loop' I seem to be in.

 

Edit: There's a huge Lost & Found folder, so looks like I'll be able to recover most data. So just need to work out how to sort out the unmountable drive. Think I have a plan though - fresh drive, re-build the drive - then remove it out of the array and rebuild parity without it.

re-format the disk, add it again - then copy files across from the repaired drive.

Edited October 7, 2018 by extrobe

[itimpi]

It is worth pointing out that doing a rebuild will never fix file system corruption - it will just replicate it onto the rebuilt disk.   It takes running the file system repair tools to fix such corruption.   The fact you have a lost+found folder suggests that you have already run such a tool.

[JorgeB]

Something missing here, formatting a disk will fix any file system issues (and delete all data), if Unraid fails to format the disk post diagnostics after trying to do it.