Hello,  I've just started looking into this issue, but I'm going to ask the question here first in case it isn't so simple.

 

I'm running version 6.3.5 with six drives (four data disks, parity, and cache), Plex and Owncloud Dockers, and 7 VMs on a Dell R510 server.  The issue I'm having lately is directories disappearing from my shares until I restart the server.

 

For example, share 1 would originally have 5 directories (docs, movies, shows, pictures, and music).  After about 12 hours of being powered on share 1 will suddenly have a single directory (docs) and nothing else.  I assume that these missing directories are ones that have been split across different disks.  My movies folder that is used by Plex will still have content, but only about 1/3 of what is supposed to be there.  If I restart the array all directories will come back like normal but disappear again after a few hours.

 

I haven't done anything new recently besides creating a few new VMs and copying files to the NAS like normal.  My total utilization is ~9TB out of 13TB.

 

I know I'm not supposed to post excerpts from the log, but below is one of them.  Most of the log are these same errors repeating for different directories/files.

 

Edit: Full log is attached

 

 

 

 

Nov 22 03:42:36 December-Server root: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
Nov 22 03:42:36 December-Server move: file ./Files/Critical Docs/Monies (On Server Now).xlsx rsync error: 23
Nov 22 03:42:36 December-Server root: Cannot stat file /proc/2474/fd/20: Input/output error
Nov 22 03:42:36 December-Server root: Cannot stat file /proc/2611/fd/20: Input/output error
Nov 22 03:42:36 December-Server root: Cannot stat file /proc/2721/fd/20: Input/output error
Nov 22 03:42:36 December-Server root: Cannot stat file /proc/2865/fd/18: Input/output error
Nov 22 03:42:36 December-Server root: Cannot stat file /proc/2865/fd/19: Input/output error
Nov 22 03:42:36 December-Server root: Cannot stat file /proc/7221/fd/20: Input/output error
Nov 22 03:42:36 December-Server root: Cannot stat file /proc/7251/fd/20: Input/output error
Nov 22 03:42:36 December-Server root: Cannot stat file /proc/7277/fd/20: Input/output error
Nov 22 03:42:36 December-Server root: rsync: recv_generator: mkdir "/mnt/user0/Files" failed: Input/output error (5)
Nov 22 03:42:36 December-Server root: *** Skipping any contents from this failed directory ***
Nov 22 03:42:36 December-Server root: cd+++++++++ Files/
Nov 22 03:42:36 December-Server root: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
Nov 22 03:42:36 December-Server move: file ./Files/Critical Docs/Premium Programs.docx rsync error: 23
Nov 22 03:42:36 December-Server root: rsync: recv_generator: mkdir "/mnt/user0/Files" failed: Input/output error (5)
Nov 22 03:42:36 December-Server root: *** Skipping any contents from this failed directory ***
Nov 22 03:42:36 December-Server root: cd+++++++++ Files/
Nov 22 03:42:36 December-Server root: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
Nov 22 03:42:36 December-Server move: file ./Files/Critical Docs rsync error: 23
Nov 22 03:42:36 December-Server root: rsync: recv_generator: mkdir "/mnt/user0/Files" failed: Input/output error (5)
Nov 22 03:42:36 December-Server root: *** Skipping any contents from this failed directory ***
Nov 22 03:42:36 December-Server root: cd+++++++++ Files/
Nov 22 03:42:36 December-Server root: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
Nov 22 03:42:36 December-Server move: file ./Files/transfer rsync error: 23
Nov 22 03:42:36 December-Server root: rsync: recv_generator: mkdir "/mnt/user0/Files" failed: Input/output error (5)
Nov 22 03:42:36 December-Server root: *** Skipping any contents from this failed directory ***
Nov 22 03:42:36 December-Server root: cd+++++++++ Files/
Nov 22 03:42:36 December-Server root: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
Nov 22 03:42:36 December-Server move: file ./Files rsync error: 23
Nov 22 03:42:36 December-Server root: mover finished
Nov 22 04:40:01 December-Server liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.23.0" x-pid="1742" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 23 00:14:26 December-Server emhttp: err: get_fs_sizes: /mnt/user/Docker statfs: Input/output error
Nov 23 00:14:26 December-Server emhttp: err: get_fs_sizes: /mnt/user/Server statfs: Input/output error
Nov 23 00:14:26 December-Server emhttp: err: get_fs_sizes: /mnt/user/V-Disk statfs: Input/output error
Nov 23 00:14:26 December-Server emhttp: err: get_fs_sizes: /mnt/user/appdata statfs: Input/output error
Nov 23 00:14:26 December-Server emhttp: err: get_fs_sizes: /mnt/user/domains statfs: Input/output error
Nov 23 00:14:26 December-Server emhttp: err: get_fs_sizes: /mnt/user/system statfs: Input/output error
Nov 23 00:14:38 December-Server emhttp: err: get_fs_sizes: /mnt/user/Docker statfs: Input/output error
Nov 23 00:14:38 December-Server emhttp: err: get_fs_sizes: /mnt/user/Server statfs: Input/output error
Nov 23 00:14:38 December-Server emhttp: err: get_fs_sizes: /mnt/user/V-Disk statfs: Input/output error
Nov 23 00:14:38 December-Server emhttp: err: get_fs_sizes: /mnt/user/appdata statfs: Input/output error
Nov 23 00:14:38 December-Server emhttp: err: get_fs_sizes: /mnt/user/domains statfs: Input/output error
Nov 23 00:14:38 December-Server emhttp: err: get_fs_sizes: /mnt/user/system statfs: Input/output error

plex log.txt

Edited November 23, 20178 yr by Danapsta

Check filesystem on disk3 (md3):

 

https://wiki.lime-technology.com/Check_Disk_Filesystems#Drives_formatted_with_XFS

Thank you, I'll run that here in a bit.

 

Am I understanding correctly that if it finds issues and makes repairs,  the repairs will be written to parity?  Reason I ask is if it would be less destructive to replace the drive entirely and do a complete re-build.  Then just completely format disk3. 

3 minutes ago, Danapsta said:

Thank you, I'll run that here in a bit.

 

Am I understanding correctly that if it finds issues and makes repairs,  the repairs will be written to parity?  Reason I ask is if it would be less destructive to replace the drive entirely and do a complete re-build.  Then just completely format disk3. 

 

Rebuild will not typically fix filesystem corruption. You want repairs to include parity or parity will be invalid.

 

Don't know why you would even bring up format here. Do you know what format does?

That makes sense.  I wasn't bringing up formatting as a solution here.  I was just asking if re-building the data to a fresh drive would be less destructive, but if re-building doesn't solve filesystem issues then that won't matter.  I mentioned formatting only because I would wipe the existing drive and add it as an additional disk instead of repairing it.

 

Edit: I went ahead and ran the repair anyway.  It doesn't look like anything is missing, but I will see if everything is still there after a day or so.

Edited November 24, 20178 yr by Danapsta
Update