I'll start by posting it here as disk in question is part of UD, but can start new topic if needed if its just an unraid issue!!

 

I have a WD Purple 3tb drive in UD that I use to store recording from blueiris in a WIn10 VM.

 

After a couple of days the drive shows at 100% full, rendering the share inaccessible via windows share..

 

 

I know for a fact that the drive could NOT be full from recordings as it errors out only after a few days. 

I have reformatted the drive numerous times to get it back to usable state.

 

The last line of my syslog is:

make_connection_snum: '/mnt/disks/CCTV' does not exist or permission denied when connecting to [CCTV] Error was Input/output error

 

What could be causing my drive to constantly fail or seem like its full? and not respond?

 

Logs attached

 

Cheers

tower-diagnostics-20210405-2010.zip

3 minutes ago, bdydrp said:

I'll start by posting it here as disk in question is part of UD, but can start new topic if needed if its just an unraid issue!!

 

I have a WD Purple 3tb drive in UD that I use to store recording from blueiris in a WIn10 VM.

 

After a couple of days the drive shows at 100% full, rendering the share inaccessible via windows share..

 

 

I know for a fact that the drive could NOT be full from recordings as it errors out only after a few days. 

I have reformatted the drive numerous times to get it back to usable state.

 

The last line of my syslog is:

make_connection_snum: '/mnt/disks/CCTV' does not exist or permission denied when connecting to [CCTV] Error was Input/output error

 

What could be causing my drive to constantly fail or seem like its full? and not respond?

 

Logs attached

 

Cheers

tower-diagnostics-20210405-2010.zip 207.87 kB · 0 downloads

 

Your disk is not mounting:

Apr  3 23:45:09 Tower unassigned.devices: Adding disk '/dev/sdd1'...
Apr  3 23:45:09 Tower unassigned.devices: Mount drive command: /sbin/mount -t xfs -o rw,noatime,nodiratime '/dev/sdd1' '/mnt/disks/CCTV'
Apr  3 23:45:09 Tower kernel: XFS (sdd1): Mounting V5 Filesystem
Apr  3 23:45:09 Tower kernel: XFS (sdd1): Starting recovery (logdev: internal)
Apr  3 23:45:09 Tower kernel: XFS (sdd1): Corruption warning: Metadata has LSN (1:10968) ahead of current LSN (1:10840). Please unmount and run xfs_repair (>= v4.3) to resolve.
Apr  3 23:45:09 Tower kernel: XFS (sdd1): Metadata corruption detected at xfs_agi_verify+0x63/0x12e [xfs], xfs_agi block 0x575428e8 
Apr  3 23:45:09 Tower kernel: XFS (sdd1): Unmount and run xfs_repair
Apr  3 23:45:09 Tower kernel: XFS (sdd1): First 128 bytes of corrupted metadata buffer:
Apr  3 23:45:09 Tower kernel: 00000000: 58 41 47 49 00 00 00 01 00 00 00 01 0a ea 85 1b  XAGI............
Apr  3 23:45:09 Tower kernel: 00000010: 00 00 00 40 00 00 00 06 00 00 00 01 00 00 00 18  ...@............
Apr  3 23:45:09 Tower kernel: 00000020: 00 00 00 80 ff ff ff ff ff ff ff ff ff ff ff ff  ................
Apr  3 23:45:09 Tower kernel: 00000030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
Apr  3 23:45:09 Tower kernel: 00000040: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
Apr  3 23:45:09 Tower kernel: 00000050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
Apr  3 23:45:09 Tower kernel: 00000060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
Apr  3 23:45:09 Tower kernel: 00000070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
Apr  3 23:45:09 Tower kernel: XFS (sdd1): metadata I/O error in "xfs_read_agi+0x7c/0xc8 [xfs]" at daddr 0x575428e8 len 8 error 117
Apr  3 23:45:10 Tower rsyslogd: [origin software="rsyslogd" swVersion="8.2002.0" x-pid="6459" x-info="https://www.rsyslog.com"] start
Apr  3 23:45:11 Tower kernel: e1000e 0000:00:1f.6 eth0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Apr  3 23:45:11 Tower kernel: bond0: (slave eth0): link status definitely up, 1000 Mbps full duplex
Apr  3 23:45:11 Tower kernel: bond0: (slave eth0): making interface the new active one
Apr  3 23:45:11 Tower kernel: device eth0 entered promiscuous mode
Apr  3 23:45:11 Tower kernel: bond0: active interface up!
Apr  3 23:45:11 Tower kernel: br0: port 1(bond0) entered blocking state
Apr  3 23:45:11 Tower kernel: br0: port 1(bond0) entered forwarding state
Apr  3 23:45:11 Tower kernel: br0.10: port 1(bond0.10) entered blocking state
Apr  3 23:45:11 Tower kernel: br0.10: port 1(bond0.10) entered forwarding state
Apr  3 23:45:11 Tower kernel: br0.20: port 1(bond0.20) entered blocking state
Apr  3 23:45:11 Tower kernel: br0.20: port 1(bond0.20) entered forwarding state
Apr  3 23:45:13 Tower ntpd[2022]: Listen normally on 4 br0 192.168.1.4:123
Apr  3 23:45:13 Tower ntpd[2022]: new interface(s) found: waking up resolver
Apr  3 23:45:53 Tower unassigned.devices: Mount of '/dev/sdd1' failed. Error message: mount: /mnt/disks/CCTV: mount(2) system call failed: Structure needs cleaning. 
Apr  3 23:45:53 Tower unassigned.devices: Partition 'WDC_WD30PURX-64P6ZY0_WD-WCC4N1YZ4A93' cannot be mounted.

There is a protection on UD disks that prevents the tmpfs file system from filling up when a UD disk is written to but not mounted.  It's very small, that's why you are seeing the 100% full message.

 

Click on the check mark next to the disk partition to attempt a repair.  Make sure you can mount the device before the VM writes to the disk.

 

You should move this disk to a pool device to prevent this kind of problem.

13 minutes ago, dlandon said:

 

Your disk is not mounting:

Apr  3 23:45:09 Tower unassigned.devices: Adding disk '/dev/sdd1'...
Apr  3 23:45:09 Tower unassigned.devices: Mount drive command: /sbin/mount -t xfs -o rw,noatime,nodiratime '/dev/sdd1' '/mnt/disks/CCTV'
Apr  3 23:45:09 Tower kernel: XFS (sdd1): Mounting V5 Filesystem
Apr  3 23:45:09 Tower kernel: XFS (sdd1): Starting recovery (logdev: internal)
Apr  3 23:45:09 Tower kernel: XFS (sdd1): Corruption warning: Metadata has LSN (1:10968) ahead of current LSN (1:10840). Please unmount and run xfs_repair (>= v4.3) to resolve.
Apr  3 23:45:09 Tower kernel: XFS (sdd1): Metadata corruption detected at xfs_agi_verify+0x63/0x12e [xfs], xfs_agi block 0x575428e8 
Apr  3 23:45:09 Tower kernel: XFS (sdd1): Unmount and run xfs_repair
Apr  3 23:45:09 Tower kernel: XFS (sdd1): First 128 bytes of corrupted metadata buffer:
Apr  3 23:45:09 Tower kernel: 00000000: 58 41 47 49 00 00 00 01 00 00 00 01 0a ea 85 1b  XAGI............
Apr  3 23:45:09 Tower kernel: 00000010: 00 00 00 40 00 00 00 06 00 00 00 01 00 00 00 18  ...@............
Apr  3 23:45:09 Tower kernel: 00000020: 00 00 00 80 ff ff ff ff ff ff ff ff ff ff ff ff  ................
Apr  3 23:45:09 Tower kernel: 00000030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
Apr  3 23:45:09 Tower kernel: 00000040: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
Apr  3 23:45:09 Tower kernel: 00000050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
Apr  3 23:45:09 Tower kernel: 00000060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
Apr  3 23:45:09 Tower kernel: 00000070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
Apr  3 23:45:09 Tower kernel: XFS (sdd1): metadata I/O error in "xfs_read_agi+0x7c/0xc8 [xfs]" at daddr 0x575428e8 len 8 error 117
Apr  3 23:45:10 Tower rsyslogd: [origin software="rsyslogd" swVersion="8.2002.0" x-pid="6459" x-info="https://www.rsyslog.com"] start
Apr  3 23:45:11 Tower kernel: e1000e 0000:00:1f.6 eth0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Apr  3 23:45:11 Tower kernel: bond0: (slave eth0): link status definitely up, 1000 Mbps full duplex
Apr  3 23:45:11 Tower kernel: bond0: (slave eth0): making interface the new active one
Apr  3 23:45:11 Tower kernel: device eth0 entered promiscuous mode
Apr  3 23:45:11 Tower kernel: bond0: active interface up!
Apr  3 23:45:11 Tower kernel: br0: port 1(bond0) entered blocking state
Apr  3 23:45:11 Tower kernel: br0: port 1(bond0) entered forwarding state
Apr  3 23:45:11 Tower kernel: br0.10: port 1(bond0.10) entered blocking state
Apr  3 23:45:11 Tower kernel: br0.10: port 1(bond0.10) entered forwarding state
Apr  3 23:45:11 Tower kernel: br0.20: port 1(bond0.20) entered blocking state
Apr  3 23:45:11 Tower kernel: br0.20: port 1(bond0.20) entered forwarding state
Apr  3 23:45:13 Tower ntpd[2022]: Listen normally on 4 br0 192.168.1.4:123
Apr  3 23:45:13 Tower ntpd[2022]: new interface(s) found: waking up resolver
Apr  3 23:45:53 Tower unassigned.devices: Mount of '/dev/sdd1' failed. Error message: mount: /mnt/disks/CCTV: mount(2) system call failed: Structure needs cleaning. 
Apr  3 23:45:53 Tower unassigned.devices: Partition 'WDC_WD30PURX-64P6ZY0_WD-WCC4N1YZ4A93' cannot be mounted.

There is a protection on UD disks that prevents the tmpfs file system from filling up when a UD disk is written to but not mounted.  It's very small, that's why you are seeing the 100% full message.

 

Click on the check mark next to the disk partition to attempt a repair.  Make sure you can mount the device before the VM writes to the disk.

 

You should move this disk to a pool device to prevent this kind of problem.

Thanks!

What would cause the disk to unmount by itself?

Now i understand why it shows as full!

This time i was able to unmount then remount the disk and it started working again.....

Which checkmark sorry?? I didnt see any previously?

 

When you say move disk to a pool device - how? As in part of the array or a pool within UD

I do plan to add an extra drive for recordings and was hoping to show both disk as 1 big disk

EDIT: Just found explanation of UD Pooling - Reading that now!

 

Edited April 5, 20215 yr by bdydrp

3 hours ago, bdydrp said:

Just found explanation of UD Pooling - Reading that now!

I don't know what you are reading. The pool he was referring to is not related to UD.

 

Unraid 6.9 and up has support for multiple pools, so you can add other pools besides "cache" pool and use those pools as you see fit. This is the (preferred) alternative to UD for permanently attached devices.

I have a problem since last Plugin Update to 2021.04.03 on Unraid 6.9.1:

 

Unassigned devices gets a timeout when starting the Unraid-Server, Server LOG:
   Apr 5 17:35:40 Ryzen7-5800X unassigned.devices: Mounting 'Auto Mount' Remote Shares...

   Apr 5 17:41:10 Ryzen7-5800X unassigned.devices: Cannot 'Auto Mount' Remote Shares. Network not available!

 

Pinging the Server is possible the whole time, and Unraid GUI is not reachable.

After this 5min 30sec the Unraid GUI is reachable, and I can manually (by hitting MOUNT-Button) mount my SMB Shares.
 

For testing I deleted my all my SMB Shares:
Same Timeout-Behaviour when rebooting Server.

ryzen7-5800x-diagnostics-20210405-1815.zip

5 hours ago, peter76 said:

Pinging the Server is possible the whole time

Do you mean you are able to ping the server that has the remote shares on it, or do you mean you can ping the Unraid server that is trying to mount the remote shares?

I can ping the Unraid server that is trying to mount the remote shares.
 

10 hours ago, bdydrp said:

What would cause the disk to unmount by itself?

It didn't.  It never mounted.

 

10 hours ago, bdydrp said:

Which checkmark sorry?? I didnt see any previously?

Click the '+' icon on the disk on the UD web page.  You'll see a check mark when the disk us not mounted.  That does a file system check.

 

10 hours ago, bdydrp said:

When you say move disk to a pool device - how?

On 6.9 there is a tab called 'Pool Devices'.  Add your disk drive there as an additional device (It's like an additional cache disk) and give it a meaningful name.  Your VM can then read/write that disk.  Unraid manages the disk.

 

10 hours ago, bdydrp said:

EDIT: Just found explanation of UD Pooling - Reading that now!

This is not recommended.  You can do this as a 'Pool Device'.

5 hours ago, peter76 said:

Unassigned devices gets a timeout when starting the Unraid-Server, Server LOG:
   Apr 5 17:35:40 Ryzen7-5800X unassigned.devices: Mounting 'Auto Mount' Remote Shares...

   Apr 5 17:41:10 Ryzen7-5800X unassigned.devices: Cannot 'Auto Mount' Remote Shares. Network not available!

UD does a network test before mounting remote shares.  The test for the network being available was not successful.  This check is after the array is started.  The network should be available then.

 

 

5 hours ago, peter76 said:

Pinging the Server is possible the whole time, and Unraid GUI is not reachable.

After this 5min 30sec the Unraid GUI is reachable, and I can manually (by hitting MOUNT-Button) mount my SMB Shares.

If the GUI is not reachable, UD cannot mount remote devices because the network is not ready.

 

The UD test for the network being available is by trying to ping your gateway device for 30 seconds.  If there is no response, UD thinks the network is not available.  Generally the gateway is your router.  Check your network setup in Unraid.

thanks a lot to dlandon, you brought me on the right path to search.

A few days ago, I did some changes on my firewall, and now pinging the gateway was not possible any more.

I solved the problem by allowing pinging the gateway.

 

Just one thing to think about:

Unraid GUI (without UD Plugin) starts without beeing able to ping the Gateway.

When installing UD Plugin AND pinging the Gateway is not possible, user gets problems (as I got..).
In my "dummy user"-opinion pinging the Gateway is maybe not the best solution for checking the network.

41 minutes ago, peter76 said:

Just one thing to think about:

Unraid GUI (without UD Plugin) starts without beeing able to ping the Gateway.

When installing UD Plugin AND pinging the Gateway is not possible, user gets problems (as I got..).
In my "dummy user"-opinion pinging the Gateway is maybe not the best solution for checking the network.

UD has no idea where the remote mount server is located.  If it is on-site (LAN), your argument makes sense.  If the server can only be reached on the Internet, then pinging the gateway makes sense.  If the gateway was not available the remote mounts would fail if they are on the Internet.

2 hours ago, dlandon said:

It didn't.  It never mounted.

It was prior to me posting! 

I had reformatted the drive and got it working again, then it "unmounted" and hence my post before going any further.

 

Now overnight i was running an extended SMART test and woke to find the drive disconnected and shown as historical on the dash

 

Obviously i have an issue somewhere with the drive itself, cabling or HBA card. Just need to narrow it down!

 

I have new diagnostics - Not sure if they show how/when drive was connected/disconnected

 

tower-diagnostics-20210406-1045.zip

Hi,
Since yesterday my mapped NAS drives have been acting strange.

They look like they are mounted correctly but "size", "used", and "free" all show 0.
 

The log shows some kind of time out error:

Apr 6 20:13:50 HighTower unassigned.devices: Error: shell_exec(/bin/df '/mnt/remotes/LAKR_NAS_Media' --output=size,used,avail | /bin/grep -v '1K-blocks' 2>/dev/null) took longer than 2s!

After a reboot the drives mounted correctly - but the error came back after a couple of hours again

 

I tried googling the error but to no avail.

 

Unraid version: 6.9.1
Unassigned Devices version: 2021.04.03
 

hightower-diagnostics-20210406-2021.zip

15 hours ago, lakr said:

Hi,
Since yesterday my mapped NAS drives have been acting strange.

They look like they are mounted correctly but "size", "used", and "free" all show 0.
 

The log shows some kind of time out error:

Apr 6 20:13:50 HighTower unassigned.devices: Error: shell_exec(/bin/df '/mnt/remotes/LAKR_NAS_Media' --output=size,used,avail | /bin/grep -v '1K-blocks' 2>/dev/null) took longer than 2s!

After a reboot the drives mounted correctly - but the error came back after a couple of hours again

 

I tried googling the error but to no avail.

 

Unraid version: 6.9.1
Unassigned Devices version: 2021.04.03
 

hightower-diagnostics-20210406-2021.zip 144.99 kB · 1 download

This generally is from network issues.  Are you using Jumbo frames?

On 4/6/2021 at 7:50 AM, dlandon said:

Click the '+' icon on the disk on the UD web page.  You'll see a check mark when the disk us not mounted.  That does a file system check.

After rebooting system because disk went offline, I clicked the check mark to run file system check

This is the result:

Same result if i press 'run with correct flag'

 

I'm going to assume here my disk is pretty much toast??

2 hours ago, bdydrp said:

I'm going to assume here my disk is pretty much toast??

Post SMART for that disk.

6 hours ago, dlandon said:

This generally is from network issues.  Are you using Jumbo frames?

MUT is set to 1.500 (I think this is default - not something I did)

I'm also hitting the "Cancelling wait for mid" issue on my Unassigned Disk remote SMB share connected to a Mac:  

 

Apr  7 14:35:03 Tower kernel: CIFS: VFS: \\MACMINI-581918\DATA Close unmatched open for MID:ad201
Apr  7 14:35:03 Tower kernel: CIFS: VFS: \\MACMINI-581918\DATA Close cancelled mid failed rc:-9
Apr  7 14:37:35 Tower kernel: CIFS: VFS: \\MACMINI-581918\DATA Close interrupted close
Apr  7 14:42:37 Tower kernel: CIFS: VFS: \\MACMINI-581918 Cancelling wait for mid 1443535 cmd: 6
Apr  7 14:42:37 Tower kernel: CIFS: VFS: \\MACMINI-581918\DATA Close interrupted close
Apr  7 14:42:37 Tower kernel: CIFS: VFS: \\MACMINI-581918\DATA Close cancelled mid failed rc:-9
Apr  7 14:45:08 Tower kernel: CIFS: VFS: \\MACMINI-581918 Cancelling wait for mid 1682517 cmd: 5
Apr  7 14:45:08 Tower kernel: CIFS: VFS: \\MACMINI-581918 Cancelling wait for mid 1682518 cmd: 14
Apr  7 14:45:08 Tower kernel: CIFS: VFS: \\MACMINI-581918\DATA Close unmatched open for MID:19ac55

This goes on and on in my log.  Diagnostics attached...thanks!

tower-diagnostics-20210407-1631.zip

20 hours ago, trurl said:

Post SMART for that disk.

So after shutting down server to remove a different failed drive, upon reboot i seem t o able to access smarts which indicates passed without error

also, ran filesystem check again, with different results

tower-smart-20210408-2144.zip

3 hours ago, bdydrp said:

So after shutting down server to remove a different failed drive, upon reboot i seem t o able to access smarts which indicates passed without error

Doesn't look like it has ever completed an extended SMART test.

3 hours ago, bdydrp said:

ran filesystem check again

You have to remove -n (nomodify) flag or nothing will be repaired.

Hello,

 

I have a similar problem as @wibble with the new version 

 

 

No error on disk log information

 

Thank

 

 

 

disklog.txt MountingLog.txt

On 4/7/2021 at 4:32 PM, Andiroo2 said:

I'm also hitting the "Cancelling wait for mid" issue on my Unassigned Disk remote SMB share connected to a Mac:  

 

Apr  7 14:35:03 Tower kernel: CIFS: VFS: \\MACMINI-581918\DATA Close unmatched open for MID:ad201
Apr  7 14:35:03 Tower kernel: CIFS: VFS: \\MACMINI-581918\DATA Close cancelled mid failed rc:-9
Apr  7 14:37:35 Tower kernel: CIFS: VFS: \\MACMINI-581918\DATA Close interrupted close
Apr  7 14:42:37 Tower kernel: CIFS: VFS: \\MACMINI-581918 Cancelling wait for mid 1443535 cmd: 6
Apr  7 14:42:37 Tower kernel: CIFS: VFS: \\MACMINI-581918\DATA Close interrupted close
Apr  7 14:42:37 Tower kernel: CIFS: VFS: \\MACMINI-581918\DATA Close cancelled mid failed rc:-9
Apr  7 14:45:08 Tower kernel: CIFS: VFS: \\MACMINI-581918 Cancelling wait for mid 1682517 cmd: 5
Apr  7 14:45:08 Tower kernel: CIFS: VFS: \\MACMINI-581918 Cancelling wait for mid 1682518 cmd: 14
Apr  7 14:45:08 Tower kernel: CIFS: VFS: \\MACMINI-581918\DATA Close unmatched open for MID:19ac55

This goes on and on in my log.  Diagnostics attached...thanks!

tower-diagnostics-20210407-1631.zip 147.57 kB · 0 downloads

I have no idea what is causing this and a Google search does not present any meaningful information

6 hours ago, rjulie95 said:

Hello,

 

I have a similar problem as @wibble with the new version 

 

 

No error on disk log information

 

Thank

 

 

 

disklog.txt 1.62 kB · 0 downloads MountingLog.txt 608 B · 0 downloads

The file system has a problem.  You need to take it to a Windows computer and see if it can be sorted out there,

15 hours ago, dlandon said:

The file system has a problem.  You need to take it to a Windows computer and see if it can be sorted out there,

under windows the disk has no problem

6 hours ago, rjulie95 said:

under windows the disk has no problem

Linux is having an issue with the disk.  UD can't do anything if Linux refuses to mount it.