UNRAID can't save files anymore (shfs command failed)

[marco_yang]

Problem: cannot save files to UNRAID server

 

Hello everyone, mine UNRAID was working fine but today I got a serious problem where none of my files can be uploaded to the server anymore. I've been trying to use docker "File Browser", plug-in "Dynamix File Manager", as well as the WEBDAV & SMB sharing from windows, but none of them works.

When I was trying see the syslog, it appears that the system was reporting:

May 29 19:24:56 NorthernLight3s shfs: command failed: 1

Anyone has any idea?

northernlight3s-diagnostics-20230529-1937.zip

Edited May 29, 2023 by marco_yang

[JorgeB]

Please post output of

zfs list

 

Also, if you set the share software to use the array as the primary storage option, can you then copy new data?

[marco_yang]

@JorgeB 

Hi thanks for your reply! I really appreciate your help!!!

Here is the out put:

NAME             USED  AVAIL     REFER  MOUNTPOINT
cache           5.21M   231G       96K  /mnt/cache
cache/a           96K   231G       96K  /mnt/cache/a
cache/else        96K   231G       96K  /mnt/cache/else
cache/kelsey      96K   231G       96K  /mnt/cache/kelsey
cache/photo       96K   231G       96K  /mnt/cache/photo
cache/software    96K   231G       96K  /mnt/cache/software
media            210G   650G      104K  /mnt/media
media/music      210G   650G      210G  /mnt/media/music
sys              305G  1.38T      144K  /mnt/sys
sys/appdata     37.9G  1.38T     37.9G  /mnt/sys/appdata
sys/docker      4.05M  1.38T     4.05M  /mnt/sys/docker
sys/domains      205G  1.38T      205G  /mnt/sys/domains
sys/isos        45.0G  1.38T     45.0G  /mnt/sys/isos
sys/system      17.0G  1.38T     17.0G  /mnt/sys/system
sys/work        61.7M  1.38T     61.7M  /mnt/sys/work

 

I tried to set "Array" as primary, and "none" as secondary, and then upload works fine! Do you think this may be a BUG or something because I am using 6.12.0-rc6?

[marco_yang]

I also want to mention that this problem seems only occur when UNRAID server already runs for a period of time.

 

When I just reboot, it seems I can upload and download as normal. But some time (last time it was a few hours after reboot, but this time it was almost 5 days) later this problem will happen and I will not be able to manage my files.

[JorgeB]

15 minutes ago, marco_yang said:

Do you think this may be a BUG

Possibly, though I still cannot reproduce, investigating.

[JorgeB]

May 26 00:00:01 NorthernLight3s shfs: /usr/sbin/zfs destroy -r 'cache/a'
May 26 00:00:01 NorthernLight3s shfs: error: retval 1 attempting 'zfs destroy'
May 26 00:00:01 NorthernLight3s shfs: /usr/sbin/zfs destroy -r 'cache/else'
May 26 00:00:01 NorthernLight3s shfs: error: retval 1 attempting 'zfs destroy'
May 26 00:00:01 NorthernLight3s shfs: /usr/sbin/zfs destroy -r 'cache/kelsey'
May 26 00:00:01 NorthernLight3s shfs: error: retval 1 attempting 'zfs destroy'
May 26 00:00:01 NorthernLight3s shfs: /usr/sbin/zfs destroy -r 'cache/photo'
May 26 00:00:01 NorthernLight3s shfs: error: retval 1 attempting 'zfs destroy'
May 26 00:00:01 NorthernLight3s shfs: /usr/sbin/zfs destroy -r 'cache/software'
May 26 00:00:01 NorthernLight3s shfs: error: retval 1 attempting 'zfs destroy'

 

Do you have a script running at midnight that would explain these? It should not be the mover since there's nothing logged about that.

[JorgeB]

2 hours ago, JorgeB said:

Do you have a script running at midnight that would explain these? It should not be the mover since there's nothing logged about that.

I see this is the mover, now if mover logging is disabled there's nothing logged.

 

Please post the output of:

zfs get all cache/software

 

just to make sure there's nothing out of the ordinary with the dataset, something that explains why it was not destroyed by the mover.

[marco_yang]

@JorgeB

Here is the output:

root@NorthernLight3s:~# zfs get all cache/software
NAME            PROPERTY              VALUE                  SOURCE
cache/software  type                  filesystem             -
cache/software  creation              Tue May 23 11:56 2023  -
cache/software  used                  96K                    -
cache/software  available             231G                   -
cache/software  referenced            96K                    -
cache/software  compressratio         1.00x                  -
cache/software  mounted               no                     -
cache/software  quota                 none                   default
cache/software  reservation           none                   default
cache/software  recordsize            128K                   default
cache/software  mountpoint            /mnt/cache/software    inherited from cache
cache/software  sharenfs              off                    default
cache/software  checksum              on                     default
cache/software  compression           off                    inherited from cache
cache/software  atime                 off                    inherited from cache
cache/software  devices               on                     default
cache/software  exec                  on                     default
cache/software  setuid                on                     default
cache/software  readonly              off                    default
cache/software  zoned                 off                    default
cache/software  snapdir               hidden                 default
cache/software  aclmode               discard                default
cache/software  aclinherit            restricted             default
cache/software  createtxg             320027                 -
cache/software  canmount              on                     default
cache/software  xattr                 sa                     inherited from cache
cache/software  copies                1                      default
cache/software  version               5                      -
cache/software  utf8only              on                     -
cache/software  normalization         formD                  -
cache/software  casesensitivity       sensitive              -
cache/software  vscan                 off                    default
cache/software  nbmand                off                    default
cache/software  sharesmb              off                    default
cache/software  refquota              none                   default
cache/software  refreservation        none                   default
cache/software  guid                  15239827051366107895   -
cache/software  primarycache          all                    default
cache/software  secondarycache        all                    default
cache/software  usedbysnapshots       0B                     -
cache/software  usedbydataset         96K                    -
cache/software  usedbychildren        0B                     -
cache/software  usedbyrefreservation  0B                     -
cache/software  logbias               latency                default
cache/software  objsetid              396                    -
cache/software  dedup                 off                    default
cache/software  mlslabel              none                   default
cache/software  sync                  standard               default
cache/software  dnodesize             auto                   inherited from cache
cache/software  refcompressratio      1.00x                  -
cache/software  written               96K                    -
cache/software  logicalused           42K                    -
cache/software  logicalreferenced     42K                    -
cache/software  volmode               default                default
cache/software  filesystem_limit      none                   default
cache/software  snapshot_limit        none                   default
cache/software  filesystem_count      none                   default
cache/software  snapshot_count        none                   default
cache/software  snapdev               hidden                 default
cache/software  acltype               posix                  inherited from cache
cache/software  context               none                   default
cache/software  fscontext             none                   default
cache/software  defcontext            none                   default
cache/software  rootcontext           none                   default
cache/software  relatime              off                    default
cache/software  redundant_metadata    all                    default
cache/software  overlay               on                     default
cache/software  encryption            off                    default
cache/software  keylocation           none                   default
cache/software  keyformat             none                   default
cache/software  pbkdf2iters           0                      default
cache/software  special_small_blocks  0                      default

 

I have the following things run at midnight:

 

1. 23:50 PM

my Synology server (under the same network with my UNRAID server) will power on

 

2. 23:55 PM 

my UNRAID server will use "user script" plug-in to make a direction under "/mnt/remotes/" and then mount my Synology server share folder to this location

 

3. 00:00 AM

my UNRAID server will:

check disk SMART information

start cache mover

start to use "Appdata backup" plug-in to backup my dockers and flash

my Synology server will:

start to backup shared folders via "Active backup for business"

 

4. 02:00 AM

my Synology server will power down

Edited May 30, 2023 by marco_yang

[JorgeB]

4 hours ago, marco_yang said:

cache/software  mounted               no         

This is not correct, and would explain why you cannot write to the share with cache=yes, the question is what is unmounting the dataset, type:

zfs mount -a

 

Now you should be able to write to the pool again, then try to find when the problem starts again.

[JorgeB]

Please update to rc7 and re-test, if there are still issues it may help manually deleting the existing datasets so they can be recreated, but please post new diags in that case.