Hello, recently I have been getting stranger errors in my syslog. These errors come up almost every second my log will will full up with all these errors. It seems my cache drive is getting these errors. my cache drive setup is btrfs raid1 2x500gb WD Blue. Not sure if one of my cache drives is failing or what. sdf1 and sdg1 are my cache drives.

Steps i've taken - i ran a scrub / ran a repair none of those help.

i have attached my diagnostics file. 

any help would greatly appreciated thank you

 

Sep  7 13:19:40 XENON-NAS kernel: BTRFS warning (device sdg1): lost page write due to IO error on /dev/sdf1 (-5)
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:40 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:41 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:41 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:41 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:41 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:41 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:41 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:41 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:41 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:41 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:41 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:42 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:42 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:42 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:42 XENON-NAS kernel: repair_io_failure: 10 callbacks suppressed
Sep  7 13:19:42 XENON-NAS kernel: BTRFS info (device sdg1): read error corrected: ino 0 off 1445825347584 (dev /dev/sdg1 sector 7906304)
Sep  7 13:19:42 XENON-NAS kernel: BTRFS info (device sdg1): read error corrected: ino 0 off 1445825351680 (dev /dev/sdg1 sector 7906312)
Sep  7 13:19:42 XENON-NAS kernel: BTRFS info (device sdg1): read error corrected: ino 0 off 1445825355776 (dev /dev/sdg1 sector 7906320)
Sep  7 13:19:42 XENON-NAS kernel: BTRFS info (device sdg1): read error corrected: ino 0 off 1445825359872 (dev /dev/sdg1 sector 7906328)
Sep  7 13:19:42 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:42 XENON-NAS kernel: btrfs_dev_stat_inc_and_print: 95576 callbacks suppressed
Sep  7 13:19:42 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449115750, rd 163556857, flush 2691324, corrupt 119303, gen 44
Sep  7 13:19:42 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449115751, rd 163556857, flush 2691324, corrupt 119303, gen 44
Sep  7 13:19:42 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449115752, rd 163556857, flush 2691324, corrupt 119303, gen 44
Sep  7 13:19:42 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449115753, rd 163556857, flush 2691324, corrupt 119303, gen 44
Sep  7 13:19:42 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449115754, rd 163556857, flush 2691324, corrupt 119303, gen 44
Sep  7 13:19:42 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449115755, rd 163556857, flush 2691324, corrupt 119303, gen 44
Sep  7 13:19:42 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449115756, rd 163556857, flush 2691324, corrupt 119303, gen 44
Sep  7 13:19:42 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449115757, rd 163556857, flush 2691324, corrupt 119303, gen 44
Sep  7 13:19:42 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449115758, rd 163556857, flush 2691324, corrupt 119303, gen 44
Sep  7 13:19:42 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449115759, rd 163556857, flush 2691324, corrupt 119303, gen 44
Sep  7 13:19:43 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:43 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:43 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:43 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:43 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS info (device sdg1): read error corrected: ino 0 off 1345683456 (dev /dev/sdg1 sector 2587328)
Sep  7 13:19:44 XENON-NAS kernel: BTRFS info (device sdg1): read error corrected: ino 0 off 1345687552 (dev /dev/sdg1 sector 2587336)
Sep  7 13:19:44 XENON-NAS kernel: BTRFS info (device sdg1): read error corrected: ino 0 off 1345691648 (dev /dev/sdg1 sector 2587344)
Sep  7 13:19:44 XENON-NAS kernel: BTRFS info (device sdg1): read error corrected: ino 0 off 1345695744 (dev /dev/sdg1 sector 2587352)
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: btrfs_end_super_write: 71 callbacks suppressed
Sep  7 13:19:44 XENON-NAS kernel: BTRFS warning (device sdg1): lost page write due to IO error on /dev/sdf1 (-5)
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:44 XENON-NAS kernel: BTRFS warning (device sdg1): lost page write due to IO error on /dev/sdf1 (-5)
Sep  7 13:19:44 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:45 XENON-NAS kernel: BTRFS warning (device sdg1): lost page write due to IO error on /dev/sdf1 (-5)
Sep  7 13:19:45 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:45 XENON-NAS kernel: BTRFS warning (device sdg1): lost page write due to IO error on /dev/sdf1 (-5)
Sep  7 13:19:45 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:45 XENON-NAS kernel: BTRFS warning (device sdg1): lost page write due to IO error on /dev/sdf1 (-5)
Sep  7 13:19:45 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:45 XENON-NAS kernel: BTRFS warning (device sdg1): lost page write due to IO error on /dev/sdf1 (-5)
Sep  7 13:19:45 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:45 XENON-NAS kernel: BTRFS warning (device sdg1): lost page write due to IO error on /dev/sdf1 (-5)
Sep  7 13:19:45 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:45 XENON-NAS kernel: BTRFS warning (device sdg1): lost page write due to IO error on /dev/sdf1 (-5)
Sep  7 13:19:45 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:46 XENON-NAS kernel: BTRFS warning (device sdg1): lost page write due to IO error on /dev/sdf1 (-5)
Sep  7 13:19:46 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:46 XENON-NAS kernel: BTRFS warning (device sdg1): lost page write due to IO error on /dev/sdf1 (-5)
Sep  7 13:19:46 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): error writing primary super block to device 1
Sep  7 13:19:47 XENON-NAS kernel: btrfs_dev_stat_inc_and_print: 96788 callbacks suppressed
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449212724, rd 163557011, flush 2691383, corrupt 119303, gen 44
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449212725, rd 163557011, flush 2691383, corrupt 119303, gen 44
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449212726, rd 163557011, flush 2691383, corrupt 119303, gen 44
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449212727, rd 163557011, flush 2691383, corrupt 119303, gen 44
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449212728, rd 163557011, flush 2691383, corrupt 119303, gen 44
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449212729, rd 163557011, flush 2691383, corrupt 119303, gen 44
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449212730, rd 163557011, flush 2691383, corrupt 119303, gen 44
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449212731, rd 163557011, flush 2691383, corrupt 119303, gen 44
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449212732, rd 163557011, flush 2691383, corrupt 119303, gen 44
Sep  7 13:19:47 XENON-NAS kernel: BTRFS error (device sdg1): bdev /dev/sdf1 errs: wr 449212733, rd 163557011, flush 2691383, corrupt 119303, gen 44

 

xenon-nas-diagnostics-20240907-0520.zip

Looks like one of the pool devices dropped offline, but the syslog already rotated, so we can't see the start of the issue, reboot and post new diags after array start.

Hello JorgeB

Thank you for taking the time to look at the issue.

The new thing i noticed is a had a notification come up saying can not write to USB now , and i can not download the new update for COMMUNITY APP. 

let me know what you think

I have a attached a new diag file after a reboot.

xenon-nas-diagnostics-20240909-0306.zip

Edited September 9, 20241 yr by poniz

Cache1 dropped offline again:

 

Sep  9 03:05:41 XENON-NAS kernel: ata2: softreset failed (device not ready)
Sep  9 03:05:41 XENON-NAS kernel: ata2: reset failed, giving up
Sep  9 03:05:41 XENON-NAS kernel: ata2.00: disable device
Sep  9 03:05:41 XENON-NAS kernel: sd 2:0:0:0: rejecting I/O to offline device

 

Check/replace cables and post new diags after array start.

Hello, i replaced cables as you asked. and I have attached the file

xenon-nas-diagnostics-20240909-1918.zip

Looks like the drive might be ok after swap out the cable but now my docker will not start. Do i have to remake my .img file again or is there another way to get the docker service to start ?

should i be looking at replacing my cache drive or you think it was just a bad cable?

 

Sep 10 01:26:44 XENON-NAS root: Starting Avahi mDNS/DNS-SD DNS Server Configuration Daemon:  /usr/sbin/avahi-dnsconfd -D
Sep 10 01:26:44 XENON-NAS avahi-dnsconfd[21838]: Successfully connected to Avahi daemon.
Sep 10 01:26:44 XENON-NAS emhttpd: shcmd (77): /usr/local/sbin/mount_image '/mnt/cache/docker/docker.img' /var/lib/docker 20
Sep 10 01:26:44 XENON-NAS kernel: loop2: detected capacity change from 0 to 41943040
Sep 10 01:26:45 XENON-NAS root: mount: /var/lib/docker: wrong fs type, bad option, bad superblock on /dev/loop2, missing codepage or helper program, or other error.
Sep 10 01:26:45 XENON-NAS root:        dmesg(1) may have more information after failed mount system call.
Sep 10 01:26:45 XENON-NAS kernel: squashfs: Unknown parameter 'space_cache'
Sep 10 01:26:45 XENON-NAS kernel: fuseblk: Unknown parameter 'space_cache'
Sep 10 01:26:45 XENON-NAS kernel: UDF-fs: bad mount option "space_cache=v2" or missing value

 

xenon-nas-diagnostics-20240910-0137.zip

First run a correcting scrub on the pool and post the results and new diags.

Hello, ok no problem. while running scrub my box went to sleep while i have the plugin disable thats a separate issues.

here are the results of the scrub before it adborted

 

UUID:             de68d996-50ba-4769-822a-45f1c30c1dd3
Scrub started:    Tue Sep 10 01:40:07 2024
Status:           aborted
Duration:         0:18:06
Total to scrub:   526.00GiB
Rate:             249.38MiB/s
Error summary:    verify=15251 csum=6891912
  Corrected:      6907161
  Uncorrectable:  2
  Unverified:     0

 

ran scrub again this time the box didn't not go to sleep and it completed

 

UUID:             de68d996-50ba-4769-822a-45f1c30c1dd3
Scrub started:    Tue Sep 10 18:05:51 2024
Status:           finished
Duration:         0:09:27
Total to scrub:   526.00GiB
Rate:             949.96MiB/s
Error summary:    csum=4
  Corrected:      0
  Uncorrectable:  4
  Unverified:     0

 

 

xenon-nas-diagnostics-20240910-2051.zip

Delete/restore this files from a backup and re-run the scrub to confirm 0 errors:

 

path: docker_appdata/lidarr/MediaCover/Albums/3029/cover.jpg

 

ok no problem .. still got 4 uncorrectable. FYI had a power outage so i had a unclean shutdown but here are the results.

 

to get my docker working again do you think i need to recreate my .img file?

 

UUID:             de68d996-50ba-4769-822a-45f1c30c1dd3
Scrub started:    Wed Sep 11 21:48:45 2024
Status:           finished
Duration:         0:09:28
Total to scrub:   526.00GiB
Rate:             948.29MiB/s
Error summary:    csum=4
  Corrected:      0
  Uncorrectable:  4
  Unverified:     0

 

 

xenon-nas-diagnostics-20240911-2204.zip

Edited September 12, 20241 yr by poniz

It's not showing the corrupt file(s) path, reboot and try again, scrub and new diags.

Hello again, docker still wont start, just wanted to say thank you for all your help

 

UUID:             de68d996-50ba-4769-822a-45f1c30c1dd3
Scrub started:    Thu Sep 12 13:30:26 2024
Status:           finished
Duration:         0:09:36
Total to scrub:   526.00GiB
Rate:             935.12MiB/s
Error summary:    no errors found

 

 

xenon-nas-diagnostics-20240912-1923.zip

Edited September 12, 20241 yr by poniz
uploading more recent .zip

Now that the pool corruption is resolved recreate the docker image:

 

https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file
Then:
https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications
Also see below if you have any custom docker networks:
https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks

 

 

Thank you for all your help!