Helmer

I’m happy to be a tester as well.

Great news! Any updates yet about the release date?

I'm troubleshooting a Disabled disk (on of my cache drives), but I'm unable to stop the array. I've attached the diagnostics. zbnas-diagnostics-20231019-1148.zip

Two years ago I started with 2 Basic keys on two servers. After a while I consolidated the two servers to just one and upgraded that one to a Plus key. Now I still have one Basic key which I donated to my brother in law. I found the "my Keys" button on https://connect.myunraid.net/keys, and downloaded the key that was mentioned below the Signed Out Keys (Possibly available for use on other servers). But when he tried to put that key on his USB stick, he got an error mentioning that the license is ineligible for replacement. Since I moved it to a USB stick earlier this year, but have no clue which stick it is on. What is the best way to get that license key replaced so it can be used on another server again?

I know, did that on purpose. That cache pool is only hosting my /appdata and the vm disks. But the appdata is backupped twice a week, so I can restore a backup from earlier this week. The VM disks I was able to save when the disk was outside of the cache pool (during the actions you asked me to perform). Not sure if those disks are corrupted, but it's not a big set of VM's and I can rebuild them (it's part of my IT lab environment 🙂 no nothing really important). I have formatted the cache pool and will also change it to raid1, 2 TB seems to be enough storage for everything for now.

UUID: 8b5a7d47-7fb9-430f-b38c-0b2eda367674 Scrub started: Thu May 11 17:28:57 2023 Status: finished Duration: 0:03:49 Total to scrub: 1023.00GiB Rate: 4.47GiB/s Error summary: csum=359110 Corrected: 12957 Uncorrectable: 346153 Unverified: 0

Rebooted the NAS, but VM and docker service are not starting. I do see a lot of errors in the log: May 11 17:19:37 ZBNAS kernel: BTRFS warning (device nvme1n1p1): csum failed root 5 ino 6018320 off 2222407680 csum 0x28e26401 expected csum 0x4cb2d1de mirror 1 May 11 17:19:37 ZBNAS kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 80631, gen 0 May 11 17:19:37 ZBNAS kernel: BTRFS warning (device nvme1n1p1): csum failed root 5 ino 6018320 off 2204942336 csum 0x96fcc2ea expected csum 0x969ee1c9 mirror 1 May 11 17:19:37 ZBNAS kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 80632, gen 0 May 11 17:19:37 ZBNAS kernel: BTRFS warning (device nvme1n1p1): csum failed root 5 ino 6018320 off 38928384 csum 0x54e56280 expected csum 0x2bf72d21 mirror 1 May 11 17:19:37 ZBNAS kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 80633, gen 0 May 11 17:19:37 ZBNAS kernel: BTRFS warning (device nvme1n1p1): csum failed root 5 ino 6018320 off 1740210176 csum 0x75f59339 expected csum 0x46d9cb01 mirror 1 May 11 17:19:37 ZBNAS kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 80634, gen 0 May 11 17:19:37 ZBNAS root: mount: /var/lib/docker: wrong fs type, bad option, bad superblock on /dev/loop2, missing codepage or helper program, or other error. May 11 17:19:37 ZBNAS root: dmesg(1) may have more information after failed mount system call. May 11 17:19:37 ZBNAS kernel: I/O error, dev loop2, sector 3398848 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0 May 11 17:19:37 ZBNAS kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 1, flush 0, corrupt 0, gen 0 May 11 17:19:37 ZBNAS kernel: BTRFS warning (device nvme1n1p1): csum failed root 5 ino 6018320 off 2008645632 csum 0x8ebbaacf expected csum 0x46d9cb01 mirror 1 May 11 17:19:37 ZBNAS kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 80635, gen 0 May 11 17:19:37 ZBNAS kernel: I/O error, dev loop2, sector 3923136 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0 May 11 17:19:37 ZBNAS kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 2, flush 0, corrupt 0, gen 0 May 11 17:19:37 ZBNAS kernel: BTRFS error (device loop2): failed to read block groups: -5 May 11 17:19:37 ZBNAS kernel: BTRFS error (device loop2): open_ctree failed May 11 17:19:37 ZBNAS root: mount error

Between the unassign and re-assign I was able to 'secure' that data that was on the cache pool, so it took me some time to perform the re-assign and start array steps, but here you have the new diags. After the re-assign (and starting the array) the cache pool reported that it was working again. Now rebooting the NAS to see if everything boots up nicely. I'll update this post when the NAS has completed the boot succesfully (or not). zbnas-diagnostics-20230511-1701.zip

root@ZBNAS:~# btrfs-select-super -s 1 /dev/nvme1n1p1 checksum verify failed on 1299759104 wanted 0xd465fb25 found 0x47a2dcda checksum verify failed on 421670961152 wanted 0xc64603a8 found 0xc95e61cb checksum verify failed on 1816412160 wanted 0xb94f3c8c found 0x13339a15 checksum verify failed on 421646434304 wanted 0x19aa0693 found 0x0b0110ca checksum verify failed on 421591498752 wanted 0xdece85af found 0x2187a1d3 checksum verify failed on 421406703616 wanted 0x14296af2 found 0xac0bef74 checksum verify failed on 1886011392 wanted 0x7b8ecefd found 0x46da7735 checksum verify failed on 1867169792 wanted 0x8ab7b210 found 0x6b55c8fd checksum verify failed on 1516126208 wanted 0x011be66e found 0x064c0005 checksum verify failed on 421884493824 wanted 0x573e1203 found 0xd40c399a checksum verify failed on 421884510208 wanted 0x810cc1b7 found 0xd27d5687 checksum verify failed on 420328275968 wanted 0x99a9e9bc found 0x389121ca checksum verify failed on 421591515136 wanted 0xda20563d found 0x60a15cde checksum verify failed on 421781471232 wanted 0x41d17f4f found 0x203dfd7d checksum verify failed on 421670977536 wanted 0x13179228 found 0xd0922978 checksum verify failed on 2162016256 wanted 0xa040bce0 found 0x430ce001 checksum verify failed on 421576032256 wanted 0xc0e490a6 found 0xb79f4696 checksum verify failed on 421671190528 wanted 0xef6b14e0 found 0xbf88a46d checksum verify failed on 421221138432 wanted 0x6492f8a7 found 0xfc7cba01 checksum verify failed on 415100780544 wanted 0x55f65470 found 0xb96af1f3 checksum verify failed on 414998528000 wanted 0x1832c57f found 0x9c8cce3b checksum verify failed on 1927266304 wanted 0xf3f10cce found 0xd0166667 checksum verify failed on 1849573376 wanted 0xad0bf53c found 0xd5b4bb1c checksum verify failed on 421671501824 wanted 0x318a24b4 found 0xca34da93 checksum verify failed on 421788499968 wanted 0x9e9ec89d found 0x9ff202d5 checksum verify failed on 421948309504 wanted 0xf44bcac0 found 0x1cbf7cb2 checksum verify failed on 420552065024 wanted 0x2f46dbb0 found 0x56a12873 checksum verify failed on 420041588736 wanted 0x582d0ad1 found 0xb9a05c94 checksum verify failed on 414643486720 wanted 0x9e364213 found 0x40d77b05 checksum verify failed on 421078859776 wanted 0xb3bd659b found 0xaa7668a3 checksum verify failed on 421068668928 wanted 0xb0fa1bd9 found 0x3feb86cf checksum verify failed on 421068914688 wanted 0x10fadd3f found 0xa3a11b43 checksum verify failed on 1923645440 wanted 0x277c361b found 0x200b0f4f checksum verify failed on 420052418560 wanted 0x6e90d939 found 0x75a35a04 checksum verify failed on 421068996608 wanted 0x707d8ca3 found 0x24458d28 checksum verify failed on 421623431168 wanted 0x93185c0b found 0x159efc1d checksum verify failed on 415573114880 wanted 0x3052b624 found 0xdbe978e2 checksum verify failed on 414604705792 wanted 0x787232bd found 0x02a0ca26 checksum verify failed on 1845215232 wanted 0x9538a137 found 0x3f6e52e4 checksum verify failed on 414799347712 wanted 0xa6984eee found 0x5820df1c checksum verify failed on 415573131264 wanted 0x91cdbf5a found 0xb1c9f157 checksum verify failed on 415580962816 wanted 0xb86acd59 found 0xf72287ec checksum verify failed on 414799364096 wanted 0xca5d91ed found 0xa0fe3fee checksum verify failed on 414799380480 wanted 0x16b5be7a found 0xbc434744 checksum verify failed on 414604804096 wanted 0x4e8947a1 found 0xf5522091 checksum verify failed on 414799396864 wanted 0x1e8376d8 found 0x180d2de4 checksum verify failed on 421592268800 wanted 0x4e53b91b found 0xa8c3bed3 checksum verify failed on 421592301568 wanted 0xaa36c435 found 0x03d3b97a checksum verify failed on 415574638592 wanted 0x59f15708 found 0x20e0df3e checksum verify failed on 415574835200 wanted 0xb67ccc2a found 0x3013e72c checksum verify failed on 414799429632 wanted 0x1615ddfb found 0x428e20a2 checksum verify failed on 421584502784 wanted 0xfbce1b1a found 0xc0bc8694 checksum verify failed on 421458657280 wanted 0x19d73196 found 0x13825018 checksum verify failed on 415254822912 wanted 0x9d937349 found 0x0a5a7aa1 checksum verify failed on 421671665664 wanted 0x5bfe79eb found 0x8d682a06 checksum verify failed on 421592334336 wanted 0x16350f94 found 0x4d474947 checksum verify failed on 421221400576 wanted 0xd91c7663 found 0xab9b2ff9 checksum verify failed on 415584616448 wanted 0x9c086e31 found 0x3eb3164f checksum verify failed on 421594808320 wanted 0x86470ca4 found 0x910c25e1 checksum verify failed on 415584649216 wanted 0x7279e46d found 0x9f25ce33 checksum verify failed on 415584681984 wanted 0x61bcc717 found 0x6478cb23 checksum verify failed on 2022965248 wanted 0x1543c82c found 0xd86807d6 checksum verify failed on 1244151808 wanted 0xd3e29c74 found 0x9c6c9cf0 checksum verify failed on 421302124544 wanted 0x7dcc1e3f found 0x55ff9221 checksum verify failed on 1671987200 wanted 0x13951b2f found 0x0db377af checksum verify failed on 1788919808 wanted 0x5d81243c found 0xa338812e checksum verify failed on 421601869824 wanted 0x02245ac7 found 0xb77f5948 checksum verify failed on 421601886208 wanted 0x8291c629 found 0x85cbe7f6 checksum verify failed on 420379148288 wanted 0x595a869f found 0xc9c0cb39 checksum verify failed on 420204888064 wanted 0x28d3a4c7 found 0x184886d9 checksum verify failed on 420339695616 wanted 0x07eb80eb found 0x78127689 checksum verify failed on 420379164672 wanted 0x6087a109 found 0xa6278ec4 checksum verify failed on 421633081344 wanted 0x61c67716 found 0x0a73bf15 checksum verify failed on 1906950144 wanted 0x61fe4f0f found 0x2bb82be7 checksum verify failed on 421165268992 wanted 0x3083eea8 found 0xd80eadde checksum verify failed on 414641225728 wanted 0x914a1559 found 0x8f5e1d00 checksum verify failed on 414643339264 wanted 0xd05040e8 found 0x5fc16bbe checksum verify failed on 1844756480 wanted 0xf5d5d2d9 found 0x693cbafe checksum verify failed on 420863655936 wanted 0x72f9970e found 0xb36a55af checksum verify failed on 414643421184 wanted 0x52506264 found 0x77690936 checksum verify failed on 421433589760 wanted 0x69ed14a8 found 0x825714f9 checksum verify failed on 420061380608 wanted 0x27fb43d7 found 0x5875d859 checksum verify failed on 415592038400 wanted 0xf8e2ba89 found 0x06579b79 checksum verify failed on 415592071168 wanted 0x57b3d5bd found 0x483cad1a checksum verify failed on 421661523968 wanted 0x6e085b5f found 0x628aac69 checksum verify failed on 420562501632 wanted 0x621c3310 found 0xfdc374d9 checksum verify failed on 1910734848 wanted 0x99137b8c found 0x605a5e89 checksum verify failed on 2174320640 wanted 0xaae601bf found 0x7695fdc4 checksum verify failed on 421628461056 wanted 0xa014a180 found 0xbdb8b382 checksum verify failed on 414633443328 wanted 0xa7a3fa68 found 0x56e39e2c checksum verify failed on 421347229696 wanted 0x127bc17d found 0xfcd95497 checksum verify failed on 421134319616 wanted 0x0ae4ba36 found 0x99f439ca checksum verify failed on 1269399552 wanted 0xced4c3a1 found 0xd11901f3 checksum verify failed on 421511544832 wanted 0xb7184fe8 found 0x7bf20822 checksum verify failed on 420271472640 wanted 0x122273ce found 0xe6861fba checksum verify failed on 1913225216 wanted 0xe2c5b2e7 found 0xa2e97bf5 checksum verify failed on 1913274368 wanted 0x0d8b6b62 found 0x4ed3db76 checksum verify failed on 2076622848 wanted 0x34dac554 found 0x5c425262 checksum verify failed on 420195139584 wanted 0x96e85d93 found 0x60b19d2a checksum verify failed on 420532240384 wanted 0xfa5fc983 found 0x0a80be8a checksum verify failed on 420504567808 wanted 0x28de045f found 0x9330ab56 checksum verify failed on 420504600576 wanted 0xf051439e found 0x8508ac7f checksum verify failed on 420335763456 wanted 0x0dae4620 found 0xe85f1880 checksum verify failed on 420532256768 wanted 0x061858ac found 0x34079075 checksum verify failed on 421633163264 wanted 0x2b9f756e found 0x4c42b9cd checksum verify failed on 421628575744 wanted 0x6d03d9cf found 0x533f22a1 checksum verify failed on 1913454592 wanted 0x49f18ddf found 0xebacb751 checksum verify failed on 420944936960 wanted 0x03812c54 found 0xc4c3ebec checksum verify failed on 420939825152 wanted 0x88a6818b found 0xf6cc5a90 checksum verify failed on 421603753984 wanted 0xd71165a9 found 0x0713dc2b checksum verify failed on 420391960576 wanted 0xaa90e912 found 0xaf58fc74 checksum verify failed on 421347262464 wanted 0xe83fce69 found 0x489ceda3 checksum verify failed on 421661622272 wanted 0xec6dd614 found 0x12f14713 checksum verify failed on 421660901376 wanted 0x4b7b5ad0 found 0x805fe65a checksum verify failed on 421139578880 wanted 0x01875a4b found 0x6a586034 checksum verify failed on 421661638656 wanted 0xb09aa9ac found 0xe06ed8fe checksum verify failed on 421601968128 wanted 0x10402d68 found 0x6ffc804c checksum verify failed on 421633245184 wanted 0x1c39fd89 found 0x476b63e2 checksum verify failed on 415230541824 wanted 0x22f87059 found 0x36709939 checksum verify failed on 421661671424 wanted 0x30ab28a7 found 0xe6a3297c checksum verify failed on 421601984512 wanted 0x9fdf48cc found 0xc5022eb5 checksum verify failed on 421677547520 wanted 0x4edcdc27 found 0x96955433 checksum verify failed on 421633327104 wanted 0x8464365f found 0x8d3c8a69 checksum verify failed on 421633343488 wanted 0x20b3d1f6 found 0x77285e63 checksum verify failed on 421602033664 wanted 0x1d2db897 found 0xd0188bd8 checksum verify failed on 421385961472 wanted 0xf61d72d0 found 0x7b1f1485 checksum verify failed on 421675483136 wanted 0xf7f5b9a6 found 0xe0157a39 checksum verify failed on 421633376256 wanted 0x25d19400 found 0x04719ec2 checksum verify failed on 421629673472 wanted 0xdbc78ebe found 0x45480996 checksum verify failed on 421633392640 wanted 0x9b730c77 found 0xe00dc8a6 checksum verify failed on 421633409024 wanted 0x8aea154d found 0x1e831409 checksum verify failed on 421629788160 wanted 0x32b4a8d9 found 0x4d9789a0 checksum verify failed on 421547933696 wanted 0x90c39a53 found 0xffb3ff3d checksum verify failed on 421229543424 wanted 0x8b653bf4 found 0x19c16685 checksum verify failed on 2156232704 wanted 0xffb76915 found 0xc75b7c42 checksum verify failed on 421677563904 wanted 0xff12648d found 0x0b5c86ec checksum verify failed on 421486575616 wanted 0xee4f5c48 found 0x4ba46013 checksum verify failed on 421665931264 wanted 0xbf7e41cf found 0x6ada00d4 checksum verify failed on 421665947648 wanted 0xf6c8ea0a found 0xed69b8f9 checksum verify failed on 421665964032 wanted 0xf644f775 found 0x43d93890 checksum verify failed on 421665980416 wanted 0x63dac4c9 found 0x87732c68 checksum verify failed on 421662277632 wanted 0xfdec4d59 found 0x90d4528e checksum verify failed on 421665996800 wanted 0x486143ee found 0xd8eec1c0 checksum verify failed on 420297474048 wanted 0x5a5962a7 found 0x68fdd02c checksum verify failed on 421548064768 wanted 0xbe8f389f found 0x0692f589 checksum verify failed on 421677580288 wanted 0x7ad81e4a found 0x1eb6e496 checksum verify failed on 421677613056 wanted 0xc9bb1915 found 0x32037e04 checksum verify failed on 421677629440 wanted 0xf68f552a found 0xbad1fbc7 checksum verify failed on 421666062336 wanted 0x6df3d994 found 0xd3a8c7a8 checksum verify failed on 421663031296 wanted 0xcf7e0371 found 0xbd43d610 checksum verify failed on 421548163072 wanted 0x56f8daca found 0x8b4aed7b checksum verify failed on 421548212224 wanted 0x3a224ea7 found 0x71e9bd7d checksum verify failed on 421666095104 wanted 0x298b4cd3 found 0x38534953 checksum verify failed on 421548244992 wanted 0x0416829c found 0x70e19040 checksum verify failed on 421548261376 wanted 0x20abe61d found 0xd9d349de checksum verify failed on 421677645824 wanted 0x357e3348 found 0xf76f24ea checksum verify failed on 421666111488 wanted 0x11da7a3d found 0x8cc58b39 checksum verify failed on 421676400640 wanted 0x2b474764 found 0x4806ce4f checksum verify failed on 421677711360 wanted 0xcc57e1a7 found 0xc7ab08ec checksum verify failed on 421601558528 wanted 0x2604ee91 found 0x6976e7d3 checksum verify failed on 421545295872 wanted 0x2a838ea9 found 0x15d72ba1 checksum verify failed on 421302829056 wanted 0x6cbe7e30 found 0x3ea653cf using SB copy 1, bytenr 67108864 root@ZBNAS:~#

Earlier today I noticed that my UNRAID server was not accessable anymore (not through the webUI, not through SMB and even with the PING command I did not get a reply). That was the reason I decided to (shortly) press the power button, to perform a clean shutdown. After around 15 minutes I noticed that nothing happened. I decided to press the power button longer, which made the machine turn off. When I turned it on again, the UI became available after a few minutes and I directly created the diagnostics file, that I have attached below. After this I noticed that one of the two SSD's was dropped out of the cache pool and was shown at the Unassigned drives. The other disk (WDC NVME SSD) was still part of the cache pool. I stopped the pool and tried to select the missing disk and started the pool. It was shown as 'unmountable - no file system', I have no clue how to fix this. I hope you can assist me. Let me know if I need to add more info. zbnas-diagnostics-20230510-2317.zip