WEHA

Looks like this needs to be fixed, but tbh I don't understand what I'm reading... From what I gather it is indeed partly RAID1? Overall: Device size: 7.28TiB Device allocated: 1.92TiB Device unallocated: 5.36TiB Device missing: 1.82TiB Used: 1.82TiB Free (estimated): 2.73TiB (min: 2.73TiB) Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Data Metadata System Id Path RAID1 RAID1 RAID1 Unallocated -- -------------- --------- -------- --------- ----------- 3 /dev/nvme0n1p1 250.00GiB 2.00GiB - 1.57TiB 4 /dev/nvme0n1p1 43.00GiB - - 1.78TiB 2 /dev/nvme1n1p1 980.00GiB 3.00GiB 32.00MiB 879.99GiB 1 missing 687.00GiB 1.00GiB 32.00MiB -688.03GiB -- -------------- --------- -------- --------- ----------- Total 980.00GiB 3.00GiB 32.00MiB 3.54TiB Used 930.08GiB 1.13GiB 176.00KiB Balance status: No balance found on '/mnt/cache'

So I tried figuring it out (made a backup first): removing the "faulty" cache disk results in not being able to read the cache assigning 1n1 from slot 2 to slot 1 has the same result reassigning 0n1 to slot 1 and 1n1 to slot 2 which then says 0n1 will lose all data: now I can read the cache However: now it says 4TB protected, but these are 2 2TB drives? Usually when this happens the size changes when it's "recovering" but this does not happen now, it remains the same. How can I correct this? (this is balancing right?)

I went back to unraid and saw this meesgae: Unraid Cache disk message: 16-12-2019 20:31 Warning [TOWER] - Cache pool BTRFS missing device(s) Samsung_SSD_970_EVO_2TB (nvme0n1) So I'm assuming I better not use disk 0n1, can I just unassign 0n1 and just use 1n1? EDIT2: according to this I can just unassign one disk and continue on, can you just tell me what the safest option is? https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?tab=comments#comment-480418 EDIT: With mount -o degraded,usebackuproot,ro /dev/sdX1 /x I can read 1n1 so it seems fine.

Ok so I have now reset the system and have the array in a stopped state. The cache disks appear normal but can I trust unraid to mount it properly / fix it properly or how can I check that it will or will not mount properly? thank you so far for your assistance.

oh dear.. I didn't know there was a problem like this... But how do I continue now? Is there any way to get it properly restarted? Or do I just force a reboot? Stopping / Killing anything which is running on the cache just hangs, nothing stops or gets killed.

tower-syslog-20191216-1145.zip

So 2 days ago I had to change the network configuration (enable VLAN) so I decided to update as well. From the moment I disabled the array, 1 nvme cache drive was "missing"... I had this before but only after a reboot, I already updated BIOS & have new nvme ssd's (Intels before, now Samsung) After a cold boot, the cache drive was back, reassigned and unraid updated from 6.7.0 to 6.7.2. Everything went well until a few hours ago I could not connect to one VM and SMB shares do not seem to work either. I connected on SSH to find the load above 280 (now 303 and steadily rising). Processes itself don't seem to use that much of the cpu so I'm guessing IO. Every tool checking IO status hangs so unable to see what is going on. On the GUI it said mover running, got that to stop, then it said BTRFS operation running. Found this thread and executed btrfs balance status /mnt/cache --> No balance found on '/mnt/cache' I also found a Kernel oops in dmesg (below) His problem was solved after a reboot, but he did not have a cache pool. Am I good to execute a reboot here? I won't lose my data? Diagnostics download is running, not sure if it will finish... [71615.611132] BUG: unable to handle kernel NULL pointer dereference at 0000000000000080 [71615.611268] PGD 8000000f138a1067 P4D 8000000f138a1067 PUD f68c96067 PMD 0 [71615.611341] Oops: 0000 [#1] SMP PTI [71615.611431] CPU: 0 PID: 26293 Comm: fstrim Not tainted 4.19.56-Unraid #1 [71615.611515] Hardware name: ASUSTeK COMPUTER INC. P10S WS/P10S WS, BIOS 3402 07/12/2018 [71615.611590] RIP: 0010:btrfs_trim_fs+0x166/0x369 [71615.611658] Code: 00 00 48 c7 44 24 38 00 00 00 00 49 8b 45 10 48 c7 44 24 40 00 00 00 00 48 c7 44 24 30 00 00 00 00 48 89 44 24 20 48 8b 43 68 <48> 8b 80 80 00 00 00 48 8b 80 f8 03 00 00 48 8b 80 a8 01 00 00 0f [71615.611873] RSP: 0018:ffffc9002eaa7c90 EFLAGS: 00010297 [71615.611942] RAX: 0000000000000000 RBX: ffff8890339ae400 RCX: 0000000000000000 [71615.612014] RDX: ffff888fff1e9c00 RSI: 00000192c100d000 RDI: ffff88901ee0e378 [71615.612086] RBP: 00000000ffffffe4 R08: 0000606fc0a09bb0 R09: ffffffff8122acea [71615.612158] R10: ffffea000d907200 R11: ffff88903f220b80 R12: ffff88901ee0e000 [71615.612230] R13: ffffc9002eaa7d20 R14: 0000000000000000 R15: ffff8887bec88000 [71615.612302] FS: 000014a5d096c780(0000) GS:ffff88903f200000(0000) knlGS:0000000000000000 [71615.612375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [71615.612468] CR2: 0000000000000080 CR3: 000000029787c001 CR4: 00000000003626f0 [71615.612553] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [71615.612625] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [71615.612697] Call Trace: [71615.612763] btrfs_ioctl_fitrim.isra.7+0xfe/0x135 [71615.612832] btrfs_ioctl+0x4f6/0x28ad [71615.612900] ? queue_var_show+0x12/0x15 [71615.612967] ? _copy_to_user+0x22/0x28 [71615.613035] ? cp_new_stat+0x14b/0x17a [71615.613102] ? vfs_ioctl+0x19/0x26 [71615.613167] vfs_ioctl+0x19/0x26 [71615.613233] do_vfs_ioctl+0x526/0x54e [71615.613300] ? __se_sys_newfstat+0x3c/0x5f [71615.613368] ksys_ioctl+0x39/0x58 [71615.613434] __x64_sys_ioctl+0x11/0x14 [71615.613524] do_syscall_64+0x57/0xf2 [71615.613604] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [71615.613674] RIP: 0033:0x14a5d0a9e397 [71615.613741] Code: 00 00 90 48 8b 05 f9 2a 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d c9 2a 0d 00 f7 d8 64 89 01 48 [71615.613956] RSP: 002b:00007fff498e2be8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [71615.614029] RAX: ffffffffffffffda RBX: 00007fff498e2d40 RCX: 000014a5d0a9e397 [71615.614123] RDX: 00007fff498e2bf0 RSI: 00000000c0185879 RDI: 0000000000000003 [71615.614208] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000416bb0 [71615.614279] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000415aa0 [71615.614351] R13: 0000000000415a20 R14: 0000000000415aa0 R15: 000014a5d096c6b0 [71615.614423] Modules linked in: vhost_net tun vhost tap kvm_intel kvm xt_nat veth xt_CHECKSUM ipt_MASQUERADE ipt_REJECT ip6table_mangle ip6table_nat nf_nat_ipv6 iptable_mangle iptable_nat nf_nat_ipv4 nf_nat ip6table_filter ip6_tables iptable_filter ip_tables xfs md_mod nfsd lockd grace sunrpc ipmi_devintf bonding igb i2c_algo_bit x86_pkg_temp_thermal intel_powerclamp coretemp hid_logitech_hidpp wmi_bmof mxm_wmi crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper intel_cstate intel_uncore pcc_cpufreq intel_rapl_perf i2c_i801 ahci ie31200_edac libahci video acpi_pad button hid_logitech_dj i2c_core nvme nvme_core aacraid cp210x usbserial cdc_acm wmi backlight [last unloaded: kvm] [71615.615190] CR2: 0000000000000080 [71615.615725] ---[ end trace eb0e9ccf73a2e8b9 ]---

It's normalized now. Too bad the gui doesn't show the "in progress" state. Anyway, thank you for your assistance!

I wasn't aware of a new version until today since it did not (still isn't) mention this in the GUI like it did last time. Updating is the last on my todo list So next problem, I rebooted and offcourse my second cache drive got "undetected" again. Updated ssd firmware this time (bios this morning) hoping this fixes it. Reboot, reassigned second drive... and now it's showing as RAID 0 in terms of space. Tried rebalance with -dconvert=raid1 -mconvert=raid1 but did nothing. Do I have to convert it to single first? Dashboard shows 1.5TB size, 785GB in use Status (these are 2 x 1TB nvme SSD's fyi): btrfs filesystem df: Data, RAID1: total=732.00GiB, used=731.15GiB System, RAID1: total=32.00MiB, used=144.00KiB Metadata, RAID1: total=2.00GiB, used=908.31MiB GlobalReserve, single: total=512.00MiB, used=0.00B btrfs balance status: No balance found on '/mnt/cache'

So I did the "have you tried turrning it off and on again" scenario, shares are back! Now I'm getting to work on the dockers and vm's, will update soon.

Everything is now copied back, stopped and started array: no exportable shares. New diag attached tower-diagnostics-20180414-1538.zip Something strange though when I "ls /mnt": 16K drwxrwxrwx 1 nobody users 106 Apr 14 15:32 cache/ 0 drwxrwxrwx 3 nobody users 19 Apr 14 10:00 disk1/ 0 drwxrwxrwx 4 nobody users 43 Apr 14 10:00 disk2/ 0 drwxrwxrwx 3 nobody users 19 Apr 14 10:00 disk3/ 0 drwxrwxrwx 3 nobody users 19 Apr 14 10:00 disk4/ 0 drwxrwxrwx 11 nobody users 167 Apr 14 10:00 disk5/ 0 drwxrwxrwx 5 nobody users 100 Apr 14 15:35 disks/ ? d????????? ? ? ? ? ? user/ 0 drwxrwxrwx 1 nobody users 19 Apr 14 10:00 user0/ /bin/ls: cannot access 'user': Transport endpoint is not connected When I "ls user0", those contain the non-cache shares

Attached tower-diagnostics-20180414-1532.zip

I mean the share configuration. When I go to shares no in the top menu, it's empty

There was 100GB of free space? How is there no space left? Pool has been formatted, restoring data currently. Shares are emtpy but appdata & system was on my cache. This will restore itself once I restart unraid?

Thank you for your quick reply. I was able to copy the data earlier, I hope there is no corruption. What are ENOSPC errors exactly? Can't seem to find a simple description What would be the best way to format the drivers: - using the gui: just delete the partition and re-add as cache? - is a full wipe necesarry? - a btrfs command thanks!

Hello So I had this issue where unraid started throwing errors, I believe because of cache drive disconnect (if that's possible, doesn't it run from the usb drive?) I set the array to not start on boot but because of the errors it apparently didn't save this preference. Anyway, after rebooting (not detecting the second cache drive) it started the array, without the second cache drive. (raid1) It was mounted read only, I guess because of the disconnect earlier. I rebooted to get the second drive detected again and I tried to re-add the drive in unraid. It remained mounted read only and btrfs was still saying that it was missing the drive, even though it was looking correct in the unraid gui. After searching the internet I found the command to replace the drive in the raid 1 (adding redetected drive and removing missing drive). But I'm still at the problem where it says the drive is mounted read only. When I execute mount it says rw, so it's btrfs not allowing me to write. The only thing I could find for this situation is that there needs to be a kernel patch to get this working. I'm not familiar how to check or install this patch in unraid. Source: https://www.mail-archive.com/search?l=linux-btrfs%40vger.kernel.org&q=subject:"raid1\%3A+cannot+add+disk+to+replace+faulty+because+can+only+mount+fs+as+read\-only."&o=newest https://www.mail-archive.com/[email protected]/msg60979.html Any suggestions? EDIT: added diagnostics tower-diagnostics-20180414-0850.zip

Would it be possible to add an included shares like the excluded shares? Or can you tell me what I need to add to my manual change in the smb config? thanks!

I have a problem that a share is not detected. It is one that is not exported via the unraid gui but rather manual config change of samba. The reason for this is that it's a users share where the subfolders have different permissions which is not possible from the gui. Is there a way to enable the recycle bin for this share? The plugin does not see it. thanks!

Ok well that's not that interesting I'm sure there are reasons for this... anyway, thanks for assisting me!

Just as I suspected, but then unraid showing it protected is a bug then no? Balance status: Data, single: total=871.00GiB, used=801.92GiB System, single: total=32.00MiB, used=128.00KiB Metadata, single: total=4.00GiB, used=1.25GiB GlobalReserve, single: total=512.00MiB, used=0.00B So nothing is raid 1...

Alright Thanks, 2 questions though. If you don't mind. Do I need to move the data first or can I be sure not to loose any data with the conversion? This is not mentioned in the post so I'd like to be sure, I can imagine btrfs is smart enough to do this when enough space is available. Reading your faq post properly I see that metadata can be in raid 1 mode separately, what exactly does this mean -> are the files protected or not? Unraid indicates they are but I would think metadata is not enough, this confuses me as you would surely understand. thanks again!

Do I have to use : -dconvert=single -mconvert=raid1 single as in single disk or as in raid 0? thanks!

Fair enough but why do the shares indicate that they are protected?

So I was adding a pcie card in the unraid system and booted it back up. Array was on auto start so this started but I noticed one of the cache disks was missing. Not sure why this would be allowed to happen as an array disk missing would prevent the array from starting, is this normal behavior? Anyway, I removed the card and I got the cache disk back and it was balancing. The strange was that the used data status was dropping so it scared me at first but when checking with du, nothing changed. The balance finished and now I have a 2TB cache disk where this should be 1TB (2 x 1TB nvme sdd). Cache shares are still showing green meaning protected but when I check balance on the cache page it says no balance found. Stopping and starting the array does not change anything. So 2 questions: - How do I fix this? unless the only way is remaking the cache... - How can I make unraid not mount the cache when one of the disks is missing? Diagnostics attached tower-diagnostics-20170730-1757.zip

So, I made a cache with 2 ssd's of 120 GB (112GB irl) I made a VM with a 100GB img file. When I put the VM on the cache, everything is fine. When things are being written to the image the cache fills up but the img file is still 100GB If I move it to the array and back to the cache I can use it again until the remaining 12GB get's written again. The img file is the only thing on the cache: /dev/sde1 112G 112G 72K 100% /mnt/cache 100G -rwxrwxrwx 1 root users 100G Jul 22 10:01 vdisk1.img* Can anyone explain to me why this is happening and how I can stop it? thanks!