Plak

I seem to be having problems with unbalanced and ZFS. I moved all my disks to ZFS one by one. I did move the majority of the data around with unbalanced. If you have ZFS Master installed you can see that some top level folders on my disks that were moved by unbalanced are not properly set up in ZFS on that disk. They are missing from the output of the command "zfs list" as well. It looks like unbalanced does not create the top level folders in zfs as unraid is expecting it. Mover seems to do that fine. If a top level folder was present on the destination disk before unbalance move was started, everything seems fine. If unbalance needs to create the top level folder, then the folder does not show up properly in ZFS Master. Here is a screenshot showing the problem: everything on disk8 and the folder data on disk7 was moved there by unbalanced. Any ideas on how to fix this beside evacuating data each disk with issues by using mover?

I want to enable remote write reception on my prometheus. I added the flag to post arguments --web.enable-remote-write-receiver This leads to the following error: level=ERROR source=main.go:593 msg="Error loading config (--config.file=prometheus.yml)" file=/prometheus/prometheus.yml err="open prometheus.yml: no such file or directory" Appdata is set to /mnt/user/appdata/prometheus/data Config is set to /mnt/user/appdata/prometheus/etc Container runs fine without the post argument. How can I get prometheus running with remote write reception enabled?

I added more disks to the array and was stuck with 15MB/s write speed on the array for 2 weeks now. Today I looked through the syslog and I found errors like "BAR 7: no space for [mem size 0x00100000 64bit pref]". A search led me to these threads: Followed the advice and found SR-IOV Support in BIOS Disabled Enabled it and put in pci=realloc=off into /boot/syslinux.cfg- Array is back to high speed. Parity sync running at 200-280 MB/S. Once the sync is done I will reboot without pci=realloc=off and see if the BIOS setting alone is helping.

My server was in the same state (hanging at the "Starting diagnostics collection" after reboot command) and letting it sit there for around 30 minutes resulted in a proper restart and a diagnostic zip file as well as the syslog files in the /boot/log directory. I think the process of sanitizing the diags leads to long gathering times when mover logging is enabled and a lot of moving actions were performed. (I did resize the /var/log because the syslogs filled up the initial 128MB with mover log entries).

The strangest thing happened. The parity sync speed recovered after I posted the diag. Currently it runs with up to 275MB/s. Preclear on the other drive is still stuck at 25MB/s.

Hi, I was running a preclear on 2 drives as they suddenly got a severe drop in write performance. Write speed dropped from around 200MB/s to 25MB/s on both drives. IIRC there was an update for "Unassigned Devices" on that day (2024-12-09). Since then I have added one of those as parity to my array and the parity rebuild is going ahead with 25MB/s at best times. The other one is in a preclear operation at 25MB/s. unraid-01-diagnostics-20241211-1123.zip

I have an update to this problem. The server recently went into throwing lots of those errors on all disks. even leading to read errors on the array and failing preclearing runs. Restarting the server or shutting it down for a couple minutes does not get rid of the errors. "Refresh Topology" did not help either. Switching the server off for a few hours (Power connected and IPMI online) and back on seems to solve the problem until it starts occurring again. Preclears are running without any issues again. I am still at a loss as to what causes the errors to start. For now I am running the server with all fans on max to make sure temperature is not the issue. I will provide updates.

UFS Explorer did work out. I got an additional drive to recover the data to from the formated one. Most of the data was recovered with correct folder and file names. only around 250GB out of 17TB were "lost-files". I got both disks back in the system. Disk1 was precleared and added to the array. I was able to restore the data from the new disk to the array at the same time. Data copy to the array with the emulated disk1 had no speed issues. (single disk -> cache at 140MB/s, cache -> array at 120 MB/s, at 95% fill rate of the array the speed went down to around 70MB/s which is to be expected) Everything is back up and running. The new disk is in preclearing process to add it to the array.

It looks like "Refresh Topology" in the HBA BIOS solved the problem. (https://www.h3c.com/en/d_202207/1655193_294551_0.htm#_Ref533597481) Since that action I have done a lot of heavy writing operations on the system without increase in UDMA CRC Errors on any attached disk. Operations run so far: Rebuild of parity moving 17,5 GB of Data from single disk (new disk6) to Array (Array emulating disk1). preclearing disk1 and adding it to array with rebuild of its contents

Strangely the system is running fine currently. Only thing I did was to gather the HBA information in the BIOS. Since that I do not see any increase in UDMA CRC Errors on the disks and the preclear is doing fine. I did see strange behavior of smartctl in the preclear (18TB TOSHIBA MG09ACA18TE disk): Oct 20 22:14:15 preclear_disk_SN_6258: Preclear Disk Version: 1.0.30 Oct 20 22:14:16 preclear_disk_SN_6258: Disk size: 18000207937536 Oct 20 22:14:16 preclear_disk_SN_6258: Disk blocks: 4394582016 Oct 20 22:14:16 preclear_disk_SN_6258: Blocks (512 bytes): 35156656128 Oct 20 22:14:16 preclear_disk_SN_6258: Block size: 4096 Oct 20 22:14:16 preclear_disk_SN_6258: Start sector: 0 Oct 20 22:14:21 preclear_disk_SN_6258: Pre-read: pre-read verification started 1 of 5 retries... Oct 21 02:36:48 preclear_disk_SN_6258: Pre-Read: progress - 25% read @ 277 MB/s Oct 21 07:23:18 preclear_disk_SN_6258: Pre-Read: progress - 50% read @ 246 MB/s Oct 21 12:59:07 preclear_disk_SN_6258: Pre-Read: progress - 75% read @ 198 MB/s Oct 21 20:39:07 preclear_disk_SN_6258: Pre-Read: elapsed time - 22:24:43 Oct 21 20:39:07 preclear_disk_SN_6258: Pre-read: pre-read verification completed! Oct 21 20:39:08 preclear_disk_SN_6258: Zeroing: zeroing the disk started 1 of 5 retries... Oct 21 20:39:08 preclear_disk_SN_6258: Zeroing: emptying the MBR. Oct 22 01:00:29 preclear_disk_SN_6258: Zeroing: progress - 25% zeroed @ 278 MB/s Oct 22 05:46:46 preclear_disk_SN_6258: Zeroing: progress - 50% zeroed @ 244 MB/s Oct 22 11:21:50 preclear_disk_SN_6258: Zeroing: progress - 75% zeroed @ 201 MB/s Oct 22 19:01:35 preclear_disk_SN_6258: killing smartctl with pid 26573 - probably stalled ... Oct 22 19:01:35 preclear_disk_SN_6258: Pause (smartctl run time: 34s) Oct 22 19:01:35 preclear_disk_SN_6258: Paused Oct 22 19:01:56 preclear_disk_SN_6258: killing smartctl with pid 26573 - probably stalled ... Oct 22 19:02:18 preclear_disk_SN_6258: killing smartctl with pid 26573 - probably stalled ... Oct 22 19:02:40 preclear_disk_SN_6258: killing smartctl with pid 26573 - probably stalled ... Oct 22 19:03:02 preclear_disk_SN_6258: killing smartctl with pid 26573 - probably stalled ... Oct 22 19:03:23 preclear_disk_SN_6258: killing smartctl with pid 26573 - probably stalled ... Oct 22 19:03:45 preclear_disk_SN_6258: killing smartctl with pid 26573 - probably stalled ... Oct 22 19:04:07 preclear_disk_SN_6258: Resumed Oct 22 19:04:09 preclear_disk_SN_6258: Zeroing: zeroing the disk completed! Oct 22 19:04:10 preclear_disk_SN_6258: Signature: writing signature... Oct 22 19:04:12 preclear_disk_SN_6258: Signature: verifying Unraid's signature on the MBR ... Oct 22 19:04:13 preclear_disk_SN_6258: Signature: Unraid preclear signature is valid! Oct 22 19:04:14 preclear_disk_SN_6258: Post-Read: post-read verification started 1 of 5 retries... Oct 22 19:04:14 preclear_disk_SN_6258: Post-Read: verifying the beginning of the disk. Oct 22 19:04:14 preclear_disk_SN_6258: Post-Read: verifying the rest of the disk. Oct 22 23:26:41 preclear_disk_SN_6258: Post-Read: progress - 25% verified @ 273 MB/s Oct 23 04:13:16 preclear_disk_SN_6258: Post-Read: progress - 50% verified @ 248 MB/s I will get back with updates.

I have a huge problem and do not know how to solve it. I bought a refurbished Supermicro Server with a LSI SAS 9400-16i HBA. (01/24) I got several 18TB Hitachi drives at different times to grow the system. (4 in 07/23, 2 in 2/24, 1 in 7/24 and 1 in 8/24) (2 RMAs due to sector relocates) started with 3 disks + parity and 2 Samsung 990 Pro 2TB as Cache Pool on Supermicro AOC-SLG3-2M2 PCI card. grew to 5 disks + parity added parity2 From the beginning I had lots of UDMA CRC Errors on all drives. (almos 20k on a drive that has 10k power on hours) The UDMA CRC Errors only increase when there are write operations on the disks. Recently I noticed that the drives are getting loads of WRITE FPDMA QUEUED and ICRC, ABRT at LBA errors ( 300-500 in oldest drives, 47 in my latest parity2 drive). Currently disk1 is not in the system and is beeing emulated, as I needed to recover data from it after a stupid mistake I made. ( After a clean shutdown due to a power failure (UPS covered) parity1 was removed from the array due to errors on the next boot. Then I saw loads of errors piling up on parity2 and I decided to shutdown. Since the mover was running at that time and the shutdown did no go through and the system became unresponsive even in IMPI KVM console, I switched the power off through IPMI. I did try to preclear parity1 to get it back into the array, but the zeroing failed. Extended SMART Test on parity1 completed without error. Currently I am preclearing parity1 again to get it back into the array, because the array is currently without any redundancy. (disk1 and parity1 removed). Since the system is around 800km away from me and I have no spare HBA, SATA cables or PSU I can not really test a lot. Details of the Server: Supermicro CSE-829U X10DRU-i+ 19" 2U 12x 3,5" LFF 2x Intel XEON E5-2600 v3 v4 DDR4 ECC Raid 4x 10GbE X540 2x PSU 2 x Intel Xeon E5-2698V4 SR2JW 20C Server Prozessor 20x 2,20 GHz 50MB 2011-3 128GB Registered ECC DDR4 SDRAM (2x64GB DIMM) LSI SAS 9400-16i PCIe x8 4x SFF-8643 12G SAS3 NVMe HBA HDD SSD NVMe JBOD Tri-Mode Controller (ZFS, Ceph, MS Storage Spaces) 2 x Supermicro 1000W PSU Netzteil PWS-1K02A-1R (redundant PSU) BlueWalker PowerWalker VFI 3000 ICR IoT, USV unraid-01-diagnostics-20241020-2232.zip

Quick Update: I still did not have the opportunity to get the disks back to the server. Will update once I put all disks in.

Currently UFS Explorer is restoring data from the formated disk to a new drive. Part of that data already has helped in getting my backups running again (restic config and data was recovered and moved back to unraid). Parity2 is working again. Once I am sure I can not get back any more data from disk1, it will be added back to the server and then the recovered data will be moved from the new drive to the array Due to the size of the disks and the remote location of the server, this process will take a couple weeks. Thanks for the help and the information. What I take with me for the future is: format is not a solution to any problems with unraid.

So I added parity2 to the array and it is rebuilding. Disk1 is emulated and I will pull it and check if any data can be recovered. From what I gather it would be ok to recover to the array directly and not to the disk1 in particular, right?

Yes sdc1 is disk1 Will try to get back my data with UFS Explorer. Next steps: * Remove disk1 and recover data with UFS Explorer (fingers crossed for this one) * Order new disk for use as new disk1 * Put parity2 back in (not sure on when to do this. I am not clear on how to get the data from the recovery into the new disk 1. Do I keep the array in maintenance mode and mount the new disk1 and put the recovered data on it and run a parity check afterwards or do I need another route?