Jetro

Thank you, let me know if you need further diagnostic

I wouldn't even for home solutions. There are problems to sort, I bought 2 Pro and 1 Basic licenses but I'm putting them down for now, passing to TrueNAS and Synology/Qnap in new purchases. I tought Unraid'd have been a good solution to re-use old hardware or even to build new NASes because internet communities are pointing at it as the best thing in the world. After trying it out (maybe because I'm used to professional devices) the feeling is that there is too much work to be carried out before this project worth current prices.

I got the right output: 147621332 /mnt/disk1/TheHub/Truenas4/./filename.mp4 976276944 /mnt/disk1/TheHub/Truenas4/./filename.mp4 3409636407 /mnt/disk1/TheHub/Truenas4/./filename.mp4 209565299 /mnt/disk1/TheHub/Truenas4/./filename.mp4 272780277 /mnt/disk1/TheHub/Truenas4/./filename.mp4 320218609 /mnt/disk1/TheHub/Truenas4/./filename.mp4 172232382 /mnt/disk1/TheHub/Truenas4/./filename.mp4 Then I also got folders which dimension is 6

Unfortunately same behaviour. I proceeded moving manually those files, meanwhile, a disk failed, so now I am trying to move the entire content of disk 6 (12TB) to disks 11 and 12 (8 TB + 8TB) and I'm getting the same problem. It stays a while "checking issues" then the usual 1 byte transfer. Is there some logging that would be useful to you? In this case: "The following items will not be transferred, because there's not enough space in the target disks" But I'm trying to copy 11.86TB in 15+ TB drives:

$RECYCLE.BIN is empty and not expanded, selected files belongs to "Truenas4" directory and I can't select it because it's something like 12TB while I need to transfer aroung 30GB. Tried also copying files from the bottom (names starts with "z") and same behaviour

Sure, i set it to verbose, but i don't see anything useful: logounbalanced SCATTERGATHERHISTORYSETTINGSLOG 2024/03/20 08:18:16 2024/03/20 08:18:16 ========================================================= 2024/03/20 08:18:16 disk(/mnt/disk4):no-items:currentFree(51.09 GB) 2024/03/20 08:18:16 --------------------------------------------------------- 2024/03/20 08:18:16 --------------------------------------------------------- 2024/03/20 08:18:16 2024/03/20 08:18:16 ========================================================= 2024/03/20 08:18:16 disk(/mnt/disk5):no-items:currentFree(50.99 GB) 2024/03/20 08:18:16 --------------------------------------------------------- 2024/03/20 08:18:16 --------------------------------------------------------- (cut) 2024/03/20 08:18:16 ========================================================= 2024/03/20 08:18:16 Bytes To Transfer: 1.00 B 2024/03/20 08:18:16 --------------------------------------------------------- As you can see, it say "Bytes To Transfer: 1.00 B", then the rest looks ok Then: Here you can see history: Settings I choose:

Hello guys, I'm trying to use unbalanced to free up some space on my disk1, however after selecting files and destination disk it only transfer 1 byte and say everything is done, while it's not. I tried many times, and in one case it transferred files, in the others no. What am I doing wrong?

I was looking at the dashboard plugin. Now it makes sense and perfectly explain performance drop after 200GB! Finally I have clear ideas on what's happening on Unraid

I have the opposite problem: I was expecting to see all ram used, instead of that only 4GB out of 256 are used. Do I missed some options?

Even if RAM is always shown < 4GB used? Good to know, at least all that RAM is useful!

ZFS Zpools may have redundancy? (ex. RAID Z1) If not, once finished transfer i can setup a zfs pool with a replica task for most used files

Does the 1% apply also for bigger array? I have 382 TB now, an plan to go to 500TB with final disk... That mean I'd need 5 TB RAM... Also, i rolled back from ZFS to XFS because of slowliness

Got that. So it makes sense, now I'm writing as low as 60 MB/s when I'm near the end of the disk (so 180 MB/s) and 90 MB/s when in faster zone. Also there was some kind of problem with the other machine, and it died today. I replaced mobo/cpu/ram at once (no time to investigate, data transfer is long enough) and it seem more stable now, acting as expected. Building the new one I'll try to put only new and equal disks, so it will make sense keeping turbo write on and transfer @ full speed! Another quick question 1 : does having 256GB of RAM makes sense? I've never seen more than 4GB used. (Note it will never run containers nor VMs as there's a separate cluster for that). Question 2: How many free space is it best to leave in disks?

That's why I'm not understanding low speed: parity drives are the faster one (min. 150 MB/s) and I'm moving big video files, so why I'm not going at full speed? And, above all, why first hundred of GB go at full speed? Maybe there's some thermal trottling involved (all temperatures seem low)? Or some caching issues?

I'm not getting this: - Before: 28 disks (actual ones + smaller and slower disks which have never been used) - Now: 28 disks (removed some small and slow disks and replaced with 18TB/280MB/s drives) - Having turbo write off, I'm only using 3 disk at a time. Parity disks have always been 2 x 18 TB Exos drives. May the overall array size be the culprit? I'm gonna build a twin machine, and I'd like to avoid bottlenecks...

Nope, initially there was the slow ones with other 5-6-8 TB slower disks which were never be used. Finished moving from first NAS I removed from the old one the 18TB disks, formatted and installed on Unraid machine

It makes sense, as there is a big heatsink on front backplate, like there's some chipset behind it. I'll make a check next time I'll be there. However, I can't understand how I moved something around 180 TB at full speed (> 150 MB/S) to slower disks (14 TB and lower) then now it seem I'm stuck at half that speed

No really, there is only a single cable going under the mainbord, I assume to something like what HP calls an expander board. To the front backplane I can see two cables linked (24 disks). To access rear backplane or the expander I have to remove the MoBo.

There are two backplanes on that Supermicro server, one to the back (12 Disks) and one to the front (24 disks). I have to take them off to read the P/N, I think I could do that in the afternoon. 01:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02) Subsystem: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 32 bytes Interrupt: pin A routed to IRQ 24 NUMA node: 0 IOMMU group: 64 Region 0: I/O ports at 6000 [size=256] Region 1: Memory at c7300000 (64-bit, non-prefetchable) [size=64K] Region 3: Memory at c7200000 (64-bit, non-prefetchable) [size=1M] Expansion ROM at c7100000 [disabled] [size=1M] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s <2us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x8 TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR- 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- TPHComp- ExtTPHComp- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, AtomicOpsCtl: ReqEn- LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+ EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [d0] Vital Product Data Not readable Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [c0] MSI-X: Enable+ Count=97 Masked- Vector table: BAR=1 offset=0000e000 PBA: BAR=1 offset=0000f000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr+ BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [1e0 v1] Secondary PCI Express LnkCtl3: LnkEquIntrruptEn- PerformEqu- LaneErrStat: 0 Capabilities: [1c0 v1] Power Budgeting <?> Capabilities: [148 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Kernel driver in use: megaraid_sas Kernel modules: megaraid_sas

How can I understand where's the controller bottleneck? Basing on 18 TB drives (which I'm using now) speeds seem to be much higher. Also, I'm reading/writing to 3 disks only, CPU usage is < 7 %, is it normal I'm getting these speeds? If it is, is there any suggestion for an HBA controller which wouldn't cause bottleneck? He's okay if there are some slow disks, but I prefer getting full speeds when working on new disks

Thank you for your reply, It looks like I have some really slow disk, which are the already filled ones. As I'm now working on the fastest drives, maybe it's better to keep turbo write off. For example I was copying to disk 2, which is rated > 250 MB/s like also the parity drives, is it normal I was writing at < 50 MB/s? Controller Benchmark: Disk benchmark: Disk 20 detail: Speed after resuming transfer: Tomorrow I'll update with transfer speed Edit: here's what hapens: FIle are only videos.

Hello everyone, I'm moving a lot of files for a friend, from 6 Truenas devices to 2 Unraid. Now only the first is "alive", with following hardware: - Supermicro X10DRi motherboard - Supermicro controller in HBA mode (don't remember exact model) - 416 GB RAM ECC (has to be 512GB but we've got some defective modules. Will go down to 256GB with the second unraid) - 2 x CPU E5-2690 v3 - 36 Disk of various capacity, that will be moved during transfer. (actually only 28 + 2 parity in use) - 10 Gbit SFP+ dual port network card - 1 Gbit dual port integrated network card I've set up the 1 Gbit network for OOB management, while 1 x 10 Gbit goes to LAN and 1 x 10 Gbit goes to the Truenas moving files. First Truenas has been moved at approx. 145 MB/s, which is the limit of destination hard drive on unraid. Parity was disabled. Write mode was "reconstruct write". Now, I've formatted first truenas Segate Exos HDDs (18TB) and installed them on unraid, and I'm transferring second NAS. Speed start at 145 MB/s then drop after some times to 50 MB/s, which means more than a month of transfer instead of 2 weeks. As the speed was slow, I integrated parity, let the parity set and then restarted transfer with the exactly same speed. Tried moving from reconstruct write to "auto", and speed increased from 50 to 60 MB/s. Attached the diagnostics Many thanks mmpl-2-diagnostics-20240124-2003.zip

Edit: installed the Supermicro branded LSI 9207-8i and it works like a charm. With 32GB of RAM it's okay as home media server now, and I'm planning to install some more containers in it. So the big differences was: - Switching back to btrfs instead of using zfs (which prevent files copy) - Enabling turbowrite (eliminated the stalls after 3 GB of copy) - Installing a dedicated PCIe SAS/SATA controller (improved general transfer speed)

I noted there was only the empty folder left on "disk1", deleted it and now everything seem ok!

Another question: I see two shares which have some bytes left on Disk 1. How can I remove residual data from disk 1?