unRAID Server release 4.5-beta11 available


limetech

Recommended Posts

I'm very interested in this stutter problem because this is the only thing that puzzles me with the unRAID package. I'm using two full blown LimeTech machines (15 drives each) and let me tell you my experience.

 

Paused streams on a spun-up drive depend on the count of drives that need to spin-up whenever another folder/file is in access. In addition the type of drives seems to count in too.

 

a.) Consider a user share spread over say 10 drives. A stream is taking place from one already spun-up drive. The 10 drives are spun-down. A simple access to contents of the spun-down drives may result in a.) The stream is paused (always) b.) The 10 drives are started one after another. This may take seconds up to minutes depending on how fast the drives spin up. My latest drives (1.5TB Seagate) take nearly 10 seconds each. 10 drives a 10 seconds are 100 seconds. There is a high risk that all connects crash - the stream and the new access. I have to close the applications and restart again. I had to experience this so many times that I'm currently checking alternatives.

 

b.) If I go to the drives directly the pause is reduced to the individual drives spin-up time. Usually 10 seconds here.

 

c.) If you operate on mass data - I'm transfering not below 50GB packages (database contents) - the chance of empty caches is nearly 100%. So even a single directory lookup takes minimum 10 seconds here - usually more.

 

So, the more data you transfer, the more disks you have, the more disks are in user shares - the problem increases linear.

 

My hair is getting pulled nearly daily because I forget that my TotalCommander is sitting in the background standing on a big user share and I bring it in front. This may crash an already running transfer after half an hour and I have to repeat this.

 

I can reproduce that at will for Windows XP, Windows Vista (32 and 64) and Windows 7.

 

I'm no kernel programmer, I even don't have any Linux experience but if I had I would put at least the directory contents of the disks/user shares into some sort of private memory that won't be paged out. My systems hold 500.000 files - say 250.000 files each. Now take 1K for each file with some overhead. My 4GB RAM wouldn't even notice this 0.25GB.

 

Thanks.

Harald

 

Link to comment
  • Replies 248
  • Created
  • Last Reply

Top Posters In This Topic

I believe there are (at least) two different causes of the same symptom.

 

The controller issue Tom identified is real... I've found the EXACT same thing some time ago.

 

But I believe there is also a Samba issue, when a drive is spun up due to a Samba request.  I've seen the same stutters and pauses when all I do is list a directory via Samba, where I can do a ls of the same directory from a console prompt to spin up a drive and it creates no stutter (unless it is a paired drive on the same controller).  I also do not get the stutter if I access a drive via NFS (again, unless it is a paired drive on the same controller) unless I open a large file.

 

You also can not discount ethernet congestion is these situations... a lot of retail GB switches out there are CRAP when you get into multiple client operations particularly if you are pushing serious bandwith.  I've had a couple of those myself.

 

Also, if you are seeing this with the PCH or other boxen based on the same chipset, I had stuttering in many circumstances, even when not spinning up a drive, until I added a 10-100 hub between the PCH and the GB switch.  It was like magic.

 

IIRC, there was some noise elsewhere about some inefficiencies in Samba that was related to this situation, and that it is targeted to be addressed in Samba 4.x.  Remember that Samba is a reverse-engineered kluge to provide compatibility with Microsoft SMB... but without the docs or other standards from MS.  A little less than 2 years ago, MS relented and promised to share with the Samba dev team, so I expect by next year and Samba 4.x, we should see some more fruits of that sharing.

 

The fact that the same behavior (stuttering when accessing a different but spun-down drive) may have multiple causes makes debugging, especially remotely with beta users on different hardware, fraught with peril.

 

Short of a debug version of Samba and a profiler, I'm not sure this is something that unRAID can do about it.

 

Referencing spinning up via MyMain, spinning up by reading one random sector, or with an explicit spin-up command, is very different from causing a spin-up to get a (potentially) large directory listing or opening a file.  I believe there may also be some activity of Samba to dive into sub-directories in some instances too.

 

Clear as mud... eh?

Link to comment
Referencing spinning up via MyMain, spinning up by reading one random sector, or with an explicit spin-up command, is very different from causing a spin-up to get a (potentially) large directory listing or opening a file.  I believe there may also be some activity of Samba to dive into sub-directories in some instances too.

 

That's why I did both tests. MyMain is operating on the server and using Windows Explorer is operating over the network. In my case, it's not the spin-up of the drive causing the problem but something to do with either accessing a certain amount of data on the drive or something to do with the networking - I would agree that the likely suspect is Samba.

 

I'll try to telnet to the box and access a spun-down drive to see what happens.

 

Peter

Link to comment

With 4.5 beta my SSH daemon refuses to start. Seems that the permissions for the key files are wrong. This worked for all unRAID releases before 4.5-beta. What's the correct permission?

 

root@Tower:/boot/custom/packages# /etc/rc.d/rc.sshd start
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0755 for '/etc/ssh/ssh_host_rsa_key' are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.
bad permissions: ignore key: /etc/ssh/ssh_host_rsa_key
Could not load host key: /etc/ssh/ssh_host_rsa_key
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0755 for '/etc/ssh/ssh_host_dsa_key' are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.
bad permissions: ignore key: /etc/ssh/ssh_host_dsa_key
Could not load host key: /etc/ssh/ssh_host_dsa_key
Disabling protocol version 2. Could not load host key
sshd: no hostkeys available -- exiting.

 

Thanks

Harald

 

Link to comment

For me it also seems to be primarily a samba / OS X Finder issue. Using MC to transfer files during a running stream or to spin up a disk for a directory listing doesn't cause stuttering. A file listing from the Finder causes both music streams and video streams to stutter if the disk needs to spin up.

Link to comment

Guys,

 

Please start describing your system inc. hardware and controllers. Saying I have this issue or writin vague descriptions doesnt help. Post your devices setup and a reproducible scenario.

 

Tell us what disks and in what combinations.

 

Accessing disk shares, user shares. How many files in the user share, how many disks. Streaming media from host a, on user share media, accessing user share media from host a in finder/explorer.

 

Please post as much info as possible. For instance is the disk from which media is being streamed is also in the share that is being accessed? Are you accessing the share from the same host as the data is being streamed too. Does the issue occur when accessing from a different host. 

 

At the moment we are getting very general descriptions. I agree with the summary that there is more than one issue (there maybe several more) until we start getting reproducible scenarios it is hard to work out what they might be.

 

Please do try to explain the scenario in which you can reproduce the issue. Also including the hardware description in use means other users can try the same scenario and confirm. I'm quite happy to act as a QA resource given a reasonable scenario. 

 

Apologies in advance to btlupin you happen to be the last poster, I'm not picking on you, honest!

 

For me it also seems to be primarily a samba / OS X Finder issue. Using MC to transfer files during a running stream or to spin up a disk for a directory listing doesn't cause stuttering. A file listing from the Finder causes both music streams and video streams to stutter if the disk needs to spin up.

 

This description doesnt describe the server hardware. The devices config or the disks that were being used to stream the music or video, or indeed what disks were being spun up to cause the issue. Whether the music/video streams were from user shares or disk shares and whether the accessed "directory" is on a user share that also includes the streaming disk.

 

From what I am getting both windows and mac/os x hosts when streaming media and then accessing another user share causes via a finder or explorer window causes stuttering. The important missing item here for instance is that the streaming host and browsing host are the same. 

 

 

 

 

 

 

 

Link to comment

Harald,

 

This doesnt make sense...

 

a.) Consider a user share spread over say 10 drives. A stream is taking place from one already spun-up drive. The 10 drives are spun-down. A simple access to contents of the spun-down drives may result in a.) The stream is paused (always) b.) The 10 drives are started one after another. This may take seconds up to minutes depending on how fast the drives spin up. My latest drives (1.5TB Seagate) take nearly 10 seconds each. 10 drives a 10 seconds are 100 seconds. There is a high risk that all connects crash - the stream and the new access. I have to close the applications and restart again. I had to experience this so many times that I'm currently checking alternatives.

 

I realise english is not your first language, this makes technical descriptions difficult, but this is flawed.

 

Here is why.

 

A stream is taking place from one already spun up drive.

The ten drives are spun down.

 

[glow=red,2,300]Nope, if a stream is being read from a drive that drive is spun up, max of nine drives can be spun down. [/glow] 

 

A simple access to contents of the spun-down drives may result in a.) The stream is paused (always).

 

[glow=red,2,300]Please describe further, the known issue we have precludes this behavior. Tell us what host is streaming. What host is browsing and a list of devices from the unraid box and controller info. Also advise disk allocations to user shares. [/glow]

 

b.) The 10 drives are started one after another. This may take seconds up to minutes depending on how fast the drives spin up. My latest drives (1.5TB Seagate) take nearly 10 seconds each. 10 drives a 10 seconds are 100 seconds.

 

[glow=red,2,300] This description sounds like a command based controller, spining up ten drives with ten second startup should take ten seconds. Alright maybe twelve.  ;D [/glow]

 

If the cache is flushed then the disks contents will need to be re-read. Perhaps Tomm can look at maintaining a copy of folders/directories in memory rather than relying on caching.

 

There is a high risk that all connects crash - the stream and the new access. I have to close the applications and restart again. I had to experience this so many times that I'm currently checking alternatives.

 

[glow=red,2,300] Please describe the clients and applications that are crashing. We need to know the unRaid hardware that you are describing. If they are limetech servers model numbers would suffice. [/glow]

 

A posting of your devices data would be helpful.

 

 

 

 

 

 

 

Link to comment

I am running 6 data + 1 parity drive, there is a cache drive but all shares have it turned off.  All disks are 1.5 Seagates (all firmware CC1H).  The SATA controller is an onboard ICH10, and a 2 port sata + 1 pata jmicron 363.  5 disks and parity are on the ICH10, 1 disk is on the jmicron (disk6).  I am using user shares.  I was copying data to unraid from a windows XP machine over gigabit.  At this point disk0 (parity, ICH10) and disk6 (on the jmicron) are spinning, the rest are spun down.  While monitoring network activity (the file being written) I used a telnet session to copy a file to a new name in the same folder on the unraid box causing one of the other disks (disk4) to spin up.  Network activity was unaffected.  I then started a copy to the same host doing the writing of a file causing disk3 to spin up, during the spinup (~10 sec) network activity dropped to zero and came back up again to a lower level.  If it is of any help, on another occasion (last night) I happened to be using the unraid web interface (regular one) (laptop,wireless) while watching a movie on a vista based HTPC using XBMC (wired,gigabit) when a stutter occurred, and during the stutter the web interface was unresponsive and firefox kept coming back with "connection reset", after a few seconds, XBMC returned to the file selection screen (as if the movie had ended) and the unraid interface started working again, I was then also able to restart playback on xbmc.  If you need anything else, let me know!

 

here is an lspci of the unraid box:

 

00:00.0 Host bridge: Intel Corporation 4 Series Chipset DRAM Controller (rev 03)
Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Flags: bus master, fast devsel, latency 0
Capabilities: [e0] Vendor Specific Information 

00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4 (prog-if 00 [uHCI])
Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Flags: bus master, medium devsel, latency 0, IRQ 16
I/O ports at e000 [size=32]
Capabilities: [50] PCIe advanced features 
Kernel driver in use: uhci_hcd

00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5 (prog-if 00 [uHCI])
Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Flags: bus master, medium devsel, latency 0, IRQ 21
I/O ports at e100 [size=32]
Capabilities: [50] PCIe advanced features 
Kernel driver in use: uhci_hcd

00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 (prog-if 00 [uHCI])
Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Flags: bus master, medium devsel, latency 0, IRQ 18
I/O ports at e200 [size=32]
Capabilities: [50] PCIe advanced features 
Kernel driver in use: uhci_hcd

00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2 (prog-if 20 [EHCI])
Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Flags: bus master, medium devsel, latency 0, IRQ 18
Memory at f8101000 (32-bit, non-prefetchable) [size=1K]
Capabilities: [50] Power Management version 2
Kernel driver in use: ehci_hcd

00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 1 (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 0000b000-0000bfff
Memory behind bridge: f6000000-f7ffffff
Prefetchable memory behind bridge: 0000000080000000-00000000800fffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+
Capabilities: [90] Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Capabilities: [a0] Power Management version 2
Capabilities: [100] Virtual Channel 
Capabilities: [180] Root Complex Link 
Kernel driver in use: pcieport-driver

00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 5 (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: 0000c000-0000cfff
Memory behind bridge: f4000000-f4ffffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+
Capabilities: [90] Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Capabilities: [a0] Power Management version 2
Capabilities: [100] Virtual Channel 
Capabilities: [180] Root Complex Link 
Kernel driver in use: pcieport-driver

00:1c.5 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 6 (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: 0000d000-0000dfff
Memory behind bridge: f5000000-f5ffffff
Prefetchable memory behind bridge: 00000000f8000000-00000000f80fffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+
Capabilities: [90] Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Capabilities: [a0] Power Management version 2
Capabilities: [100] Virtual Channel 
Capabilities: [180] Root Complex Link 
Kernel driver in use: pcieport-driver

00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 (prog-if 00 [uHCI])
Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Flags: bus master, medium devsel, latency 0, IRQ 23
I/O ports at e300 [size=32]
Capabilities: [50] PCIe advanced features 
Kernel driver in use: uhci_hcd

00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 (prog-if 00 [uHCI])
Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Flags: bus master, medium devsel, latency 0, IRQ 19
I/O ports at e400 [size=32]
Capabilities: [50] PCIe advanced features 
Kernel driver in use: uhci_hcd

00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 (prog-if 00 [uHCI])
Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Flags: bus master, medium devsel, latency 0, IRQ 18
I/O ports at e500 [size=32]
Capabilities: [50] PCIe advanced features 
Kernel driver in use: uhci_hcd

00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 (prog-if 20 [EHCI])
Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Flags: bus master, medium devsel, latency 0, IRQ 23
Memory at f8100000 (32-bit, non-prefetchable) [size=1K]
Capabilities: [50] Power Management version 2
Kernel driver in use: ehci_hcd

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90) (prog-if 01 [subtractive decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=04, subordinate=04, sec-latency=32
Memory behind bridge: ec000000-f3ffffff
Prefetchable memory behind bridge: 0000000080100000-00000000801fffff
Capabilities: [50] Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard

00:1f.0 ISA bridge: Intel Corporation 82801JIB (ICH10) LPC Interface Controller
Subsystem: Giga-byte Technology Unknown device 5001
Flags: bus master, medium devsel, latency 0
Capabilities: [e0] Vendor Specific Information 

00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller (prog-if 01 [AHCI 1.0])
Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 27
I/O ports at e600 [size=8]
I/O ports at e700 [size=4]
I/O ports at e800 [size=8]
I/O ports at e900 [size=4]
I/O ports at ea00 [size=32]
Memory at f8102000 (32-bit, non-prefetchable) [size=2K]
Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/4 Enable+
Capabilities: [70] Power Management version 3
Capabilities: [a8] SATA HBA 
Capabilities: [b0] PCIe advanced features 
Kernel driver in use: ahci
Kernel modules: ahci

00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Flags: medium devsel, IRQ 5
Memory at f8103000 (64-bit, non-prefetchable) [size=256]
I/O ports at 0500 [size=32]

01:00.0 SATA controller: JMicron Technologies, Inc. JMicron 20360/20363 AHCI Controller (rev 03) (prog-if 01 [AHCI 1.0])
Subsystem: JMicron Technologies, Inc. JMicron 20360/20363 AHCI Controller
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at f7000000 (32-bit, non-prefetchable) [size=8K]
[virtual] Expansion ROM at 80000000 [disabled] [size=64K]
Capabilities: [68] Power Management version 2
Capabilities: [50] Express Legacy Endpoint, MSI 01
Kernel driver in use: ahci
Kernel modules: ahci

01:00.1 IDE interface: JMicron Technologies, Inc. JMicron 20360/20363 AHCI Controller (rev 03) (prog-if 85 [Master SecO PriO])
Subsystem: JMicron Technologies, Inc. JMicron 20360/20363 AHCI Controller
Flags: bus master, fast devsel, latency 0, IRQ 17
I/O ports at b000 [size=8]
I/O ports at b100 [size=4]
I/O ports at b200 [size=8]
I/O ports at b300 [size=4]
I/O ports at b400 [size=16]
Capabilities: [68] Power Management version 2
Kernel driver in use: JMicron IDE
Kernel modules: pata_jmicron, jmicron

02:00.0 IDE interface: JMicron Technologies, Inc. JMB368 IDE controller (prog-if 85 [Master SecO PriO])
Subsystem: Giga-byte Technology Unknown device b000
Flags: bus master, fast devsel, latency 0, IRQ 16
I/O ports at c000 [size=8]
I/O ports at c100 [size=4]
I/O ports at c200 [size=8]
I/O ports at c300 [size=4]
I/O ports at c400 [size=16]
Capabilities: [68] Power Management version 2
Capabilities: [50] Express Legacy Endpoint, MSI 01
Kernel driver in use: JMicron IDE
Kernel modules: pata_jmicron, jmicron

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 02)
Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Flags: bus master, fast devsel, latency 0, IRQ 28
I/O ports at d000 [size=256]
Memory at f8010000 (64-bit, prefetchable) [size=4K]
Memory at f8000000 (64-bit, prefetchable) [size=64K]
[virtual] Expansion ROM at f8020000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1 Enable+
Capabilities: [70] Express Endpoint, MSI 01
Capabilities: [b0] MSI-X: Enable- Mask- TabSize=2
Capabilities: [d0] Vital Product Data 
Capabilities: [100] Advanced Error Reporting 
Capabilities: [140] Virtual Channel 
Capabilities: [160] Device Serial Number 78-56-34-12-78-56-34-12
Kernel driver in use: r8169
Kernel modules: r8169

04:00.0 VGA compatible controller: S3 Inc. 86c325 [ViRGE] (rev 06) (prog-if 00 [VGA controller])
Flags: bus master, medium devsel, latency 32, IRQ 12
Memory at ec000000 (32-bit, non-prefetchable) [size=64M]
[virtual] Expansion ROM at 80100000 [disabled] [size=64K]

Link to comment

I only use user shares. My Unraid machine is based on the Asus P5B-VM DO board that Tom used to use. The output from devices tab:

 

parity device: pci-0000:00:1f.2-scsi-0:0:0:0 (sda) ata-WDC_WD15EADS-00H7B0_WD-WCAUP0024759

disk1 device: pci-0000:00:1f.2-scsi-1:0:0:0 (sdb) ata-WDC_WD10EADS-00L5B1_WD-WCAU45051875

disk2 device: unassigned

disk3 device: pci-0000:00:1f.2-scsi-3:0:0:0 (sdc) ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT0751279

disk4 device: pci-0000:00:1f.2-scsi-4:0:0:0 (sdd) ata-WDC_WD5000AAKS-00YGA0_WD-WCAS80541016

disk5 device: pci-0000:00:1f.2-scsi-5:0:0:0 (sde) ata-WDC_WD10EADS-65L5B1_WD-WCAU48062314

disk6 device: pci-0000:04:04.0-scsi-0:0:0:0 (sdg) ata-WDC_WD7500AACS-65D6B0_WD-WCAU80040406

disk7 device: pci-0000:04:04.0-scsi-3:0:0:0 (sdj) ata-WDC_WD10EADS-65L5B1_WD-WCAU48049990

disk8 device: pci-0000:04:04.0-scsi-2:0:0:0 (sdi) ata-WDC_WD7500AACS-00D6B0_WD-WCAU50028552

cache device: pci-0000:04:04.0-scsi-1:0:0:0 (sdh) ata-Maxtor_7L300S0_L603LVAG

 

As far as I can tell with beta11 this is only occurring when I am watching/listening on my Mac while simultaneously spinning up a disk to get a directory though the Finder.

Link to comment

I believe there are (at least) two different causes of the same symptom.

 

The controller issue Tom identified is real... I've found the EXACT same thing some time ago.

 

But I believe there is also a Samba issue, when a drive is spun up due to a Samba request.  I've seen the same stutters and pauses when all I do is list a directory via Samba, where I can do a ls of the same directory from a console prompt to spin up a drive and it creates no stutter (unless it is a paired drive on the same controller).  I also do not get the stutter if I access a drive via NFS (again, unless it is a paired drive on the same controller) unless I open a large file.

 

You also can not discount ethernet congestion is these situations... a lot of retail GB switches out there are CRAP when you get into multiple client operations particularly if you are pushing serious bandwith.  I've had a couple of those myself.

 

Also, if you are seeing this with the PCH or other boxen based on the same chipset, I had stuttering in many circumstances, even when not spinning up a drive, until I added a 10-100 hub between the PCH and the GB switch.  It was like magic.

 

IIRC, there was some noise elsewhere about some inefficiencies in Samba that was related to this situation, and that it is targeted to be addressed in Samba 4.x.  Remember that Samba is a reverse-engineered kluge to provide compatibility with Microsoft SMB... but without the docs or other standards from MS.  A little less than 2 years ago, MS relented and promised to share with the Samba dev team, so I expect by next year and Samba 4.x, we should see some more fruits of that sharing.

 

The fact that the same behavior (stuttering when accessing a different but spun-down drive) may have multiple causes makes debugging, especially remotely with beta users on different hardware, fraught with peril.

 

Short of a debug version of Samba and a profiler, I'm not sure this is something that unRAID can do about it.

 

Referencing spinning up via MyMain, spinning up by reading one random sector, or with an explicit spin-up command, is very different from causing a spin-up to get a (potentially) large directory listing or opening a file.  I believe there may also be some activity of Samba to dive into sub-directories in some instances too.

 

Clear as mud... eh?

 

agreed, however the samba issue doesn't really explain why the unraid webpage also locks up for a few seconds while a drive is spinning up

 

either way this problem isnt really an issue for me. i can quite happily live with the pausing

Link to comment

"A stream is taking place from one already spun up drive.  The ten drives are spun down. "

 

[glow=red,2,300]Nope, if a stream is being read from a drive that drive is spun up, max of nine drives can be spun down. [/glow] 

 

Hmm, what's wrong with my statement? Please read it as I wrote it. A stream is taking place from a spun-up drive. 10 more drives are sleeping. Now I use whatever I want (TotalCommander, Explorer, ...) to touch something from the sleeping share. The stream pauses until the 10 drives are spun-up. It doesn't make any difference on my system if the streaming drive is part of the share or if the streaming drive is part of another share - the stream pauses.

 

 

"A simple access to contents of the spun-down drives may result in a.) The stream is paused (always)."

 

[glow=red,2,300]Please describe further, the known issue we have precludes this behavior. Tell us what host is streaming. What host is browsing and a list of devices from the unraid box and controller info. Also advise disk allocations to user shares. [/glow]

 

As I wrote: The two machines are full blown original LimeTech machines. One is 1500/LL with 15 drives Seagate 1.5TB 341AS. The other one is the older original LimeTech machine with 15 drives mixed from Samsung and WD. They are connected from Windows machines. For my system it doesn't make any difference if I'm streaming or if I'm copying files. If disks spin-up the running connection pauses. If the pause is longer than some minutes the connections crash (this at least is a Windows feature 8-().

 

Before somebody points on my environment ;-) This happened in our old house (CAT5e) and it happens in our new house with new Cat7 (I think). All hardware was and is connected to a professional GB switch. The incoming VDSL 50MBit comes from a router and ends in a switch. The router changed in the new house and the switch changed in the new house too. It didn't make any difference. If all drives are spun-up I never see any problems ...

 

 

"The 10 drives are started one after another. This may take seconds up to minutes depending on how fast the drives spin up. My latest drives (1.5TB Seagate) take nearly 10 seconds each. 10 drives a 10 seconds are 100 seconds."

 

[glow=red,2,300] This description sounds like a command based controller, spining up ten drives with ten second startup should take ten seconds. Alright maybe twelve.  ;D [/glow]

 

Yes, if I go downstairs in front of the machines I can hear a click for the drive spinning up, then comes a pause, then the next click and so on ...

 

 

"There is a high risk that all connects crash - the stream and the new access. I have to close the applications and restart again. I had to experience this so many times that I'm currently checking alternatives."

 

[glow=red,2,300] Please describe the clients and applications that are crashing. We need to know the unRaid hardware that you are describing. If they are limetech servers model numbers would suffice. [/glow]

 

As I wrote in my original post: I can reproduce this at will for all Windows (XP, Vista (32 and 64), Windows 7). I can reproduce it in every application I use (e.g. Windows Explorer, TotalCommander, and Windows CMD for example).

 

 

I just changed to Beta-11. I usually never use Beta releases but the current GA is so painful for me - regular business is not possible. Yes, for some months I had the drives spun-up all day - but this doesn't help to find and fix the problem. Beta11 is much more faster and snappier - I will test further.

 

Is there some kind of Samba log that could be enabled? The syslog never shows such crashing connections ...

 

Regards

Harald

 

 

Machine 1:

parity device:  	pci-0000:00:1f.2-scsi-0:0:0:0 (sdk) ata-SAMSUNG_HD103UJ_S13PJ1KQ101691
disk1 device: 	pci-0000:00:1f.2-scsi-0:0:1:0 (sdl) ata-SAMSUNG_HD753LJ_S13UJ1KPC13868
disk2 device: 	pci-0000:00:1f.2-scsi-1:0:0:0 (sdm) ata-SAMSUNG_HD103UJ_S13PJ1BQA04578
disk3 device: 	pci-0000:00:1f.2-scsi-1:0:1:0 (sdn) ata-SAMSUNG_HD753LJ_S13UJ1KQ100980
disk4 device: 	pci-0000:00:1f.5-scsi-0:0:0:0 (sdo) ata-SAMSUNG_HD103UJ_S13PJ1CQA06295
disk5 device: 	pci-0000:00:1f.5-scsi-1:0:0:0 (sdp) ata-SAMSUNG_HD103UJ_S13PJ1EQ309596
disk6 device: 	pci-0000:02:00.0-scsi-1:0:0:0 (sda) ata-SAMSUNG_HD753LJ_S13UJ1KQ100993
disk7 device: 	pci-0000:04:03.0-scsi-0:0:0:0 (sdb) ata-SAMSUNG_HD753LJ_S13UJ1KQ100982
disk8 device: 	pci-0000:04:03.0-scsi-1:0:0:0 (sdc) ata-SAMSUNG_HD103UJ_S13PJ1BQA04574
disk9 device: 	pci-0000:04:03.0-scsi-2:0:0:0 (sdd) ata-SAMSUNG_HD753LJ_S13UJ1KQ100991
disk10 device: 	pci-0000:04:03.0-scsi-3:0:0:0 (sdf) ata-SAMSUNG_HD753LJ_S13UJ1KQ100992
disk11 device: 	pci-0000:04:04.0-scsi-0:0:0:0 (sdg) ata-SAMSUNG_HD753LJ_S13UJ1KQ117100
disk12 device: 	pci-0000:04:04.0-scsi-1:0:0:0 (sdh) ata-WDC_WD10EACS-00ZJB0_WD-WCASJ0802616
disk13 device: 	pci-0000:04:04.0-scsi-2:0:0:0 (sdi) ata-WDC_WD7500AACS-00ZJB0_WD-WCASJ0768258
disk14 device: 	pci-0000:04:04.0-scsi-3:0:0:0 (sdj) ata-SAMSUNG_HD753LJ_S13UJ1KQ100981

 

Machine 2:

parity device:  	pci-0000:00:1f.2-scsi-0:0:0:0 (sdl) ata-ST31500341AS_9VS04BKQ
disk1 device: 	pci-0000:00:1f.2-scsi-1:0:0:0 (sdn) ata-ST31500341AS_9VS06X0B
disk2 device: 	pci-0000:00:1f.2-scsi-0:0:1:0 (sdm) ata-ST31500341AS_9VS0C3DK
disk3 device: 	pci-0000:00:1f.2-scsi-1:0:1:0 (sdo) ata-ST31500341AS_9VS0DLF2
disk4 device: 	pci-0000:00:1f.5-scsi-0:0:0:0 (sdp) ata-ST31500341AS_9VS0GRGL
disk5 device: 	pci-0000:00:1f.5-scsi-1:0:0:0 (sdq) ata-ST31500341AS_9VS0H714
disk6 device: 	pci-0000:02:00.0-scsi-3:0:0:0 (sdh) ata-ST31500341AS_9VS0H7GB
disk7 device: 	pci-0000:02:00.0-scsi-2:0:0:0 (sdg) ata-ST31500341AS_9VS0H70Q
disk8 device: 	pci-0000:02:00.0-scsi-1:0:0:0 (sdf) ata-ST31500341AS_9VS0H70W
disk9 device: 	pci-0000:02:00.0-scsi-0:0:0:0 (sde) ata-ST31500341AS_9VS0HBQ0
disk10 device: 	pci-0000:01:00.0-scsi-3:0:0:0 (sdd) ata-ST31500341AS_9VS0HBRN
disk11 device: 	pci-0000:01:00.0-scsi-2:0:0:0 (sdc) ata-ST31500341AS_9VS0HLYC
disk12 device: 	pci-0000:01:00.0-scsi-1:0:0:0 (sdb) ata-ST31500341AS_9VS0HM39
disk13 device: 	pci-0000:01:00.0-scsi-0:0:0:0 (sda) ata-ST31500341AS_9VS0HM70
disk14 device: 	pci-0000:03:00.0-scsi-1:0:0:0 (sdk) ata-ST31500341AS_9VS0HRKA

Link to comment

What could possibly be causing this decrease in read speeds from the server?

Could this have something to do with changes in Samba?

 

Please download the attached file and place in 'config' directory of Flash, then Stop/Start array & let me know if this makes any difference.

 

Thanks for this tweak! I managed to write a 25GB directory to a parity protected drive at 42.5MB/s -- sustained average over 30s measured by bwm-ng.

Is that some kind of record? Who cares, it's fast!

Link to comment

4.5-beta11 still not any faster for me with regards to uploading. Only getting 14 MB/s to the cache. unRAID 4.4.2 gives me 38+ MB/s.  :(

 

That should not be the case - please try -beta11 again, do a transfer, and then post your system log.

 

http://pastebin.com/fc7da351 I'm not sure if this is all the info necessary because it didn't seem to change after I uploaded a new file to the array. I pulled the Syslog from unmenu. The file was 2.65 GB and it took 3:12 to up it to the array.

Hope it's cool to bump this. Anybody have an idea as to why my upload speed is only 14 MB/s? I always just replace bzimage and bzroot files when updating. Is there some other sort of refresh I can do? I also tried that smb-extra.conf file without success. Uploading starts out at near 30 MB/s but within 5 seconds slows to 14 MB/s. If I replace the bzimage and bzroot files back to 4.4.2 I get near 40 MB/s with bursts up to 50. Any ideas would be greatly appreciated.

Link to comment

OK, After 5 hours of testing I can reproduce at will three scenrios that can cause a stutter with a streaming video. I dropped the memory in the box to 1GB - copying 50Gb each time to run a test didn't appeal.

 

Stream video from unraid to a media streamer. Then access a share (user) from another host. Depending on the cache status of that share, you will experience anything from a mild stutter to the stream stopping. The unraid UI becomes unresponsive to the same extent as the stuttering, does recover though. My egreat requires restarting if the streaming stops rather than just stuttering (drop_caches). Unmenu UI works no problem.

 

If I'm not running a stream, running a sync then drop_caches cant get a stall condition (even with a sync), I'm assuming because the cache isnt being flushed. No disks are spinning up indicating the directory info is remaining in memory.

 

When streaming to a windows box and accessing a disk share from the same windows box a stutter is possible. Really large directory can cause some serious stuttering. Especially when browsed to as above, going direct is better (less stutter) than browsing a large share. \\tower\disk2 then explorer window to browse to music folder is bad... \\tower\disk2\music is better.

 

If steaming to a different client. Accessing a disk share from the windows box doesnt cause a stutter. Even when the explorer window becomes unresponsive for a long period. Unresponsiveness occurs whether disk is spun up or not. I believe this is a seperate issue but a symptom some people may be suffering from. I have reported this before.

 

When using a disk share, the more the files and directories the larger the effect. With a user share simply having a spun down disk then accessing the user share is all that is needed. 

 

I didnt previously notice this behaviour because I had lots of memory in my unraid box (was running 4GB) and my server runs 24/7 it caches and its data stays cached. I only copy data to and stream from. My shares are all on the same disks so accessing from one room would spin up any thet werent cached. Watching a movie in a second room the disks would already be cached. I dont stream media to my windows boxes. I have dedicated h/w media streamers and big TVs, why use a 19" monitor!

 

What I did discover is that the above scenarios can cause my NMT to drop of the network even if just sitting idle at a shares directory, it doesnt need to be streaming, if the unraid server dissappears it can leave the NMT unresponsive to network commands.

 

Now accessing a movie from a spun down disk without a directory listing of the share doesn't cause a stutter. A sync and drop caches without spun down disks, then requesting a large directory listing also doesn't cause a stutter.

 

So quick recap, three scenrios:

 

User share info must not cached, directory listing must trigger at least one disk to spinup, causes a stutter. In the case of many disks spinning up can causes streaming to stop or other network transfers to timeout.

 

Disk share access, streaming to a host, accessing a spun down disk, via a disk share from that same host can cause a stutter. Again cache must be flushed.

 

Disk share access, streaming to a host, accessing a large disk share from the same host causes the stream to stutter. Browsing a share is worse than directly accessing the share.

 

Other side effects noticed unraid UI becomes unresponsive during these pauses. Unmenu is unaffected.

 

   

Link to comment

OK, After 5 hours of testing I can reproduce at will three scenrios that can cause a stutter with a streaming video.

 

Kaygee, thank you for all this hard work!  I very much appreciate it.  Now I wonder if I can ask you to do some more?  ;D

 

What I would like to see is, with keeping all disk spun up all the time, under what circumstances can you get the video to stutter.

 

This will help me to isolate protocol and file system bottlenecks without having to worry about huge delays introduced by disk spinup.  This is because, as I described earlier in the thread, depending on your server hardware, if you are streaming on one 'port' of a particular disk controller, and an I/O gets sent to the 'other' port of the same disk controller, it will be impossible to prevent stutter if the drive on the other port is spun down when it receives that I/O.  I can not tell from your report how much stutter is being caused by this phenomena.

 

To help solve this issue with multiple drives on the same controller, I will be releasing -beta12 which has a new feature called "spinup groups".  This will let you group disks so that they are always spun up and down together.  For example, you might define a group called "host1" into which you put say disk1 and disk2.  Now when it's time to spin down disk1, disk2 will also be spun down; and vice-versa.  Similarly if I/O is received for a disk, if that disk, or any disk in it's group is spun down, all disks will be spun up.  The main use of this function would be to define controller groups - that is, define sets of drives on the same controller.  A second use of this function would be to perhaps group by user share, which would be useful especially for 'most free' allocation method.  Anyway, more on this later.

 

One more thing: what media players are you using for streaming video?

Link to comment

With disks spun up only stutter was possible with same host streaming. xp/vlc or xp/media player streaming, browse v. large disk share via explorer. Same scenario external streamer, no problem. Repro xp, 2000 and vista.

 

hardware media streamers are dlink dsm320rd, egreat nmt, netgear mp101 mp3, xmbc.

 

I dont think your idea will work, I had similar thought ysterday and tried several versions, variants and potential work arounds.

 

I tried removing disks from user shares, no difference. I tried reading disk shares rather than user shares, no difference. I tried streaming media from different user share and accessing another user share.

 

I also accessed hdds on different controllers, again same issue stutter. I just did another test to confirm, removing the hdd on the seprate controller from the big media user share, stream from the seperate controller, access the big media share and the stream on a seperate controller stutters or stops altogether.

 

The mb is pci-e, the other controller is a PCI controller so is on a seperate bus.

 

I'm copying a load more data to the other controller HDD and will re-test in reverse, stream from the big media share, then access the seprate controller to see if the media stream stutters. I tried this but there are only a few movies on the other disk and these seemed to be cached (disk didnt spin up) browsing the share causes the disk to spin up but no stutter seen on the streaming device. Hence copying more data.

 

All the tests apart from same host media streaming tests are conducted using the NMT streamer.

 

I also checked PCI devices to see what the onboard SATA controllers identifies as and noticed it was in non AHCI mode. I have swapped the bios back to non AHCI and will run some retests.

 

Here is AHCI mode info.

 

00:09.0 SATA controller: nVidia Corporation MCP78S [GeForce 8200] AHCI Controller (rev a2) (prog-if 01 [AHCI 1.0])

Subsystem: nVidia Corporation Unknown device cb84

Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 28

I/O ports at b480

I/O ports at b400

I/O ports at b080

I/O ports at b000

I/O ports at ac00

Memory at fbe7a000 (32-bit, non-prefetchable)

Capabilities: [44] Power Management version 2 Capabilities: [8c] SATA HBA

Capabilities: [b0] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3 Enable+

Capabilities: [ec] HyperTransport: MSI Mapping Enable+ Fixed+

Kernel driver in use: ahci

Kernel modules: ahci 

 

 

 

 

Link to comment

I have about ten user shares defined. Some single disk, some multiple disk (disk1, disk3), some multiple disk (disk4-6) and one with multiple disks and an exclusion (disk1-5, ex disk2).

 

Hdd 1-6 on motherboard inc parity.

Hdd 7 on SIL3124 PCI controller.

 

In devices page none of the disk appear in ide master/slave mode in either SATA or AHCI mode.

 

Still stutters in AHCI mode with force NCQ disabled = no. No idea when I swapped it to sata mode or indeed why.

 

Tried ncq queue disabled = yes

 

Couldnt reproduce stutter from big media share when accessing disk6 on a seprate controller, no stutter from user share just on disk2 when accessing disk6 user share either. Streaming from big media share and accessing disk2 user share genrates small stutter (1/4 second maybe).

 

If the directory request doesnt spin up the disk, browsing a share doesnt cause a stutter but does spin up the disk. 

 

It is definetly cache related more than the disk having to spin up. Having disk 6 ( on a seperate controller spun down) accessing the user share that is only on disk 6, doesnt stutter media streaming from a different controller/user share.

 

Using hdparam to spin up or down disks doesnt cause any stuttering.

 

Graded worst to best.

sata mode with ncq disabled no, ahci mode with ncq disabled no, ahci mode with with ncq disabled yes.

 

I hope this helps. 

 

 

 

 

 

 

 

Link to comment

Seeing some curious and inconsistent behaviour in AHCI with ncq disabled = yes. Everyone once in a while accessing a user share will cause other drives to spin up.

 

user share media = disk 6

user share music = disk 2

user share movies = disk 1,3,4,5.

 

When accessing music or media a stream playing from movies will occasionally stutter or stop, perhaps 75-80% time this doesnt occur. When it does several disks will have spun up rather than just disk2 or disk 6 as expected.

 

The main menu read stats for those disks also increase. Sometime just a few byte, a single disk and a small stutter. Sometimes several disks several bytes and streaming stops.

 

Dont remember seeing this behaviour in SATA mode during previous testing.

 

Streaming from Disk3 media share.

Disk1, 2, 4,5,6 all spun down.

Open the user share on disk2 called music.

 

4 bytes read from disk1 and it is now spun up.

disk2 read as expected and spun up.

3 bytes read from disk4 and it is now spun up.

3 bytes read from disk 5 and it is now spun up.

disk6 stays spun down.

 

Previously I would expect to see only disk2 spin up.       

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.