johnodon

I just started reading about the "rcu_sched detected stalls on CPUs/tasks" error and have seen quite a few articles about an issue with this in Ubuntu VMs. Here is one: http://www.gossamer-threads.com/lists/openstack/operators/35716 This makes me wonder if my XBMCBuntu VMs are causing the crashes. EDIT: mhoober has similar (not identical) errors in his syslog and is experiencing crashes: http://lime-technology.com/forum/index.php?topic=37263.0

I have been experiencing hard crashes fairly frequently...maybe every few days. I finally had the log window open when the last one occurred a few minutes ago. Here is what was captured. Does it lend any clue as to what the culprit is? I am going to disable my nzbget, nzbdrone and xbmcserver dockers. I need to leave the mariadb one running. The only plugins I have are libvirt, virtman, dynamix and SNAP (which I just removed). Dec 21 08:33:21 unRAID kernel: perf interrupt took too long (2507 > 2500), lowering kernel.perf_event_max_sample_rate to 50000 Dec 21 12:02:42 unRAID kernel: br0: port 2(eth1) neighbor 8000.00:25:90:64:a7:d8 lost Dec 21 12:02:42 unRAID kernel: br0: port 2(eth1) entered listening state Dec 21 12:02:57 unRAID kernel: br0: port 2(eth1) entered learning state Dec 21 12:03:12 unRAID kernel: br0: topology change detected, propagating Dec 21 12:03:12 unRAID kernel: br0: port 2(eth1) entered forwarding state Dec 21 12:03:22 unRAID kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 11, t=6002 jiffies, g=306012, c=306011, q=13917) Dec 21 12:03:22 unRAID kernel: Task dump for CPU 0: Dec 21 12:03:22 unRAID kernel: swapper/0 R running task 0 0 0 0x00000008 Dec 21 12:03:22 unRAID kernel: ffffffff817ebed8 ffffffff815e64ef ffffffff817ebfd8 ffffffff817fe440 Dec 21 12:03:22 unRAID kernel: 0000000000012c80 ffff88087cd9d8b0 ffffffff817ebe40 ffffffff81085c44 Dec 21 12:03:22 unRAID kernel: 0000000000000005 ffffffff817ebe8c 0000000000000046 ffffffff817ebe48 Dec 21 12:03:22 unRAID kernel: Call Trace: Dec 21 12:03:22 unRAID kernel: [] ? __schedule+0x43f/0x6a4 Dec 21 12:03:22 unRAID kernel: [] ? tick_broadcast_oneshot_control+0x13d/0x19e Dec 21 12:03:22 unRAID kernel: [] ? pick_next_task_fair+0x38b/0x40f Dec 21 12:03:22 unRAID kernel: [] ? cpuidle_enter_state+0x49/0x9d Dec 21 12:03:22 unRAID kernel: [] ? cpuidle_enter+0x12/0x14 Dec 21 12:03:22 unRAID kernel: [] ? cpu_startup_entry+0x17a/0x22e Dec 21 12:03:22 unRAID kernel: [] ? rest_init+0x72/0x74 Dec 21 12:03:22 unRAID kernel: [] ? start_kernel+0x400/0x40c Dec 21 12:03:22 unRAID kernel: [] ? set_init_arg+0x53/0x53 Dec 21 12:03:22 unRAID kernel: [] ? early_idt_handlers+0x120/0x120 Dec 21 12:03:22 unRAID kernel: [] ? x86_64_start_reservations+0x2a/0x2c Dec 21 12:03:22 unRAID kernel: [] ? x86_64_start_kernel+0xee/0xfb Dec 21 12:06:22 unRAID kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 11, t=24007 jiffies, g=306012, c=306011, q=54217) Dec 21 12:06:22 unRAID kernel: Task dump for CPU 0: Dec 21 12:06:22 unRAID kernel: swapper/0 R running task 0 0 0 0x00000008 Dec 21 12:06:22 unRAID kernel: ffffffff817ebed8 ffffffff815e64ef ffffffff817ebfd8 ffffffff817fe440 Dec 21 12:06:22 unRAID kernel: 0000000000012c80 ffff88087cd9d8b0 ffffffff817ebe40 ffffffff81085c44 Dec 21 12:06:22 unRAID kernel: 0000000000000005 ffffffff817ebe8c 0000000000000046 ffffffff817ebe48 Dec 21 12:06:22 unRAID kernel: Call Trace: Dec 21 12:06:22 unRAID kernel: [] ? __schedule+0x43f/0x6a4 Dec 21 12:06:22 unRAID kernel: [] ? tick_broadcast_oneshot_control+0x13d/0x19e Dec 21 12:06:22 unRAID kernel: [] ? pick_next_task_fair+0x38b/0x40f Dec 21 12:06:22 unRAID kernel: [] ? cpuidle_enter_state+0x49/0x9d Dec 21 12:06:22 unRAID kernel: [] ? cpuidle_enter+0x12/0x14 Dec 21 12:06:22 unRAID kernel: [] ? cpu_startup_entry+0x17a/0x22e Dec 21 12:06:22 unRAID kernel: [] ? rest_init+0x72/0x74 Dec 21 12:06:22 unRAID kernel: [] ? start_kernel+0x400/0x40c

Request for the Create VM tab... Under USB Devices, can you include the bus and device IDs as they are shown in System Devices? Bus 002 Device 004: ID 05dc:a817 Lexar Media, Inc. Bus 002 Device 006: ID 20a0:0001 Clay Logic Bus 002 Device 003: ID 1a40:0101 Terminus Technology Inc. Hub Bus 002 Device 005: ID 20a0:0001 Clay Logic Bus 002 Device 002: ID 1a40:0101 Terminus Technology Inc. Hub This would help to differentiate multiple devices with same vendor/product IDs. John

I'll try that. Brit...what are your thoughts on this? Just curious. John

Hey sparky. First...thanks for this! Does your XBMC Server container include an advancedsettings.xml? Werner had provided one (not actually in the container) that turned off a lot of unnecessary services (as well as other stuff). John

I just performed another test... I copied an MKV from a disk share (DISK11) of a drive that is pretty full down to my desktop. I deleted the file from the disk share. I copied the file back from my desktop to the disk share. Right at the time the Windows transfer completed, I refreshed the unraid Main screen and saw ~11,000 writes to both parity and disk. I then deleted the file from my desktop. The parity/disk writes continued to increase over the next 1.5 minutes to 41,000 writes. So, at least I know that the transfer is still not going on behind the scenes when windows said it was completed. I am left to believe the the RAM is being used. If this is the case, my concern is what would happen if I shut the server down in the middle of that 1.5 minutes? John

Based on what Weebo said, that seems likely. That's extremely intresting if that is what's going on... becasue it means that unRAID is using extra ram like a temp cache disk and might reduce the need of a cache disk for systems with infrequent large writes. Now are there any drawbacks to this, for example if your not using ECC ram does this increase the risk of bitrot? I never look a gift horse in the mouth so I should stop asking questions. I do use ECC so at least I know I am covered there. But again...I am pretty damn sure I did not see these speeds in v6 until I upgraded the parity disk and migrated to XFS. Maybe XFS is playing a part with the memory caching. I don't know.

Is it possible that the file is being written to memory first since I have 48GB? Once the transfer was complete, both DISK1 and Parity showed ~11,000 writes. I went back a minute later and the number of writes had increased to what you see above (~33,000). This snapshot was taken almost immediately after Windows had said the transfer was completed: This one (also shown above) was take about a minute later: I have no other activity on my server ATM other than NZBDrone updating some metadata (DISK6). John

EACH of these has an impact. The 4TB 7200 RPM HGST is capable of 160MB/s on the outer tracks. uRAID6 allows more buffering of writes in the buffer cache, especially with a large amount of ram. Perhaps housekeeping on XFS is managed better. There used to be a single kernel lock in reiserfs for years that hurt performance. The destination drive type plays a big role in this too. I'm on unRAID 5 and I burst at 110MB/s from windows with data on SSD to an HP micro server. There were kernel tunings I used plus I always make sure I purchase a high speed parity drive. Thx Weebo. The thing that confused me was that even with the new parity drive, my parity check speeds still clocked in around 75MB/s. Would the outer tracks not be a factor here also? John I am having a hard time believing this as well. First a speed of 113MB/s is basically saturating a gigabit LAN connection. Second that matches the speeds I get writing to a SSD cache drive. I thougth the parity calculations slowed writes to the array down somewhat... (This was my experience pre-cache) which is the basis of why people use cache drives and mover in the first place. If you can write to the array at the same speed (or better) which you can transfer data over your LAN to the array why bother with a cache and open yourself up to data loss. I had the same exact concerns. I guess the only way to really prove that the file I am testing with exists in parity would be to pull the drive that it lives on and replace with a new drive. If the drive is rebuilt and the file exists then we will have our answer. Other than that my only verification is watching the "Writes" indicator for the parity drive on the Main screen update as I write the file to the array (which it does). John EDIT: I copied that MKV down to my dekstop, cleared the stats in unRAID and then copied the file back to the user share (/mnt/user/Movies). The data was written to DISK1 and Parity as shown below:

EACH of these has an impact. The 4TB 7200 RPM HGST is capable of 160MB/s on the outer tracks. uRAID6 allows more buffering of writes in the buffer cache, especially with a large amount of ram. Perhaps housekeeping on XFS is managed better. There used to be a single kernel lock in reiserfs for years that hurt performance. The destination drive type plays a big role in this too. I'm on unRAID 5 and I burst at 110MB/s from windows with data on SSD to an HP micro server. There were kernel tunings I used plus I always make sure I purchase a high speed parity drive. Thx Weebo. The thing that confused me was that even with the new parity drive, my parity check speeds still clocked in around 75MB/s. Would the outer tracks not be a factor here also? John

This thread used to piss me off because I was the typical unRAID'er that would see ~30MB/s - 40MB/s when writing to a parity protected array without the use of a cache drive. I have since made 3 major changes to my server: 1. Upgraded to unRAID v6 - Honestly, I did not see an increase in speed after migrating to v6 and prior to the next to items listed below being completed. 2. Upgraded my parity drive from a 2TB 7200 RPM 6Gb/s Hitachi to a 4TB 7200 RPM 6Gb/s HGST 3. Migrated all data drives from RFS to XFS (amount of data on each drive pretty much remained the same) Here is what I now see when reading and writing to the array: READING FROM ARRAY WRITING TO ARRAY Which of the factors listed above do you think provided the largest impact? I cannot explain the speed boost. In fact, I found this to be so alien to me that I verified that I did not accidentally turn on caching and that the parity drive was being written to. And yes...I did time the transfers and they really were that fast. John

memtest completed 6 passes without error. All of the drives that had issues were WD green drives although varying models...WD20EARS and WD20EARX. I also had a 1.5TB WDEARS drive that I had to pull from the array some time ago. No more WD drives for me again...ever!

Finally finished migrating all of my data disks from RFS to XFS. Parity sync is running now. Thank you to everyone for all of your help! John

Seeing an odd issue that I think I read about in another thread. I'll see if I can find it. Per Rob's suggestion, I added my spare disk in the DISK11 slot, changed the format to XFS and started the array. As expected, unRAID saw the drive and unformatted so I formatted it. I then began the rsync process from DISK3 to DISK11. When rsync was complete, I stopped the array, unassigned both DISK3 and DISK11 and swapped them. When I went to start the array, there was a note the said disk assignment was wrong and a new assignment would be created is the array is started. I started the array but now DISK3 showed as unformatted. I put the disks back in their original slots and DISK3 showed as formatted as RFS (as it had been). I did the swap a few times and DISK3 always came back as unformatted when the disks were swapped into the new assignments. In the end, I just stopped the array, formatted DISK3 as XFS and rsync'd DISK11 to DISK3. When I move onto the next disk, I'll see if I can duplicate. Have you guys seen this before? John

That makes completes sense! Thx Rob.

So v7 the architecture couldn't be reconstructed?

I know this is a BIG one and I have no idea if this has been considered but I know it is something I would be interested in... I would love the option of having my parity being either realtime (like unraid) or scheduled (like snapraid). The customer would have the ability to decide how they want to configure their parity. The more I become educated in the world of parity and data protection, I have come to realize that a backup scheme like snapraid may better suit my environment...vastly movies/tv shows. Any *important* data on my server is backed-up via other means. I don't think I have a true need for real-time parity but love everything else that unraid has to offer: webui, visualization, etc. Also, with a backup parity scheme, speed is not being sacrificed as it is on real-time parity. I know that one main arguments against this will he "install a cache drive". I am limited by physical bays so this is not really an option. Thoughts? John

OK guys...I can't take it anymore. Copying data from the spare disk back to the newly XFS formatted DISK1 with parity enabled averaged an awful 16MB/s. When I copied the other way (DISK1 --> spare disk) I saw an average of 90MB/s with some rates hovering around 125MB/s. I have no idea what the issue is but it seems to be tied directly to parity. I have since ordered a new parity drive (4TB HGST NAS drive) to see if it is drive related. Anyway, I will be performing the rest of my migration sans parity. I understand the risks but I just can't accept a total of ~32 hours to transfer data back and forth (array --> spare = ~6 hours, spare --> array = 26 hours). Please let me know if this process will work OK. The part that I am struggling with is when to run the New Config tool. After all disks are transferred and assigned? After each disk transfer and assignment? - Unassign parity disk - Format spare disk as XFS (mkfs.xfs /dev/sdl1) - Mount spare drive via SNAP - rsync data from DISK2 --> SNAP - Unmount SNAP disk - Stop array - Unassign DISK2 - Assign SNAP disk as DISK2 - Start array - Format old DISK2 as XFS - Mount old DISK2 via SNAP - rsync data from DISK3 --> SNAP ---- Continue with remaining disks

Only 160GB more to go. As soon as this is finished I want to poke around in the BIOS for both the MB and the AOC cards to make sure everything looks OK.

Doing a little more reading and found this: http://lime-technology.com/forum/index.php?topic=22095.0 Another user with 2x AOC-SASLP-MV8 and was experiencing the same slow write speeds as me.

Pretty vanilla... rsync -arv --stats --progress /mnt/disk/transfer/disk1/ /mnt/disk1/

Why does it always seem that it is my system that suck at transfer speeds?!?!

You left parity enabled? Do you remember what your writes speeds were? I'd like to blame the fact that I an reading from and writing to WD green drives 20MB/s seems slow to me. My parity drive is a Hitachi 7200 RPM. EDIT: Hell, now that I look I'm not even getting 20MB/s: Movies/The Bridge on the River Kwai (1957)/ Movies/The Bridge on the River Kwai (1957)/The Bridge on the River Kwai (1957).mkv 32,519,843,189 100% 20.45MB/s 0:25:16 (xfr#1148, to-chk=678/2100) Movies/The Croods (2013)/ Movies/The Croods (2013)/The Croods (2013).mkv 21,286,252,532 100% 19.16MB/s 0:17:39 (xfr#1149, to-chk=677/2100) Movies/The Departed (2006)/ Movies/The Departed (2006)/The Departed (2006).mkv 22,127,628,284 100% 14.46MB/s 0:24:18 (xfr#1150, to-chk=676/2100) Movies/The Evil Dead (1981)/ Movies/The Evil Dead (1981)/The Evil Dead (1981).mkv 17,952,888,479 100% 16.43MB/s 0:17:21 (xfr#1151, to-chk=675/2100) Movies/The Hobbit - The Desolation of Smaug (2013)/ Movies/The Little Drummer Boy (1968)/ Movies/The Lord of the Rings - The Return of the King (2003)/ Movies/The Lord of the Rings - The Return of the King (2003)/The Lord of the Rings - The Return of the King (2003).mkv 81,323,129,520 100% 13.76MB/s 1:33:56 (xfr#1152, to-chk=674/2100) Movies/The Passion of the Christ (2004)/ Movies/The Passion of the Christ (2004)/The Passion of the Christ (2004).mkv 25,001,766,292 100% 16.85MB/s 0:23:35 (xfr#1153, to-chk=673/2100)

rsync

I am in the process now of migrating all of my data disks from RFS to XFS. I essentially mounted a spare drive using SNAP, copied all contents from DISK1 to it, formatted DISK1 as XFS and copied everything back to it. The important part is to leave the parity drive enabled so the new bits are written to it. Unfortunately, this makes the copy-back process SLOOOOOOOOOW! I'm probably averaging ~20MB/s so you can imagine how long 1.8TB will take. Right now i'm looking at ~25 hours. Of course, you can take parity out of the mix and speed things up but bad things could happen. 2 down...8 to go.