January 29, 201214 yr Hi Gurus! Introduction: I am new to unRAID as can be recognised by my post count, but have a decent amount of IT experience (non-linux though). My old QNAP had about 5TB of media on it and it was getting full, so have built an unRAID server based on a Norco 4224. Specs: Case: Norco 4224 M/B: SuperMicro X8SIL CPU: Intel [email protected] RAM: 3GB DDR3 HDD: 6 x 3TB WD Caviar Green (Spread across 3 x Supermicro SAS Controllers) Cache: 1 x 60GB OCZ SSD (Onboard SATA Controller) unRAID: 5.0-beta14 I have set up a number of shares and am comfortable with the understanding of split levels. Current Issue: I have been running this setup for about a week and today the server is practically unresponsive. By this I mean that accessing webpages on the server, such as the unraid page or unmenu page, as well as accessing files, is practically impossible, being anything from a minute or two wait, up to simply timing out. Looking to find out what went wrong, how to fix it and how to avoid it happening again. A key result of this issue is that I see the cache drive going at about 1.5MB/s, which is incredibly poor for a snappy new SSD, I am hoping there is a system wide issue and this improves back to the 100+MB/s that was happening during the initial data migration. History: A couple of days ago (being a couple of days after I built the server) my system was running smoothly, I had migrated the 5TB across and unRAID was all green lights. Then in the morning I noticed things running poorly, noticed that a disk was showing DSBL_NP, etc. I googled this issue and found it was a dead drive, which indeed was the case. So had the drive replaced, yadda, yadda, get the drive in, rebuild the array, all green lights again. Parity check to be sure, 0 errors. Great! Today, my server is unresponsive, by this, I mean that as I bounce around the unRAID menu, it would take 1 minute to load a page, sometimes even time out. To access files was impossible. During this time I was not attempting to do anything else on the NAS. Not trying to copy files, view files, etc. I have taken a number of screen captures to compliment my descriptions: Image Gallery: http://imgur.com/a/Kyn4G Image #1 is my top during this time. As you can see, no cpu usage, all looks ok. Image #2 shows disk performance page from unmenu, as you can see no disk activity. Image #3 shows a shell version of bwm-ng, which agrees with unmenu. After googling for hours and asking linux friends, I didn't have much luck, but then noticed in the image showing my top process, there is a large % for %wa (78.6 in the image). Which googling and logic seems to mean that the IO is bottlenecked. So at least I had a lead. In between all the spindowns being written to the syslog, I eventually noticed that "cache disk full" is being spammed to the syslog. Another lead! Viewing the contents of the cache drive, it was indeed full of random media files, which is bizarre, I haven't added to the disk for days, the mover process should have been done and dusted, so I thought maybe mover had somehow crashed and killed the process and re-ran it manually, so I could see the results. It attempts to start moving files, but incredibly slowly. Image #4 While doing this, I jump to bwm-ng and set it to 10s avg and I see a quite regular ~1400KB/s Tx from the cache drive. Image #5 Every now and then, the other drives would all jump up like crazy. Scrolling back through my syslog, at 3:40am mover started and everything looks just fine until 5:22am, where syslog gets very red. Pastebin of syslog cut http:// pastebin.com / WNaVF6tk (FYI, the forums 404 when a pastebin link is given) Then after that, it's all just fine, it slowly grinds through files and pops out a regular "cache disk full". At the time of writing this, things are still as described, but unraid thinks things are just fine. http://imgur.com/uQN3t Question: So, after all this information, I am curious whether anybody has any hints as to what may have gone wrong? Whether my system will be fine recovering itself after some time or whether I need to perform some process to recover the system. Whether this will happen again? Thanks in advance for any responses. My fingers are crossed UPDATE: 30 hours since it first started, issue still remains. SMART reports posted in thread. pastebin urls still 404 a forum post. UPDATE: I have traced the source of the issue down to the fact that *all* copy operations to the unraid share cause the system to grind to a halt, my posts further down explain this in detail. I have re-labelled the thread, since it seems it was not accurate, the issue was far more broad. The original issue was just one symptom from the root cause. Cheers, Matthew.
January 29, 201214 yr Author Hopefully the following reports are what you mean. Thanks for the help. Please remove the spaces around the forward slash to view. The unRAID forums 404 when the spaces are included. sda (Cache) - http://pastebin.com / kCkVkx3v sdb (Flash) sdc (Disk 3) - http://pastebin.com / z4e8k3gz sdd (Disk 4) - http://pastebin.com / JbrwX66U sde (Disk 1) - http://pastebin.com / GwFaGtnd sdf (Disk 2) - http://pastebin.com / SR2UMSFB sdg (Disk 5) - http://pastebin.com / SjbRrKCy sdh (Disk 6) - http://pastebin.com / UEZGXxKD
January 30, 201214 yr The URLs don't work. Please attach the files to a post or paste each report in a post.
January 30, 201214 yr Author Apologies, I will try to remember to add a note to remove the spaces whenever I post pastebin links. If you try posting a pastebin url into a forum post here, the forums will 404, so the spaces are sadly an annoying necessity.
February 2, 201214 yr Author Hmm, 3 days, 66 views and no thoughts from the community on what could be the issue. As the issue is still occurring, I'm obviously still open on working out what is going wrong. I'll spend a bit of time today going through logs and see if I can gather up some more info to help out. And I'm back. Alrighty, does anyone know how unRAID handles things when a drive goes down and you rebuild it. Specifically regarding moving files across a split share? Does unRAID do "file balancing" or something (my understanding is that it does not)? Or is there some other time that unRAID moves files around, and in the process, for some reason, sends them out through the user share, thereby running through the cache drive? The situation seems to be, mover keeps moving my files off from the cache drive, which clears some space and then the system re-fills the cache drive. All this accessing keeps filling the cache drive and the system crawls to a halt as IO threads get bottlenecked. For an extra level of diagnosis, is there a way I can see what files are being copied where? I can work out where the mover script is moving files to, as I know where my split levels are set, so file are allocated with a known directory to drive link. But the file that gets chucked onto the cache drive, to fill the new free space, straight away, I have no idea which file gets put on there or by what process. Thoughts? Cheers.
February 3, 201214 yr Author Alrighty, the situation is half resolved. I now know what is specifically occurring in the system (thanks to lots of grep and lsof), I just don't know why it is occurring. Perhaps someone here can help out with that part of it. The situation seems to be that I have a bunch of files that are sitting in the sabnzbd complete folder, ready for sickbeard to process. But the cache drive is full, so SB just sits and waits for some free space. As mover, clears space, SB fills it back up. Last night, I killed SB and left mover to do its thing, by the morning it had emptied the 60GB drive and all was peachy. I just re-started SB and now the transfer of files from the SAB complete folder to the cache drive is still only 1-2MBps and 'top' shows >50%wa and the system is crawling again. The crux of the issue is, that this *should* occur very quickly, the cache drive is an SSD, yet the read/write operations are going at 1-2MBps If there is a bunch of stuff to process, and the drive is running at about 1% performance, you can see where this starts to go pear-shaped. So the question remains, why is my cache drive only transferring at 1-2MBps? UPDATE: 12 hours later, with SB running, the cache drive has been refilled, at 1-2MBps, the whole time the server was practically unresponsive. I have now shutdown SB and run mover, the cache drive is now moving stuff off to the array at ~30MBps. Still no closer to resolving the issue, since I know as soon as I turn SB on, the system will fall over again. Cheers.
February 3, 201214 yr I would say you have some settings wrong between sab and sickbeard. I have both installed and don't have any issues. Sent from my ADR6350 using Tapatalk
February 23, 201214 yr Author I've given the issue a fair bit of time now and it still persists. It is not related to sb or sab. I have found I can simply copy a file from a disk location to the unraid share and it will cause the issue. The original issue of the cache drive filling up, was simply a consequence of the system grinding to a halt when doing copying operations and not being able to clear stuff in time before more was added. If anybody has any diagnostic techniques to be able to get more information about the root cause of this, I'll try things out and let you know. But currently, I feel it is an issue of unraid and the config here. Update: I have found a great, reproducible test. cp <file> /mnt/disk1/TV/ <-- This copies a file full speed, a 300mb file transfers in a few seconds cp <file> /mnt/user/TV/ <-- This causes the issue, grinds the system to a halt, sends sysload up to 5 or 6 and beyond, and takes about a minute to complete. Update: Below are rsync results for 3 different file operations rsync -Ph <file> /mnt/disk2/TV - 365.65M 00:12 27.09M bytes/sec rsync -Ph <file> /mnt/user/TV - 365.65M 01:12 5.04M bytes/sec rsync -Ph <file> /mnt/user0/TV - 365.65M 00:21 17.01M bytes/sec Ideas?
February 23, 201214 yr Author Alrighty, I am going to try and get as much information as possible, by doing a pretty comprehensive set of rsyncs to show where the file operation issues are occurring. I am finding that files move around in bursts, so different file sizes get affected in different ways. The test: I have 3 files called 5MB File.avi, 50MB File.avi and 350MB File.avi (they are not quite the exact size, but close enough from the files I had lying around) I have placed these 3 files under /mnt/disk1/TV_Series/test I have created a second folder /mnt/disk2/TV_Series/test2 The "TV_Series" share is setup with split level 1, so anything posted to /mnt/user/TV_Series/test2, will be forcibly placed on disk2. The share goes from disk1-4, so the disk5 test is in place to test outside of the share. The "TV_Series" share is setup with a cache disk, which as described originally, is a 60GB SSD. There is no sickbeard or sabnzbd process running during any of these tests. An issue I foresee, is that all file operations to the share, will simply dump onto the cache ssd, so don't really show up any issues from the array, etc. This is ok, since the real issue that is occurring, is happening during this initial move. The purpose of this test is to get results that show the issue, and they will. Additionally, I will do a dump to mnt/cache to compare results I will perform rsyncs and post the results for the following operations: disk1 -> disk2 disk1 -> disk5 disk1 -> mnt/user/TV_Series disk1 -> mnt/user0/TV_Series disk1 -> cache Results attached in log.txt for all 5 tests. Reviewing the results: It is clear that direct transfers from disk to disk copy just fine (as per disk2/disk5/user0 tests). They do trigger a slight increase in loadavg and %wa in top, but overall, they get files moved around. The two failed tests are user/ and cache/ which to me points towards the cache drive. However, I am not yet convinced it is the disk itself, as it starts transferring just fine. What occurs is, that the system loadavg skyrockets and using 'top' shows a %wa heading up to 100%. Therefore, the system grinds to a halt and is unable to make good use of the disk. The underlying system slowdown is present in all 5 tests, it just seems to be exaggerated with the 2 tests on the cache disk. I am pretty sure the cache drive is not the culprit. I have attached smart and hdparm files of the cache drive. And for reference, I did a quick benchmark on the ssd, twice. hdparm -tT /dev/sda /dev/sda: Timing cached reads: 10260 MB in 2.00 seconds = 5135.64 MB/sec Timing buffered disk reads: 318 MB in 3.00 seconds = 105.95 MB/sec hdparm -tT /dev/sda /dev/sda: Timing cached reads: 10574 MB in 2.00 seconds = 5293.06 MB/sec Timing buffered disk reads: 320 MB in 3.01 seconds = 106.22 MB/sec Cheers, Matt. log.txt hdparm.txt smart.txt
February 23, 201214 yr Can you disable the cache and try a copy test to the share? Just trying to rule things out. I wonder if performance would improve if the cache had more free space. Try that test if disabling the cache improves things.
February 23, 201214 yr Author Yep, no probs. I did try that at one point during the testing as I started to lean towards the cache drive, but didn't take much notice of the results. I'm loading the shares page while typing this... Alright, turned off. Okay, so copying across in windows, which I was doing at the time I started writing this, hasn't improved. To be certain, let me move the 3 test files across and see how it goes... Nope. Same as before, it is able to send off about 80MB or so before the speed drops to a crawl for the remainder, taking minutes to copy just a few hundred mb. I can't say precisely if it is any slower or faster, since the numbers always fluctuated, both before and now, but ultimately I place it as there being no change. FYI, the loadavg at the end of getting through the 350MB File was about 10, 7, 4. Just insanely bringing the system to it's knees, with nothing more than a local, single file copy. I did try copying a 1.3GB File earlier from windows, it can't even complete it, it just times out. Cheers.
February 23, 201214 yr How full was the cache disk during these tests? Have you set a minimum free space for the cache drive? How are you trying to run the server - download and process with the mover running every so many minutes? I expect you need a bigger cache disk to run those apps. You shouldn't be running out of disk space while downloading and processing if you want everything to work OK. I use a 500gig drive and do a move once a week. I would suggest you use a larger drive and set the mover to run as necessary. Other people have complained the server and the shares become very slow if a data disk is filled very close to full. This may be the issue you are running into with the cache disk always being full. Or else, your SSD or the connections or SATA port is simply bad. Peter
February 23, 201214 yr Author Hi lionelhutz, Thanks for your suggestions. I'm going to presume you have skimmed through this thread and tried to respond with the basics, which in some cases can work when the original poster doesn't bother to spend time doing their research, such as googling for similar people having similar issues. But, it would seem you haven't read all of the posts in this thread thoroughly, some of which already address your suggestions. The cache disk has been empty the entire time during all of these tests. The cache disk has a minimum free space set as 2000000. It seems to do it's job of filling up through the day and emptying at night. The server is quite vanilla, I do download and let the mover do it's job. With the smallish amount of downloading I do, it seems to be able to hide it's pain from me through the night. I know that it struggles as it does move things around, but out of sight, out of mind. The real crunch comes when I try to move a file to the NAS and it is just unusable. (Note: What I am getting at is, it would be unusable when the mover is doing it's job, I just tend to be asleep at that time) re: bigger cache drive, If I was constantly hitting full on the cache drive, I would agree with you. But as this thread has developed (and I have mildly updated the first post and even the thread subject with these details) it is clear that the cache drive has little or nothing to do with the issues I am suffering. The cache drive filled up because the system is not performing file operations properly and I flooded the array at the beginning of it's life. The cache drive no longer gets anywhere near full. It would be lucky if it grabs 10GB per day. Which it clears out at 3am in the morning. If I turn the cache drive off (as was tested in the post before yours), the array still copies things at less than 1MB/s locally. I have 6x3TB drives, 1 of which is parity. Of the 5 data drives, they are all over half empty (two of the drives are practically untouched). The cache drive is rarely even reaching 10-15GB out of 60GB per night. The connections would make sense if hdparm didn't come back with full hdd speed stats. Or more importantly I have logs here of copy operations working just fine from one disk to another. Please review the attached log files for all of this information available. Cheers.
February 23, 201214 yr I'm curious of the no cache test. I'm thinking if the cache is disabled the the behavior remains we may need to focus on the OS and motherboard combo.
February 23, 201214 yr I did look through most of your postings and not much stuck out. The problem described points to your cache drive having failed and no longer able to accept writes properly. I suggest you test the cache drive further, maybe even pull it out and try formatting with a different file system and do read and write tests using another machine. Your comment about the cache disk disables tests was and odd addition (suddenly every drive has issues???) but I pretty much ignored it due to the other testing you had already posted about which pointed to the cache drive being at fault. Also, why didn't you completey remove it from the device assignment or better yet unassign it and unplug it to do those tests? Try posting a syslog so readers can look at it. Have you run a filesystem check on each disk? Also, you do know that /mnt/disk1/TV and /mnt/cache/TV are both a subset of the /mnt/user/TV share? I don't know what will happen when you try to copy from the share to the share location on a disk or from a disk share to the same user share but I doubt anything good will come of it. I don't believe you are doing this, but you didn't really provide enough info to know. Peter
February 24, 201214 yr I'm going to presume you have skimmed through this thread and tried to respond with the basics, which in some cases can work when the original poster doesn't bother to spend time doing their research, such as googling for similar people having similar issues. But, it would seem you haven't read all of the posts in this thread thoroughly, some of which already address your suggestions. Just for the record, could you point out in where in your posts you; Posted that the cache drive was empty when you did the write tests to it? Posted that you had set a minimum free space on the cache drive? Posted how much you were downloading and how often you had the mover set to run? You did call me out as if I didn't read any of your postings and was providing "generic" answers that had nothing to do with your issue. Yet, not of the questions or info I posed had been addressed by you in your posts. I just happened to have a little extra free time to go back and read through them again to make sure. I did find that you posted this; UPDATE: 12 hours later, with SB running, the cache drive has been refilled, at 1-2MBps, the whole time the server was practically unresponsive. I have now shutdown SB and run mover, the cache drive is now moving stuff off to the array at ~30MBps. Still no closer to resolving the issue, since I know as soon as I turn SB on, the system will fall over again. Which oddly enough strongly suggests you did download enough to fill the cache drive within 12 hours. Yet, you responded to my post and said that never happens. How would I know that never happens when you first posted info directly contrary to it never happening? Just for curiousity, was there any share data still on the cache drive when you disabled it from the share. That would screw up the share because that data would still be accessed and included as part of the share even if the cache was no longer used for new writes. Same as excluding a disk from a share just stops new writes to the disk but leaves the existing data as part of the share. I'm with dgaschk and I'll read your post but I'm not spending the time to work those links to get to those SMART reports and without looking at them it could be any of the disks which have a problem. You should make it easier for others to access the data which could help you. I would first pull the SDD completely just to see what happens. I would also do a reiserfsck filesystem check on every data disk. I would also post a syslog from the server after it has been running and acting up for a while (probably 1/2 hour of the programs running with the bad behavour would do). Peter
February 28, 201214 yr Author I'm curious of the no cache test. I'm thinking if the cache is disabled the the behavior remains we may need to focus on the OS and motherboard combo. The O/S is straight from an UNRaid flash stick, I personally believe this is indeed an O/S problem. Something related to drive settings perhaps, or to do with UNRaid's handling of 3TB drives. The M/B is SuperMicro X8SIL, which is straight from the 20 drive beast build on the wiki. The things that points me away from the M/B is the hdparm tests that show things are great and the fact that logs of copy processes show that speed is high when not copying to an unraid share. As wello as the fact that copies to the unraid share show an initial burst. There is also a new test that I did, which I'll put into one of my main posts, where I can copy a directory with 30X50mb files and it will transfer at full speed, to the unraid share. Whereas if there is a 1x1.5gb file (same overall filesize quantity) it will usually not even finish, will most definitely take minutes to copy, but mostly times out.
February 28, 201214 yr Author Please attach reports and logs to post. I don't have time to fix the URLs. You have time to download files and open in a 3rd party app, but can't remove a space in a url? Fair enough. You know what would be great? If these forums didn't 404 every time you type an actual pastebin url in to a post. Please find attached, the 7 smart reports, being my 6 data/parity drives and cache drive. I would have kept them as txt files, but the forum also has a 4 file attachment limit. Oh and rar files, can't add them either... smart_logs.zip
February 28, 201214 yr Author Hi All, How long does it take for a cache disk to be removed from a share? As in, if I go to the GUI, Shares, Select Cache disk to No and hit Done. How long does it take for that action to take place? Last week, when daniel.boone suggested to remove the cache disk and re-test the share, I did so, and performance did not change. Even two days ago it hadn't changed. Yet now it has, copying files is fine. The system hasn't rebooted, infact, the system has been idle for over a couple of weeks (since I have no quota) except for copying files locally as part of these tests. I'm assuming what lionelhutz mentioned about the cache disk still being accessed if there was data on it, could have kept the cache disk involved. Although I would have expected that to last no longer than a night, when the mover script cleaned things up. Even more bizarre is that for a few days it didn't change, then out of the blue it now seems to have changed. A test that I am running right now, shows that the following: 1GB File being transferred from mnt/disk1 to mnt/user/TV copies at ~30MB/s which is expected. Also doesn't add much system load, at least not a detrimental amount. 1GB File being transferred from mnt/disk1 to mnt/cache copies at ~2MB/s which is what has been the issue all along. It also brings the system to a loadavg of 9+. So, ignoring all of the evolution of information from this thread (it went through a lot of ups and downs of change, most of it is now misinformation in the current context), based on the current, clear evidence of the issue writing to the cache drive, yet smart reports ok, hdparm reports ok, what may be the issue with the cache drive? Cheers, Matt.
February 28, 201214 yr No idea what the problem is with it. If the SMART tests say it's OK but it won't work then it's probably a firmware bug or it's just screwed up inside. I do recall about a week ago suggesting you remove it completely to test.... I also recall about a week ago suggesting to put it in a different machine and testing it there...
February 28, 201214 yr Add the cache back but dont enable any of the apps and retest. I would leave it in place for a couple of days. If the problem returns you know it's the cache drive. As mentioned could be the firmware or even unRaid. I'm sure more testing at this point may not be what you'd hope but it's the only way to be sure. Glad things are looking better
Archived
This topic is now archived and is closed to further replies.