JustinChase Posted September 21, 2014 Share Posted September 21, 2014 I'm looking at 2 red-balled drives, one has a drive in the box, the other says missing disk. I've checked the cables twice, and the third time I moved the drives around, to connect new/different SATA and power cables to both drives. The errors followed the drives, so I doubt it's a cable issue at this point. How I got here... I had to hard boot the machine a few days ago, and just remembered last night to start the parity check I cancelled because I couldn't watch anything with parity check running the night I had to hard boot. So, I started it at about 10pm last night, and at 8pm this morning, parity was 4% finished, which is crazy slow. I cancelled that parity check, then noticed a couple thousand sync errors (not happy). I rebooted again, then started the parity check again. It started off fine, but after about an hour, was going very slow again. During that hour, I tried to launch my deluge docker, but it wouldn't load. I looked at the docker plugin and it said it was started, but still I couldn't connect. I tried to uninstall the docker, but that gave me an error. I grabbed a syslog, and it looks like disk10 (where the docker image resides) is having problems. I rebooted again, and the docker plugin won't even start now. My son started watching a movie, and it was stuttering frequently (not an uncommon occurrence unfortunately), but I suspected it might have to do with disk10 being sketchy, so I shut down the server, and checked all the cables. I remembered I had a new drive sitting 'loose' in the server from when I had the last red ball, about 2 weeks ago. I removed the old red-balled drive from its caddy, and put the new drive in its place, checked all SATA and power cables, and started the server. I ended up with 2 red-balled drives, one as missing and the other listing the drive. I shutdown, moved the drives around, and started again. red-balls followed drives. Before I make things any worse, what should I do at this point? I don't currently have any blank drives to install, but I do have the old 2TB drive that was previously red-balled, most likely due to a cable issue. I had planned to preclear it to test it before using it in something else, but even if that worked, it's smaller than both current red-balled drives. So, what do I do now? syslog.zip Quote Link to comment
jphipps Posted September 21, 2014 Share Posted September 21, 2014 From your syslog and the image, it looks like the disk listed in the drop down is the same serial number as the one it says is missing on disk5. Can you assign that disk and it show up as the correct disk before starting the array? Quote Link to comment
JustinChase Posted September 21, 2014 Author Share Posted September 21, 2014 I couldn't assign it. When I selected it, it just didn't 'accept' that. I finally decided that rebooting one more time couldn't hurt, and when I did, it found that disk, and added it just fine. Now, I just have the one red-balled drive, disk 9. That allowed the array to start, so now I'm moving all the data off of disk9 to the other disks, and will try to pre-clear disk9 once finished, and see if it 'passes' without error. If so, I'll format it to (probably) XFS, then begin the transferring of data onto that disk, and continue to change formatting of the rest of my drives. Hopefully this is the end of the red-ball drives I'll need to deal with for quite some time. Quote Link to comment
jphipps Posted September 21, 2014 Share Posted September 21, 2014 Glad to hear.. Sounds like a good plan... Quote Link to comment
JustinChase Posted September 21, 2014 Author Share Posted September 21, 2014 Thanks. I wonder; is Midnight Commander the 'best' way to move the files around? I've seen rsync mentioned, but am not sure how to access/use it, or if it's worth my time to learn. MC is going pretty slow (~15Mb/sec) right now, but I think that's likely a function of it actually using parity to move the data, since it thinks the drive is missing. Either way, I would like to use whatever tool is fastest, but safe. The sooner I get everything off the red-balled drive, the better I'll feel. Quote Link to comment
eroz Posted September 21, 2014 Share Posted September 21, 2014 I thought I saw a post saying to use rsync instead of mc. Then there would be 2 ways to do it. One that copied the files to the new drive and then deleted the files from the old drive. and the second method to just copy the files to the new drive and then reformat the drive once you verified all the files were copied. Quote Link to comment
jphipps Posted September 21, 2014 Share Posted September 21, 2014 I have been using rsync like crazy moving things around... I have used it for years and it has always worked great for me... I usually use: rsync -av --progress --remove-source-files /mnt/diskX/ /mnt/diskY/ to migrate from diskX to diskY It removes the files from the source drive when they are completed successfully. Nice thing is you can restart it anytime and run a final rsync before formatting to make sure you have all your files moved. Some skip the --remove-source-files because of the over head of deleting since you are formatting the source drive anyway, but I like the sanity of knowing all the files have been removed/moved. You can either use screen to run them, but I just run them from a normal ssh sessions. If it dies, you can aways restart and it will start were it left off.. Quote Link to comment
JustinChase Posted September 21, 2014 Author Share Posted September 21, 2014 Okay, I'll try that when this mc batch is done. Sadly, I don't have enough room to move everything to one drive, none have enough room, so I'm going to have to split it into more batches. I'll try the rsync next. thanks again. Quote Link to comment
jphipps Posted September 21, 2014 Share Posted September 21, 2014 No problem.. You can also specify more exact criteria on the source/destination. I had the same issue with space, I did it down to the directory level to split the data among other drives. Quote Link to comment
JustinChase Posted September 22, 2014 Author Share Posted September 22, 2014 so, rsync is moving so slow, and now I've noticed some issues as it's going along. One example is this... rsync: sender failed to remove Formula 1/Season 2010/Formula 1 - s2010e18 - PreRace.mp4: Read-only file system (30) Also, after it's finished moving some directories, it's left some/several directories behind. They are blank, but there are still lots of empty directories in the folders that rsync has finished with. I then tried to 'move' the directories with MC, but it said "cannot move directory ... Read-only file system (30)" I've been getting some similar errors when updating tags in my media player, so I suspect there are more issues here. So, how to finish getting all my files moved, so I can get this drive cleared? Quote Link to comment
itimpi Posted September 22, 2014 Share Posted September 22, 2014 In my experience, rsync runs as fast if not faster than the 'cp' command and has major advantages. If rsync is running slow then there may be problems reading the source files. A syslog may well show errors being reported. It is normal for rsync to leave empty directories around if you have used the --remove-source-files option. The fact that the disk appears to be treated mounted as read-only will stop rsync from removing source files, or from you manually removing files after copying them. Quote Link to comment
JustinChase Posted September 22, 2014 Author Share Posted September 22, 2014 so, how do I change the "read only" nature of the files on this disk? the newperms script doesn't do it, so I don't know how to 'free up' my files, so I can move them off of this disk. Quote Link to comment
itimpi Posted September 22, 2014 Share Posted September 22, 2014 so, how do I change the "read only" nature of the files on this disk? the newperms script doesn't do it, so I don't know how to 'free up' my files, so I can move them off of this disk. The only way is to fix whatever is causing the disk to be mounted as read-only. If you are going to replace or rebuild the disk why is it important that you remove them? Surely the important thing is to have copied them. The fix process will almost certainly be to run the reiserfsck utility against the disk to rebuild the file system control structures. However that takes a long time and if the disk is already prone to failing could easily push it over the edge. Quote Link to comment
JustinChase Posted September 22, 2014 Author Share Posted September 22, 2014 some of the files failed to move, not just be removed after. I ended up rebooting the server, then ran newperms again, and it didn't obviously fail this time. I'm working on moving the files again now. Sadly, it looks like this could take a few days to finish Quote Link to comment
JustinChase Posted September 22, 2014 Author Share Posted September 22, 2014 okay, it's getting worse When I try to move files, it works for a short time, then fails with 'read only' permissions errors. If I stop the array, disk5 shows as missing again. if I reboot, it comes back, but then when I try to move files off of disk9, it works for a bit, then fails again. The time it works before failing is getting shorter each time. I really don't want to make things worse, but I do want to get everything off disk9, so that at least I'd be down to one disk giving me issues. Ideas on how to proceed? Quote Link to comment
jphipps Posted September 22, 2014 Share Posted September 22, 2014 If you run the rsync without the --remove-source-files does it still fail? Can you post your syslog when you get the read-only failure? I found it does take about 24 hours for about 2TB on a single copy... Quote Link to comment
itimpi Posted September 22, 2014 Share Posted September 22, 2014 You will not be able to MOVE files as the disk is mounted read-only. Any attempt to so is just likely to aggravate the issue. You just want to copy them (if you can). It is possible that some will not be copyable because of your current problems. Quote Link to comment
JustinChase Posted September 22, 2014 Author Share Posted September 22, 2014 Hmmm, I've moved several hundred gigs already, it's just that occasionally I lose connectivity and have to reboot the server, then it's all good again, until it's not; and I have to reboot again. Quote Link to comment
itimpi Posted September 22, 2014 Share Posted September 22, 2014 Hmmm, I've moved several hundred gigs already, it's just that occasionally I lose connectivity and have to reboot the server, then it's all good again, until it's not; and I have to reboot again. I believe that you have file system corruption on that disk. You are OK until that is noticed when accessing a file, and then the disk is changed to read-only. The danger with removing files is that it might worsen the corruption and thus make other files inaccessible. However, you can keep going the way you are knowing the risks. Quote Link to comment
JustinChase Posted September 22, 2014 Author Share Posted September 22, 2014 that seems like a likely explanation. Assuming so, how might I fix the problem, instead of making it worse? From what I can tell, I've gotten all the 'important' stuff off the drive and anything still on there, can be replaced, but will not be fun or easy. Quote Link to comment
jphipps Posted September 22, 2014 Share Posted September 22, 2014 I am having a similiar issue right now. I had converted most of my drives to BTRFS, and after several crashes, when I run a scrub on them, it states there is unrecoverable errors on all of the filesystem. I am just rsyncing everything I can over to XFS and than deal with the corrupted files at the end... Quote Link to comment
JustinChase Posted September 22, 2014 Author Share Posted September 22, 2014 Yeah, I think XFS is going to be my choice for now also. once I get disk9 emptied, I'll preclear, format to XFS, then start moving files onto that drive. I'm tempted to buy a 4TB drive to use as a cache for a short time, so I can move user shares onto the cache, then move from cache to the formatted drive. Reason: I've seen that my files have gotten rather scattered across all my drives. I have a video on one drive, the xml for that video on another drive, and the jpg on yet another drive. I imagine this doesn't help when browsing or watching my library. I guess I'll worry about that once I have disk9 emptied. Quote Link to comment
eroz Posted September 22, 2014 Share Posted September 22, 2014 Justin, I don't think you have to preclear a drive that was already precleared and is in use right now to convert it to XFS. I would only run the preclear to stress the drive if you are worried about it. I'm also in the process of switching over to XFS, currently preclearing a new 4TB drive. Quote Link to comment
JustinChase Posted September 22, 2014 Author Share Posted September 22, 2014 Justin, I don't think you have to preclear a drive that was already precleared and is in use right now to convert it to XFS. I would only run the preclear to stress the drive if you are worried about it. No, I don't have to, but I do want to stress the drive, since it red-balled on me. I just want to be sure it's okay before putting it back into service. Quote Link to comment
JustinChase Posted September 23, 2014 Author Share Posted September 23, 2014 AARRRgggghhhhhh!!!!! So, now disk 9 is showing as unformatted, meaning I can't access it at all. Can/should I run reiserfsck --check /dev/md9 on it in this state, or do I just accept that the 1TB of data still on it is lost, and move on? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.