trurl Posted August 7, 2017 Share Posted August 7, 2017 2 hours ago, ptirmal said: At the time I thought maybe the parity disk wasn't updating during the transfer. You can forget about that ever happening unless parity is disabled. Quote Link to comment
ptirmal Posted August 7, 2017 Share Posted August 7, 2017 (edited) So I'm not quite sure what's going on. I ran docker safe permissions, nothing changed. Crashplan shows missing files same as when I access via samba or console so I'm guessing they didn't move before the format? . I couldn't see anything that looks like it didn't copy, and unraid is showing the capacity was used. However, it appears Crashplan is not showing the data is there, but it's telling me there isn't enough space to restore (attached) which makes it seem permission related. Backup is the share with most of the stuff that's missing. Thoughts? Edited August 8, 2017 by ptirmal Quote Link to comment
jbrodriguez Posted August 7, 2017 Author Share Posted August 7, 2017 Ok, I checked the logs There were two separate errors transferring files The first one I: 2017/08/06 07:27:43 core.go:740: Running Move operation ... I: 2017/08/06 07:27:43 core.go:857: Command Started: rsync -avPRX "Media/movies" "/mnt/disk5/" W: 2017/08/06 12:39:00 server.go:259: closing socket (&{id: ws:0xc4200e4240 onMessage:0x6cf050 onClose:0x6cf0a0}): EOF I: 2017/08/06 16:15:34 core.go:939: Command Finished I: 2017/08/06 16:15:34 core.go:945: Current progress: 64.79% done ~ 4h46m52s left (39.78 MB/s) I: 2017/08/06 16:15:34 core.go:957: Removing: rm -rf "/mnt/disk2/Media/movies" W: 2017/08/06 16:15:34 shell.go:53: transferProgress:: rm: cannot remove '/mnt/disk2/Media/movies/Cake (2014)/Cake (2014) 720p.mkv': Permission denied W: 2017/08/06 16:15:35 shell.go:101: transferProgress:: waitError: signal: broken pipe W: 2017/08/06 16:15:35 core.go:968: Unable to remove source folder (/mnt/disk2/Media/movies): signal: broken pipe It fully copied "Media/movies" to disk5, but failed to delete the source folder (/mnt/disk2/Media/movies), due to a permission issue. In this case, the "Media/movies" data is in disk 5. Run $ ls -al /mnt/disk5/Media/movies to list the files. After you've double checked that, you can delete the files from disk2 (rm -rf "/mnt/disk2/Media/movies"), so you can further debug the permission issue. The second error I: 2017/08/06 16:15:35 core.go:857: Command Started: rsync -avPRX "Media/shows" "/mnt/disk5/" I: 2017/08/06 17:00:16 core.go:143: Sending config W: 2017/08/06 19:57:57 server.go:259: closing socket (&{id: ws:0xc4201841b0 onMessage:0x6cf050 onClose:0x6cf0a0}): EOF I: 2017/08/06 19:57:58 core.go:143: Sending config W: 2017/08/06 20:16:13 server.go:259: closing socket (&{id: ws:0xc4201842d0 onMessage:0x6cf050 onClose:0x6cf0a0}): EOF I: 2017/08/06 20:16:14 core.go:143: Sending config W: 2017/08/06 20:22:59 core.go:929: Command Interrupted: rsync -avPRX "Media/shows" "/mnt/disk5/" (exit status 23 : Partial transfer due to error) I: 2017/08/06 20:22:59 core.go:1039: unBALANCE - MOVE operation INTERRUPTED Started: Aug 6, 2017 07:27:43 Ended: Aug 6, 2017 20:22:59 Elapsed: 12h55m16.176s Command Interrupted: rsync -avPRX "Media/shows" "/mnt/disk5/" (exit status 23 : Partial transfer due to error) happened while transferring "Media/shows", so in this case the data still exists whole in disk2. Run $ ls -al /mnt/disk2/Media/shows After you've checked that, you can delete the incomplete transfer on disk5 As for the other share (Backup), the app didn't get to transfer it (because of the previous error), so the files are still present in disk2. Check that by running $ ls -al /mnt/disk2/Backup Not sure what you mention about crashplan, but the data in the raw disk should be there as per above. Also is it possible that you were receiving crashplan data at the same time as the app was running ? The app calculated these sizes for the folders you've shown in the screenshot, while the screenshot shows biggers folder sizes ? [64.9G] /mnt/disk2/Backup/./Pictures [40.4G] /mnt/disk2/Backup/./Documents [263.2M] /mnt/disk2/Backup/./Videos Quote Link to comment
ptirmal Posted August 8, 2017 Share Posted August 8, 2017 Those disks were already formatted... I assumed (incorrectly) that the disks had moved the files because it said it was complete. I'm having a separate issue restoring from my crashplan backup. Quote Link to comment
jbrodriguez Posted August 8, 2017 Author Share Posted August 8, 2017 11 hours ago, ptirmal said: Those disks were already formatted Mmm, that's definitely not good Not sure if your notifications were enabled, but you'd have been able to read that the transfer was interrupted Quote Link to comment
ptirmal Posted August 8, 2017 Share Posted August 8, 2017 (edited) 5 minutes ago, jbrodriguez said: Mmm, that's definitely not good Not sure if your notifications were enabled, but you'd have been able to read that the transfer was interrupted Yea it sucks but that's what backups are for right? I had unbalance on the default settings so I don't know if notifications are enabled by default. I did notice when it was complete it wasn't let me click validate or anything else. I should have looked into it further but I figured it transferred it all since unraid was showing what looked like the correct percentage increase. Oh well, live and learn, could have been worse. And it looks like I'm restoring from my backup so whatever I deemed important survives. Edited August 8, 2017 by ptirmal Quote Link to comment
jbrodriguez Posted August 8, 2017 Author Share Posted August 8, 2017 1 minute ago, ptirmal said: I had unbalance on the default settings so I don't know if notifications are enabled by default No, they're not enabled by default. In any case, this gives me the idea to implement a history (similar to the parity history) in an upcoming release. Quote Link to comment
trurl Posted August 8, 2017 Share Posted August 8, 2017 1 hour ago, ptirmal said: I don't know if notifications are enabled by default Do you have unRAID Notifications setup? It is very important that unRAID be able to notify you by email or other agent if it detects problems. Many people have allowed themselves to get multiple disks with problems, for example, because they didn't have unRAID configured to notify them. I haven't used this plugin lately, but I assume it uses unRAID's builtin notifications to notify you, which can't happen if you haven't configured unRAID Notifications. Quote Link to comment
ptirmal Posted August 8, 2017 Share Posted August 8, 2017 (edited) 47 minutes ago, trurl said: Do you have unRAID Notifications setup? It is very important that unRAID be able to notify you by email or other agent if it detects problems. Many people have allowed themselves to get multiple disks with problems, for example, because they didn't have unRAID configured to notify them. I haven't used this plugin lately, but I assume it uses unRAID's builtin notifications to notify you, which can't happen if you haven't configured unRAID Notifications. Yes I do. I currently have them notify me via Pushbullet. I also check in on it pretty regularly. Edited August 8, 2017 by ptirmal Quote Link to comment
dabl Posted August 12, 2017 Share Posted August 12, 2017 Is it possible to use this plugin to move files from a disabled disk to a good disk? Quote Link to comment
trurl Posted August 12, 2017 Share Posted August 12, 2017 2 minutes ago, dabl said: Is it possible to use this plugin to move files from a disabled disk to a good disk? The emulated disk can be read and written just like a real disk. It won't be very efficient since all other drives are used to emulate the disabled disk. Why do you want to do this? Rebuilding the disabled disk is the normal way to keep that data. Quote Link to comment
dabl Posted August 12, 2017 Share Posted August 12, 2017 (edited) 15 hours ago, trurl said: The emulated disk can be read and written just like a real disk. It won't be very efficient since all other drives are used to emulate the disabled disk. Why do you want to do this? Rebuilding the disabled disk is the normal way to keep that data. Thanks, that's what I figured but for whatever reason when click the CALCULATE button nothing is happening except the CALCULATE button then as expected becomes disabled. I get no console as shown in the doc. Nothing shows in the Log page and the REFRESH button on that page is disabled. Attached is a screenshot. The reason I want to do this is I was headed down the path of converting my reiserfs drives to xfs using a new empty 8 TB drive. The disk 3 that just became disabled is 1.5 TB and I have space available on the 8 TB disk 8. The 8 TB disk 9 is my new empty drive. I'd prefer not to have to rebuild the 1.5 TB reiserfs disk 3 using disk 9 but perhaps that is the best thing to do at this point and then move the data to disk 8 and start the reiserfs to xfs conversion again. To complicate matters disk 6 was showing signs of going bad so I moved the data off that disk using the krusader docker. Disk 6 is now empty and I would like to take it out of service so things don't become even worse if/when disk 6 gets disabled, right in the middle of a move operation for example. I'm open to any/all suggested approaches. EDIT, UPDATE: I turned notifications on and restarted the plugin and am now seeing the console on CALCULATE and output on the Log page so the plugin does appear to at least be working for me now. Am still interested in general input on suggested approaches to my situation. Edited August 12, 2017 by dabl Quote Link to comment
jbrodriguez Posted August 12, 2017 Author Share Posted August 12, 2017 2 minutes ago, dabl said: Nothing shows in the Log Can you run a quick $ tail -f /boot/logs/unbalance.log Just to check what it's doing. If you already stopped the process, then just check the end of the log. Quote Link to comment
dabl Posted August 12, 2017 Share Posted August 12, 2017 (edited) 1 hour ago, jbrodriguez said: Can you run a quick $ tail -f /boot/logs/unbalance.log Just to check what it's doing. If you already stopped the process, then just check the end of the log. Sorry, at the time of the problem CALCULATE run I did not know to look for the log and have since successfully run CALCULATE. For what it's worth, attached is my latest unbalance.log from a successful run after fixing some permissions on my disabled disk 3 per the OPERATION FEEDBACK screen. Also attached are screenshots from the successful run of CALCULATE with the OPERATION FEEDBACK and the run after fixing the permissions. I wanted to avoid any kind of system wide newperms run so ran it manually from an ssh session with newperms /mnt/disk3 After doing that I now seem to be getting a 'clean' CALCULATE with no orange OPERATION FEEDBACK screen. One thing I don't understand is in the unbalance.log after running newperms /mnt/disk3 it still seems to be showing a (group?) [dabl] for some files but when I look at those via ssh ls -l it shows the group as 'users'. Maybe I'm just confused on that. EDIT: also attaching the unbalance.log generated during the run with the OPERATION FEEDBACK screen since I had saved that one. EDIT: Hmm comparing the .log files I see the second run just got appended so that was probably uneccessary (duh) Also what I was seeing with the [dabl] group was actually from the first run and was NOT in the second run...... unbalance.log unbalance log from CALCULATE results OPERATION FEEDBACK .log Edited August 12, 2017 by dabl Quote Link to comment
jbrodriguez Posted August 12, 2017 Author Share Posted August 12, 2017 Ok, well looks like the plugin is doing what it's supposed to Not sure how to proceed though, knowing your disk is in 'bad' shape as you mention. rsync will probably generate some error and interrupt any operation you run. Since you have enough space in at least one disk, why don't you run rsync from a (tmux/screen) command line, so you can monitor progress more closely. Quote Link to comment
jbrodriguez Posted August 14, 2017 Author Share Posted August 14, 2017 v3.3.0 is now available ! 2017-08-14 - 3.3.0 GATHER operation is now implemented !! Consolidate folders from your user shares into a single drive, always checking that enough space is available for the operation. Also, as soon as any transfer operation gets underway, a notification is sent showing which rsync commands will be executed (this happens after you click Move, Copy or Proceed). Additionally, the following changes were made: - Verbosity setting controls permission issue logging - Fix issue with console panel not allowing mouse scroll - Drive sizes in logs are shown in 'base 1000' - Update internal components The underlying transfer is the exact same as the one used for SCATTER. Let me know how it works for you. The next planned changes are: - Implement an operation history - Run docker safe new permissions (at the source folder level) (needs discussion with Squid 1 Quote Link to comment
stefer Posted August 14, 2017 Share Posted August 14, 2017 I made the mistake (from what I gather after reading some posts) of setting my default fs to btrfs... Now i came across this post and I'm wondering how to get this done (converting to xfs). I would use unbalance to get stuff off one disk, then format it to xfs? how? stop array, ssh to array, format it with fdisk? or is there a tool in the webgui that I don't see, or a plugin to help? Then what? When I start the array, unraid will use the newly formatted drive in the storage pool and then I unbalance the others one by one until I have formattted all of them to xfs? OR does unraid create a 2nd pool that is xfs formatted? Or should I not fret and keep it as btrfs? Quote Link to comment
JonathanM Posted August 14, 2017 Share Posted August 14, 2017 29 minutes ago, stefer said: I would use unbalance to get stuff off one disk, then format it to xfs? how? stop array, ssh to array, format it with fdisk? or is there a tool in the webgui that I don't see, or a plugin to help? Then what? When I start the array, unraid will use the newly formatted drive in the storage pool and then I unbalance the others one by one until I have formattted all of them to xfs? Very close, but it's easier. With the array stopped, when you click on the disk in question you can change the format to XFS. Then when you start the array, that disk (and hopefully ONLY that disk) will show as unmountable, with an option to format ALL unmountable disks. You can then use the newly empty XFS disk as a target for copying the contents of one of your other disks, lather, rinse, repeat. If you have a stable server with no power issues, no forced shutdowns or crashes, then BTRFS is fine, great even. It has several benefits, but I would describe it as fragile, or brittle. It doesn't survive crashes well at all, and data recovery tools on BTRFS are sketchy as well. If you keep current backups of everything you care about, which you should be doing anyway, then don't fret and just leave it BTRFS. The chances of you messing something up in the conversion are non-zero, so backup, backup, backup. Quote Link to comment
stefer Posted August 14, 2017 Share Posted August 14, 2017 I have very stable power (now), but power failures do occur from time to time... Twice this year so far. And my UPS is not strong enough to support the PC for more than 5 seconds... My important stuff is on dropbox. Yeah the conversion sounds painful... even if it's easier like you explained. Since i get this error when I run the calculate option : There are some permission issues with the folders/files you want to move 5080 file(s)/folder(s) with an owner other than 'nobody' 0 file(s)/folder(s) with a group other than 'users' 268 folder(s) with a permission other than 'drwxrwxrwx' 1356 files(s) with a permission other than '-rw-rw-rw-' or '-r--r--r--' You can find more details about which files have issues in the log file (/boot/logs/unbalance.log) At this point, you can move the folders/files if you want, but be advised that it can cause errors in the operation You are STRONGLY suggested to install the Fix Common Problems plugin, then run the Docker Safe New Permissions command I'm thinking it's either my Dropbox docker, or NZBget/Sickbeard/Couchpotato dockers... I don't really know how to find out though, I checked the log and i'm not really sure what to look for. Quote Link to comment
stefer Posted August 14, 2017 Share Posted August 14, 2017 (edited) From what i can see when I ssh to my server, some shares have folders that are owned by nobody, and some are owned by a user that i created just for myself for personal stuff, budget etc... BUT the items that are owned by my user are nowhere near the personal folder. I wonder if it's because I used a Windows machine to copy stuff over those shares and I might have used that username when I mapped the drive in explorer.... which would explain the ownership. I COULD fix it with the Docker Safe New Permissions command... Question, when you say : Quote You can then use the newly empty XFS disk as a target for copying the contents of one of your other disks, lather, rinse, repeat. Once it's formatted... the array is started, but it won't part of it? Or will it? And how do you copy content over? Using unBalance again? At what point do the XFS disks become part of the array and are protected by the parity? Once they've all been formatted? Edited August 15, 2017 by stefer Quote Link to comment
JonathanM Posted August 15, 2017 Share Posted August 15, 2017 15 minutes ago, stefer said: Once it's formatted... the array is started, but it won't part of it? Or will it? And how do you copy content over? Using unBalance again? At what point do the XFS disks become part of the array and are protected by the parity? Once they've all been formatted? You can copy from disk to disk using any copy tool, rsync, unbalance (which is a gui frontend for rsync commands), krusader or dolphin docker, cp command, whatever. Parity doesn't protect files or file systems, it protects entire disks. Content and format are irrelevant to parity. All disks assigned to data slots are part of the array, whether or not they participate in user shares is up to the settings. I recommend disabling user shares and turning on disk shares while working with converting disks, until you understand what user shares are and how they work. Since user shares are a consolidated view of the disks, it's possible to wipe out data if you mix copying between user and disk shares. Quote Link to comment
stefer Posted August 15, 2017 Share Posted August 15, 2017 (edited) Thanks. I understand what user shares are vs disk shares. It's the whole adding it back to the pool ordeal that confused me... I'm used to my NASes and raid in general... Using unbalance should fix the issue as long as I stop my dockers so they don't write anything new and to make sure the drive is completely empty before stopping the array and formatting. It's funny you replied as I was reading this (one of your replies) in another tab : Will have to run the script to fix my access rights first though... After reading a bunch of posts, I think I will stick with btrfs... maybe down the line invest in a good APC ups... I have to upgrade most of my drives sooner than later anyway so I could format the new ones as XFS as I replace them. Edited August 15, 2017 by stefer Quote Link to comment
wreave Posted August 17, 2017 Share Posted August 17, 2017 (edited) Having an issue when trying to use the gather operation. It is scanning for the show and detects that it is on 6 drives. When I select the drive to move the data to and run the move operation (dry run is disabled) it quickly spits some logs and goes back to the main gather page. The logs (attached) seem to indicate it should work but is showing silly fast transfer speeds and nothing is actually being moved. Transferred 4.68 GB at ~ 131030.23 MB/s Any help would be appreciated, been waiting for the gather operation for a while Edit: I am on unRAID 6.4.0-rc7a unblanace_log.txt Edited August 17, 2017 by wreave Quote Link to comment
jbrodriguez Posted August 17, 2017 Author Share Posted August 17, 2017 Hi wreave, you didn't send the whole log, but what I can see it's only transferring 4.68gb, it does seem like it's running a dry-run. Can you send the first part of the log, with the calculate section ? Quote Link to comment
abs0lut.zer0 Posted August 17, 2017 Share Posted August 17, 2017 Hi All I know that i am going to get flammed for this but is there a PLUGIN that does the exact opposite of unbalance. Thanks Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.