JonathanM Posted March 19, 2023 Share Posted March 19, 2023 On 3/18/2023 at 8:40 AM, Hellomynameisleo said: I want to do this so it increase the I/O performance when accessing multiple files it will spin up and use multiple HDD instead of just a single one taking all the load. You do realize this will still limit your writes, since every write is updating the parity disk(s). Have you looked into turbo write? Quote Link to comment
itimpi Posted March 19, 2023 Share Posted March 19, 2023 On 3/18/2023 at 12:40 PM, Hellomynameisleo said: Is there a way to scatter where all the files are spread across all disks in the array similar to RAID (how it sripes all files across all disks) but with files instead. File-level striping at the file level rather than at the block level. I want to do this so it increase the I/O performance when accessing multiple files it will spin up and use multiple HDD instead of just a single one taking all the load. I want this as I perfer having higher I/O performance and don't mind the drives spinning up. I don't want to use actual RAID because of how inflexable it is with different drive sizes. Looks like this can be done in python, I used user script files plugin and just replace the "Disk = "with your dir destination location and "src_dir = " with your source directory. import os import shutil # Define the paths of the disks disks = ['D:/test', 'E:/test', 'F:/test'] # Define the path of the source directory src_dir = 'D:/test' # Define a variable to keep track of the current disk index current_disk_index = 0 # Recursively walk through the source directory and move files for dirpath, dirnames, filenames in os.walk(src_dir): for filename in filenames: src_path = os.path.join(dirpath, filename) # Get the next disk path in a round-robin fashion current_disk = disks[current_disk_index] current_disk_index = (current_disk_index + 1) % len(disks) dst_path = os.path.join(current_disk, os.path.relpath(src_path, start=src_dir)) # Create the destination directory if it doesn't exist if not os.path.exists(os.path.dirname(dst_path)): os.makedirs(os.path.dirname(dst_path)) # Move the file to the destination directory shutil.move(src_path, dst_path) print("Moved {} to {}".format(src_path, dst_path)) Have you checked this actually works as expected? I see a potential problem? You have the same drive in source and destination list. Copying a file to itself can.end up truncating the file to zero length. Quote Link to comment
Hellomynameisleo Posted March 19, 2023 Share Posted March 19, 2023 (edited) 7 hours ago, JonathanM said: You do realize this will still limit your writes, since every write is updating the parity disk(s). Have you looked into turbo write? I'll take a look at turbo write but it looks complicated in how it works. I'll try the plugin out. 7 hours ago, itimpi said: Have you checked this actually works as expected? I see a potential problem? You have the same drive in source and destination list. Copying a file to itself can.end up truncating the file to zero length. Yes it works as expected, just don't use /mnt/user as the source and then /mnt/disk1 /mnt/disk2 etc. as the destination otherwise it will end up deleting the file. Also it seems if you want to use /mnt/user/ where you just move the files to the folder name and add a "1" or something extra to the folder name so there's no conflict, then at the end of the script move it will rename the folder back to the original folder name. below example where it will move the folder music to music1 to the 4 disks in a round robin when its done rename folder "music1" to "music". import os import shutil # Define the paths of the disks disks = ['/mnt/disk1/Backup/Music1', '/mnt/disk2/Backup/Music1', '/mnt/disk3/Backup/Music1', '/mnt/disk4/Backup/Music1'] # Define the path of the source directory src_dir = '/mnt/user/Backup/Music' # Define a variable to keep track of the current disk index current_disk_index = 0 # Recursively walk through the source directory and move files for dirpath, dirnames, filenames in os.walk(src_dir): for filename in filenames: src_path = os.path.join(dirpath, filename) # Get the next disk path in a round-robin fashion current_disk = disks[current_disk_index] current_disk_index = (current_disk_index + 1) % len(disks) dst_path = os.path.join(current_disk, os.path.relpath(src_path, start=src_dir)) # Create the destination directory if it doesn't exist if not os.path.exists(os.path.dirname(dst_path)): os.makedirs(os.path.dirname(dst_path)) # Move the file to the destination directory shutil.move(src_path, dst_path) print("Moved {} to {}".format(src_path, dst_path)) # Rename the Music1 folder to Music old_dir = '/mnt/user/Backup/Music1' new_dir = '/mnt/user/Backup/Music' os.rename(old_dir, new_dir) print("Finished") Edited March 19, 2023 by Hellomynameisleo Quote Link to comment
daTroll Posted March 26, 2023 Share Posted March 26, 2023 Tiny cosmetic request: the next time you update this plugin, can you please remove the "v" prefix in the version designation? Most of my other plugins use the same YYYY.MM.DD format but without any "v" prefix. Occasionally I like to sort the list of plugins in the WebUI based on version, to quickly identify the oldest/newest. Cheers, T Quote Link to comment
quimnut Posted April 18, 2023 Share Posted April 18, 2023 I see unbalance writing logs to my usb stick, it is the only software that writes logs there and I feel it's not appropriate. Additionally if I was to enable cloud backups of my boot stick it would leak what secret files I have moved. Is it possible to stop logging. Perhaps even prefix it into console logs/dmesg? Thanks. Quote Link to comment
cobalt027 Posted April 22, 2023 Share Posted April 22, 2023 I just upgraded from 6.9.2 (unbalance working) to 6.11.5 and now unbalance wont show the UI. Stop, start, change port, uninstall, reinstall. It just doesn't show. Any ideas where i can look to diagnose the problem and fix it? Thank you for this plugin. I use it all the time. Quote Link to comment
cobalt027 Posted April 22, 2023 Share Posted April 22, 2023 9 minutes ago, cobalt027 said: I just upgraded from 6.9.2 (unbalance working) to 6.11.5 and now unbalance wont show the UI. Stop, start, change port, uninstall, reinstall. It just doesn't show. Any ideas where i can look to diagnose the problem and fix it? Thank you for this plugin. I use it all the time. Nevermind. It was chrome forcing it to https. once i changed it manually to http it worked. Also works completely normal in firefox. On another note, I have 2 cache pools setup. Unblanace only see one of them. The one it sees is a normal 1 SSD pool. The one it doesn't see is 2 SSD drives in a RAID1. Does that make a difference? Quote Link to comment
bobC Posted May 2, 2023 Share Posted May 2, 2023 On 2/9/2023 at 12:48 PM, Cull2ArcaHeresy said: no but there was talk about multiple pools previously (1 or 2 years ago i think), but boiled down to it being hard to identify pools that aren't "cache" Yes, it does not seem to recognize that there is a zfs pool to move/copy to/from , it would be wonderful to see an update on this fine app, Quote Link to comment
redwolf Posted May 4, 2023 Share Posted May 4, 2023 hello, i am getting this message when i press the plan button on scatter, does it matter? Quote Link to comment
kiki Posted May 5, 2023 Share Posted May 5, 2023 (edited) 13 hours ago, redwolf said: hello, i am getting this message when i press the plan button on scatter, does it matter? I install today the plugin just to test it for a future migration, same thing, i install the Fix common Problems plugin and launch the Docker Safe new permissions (in Tools tab of the Unraid WebUi) but the message stay. The worst part is if a launch a migration, the mover doesn't.. move anything. but it tell me i succeed to move 50GB in less than 1 second.. 😄 Edited May 5, 2023 by kiki orthograph edit Quote Link to comment
veri745 Posted May 10, 2023 Share Posted May 10, 2023 (edited) On 4/22/2023 at 9:57 AM, cobalt027 said: On another note, I have 2 cache pools setup. Unblanace only see one of them. The one it sees is a normal 1 SSD pool. The one it doesn't see is 2 SSD drives in a RAID1. Does that make a difference? I just came here to report this same issue. I installed a second ssd and set it up as a second cache pool (btrfs), but it is not visible from Unbalance Edited May 10, 2023 by veri745 Quote Link to comment
TheGleaner Posted May 23, 2023 Share Posted May 23, 2023 I am having the issue of when trying to transfer from a disk to the cache, some files will not transfer as it says there is not room in the cache, but the cache does have room and the files are small, small enough they could all fit on the cache. version v2021.04.21 (5.6.4) Unraid 6.11.5 Quote Link to comment
itimpi Posted May 23, 2023 Share Posted May 23, 2023 4 hours ago, TheGleaner said: I am having the issue of when trying to transfer from a disk to the cache, some files will not transfer as it says there is not room in the cache, but the cache does have room and the files are small, small enough they could all fit on the cache. version v2021.04.21 (5.6.4) Unraid 6.11.5 Do you have a Minimum Free Space setting on the cache pool or on the User Share involved that my be kicking in and stopping the transfer? You are likely to get better informed feedback if you attach your system’s diagnostics zip file to your next post in this thread. Quote Link to comment
Vetteman Posted June 14, 2023 Share Posted June 14, 2023 When I run UNBALANCE and attempt to move a folder I get a warning about user nobody not having enough rights. When I go into the settings for UNBLANCE I on see Enable server and port. I do not see the user setting that is mentioned earlier in the thread. I am assuming I need to change the user to root or an equivalent admin. How do I run UNBALANCE as root? Thanks Quote Link to comment
TheGleaner Posted June 15, 2023 Share Posted June 15, 2023 On 5/23/2023 at 5:36 AM, itimpi said: Do you have a Minimum Free Space setting on the cache pool or on the User Share involved that my be kicking in and stopping the transfer? You are likely to get better informed feedback if you attach your system’s diagnostics zip file to your next post in this thread. No. I was meaning I was trying to transfer MBs worth of files to a 2 TB SSD cache drive, but I have not had the issue since I posted. Go figure. Quote Link to comment
dopeytree Posted June 15, 2023 Share Posted June 15, 2023 Is this all good to use with 6.12? the last update was v2021-04-21 1 Quote Link to comment
isvein Posted June 21, 2023 Share Posted June 21, 2023 On 6/15/2023 at 4:57 PM, dopeytree said: Is this all good to use with 6.12? the last update was v2021-04-21 I have tested it now, it works just as usual, but if you try to use it to move data to an ZFS formatted disk, each folder does not become its own dataset as it does if you create a new share on an zfs drive. All folders and files just gets moved/copied to the pool root 1 Quote Link to comment
jbrodriguez Posted June 21, 2023 Author Share Posted June 21, 2023 1 hour ago, isvein said: each folder does not become its own dataset as it does if you create a new share on an zfs drive creating a dataset on a zfs pool is a design decision from the zfs admin (user :)), unbalance will not see datasets only shares, so it's up to the user to create shares that unbalance can use (not sure if i'm missing something here) 2 Quote Link to comment
isvein Posted June 21, 2023 Share Posted June 21, 2023 2 hours ago, jbrodriguez said: creating a dataset on a zfs pool is a design decision from the zfs admin (user :)), unbalance will not see datasets only shares, so it's up to the user to create shares that unbalance can use (not sure if i'm missing something here) That makes sense, but can unbalanse put data into shares on another disk? I thought it just copied or moved folders from root of one disk to root of another, since this works just fine on say xfs disks. Quote Link to comment
madejackson Posted June 26, 2023 Share Posted June 26, 2023 So as with isvein discussed on reddit (here and here), unBALANCE does not create the nececcary datasets in advance of moving the files over (unlike core unRAID functions like mover that do work as they should). But I also experienced another issue with zfs and unBALANCE: unBALANCE is unable to read the size of a zfs-formated disk correctly. As soon as there is some data filled onto the zfs formated disk, it just shows the remaining free space as "size": Quote Link to comment
jbrodriguez Posted June 26, 2023 Author Share Posted June 26, 2023 3 hours ago, madejackson said: ... unBALANCE is unable to read the size of a zfs-formated disk correctly. As soon as there is some data filled onto the zfs formated disk, it just shows the remaining free space as "size": what is actual free space on these disks (i understand you're referring to disk 1 and 2 ) ? Quote Link to comment
madejackson Posted June 27, 2023 Share Posted June 27, 2023 (edited) 19 hours ago, jbrodriguez said: what is actual free space on these disks (i understand you're referring to disk 1 and 2 ) ? Free Space calculation is fine as far as i can see. It's just the total size that's wrong. Updated screenshots, unRAID and ZFS-Master as comparison: Edited June 27, 2023 by madejackson Quote Link to comment
isvein Posted June 28, 2023 Share Posted June 28, 2023 (edited) Im getting this message now when I try to move data from my last xfs drive (the last one does not have much free space) to zfs drives. Tried different things but same happens Disk9 is xfs and disk8 is zfs Edit: Hmmm looks like this may be because of something else, but not sure why Edited June 29, 2023 by isvein Quote Link to comment
Can0n Posted July 10, 2023 Share Posted July 10, 2023 (edited) Hello @jbrodriguez love the unbalance plugin however I am seeing one issue one of my cache pools is now showing up I have 2 party, 22 array disks, 1 ZFS pool with two 1TB SSD's and the pool not showing is a second ZFS pool with two 4TB spinners in ZFS it is also reading the size of the ZFS SSD's incorrectly as 690GB in size the pool is called download-cache so not sure if there is a character limit for it to display or what the issues is and wondering if you could take a look Edited July 10, 2023 by Can0n Quote Link to comment
madejackson Posted July 11, 2023 Share Posted July 11, 2023 (edited) Edit: solved. I was able to find the culprit: It's the hard-drives from WD (WD50NPZZ). I can write to my Toshibas and Seagate ST5000LM000 with about 35Mbit/s, but the WD (WD50NPZZ) is stuck at <5MB/s. I'm on 6.12.2 now and suddenly have some very slow speeds (<4MB/s), when trying to copy from btrfs pool to my newly zfs formatted array disk. I am 100% sure that nothing else is accessing my disks (fresh reboot, docker+VM stopped). Any Idea what this could be? In the last couple of transfers on 6.12.1 from xfs disk to zfs disk I usually got 20-30MB/s. Speed Fluctuate a bit. Sometimes it goes up to 25MB/s for a short time only for then to fall back down to 3MB/s, grinding to a complete halt and then speed up again. The Files are regular sized MP4's and maybe some .srt subtitels. There are no errors in the logs. Diskspeed lists the speed of this particular WD disk at about 90-150MB/s unbalance.log Edited July 11, 2023 by madejackson Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.