-
Posts
616 -
Joined
-
Last visited
Content Type
Profiles
Forums
Downloads
Store
Gallery
Bug Reports
Documentation
Landing
Posts posted by Can0n
-
-
On 7/6/2023 at 6:41 PM, Veah said:
Try using Unbalance plugin. It will kind of let you see what is moving.
unfortunately unbalance doesnt support datasets just shares so that could explain why i could not see the temp location i had moved my appdata to
-
Hello @jbrodriguez
love the unbalance plugin however I am seeing one issue
one of my cache pools is now showing up
I have 2 party, 22 array disks, 1 ZFS pool with two 1TB SSD's and the pool not showing is a second ZFS pool with two 4TB spinners in ZFS it is also reading the size of the ZFS SSD's incorrectly as 690GB in size
the pool is called download-cache so not sure if there is a character limit for it to display or what the issues is and wondering if you could take a look
-
On 4/28/2020 at 2:41 PM, golli53 said:
Ok, finally solved it. In case anyone runs into this, `umount -l /dev/loop2` worked
thank you for this
I was pulling a some drives from a zpool to redo them from a 4 drive config to a 2 drive config and started and stopped too soon and docker.img held up the stop process as soon as i saw your command ran it and everything unmounted clean
Still relevant 3 years later !thank you!
-
4 hours ago, Veah said:
Try using Unbalance plugin. It will kind of let you see what is moving.
So got Plex in appdata working now used mover and rsync -avh and then a quick move from /mnt/cache/apps2/PlexMediaServer to /mnt/cache/appdata/PlexMediaServer
after 15 hours of mover started using rsync for all the tiny meta data files and found a vast majority of the 214GB was old PhotoCache i removed all that and it made Plex about 64GB -
23 hours ago, Can0n said:
i used rsync becasue mover wont move files off of zfs
the rsync moved all files from apps2 on the download-cache to appdata on the cache it retained the timestamps, permissions and users/groups from the download cache
my plex config folder is approx 214GB so rather large no way to move it from one cache to other and it took over 4 hours to for mover to move from cache to array and array to download-cache
once i tried to move it back to array it completely failed mover just stops for no reason so was stuck using rsync which actually took about 45min to copy all the containers from appdata that i put to "apps"
I verified database files are there and all the files that should be there are...
when i load from a URL i configured to get me there using internet DNS i get the interface and all my libraries have a triangle
when i map the plex contain back to the apps folder on download-cache it loads my server and libraries just fineWell it was the most painful move ever 15 hours of mover trying to move 214GB plex folder. I ultimately using rsync with -avh to move the folders with smaller and more files and let mover handle the database files
took nearly 24 hours but I got Plex up and running on a ZFS pool now with no watched status lost for myself ore my users! I took a snapshot right away
-
25 minutes ago, Veah said:
Try using Unbalance plugin. It will kind of let you see what is moving.
I do have that but it doesnt see the download-cache only the cache and my disks...odd
-
2 hours ago, Vr2Io said:
Opps 🤪
mover seems to have gotten stuck it says its still moving but no dat is being moved in meantime only things not moved for Plex is the Cache/Phototranscoder and gigs and gigs in it I might try moving the contents outside of plex and that way mover running plex back to cache pool might be quite a bit faster
-
25 minutes ago, Vr2Io said:
I have 64G, but mainly because just several docker and not use more then 6GB. Remain memory use for CCTV recording.
my Plex Folder alone is 214GB
i have 31 containers installed and run about 20 of the full time -
1 minute ago, Vr2Io said:
I always use rsync -ah ( same as -avh ) to sync appdata between disk and /tmp ( ram ) because I run docker in ram disk, it never have problem, but I will stop all docker & docker service before backup.
yeah the -v is just verbose so you can see what it is spitting out.
wish I had enough ram for that I have 64GB max for my i7 10700k
i am running a plex transcoder scratch disk
write reconstruction seems to be helping-
1
-
-
39 minutes ago, Vr2Io said:
md_write_method="auto"
Try enable reconstruction write first.
Even that, it seems a bit slow, less then 30MB/s, but this also depends on large / small file size.
yeah 400GB 4 hours isnt bad Plex has millions of little files but moving from ZFS to XFS is dog slow since its just plex now and mover
i moved all the other docker appdata using rsync -avh /mnt/download-cache/apps /mnt/cache/appdata and it was from ZFS pool to ZFS pool at about 45min
plex went as well but when i started the container up it was trying to set me up as a new server. when i mapped the contain back to /mnt/download-cache/apps/PlexMediaServer
it fired right up so rsync is breaking something in plex
i enabled reconstruction and monitoring thanks for the tip I thought i had that on a long time ago
-
its beautiful I would love to add that now! i have been trying to move a 214GB plex appdata folder from my ZFS 4 drive zpool to the array trying to get it on my newly formatted SSD in ZFS) for over 13 hours its moving sooo slowly
i need to see the status I have dozens of messages coming in why my plex is down...it litterally not showing in the logs that its moving
I see trickles of data on Disk1 and the ZFS cache-pool here and there but it is taking forever....the initial appdata move of over 500GB from BTRF to array took 4 hours -
So here is the story.
My current server (6.12.2) has two 10TB parity disks and 22x6,8 and 10TB XFS disks for the array
I used two 1TB SSD in BTRFS RAID1 and two 4TB Spinners as a secondary pool called download-cache using BTRFS
I also have 2 USB3 4TB drives attached with unassigned devices
I wanted to convert both pools to ZFS so here is what i did.
set mover for the download-cache to move all the data to the array stopped all applications using the download-cache so no files were locked. ran mover when it was done and set the share (Media and a couple other shares) to array and primary no no secondary storage.
Once the download-cache was empty i let the apps use it again as the data would only go to the array.
I stopped the array and removed the 2x4TB disks and started the array without them then stopped it again and assigned them setting the file system to ZFS then formatted them adding the 2x USB3 4TB drives as well for a raidz 1 group of 4 disks.
once done my next step was appdata, domains and system shares (current on BTRFS)
i shutdown vm manager and docker and set the shares to move to the array. once done I did same process to remove cache, re-add and format as ZFS Mirror 1 group of 2 devices.
i decided to rsync it back to the cache from array as Mover took over 4 hours rsync took about 1.5 hours. most of my dockers failed to work correctly and i saw perms completely borked.
i ran mover to move them to the download-cache to test them and they all worked albeit slower being on spinner drives.
now i used a different rsync option to keep time, owner/groups and permissions rsync -avz /mnt/download-cache/apps /mnt/cache/appdatathat seemed to copy ok and all my dockers except Plex worked. Plex is acting like its a new server despite database and all files seeming to be there. I updated the config to point back to download-cache where the folder apps still had all the data
Plex is working albeit slower
so i decided to run mover on it last night the only thing moving is the Plex folder and its been 12 hours and still moving
Attached diagnostics but would like to know why BTRFS over 400GB took 4 hours to move and ZFS to XFS 214GB is taking more than 12 hours
some takeaways:
Would be nice to have a progress bar on Mover to see its status and potentially pause it if needing some bandwidth on the drives.
if a progress bar is not easily able to be added perhaps an estimated time to completion based on its current file list -
7 hours ago, itimpi said:
I would suggest searching the boot drive as indicated using 'Freya_Backup' as the search string as that looks unlikely to give many false positives.
found it under /boot/config/plugins/dynamix/thor2freya.cron
renamed it thor2freya.cron.oldI have other crons I created there I am going to remove
I created that almost 5 years ago when i had 3 unraid servers. the Freya ultimately died and i recycled it
the 3rd one has become a windows Server 2022 Printer Server as the case drive bays was killing drives at an alarming rate. now its just a single 256GB SSD laying inside the case -
12 hours ago, Delarius said:
It's definitely somewhere on that /boot drive, just a matter of figuring out how to search for it. Not sure using the old servername is the right approach. The cronjob will no doubt look something like this:
5 4 * * 0 /root/backup.sh
So in this instance I'd try using "backup.sh" as my keyword.
found it!! thank you for your help
root@Thor:~# nano /etc/cron.d/root
grep -r "Backup Share to Freya" /boot/
/boot/config/plugins/dynamix/thor2freya.cron:#*** rsync Thor's Backup Share to Freya -
7 hours ago, mgutt said:
Then Plex has created appdata files of the new Installation. Check with the File Manager Plugin or similar where the files have been created.
And I'm not sure why you used rsync. If you are changing the cache pool of appdata, you simply need to select the new pool in the appdata share settings and then start the mover again.
so looking up mover, now running it and its been over 7 hours trying to move to the array
appears mover is just using rsync anyways so wondering what the full rsync string is in mover so I can try and do a copy not move using the same options as mover so I can keep a working copy of plex on the array should the move to the cache fail again for whatever reason
is it by chance this script here?
https://gist.github.com/fabioyamate/4087999 -
10 minutes ago, mgutt said:
Then Plex has created appdata files of the new Installation. Check with the File Manager Plugin or similar where the files have been created.
And I'm not sure why you used rsync. If you are changing the cache pool of appdata, you simply need to select the new pool in the appdata share settings and then start the mover again.
i used rsync becasue mover wont move files off of zfs
the rsync moved all files from apps2 on the download-cache to appdata on the cache it retained the timestamps, permissions and users/groups from the download cache
my plex config folder is approx 214GB so rather large no way to move it from one cache to other and it took over 4 hours to for mover to move from cache to array and array to download-cache
once i tried to move it back to array it completely failed mover just stops for no reason so was stuck using rsync which actually took about 45min to copy all the containers from appdata that i put to "apps"
I verified database files are there and all the files that should be there are...
when i load from a URL i configured to get me there using internet DNS i get the interface and all my libraries have a triangle
when i map the plex contain back to the apps folder on download-cache it loads my server and libraries just fine -
4 hours ago, Delarius said:
It's definitely somewhere on that /boot drive, just a matter of figuring out how to search for it. Not sure using the old servername is the right approach. The cronjob will no doubt look something like this:
5 4 * * 0 /root/backup.sh
So in this instance I'd try using "backup.sh" as my keyword.
So just noticed it showed up again with no stop n start of the array
here is a screnshot of the the /etc/cron.d/root file
and maybe i using grep wrong but no results using the cron schedule with rsync command and its options (Attached too)
-
So a few days ago I decided I wanted to migrate my old dual 1TB Cache SSD’s from BTRFS to ZFS (unraid 6.12.2) I had already set a 4 disk zool pool i call download-cache
I stopped docker and VM manager and set the share to migrate to the array (which is on xfs) when mover is invoked. I used mover to move the appdata folder to the array then used rsync -avz /mnt/user/appdata /mnt/download-cache/apps/i updated the mapping to the for Plex to /mnt/user/apps/PlexMediaServer thats on the download-cache and saw Plex worked.
I since shut it all down and used rsync again to move from the 4drive zpool (spinner disks) to the 2 Drive zPool (formerly a BTRFS pool) called cache
Now when i launch plex it is setting up like a new server with a random name… if i stop it and update the config path to the 4 drive pool it works again.
The claim code in the container config has not changed not sure where to look to make sure it launches the same name and instance of Plex on the new location with the 2 drivesreally struggling with trying to get this back on my SSD's if cant be the filesystem as the download-cache is zfs as is the cache
when it didnt work the first time i went back to BTRFS and it worked again on the cache
any help would be greatly appreciated. -
15 minutes ago, Delarius said:
It's definitely somewhere on that /boot drive, just a matter of figuring out how to search for it. Not sure using the old servername is the right approach. The cronjob will no doubt look something like this:
5 4 * * 0 /root/backup.sh
So in this instance I'd try using "backup.sh" as my keyword.
yeah i will make a better note when it comes back after a stop and start of the array
no luck finding it so far -
51 minutes ago, Delarius said:
It's coming from your /boot volume (the flash drive) - probably easiest to determine a keyword in your cronjob - something somewhat unique if possible. Then you can do something like this to figure out where that root cron is coming from:
$ grep -r "mykeyword" /boot/
That should give you a list of the possible matches and hopefully it's easy to determine where the root cron is coming from.
sadly putting the old server name in the keywords only pulls old syslog files i might have to stop and start the array again to get it to show up in /etc/cron.d/root againi have a sneaking suspecision its an old user.script plugin file lingering and staying active but could be something else
-
i checked my go...issue with mine is the old job keeps coming back....its not in user scripts and not in my go file. if i stop the array and start it its back
-
still getting this too using unRAID 6.12.2
just stopped VM service to move domains from a zfs pool/dataset to array (mover isnt working to move data from ZFS datasets to xfs array by the way) used rsync -avx to copy the domains
tried to restart libvirt and it fails with same umount error
long standing but buit only docker that was running was Plex and i tied the main pid to that but could stop it as it was in use -
On 6/7/2023 at 8:21 PM, CyberMew said:
No, it doesn't get saved for me. I have a similar issue and I have no idea where I added that cron job, and no matter how I modify the /etc/cron.d/root file, it always revert back after I restart the server or start the array. I took some instructions on this forum very long ago as there was no such thing as user scripts back then and have no idea how to reset/remove it now
it saves for me so not sure what voodoo magic my server is doing but i got some old Cron jobs there before I was using before User scripts came out and i even commented it out due to it not longer being valid and ocassionaly showing the invalid path in my logs when it ran. but after a massive issue last night with BTRFS errors flooding my logs and causing my VM and dockers to all crash I had to reboot forcibly due to stopping array failing and the comment # is still there so not sure what is happening to keep mine there
User Scripts even puts its schedules in there -
I even had a very old from to restart a docker container ever second day at 6am
Mover moves data from ZFS pool to XFS array extremely slowly
in General Support
Posted
Mover still incredibly slow moving any thing even large linux isos to ZFS from XFS or XFS to ZFS ZFS is using datasets so i think its hanging on the delete portion of the mover script as it not checking for datasets and doing a recursive zfs destroy command