mgutt Posted August 9, 2020 Share Posted August 9, 2020 (edited) I'm moving some movie files from disk2 to disk3 with the following command: mv -v "/mnt/disk2/Movie/IJ/"* /mnt/disk3/Movie/IJ At the same moment I'm watching a movie through Plex from disk2 and its stuttering. iotop returns the following: If I pause the movie the IO load of "mv" falls to around 60%: If I resume the movie multiple SHFS processes kick in and produce a massive load so everything raises up (up to 99.99%, see first screenshot): Why are there so many of them producing so much IO load? I mean its only a movie with a bitrate of around 40 Mbit/s (5 MB/s) and it should be an easy task for a 7200 rpm HDD to have one file transfer and one movie stream at the same time. Server specs Supermicro A2SDi-8C+-HLN4F Intel Atom C3758 (4675 cpubenchmark points) 32 GB RAM 10x Western Digital Ultrastar DC HC520 12TB XFS formatted Sorry I can not provide the full diagnostics as altough unraid states, that they are anonymized, they contain file names that are private. So please ask for specific log files and I will censor them manually, before posting them. Edited August 9, 2020 by mgutt Quote Link to comment
ogi Posted March 12, 2021 Share Posted March 12, 2021 Hi @mguttI'm curious if you got some resolution here... I've noticed my video streams have to pause/buffer while another file is transferring to the array. I'm hoping to come up with some settings to prioritize plex streams. Quote Link to comment
primeval_god Posted March 12, 2021 Share Posted March 12, 2021 @ogi Its not really an answer to why the above happens but you might want to look at using ionice on you cmd line copy operations https://linux.die.net/man/1/ionice Quote Link to comment
ogi Posted March 12, 2021 Share Posted March 12, 2021 thanks for the suggestion @primeval_godI wasn't even sure where to begin w/ looking for issues here. Quote Link to comment
mgutt Posted March 13, 2021 Author Share Posted March 13, 2021 On 3/12/2021 at 6:18 AM, ogi said: Hi @mguttI'm curious if you got some resolution here... I've noticed my video streams have to pause/buffer while another file is transferring to the array. I'm hoping to come up with some settings to prioritize plex streams. I retested this scenario as my server specs changed as follows: Gigabyte C246N-WU2 Xeon E-2146G 64 GB RAM 10x Western Digital Ultrastar DC HC520 12TB XFS formatted Now this happens only if I transcode and if I don't use a parity disk: This seems logical as a parity disk limits writes to a target disk to 80 MB/s which limits reads from source disk to 80 MB/s, too. But I don't know why it happens only while transcoding. Are you using a parity disk? Which CPU? Another solution to solve this, could be to bypass the Unraid SHFS overhead by changing Plex container paths from /mnt/user/movies to /mnt/diskX/movies. If your movies are on multiple disks, you need to add multiple paths. I do not use this solution as this would cause a full rescan in Plex if I move my movies to more or less disks. Quote Link to comment
ogi Posted March 13, 2021 Share Posted March 13, 2021 I have a 2x E5-2680 v2 CPUs; I use dual parity drives inside a Supermicro CSE-846 chassis. My situation I don't think is Plex specific; and unfortunately I couldn't replicate last night while watching a high bitrate video (video is 82615 kb/s), but I did just upgrade from unraid 6.8.3 to 6.9.1. When streaming very high bitrate video (on VLC for examples) periodically the video stream will outright stop for ~10 seconds... and then resume. I have assumed this was due to some other write operation to the array; but I cannot be certain. I should point out things did get better over NFS vs. SMB; but I suspect that's just because the faster file transfer allowed for faster buffering (getting NFS mounts on macOS took me a while to sort out, I should probably post a guide about that!) I'll try and monitor things in iotop and the "System Stats" page on the webui and see if I can get a better idea of what else is happening when this occurs. As your issue involved Plex transcoding, and mine seems to have to do with just general file read operations, I'll create a new post if I'm able to reproduce more consistently. Thanks for chiming in! Quote Link to comment
mgutt Posted March 13, 2021 Author Share Posted March 13, 2021 Are your disks encrypted? Please enable disk shares and start playing a movie directly from this disk share while running the disturbing write operation. Example: dd if=/dev/random of=/mnt/diskX/newshare/random.file bs=128k iflag=count_bytes count=128GB Quote Link to comment
ogi Posted March 13, 2021 Share Posted March 13, 2021 (edited) It's running now; I'm getting occasional hangups in the stream; here is a screenshot when during one of those freezes... for the most part its playing well. The `dd` command finished root@Tower:~# dd if=/dev/random of=/mnt/disk18/movies/random.file bs=128k iflag=count_bytes count=128GB 976562+1 records in 976562+1 records out 128000000000 bytes (128 GB, 119 GiB) copied, 2241.48 s, 57.1 MB/s root@Tower:~# Given the movie (mostly) played w/o interruption, not sure what to make of it. Edited March 13, 2021 by ogi moreinformation Quote Link to comment
mgutt Posted March 13, 2021 Author Share Posted March 13, 2021 Now repeat the test, but replace /mnt/disk18/movies against /mnt/user/movies in the dd command. Quote Link to comment
ogi Posted March 13, 2021 Share Posted March 13, 2021 (edited) Thanks for chiming in @mgutt It's been running for ~10 minutes w/o issue; have to go back to monitoring the kids.... will edit this post later after the dd command returns. EDIT: kids now want to watch a movie too; so this will be a real stress test haha. EDIT2: dd command finished root@Tower:~# dd if=/dev/random of=/mnt/user/movies/random.file bs=128k iflag=count_bytes count=128GB 976562+1 records in 976562+1 records out 128000000000 bytes (128 GB, 119 GiB) copied, 1954.55 s, 65.5 MB/s Everything seems to be working remarkably well ...two movies playing (one on Plex via transcoding, BTW thanks for your in-memory decoding guide)... I guess whatever I have had going on before, it's no longer an issue? ...maybe I should test some CPU intensive tasks too...but I can do that later. EDIT 3: Just wanted to say thanks again for your debugging steps. Edited March 14, 2021 by ogi Quote Link to comment
mgutt Posted March 14, 2021 Author Share Posted March 14, 2021 7 hours ago, ogi said: Everything seems to be working remarkably well Only to be sure: The random.file was written to the disk where the movie of the kids was located, too? Quote Link to comment
ogi Posted March 14, 2021 Share Posted March 14, 2021 3 minutes ago, mgutt said: Only to be sure: The random.file was written to the disk where the movie of the kids was located, too? Sorry for the confusion; random.file was written to the same disk that a very high bitrate movie was streaming from (disk 18). I'm not sure which disk random.file was written to when I aimed it at /mnt/user/movies/random.file (haven't looked for it yet). The kids movie, I'm fairly certain is on another disk in the array. All operations went smoothly so now I am having trouble even replicating the issue I began with (where the movie would freeze for 5-10 seconds, and there would be no data coming in via the network during that time). Quote Link to comment
mgutt Posted March 14, 2021 Author Share Posted March 14, 2021 1 hour ago, ogi said: I'm not sure which disk random.file was written to when I aimed it at /mnt/user/movies/random.file (haven't looked for it yet). I missed to say that. You need to check through the Unraid disk overview which disk is writing and start a movie from that disk. Quote Link to comment
ogi Posted March 14, 2021 Share Posted March 14, 2021 7 hours ago, mgutt said: I missed to say that. You need to check through the Unraid disk overview which disk is writing and start a movie from that disk. Ahh that makes more sense, I see what you're getting at. Will test this when I get some time later today. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.