Mounting USB Drive Creates large IOwait


Koshy

Recommended Posts

[Unraid 6.8.2]

 

When I mount an External USB HardDrive via unassigned Devices it creates large IOwait

 

Before Mounting 

748659954_2020-01-27(29).thumb.png.8d83cf9a548c5cb55c964472ad1b46f8.png

After Mounting 

1960083436_2020-01-27(35).thumb.png.56534c771aa03632f78bb7e173798aa3.png

 

I'm mounting this drive to transfer some data from the unraid array to the External Drive using binhex-Krusader. These screenshots are from before I've started any transfers. Once I start a transfer it will copy at a good speed for a while 

 

image.thumb.png.b95b0d611d5ce5db7f89fe8b7f2164ef.png

 

Then IOwait will go up and the transfer speed will crash 

 

image.thumb.png.f942aeaf4c415548b51611f0f9be0db0.png

 

image.thumb.png.454dcf6b5286105947b54ecb5aba1572.png

 

Even after I pause the Copy the IOwait times don't go down

 

image.thumb.png.b9240d1c1854d9a69d0f861b0e0ff00d.png

 

When I dismount the Drive the IOWait goes down to Zero

The backlog and drive utilization shoots up immediately when I remount the drive

 

 2144834114_2020-01-27(25).thumb.png.7f8d4e22a36be87d5bd22fa00a14c354.png

 

I have attached the diagnostics

 

Edit : 

 

iotop while drive mounted and with NO file Transfer(Not copying files with Krusader)

 

 image.thumb.png.cded7ef639b714ee6d5969989aecce83.png

 

Edit 2: 

lsof shows unrelated files being accessed. The file that is being copied is in the Yellow Box, All the files in the Red Box are untreated to the Copy

 

1084457139_2020-01-27(75).thumb.png.3a7d5e0f48db280f6a9b294056a20e6b.png

 

Edit 3:

 

This is when no transfer is taking place. These random files are being accessed from my mounted external USB Hard Drive. 

 

image.thumb.png.d4a85c6eefd72945c044594a8e8e56d6.png

 

Edit 4:

 

It was dynamic cache dirs scanning the drive that was the problem. Thank you robobub [https://forums.unraid.net/profile/100006-robobub/] for figuring it out. I excluded the External Drive from the Cache Dirs and the problem is now resolved.

 

Edit 5:

 

I think this was only a partial fix. I'm still getting High iowait and slow transfer speeds.[I'm getting faster speeds than before that is between 2 and 35 MegaBytes per second once it has slowed down earlier I was seeing speeds in the KiloBytes per second or even Bytes per second after it slows down and iowait is staying below 50% earlier I've seen it go to upto 65%]. I'm not getting hit with high iowait before transfers but after some minutes after I start a transfer the issue returns.

479753027_2020-01-28(10).thumb.png.7fa07c106f9e0ac0c62e1e998c2f0b60.png

 

lsof isn't showing other unrelated files being accessed though 

 

image.thumb.png.c16924fc4ea6f7b29cd906fcc8d17e7c.pngimageproxy.php?img=&key=e5eec7c5c933ca16

 

iotop

 

image.thumb.png.39e7f4d527f73554422b552260b9105a.png

 

I have also attached new Diagnostics

 

tower-diagnostics-20200127-1248.zip

tower-diagnostics-20200128-1247.zip

Edited by Koshy
More Info
Link to comment

One thing you can try is adjusting the I/O Scheduler and queue size on the USB drive (and perhaps even the array drives being written to)  I had an issue where large writes would lock up my system with high I/O wait because perhaps the drive controller didn't like it.  

 

# Enumerate the drives being written to and read from, or you could do all of them with sd*
# note, changing scheduler will change nr_requests

echo none > /sys/block/sd[c,d,e]/queue/scheduler
echo 4 > /sys/block/sd[c,d,e]/queue/nr_requests

 

Edited by robobub
Link to comment
2 minutes ago, robobub said:

One thing you can try is adjusting the I/O Scheduler and queue size on the USB drive (and perhaps even the array drives being written to)  I had an issue where large writes would lock up my system with high I/O wait.  

 


# Enumerate the drives being written to and read from, or you could do all of them with sd*
# note, changing scheduler will change nr_requests

echo none > /sys/block/sd[c,d,e]/queue/scheduler
echo 4 > /sys/block/sd[c,d,e]/queue/nr_requests

 

I'm copying data from the array to the USB drive not the other way round. I'm not writing anything to the array.

Link to comment
5 minutes ago, Koshy said:

I'm copying data from the array to the USB drive not the other way round. I'm not writing anything to the array.

Okay? Your USB drive will have a device ID, and you can change the I/O Scheduler and Queue size for that device.  I suggested changing both the source and destination drives.

 

Not that it's too relevant, but the post I linked was moving data from the array to the cache drive.  My suggestion applies with any data transfer.  It's a simple test that has a chance of helping.  

Edited by robobub
Link to comment
1 minute ago, robobub said:

Okay? Your USB drive will have a device ID, and you can change the I/O Scheduler and Queue size for that device.  I suggested changing both the source and destination drives.

And also iowait becomes large ~20% even before I start a transfer(just after I mount the drive). Will your fix still work?

 

Thank you.

Link to comment
2 minutes ago, Koshy said:

And also iowait becomes large ~20% even before I start a transfer(just after I mount the drive). Will your fix still work?

 

Thank you.

I don't have your exact situation, so I'm not guaranteeing it will fix it.  But it takes 1 second to try, and you can always change it back (you can just cat that same location to see what it was before changing). 

 

It is a bit odd that it goes up without starting any copying.  I would look to see if something is scanning it (e.g. iotop+lsof, or a fancier tool)

Edited by robobub
Link to comment
1 hour ago, robobub said:

I don't have your exact situation, so I'm not guaranteeing it will fix it.  But it takes 1 second to try, and you can always change it back (you can just cat that same location to see what it was before changing). 

 

It is a bit odd that it goes up without starting any copying.  I would look to see if something is scanning it (e.g. iotop+lsof, or a fancier tool)

Thank you.

 

Changing the I/O Scheduler and queue size unfortunately did not make a difference. 

 

update: iowait went down to 0, but I don't know why. Will Investigate further.

update 2:It went up again

Edited by Koshy
Link to comment

 

3 hours ago, robobub said:

I don't have your exact situation, so I'm not guaranteeing it will fix it.  But it takes 1 second to try, and you can always change it back (you can just cat that same location to see what it was before changing). 

 

It is a bit odd that it goes up without starting any copying.  I would look to see if something is scanning it (e.g. iotop+lsof, or a fancier tool)

image.thumb.png.67d9bdb049a9a88506347588ba01f6cb.pngimageproxy.php?img=&key=e5eec7c5c933ca16imageproxy.php?img=&key=e5eec7c5c933ca16

 

Any clues on what I should do?

 

Edit: 

 

lsof shows unrelated files being accessed. The file that is being copied is in the Yellow Box, All the files in the Red Box are untreated to the Copy 

 

543389025_2020-01-27(75).thumb.png.6e47fcb7ef301748415453e32a5b7a0f.png

 

Edited by Koshy
Link to comment

Something is scanning your /mnt/disks/Seagate_Backup_Plus_Drive with the command find with PID 25568 (the 2nd column in your screenshot, PID)

 

This doesn't show up on your diagnostics what process it is and what spawned it.  

 

You can try to trace what was running by running 

ps axjf | egrep 25568

The first 4 columns will list other processes associated with it, so then you add that to the egrep command.  If any of the columns are 0,1,2, I would exclude it as it is one of the root processes.

ps axjf | egrep '25568|<column 1>|<column2>|<column 3>'

The process number 25568 could change, so make sure to run lsof again to find out what command and PID is currently accessing it.  

 

 Also, the command I listed earlier to change the schedulers doesn't work.  Forgot that you need to use tee if outputting to multiple devices.  Though of course what I mentioned earlier about finding what is scanning your drive is more important.

# Change all of them excluding the flash drive with sd[b-z], or enumerate specific ones with sd[b,c,e]
# note, changing scheduler will change nr_requests

echo none | tee /sys/block/sd[b-z]/queue/scheduler
echo 4 | tee /sys/block/sd[b-z]/queue/nr_requests

 

Edited by robobub
Link to comment
11 hours ago, robobub said:

Something is scanning your /mnt/disks/Seagate_Backup_Plus_Drive with the command find with PID 25568 (the 2nd column in your screenshot, PID)

 

This doesn't show up on your diagnostics what process it is and what spawned it.  

 

You can try to trace what was running by running 


ps axjf | egrep 25568

The first 4 columns will list other processes associated with it, so then you add that to the egrep command.  If any of the columns are 0,1,2, I would exclude it as it is one of the root processes.


ps axjf | egrep '25568|<column 1>|<column2>|<column 3>'

The process number 25568 could change, so make sure to run lsof again to find out what command and PID is currently accessing it.  

 

 Also, the command I listed earlier to change the schedulers doesn't work.  Forgot that you need to use tee if outputting to multiple devices.  Though of course what I mentioned earlier about finding what is scanning your drive is more important.


# Change all of them excluding the flash drive with sd[b-z], or enumerate specific ones with sd[b,c,e]
# note, changing scheduler will change nr_requests

echo none | tee /sys/block/sd[b-z]/queue/scheduler
echo 4 | tee /sys/block/sd[b-z]/queue/nr_requests

 

Here is what I got.

 

512243974_2020-01-28(5).thumb.png.caf0eb866d68615306b52c14a975b87d.png

 

I have also attached new diagnostics.

 

Your help is much apricated. Thank You.

tower-diagnostics-20200128-1103.zip

Edited by Koshy
Link to comment
2 hours ago, robobub said:

You can see what is launching that find process by going up a few lines: dynamic cache dirs plugin.

Add your external hard drive to the excluded folders path of that folder caching plugin

 

I think this was only a partial fix. I'm still getting High iowait and slow transfer speeds[I'm getting faster speeds than before that is between 2 and 35 MegaBytes per second once it has slowed down earlier I was seeing speeds in the KiloBytes per second or even Bytes per second after it slows down and iowait is staying below 50% earlier I've seen it go to upto 65%]. I'm not getting hit with high iowait before transfers but after a couple minutes after I start a transfer the issue returns.

 

1366446214_2020-01-28(10).thumb.png.bfb7e3d527a1aafa5280b429653de0a6.png

lsof isn't showing other unrelated files being accessed though 

 

851741096_2020-01-28(11).thumb.png.725c9cfdcb6dab26341fa1c9fbd494c0.png

 

iotop

 

125551121_2020-01-28(12).thumb.png.a3d6b0f721690b635820839554fb9f49.png

 

2 hours ago, Koshy said:

 Also, the command I listed earlier to change the schedulers doesn't work.  Forgot that you need to use tee if outputting to multiple devices.  Though of course what I mentioned earlier about finding what is scanning your drive is more important.


# Change all of them excluding the flash drive with sd[b-z], or enumerate specific ones with sd[b,c,e]
# note, changing scheduler will change nr_requests

echo none | tee /sys/block/sd[b-z]/queue/scheduler
echo 4 | tee /sys/block/sd[b-z]/queue/nr_requests

I get this when I try that

 

1765344613_2020-01-28(14).thumb.png.6f73007075c61af5edf2debfec2fe268.png

Edited by Koshy
Link to comment
  • 10 months later...
3 hours ago, Nanobug said:

How do you do that?

The Dynamix Cache Directories plugin.  Settings->Folder Caching.

 

I think the better way to do it is to just set the folders you want to cache, rather than let it cache everything.  It's the 'Included Folders' setting.  As this user found out, Cache Directories will add UD disks unless you either set it to only include certain folders, or exclude the UD mounted disk specifically.

Link to comment
37 minutes ago, dlandon said:

The Dynamix Cache Directories plugin.  Settings->Folder Caching.

 

I think the better way to do it is to just set the folders you want to cache, rather than let it cache everything.  It's the 'Included Folders' setting.  As this user found out, Cache Directories will add UD disks unless you either set it to only include certain folders, or exclude the UD mounted disk specifically.

I've tried excluding the two folders it's being used for.

I'll get back here when I know if it helped or not.

Link to comment

Before I did the change, it was always high on CPU usage with IOwait.
Now, it's only when it does something with that disk the IOwait goes up.
So the overall performance is better, but when it needs to read or write to that disk, it goes up by a lot.

 

I guess I'll have to live with this, until I get a replacement disk.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.