Mover Not Working

[TechTechTech]

My mover has stopped working. I have attached my diagnostics into this thread. I have uninstalled Mover Tuning.

 

When I try to move files I'm met with the following: "emhttpd: shcmd (9106): /usr/local/sbin/mover &> /dev/null &" -- there is no movement after this line on the system log.

 

Sometimes when I stop the mover in the command line I'm getting the following message:

• /usr/local/sbin/mover: line 102: kill: (18148) - No such process

 

I'm in the process of replacing my cache pool as well as my SAS cable as I was getting the following errors/warnings:

• blk_update_request: I/O error
• BTRFS info (device dm-19): read error corrected
• No reference found at driver, assuming scmd(0x00000000de36383e) might have completed
• sd 11:0:5:0: device reset: FAILED scmd(0x000000009844f7c0)
• blk_update_request: critical target error, dev sdt, sector 1953269757 op 0x3:(DISCARD) flags 0x800 phys_seg 1 prio class 0
• 2022/05/28 06:13:13 [error] 10318#10318: *1973802 upstream timed out (110: Connection timed out) while reading response header from upstream

 

Thoughts on what might be causing this issue?

 

Any help is much appreciated!

 

diagnostics-20220528-0624.zip

Edited May 28, 2022 by TechTechTech
Uploaded file

[JorgeB]

First would recommend connecting the SSDs to the Intel SATA ports, if that's not possible at least update LSI firmware to latest to see if all the errors go away, then enable the mover logging, run it and post new diags.

[TechTechTech]

Thanks for the quick response Jorge! I am unable to connect the SSDs to the SATA ports so I will work on updating the LSI firmware. Does the firmware look old to you? Dumb question, but what makes you think it has to do with the HBA?

 

Second question is this the firmware I'm looking for: https://docs.broadcom.com/docs/9305_16e_Pkg_P16.12_IT_FW_BIOS_for_MSDOS_Windows.zip

 

Or would it be this one: https://docs.broadcom.com/docs/9305_16i_Pkg_P16.12_IT_FW_BIOS_for_MSDOS_Windows.zip

 

My HBA is 9305-16i.

 

Thanks!

Edited May 28, 2022 by TechTechTech
Updated to add the 9305_16I link

[JorgeB]

13 hours ago, TechTechTech said:

Does the firmware look old to you?

Yes, FWVersion(09.00.100.00)

 

They are probably the same, but get the second one, 1st one is for the 9305-16e.

 

 

[TechTechTech]

Another dumb question for you, how do I make the temporary folder in my ram? I tried using the command line, but I'm getting the following when trying to follow your other thread:

• mkdir: cannot create directory ‘/lsi’: File exists

 

I also tried making a lsi folder in tmp using krusader, but when I try chmod +x sas3flash step I'm getting:

• chmod: cannot access 'sas3flash': No such file or directory

[JorgeB]

3 hours ago, TechTechTech said:

mkdir: cannot create directory ‘/lsi’: File exists

This means the folder already exists, you can reboot, remove that one or create a different named folder.

[TechTechTech]

Thank you for the guide on how to flash LSI firmware! I was able to successfully flash the HBA and I am not seeing the same errors I was seeing before. I am currently running mover, and I have attached my diagnostics.

 

The mover is still running but turning on mover logging fills up my log (which caused me to need to reboot to clear out the log). I'm getting the following messages:

• Jun  6 15:49:06 move: move_object: /mnt/cache/appdata/plex/Library/Application Support/Plex Media Server/Media/localhost/e/880e427ff2c9109a3f0b808fff11d7ad55855bc.bundle/Contents/Thumbnails/thumb3.jpg File exists
• Jun  6 15:49:06 move: file: /mnt/cache/appdata/plex/Library/Application Support/Plex Media Server/Media/localhost/e/880e427ff2c9109a3f0b808fff11d7ad55855bc.bundle/Contents/GoP-0.xml
• Jun  6 15:49:06 move: move_object: /mnt/cache/appdata/plex/Library/Application Support/Plex Media Server/Media/localhost/e/880e427ff2c9109a3f0b808fff11d7ad55855bc.bundle/Contents/GoP-0.xml File exists
• Jun  6 15:49:06 move: file: /mnt/cache/appdata/plex/Library/Application Support/Plex Media Server/Media/localhost/e/00c7849238a7a8437764ddf3cdf38a1bc7ba383.bundle/Contents/Thumbnails/thumb1.jpg
   

So far nothing has moved from the cache to the array.

 

diagnostics-20220606-1203.zip

Edited June 6, 2022 by TechTechTech

[ChatNoir]

6 hours ago, TechTechTech said:

• Jun  6 15:49:06 move: file: /mnt/cache/appdata/plex/Library/Application Support/Plex Media Server/Media/localhost/e/880e427ff2c9109a3f0b808fff11d7ad55855bc.bundle/Contents/GoP-0.xml
• Jun  6 15:49:06 move: move_object: /mnt/cache/appdata/plex/Library/Application Support/Plex Media Server/Media/localhost/e/880e427ff2c9109a3f0b808fff11d7ad55855bc.bundle/Contents/GoP-0.xml File exists

This means that the file already exist on the destination. The mover will never overwrite an existing file.

 

You will have to determine what is the more recent file and delete the other one.

[TechTechTech]

Flashing the LSI with updated firmware and bios + increasing the size of my log allowed the mover to start working again!

 

JorgeB and ChatNoir thank you both for your help it is so appreciated! People like you are what makes Unraid truly great.

[trurl]

Your appdata, domains, system shares are on the array. Best if these are on fast pool (cache) and set to stay there for better docker/VM performance and so array disks can spindown since these files are always open. After you get other shares moved from cache to make room, setting these to cache-prefer will let mover try to move them to cache, but nothing can move open files so you will have to disable Docker and VM Manager in Settings to get these moved. 

[TechTechTech]

Thanks for the response trurl! I ended up changing all shares outside of media and downloads to be cache only so that they'll only live on the cache. They were previously set that way but I was trying to make different changes (to no avail) to the cache settings to see if I could get the mover working.

[trurl]

I see your dockers are configured to use a share named docker on cache, but that share also has files on the array since you have configured it to be moved to the array.

[trurl]

1 minute ago, TechTechTech said:

changing all shares outside of media and downloads to be cache only

Mover ignores cache-only shares so to get these moved to cache you have to make them cache-prefer,

8 minutes ago, trurl said:

but nothing can move open files so you will have to disable Docker and VM Manager in Settings to get these moved. 

 

[TechTechTech]

Ah okay! Good catch! Thank you for the great and simple explanation and taking time to read/respond to the thread. I'll follow the steps that you listed!