Mover only moving some, not all, files

[VelcroBP]

I think my mover or cache config isn’t correct. I have some large downloads that I’m expecting Mover to transfer to the array. Only some of them moved, despite having manually invoked Mover several times.

 

I am expecting the files in the DownDownDown share to be moved, but some remain. I’ve turned on Mover logging and attached diagnostic.

 

 

mootower-diagnostics-20220220-2016.zip

Edited February 21, 2022 by VelcroBP
Wrong dx file attached

[VelcroBP]

Ok finally attached a proper dx zip with mover logs included.

[trurl]

If you aren't trying to help debug Mover Tuning plugin, remove it.

 

If you are trying to help debug Mover Tuning plugin, you should post on its support thread.

[Frank1940]

When this has happened to me, I generally find that the file(s) all ready exist on the share.  I have always assumed that mover will not overwrite an existing file.  As far as I know, most modern OS default to not overwriting an existing file without requiring confirmation from the user that that is the action that he really wants!

[VelcroBP]

46 minutes ago, Frank1940 said:

When this has happened to me, I generally find that the file(s) all ready exist on the share.  I have always assumed that mover will not overwrite an existing file. 

This seems to be the case. I've found that the folder exists on Array disc 8, but only the .nfo file is there. Only the .mkv file is in the folder on Cache. How do I correct this?

[Frank1940]

I looked through your syslog and found you have a tremendous number of errors of this type: 

 

Feb 20 18:11:16 mooTower kernel: blk_update_request: critical space allocation error, dev loop2, sector 8248824 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Feb 20 18:11:16 mooTower kernel: blk_update_request: critical space allocation error, dev loop2, sector 8248920 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Feb 20 18:11:16 mooTower kernel: blk_update_request: critical space allocation error, dev loop2, sector 6370328 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Feb 20 18:11:16 mooTower kernel: blk_update_request: critical space allocation error, dev loop2, sector 6130928 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Feb 20 18:11:16 mooTower kernel: blk_update_request: critical space allocation error, dev loop2, sector 8248824 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Feb 20 18:11:16 mooTower kernel: blk_update_request: critical space allocation error, dev loop2, sector 8248920 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Feb 20 18:11:16 mooTower kernel: blk_update_request: critical space allocation error, dev loop2, sector 6370328 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Feb 20 18:11:16 mooTower kernel: blk_update_request: critical space allocation error, dev loop2, sector 6130928 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Feb 20 18:11:16 mooTower kernel: blk_update_request: critical space allocation error, dev loop2, sector 6370328 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Feb 20 18:11:16 mooTower kernel: blk_update_request: critical space allocation error, dev loop2, sector 8248824 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Feb 20 18:11:23 mooTower kernel: print_req_error: 41 callbacks suppressed
Feb 20 18:11:23 mooTower kernel: blk_update_request: critical space allocation error, dev loop2, sector 8248824 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
Feb 20 18:11:23 mooTower kernel: blk_update_request: critical space allocation error, dev loop2, sector 8248920 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
Feb 20 18:11:23 mooTower kernel: blk_update_request: critical space allocation error, dev loop2, sector 6370328 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
Feb 20 18:11:23 mooTower kernel: blk_update_request: critical space allocation error, dev loop2, sector 6130928 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
Feb 20 18:11:23 mooTower kernel: BTRFS error (device loop2): parent transid verify failed on 262684672 wanted 395016 found 395007
### [PREVIOUS LINE REPEATED 9 TIMES] ###

Is your array nearly full or you are trying to force data into a place where there is no room for it?  (Incorrectly setup of split levels are often responsible for this problem.)

 

You appear to  have some issues during bootup with your hardware: 

 

Feb 20 13:26:59 mooTower kernel: ata1.00: configured for UDMA/133
Feb 20 13:26:59 mooTower kernel: sd 1:0:0:0: [sdb] tag#6 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s
Feb 20 13:26:59 mooTower kernel: sd 1:0:0:0: [sdb] tag#6 Sense Key : 0x5 [current]
Feb 20 13:26:59 mooTower kernel: sd 1:0:0:0: [sdb] tag#6 ASC=0x21 ASCQ=0x4
Feb 20 13:26:59 mooTower kernel: sd 1:0:0:0: [sdb] tag#6 CDB: opcode=0x88 88 00 00 00 00 00 00 00 00 28 00 00 00 10 00 00
Feb 20 13:26:59 mooTower kernel: blk_update_request: I/O error, dev sdb, sector 40 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 0
Feb 20 13:26:59 mooTower kernel: sd 1:0:0:0: [sdb] tag#7 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s
Feb 20 13:26:59 mooTower kernel: sd 1:0:0:0: [sdb] tag#7 Sense Key : 0x5 [current]
Feb 20 13:26:59 mooTower kernel: sd 1:0:0:0: [sdb] tag#7 ASC=0x21 ASCQ=0x4
Feb 20 13:26:59 mooTower kernel: sd 1:0:0:0: [sdb] tag#7 CDB: opcode=0x88 88 00 00 00 00 00 00 00 00 48 00 00 00 30 00 00
Feb 20 13:26:59 mooTower kernel: blk_update_request: I/O error, dev sdb, sector 72 op 0x0:(READ) flags 0x80700 phys_seg 6 prio class 0
Feb 20 13:26:59 mooTower kernel: sd 1:0:0:0: [sdb] tag#8 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s
Feb 20 13:26:59 mooTower kernel: sd 1:0:0:0: [sdb] tag#8 Sense Key : 0x5 [current]
Feb 20 13:26:59 mooTower kernel: sd 1:0:0:0: [sdb] tag#8 ASC=0x21 ASCQ=0x4
Feb 20 13:26:59 mooTower kernel: sd 1:0:0:0: [sdb] tag#8 CDB: opcode=0x88 88 00 00 00 00 00 00 00 00 88 00 00 00 78 00 00
Feb 20 13:26:59 mooTower kernel: blk_update_request: I/O error, dev sdb, sector 136 op 0x0:(READ) flags 0x80700 phys_seg 15 prio class 0
Feb 20 13:26:59 mooTower kernel: sd 1:0:0:0: [sdb] tag#9 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s
Feb 20 13:26:59 mooTower kernel: sd 1:0:0:0: [sdb] tag#9 Sense Key : 0x5 [current]
Feb 20 13:26:59 mooTower kernel: sd 1:0:0:0: [sdb] tag#9 ASC=0x21 ASCQ=0x4
Feb 20 13:26:59 mooTower kernel: sd 1:0:0:0: [sdb] tag#9 CDB: opcode=0x88 88 00 00 00 00 00 00 00 01 08 00 00 00 f8 00 00
Feb 20 13:26:59 mooTower kernel: blk_update_request: I/O error, dev sdb, sector 264 op 0x0:(READ) flags 0x80700 phys_seg 31 prio class 0
Feb 20 13:26:59 mooTower kernel: ata1: EH complete
Feb 20 13:26:59 mooTower kernel: ata1.00: exception Emask 0x10 SAct 0x20 SErr 0x280100 action 0x6 frozen
Feb 20 13:26:59 mooTower kernel: ata1.00: irq_stat 0x08000000, interface fatal error
Feb 20 13:26:59 mooTower kernel: ata1: SError: { UnrecovData 10B8B BadCRC }
Feb 20 13:26:59 mooTower kernel: ata1.00: failed command: READ FPDMA QUEUED
Feb 20 13:26:59 mooTower kernel: ata1.00: cmd 60/08:28:10:01:00/00:00:00:00:00/40 tag 5 ncq dma 4096 in
Feb 20 13:26:59 mooTower kernel:         res 40/00:28:10:01:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Feb 20 13:26:59 mooTower kernel: ata1.00: status: { DRDY }
Feb 20 13:26:59 mooTower kernel: ata1: hard resetting link
Feb 20 13:26:59 mooTower kernel: ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Feb 20 13:26:59 mooTower kernel: ata1.00: configured for UDMA/133

 

You have a equipment listing in your signature.  Is it  up to date?  (As I recall, SASLP-MV8 boards often cause problems with later releases of Unraid.)

[VelcroBP]

I still have the SASLP-MV8 board installed. 

As for the critical space issue: I have ~2TB left in the array. My cache was almost full yesterday after several large downloads and issued a warning in the UI. That is what prompted me to notice what was remaining on the cache after Mover ran.

[Frank1940]

Have you set the 'Minimum free space:' parameter (for the share in question) to be about twice the size of the largest file that you intend to commit to that share?

 

11 hours ago, VelcroBP said:

I've found that the folder exists on Array disc 8, but only the .nfo file is there.

 

Did you check every other disk to see if that folder exists on some other disk besides disk 8? ...Or did you stop looking when you found it on disk 8?  (By default, contents of folders can be span-- or split across --several different disks.  The 'Split level' parameter can prevent this but it is a two edge tool and you have to know exactly why you need to employ it or it can also cause problems in the future.)

[trurl]

2 hours ago, Frank1940 said:

tremendous number of errors of this type

/dev/loop2 is the docker.img vdisk. It wasn't currently full when the diagnostics were taken, but I noticed it was set somewhat larger, 30G, than default 20G, and using 18G of that. I have 15-20 containers and only use about half of 20G. Have you had problems filling it? Making it larger won't fix that, it will only make it take longer to fill. A common cause of filling docker.img is an application writing to a path that isn't mapped to host storage.

 

2 hours ago, Frank1940 said:

some issues during bootup with your hardware

Possibly just connection problems on disk3 and it only shows up at boot because it isn't accessed after. I see it has a number of CRC errors which would also suggest connection issues.

 

You should see SMART warnings for disk3 on the Dashboard page. Do any of your other disks have SMART warnings? Do you have Notifications setup?

 

11 hours ago, VelcroBP said:

I've found that the folder exists on Array disc 8, but only the .nfo file is there. Only the .mkv file is in the folder on Cache. How do I correct this?

But that doesn't mean mover can't move a file because the file already exists, which is what we were referring to.

 

Did you remove the Mover Tuning plugin?

[VelcroBP]

1 hour ago, Frank1940 said:

Have you set the 'Minimum free space:' parameter (for the share in question) to be about twice the size of the largest file that you intend to commit to that share?

I did not have that set. I just set it to 100GB.

 

1 hour ago, Frank1940 said:

Did you check every other disk to see if that folder exists on some other disk besides disk 8? ...Or did you stop looking when you found it on disk 8? 

I checked all disks. That folder exists on another Disk, but containing one of the other movie files. The .MKV that is on Cache is not on an Array disk

[VelcroBP]

47 minutes ago, trurl said:

A common cause of filling docker.img is an application writing to a path that isn't mapped to host storage.

Is there an easy way for me to tell what applications are writing to the docker.img instead of an Array path?

 

50 minutes ago, trurl said:

Do any of your other disks have SMART warnings? Do you have Notifications setup?

Only Disk 3 shows a SMART error on the Dashboard.

Notifications seem to be set to DEFAULT for all disks.

 

51 minutes ago, trurl said:

Did you remove the Mover Tuning plugin?

I did remove it.

[VelcroBP]

2 hours ago, trurl said:

Possibly just connection problems on disk3

Don't know if it's relevant at all, but I just noticed that Disk 3 is formatted XFS, and all my others are Resierfs. Just wanted to point that out. This drive is the latest one installed.

[itimpi]

Just now, VelcroBP said:

Don't know if it's relevant at all, but I just noticed that Disk 3 is formatted XFS, and all my others are Resierfs. Just wanted to point that out. This drive is the latest one installed.

The format is irrelevant other that reiserfs doesn’t not handle disks getting very full very well (as well as being deprecated for new drives).

[Frank1940]

I was playing around and have a suggestion.  Change the extension of the file on the cache.  (I would suggested doing it this way:  Filename.mkv   to   filename.mkv.NEW   )

 

Spin up all of the array disks-- you can do this from the Main tab on the GUI.  Then using the Windows Explorer search function to search the share in question, search for the string  file   and see what you get.   (Just make sure that the search string (file) is unique enough to limit the number of positive results.) 

 

PS---  The reason for adding the new extension is because Windows will only find the first occurrence of a duplicated file name.  It ignores any subsequent file with an identical name-- and the extension is a part of the name.  Oh, BTW, Windows considers File.txt and file.txt to be the same file (name) as it ignores capitalization in all file names. (Linux and Unix would consider these to be different files and that fact just adds to the confusion!) 

[VelcroBP]

3 hours ago, VelcroBP said:

I did not have that set. I just set it to 100GB.

Don't know if this change fixed it, but the file has successfully moved into the Array.

[VelcroBP]

7 hours ago, Frank1940 said:

As I recall, SASLP-MV8 boards often cause problems with later releases of Unraid.

I'm not sure how to troubleshoot this further. Specifically, if this is a bad HDD, bad controller, bad cable, or simply a bad connection. I will open it up and make sure the cables are all firmly seated, as I did relocate the server about 1.5 months ago.

 

If I determine this needs replaced, is the LSI SAS 9200-8i a solid choice?

[Frank1940]

One more possibility you might want to consider.  The file in question was still being held open by an other application and was locked from being modified--- Moving is a modification.  Linux (as are almost every modern OS's) is a multi user OS and has provision for file locking to prevent two applications from modifying the file at the same time. 

 

I would assume that the Mover utility would honor a file lock and not move a file in that condition.

[trurl]

5 hours ago, VelcroBP said:

Is there an easy way for me to tell what applications are writing to the docker.img instead of an Array path?

Container Size button on the Dockers page might tell something.

[VelcroBP]

4 hours ago, trurl said:

Container Size button on the Dockers page might tell something.

Here are the results. Are they way out of line with expected?

 

Edited February 22, 2022 by VelcroBP
removed bad attachment (xlsx)