I was looking at my array when I noticed one of my disks was 100% used, and when I realized it was one of the disks dedicated to TV shows I got concerned and started looking in the latest season folders and noticed that some of the shows are zero byte file sizes because the disk was full.

 

I use a cache disk, so was hopeful that the episodes would still be stored in cache as the copy would have failed, however, it is empty as well, so I am guessing that even though there was zero space, UnRAID still thinks that it successfully copied the files over.

 

Is it just me, or does this seems like it should be a bug? Ideally I would have hoped it would have kept the file on the cache drive as I don't believe the copy should have been successful. When I look at my syslog sure enough I see the following:

 

Mar 10 03:45:56 CydStorage logger: rsync error: error in file IO (code 11) at receiver.c(389) [receiver=3.1.0] (Errors)

Mar 10 03:45:56 CydStorage logger: ./TV/Justified/Season 6/Justified S06E07 - The Hunt.mkv

Mar 10 03:45:56 CydStorage logger: >f.stpog... TV/Justified/Season 6/Justified S06E07 - The Hunt.mkv

Mar 10 03:45:56 CydStorage shfs/user0: shfs_write: write: (28) No space left on device

Mar 10 03:45:56 CydStorage shfs/user0: shfs_write: write: (28) No space left on device

Mar 10 03:46:09 CydStorage logger: rsync: write failed on "/mnt/user0/TV/Justified/Season 6/Justified S06E07 - The Hunt.mkv": No space left on device (28) (Minor Issues)

 

So, a couple of points....

 

1) If it failed to copy, why does it get wiped off of the cache drive?

2) This seems like something that a notification should pop up for, but when I log into the GUI I only get green notification boxes telling me life is good.

 

This seems like a pretty big oversight, and am not sure if it's always been this way, or if something has changed, but thought it should be raised. I have to redo 8-10 days of TV shows to those folders, but it could have been much worse.

 

Encountered the exact same scenario of some disks associated with my TV user share getting full. 

 

I haven't yet encountered this on v6, but on v5 I would get a disk full error in the log, but the files would remain on the cache.  Luckily I caught it within a week, so the mover only attempted the move once.  How long do you think your disk has been full? 

 

Do you have other disks included in the same share?

 

One of the key benefits of UnRAID is the user shares the eliminate the artificial constraints of a single disk's size by automatically using space on other disks when needed.    Not taking advantage of this can lead to exactly what you've noted here.

 

I agree the system shouldn't delete things from the cache drive if they aren't successfully copied ... but you can clearly eliminate the likelihood of this by not restricting the share from using your available disks.    If you like to have things limited to specific disks, then at least keep one "overflow disk" that's always empty, but is included in ALL of your shares.    Then you simply need to look at it from time-to-time to see if something's being copied to it; and then you'd know you needed to add another disk for the involved shares.

 

I do have multiple disks in the share, however have the split set up so that any given show is constrained to a single disk. Even if I took this one level lower I still would have been screwed as it was part way through the season, so even if I split at a lower level I still would have had the same issue. I have several other disks with free space, but given the way the data is stored they are only helping now as I manually move data around.

 

I am reading you comments and assuming that split levels is the only way to manage this, which I don't see helping for the reasons above. Is there something above and beyond that that you are referring to?

 

As to kaiguy's question It looks like it's been roughly 8-10 days. I am not even sure if the log is showing me all the files/folders in question so I am going to have to go through each active show and investigate. Definitely a pain in the ass, but could be worse.

 

I definitely think the mover script should leave the files alone if they don't successfully copy. This would have at least saved me a bunch of headaches.

 

 

 

 

I definitely think the mover script should leave the files alone if they don't successfully copy. This would have at least saved me a bunch of headaches.

 

 

If mover attempted to move the files only to fail do to lack of space and the files then are no longer on the cache (like they should be on a failed transfer) then this is a bug.

... have the split set up so that any given show is constrained to a single disk ...

 

Do you mean show or episode?    If you're constraining a show that's really too restrictive.  An episode is fine -- it's in fact a good idea to have the full episode on a single disk ... same as for a movie ... so only one disk needs to spin up to watch that episode/movie.

 

You can completely eliminate the issue you had by a combination of split level and min free space ... as long as the min free space is larger than any episode could be, the copy will succeed for any episode;  and when the next episode needs to be written, it will go on a different disk if the min free is no longer available.

 

I agree, however, that it's a bug that the mover isn't leaving the unsuccessfully copied file on the cache.  I suspect this is related to the user share copy issue, since the result is the same (zero length files).

 

... have the split set up so that any given show is constrained to a single disk ...

 

Do you mean show or episode?    If you're constraining a show that's really too restrictive.  An episode is fine -- it's in fact a good idea to have the full episode on a single disk ... same as for a movie ... so only one disk needs to spin up to watch that episode/movie.

 

You can completely eliminate the issue you had by a combination of split level and min free space ... as long as the min free space is larger than any episode could be, the copy will succeed for any episode;  and when the next episode needs to be written, it will go on a different disk if the min free is no longer available.

 

I agree, however, that it's a bug that the mover isn't leaving the unsuccessfully copied file on the cache.  I suspect this is related to the user share copy issue, since the result is the same (zero length files).

 

Prior to this I had never thought constraining a show was too restrictive. It just seemed like good housekeeping. It just goes against my ordered mentality to let episodes go wherever and have a single show span a bunch of disks.

 

I guess I will have to pay better attention to how I have things structured (though I did a full clean-up 6 months ago).

 

Hopefully Tom & team can fix the mover issue though to minimize the impact if this happens again.

 

I wanted my shows to reside on the same drive as well, so watching one episode doesn't spin up multiple disks.

 

Hopefully Tom & team can fix the mover issue though to minimize the impact if this happens again.

 

If you want to help them along with this process, please file a bug report in the bug report form.

 

It seems strange that this would be a user share copy issue since mover uses user0 to avoid this, but that's what it seems like.

Okay.... I hate when I do this, but it appears I am an idiot. When filling in the defect bug report I started to explain the situation and realized that at least some of the data is actually on the cache drive.

 

I got mixed up as I had been looking in the complete folder from SAB/SB, and NOT in the actual TV folder in the cache drive. Looking through the folders there is at least a good chunk of the files there. I am going to need to dig into this further before I bother filing a bug as it may just be a PEBCAK error.

 

Those pesky PEBCAK errors can get you every time 

 

You might want to do a bit of adjustment on your split levels & min free settings so this won't happen again.

 

 

Yup. They are my worst enemy.

 

I am going to re-evaluate my settings to save me from headaches like this again. I still want things neat, but need to put a bit more thought into the layout.

 

... I still want things neat, but need to put a bit more thought into the layout.

 

I presume you realize that "neat" is more a function of how you've logically organized your data (i.e. the shares and folder structures) than what physical disk something's on.    One of the key benefits of UnRAID is the user share feature ... which provides a unified view of data that can be spread across many physical disks.    This ability to have what in effect is an "unlimited" disk size, which can grow as necessary (by simply adding more disks to the array) allows you to organize your data as you like without the need to be concerned with which physical device it's stored on.    The "split level" is primarily designed to allow key elements of a structure to be stored on the same physical disk (i.e. the .VOB's associated with a single DVD movie), so only one disk need be spun up to access that entity.    Setting it at a level that effectively disables the ability to take advantage of the user share feature is more "messy" than neat 

... I still want things neat, but need to put a bit more thought into the layout.

 

I presume you realize that "neat" is more a function of how you've logically organized your data (i.e. the shares and folder structures) than what physical disk something's on.    One of the key benefits of UnRAID is the user share feature ... which provides a unified view of data that can be spread across many physical disks.    This ability to have what in effect is an "unlimited" disk size, which can grow as necessary (by simply adding more disks to the array) allows you to organize your data as you like without the need to be concerned with which physical device it's stored on.    The "split level" is primarily designed to allow key elements of a structure to be stored on the same physical disk (i.e. the .VOB's associated with a single DVD movie), so only one disk need be spun up to access that entity.    Setting it at a level that effectively disables the ability to take advantage of the user share feature is more "messy" than neat 

 

Good point. It's just a perception thing. While I like the idea of splitting the share over multiple disks, I prefer a show being grouped together - though you are right - since I access the data via the share 99% of the time, where it sits is really academic.

Just to confirm my fidnings.

 

If the mover trys to move files from the cache drive to the main array and fails due to lack of disk space (i.e. the physical disk it is copying to becomes full)....

 

then the mover will fail with this error "hfs_write: write: (28) No space left on device".

 

You now have the original file on the cache drive and a partial copy on the array drive. unRAID does not dleete this partial file and forever more will try to rsync the file every night and fail.

 

Until you manually delete the partial file or make space on that specific physical drive it will fail. Even if you have 100TB free on other drives it will fail with an out of space error.

 

Come to think of it this is a bug. Partial rsyncs should be deleted.

Just to confirm my fidnings.

 

If the mover trys to move files from the cache drive to the main array and fails due to lack of disk space (i.e. the physical disk it is copying to becomes full)....

 

then the mover will fail with this error "hfs_write: write: (28) No space left on device".

 

You now have the original file on the cache drive and a partial copy on the array drive. unRAID does not dleete this partial file and forever more will try to rsync the file every night and fail.

 

Until you manually delete the partial file or make space on that specific physical drive it will fail. Even if you have 100TB free on other drives it will fail with an out of space error.

 

Come to think of it this is a bug. Partial rsyncs should be deleted.

 

This was what highlighted the issue for me to begin with. Looking in the directory on the array I had a number of zero byte files. Thankfully once I freed up space and re-ran the mover it replaced them. It also looks like the zero byte nfo file and thumb picture have been updated.

 

 

... and you should be able to completely avoid this ever happening with the correct settings for Min Free and split levels => i.e. if the Min Free is greater than the size of any file you'll write then there will always be room (if there's not, it will use a different disk) as long as the split level isn't restricting it from using a different disk.

 

You can never "completely avoid this" because min free only works in the instant you go to add a file e.g it doesn't work at all if a file grows.

 

Fundamentally the problem is a failed mover should not leave partial files around the array.