Unraid 653 locks up after deletion of many files

[CDebarry]

Hi Guys,

 

I'm having trouble with my server.

Every time I start a deletion of a very large amount of files the server stops responding and I have to hard reset it.

Its a production server with many files and I'm really scared of data corruption.

 

Its somting like 20Gb of data, but 100k files.

 

Everything stops responding: file shares, the VMs, docks and GUI.

 

Any advise?

Thank you!!

 

Update:
Seams I can log in via SSH. But neither shutdown nor reboot commands works. Server does not do anything.

Edited March 6, 2019 by CDebarry

[Tom3]

It may depend on how you are doing the deletion.  For example the Ubuntu GUI File manager will by default

put each file in the 'Trash' directory so they can be recovered later.  This slows down massive file deletions terribly.

There is a shift-delete on Ubuntu GUI that just deletes the files without this extra step, and it runs much

faster.

 

If you are deleting files using the command line from the terminal, then by default it doesn't use the trash

directory.

 

-- Tom

 

[John_M]

Since you're deleting a very large number of files you are making a very large number of small changes to the file system metadata. It isn't the amount of data that matters but the number of changes. Have you tried just leaving it to get on with its job? That is much less likely to result in data corruption than resetting it. Then again, your server might have a real problem but with no information or diagnostics it's impossible to tell.

[CDebarry]

14 minutes ago, John_M said:

Since you're deleting a very large number of files you are making a very large number of small changes to the file system metadata. It isn't the amount of data that matters but the number of changes. Have you tried just leaving it to get on with its job? That is much less likely to result in data corruption than resetting it. Then again, your server might have a real problem but with no information or diagnostics it's impossible to tell.

I did, actually. It run for 12 hours.

But since its production I had to reset it. (And 12 hours is plenty of time, right?)

 

[CDebarry]

27 minutes ago, Tom3 said:

It may depend on how you are doing the deletion.  For example the Ubuntu GUI File manager will by default

put each file in the 'Trash' directory so they can be recovered later.  This slows down massive file deletions terribly.

There is a shift-delete on Ubuntu GUI that just deletes the files without this extra step, and it runs much

faster.

 

If you are deleting files using the command line from the terminal, then by default it doesn't use the trash

directory.

 

-- Tom

 

Deletion is happening from Windows Explorer though a share.

[kizer]

Have you attempted to delete less than that at a time? Maybe do them in chunks?

I know when I use windows and and right mouse sometimes my File Explorer simply crashes and then sometimes resumes. 

 

Normally if I'm doing a lot I'll fire up putty and use MC aka Midnight Commander to do a lot. However if your having users access the files and delete them they might not have access nor the knowledge for quick get-arounds to things. 

 

As well are the files on a particular drive that might be having issues?

[CDebarry]

2 hours ago, kizer said:

Have you attempted to delete less than that at a time? Maybe do them in chunks?

I know when I use windows and and right mouse sometimes my File Explorer simply crashes and then sometimes resumes. 

 

Normally if I'm doing a lot I'll fire up putty and use MC aka Midnight Commander to do a lot. However if your having users access the files and delete them they might not have access nor the knowledge for quick get-arounds to things. 

 

As well are the files on a particular drive that might be having issues?

I have... and that's what I'm doing.
But sometimes I misjudge the size of the folder to be deleted and I find my self in trouble.

 

 

 

[CDebarry]

Thanks everyone for the help!

 

I'm still tying to troubleshoot this issue.

Maybe its a bug, maybe hardware falt/limitation?

 

Would log help?

[JonathanM]

What filesystem?

[trurl]

3 hours ago, CDebarry said:

Would log help?

7 hours ago, John_M said:

with no information or diagnostics it's impossible to tell.

John_M hinted at it, but we somehow made it to the 10th post in this thread without someone saying:

 

@CDebarry

Go to Tools - Diagnostics and attach the complete diagnostics zip file to your next post.

 

[CDebarry]

8 minutes ago, jonathanm said:

What filesystem?

Hi, it's XFS.

 

Thank you.

[trurl]

OK, so no GUI.

 

Do you have an attached keyboard and monitor?

[CDebarry]

I don't. But I can plug it in if needed.

[CDebarry]

4 minutes ago, trurl said:

John_M hinted at it, but we somehow made it to the 10th post in this thread without someone saying:

 

@CDebarry

Go to Tools - Diagnostics and attach the complete diagnostics zip file to your next post.

 

 

unviper-diagnostics-20190306-2129.zip

[trurl]

11 minutes ago, CDebarry said:

Hi, it's XFS.

According to your syslog, it was your cache drive that was full, and it is btrfs. Possibly you have corrupted cache by overfilling it.

 

But this is the first mention in this thread of cache. Do you actually know which disk was involved?

[CDebarry]

1 hour ago, trurl said:

According to your syslog, it was your cache drive that was full, and it is btrfs. Possibly you have corrupted cache by overfilling it.

 

But this is the first mention in this thread of cache. Do you actually know which disk was involved?

I have just notice my cache fs is different from other data disks.. it's this ok?

 

Cache is an SSD and I THINK I disabled cache for most shares.

 

Why would cache be used on deletion?

[trurl]

1 hour ago, CDebarry said:

Why would cache be used on deletion?

Cache is part of the user shares. If you were deleting data from a user share, and that data was still on the cache disk, then of course it would be involved in the deletion.

 

I thought perhaps the whole reason you were deleting the data is because you had filled cache somehow and were trying to deal with that problem.

 

 

[trurl]

1 hour ago, CDebarry said:

I have just notice my cache fs is different from other data disks.. it's this ok?

Yes. In fact, btrfs is required for cache if you want to have more than one disk in the cache pool.

[trurl]

1 hour ago, CDebarry said:

I THINK I disabled cache for most shares.

You certainly had some shares configured to use cache. Unfortunately, I can't tell whether those shares still exist and what disks they might be using, because the diagnostics from that OLD version of Unraid you are running don't tell me enough information.

[CDebarry]

4 minutes ago, trurl said:

Cache is part of the user shares. If you were deleting data from a user share, and that data was still on the cache disk, then of course it would be involved in the deletion.

 

I thought perhaps the whole reason you were deleting the data is because you had filled cache somehow and were trying to deal with that problem.

 

 

Oh I see what you mean.

That's not the issue, tough.

 

Cache is not full. All data has already been moved from cache to data disks. They reside only in data disks so I guess cache is not really used, is it?

 

Thank you!

[CDebarry]

Tomorrow I'll try disabling cache drive.

Let's see what happens.

 

Thanks everyone for the help!!

[CDebarry]

6 minutes ago, trurl said:

You certainly had some shares configured to use cache. Unfortunately, I can't tell whether those shares still exist and what disks they might be using, because the diagnostics from that OLD version of Unraid you are running don't tell me enough information.

Just realized I missed some upgrades. 653 is only two versions behind, isn't it? Will investigate what's changed.

 

Thank you!

[trurl]

5 minutes ago, CDebarry said:

Cache is not full. All data has already been moved from cache to data disks. They reside only in data disks so I guess cache is not really used, is it?

It wasn't full at the time your diagnostics were taken, but it also wasn't empty. And I wouldn't expect it to be, nor would you want it to be since that is where you should have your dockers setup.

 

But, according to the syslog in those same diagnostics, you had a lot of errors occurring earlier because cache was completely full. You mustn't let any disk get completely full.

2 minutes ago, CDebarry said:

653 is only two versions behind, isn't it?

Well, that depends on how you want to count it I guess. There were a lot of releases between what you have and what is available now but they aren't all available for download anymore.

 

[trurl]

9 minutes ago, CDebarry said:

Tomorrow I'll try disabling cache drive.

Why? Did anyone suggest that? What do you even mean by "disabling cache" anyway?

[CDebarry]

2 minutes ago, trurl said:

It wasn't full at the time your diagnostics were taken, but it also wasn't empty. And I wouldn't expect it to be, nor would you want it to be since that is where you should have your dockers setup.

 

But, according to the syslog in those same diagnostics, you had a lot of errors occurring earlier because cache was completely full. You mustn't let any disk get completely full.

Well, that depends on how you want to count it I guess. There were a lot of releases between what you have and what is available now but they aren't all available for download anymore.

 

Ohhh so maybe cache is used when deleting files even though these folks are not on cache? Investigating now.