I have no idea what is going on

 

I moved a 255G folder off the pool (I deactivated the cache for the backup share first, so straight to my array)

 

 

Then i got this "No such file or directory" error which is odd. The pool data usage in my unRaid UI went from 7.87TB down to 7.85TB which is 20GB down while i moved 255GB data, which is odd again.

Then i wanted to move the other 72G folder off as well. I executed the move command. While it was running the terminal window suddenly closed, and now my server is unreachable and the monitor through my pikvm looks super weird

 

 

I think it completely froze, because also when i enter something and press enter a couple times nothing happens. So i hard reset it and let it boot up again. If useful for whatever reason i took a diagnostics right after startup before starting up the array again: tower-diagnostics-20240627-1647.zip

 

Then i started the array and cancelled the parity check (i should probably do one but once this has all been resolved i think is better, otherwise im doing 10 of them and waiting for each one every time). Now the nas-backup pool is back at 7.87TB used

 

And the folder that i just moved off is back again!!

 

What the hell is going on

 

Also to you all of these issues must sound like my hardware is on the brink of survival but before i started this whole pool thing with the drive that is too full i never had random freezes and crashes like this. I used to have an uptime of over a year a couple months ago.

 

Edit: And the 255G folder is also in the location i moved it to, just like what happened before:

Edited June 27, 20242 yr by YourNightmar3

The filesystem is possibly crashing and going read only during the move, enable the mirror to flash drive option in the syslog server, try moving the data again and post that log if it crashes.

On 6/27/2024 at 5:57 PM, JorgeB said:

The filesystem is possibly crashing and going read only during the move, enable the mirror to flash drive option in the syslog server, try moving the data again and post that log if it crashes.

I enabled the mirrror syslog to usb option, then did the move again, and got this same error:

 

 

Then moved the Photo folder and got this:

 

 

Also a BTRFS error on my unraid machine:

 

 

 

But there was no kernel crash this time, it just turned to readonly again. In total i tried to move almost 1TB of data off the pool, but the "used" number in the unRaid GUI only went from 7.87TB to 7.85TB. My Photo folder is also still in full on the disk, and when i try `du -sh` on the P### folder i get a message that there is no such file or directly while `ls` still lists it.

 

Here is my syslog: syslog

Really not sure what to do. Im considering just completely wiping the drive and starting the backup over from the start, if that's even possible at the moment. But i would highly prefer to not have to send all 8TB over the internet again, it'll take weeks. The internet connection at the unRaid server's side is not the fastest.

 

 

Edited July 2, 20242 yr by YourNightmar3

At this point the pool appears to be beyond fixable, I would recommend mounting it read-only, then copy all the data somewhere else, then re-format.

Alright that is a good idea but unfortunately i don't have available hard drive space on my array for all the data. I will copy the Video folder (6.2TB) to my array and then redo the other 1.5TB of backup data over the internet.

 

How can i re-format the drive(s) of the pool? And then how do i prevent this from ever happening again? Should i enable a "Minimum free space" on the drive(s)?

 

Edit: And what would be a good way of copying the folder from the pool to my array? If i just use 'mv' or 'cp' i can't see any progress which would be kind of difficult with 6TB of data because it'll take a while.

Edited July 2, 20242 yr by YourNightmar3

1 hour ago, YourNightmar3 said:

How can i re-format the drive(s) of the pool?

You can click "erase" in the pool settings, then format after array start.

 

1 hour ago, YourNightmar3 said:

Should i enable a "Minimum free space" on the drive(s)?

Yes, you can set that for the pool.

 

1 hour ago, YourNightmar3 said:

And what would be a good way of copying the folder from the pool to my array?

You can use midnight commander (mc on the console) or the Dynamix file manager plugin

I used midnight commander to copy the 6.1TB "video" folder from the pool to my array. I then did as you said: clicked "erase" in the pool settings and then clicked "format" after array start. I then used midnight commander again to copy the 6.1TB back from the array to the pool, and after 3.06TB copied, my server crashed again.

 

I will continue the copy action later, for now i need it to stay up for a bit.

 

Here are the two syslog files on my usb:

 

syslogsyslog-previous

 

Can you determine what went wrong?

11 hours ago, YourNightmar3 said:

my server crashed again.

 

Jul  4 20:42:33 Tower kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Jul  4 20:42:33 Tower kernel: ? _raw_spin_unlock+0x14/0x29
Jul  4 20:42:33 Tower kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

 

Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), then reboot.

On 7/5/2024 at 9:55 AM, JorgeB said:

 

Jul  4 20:42:33 Tower kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Jul  4 20:42:33 Tower kernel: ? _raw_spin_unlock+0x14/0x29
Jul  4 20:42:33 Tower kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

 

Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), then reboot.

 

Thank you, that seems to have fixed that. The backup is running again, everything seems good so far. Thanks a lot for all the help.