How can i make sure it never happens again? Can i set a minimum free size on the pool?

 

Also it looks like it turned back to readonly?

 

 

diags:tower-diagnostics-20240626-1339.zip

Try stopping and restarting the array, then run the balance again, if it still goes read-only, you will need to free up some more space.

Restarting the array seems to have worked, is 0 relocated good?

 

 

New diags: tower-diagnostics-20240626-1349.zip

Edited June 26, 20242 yr by YourNightmar3

It needs to relocate a few chunks, try with -dusage=20, and if still 0 relocated chunks, try 30, etc

 

ehhhh??? im confused. Do i restart the array again and try 100 again? Do i move more off the pool?

That's weird, re-start the array and post new diags.

alright i restarted the array and here are the new diags

 

tower-diagnostics-20240626-1450(1).zip

Huh.... i just noticed the 72G folder that i moved off the pool is back on it. But it's also still in the folder that i moved it to. And both are 72G

 

Edited June 26, 20242 yr by YourNightmar3

6 minutes ago, YourNightmar3 said:

i just noticed the 72G folder that i moved off the pool is back on it.

Possibly the pool was read-only when you tried before, try again now, and post new diags after the move.

That's odd because im pretty sure  saw the used space in the UI benig a bit less on the drive that i moved the folder off of.

 

 

This was a screenshot from earlier:

 

 

so it doesnt seem like it was readonly.

 

And now i got this:

 

 

Edited June 26, 20242 yr by YourNightmar3

The entire server just crashed suddenly, i've never seen this before.

 

Here are the diags after a reboot:

 

tower-diagnostics-20240626-1532.zip

Do you know which device is this?

 

Jun 26 15:29:01 Tower kernel: sd 5:0:0:0: [sdb] tag#4 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
Jun 26 15:29:01 Tower kernel: sd 5:0:0:0: [sdb] tag#4 CDB: opcode=0x28 28 00 74 6f 6f 80 00 00 08 00
Jun 26 15:29:01 Tower kernel: blk_print_req_error: 4 callbacks suppressed
Jun 26 15:29:01 Tower kernel: I/O error, dev sdb, sector 1953460096 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
Jun 26 15:29:01 Tower kernel: sd 5:0:0:0: [sdb] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
Jun 26 15:29:01 Tower kernel: sd 5:0:0:0: [sdb] tag#5 CDB: opcode=0x28 28 00 74 6f 6f 80 00 00 08 00
Jun 26 15:29:01 Tower kernel: I/O error, dev sdb, sector 1953460096 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jun 26 15:29:01 Tower kernel: Buffer I/O error on dev sdb1, logical block 244182256, async page read
Jun 26 15:29:01 Tower kernel: sd 5:0:0:0: [sdb] tag#8 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
Jun 26 15:29:01 Tower kernel: sd 5:0:0:0: [sdb] tag#8 CDB: opcode=0x28 28 00 74 6f 6f 80 00 00 08 00
Jun 26 15:29:01 Tower kernel: I/O error, dev sdb, sector 1953460096 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
Jun 26 15:29:01 Tower kernel: sd 5:0:0:0: [sdb] tag#10 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
Jun 26 15:29:01 Tower kernel: sd 5:0:0:0: [sdb] tag#10 CDB: opcode=0x28 28 00 74 6f 6f 80 00 00 08 00
Jun 26 15:29:01 Tower kernel: I/O error, dev sdb, sector 1953460096 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jun 26 15:29:01 Tower kernel: Buffer I/O error on dev sdb1, logical block 244182256, async page read
Jun 26 15:29:02 Tower emhttpd: device /dev/sdb problem getting id

 

Likely a USB device

Could it be my boot USB? I have been suspecting it's dying for a little while now because sometimes i have trouble getting the server started up after a shutdown. I am planning to replace it.

 

But that should be unrelated to my drive pool issue because the boot issue was already present before i added the extra hard drive for the pool. It is also the only USB storage device i have plugged into the server.

 

And immediately after starting the array, the pool is readonly again

 

I also immediately got this error on my kvm:

that is the 8TB drive in the nas-backup pool that is full.

 

Attached diags again in case its nessecary tower-diagnostics-20240626-1603.zip

Edited June 26, 20242 yr by YourNightmar3

It's not the flash drive, looks like it's a SanDisk_SSD_PLUS_1000GB, and it's not USB, but it has the same ID as another one, that's not normal with SATA devices. 

Ah, i took a look into the physical machine and you're right. There is a SanDisk_SSD_PLUS_1000GB still connected that has died. I used to have two of them for my cache, i now have two nvme ssd's for my cache and both of those Sandisk SATA ssd's are still wired up while one is dead. It's been in there for i think well over a year at this point.

Edited June 27, 20242 yr by YourNightmar3

It would be better to disconnect it, if nothing else to stop the log spam.

Alright i'll take it out when i have physical access to the machine again.

 

I wanted to try the whole thing again from start to finish, this time moving a bigger folder off the pool. I restarted my array, then did:

mkdir /x
mount -t btrfs -o skip_balance /dev/sdh1 /x

 

and i get

mount: /x: wrong fs type, bad option, bad superblock on /dev/sdh1, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.

 

Which last time meant the /dev/sdh1 wasn't correct, but this time it is:

 

So what is going wrong now?  tower-diagnostics-20240627-1313.zip

Edited June 27, 20242 yr by YourNightmar3

7 minutes ago, YourNightmar3 said:

mount -t btrfs -o skip_balance /dev/sdh1 /x

This won't help now, since there's no balance running.

Oh alright, so then what can i do now to fix the read only file system error?

 

2 minutes ago, JorgeB said:

This won't help now, since there's no balance running

Scratch that, there is balance running, the last one you tried, reboot, son't start the array, and try the skip balance again, using the correct drive letter, but use:

 

mount -vt btrfs -o skip_balance /dev/sdX1 /x

 

It should give more info if it still fails.

Alright the disk is now sdf:

So i did:

 

What is next?  

 

On 6/26/2024 at 9:00 AM, JorgeB said:

btrfs balance cancel /x
umount /x

 

The start the array and try again to move some data from the pool.

Alright thank you, i got this message after the first command which i got last time as well but i closed my console without taking a screenshot of it:

 

Should i still continue with umount /x or should i do something else because of this warning?

Yes, you can continue, that is what we are trying to fix, but need to free up some space first.