Unable to format new drives, system locks up (RC5)

9 minutes ago, je82 said:

The other guy who posted in this thread that this happened to used XFS but not encrypted as far as i know so probably no

Correct, array is not encrypted. 

 

Original array was created with 6.7 series.

22 minutes ago, je82 said:

Don't know, the two times it happened for me was:

1. I have an already working array, with 2x cache btrfs drives and 2x parity drives.

2. I add a totally new drive

3. Start array, preclear starts..

4. Click format, that's when it starts to crash. Webgui will respond in the beginning but the longer it goes the less things are responding and it seems to be stuck formating, the format never completes it just says "Formating...".

5. You eventually try to do a graceful shutdown, it wont work.

I used the exact process when this happened, only I have 1 parity and I was adding 2 drives to the array at the same time. And I had the exact same result

 

2 hours ago, bonienl said:

Preclearing is not needed

As stated, formatting on an already pre-cleared drive works (I tried formatting after my server came back up after the crash).  

Could reproduce this issue, formatting a device xfs-encrypted after clearing resulted in a server crash, unable to get diags, this is what was on the screen:

 

 

Will now try with btrfs, to see if it's a general clear related bug.

19 minutes ago, johnnie.black said:

Will now try with btrfs, to see if it's a general clear related bug.

Same thing happened when trying to format btrfs-encrypted after clearing, and since the same happens without encryption to @civic95man it appears to be a bug with the clear process.

Was able to get the start of the error by tailing the syslog, might help see what the problem is, this time was after formatting with btrfs, no encryption, after a clearing.

syslog.txt

3 hours ago, johnnie.black said:

Was able to get the start of the error by tailing the syslog, might help see what the problem is, this time was after formatting with btrfs, no encryption, after a clearing.

Thank you, looking into this..

Regarding the additional used space after the format (in my case, after rebooting the server after the initial format resulted in a freeze):   

 

I found mention of

reflink=1

during mkfs.xfs time resulting in the extra metadata creation/reservation (such as 78G for a 12TB drive), whereas

reflink=0

resulted in the standard 1GB per TB we were used to.  @johnnie.black, do you still have your cleared xfs drive and if so, could you format it and see what if reflink=1 was used?  Just an idea as to solve the mystery of the increase in initial "used" space after format.

 

Edit:  Would this mean that any disks formatted with this option (i.e. in 6.8 series) would be incompatible with older versions of unraid (6.7.2 and older)?

Edited November 20, 2019 by civic95man

1 hour ago, civic95man said:

I found mention of

reflink=1

during mkfs.xfs time resulting in the extra metadata creation/reservation (such as 78G for a 12TB drive), whereas

reflink=0

resulted in the standard 1GB per TB we were used to

Where do you see 'reflink=1'?

21 minutes ago, limetech said:

Where do you see 'reflink=1'?

I don't.  I was questioning if it was set.  I already removed the 2 drives I had formatted otherwise I would have checked for myself.  This was merely a suggested cause of the increased "used" space after formatting, and a quick google search pulled up this:  https://serverfault.com/questions/983907/newly-created-xfs-filesystem-shows-78-gb-used

 

I guess I'm nervous about formatting and beginning to fill up a drive that *possibly* would become invalid in the future.

9 hours ago, civic95man said:

could you format it and see what if reflink=1 was used?

It wasn't used on the command but it was used for the format, so possibly it's the default now:

 

Quote

Nov 21 07:31:33 Test emhttpd: shcmd (167): mkfs.xfs -m crc=1,finobt=1 -f /dev/md1
Nov 21 07:31:34 Test root: meta-data=/dev/md1               isize=512    agcount=4, agsize=15262410 blks
Nov 21 07:31:34 Test root:          =                       sectsz=512   attr=2, projid32bit=1
Nov 21 07:31:34 Test root:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
Nov 21 07:31:34 Test root:          =                       reflink=1
Nov 21 07:31:34 Test root: data     =                       bsize=4096   blocks=61049638, imaxpct=25
Nov 21 07:31:34 Test root:          =                       sunit=0      swidth=0 blks
Nov 21 07:31:34 Test root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Nov 21 07:31:34 Test root: log      =internal log           bsize=4096   blocks=29809, version=2
Nov 21 07:31:34 Test root:          =                       sectsz=512   sunit=0 blks, lazy-count=1
Nov 21 07:31:34 Test root: realtime =none                   extsz=4096   blocks=0, rtextents=0

 

8 hours ago, civic95man said:

I guess I'm nervous about formatting and beginning to fill up a drive that *possibly* would become invalid in the future.

I already posted earlier that a xfs drive formatted with v6.8 mounts normally with v6.7

8 hours ago, johnnie.black said:

I already posted earlier that a xfs drive formatted with v6.8 mounts normally with v6.7

Mostly to satisfy my curiosity, and sorry for being off topic since this bug report doesn't have anything to do with xfs, I also tested earlier Unraid releases:

 

v6.2.4 - kernel 4.4.30 - doesn't mount, only read-only mount is possible:

Nov 21 11:33:21 Tower9 kernel: XFS (md2): Superblock has unknown read-only compatible features (0x4) enabled.
Nov 21 11:33:21 Tower9 kernel: XFS (md2): Attempted to mount read-only compatible filesystem read-write.
Nov 21 11:33:21 Tower9 kernel: XFS (md2): Filesystem can only be safely mounted read only.

v6.3.5 - kernel 4.9.30 - mounts normally with a warning:

Nov 21 11:38:54 Tower9 kernel: XFS (md2): EXPERIMENTAL reflink feature enabled. Use at your own risk!

The warning remains until v6.6.7 with kernel 4.18.20 where it mounts normally without any warning, since I only tested one release from each v6.x series it's likely the warning was removed on kernel 4.8 series.

 

 

Edited November 21, 2019 by johnnie.black

7 hours ago, johnnie.black said:

It wasn't used on the command but it was used for the format, so possibly it's the default now

Interesting... Is this an option to enable as default at compile time?  Its my understanding that reflink is similar to cow on btrfs?  Is this accurate?

 

Thanks for testing the previous versions.  I guess that helps put my mind at ease.  

 

Correct, the xfs free space doesn't directly affect this bug report.  I was thinking of @je82's original post mentioning this issue along side the preclear/format issue.

10 minutes ago, civic95man said:

Interesting... Is this an option to enable as default at compile time?

I would assume it's enable by default on newer kernels.

 

11 minutes ago, civic95man said:

Its my understanding that reflink is similar to cow on btrfs?  Is this accurate?

From my understating it allows using e.g. cp --reflink-always with xfs, also de-duplication support, but I don't keep up with xfs development.

 

 

Changed Status to Retest

@limetech, all seems good now. Much appreciated!  I take it that reflink=1 is the new default for xfs? 

10 hours ago, civic95man said:

I take it that reflink=1 is the new default for xfs?

If another option, 'crc=1' is set (which we do set since feature has been available) then reflink=1 is the default.

Changed Status to Solved