August 3, 20178 yr New config -> Retain All -> With or without "parity is already valid" checked array starts with all disk unmountable with the following errors: Aug 3 11:44:10 Tower8 emhttpd: shcmd (515): mkdir -p /mnt/disk1 Aug 3 11:44:10 Tower8 emhttpd: shcmd (516): mount -t btrfs,xfs,reiserfs -o noatime,nodiratime /dev/ /mnt/disk1 Aug 3 11:44:10 Tower8 root: mount: /mnt/disk1: /dev is not a block device. Aug 3 11:44:10 Tower8 emhttpd: shcmd (516): exit status: 32 Aug 3 11:44:10 Tower8 emhttpd: mount error: No file system (32) Aug 3 11:44:10 Tower8 emhttpd: shcmd (517): umount /mnt/disk1 Aug 3 11:44:10 Tower8 root: umount: /mnt/disk1: not mounted. Aug 3 11:44:10 Tower8 emhttpd: shcmd (517): exit status: 32 Aug 3 11:44:10 Tower8 emhttpd: shcmd (518): rmdir /mnt/disk1 Aug 3 11:44:10 Tower8 emhttpd: shcmd (519): mkdir -p /mnt/disk2 Aug 3 11:44:10 Tower8 emhttpd: shcmd (520): mount -t btrfs,xfs,reiserfs -o noatime,nodiratime /dev/ /mnt/disk2 Aug 3 11:44:10 Tower8 root: mount: /mnt/disk2: /dev is not a block device. Aug 3 11:44:10 Tower8 emhttpd: shcmd (520): exit status: 32 Aug 3 11:44:10 Tower8 emhttpd: mount error: No file system (32) Aug 3 11:44:10 Tower8 emhttpd: shcmd (521): umount /mnt/disk2 Aug 3 11:44:10 Tower8 root: umount: /mnt/disk2: not mounted. Aug 3 11:44:10 Tower8 emhttpd: shcmd (521): exit status: 32 Aug 3 11:44:10 Tower8 emhttpd: shcmd (522): rmdir /mnt/disk2 Aug 3 11:44:10 Tower8 emhttpd: shcmd (523): mkdir -p /mnt/disk3 Aug 3 11:44:10 Tower8 emhttpd: shcmd (524): mount -t btrfs,xfs,reiserfs -o noatime,nodiratime /dev/ /mnt/disk3 Aug 3 11:44:10 Tower8 root: mount: /mnt/disk3: /dev is not a block device. Aug 3 11:44:10 Tower8 emhttpd: shcmd (524): exit status: 32 Aug 3 11:44:10 Tower8 emhttpd: mount error: No file system (32) Aug 3 11:44:10 Tower8 emhttpd: shcmd (525): umount /mnt/disk3 Aug 3 11:44:10 Tower8 root: umount: /mnt/disk3: not mounted. Aug 3 11:44:10 Tower8 emhttpd: shcmd (525): exit status: 32 Aug 3 11:44:10 Tower8 emhttpd: shcmd (526): rmdir /mnt/disk3 Stopping and re-starting the array fixes the problem, full diags attached. tower8-diagnostics-20170803-1147.zip
August 3, 20178 yr Possibly(???) related to this issue that I had: Of course, Once it was detected and assigned again, the issue has not happened again. (I have rebooted the server several times since then.) So apparently it to me to be a one time thing. Edited August 3, 20178 yr by Frank1940
August 4, 20178 yr @johnnie.black Interesting logs ...they look very much like mine with the key exception being that subsequent attempts to start my array did not yield a different result. I can see in your logs, that your drives mounted in 'auto' mode on the second attempt, yet not sure why it didn't work on the first attempt. It wasn't until I changed mine to 'xfs' that my array mounted. Not sure if it matters you're using reiserfs vs xfs in my case. As to the question Frank1940 raises, I'm not aware the mount command, or auto, caches any detected file-system time to make subsequent attempts work. mount as a process carries no persistence between executions. @Frank1940 I've seen the issue you've detailed as well but I think yours is different root cause. A drive missing vs a file system not mounting are two different beasts all together. I can't consistently reproduce your missing cache disk issue to any effect, too infrequent. I do know that when it happens, it's something specific to the cache disk only, I have NOT seen other disk types (data or parity) go missing.
August 6, 20178 yr @johnnie.black I moved all my data and parity disks to a new server this evening. Encountered what you described. The second attempt to mount worked and the mount command got the XFS mounted. tower2-diagnostics-20170805-2253.zip
August 14, 20178 yr Same thing again tonight... disk11 near the end of log. tower2-diagnostics-20170813-2236.zip
August 14, 20178 yr I have found that with 6.4 rc7a if I do a New Config and leave all disks set to the default of ‘auto’ then the first time I start the array all disks show as unmountable. Stopping and restarting the array resolves the problem and they now all show as XFS. I find this behaviour completely reproducible. Whether it was there in an earlier release I am not sure as it is not often that I do a New Config. It looks like there must be some sort of race condition between the code that detects the format type and the mount operation? Although the array stop/restart is an easy workaround I could see this leading to some users panicking and therefore taking inappropriate action that might lead to data loss.
August 14, 20178 yr 12 hours ago, itimpi said: Although the array stop/restart is an easy workaround I could see this leading to some users panicking and therefore taking inappropriate action that might lead to data loss. Yes and yes. The example in my logs attached above didn't involve 'new config' as the prior cases have. This new example may be effecting anyone, but if so then reports of it would have been more widespread. Unless just not enough time has passed that a large group of users on 6.4rc7a have used new config, or in my most recent case, added an additional new disk to an existing array. Here's what in the log attached above... Existing array added precleared disk11 array was started I picked format disk11, after format it wasn't mounted (it tried 'auto'). array stopped. array started, disk11 didn't mount (again it tried 'auto') array stopped array started, and then it mounts (this time it did 'xfs', 'not auto')
September 3, 20178 yr Author There's a new auto filesystem setting for the cache pool, which is the default after doing a new config, but cache won't mount ( Unmountable: not btrfs pool ) if cache slots >1 and fs=auto, user needs to change fs to btrfs.
September 3, 20178 yr Author 2 minutes ago, bonienl said: Is it the same with a single cache device? Yes, if cache slots > 1
September 3, 20178 yr 3 minutes ago, johnnie.black said: Yes, if cache slots > 1 Can I conclude: when cache slots = 1, it is okay (didn't test myself)?
September 3, 20178 yr Author Just now, bonienl said: Can I conclude: when cache slots = 1, it is okay (didn't test myself)? Yes, auto works with slots=1
September 9, 20178 yr Almost a month later and no one else seeing this? I hit it consistently on 2 out of 2 servers. I'm going to have to check my setups, maybe I'm doing something outside of the norm. Next time I'll be sure to use a completely new installation straight from the USB creator tool with no previous files manually copied over. Might help rule something out with that.
September 9, 20178 yr Author 19 minutes ago, Lev said: Almost a month later and no one else seeing this? I hit it consistently on 2 out of 2 servers. I'm going to have to check my setups, maybe I'm doing something outside of the norm. Next time I'll be sure to use a completely new installation straight from the USB creator tool with no previous files manually copied over. Might help rule something out with that. Fixed on rc8
September 9, 20178 yr 3 hours ago, johnnie.black said: Fixed on rc8 Thank you sir, I'll give it a test.
September 24, 20178 yr I've just reset config on my system. Added a new Parity drive 8TB, (it was precleared) Added the old parity drive as a data disk. (Was not cleared) Added another data disk 8TB (it was precleared) What I'm left with is a system doing a parity sync and two new data drive that state they are unmountable: no file system. Also the cache drive states Unmountable: not btrfs pool Any ideas what I need to do in order to format the drives, I really don't want to sit through the full parity sync and then have to start it again. tower-diagnostics-20170924-1339.zip Edited September 24, 20178 yr by Tuftuf
September 24, 20178 yr Guess I can only do a hard reboot to stop the parity sync. I may of been blind.
September 24, 20178 yr 1 minute ago, Tuftuf said: Guess I can only do a hard reboot to stop the parity sync. I may of been blind. You don't need to stop the sync.
September 24, 20178 yr Author Just now, Tuftuf said: Guess I can only do a hard reboot to stop the parity sync There's a cancel button.
September 24, 20178 yr Just now, Squid said: You don't need to stop the sync. It wants to format my cache drive as well. I need to get that mountable first.
September 24, 20178 yr 10 minutes ago, johnnie.black said: There's a cancel button. Ok thanks! Distracted by other things and didn't expect this go wrong. Will use a bit more focus now. EDIT - Current issue : Unmountable: not btrfs pool This happened after creating a new config, to remove a failed disk that I had removed all data off. The cache was mounted prior to creating a new config. I cannot format the new disks until I fix the cache drive. UNMOUNTABLE DISKS PRESENT:DISK 2 • ST8000VN0022-2EL112_ZA18PB9C (SDF)DISK 3 • WDC_WD40EFRX-68WT0N0_WD-WCC4E7TPZ2JS (SDD)CACHE • SAMSUNG_SSD_840_EVO_500GB_S1DHNSADC22098M (SDC) Edited September 24, 20178 yr by Tuftuf
September 24, 20178 yr Author Stop the array, on the main page click on cache, change the filesystem from auto to btrfs, start the array and it should mount.
September 24, 20178 yr 8 minutes ago, johnnie.black said: Stop the array, on the main page click on cache, change the filesystem from auto to btrfs, start the array and it should mount. Thank you, I just did this. Once I decided to stop panicking and started to think it became easier. I've just not had a good couple of days with changing things on this system. The disk is mounted and all my dockers reappeared. The remaining disks are now formatting. Edited September 24, 20178 yr by Tuftuf
Archived
This topic is now archived and is closed to further replies.