December 24, 20241 yr help ~~ this is log,i found that when i start array , zpool run the replace commadn , sda was my flash Dec 24 10:56:42 xBox kernel: mdcmd (31): set md_num_stripes 1280 Dec 24 10:56:42 xBox kernel: mdcmd (32): set md_queue_limit 80 Dec 24 10:56:42 xBox kernel: mdcmd (33): set md_sync_limit 5 Dec 24 10:56:42 xBox kernel: mdcmd (34): set md_write_method Dec 24 10:56:42 xBox kernel: mdcmd (35): start STOPPED Dec 24 10:56:42 xBox kernel: unraid: allocating 15750K for 1280 stripes (3 disks) Dec 24 10:56:42 xBox kernel: md1p1: running, size: 244197560 blocks Dec 24 10:56:42 xBox emhttpd: shcmd (261): udevadm settle Dec 24 10:56:43 xBox emhttpd: Opening encrypted volumes... Dec 24 10:56:43 xBox emhttpd: shcmd (262): touch /boot/config/forcesync Dec 24 10:56:43 xBox emhttpd: Mounting disks... Dec 24 10:56:43 xBox emhttpd: mounting /mnt/disk1 Dec 24 10:56:43 xBox emhttpd: shcmd (263): mkdir -p /mnt/disk1 Dec 24 10:56:43 xBox emhttpd: shcmd (264): mount -t xfs -o noatime,nouuid /dev/md1p1 /mnt/disk1 Dec 24 10:56:43 xBox kernel: SGI XFS with ACLs, security attributes, no debug enabled Dec 24 10:56:43 xBox kernel: XFS (md1p1): EXPERIMENTAL Large extent counts feature in use. Use at your own risk! Dec 24 10:56:43 xBox kernel: XFS (md1p1): Mounting V5 Filesystem Dec 24 10:56:43 xBox kernel: XFS (md1p1): Ending clean mount Dec 24 10:56:43 xBox emhttpd: shcmd (265): xfs_growfs /mnt/disk1 Dec 24 10:56:43 xBox root: meta-data=/dev/md1p1 isize=512 agcount=4, agsize=15262348 blks Dec 24 10:56:43 xBox root: = sectsz=512 attr=2, projid32bit=1 Dec 24 10:56:43 xBox root: = crc=1 finobt=1, sparse=1, rmapbt=1 Dec 24 10:56:43 xBox root: = reflink=1 bigtime=1 inobtcount=1 nrext64=1 Dec 24 10:56:43 xBox root: data = bsize=4096 blocks=61049390, imaxpct=25 Dec 24 10:56:43 xBox root: = sunit=0 swidth=0 blks Dec 24 10:56:43 xBox root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Dec 24 10:56:43 xBox root: log =internal log bsize=4096 blocks=29809, version=2 Dec 24 10:56:43 xBox root: = sectsz=512 sunit=0 blks, lazy-count=1 Dec 24 10:56:43 xBox root: realtime =none extsz=4096 blocks=0, rtextents=0 Dec 24 10:56:43 xBox emhttpd: mounting /mnt/zpool Dec 24 10:56:43 xBox emhttpd: shcmd (266): mkdir -p /mnt/zpool Dec 24 10:56:43 xBox emhttpd: shcmd (267): /usr/sbin/zpool import -f -N -o autoexpand=on -d /dev/sdc1 -d /dev/sdh1 -d /dev/sdg1 -d /dev/sdi1 -d /dev/sdj1 -d /dev/sdk1 -d /dev/sdl1 9710758304609155499 zpool Dec 24 10:56:49 xBox emhttpd: shcmd (268): /usr/sbin/zfs set mountpoint=/mnt/zpool zpool Dec 24 10:56:49 xBox emhttpd: shcmd (269): /usr/sbin/zfs set atime=off zpool Dec 24 10:56:49 xBox emhttpd: shcmd (270): /usr/sbin/zfs mount zpool Dec 24 10:56:49 xBox emhttpd: shcmd (271): /usr/sbin/zpool set autotrim=on zpool Dec 24 10:56:50 xBox emhttpd: shcmd (272): /usr/sbin/zfs set compression=off zpool Dec 24 10:56:50 xBox emhttpd: /mnt/zpool root profile: raidz1 Dec 24 10:56:50 xBox emhttpd: /mnt/zpool root groups: 1 Dec 24 10:56:50 xBox emhttpd: /mnt/zpool root width: 7 Dec 24 10:56:50 xBox emhttpd: /usr/sbin/zpool status -PL zpool 2>&1 Dec 24 10:56:50 xBox emhttpd: pool: zpool Dec 24 10:56:50 xBox emhttpd: state: ONLINE Dec 24 10:56:50 xBox emhttpd: status: One or more devices could not be used because the label is missing or Dec 24 10:56:50 xBox emhttpd: #011invalid. Sufficient replicas exist for the pool to continue Dec 24 10:56:50 xBox emhttpd: #011functioning in a degraded state. Dec 24 10:56:50 xBox emhttpd: action: Replace the device using 'zpool replace'. Dec 24 10:56:50 xBox emhttpd: see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J Dec 24 10:56:50 xBox emhttpd: scan: resilvered 81.7M in 00:00:07 with 0 errors on Wed Oct 16 20:44:48 2024 Dec 24 10:56:50 xBox emhttpd: config: Dec 24 10:56:50 xBox emhttpd: #011NAME STATE READ WRITE CKSUM Dec 24 10:56:50 xBox emhttpd: #011zpool ONLINE 0 0 0 Dec 24 10:56:50 xBox emhttpd: #011 raidz1-0 ONLINE 0 0 0 Dec 24 10:56:50 xBox emhttpd: #011 /dev/sdc1 ONLINE 0 0 0 Dec 24 10:56:50 xBox emhttpd: #011 /dev/sdh1 ONLINE 0 0 0 Dec 24 10:56:50 xBox emhttpd: #011 /dev/sdg1 ONLINE 0 0 0 Dec 24 10:56:50 xBox emhttpd: #011 /dev/sdi1 ONLINE 0 0 0 Dec 24 10:56:50 xBox emhttpd: #011 /dev/sdj1 ONLINE 0 0 0
December 24, 20241 yr Author 5 hours ago, JorgeB said: Please post the diagnostics and the output from zpool import diagnostics xbox-diagnostics-20241224-2228.zip
December 24, 20241 yr Author 5 hours ago, JorgeB said: Please post the diagnostics and the output from zpool import i restored the system from backup, restart the system, and tried to start the array again, but encountered a flash read error , and i find this log ,have /usr/sbin/zpool replace -f zpool /dev/sdc1 /dev/sdb1 command Dec 24 22:15:08 xBox emhttpd: shcmd (175): /usr/sbin/zpool replace -f zpool 9828357030404679025 /dev/sdf1 Dec 24 22:15:27 xBox emhttpd: #011 /dev/sdh1 ONLINE 0 0 0 Dec 24 22:15:27 xBox emhttpd: #011 /dev/sdg1 ONLINE 0 0 0 Dec 24 22:15:27 xBox emhttpd: #011 /dev/sdi1 ONLINE 0 0 0 Dec 24 22:15:27 xBox emhttpd: #011 /dev/sdj1 ONLINE 0 0 0 Dec 24 22:15:27 xBox emhttpd: #011 /dev/sdk1 ONLINE 0 0 0 Dec 24 22:15:27 xBox emhttpd: #011 /dev/sdl1 ONLINE 0 0 0 Dec 24 22:15:27 xBox emhttpd: #011cache Dec 24 22:15:27 xBox emhttpd: #011 /dev/sdc1 UNAVAIL 0 0 0 Dec 24 22:15:27 xBox emhttpd: shcmd (176): /sbin/wipefs -af /dev/sdb1 Dec 24 22:15:27 xBox root: /dev/sdb1: 8 bytes were erased at offset 0x00000052 (vfat): 46 41 54 33 32 20 20 20 Dec 24 22:15:27 xBox root: /dev/sdb1: 1 byte was erased at offset 0x00000000 (vfat): eb Dec 24 22:15:27 xBox root: /dev/sdb1: 2 bytes were erased at offset 0x000001fe (vfat): 55 aa Dec 24 22:15:27 xBox emhttpd: writing MBR on disk (sdb) with partition 1 offset 64, erased: 0 Dec 24 22:15:28 xBox emhttpd: re-reading (sdb) partition table Dec 24 22:15:28 xBox emhttpd: error: mkmbr, 2196: Device or resource busy (16): ioctl BLKRRPART: /dev/sdb Dec 24 22:15:28 xBox emhttpd: shcmd (177): udevadm settle Dec 24 22:15:28 xBox emhttpd: shcmd (178): /usr/sbin/zpool replace -f zpool /dev/sdc1 /dev/sdb1 Dec 24 22:15:28 xBox root: cannot open '/dev/sdb1': Device or resource busy Dec 24 22:16:00 xBox root: cannot replace /dev/sdc1 with /dev/sdb1: /dev/sdb1 is busy, or device removal is in progress Dec 24 22:16:00 xBox emhttpd: shcmd (178): exit status: 255 Dec 24 22:16:00 xBox emhttpd: errors: No known data errors Dec 24 22:16:00 xBox emhttpd: shcmd (179): /usr/sbin/zfs mount -a Dec 24 22:16:00 xBox emhttpd: shcmd (180): sync Dec 24 22:16:00 xBox emhttpd: shcmd (181): mkdir /mnt/user0 Dec 24 22:16:00 xBox emhttpd: shcmd (182): /usr/local/bin/shfs /mnt/user0 -disks 2 -o default_permissions,allow_other,noatime
December 24, 20241 yr 8 minutes ago, chinafuxi said: Dec 24 22:15:27 xBox emhttpd: #011 /dev/sdc1 UNAVAIL 0 0 0 This is the problem, remove this device from the pool before starting the array, then re-import the pool with the remaining devices, if you need more help to do that, let us know.
December 24, 20241 yr But I see that the pool is resilvering now, and with errors in multiple devices: NAME STATE READ WRITE CKSUM zpool DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 replacing-0 DEGRADED 0 0 167 9828357030404679025 UNAVAIL 0 0 0 was /dev/sdc1 sdf1 ONLINE 0 0 0 (resilvering) sdh1 ONLINE 185 0 0 (resilvering) sdg1 ONLINE 0 0 167 (resilvering) sdi1 ONLINE 0 0 167 (resilvering) sdj1 ONLINE 0 0 167 (resilvering) sdk1 ONLINE 0 0 167 (resilvering) sdl1 ONLINE 0 0 167 (resilvering) cache sdc1 UNAVAIL 0 0 0 Post a new zpool status
December 24, 20241 yr Author 3 minutes ago, JorgeB said: But I see that the pool is resilvering now, and with errors in multiple devices: NAME STATE READ WRITE CKSUM zpool DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 replacing-0 DEGRADED 0 0 167 9828357030404679025 UNAVAIL 0 0 0 was /dev/sdc1 sdf1 ONLINE 0 0 0 (resilvering) sdh1 ONLINE 185 0 0 (resilvering) sdg1 ONLINE 0 0 167 (resilvering) sdi1 ONLINE 0 0 167 (resilvering) sdj1 ONLINE 0 0 167 (resilvering) sdk1 ONLINE 0 0 167 (resilvering) sdl1 ONLINE 0 0 167 (resilvering) cache sdc1 UNAVAIL 0 0 0 Post a new zpool status i try to remove sdc1 , but return this root@xBox:/flush# zpool remove zpool sdc1 cannot remove sdc1: operation not supported on this type of pool this is new zpoo status NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zpool 25.5T 3.11T 22.4T - - 1% 12% 1.00x DEGRADED - raidz1-0 25.5T 3.11T 22.4T - - 1% 12.2% - DEGRADED replacing-0 - - - - - - - - DEGRADED 9828357030404679025 - - - - - - - - UNAVAIL sdf1 3.64T - - - - - - - ONLINE sdh1 3.64T - - - - - - - ONLINE sdg1 3.64T - - - - - - - ONLINE sdi1 3.64T - - - - - - - ONLINE sdj1 3.64T - - - - - - - ONLINE sdk1 3.64T - - - - - - - ONLINE sdl1 3.64T - - - - - - - ONLINE cache - - - - - - - - - sdc1 - - - - - - - - UNAVAIL ater restart system, I replaced the sdc1 with another disk. now zpool is recovering data
December 24, 20241 yr try: zpool remove zpool /dev/sdc1 If that also doesn't work, wait for the resilver to finish, and post a new status.
December 24, 20241 yr Author 7 minutes ago, JorgeB said: try: zpool remove zpool /dev/sdc1 If that also doesn't work, wait for the resilver to finish, and post a new status. resilver finish, use command remove sdc1 done , i try again to restore flash and reboot system this new zpool status root@xBox:/flush# zpool list -v NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zpool 25.5T 3.11T 22.4T - - 1% 12% 1.00x ONLINE - raidz1-0 25.5T 3.11T 22.4T - - 1% 12.2% - ONLINE sdf1 3.64T - - - - - - - ONLINE sdh1 3.64T - - - - - - - ONLINE sdg1 3.64T - - - - - - - ONLINE sdi1 3.64T - - - - - - - ONLINE sdj1 3.64T - - - - - - - ONLINE sdk1 3.64T - - - - - - - ONLINE sdl1 3.64T - - - - - - - ONLINE root@xBox:/flush#
December 24, 20241 yr Just now, chinafuxi said: i try again to restore flash and reboot system Make sure you reimport the pool using the same order as the zpool output: sdf sdh sdg etc
December 24, 20241 yr Author 12 minutes ago, JorgeB said: Make sure you reimport the pool using the same order as the zpool output: sdf sdh sdg etc start dnoe , error disappear ，so ，why zpool disk error affects flash ? the name of flash should be sdb or sda , i have two flash disk
December 24, 20241 yr It's a bug with v6, it can happen under certain circumstances if a pool has an unassigned device, if you want to use a cache vdev for example, that device must also be assigned to the pool, but this issue should not happen with v7.
December 25, 20241 yr Author 8 hours ago, JorgeB said: It's a bug with v6, it can happen under certain circumstances if a pool has an unassigned device, if you want to use a cache vdev for example, that device must also be assigned to the pool, but this issue should not happen with v7. V7 support set a partition of a hard disk as a cache disk？ cache disk have two partition , i add one partition to zpool cache ,because the cache disk has 1TB, cache cannot be used up partition map like this， sdd1 used for cache sdd 8:48 0 953.9G 0 disk ├─sdd1 8:49 0 500G 0 part └─sdd2 8:50 0 453.9G 0 part in addition, I also found that the drive letter seems to change after restart, which caused zpool cache disk to be mistakenly mount as another disk
December 25, 20241 yr Author 8 hours ago, JorgeB said: It's a bug with v6, it can happen under certain circumstances if a pool has an unassigned device, if you want to use a cache vdev for example, that device must also be assigned to the pool, but this issue should not happen with v7. before i update to version 7bate4 , it seems that only one disk can be configured
December 25, 20241 yr 9 hours ago, chinafuxi said: V7 support set a partition of a hard disk as a cache disk？ Nope, only one partition can be used.
December 27, 20241 yr Author On 12/25/2024 at 6:46 PM, JorgeB said: Nope, only one partition can be used. okey ... tks , i can only use command
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.