Hey all,

 

Just wondering if i've done this wrong....i'm replacing my 2 x Crucial P1 1TB NVMe's with 2 x Samsung 970 Evo Plus 2TB NVMe's in my secondary cache pool, which hosts my Dockers & VM's. The cache pool is a RAID1. 

 

I followed this FAQ, which said just to stop the array, replace the drive and select the new drive and start the array

 

In the UI I can see that the BTRFS operation is running, and I can see some activity on the NVMe drives every now and then. 

 

I can see in the log that its relocating blocks 

Dec 12 15:04:11 TOWER kernel: BTRFS info (device nvme1n1p1): found 5463 extents, stage: move data extents
Dec 12 15:04:22 TOWER kernel: BTRFS info (device nvme1n1p1): found 5463 extents, stage: update data pointers
Dec 12 15:04:29 TOWER kernel: BTRFS info (device nvme1n1p1): relocating block group 1086715854848 flags data|raid1
Dec 12 15:06:18 TOWER kernel: BTRFS info (device nvme1n1p1): found 4546 extents, stage: move data extents
Dec 12 15:06:22 TOWER kernel: BTRFS info (device nvme1n1p1): found 4546 extents, stage: update data pointers
Dec 12 15:06:44 TOWER kernel: BTRFS info (device nvme1n1p1): relocating block group 1085642113024 flags data|raid1
Dec 12 15:08:47 TOWER kernel: BTRFS info (device nvme1n1p1): found 5386 extents, stage: move data extents

 

But its excrutiatingly slow. It's been 2 hours and its only done 70GB  which means its going to take approx a full day for less than 1TB on each of the NVMe drives. It also looks like its draining the secondary drive in the mirror, and putting it on drive 3 rather than to copy it. Is this normal behavior? I thought the new device should replace device 1? 

 

root@TOWER:~# btrfs device usage /mnt/nvme_mirror
missing, ID: 1
   Device size:               0.00B
   Device slack:              0.00B
   Data,RAID1:            788.05GiB
   Metadata,RAID1:          2.00GiB
   System,RAID1:           32.00MiB
   Unallocated:          -790.08GiB

/dev/nvme1n1p1, ID: 2
   Device size:           931.51GiB
   Device slack:              0.00B
   Data,RAID1:            857.05GiB
   Metadata,RAID1:          3.00GiB
   System,RAID1:           32.00MiB
   Unallocated:            71.43GiB

/dev/nvme0n1p1, ID: 3
   Device size:             1.82TiB
   Device slack:              0.00B
   Data,RAID1:             69.00GiB
   Metadata,RAID1:          1.00GiB
   Unallocated:             1.75TiB

 

root@TOWER:~# iostat -m
Device             tps    MB_read/s    MB_wrtn/s    MB_dscd/s    MB_read    MB_wrtn    MB_dscd
loop0             0.00         0.00         0.00         0.00          0          0          0
loop1             0.00         0.00         0.00         0.00          0          0          0
loop2             0.00         0.00         0.00         0.00          0          0          0
loop3             0.00         0.00         0.00         0.00          0          0          0
nvme0n1          16.90         0.00         6.03         0.00          0         60          0
nvme1n1          78.30         0.18         4.53         0.25          1         45          2

 

Please post the diagnostics.