live4soccer7

A few here have been helping me with disk configuration, etc... with my new netapp ds4486. I settled on using the pool feature for the disks and splitting the 48 disks into 2 pools. I had already created a pool with about 12 disks to try and then added in the additional 12 after I had added 10tb of data or so to the pool. Now it is taking FOREVER to "balance" the raid1 with the new disks added. I realized that I actually want to do raid5 to increase the available capacity to me to the max and not be able to lose the entire array with a single disk failure. Ideally, raid0 would be best to give me the most space, but it seems I would lose the entire array if a single disk fails, which I don't want. I don't mind if I lose the data on that disk, but I don't want to lose the entire array/pool of data at once from one disk failure. To the point. Can I cancel the balance and then immediately switch it to raid5? The balance is at 43% done and it has been about 2 days?

I've pretty much confirmed it is literally just the spin-up time of the disks. If I do a FS check on the disk when it is spun down then I will get the errors posted above. The check will fail and the disk will be spun up moments after (automatically). If I do the check again when the disk is spun up then it works perfectly and there are no errors. It is simply a timeout issue while waiting for the disk to spin up. It takes roughly 3s more to spin up than a disk not on the netapp and that difference is the problem.

I'll be mounting the "pool share" and transferring a lot of files from Ubuntu. Would nfs be the best bet for speed? I have plenty of bandwidth on the LAN (10Gig).

well that went "buttery" smooth. HAHAHA. created a pool with 7 drives to try out. It was easy and intuitive in the gui.

Fantastic. I'm liking this a lot. Gotta shut the server down, move a couple things, etc... then get it back up and get a pool created and start transferring some files stat.

Will I be able to run smart tests on the individual disks in the pool after it has been created and the disks have been added? These disks are new to me, so I would like to run tests on them all and wondering if I should do so before creating a pool. I'm thinking the pool is the way to go for me.

Great! I'll have to look at the documentation to see how the pooling works, flexibility, etc.... Ideally, I'd like to add to the pool if necessary or even remove a disk if I have a failure without "destroying" the pool. Different drive sizes would be nice, but I suppose it wouldn't be the end of the world if they had to be the same size as long as I could remove a failed/failing drive and shrink the pool with just a loss of data on the one disk.

I have the unlimited pro. So these don't play into the "Array" limit if I were to create a btrfs pool with the drives?

Unassigned devices do not count toward that number, from my understanding.

Would that count toward the 30/31 drive limit?

I'll give it a shot. It does seem like it those errors don't constantly happen, so something it requesting and causing it. I'll do some more trouble-shooting / digging into this evening. The last batch of errors was over 2 hrs ago.

Could it be a delay in spinning up those devices and taking too long?

I'll be getting a supermicro 24 bay to house the server as well and additional drives, so it will be good to get some experience with both solutions and compare. I'm still getting errors in the logs. Any ideas? Maybe I should post this in the unassigned devices plugin thread?? Aug 23 08:16:20 Tower kernel: blk_update_request: I/O error, dev sdbo, sector 1189481720 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 23 08:16:21 Tower kernel: blk_update_request: I/O error, dev sdbo, sector 1189481720 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 23 08:16:21 Tower kernel: Buffer I/O error on dev sdbo1, logical block 148684959, async page read Aug 23 08:16:21 Tower kernel: blk_update_request: I/O error, dev sdbo, sector 1189481720 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 23 08:16:21 Tower kernel: Buffer I/O error on dev sdbo1, logical block 148684959, async page read Aug 23 08:16:21 Tower kernel: blk_update_request: I/O error, dev sdbo, sector 1189481720 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 23 08:16:21 Tower kernel: Buffer I/O error on dev sdbo1, logical block 148684959, async page read Aug 23 08:16:21 Tower kernel: blk_update_request: I/O error, dev sdbo, sector 1189481720 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 23 08:16:21 Tower kernel: Buffer I/O error on dev sdbo1, logical block 148684959, async page read Aug 23 08:16:21 Tower kernel: blk_update_request: I/O error, dev sdbo, sector 1189481720 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 23 08:16:21 Tower kernel: Buffer I/O error on dev sdbo1, logical block 148684959, async page read Aug 23 08:16:21 Tower kernel: blk_update_request: I/O error, dev sdbo, sector 1189481720 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 23 08:16:21 Tower kernel: Buffer I/O error on dev sdbo1, logical block 148684959, async page read Aug 23 08:16:22 Tower kernel: blk_update_request: I/O error, dev sdbo, sector 1189481720 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 23 08:16:22 Tower kernel: Buffer I/O error on dev sdbo1, logical block 148684959, async page read Aug 23 08:16:22 Tower kernel: blk_update_request: I/O error, dev sdbo, sector 1189481720 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 23 08:16:22 Tower kernel: Buffer I/O error on dev sdbo1, logical block 148684959, async page read Aug 23 08:16:22 Tower kernel: blk_update_request: I/O error, dev sdbo, sector 1189481720 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 23 08:16:22 Tower kernel: Buffer I/O error on dev sdbo1, logical block 148684959, async page read Aug 23 08:16:22 Tower kernel: Buffer I/O error on dev sdbo1, logical block 148684959, async page read Aug 23 08:16:44 Tower kernel: blk_update_request: I/O error, dev sdbq, sector 7814035328 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 23 08:16:46 Tower kernel: blk_update_request: I/O error, dev sdbq, sector 7814035328 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 23 08:16:46 Tower kernel: Buffer I/O error on dev sdbq1, logical block 976754160, async page read Aug 23 08:16:54 Tower kernel: blk_update_request: I/O error, dev sdbp, sector 7814035328 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 23 08:16:55 Tower kernel: blk_update_request: I/O error, dev sdbp, sector 7814035328 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 23 08:16:55 Tower kernel: Buffer I/O error on dev sdbp1, logical block 976754160, async page read

The data isn't terribly valuable, so loss is acceptable if there is an immediate failure. The drives will be accessed all the time.

Initially I formatted them all as NTFS as I do have windows machines, but I do also have linux machines as well. I was thinking maybe XFS may be better. I'm basically storing data on them and then not much else after that will be written to them. Figured I would get some input before filling them all and having it essentially be too late to change.

Yeah. I spent most of yesterday chasing it down. I'm sure this will be of some help to others that have the same netapp jbod/disk shelf.

With the changes I made that I posted up, it is working. I have ran several short tests and currently waiting to see if an extended test works. It has been about 8-9 hours on the extended test and it still says 10%. The temps are actually populating in the gui as well. I suppose it just had to poll them and I'm not sure what I have that interval set at currently. Device model:HITACHI HUS724040ALE640 Serial number:PAJEK98T LU WWN device id:5 000cca 22be258c5 Firmware version:MJAONS04 User capacity:4,000,787,030,016 bytes [4.00 TB] Sector sizes:512 bytes logical, 4096 bytes physical Rotation rate:7200 rpm Form factor:3.5 inches Device:Not in smartctl database [for details use: -P showall] ATA version:ATA8-ACS T13/1699-D revision 4 SATA version:SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local time:Mon Aug 23 08:42:39 2021 PDT SMART support:Available - device has SMART capability. SMART support:Enabled SMART overall-health:Passed 1Raw read error rate0x000b100100016Pre-failAlwaysNever0 2Throughput performance0x0005100100054Pre-failOfflineNever0 3Spin up time0x0007100100024Pre-failAlwaysNever495 4Start stop count0x0012100100000Old ageAlwaysNever18 5Reallocated sector count0x0033100100005Pre-failAlwaysNever0 7Seek error rate0x000b100100067Pre-failAlwaysNever0 8Seek time performance0x0005100100020Pre-failOfflineNever0 9Power on hours0x0012092092000Old ageAlwaysNever58421 (6y, 7m, 30d, 5h) 10Spin retry count0x0013100100060Pre-failAlwaysNever0 12Power cycle count0x0032100100000Old ageAlwaysNever14 192Power-off retract count0x0032099099000Old ageAlwaysNever2346 193Load cycle count0x0012099099000Old ageAlwaysNever2346 194Temperature celsius0x0002181181000Old ageAlwaysNever33 (min/max 22/38) 196Reallocated event count0x0032100100000Old ageAlwaysNever0 197Current pending sector0x0022100100000Old ageAlwaysNever0 198Offline uncorrectable0x0008100100000Old ageOfflineNever0 199UDMA CRC error count0x000a200200000Old ageAlwaysNever0

They are all SATA drives.

Another update. Doing some more digging, specifically on netapp hgst drives. I found that on the ds4486, you have to run a specific parameter to get the smart test info. smartctl -a -d sat /dev/your_disk This worked when I ran the test on a disk from the console. Great information. I see there is an option in the individual disks in unassigned devices when you click on the disks. Smart Controller Type --> SAT Then when you choose SAT it provides two additional fields, which I am unsure of what to put as this is, again, new to me. If I leave auto blank and choose 12, it seems to show the smart data. What are the 12 and 16 numbers representing? Any help on this one? Edit: the temps are reported in the smart test, but not on the unraid dash/gui. I believe this is likely the issue of smartmontools and link I provided earlier. I think when the next release comes out it will fix this as it seems they have smartmontools 7.2 for the next release.

I found the distro information in the install and it should be smartmontools 7.1. What I was seeing on the forums must have been either a beta or RC version.

So.... looking at another post of someone posting up a smart report, it seems like 6.9.2 should be smartclt 7.2. Can someone check this on their updated install? If that is indeed the case, how can I go about getting mine updated?

I believe I have this same issue. I wonder why unraid/limetech has not updated the smartclt/smartmontools from 7.1 when we are on 6.9.2 now.

So, I found this thread and it seems like it could be an issue in smartmontools 7.1 and HBA cards or just in certain scenarios. It appears upgrading to 7.2 fixed the issue for this person. Can I update smartmontools in unraid without screwing everything up? I'm on unraid 6.9.2. If I can, how can I go about doing that properly? https://www.smartmontools.org/ticket/1331?cversion=0&cnum_hist=4

To add some additional information. smartctl -i /dev/sdad on a drive and got the below: smartctl -i /dev/sdad smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.10.28-Unraid] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: HITACHI Product: HUS724040ALE64DB Revision: NA01 Compliance: SPC-3 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Logical block size: 512 bytes Physical block size: 4096 bytes Logical Unit id: 0x5000cca22be0f3aa Serial number: PAJAH58T Device type: disk Transport protocol: SAS (SPL-3) Local Time is: Sun Aug 22 14:10:38 2021 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled Temperature Warning: Enabled I then ran ran smartctl -a /dev/sdad and received the following: smartctl -a /dev/sdad smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.10.28-Unraid] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: HITACHI Product: HUS724040ALE64DB Revision: NA01 Compliance: SPC-3 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Logical block size: 512 bytes Physical block size: 4096 bytes Logical Unit id: 0x5000cca22be0f3aa Serial number: PAJAH58T Device type: disk Transport protocol: SAS (SPL-3) Local Time is: Sun Aug 22 14:11:55 2021 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled Temperature Warning: Enabled === START OF READ SMART DATA SECTION === SMART Health Status: OK Current Drive Temperature: 0 C Drive Trip Temperature: 0 C Read defect list: asked for grown list but didn't get it Error Counter logging not supported Device does not support Self Test logging I would think that firmware on the hard drive or something with the HBA settings (not sure how to check those).

I'm using an LSI 9206-16e HBA to connect to the netapp. There are 48 4TB disks, all formatted to 512b sectors and recognized by unassigned devices. The LSI card has been updated to the latest P20 firmware. I was unable to get lsiutility to work, so I couldn't further check the configuration on the card. If there is a way to do this within unraid console? That would be pretty epic and I would definitely like to know. I have had random issues with the drive mount button being greyed out, not being able to unmount a drive (its share doesn't show in exportfs -v), so I don't think anything could be attached to its filesystem. I have even lost the drive name_serial and then it will randomly show back up again. I am unable to do a smart test or see temps of the drives I currently have most drives formatted to NTFS, but I am thinking maybe XFS would be better. They are just used for individual storage on each drive and backup isn't important. I have attached a system log error output from unraid. A lot of this with the "jbod" is new to me. Any direction here would be great. Aug 21 18:19:10 Tower kernel: blk_update_request: I/O error, dev sdv, sector 32768 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0 Aug 21 18:19:12 Tower kernel: blk_update_request: I/O error, dev sdv, sector 32768 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 18:19:12 Tower kernel: Buffer I/O error on dev sdv2, logical block 0, async page read Aug 21 18:19:12 Tower kernel: blk_update_request: I/O error, dev sdv, sector 7814033280 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 18:19:12 Tower kernel: blk_update_request: I/O error, dev sdv, sector 7814033280 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 18:19:12 Tower kernel: Buffer I/O error on dev sdv2, logical block 976750064, async page read Aug 21 18:20:36 Tower kernel: blk_update_request: I/O error, dev sdw, sector 32768 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0 Aug 21 18:20:37 Tower kernel: blk_update_request: I/O error, dev sdw, sector 32768 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 18:20:37 Tower kernel: Buffer I/O error on dev sdw2, logical block 0, async page read Aug 21 18:20:37 Tower kernel: blk_update_request: I/O error, dev sdw, sector 7814033280 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 18:20:37 Tower kernel: blk_update_request: I/O error, dev sdw, sector 7814033280 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 18:20:37 Tower kernel: Buffer I/O error on dev sdw2, logical block 976750064, async page read Aug 21 18:20:51 Tower kernel: blk_update_request: I/O error, dev sdx, sector 32768 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0 Aug 21 18:20:52 Tower kernel: blk_update_request: I/O error, dev sdx, sector 32768 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 18:20:52 Tower kernel: Buffer I/O error on dev sdx2, logical block 0, async page read Aug 21 18:20:52 Tower kernel: blk_update_request: I/O error, dev sdx, sector 7814033280 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 18:20:52 Tower kernel: blk_update_request: I/O error, dev sdx, sector 7814033280 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 18:20:52 Tower kernel: Buffer I/O error on dev sdx2, logical block 976750064, async page read Aug 21 18:21:16 Tower kernel: blk_update_request: I/O error, dev sdy, sector 2048 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0 Aug 21 18:21:17 Tower kernel: blk_update_request: I/O error, dev sdy, sector 2048 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 18:21:17 Tower kernel: Buffer I/O error on dev sdy1, logical block 0, async page read Aug 21 18:21:17 Tower kernel: blk_update_request: I/O error, dev sdy, sector 7814035328 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 18:21:17 Tower kernel: blk_update_request: I/O error, dev sdy, sector 7814035328 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 18:21:17 Tower kernel: Buffer I/O error on dev sdy1, logical block 976754160, async page read Aug 21 19:11:12 Tower kernel: blk_update_request: I/O error, dev sdz, sector 2048 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0 Aug 21 19:11:12 Tower kernel: blk_update_request: I/O error, dev sdz, sector 2048 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 19:11:12 Tower kernel: Buffer I/O error on dev sdz1, logical block 0, async page read Aug 21 19:11:12 Tower kernel: blk_update_request: I/O error, dev sdz, sector 7814035328 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 19:11:12 Tower kernel: blk_update_request: I/O error, dev sdz, sector 7814035328 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 19:11:12 Tower kernel: Buffer I/O error on dev sdz1, logical block 976754160, async page read Aug 21 19:18:50 Tower kernel: blk_update_request: I/O error, dev sdae, sector 7814036992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 19:18:50 Tower kernel: blk_update_request: I/O error, dev sdae, sector 7814036992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 19:18:50 Tower kernel: Buffer I/O error on dev sdae, logical block 976754624, async page read Aug 21 19:18:50 Tower kernel: blk_update_request: I/O error, dev sdag, sector 7814036992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 19:18:50 Tower kernel: blk_update_request: I/O error, dev sdag, sector 7814036992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 19:18:50 Tower kernel: Buffer I/O error on dev sdag, logical block 976754624, async page read Aug 21 19:18:50 Tower kernel: blk_update_request: I/O error, dev sdaf, sector 7814036992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 19:18:50 Tower kernel: blk_update_request: I/O error, dev sdau, sector 7814036992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 19:18:50 Tower kernel: blk_update_request: I/O error, dev sdal, sector 7814036992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 19:18:50 Tower kernel: blk_update_request: I/O error, dev sdaf, sector 7814036992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 19:18:50 Tower kernel: Buffer I/O error on dev sdaf, logical block 976754624, async page read Aug 21 19:18:50 Tower kernel: blk_update_request: I/O error, dev sdau, sector 7814036992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 19:18:50 Tower kernel: Buffer I/O error on dev sdau, logical block 976754624, async page read Aug 21 19:18:50 Tower kernel: blk_update_request: I/O error, dev sdal, sector 7814036992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 19:18:50 Tower kernel: Buffer I/O error on dev sdal, logical block 976754624, async page read Aug 21 19:18:50 Tower kernel: Buffer I/O error on dev sdat, logical block 976754624, async page read Aug 21 19:18:50 Tower kernel: Buffer I/O error on dev sdbj, logical block 976754624, async page read Aug 21 19:18:50 Tower kernel: Buffer I/O error on dev sdbd, logical block 976754624, async page read Aug 21 19:18:50 Tower kernel: Buffer I/O error on dev sdao, logical block 976754624, async page read Aug 21 19:18:50 Tower kernel: Buffer I/O error on dev sdbg, logical block 976754624, async page read Aug 21 19:18:57 Tower kernel: blk_update_request: I/O error, dev sdaz, sector 7814036992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 19:18:57 Tower kernel: blk_update_request: I/O error, dev sdaz, sector 7814036992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 19:18:57 Tower kernel: Buffer I/O error on dev sdaz, logical block 976754624, async page read Aug 21 19:18:58 Tower kernel: blk_update_request: I/O error, dev sdad, sector 7814036992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 19:18:58 Tower kernel: blk_update_request: I/O error, dev sdad, sector 7814036992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 19:18:58 Tower kernel: Buffer I/O error on dev sdad, logical block 976754624, async page read Aug 21 19:18:58 Tower kernel: blk_update_request: I/O error, dev sdbb, sector 7814036992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 19:18:58 Tower kernel: blk_update_request: I/O error, dev sdam, sector 7814036992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 19:18:58 Tower kernel: blk_update_request: I/O error, dev sdbb, sector 7814036992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 19:18:58 Tower kernel: Buffer I/O error on dev sdbb, logical block 976754624, async page read Aug 21 19:18:58 Tower kernel: blk_update_request: I/O error, dev sdam, sector 7814036992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 19:18:58 Tower kernel: Buffer I/O error on dev sdam, logical block 976754624, async page read Aug 21 19:18:58 Tower kernel: blk_update_request: I/O error, dev sdbk, sector 7814036992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 19:18:58 Tower kernel: blk_update_request: I/O error, dev sdbe, sector 7814036992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 19:18:58 Tower kernel: Buffer I/O error on dev sdbk, logical block 976754624, async page read Aug 21 19:18:58 Tower kernel: Buffer I/O error on dev sdbe, logical block 976754624, async page read Aug 21 19:18:58 Tower kernel: Buffer I/O error on dev sdan, logical block 976754624, async page read Aug 21 19:18:58 Tower kernel: Buffer I/O error on dev sdaq, logical block 976754624, async page read Aug 21 19:18:58 Tower kernel: Buffer I/O error on dev sdbm, logical block 976754624, async page read Aug 21 19:18:58 Tower kernel: Buffer I/O error on dev sdbf, logical block 976754624, async page read Aug 21 20:22:29 Tower kernel: blk_update_request: I/O error, dev sdx, sector 328210992 op 0x0:(READ) flags 0x80700 phys_seg 32 prio class 0 Aug 21 20:22:30 Tower ntfs-3g[31293]: ntfs_attr_pread error reading '/plots/plot-k32-2021-08-06-06-39-fd0b4152cc6f1ed9ba0a099b9cf4be1f9d907b6922f77fe040f5e6aee3dc70cc.plot' at offset 55803117568: 131072 <> -1: Input/output error Aug 21 20:22:30 Tower kernel: blk_update_request: I/O error, dev sdx, sector 328210736 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 20:22:30 Tower kernel: Buffer I/O error on dev sdx2, logical block 41022246, async page read Aug 21 20:22:30 Tower kernel: blk_update_request: I/O error, dev sdx, sector 328210480 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 20:22:30 Tower ntfs-3g[31293]: ntfs_attr_pread error reading '/plots/plot-k32-2021-08-06-06-39-fd0b4152cc6f1ed9ba0a099b9cf4be1f9d907b6922f77fe040f5e6aee3dc70cc.plot' at offset 55802986496: 4096 <> -1: Input/output error Aug 21 20:22:30 Tower kernel: blk_update_request: I/O error, dev sdx, sector 328210480 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 20:22:30 Tower kernel: Buffer I/O error on dev sdx2, logical block 41022214, async page read Aug 21 20:22:30 Tower ntfs-3g[31293]: ntfs_attr_pread error reading '/plots/plot-k32-2021-08-06-06-39-fd0b4152cc6f1ed9ba0a099b9cf4be1f9d907b6922f77fe040f5e6aee3dc70cc.plot' at offset 55802986496: 4096 <> -1: Input/output error Aug 21 20:22:30 Tower kernel: blk_update_request: I/O error, dev sdx, sector 328210480 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 20:22:30 Tower kernel: Buffer I/O error on dev sdx2, logical block 41022214, async page read Aug 21 20:22:30 Tower ntfs-3g[31293]: ntfs_attr_pread error reading '/plots/plot-k32-2021-08-06-06-39-fd0b4152cc6f1ed9ba0a099b9cf4be1f9d907b6922f77fe040f5e6aee3dc70cc.plot' at offset 55802986496: 4096 <> -1: Input/output error Aug 21 20:22:30 Tower kernel: blk_update_request: I/O error, dev sdx, sector 328210480 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 20:22:30 Tower kernel: Buffer I/O error on dev sdx2, logical block 41022214, async page read Aug 21 20:53:11 Tower kernel: blk_update_request: I/O error, dev sdbq, sector 3072 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 20:53:13 Tower kernel: blk_update_request: I/O error, dev sdbq, sector 3072 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 20:53:13 Tower kernel: Buffer I/O error on dev sdbq1, logical block 128, async page read Aug 21 20:54:21 Tower kernel: blk_update_request: I/O error, dev sdbp, sector 3072 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 20:54:23 Tower kernel: blk_update_request: I/O error, dev sdbp, sector 3072 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 20:54:23 Tower kernel: Buffer I/O error on dev sdbp1, logical block 128, async page read Aug 21 22:28:01 Tower kernel: blk_update_request: I/O error, dev sdy, sector 976756264 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 22:28:01 Tower unassigned.devices: Error: shell_exec(/bin/df '/mnt/disks/netapp_5' --output=size,used,avail | /bin/grep -v '1K-blocks' 2>/dev/null) took longer than 2s! Aug 21 22:28:02 Tower kernel: blk_update_request: I/O error, dev sdy, sector 976756264 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 22:28:02 Tower kernel: Buffer I/O error on dev sdy1, logical block 122094277, async page read Aug 21 22:29:38 Tower kernel: blk_update_request: I/O error, dev sdw, sector 1069489368 op 0x0:(READ) flags 0x80700 phys_seg 32 prio class 0 Aug 21 22:29:38 Tower unassigned.devices: Error: shell_exec(/bin/df '/mnt/disks/netapp_2' --output=size,used,avail | /bin/grep -v '1K-blocks' 2>/dev/null) took longer than 2s! Aug 21 22:29:39 Tower ntfs-3g[2313]: ntfs_attr_pread error reading '/plots/plot-k32-2021-07-11-22-04-c57332a1f059d6d083fd731ba7acdb73fb8518907058d528454a16605b7acb19.plot' at offset 0: 131072 <> -1: Input/output error Aug 21 22:29:39 Tower kernel: blk_update_request: I/O error, dev sdw, sector 1069489368 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 22:29:39 Tower kernel: Buffer I/O error on dev sdw2, logical block 133682075, async page read Aug 21 22:29:39 Tower ntfs-3g[2313]: ntfs_attr_pread error reading '/plots/plot-k32-2021-07-11-22-04-c57332a1f059d6d083fd731ba7acdb73fb8518907058d528454a16605b7acb19.plot' at offset 0: 4096 <> -1: Input/output error Aug 21 22:29:39 Tower kernel: blk_update_request: I/O error, dev sdw, sector 1069489368 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 22:29:39 Tower kernel: Buffer I/O error on dev sdw2, logical block 133682075, async page read Aug 21 22:53:08 Tower kernel: blk_update_request: I/O error, dev sdbq, sector 3072 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 22:53:08 Tower kernel: blk_update_request: I/O error, dev sdbq, sector 3072 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 22:53:08 Tower kernel: Buffer I/O error on dev sdbq1, logical block 128, async page read Aug 21 22:53:16 Tower kernel: blk_update_request: I/O error, dev sdbp, sector 3072 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 22:53:18 Tower kernel: blk_update_request: I/O error, dev sdbp, sector 3072 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 22:53:18 Tower kernel: Buffer I/O error on dev sdbp1, logical block 128, async page read Aug 21 23:41:33 Tower kernel: blk_update_request: I/O error, dev sdac, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0 Aug 21 23:41:33 Tower kernel: blk_update_request: I/O error, dev sdac, sector 7814035328 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 21 23:41:34 Tower kernel: blk_update_request: I/O error, dev sdac, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 23:41:34 Tower kernel: Buffer I/O error on dev sdac, logical block 0, async page read Aug 21 23:41:34 Tower kernel: blk_update_request: I/O error, dev sdac, sector 7814035328 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 23:41:34 Tower kernel: Buffer I/O error on dev sdac1, logical block 976754160, async page read Aug 21 23:41:34 Tower kernel: blk_update_request: I/O error, dev sdac, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 23:41:34 Tower kernel: Buffer I/O error on dev sdac, logical block 0, async page read Aug 21 23:41:34 Tower kernel: blk_update_request: I/O error, dev sdac, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 23:41:34 Tower kernel: Buffer I/O error on dev sdac, logical block 0, async page read Aug 21 23:41:34 Tower kernel: blk_update_request: I/O error, dev sdac, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 23:41:34 Tower kernel: Buffer I/O error on dev sdac, logical block 0, async page read Aug 21 23:41:34 Tower kernel: blk_update_request: I/O error, dev sdac, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 23:41:34 Tower kernel: Buffer I/O error on dev sdac, logical block 0, async page read Aug 21 23:41:34 Tower kernel: blk_update_request: I/O error, dev sdac, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 23:41:34 Tower kernel: Buffer I/O error on dev sdac, logical block 0, async page read Aug 21 23:41:34 Tower kernel: blk_update_request: I/O error, dev sdac, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 21 23:41:34 Tower kernel: Buffer I/O error on dev sdac, logical block 0, async page read Aug 21 23:41:34 Tower kernel: Buffer I/O error on dev sdac, logical block 0, async page read Aug 21 23:41:34 Tower kernel: Buffer I/O error on dev sdac, logical block 0, async page read Aug 22 05:44:41 Tower unassigned.devices: Error: shell_exec(/bin/df '/mnt/disks/d_240' --output=size,used,avail | /bin/grep -v '1K-blocks' 2>/dev/null) took longer than 2s! Aug 22 09:20:21 Tower ntfs-3g[27765]: ntfs_attr_mst_pwrite: written=-1: Input/output error Aug 22 09:20:21 Tower ntfs-3g[27765]: Error writing $Mft record(s): Input/output error Aug 22 09:20:21 Tower kernel: blk_update_request: I/O error, dev sdy, sector 2208 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 09:20:21 Tower kernel: blk_update_request: I/O error, dev sdy, sector 977028288 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0 Aug 22 09:20:21 Tower kernel: Buffer I/O error on dev sdy1, logical block 122128280, lost async page write Aug 22 09:20:21 Tower kernel: blk_update_request: I/O error, dev sdy, sector 2208 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 09:20:21 Tower ntfs-3g[27765]: ntfs_attr_mst_pwrite: written=-1: Input/output error Aug 22 09:20:21 Tower ntfs-3g[27765]: Error writing $Mft record(s): Input/output error Aug 22 09:20:36 Tower kernel: blk_update_request: I/O error, dev sdw, sector 219283032 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 0 Aug 22 09:20:37 Tower kernel: blk_update_request: I/O error, dev sdw, sector 219283032 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 09:20:37 Tower kernel: Buffer I/O error on dev sdw2, logical block 27406283, async page read Aug 22 10:06:33 Tower kernel: blk_update_request: I/O error, dev sdbo, sector 976756264 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0 Aug 22 10:06:34 Tower kernel: blk_update_request: I/O error, dev sdbo, sector 976756264 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 10:06:34 Tower kernel: Buffer I/O error on dev sdbo1, logical block 122094277, async page read Aug 22 10:06:34 Tower kernel: blk_update_request: I/O error, dev sdbo, sector 976756264 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 10:06:34 Tower kernel: Buffer I/O error on dev sdbo1, logical block 122094277, async page read Aug 22 10:06:34 Tower kernel: blk_update_request: I/O error, dev sdbo, sector 976756264 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 10:06:34 Tower kernel: Buffer I/O error on dev sdbo1, logical block 122094277, async page read Aug 22 10:07:08 Tower kernel: blk_update_request: I/O error, dev sdaa, sector 2048 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0 Aug 22 10:07:09 Tower kernel: blk_update_request: I/O error, dev sdaa, sector 2048 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 10:07:09 Tower kernel: Buffer I/O error on dev sdaa1, logical block 0, async page read Aug 22 10:07:09 Tower kernel: blk_update_request: I/O error, dev sdaa, sector 7814035328 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 22 10:07:09 Tower kernel: blk_update_request: I/O error, dev sdaa, sector 7814035328 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 10:07:09 Tower kernel: Buffer I/O error on dev sdaa1, logical block 976754160, async page read Aug 22 10:08:15 Tower kernel: blk_update_request: I/O error, dev sdv, sector 33056 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 22 10:08:17 Tower kernel: blk_update_request: I/O error, dev sdv, sector 33056 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 10:08:17 Tower kernel: Buffer I/O error on dev sdv2, logical block 36, async page read Aug 22 10:08:17 Tower kernel: blk_update_request: I/O error, dev sdv, sector 33056 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 10:08:17 Tower kernel: Buffer I/O error on dev sdv2, logical block 36, async page read Aug 22 10:08:17 Tower kernel: blk_update_request: I/O error, dev sdv, sector 33056 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 10:08:17 Tower kernel: Buffer I/O error on dev sdv2, logical block 36, async page read Aug 22 10:08:17 Tower kernel: blk_update_request: I/O error, dev sdv, sector 33056 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 10:08:17 Tower kernel: Buffer I/O error on dev sdv2, logical block 36, async page read Aug 22 10:09:14 Tower kernel: blk_update_request: I/O error, dev sdz, sector 976756264 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 22 10:09:14 Tower kernel: blk_update_request: I/O error, dev sdz, sector 976756264 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 10:09:14 Tower kernel: Buffer I/O error on dev sdz1, logical block 122094277, async page read Aug 22 10:09:14 Tower kernel: blk_update_request: I/O error, dev sdz, sector 976756264 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 10:09:14 Tower kernel: Buffer I/O error on dev sdz1, logical block 122094277, async page read Aug 22 10:09:14 Tower kernel: blk_update_request: I/O error, dev sdz, sector 976756264 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 10:09:14 Tower kernel: Buffer I/O error on dev sdz1, logical block 122094277, async page read Aug 22 10:09:14 Tower kernel: blk_update_request: I/O error, dev sdz, sector 976756264 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 10:09:14 Tower kernel: Buffer I/O error on dev sdz1, logical block 122094277, async page read Aug 22 10:37:43 Tower kernel: blk_update_request: I/O error, dev sdbq, sector 7814035328 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 22 10:37:44 Tower kernel: blk_update_request: I/O error, dev sdbq, sector 7814035328 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 10:37:44 Tower kernel: Buffer I/O error on dev sdbq1, logical block 976754160, async page read Aug 22 10:37:52 Tower kernel: blk_update_request: I/O error, dev sdbp, sector 7814035328 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 22 10:37:54 Tower kernel: blk_update_request: I/O error, dev sdbp, sector 7814035328 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 10:37:54 Tower kernel: Buffer I/O error on dev sdbp1, logical block 976754160, async page read Aug 22 11:09:05 Tower kernel: blk_update_request: I/O error, dev sdaa, sector 7814035328 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Aug 22 11:09:05 Tower kernel: blk_update_request: I/O error, dev sdaa, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0 Aug 22 11:09:06 Tower kernel: blk_update_request: I/O error, dev sdaa, sector 7814035328 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 11:09:06 Tower kernel: Buffer I/O error on dev sdaa1, logical block 976754160, async page read Aug 22 11:09:06 Tower kernel: blk_update_request: I/O error, dev sdaa, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 11:09:06 Tower kernel: Buffer I/O error on dev sdaa, logical block 0, async page read Aug 22 11:09:06 Tower kernel: blk_update_request: I/O error, dev sdaa, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 11:09:06 Tower kernel: Buffer I/O error on dev sdaa, logical block 0, async page read Aug 22 11:09:06 Tower kernel: blk_update_request: I/O error, dev sdaa, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 11:09:06 Tower kernel: Buffer I/O error on dev sdaa, logical block 0, async page read Aug 22 11:09:06 Tower kernel: blk_update_request: I/O error, dev sdaa, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 11:09:06 Tower kernel: Buffer I/O error on dev sdaa, logical block 0, async page read Aug 22 11:09:06 Tower kernel: blk_update_request: I/O error, dev sdaa, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 11:09:06 Tower kernel: Buffer I/O error on dev sdaa, logical block 0, async page read Aug 22 11:09:06 Tower kernel: blk_update_request: I/O error, dev sdaa, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 11:09:06 Tower kernel: Buffer I/O error on dev sdaa, logical block 0, async page read Aug 22 11:09:06 Tower kernel: blk_update_request: I/O error, dev sdaa, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Aug 22 11:09:06 Tower kernel: Buffer I/O error on dev sdaa, logical block 0, async page read Aug 22 11:09:06 Tower kernel: Buffer I/O error on dev sdaa, logical block 0, async page read Aug 22 11:09:06 Tower kernel: Buffer I/O error on dev sdaa, logical block 0, async page read