Jump to content

Crashing after upgrade 6.8.0+


Recommended Posts

I upgraded to 6.8.0 then later 6.8.1 and i started to experience random server crashes, doesn't appear to be linked to anything specific at least not anything I can link it too, last night my dockers crashed and it appeared my cache drive was corrupted which i suspect is just from the server crashes, i cleared the data off it then formatted the drive and everything started working as expected.

 

Any help would be appreciated,  Thanks!

 

Errors for the cache drive

Jan 25 23:31:25 Atlas kernel: print_req_error: I/O error, dev loop2, sector 9779904
Jan 25 23:31:25 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2297, flush 0, corrupt 0, gen 0
Jan 25 23:31:25 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:25 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4298624
Jan 25 23:31:25 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4298648
Jan 25 23:31:25 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4482352
Jan 25 23:31:25 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:25 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2298, flush 0, corrupt 0, gen 0
Jan 25 23:31:27 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:27 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2299, flush 0, corrupt 0, gen 0
Jan 25 23:31:29 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:29 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2300, flush 0, corrupt 0, gen 0
Jan 25 23:31:31 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:31 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2301, flush 0, corrupt 0, gen 0
Jan 25 23:31:34 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:34 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2302, flush 0, corrupt 0, gen 0
Jan 25 23:31:36 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:36 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2303, flush 0, corrupt 0, gen 0
Jan 25 23:31:38 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:38 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2304, flush 0, corrupt 0, gen 0
Jan 25 23:31:40 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:40 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2305, flush 0, corrupt 0, gen 0
Jan 25 23:31:42 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:42 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2306, flush 0, corrupt 0, gen 0
Jan 25 23:31:44 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:44 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2307, flush 0, corrupt 0, gen 0
Jan 25 23:31:46 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:46 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2308, flush 0, corrupt 0, gen 0
Jan 25 23:31:48 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:48 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2309, flush 0, corrupt 0, gen 0
Jan 25 23:31:51 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:51 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2310, flush 0, corrupt 0, gen 0
Jan 25 23:31:53 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:53 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2311, flush 0, corrupt 0, gen 0
Jan 25 23:31:55 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:55 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2312, flush 0, corrupt 0, gen 0
Jan 25 23:31:57 Atlas kernel: print_req_error: I/O error, dev loop2, sector 4275976
Jan 25 23:31:57 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2313, flush 0, corrupt 0, gen 0
Jan 25 23:34:29 Atlas kernel: print_req_error: I/O error, dev loop2, sector 6636176
Jan 25 23:34:29 Atlas kernel: print_req_error: I/O error, dev loop2, sector 6636176
Jan 25 23:34:29 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2314, flush 0, corrupt 0, gen 0
Jan 25 23:34:29 Atlas kernel: print_req_error: I/O error, dev loop2, sector 6636176
Jan 25 23:34:29 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2315, flush 0, corrupt 0, gen 0
Jan 25 23:34:29 Atlas kernel: print_req_error: I/O error, dev loop2, sector 6631080
Jan 25 23:34:29 Atlas kernel: print_req_error: I/O error, dev loop2, sector 6631096
Jan 25 23:34:29 Atlas kernel: print_req_error: I/O error, dev loop2, sector 6652288
Jan 25 23:34:29 Atlas kernel: print_req_error: I/O error, dev loop2, sector 12139008
Jan 25 23:34:29 Atlas kernel: print_req_error: I/O error, dev loop2, sector 6631080
Jan 25 23:34:29 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2316, flush 0, corrupt 0, gen 0
Jan 25 23:34:31 Atlas kernel: print_req_error: I/O error, dev loop2, sector 6631080
Jan 25 23:34:31 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2317, flush 0, corrupt 0, gen 0
Jan 25 23:35:00 Atlas kernel: print_req_error: I/O error, dev loop2, sector 25685120
Jan 25 23:35:00 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2318, flush 0, corrupt 0, gen 0
Jan 25 23:35:00 Atlas kernel: print_req_error: I/O error, dev loop2, sector 25685120
Jan 25 23:35:00 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2319, flush 0, corrupt 0, gen 0
Jan 25 23:35:00 Atlas kernel: print_req_error: I/O error, dev loop2, sector 25685120
Jan 25 23:35:00 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2320, flush 0, corrupt 0, gen 0
Jan 25 23:35:00 Atlas kernel: print_req_error: I/O error, dev loop2, sector 25685120
Jan 25 23:35:00 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2321, flush 0, corrupt 0, gen 0
Jan 25 23:35:00 Atlas kernel: print_req_error: I/O error, dev loop2, sector 25685120
Jan 25 23:35:00 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2322, flush 0, corrupt 0, gen 0
Jan 25 23:35:00 Atlas kernel: print_req_error: I/O error, dev loop2, sector 25685120
Jan 25 23:35:00 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2323, flush 0, corrupt 0, gen 0
Jan 25 23:35:00 Atlas kernel: print_req_error: I/O error, dev loop2, sector 25685120
Jan 25 23:35:00 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2324, flush 0, corrupt 0, gen 0
Jan 25 23:35:00 Atlas kernel: print_req_error: I/O error, dev loop2, sector 25685120
Jan 25 23:35:00 Atlas kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 2325, flush 0, corrupt 0, gen 0

Panic message when the server locks up.

crash2.png.7a9127b4ee442cefd3a4df12ee40748c.png

Another crash from a different day.

crash.png.b6425ff9ce4575248bd2f8fa5d9fd6ab.png

atlas-diagnostics-20200126-161

7.zip

Edited by HotelErotica
more info
Link to comment
On 1/27/2020 at 1:25 AM, johnnie.black said:

That's not the cache files system, it's the docker image, delete and re-create.

Redid my docker image, and upgraded to the to 6.8.2 and everything appears to be good, no crashes in the last three days.

 

I am seeing these messages though, they appear to only happen on spun down drives. Doesn't seem to be causing any problems though.

 

Thanks for the help so far!

Jan 28 16:08:18 Atlas kernel: print_req_error: I/O error, dev sdf, sector 19532873536
Jan 28 16:08:28 Atlas kernel: sd 7:0:2:0: [sdd] tag#2485 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x00
Jan 28 16:08:28 Atlas kernel: sd 7:0:2:0: [sdd] tag#2485 CDB: opcode=0x88 88 00 00 00 00 04 8c 3f ff 40 00 00 00 08 00 00
Jan 28 16:08:28 Atlas kernel: print_req_error: I/O error, dev sdd, sector 19532873536
Jan 28 16:08:37 Atlas kernel: sd 7:0:5:0: [sdg] tag#2513 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x00
Jan 28 16:08:37 Atlas kernel: sd 7:0:5:0: [sdg] tag#2513 CDB: opcode=0x88 88 00 00 00 00 04 8c 3f ff 40 00 00 00 08 00 00
Jan 28 16:08:37 Atlas kernel: print_req_error: I/O error, dev sdg, sector 19532873536

 

 

Edited by HotelErotica
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...