I overflowed my cache yesterday, and once again, the Plex database got corrupted.  At least I'm getting good at restoring from an old database.

 

So I copied over an older copy, but the UI wouldn't load.  I go into the Log, and see these lines over and over...

 

Starting Plex Media Server.
Error: Unable to set up server: sqlite3_statement_backend::loadOne: attempt to write a readonly database (N4soci10soci_errorE)
Stopping Plex Media Server.
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

 

How do I enable writing to the database?  I hadn't come across this before...

3 minutes ago, Jeff in Indy said:

I overflowed my cache yesterday

Are you sure the (presumably) cache drive isn't itself in read-only mode because of this?  (BTRFS isn't the most fault tolerant FS unfortunately)

 

diagnostics would give the answer to the above

Well Squid, you tell me.

 

(After moving some files from the cache, I rebooted, and disc 7 came up as invalid.  I stopped the array, "removed" disc 7, restarted, stopped, "re-added" disc 7, and it is rebuilding as we speak.  So you'll see disc 7 as Invalid in the log.)

kaokulk-diagnostics-20220628-1402.zip

1 hour ago, Jeff in Indy said:

overflowed my cache yesterday,

Have you set a Minimum Free Space value for the cache drive to stop it getting too full?   BTRFS file systems to be prone to corruption if they run out of free space.

I was looking for that setting, when my server became unresponsive on my desktop.  My phone was still showing it working, so I rebooted my desktop.

 

Now one of my parity drives is showing it's invalid, and my rebuild of disc 7 has changed to a Read-Check, and has paused.

 

Where do I go from here?  Now I'm afraid to do anything.

kaokulk-diagnostics-20220628-1600.zip

I'm now flying without a net, and I'm not sure what to do next.  I have 2 drives that are out of sync.  Any help would be greatly appreciated.

02:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller [1b4b:9215] (rev 11)
	Subsystem: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller [1b4b:9215]
04:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller [1b4b:9215] (rev 11)
	Subsystem: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller [1b4b:9215]

Marvell controllers are NOT recommended and are a likely cause of the communication problems with multiple disks.

 

 

So I should shut down the server, and order a new controller?

 

I've ordered a new card, and it will be here tomorrow.

 

I had (as far as I know) a working Disc 7 that I was rebuilding when my second parity disc went down.  Do I assume that Disc 7 is trash, and rebuild it from my working parity, or should I assume disc 7 is OK, and rebuild parity from that?  My guess is the safer bet is to rebuild 7 from the working parity drive.

Marvell chipset isn't inherently bad, most customers are without a doubt using one if connected to their motherboards instead of dedicated HBA. In fact, it's a great option to e.g. cheap SIL which often has firmware that will park heads like a madman, Marvell won't. I find it's unfitting that a mod would blindly point fingers to it like in this thread, when the symptoms don't specifically match at all. I took a deep dive into the reference, and it's entirely IOMMU/vt-d(virtualization) related, symptoms include loosing all drives, and would have to be triggered by a system upgrade or BIOS changes if it already worked. 

 

Change the controller for troubleshooting sure, but I highly doubt it's related if you still see at least some drives connected to it. If anything I'd check if the cables and connections are good, first.

9 hours ago, Jeff in Indy said:

rebuild 7 from the working parity drive

 

1 hour ago, Corvinus said:

Change the controller for troubleshooting sure, but I highly doubt it's related if you still see at least some drives connected to it. If anything I'd check if the cables and connections are good, first.

Well, I wondered, since only 1 of those drives use the Marvell card.  The other is on the motherboard directly (disc 7, the first one to get out of sync).

 

I figured that maybe the Marvell cards were somehow screwing up other SATA connections, since I didn't know anything about the Marvell problem, and was too much in a panic to do so.

 

I'll wait until the card comes in and replace it, then rebuild disc 7 and Parity 2 from the working parity drive.

By the way, here's what I got:

 

FebSmart PCIE3.0 to 4-Ports 6Gbps SATA 3.0 Expansion Card for PCs, Servers, NAS, MAC Pros-Plug and Play on Windows, MAC OS, Linux System-ASMedia ASM1064 Non-Raid PCIE 3.0 SATA Controller (FS-S4-Pro)

https://www.amazon.com/gp/product/B097R6MQBT/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

So frustrated right now…

 

so I get 2 new SATA cards (mentioned above). Take the old Maxell cards out, put new ones in, hook everything up. Start it up, and the 2 drives that were out of sync don’t show up at all, as well as a third drive. 
 

Shut down, replace the SATA cables for those 3 drives. Start it back up, now FOUR drives are missing. They don’t seem to be on the same card, ones even in the motherboard. Aaaaack…

Your diagnostics might show something ?

Diags included.  Also a screenshot during boot, showing "SATA link down".  Not sure why it's showing that.

kaokulk-diagnostics-20220630-0831.zip

Check power connections, including splitters

Also reseat new cards

You can ignore all those SATA link down messages, it's typical with some Asmedia controllers, they are "phantom" ports, i.e., they don't exist.

I reset the cards, no luck.  Later today, I'll try seeing if Maxell cards fix it, if not, I'll recheck all the cables.

OK, I tried the Maxell cards, no difference.  I then pulled every drive out, and reset all the cables.  I think there was a MOLEX splitter that had come loose.  Now all drives are showing up, and my disc 7 and parity 2 are being rebuilt, and I have the non-Maxell cards in and running.

 

So after ALL that, I'm back to square one:  My Plex database won't run because of a read only error.

 

I'm going to set a Minimum Free Space for my cache when my rebuild is complete.

 

Squid asked if my cache is read only.  I never got an answer to that since my system went belly up.  Does anyone see anything of interest here?

kaokulk-diagnostics-20220630-2322.zip

It's not a pool problem. 

So now I'm back to this non-stop message:

 

Starting Plex Media Server.
Error: Unable to set up server: sqlite3_statement_backend::loadOne: attempt to write a readonly database (N4soci10soci_errorE)
Stopping Plex Media Server.
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

20 minutes ago, Jeff in Indy said:

Error: Unable to set up server: sqlite3_statement_backend::loadOne: attempt to write a readonly database (N4soci10soci_errorE)
Stopping Plex Media Server.

Search the plex forums for how to restore the database from backup.