Plex is in read only mode

[Jeff in Indy]

I overflowed my cache yesterday, and once again, the Plex database got corrupted.  At least I'm getting good at restoring from an old database.

 

So I copied over an older copy, but the UI wouldn't load.  I go into the Log, and see these lines over and over...

 

Starting Plex Media Server.
Error: Unable to set up server: sqlite3_statement_backend::loadOne: attempt to write a readonly database (N4soci10soci_errorE)
Stopping Plex Media Server.
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

 

How do I enable writing to the database?  I hadn't come across this before...

[Squid]

3 minutes ago, Jeff in Indy said:

I overflowed my cache yesterday

Are you sure the (presumably) cache drive isn't itself in read-only mode because of this?  (BTRFS isn't the most fault tolerant FS unfortunately)

 

diagnostics would give the answer to the above

[Jeff in Indy]

Well Squid, you tell me.

 

(After moving some files from the cache, I rebooted, and disc 7 came up as invalid.  I stopped the array, "removed" disc 7, restarted, stopped, "re-added" disc 7, and it is rebuilding as we speak.  So you'll see disc 7 as Invalid in the log.)

kaokulk-diagnostics-20220628-1402.zip

[itimpi]

1 hour ago, Jeff in Indy said:

overflowed my cache yesterday,

Have you set a Minimum Free Space value for the cache drive to stop it getting too full?   BTRFS file systems to be prone to corruption if they run out of free space.

[Jeff in Indy]

I was looking for that setting, when my server became unresponsive on my desktop.  My phone was still showing it working, so I rebooted my desktop.

 

Now one of my parity drives is showing it's invalid, and my rebuild of disc 7 has changed to a Read-Check, and has paused.

 

Where do I go from here?  Now I'm afraid to do anything.

kaokulk-diagnostics-20220628-1600.zip

[Jeff in Indy]

I'm now flying without a net, and I'm not sure what to do next.  I have 2 drives that are out of sync.  Any help would be greatly appreciated.

[trurl]

02:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller [1b4b:9215] (rev 11)
	Subsystem: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller [1b4b:9215]
04:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller [1b4b:9215] (rev 11)
	Subsystem: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller [1b4b:9215]

Marvell controllers are NOT recommended and are a likely cause of the communication problems with multiple disks.

 

 

[Jeff in Indy]

So I should shut down the server, and order a new controller?

[trurl]

 

[Jeff in Indy]

I've ordered a new card, and it will be here tomorrow.

 

I had (as far as I know) a working Disc 7 that I was rebuilding when my second parity disc went down.  Do I assume that Disc 7 is trash, and rebuild it from my working parity, or should I assume disc 7 is OK, and rebuild parity from that?  My guess is the safer bet is to rebuild 7 from the working parity drive.

[Corvinus]

Marvell chipset isn't inherently bad, most customers are without a doubt using one if connected to their motherboards instead of dedicated HBA. In fact, it's a great option to e.g. cheap SIL which often has firmware that will park heads like a madman, Marvell won't. I find it's unfitting that a mod would blindly point fingers to it like in this thread, when the symptoms don't specifically match at all. I took a deep dive into the reference, and it's entirely IOMMU/vt-d(virtualization) related, symptoms include loosing all drives, and would have to be triggered by a system upgrade or BIOS changes if it already worked. 

 

Change the controller for troubleshooting sure, but I highly doubt it's related if you still see at least some drives connected to it. If anything I'd check if the cables and connections are good, first.

[trurl]

9 hours ago, Jeff in Indy said:

rebuild 7 from the working parity drive

 

[Jeff in Indy]

1 hour ago, Corvinus said:

Change the controller for troubleshooting sure, but I highly doubt it's related if you still see at least some drives connected to it. If anything I'd check if the cables and connections are good, first.

Well, I wondered, since only 1 of those drives use the Marvell card.  The other is on the motherboard directly (disc 7, the first one to get out of sync).

 

I figured that maybe the Marvell cards were somehow screwing up other SATA connections, since I didn't know anything about the Marvell problem, and was too much in a panic to do so.

 

I'll wait until the card comes in and replace it, then rebuild disc 7 and Parity 2 from the working parity drive.

[Jeff in Indy]

By the way, here's what I got:

 

FebSmart PCIE3.0 to 4-Ports 6Gbps SATA 3.0 Expansion Card for PCs, Servers, NAS, MAC Pros-Plug and Play on Windows, MAC OS, Linux System-ASMedia ASM1064 Non-Raid PCIE 3.0 SATA Controller (FS-S4-Pro)

https://www.amazon.com/gp/product/B097R6MQBT/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

[Jeff in Indy]

So frustrated right now…

 

so I get 2 new SATA cards (mentioned above). Take the old Maxell cards out, put new ones in, hook everything up. Start it up, and the 2 drives that were out of sync don’t show up at all, as well as a third drive. 
 

Shut down, replace the SATA cables for those 3 drives. Start it back up, now FOUR drives are missing. They don’t seem to be on the same card, ones even in the motherboard. Aaaaack…

[ChatNoir]

Your diagnostics might show something ?

[Jeff in Indy]

Diags included.  Also a screenshot during boot, showing "SATA link down".  Not sure why it's showing that.

kaokulk-diagnostics-20220630-0831.zip

[trurl]

Check power connections, including splitters

[trurl]

Also reseat new cards

[JorgeB]

You can ignore all those SATA link down messages, it's typical with some Asmedia controllers, they are "phantom" ports, i.e., they don't exist.

[Jeff in Indy]

I reset the cards, no luck.  Later today, I'll try seeing if Maxell cards fix it, if not, I'll recheck all the cables.

[Jeff in Indy]

OK, I tried the Maxell cards, no difference.  I then pulled every drive out, and reset all the cables.  I think there was a MOLEX splitter that had come loose.  Now all drives are showing up, and my disc 7 and parity 2 are being rebuilt, and I have the non-Maxell cards in and running.

 

So after ALL that, I'm back to square one:  My Plex database won't run because of a read only error.

 

I'm going to set a Minimum Free Space for my cache when my rebuild is complete.

 

Squid asked if my cache is read only.  I never got an answer to that since my system went belly up.  Does anyone see anything of interest here?

kaokulk-diagnostics-20220630-2322.zip

[JorgeB]

It's not a pool problem. 

[Jeff in Indy]

So now I'm back to this non-stop message:

 

Starting Plex Media Server.
Error: Unable to set up server: sqlite3_statement_backend::loadOne: attempt to write a readonly database (N4soci10soci_errorE)
Stopping Plex Media Server.
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

[trurl]

20 minutes ago, Jeff in Indy said:

Error: Unable to set up server: sqlite3_statement_backend::loadOne: attempt to write a readonly database (N4soci10soci_errorE)
Stopping Plex Media Server.

Search the plex forums for how to restore the database from backup.