I am at a loss. I tried to the requested method of seeing if things would remedy in safe mode and still no. This happened before with a different NVME PCIE card I used for my families Minecraft server. I believed I had no other option than to format and start again. I even bought a newer PCIE adapter card but the problem persisted. Now that it is happening to the main NVME drive, I am desperate as formatting the drive would make rebuilding my local setup nothing short of impossible.

Breakdown: A few days ago is when this first happened with the first NVME drive. I happened to be using AMP (Minecraft server host) and running Chunky (world pre-generator) and mind you before this happened, I have done this more times than I can count. While running, the server at some point just stopped and started locking up. I had the dashboard open and the only thing I could see was that two CPU cores were red-lined. Which is not out of the normal given what I was doing with the Minecraft server but they were pegged at 100%. That honestly could have just been where the UI had frozen when everything locked up to be honest but I wanted to note it.

Troubleshooting : After sometime had passed I realized that the entire server had locked up. I wasn't sure what to really do here so I restarted the server with a power cycle. After doing this the first time everything seemed fine. I looked online and saw a few instances that other people were having issues that seemed similar enough to what I have been dealing with and tried doing the syslog locally to see if I could see any warning signs but nothing jumped out at me. It was at this point where last night my main NVME showed the same error 'Unmountable: no file system'. I took the advice given and restarted the server in to safe mode and with nothing running, it is still down and showing the same error.

I can not use the server at the moment seeing as the NVME drive holds all my appdata so I'll be in limbo.

I appreciate any help with this matter as again I am at a loss here.

All the best,

LoveyDov3

Please post the diagnostics

My apologies! I did not get notified that a response had been made! I will post what I can here shortly.

Here is the .ZIP

Again my apologies for not seeing this sooner!

loveyunraid-diagnostics-20250608-2236.zip

Also here is the log from the drive after stopping and starting the array.

Jun 8 22:33:55 LoveyUnRaid kernel: nvme0n1:

Jun 8 22:34:32 LoveyUnRaid emhttpd: online: CT2000P3SSD8_2305E6A6A057 (nvme0n1) 512 3907029168

Jun 8 22:34:32 LoveyUnRaid emhttpd: /bin/lsblk -lnbo TYPE,NAME,START,SIZE /dev/nvme0n1 2>&1

Jun 8 22:34:32 LoveyUnRaid emhttpd: disk nvme0n1 2000398934016

Jun 8 22:34:32 LoveyUnRaid emhttpd: device nvme0n1: no partitions

Jun 8 22:34:32 LoveyUnRaid emhttpd: import 30 pool device: (nvme0n1) CT2000P3SSD8_2305E6A6A057

Jun 8 22:34:33 LoveyUnRaid emhttpd: read SMART /dev/nvme0n1

Jun 8 22:34:37 LoveyUnRaid emhttpd: /bin/lsblk -lnbo TYPE,NAME,START,SIZE /dev/nvme0n1 2>&1

Jun 8 22:34:37 LoveyUnRaid emhttpd: disk nvme0n1 2000398934016

Jun 8 22:34:37 LoveyUnRaid emhttpd: device nvme0n1: no partitions

Jun 8 22:34:37 LoveyUnRaid emhttpd: import 30 pool device: (nvme0n1) CT2000P3SSD8_2305E6A6A057

Jun 8 22:36:36 LoveyUnRaid kernel: block nvme0n1: the capability attribute has been deprecated.

Jun 8 22:59:02 LoveyUnRaid emhttpd: read SMART /dev/nvme0n1

Jun 8 22:59:11 LoveyUnRaid emhttpd: /bin/lsblk -lnbo TYPE,NAME,START,SIZE /dev/nvme0n1 2>&1

Jun 8 22:59:11 LoveyUnRaid emhttpd: disk nvme0n1 2000398934016

Jun 8 22:59:11 LoveyUnRaid emhttpd: device nvme0n1: no partitions

Jun 8 22:59:11 LoveyUnRaid emhttpd: import 30 pool device: (nvme0n1) CT2000P3SSD8_2305E6A6A057

Jun 8 22:59:42 LoveyUnRaid emhttpd: /bin/lsblk -lnbo TYPE,NAME,START,SIZE /dev/nvme0n1 2>&1

Jun 8 22:59:42 LoveyUnRaid emhttpd: disk nvme0n1 2000398934016

Jun 8 22:59:42 LoveyUnRaid emhttpd: device nvme0n1: no partitions

Jun 8 22:59:42 LoveyUnRaid emhttpd: import 30 pool device: (nvme0n1) CT2000P3SSD8_2305E6A6A057

Jun 8 23:00:01 LoveyUnRaid emhttpd: read SMART /dev/nvme0n1

Jun 8 23:00:10 LoveyUnRaid emhttpd: /bin/lsblk -lnbo TYPE,NAME,START,SIZE /dev/nvme0n1 2>&1

Jun 8 23:00:10 LoveyUnRaid emhttpd: disk nvme0n1 2000398934016

Jun 8 23:00:10 LoveyUnRaid emhttpd: device nvme0n1: no partitions

Jun 8 23:00:10 LoveyUnRaid emhttpd: import 30 pool device: (nvme0n1) CT2000P3SSD8_2305E6A6A057

Jun 8 23:01:05 LoveyUnRaid emhttpd: read SMART /dev/nvme0n1

Jun 8 23:01:18 LoveyUnRaid emhttpd: /bin/lsblk -lnbo TYPE,NAME,START,SIZE /dev/nvme0n1 2>&1

Jun 8 23:01:18 LoveyUnRaid emhttpd: disk nvme0n1 2000398934016

Jun 8 23:01:18 LoveyUnRaid emhttpd: device nvme0n1: no partitions

Jun 8 23:01:18 LoveyUnRaid emhttpd: import 30 pool device: (nvme0n1) CT2000P3SSD8_2305E6A6A057

Jun 8 23:01:48 LoveyUnRaid emhttpd: /bin/lsblk -lnbo TYPE,NAME,START,SIZE /dev/nvme0n1 2>&1

Jun 8 23:01:48 LoveyUnRaid emhttpd: disk nvme0n1 2000398934016

Jun 8 23:01:48 LoveyUnRaid emhttpd: device nvme0n1: no partitions

Jun 8 23:01:48 LoveyUnRaid emhttpd: import 30 pool device: (nvme0n1) CT2000P3SSD8_2305E6A6A057

There's no partition on that device, any idea on that happened? Depending on what happened, it may still be possible to recover.

I really have no clue. I didn't do anything different with either instances of this happening. As mentioned with the first NVME drive this happened to, I just chalked it up to chance and formatted since it was so serious to have to redo the Minecraft instance I had been working. When it happened to my main cache NVME, that's when I reached out.

I'm not sure what to do next. If you believe there is a chance for the data to be recovered I am all ears!

Also, what is the proper way to setup the 'syslog' function? To my knowledge, this would help in instance where the server may lock up but still have a trace route showing what happened just before the lock up. I want to make sure I have my t's crossed and my I's dotted for any future issues they might occur.

All the best!

Type sfdisk /dev/nvme0n1 then type 2048 and enter, then hit CTRL + C to abort and post the output.

Below is what was out put.

root@LoveyUnRaid:~# sfdisk /dev/nvme0n1 2048

sfdisk: /dev/nvme0n1: cannot modify partition 2049: partition table contains only 4 partitions

root@LoveyUnRaid:~# sfdisk /dev/nvme0n1

Welcome to sfdisk (util-linux 2.41).

Changes will remain in memory only, until you decide to write them.

Be careful before using the write command.

Checking that no-one is using this disk right now ... OK

Disk /dev/nvme0n1: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors

Disk model: CT2000P3SSD8

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: dos

Disk identifier: 0x8eb40d79

Old situation:

Type 'help' to get more information.

>>> 2048

Created a new DOS (MBR) disklabel with disk identifier 0x3f1445dd.

Created a new partition 1 of type 'Linux' and of size 1.8 TiB.

/dev/nvme0n1p1 : 2048 3907029167 (1.8T) Linux

/dev/nvme0n1p2:

2048 is only after entering the sfdisk command, but no signature was found, try the same but with 64

Type sfdisk /dev/nvme0n1 then type 64and enter, then hit CTRL + C to abort and post the output.

So it should look like this:

sfdisk /dev/nvme0n164

or

sfdisk /dev/nvme0n1 64

?

Just a bit confused as to what to do.

When:

sfdisk /dev/nvme0n1

is typed, I get the following output below;

root@LoveyUnRaid:~# sfdisk /dev/nvme0n1

Welcome to sfdisk (util-linux 2.41).

Changes will remain in memory only, until you decide to write them.

Be careful before using the write command.

Checking that no-one is using this disk right now ... FAILED

This disk is currently in use - repartitioning is probably a bad idea.

Umount all file systems, and swapoff all swap partitions on this disk.

Use the --no-reread flag to suppress this check.

sfdisk: Use the --force flag to overrule all checks.

When:

sfdisk /dev/nvme0n1 64

is typed, this is what comes up;

root@LoveyUnRaid:~# sfdisk /dev/nvme0n1 64

sfdisk: /dev/nvme0n1: cannot modify partition 65: partition table contains only 4 partitions

Just now, LoveyDov3 said:

This disk is currently in use - repartitioning is probably a bad idea.

You can't do that with the disk in use, did you abort the previous command?

12 minutes ago, JorgeB said:

then hit CTRL + C to abort and post the output.

Then type just sfdisk /dev/nvme0n1, hit enter, only after enter 64 and enter again, then hit CTRL + C to abort and post the output.

The array is stopped and I am not sure what would be using the drive since it still shows as unmounted. Should I reboot in safe mode and try this?

I am still getting that 'currently in use...' readout.

My previous post was the only time it said something different. With the 2048.

If you are still getting drive in use, reboot first, then try again.

I will try this next. However when rebooting, it seems to hang up at the:

local_shutdown: Waiting up to 90seconds for graceful shutdown. . .

I'm not sure when this started to happen as sometimes I don't have my monitor that's connected on but I have to power cycle the server to get it back up.

I will try again after it comes back online.

root@LoveyUnRaid:~# sfdisk /dev/nvme0n1

Welcome to sfdisk (util-linux 2.41).

Changes will remain in memory only, until you decide to write them.

Be careful before using the write command.

Checking that no-one is using this disk right now ... OK

Disk /dev/nvme0n1: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors

Disk model: CT2000P3SSD8

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: dos

Disk identifier: 0x8eb40d79

Old situation:

Type 'help' to get more information.

>>> 64

Created a new DOS (MBR) disklabel with disk identifier 0x7d3e3148.

Created a new partition 1 of type 'Linux' and of size 1.8 TiB.

/dev/nvme0n1p1 : 64 3907029167 (1.8T) Linux

/dev/nvme0n1p2:

/dev/nvme0n1p2: ^C

Above is what populated when following the steps mentioned.

All the best and thank you for all the help thus far!

I also appreciate the patience! This type of troubleshooting always drives me up the wall. Hands on things, easy peasy! Having to type in code for outputs, drives me crazy because it always ends up be a number or letter out of place that makes the whole process way more difficult then it needs to be...

but seriously, thank you for the help!

I'm afraid that there's no existing filesystem signature on either of the two possible sectors, this suggests it's just not the partition that is missing, so not possible to recover that way, depending on what exactly happened, you may be able to recover with a file recovery app, like UFS explorer, the free trial should show if it can recover anything from that device.

Thanks for the heads up! I was able to scan the drive and able to access the 'docker.img' as a see that everything appears to be there. Do you know which minimum version of the program I need to get in order to save the data? Really don't want to have to drop 700$. 🫠

The docker image can easily be recreated, but if there's more data there, the $60 version should do it.

I'll go ahead and try it out and report back.