[Plugin] Nvidia-Driver

[SaltShakerOW]

31 minutes ago, ich777 said:

If you'll bind it through the Kernel command line via syslinux.conf, like you did it after reviewing the Diagnostics again, then you would still bind it to VFIO:

BOOT_IMAGE=/bzimage initrd=/bzroot pcie_acs_override=downstream vfio-pci.ids=10de:1b81,10de:10f0

 

Please remove the VFIO binding from the syslinux.conf and reboot.

It worked! Thanks for the speedy help and easy solution.

[Aspiro]

On 12/4/2022 at 7:29 AM, ich777 said:

This was misinformation, the latest driver supports Tesla cards and this is the exact same driver that you can download from Nvidia.

 

Another user with a P100 confirmed that it is working, see this post:

 

...and the user that reported the issue with it's P4 also solved the issue (he forgot to put in --runtime=nvidia in the template).

Thanks for the answer when i got time I'll install it on my unraid machine. Tahanks for the work you do.

[SimpleDino]

Hi,

 

Today I got a notification saying nvidia-plugin had crashed (don't remember the detail...and yes I am ashamed) and when I tried to go to the Nvidia-plugin under settings the whole screen/UnRaid froze! After rebooting I deleted the plugin and re-installed to the letter according to your instructions etc. No more freezing issues and the plugin is OK, but now I can not find the Nvidia P2000 gpu card under the info section in the plugin (see picture). Has anyone encountered this issue? Or have I just missed a step during uninstall & install part... Any idea?! The card is no connected to any VMs or bound to VFIO.

 

Also at htop am getting high cpu usage for this: nvidia-smi -q -x -g GPU-3181...........this is the gpu ID I guess. Could this be the p-state script for power consumptions or me trying nvidia-smi command in console and it's not finding anything?

 

Steps taken:

1. Uninstall plugin

2. Reboot

3. Install according to instructions

4. Reboot

5. Downgrade to previous driver...no luck

5.1 Reboot

6.0 Upgrade to current version

6.1 Reboot.

 

 

tower-diagnostics-20221208-1504.zip

Edited December 8, 2022 by SimpleDino

[ich777]

18 minutes ago, SimpleDino said:

Also at htop am getting high cpu usage for this: nvidia-smi -q -x -g GPU-3181...........this is the gpu ID I guess.

This is caused by the GPU Statistics plugin.

 

18 minutes ago, SimpleDino said:

Today I got a notification saying nvidia-plugin had crashed (don't remember the detail...and yes I am ashamed) and when I tried to go to the Nvidia-plugin under settings the whole screen/UnRaid froze! After rebooting I deleted the plugin and re-installed to the letter according to your instructions etc. No more freezing issues and the plugin is OK, but now I can not find the Nvidia P2000 gpu card under the info section in the plugin (see picture).

Please turn of the server completely, pull the power cord from the wall, press the power and reset button a few times to empty the caps, wait 30 seconds and then power up the server and see if this makes any difference.

 

If not the only thing that I would recommend next is pulling the GPU from the server and try it on a Desktop PC, also install the drivers and put a load on it and see if everything works.

nvidia-persistenced reports that it can't open the device.

[SimpleDino]

On 12/8/2022 at 3:39 PM, ich777 said:

This is caused by the GPU Statistics plugin.

 

Please turn of the server completely, pull the power cord from the wall, press the power and reset button a few times to empty the caps, wait 30 seconds and then power up the server and see if this makes any difference.

 

If not the only thing that I would recommend next is pulling the GPU from the server and try it on a Desktop PC, also install the drivers and put a load on it and see if everything works.

nvidia-persistenced reports that it can't open the device.

 

[SOLVED]

Thanks for all the input, unfortunately it did not help!
Figured out the issue... Apparently if you populate the last PCIe GPU slot on the X570 Taichi Motherboard then the first one becomes inactive somehow. Second one is also populated, used in VM's etc.

 

I populated the last gpu slot as mentioned with a PCIE M.2 Adapter with four 970 Plus 1TB nvme's but I did not use or activate in Unraid until the other day and that is when nvidia-plugin gave out a warning and first gpu slot (P2000) got inactive but recognized in devices.

 

I've removed the M.2 PCIE card until I get hold of an Threadripper system.

Edited December 9, 2022 by SimpleDino

[lukeasoiler]

Hi guys,

I'm sure I read about the issue before somewhere here in the forums, but I just fail to find it again (thought I bookmarked it).

I'm having the issue that on startup the plugin finds my gpu (GT 710) just fine, but once my VM ran, where I pass it through to, the plugin page or any terminal commands cannot access the gpu anymore. Is there a way around it? I'm mainly looking to always set it to persistenced mode, whenever the VM is not running.

Best,
Lukas

[ich777]

1 hour ago, lukeasoiler said:

I'm having the issue that on startup the plugin finds my gpu (GT 710) just fine, but once my VM ran

After you've shut it off?

Maybe the VM resets the graphics card not right and it is still bound to the VM.

 

1 hour ago, lukeasoiler said:

Is there a way around it?

No, at least not from my knowledge.

 

1 hour ago, lukeasoiler said:

I'm mainly looking to always set it to persistenced mode, whenever the VM is not running.

I strongly don't recommend doing that because you can simply crash your server.

But if you do so, please be sure to kill persistneced first, then start the VM and after finally shutting down the VM you can start peristenced again.

[Skitals]

Dec 15 18:34:58 Tower kernel: NVRM: GPU at PCI:0000:05:00: GPU-2e461cb5-b925-eff5-da77-bc3ab28682b1
Dec 15 18:34:58 Tower kernel: NVRM: Xid (PCI:0000:05:00): 79, pid=5103, GPU has fallen off the bus.
Dec 15 18:34:58 Tower kernel: NVRM: GPU 0000:05:00.0: GPU has fallen off the bus.
Dec 15 18:34:58 Tower kernel: NVRM: GPU 0000:05:00.0: RmInitAdapter failed! (0x25:0xf:1241)
Dec 15 18:34:58 Tower kernel: NVRM: GPU 0000:05:00.0: rm_init_adapter failed, device minor number 0
Dec 15 18:34:58 Tower kernel: NVRM: GPU 0000:05:00.0: RmInitAdapter failed! (0x22:0x56:667)
Dec 15 18:34:58 Tower kernel: NVRM: GPU 0000:05:00.0: rm_init_adapter failed, device minor number 0
Dec 15 18:34:58 Tower kernel: NVRM: GPU 0000:05:00.0: RmInitAdapter failed! (0x22:0x56:667)
Dec 15 18:34:58 Tower kernel: NVRM: GPU 0000:05:00.0: rm_init_adapter failed, device minor number 0
Dec 15 18:34:58 Tower kernel: NVRM: GPU 0000:05:00.0: RmInitAdapter failed! (0x22:0x56:667)
Dec 15 18:34:58 Tower kernel: NVRM: GPU 0000:05:00.0: rm_init_adapter failed, device minor number 0

 

Decided to install nvidia-driver to see just how bad my gt710 is at transcoding. It falls off the bus as soon as nvidia-smi is run. I can set this card as my primary output, boot unraid, run nvidia-smi (or open the plugin) and immediately my screen goes black. I am of course running driver version v470.141.03 for legacy support.

 

I have tried everything suggested. I enabled Above 4G Decoding and Resizable BAR. I disabled ASPM. I disabled C-States. The only thing I didn't try was switching from UEFI to Legacy since I have no interest in that. I turned off docker and turned off my VMs for testing.

 

Any tips would be appreciated.

tower-diagnostics-20221215-1828.zip

[ich777]

7 hours ago, Skitals said:

I have tried everything suggested. I enabled Above 4G Decoding and Resizable BAR. I disabled ASPM. I disabled C-States.

Please also check if the slot provides enough power to the card but that shouldn't be an issue.

 

Sadly enough I have no real solution to this and for some users it is enough to disable C-States in the BIOS but this seems like a AMD Platform issue in general and can happen with nearly every card, but for some users the solution was to switch to a different card.

Usually this error means that some kind of insufficient power delivery or some kind of sleep or power saving measure in terms of the PCIe bus causes this.

 

7 hours ago, Skitals said:

The only thing I didn't try was switching from UEFI to Legacy since I have no interest in that.

May I ask why? This would be nothing more than turning on Legacy Boot, nothing should change in terms of VMs and so on...

I personally prefer Legacy Boot over UEFI for Unraid.

 

7 hours ago, Skitals said:

I turned off docker and turned off my VMs for testing.

Once the card fall from the bus it's not that easy to recover it, there is certainly a way but too complicated.

 

7 hours ago, Skitals said:

Decided to install nvidia-driver to see just how bad my gt710 is at transcoding.

In terms of h264 (AVC) not that bad but this card simply can't transcode h265 (HEVC) because it is Kepler based.

 

 

Oh, I completely forgot, try to set the PCIe slot manually to PCIe Gen 3.0 or even 2.0 in the BIOS <- this can maybe also help with your issue (the GT710 is only PCIe Gen 2.0 anyways).

[Skitals]

6 hours ago, ich777 said:

Please also check if the slot provides enough power to the card but that shouldn't be an issue.

It is a pciex1 card in a pciex4 slot, so that seems unlikely. That slot gets its lanes from the chipset versus the cpu, so switching the card to a different (cpu) slot might make a difference.

 

6 hours ago, ich777 said:

May I ask why? This would be nothing more than turning on Legacy Boot, nothing should change in terms of VMs and so on...

I personally prefer Legacy Boot over UEFI for Unraid.

It took a lot of mucking around 2 years ago to dial in my settings and make gpu passthrough work with my radeon 5700xt (subject to amd reset bug). I suppose you are right it shouldn't make a difference to my VMs so I might pursue this.

 

6 hours ago, ich777 said:

Oh, I completely forgot, try to set the PCIe slot manually to PCIe Gen 3.0 or even 2.0 in the BIOS <- this can maybe also help with your issue (the GT710 is only PCIe Gen 2.0 anyways).

The motherboard only supports running in gen 3 or gen 4 natively, so it will run in backwards compat either way (card is gen 2). Afaik I can't change this on a per slot basis.

 

I went ahead and ordered a quadro p400 for $50. It's pcie gen 3 and supported in the latest nvidia drivers so hopefully I have better luck with that. I will pick up troubleshooting when that comes in a week or so.

 

Thanks for your thoughtful input!

[ich777]

7 minutes ago, Skitals said:

The motherboard only supports running in gen 3 or gen 4 natively, so it will run in backwards compat either way (card is gen 2). Afaik I can't change this on a per slot basis.

But keep in mind that this fixed issues for some users in terms of cards fall from the bus.

 

Even one user switched slots and it started working and prevented that the card fall from the bus.

 

12 minutes ago, Skitals said:

It is a pciex1 card in a pciex4 slot, so that seems unlikely.

Never say never…

Maybe it‘s related because it‘s connected to the chipset and maybe some weird or hald-baked implementation in the BIOS so that the card falls from the bus.

 

AMD did some crazy stuff with PCIe IIRC.

[Skitals]

7 minutes ago, ich777 said:

But keep in mind that this fixed issues for some users in terms of cards fall from the bus.

 

Even one user switched slots and it started working and prevented that the card fall from the bus.

 

Never say never…

Maybe it‘s related because it‘s connected to the chipset and maybe some weird or hald-baked implementation in the BIOS so that the card falls from the bus.

 

AMD did some crazy stuff with PCIe IIRC.

 

You are right, switching slots and forcing gen 3 could do the trick. Both things I will try, need be, when the new card comes in. I can probably swap the GT 710 and dell 310 without affecting my iommu/VM assignments, so I might try that in the meantime.

[Skitals]

On 12/16/2022 at 8:43 AM, ich777 said:

But keep in mind that this fixed issues for some users in terms of cards fall from the bus.

 

Even one user switched slots and it started working and prevented that the card fall from the bus.

 

Never say never…

Maybe it‘s related because it‘s connected to the chipset and maybe some weird or hald-baked implementation in the BIOS so that the card falls from the bus.

 

AMD did some crazy stuff with PCIe IIRC.

 

I think I have exhausted all options with the GT 710. CSM Support legacy boot unraid (uefi boot disabled). Forced Gen2 and Gen 3. Tried all of the above in a different pcie slot (cpu vs chipset lanes). GPU always falls off the bus. Only thing of note is a different error when pcie is set to gen2:

 

Dec 17 17:50:10 Tower kernel: NVRM: GPU at PCI:0000:0f:00: GPU-2e461cb5-b925-eff5-da77-bc3ab28682b1

Dec 17 17:50:10 Tower kernel: NVRM: Xid (PCI:0000:0f:00): 79, pid=4756, GPU has fallen off the bus.

Dec 17 17:50:10 Tower kernel: NVRM: GPU 0000:0f:00.0: GPU has fallen off the bus.

Dec 17 17:50:10 Tower kernel: pcieport 0000:00:03.2: AER: Corrected error received: 0000:00:03.2

Dec 17 17:50:10 Tower kernel: pcieport 0000:00:03.2: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)

Dec 17 17:50:10 Tower kernel: pcieport 0000:00:03.2:   device [1022:1483] error status/mask=00001000/00004000

Dec 17 17:50:10 Tower kernel: pcieport 0000:00:03.2:    [12] Timeout               

Dec 17 17:50:10 Tower kernel: NVRM: GPU 0000:0f:00.0: RmInitAdapter failed! (0x25:0xf:1241)

Dec 17 17:50:10 Tower kernel: NVRM: GPU 0000:0f:00.0: rm_init_adapter failed, device minor number 0

Dec 17 17:50:10 Tower kernel: NVRM: GPU 0000:0f:00.0: RmInitAdapter failed! (0x22:0x56:667)

Dec 17 17:50:10 Tower kernel: NVRM: GPU 0000:0f:00.0: rm_init_adapter failed, device minor number 0

 

[ich777]

6 hours ago, Skitals said:

AER: Corrected error received: 0000:00:03.2

You can try to disable AER error correction, won‘t hurt anything.

I also have to disable it on my server because otherwise my PCIe cards won‘t work properly (this is most of the times caused by some weired or better speaking wrong integration in the BIOS).

Simply add this to your syslinux.conf:

pci=noaer

 

[Skitals]

11 hours ago, ich777 said:

You can try to disable AER error correction, won‘t hurt anything.

I also have to disable it on my server because otherwise my PCIe cards won‘t work properly (this is most of the times caused by some weired or better speaking wrong integration in the BIOS).

Simply add this to your syslinux.conf:

pci=noaer

 

 

Still fell off the bus, just without the aer error. Really hoping it's just this card. I've been using it for a few years but I don't think ever with drivers in any os. Just been using it for basic vga output in unraid/gui and the odd vm here and there. Never had any issues with other pcie devices, and I ask quite a bit from this machine.

[corfoto4]

Hi all,

 

I have an issue where the Nvidia plugin no longer works after the server exits S3 sleep. I use the plugin for hardware transcoding for Tdarr and Plex. If I restart the server then it works properly until S3 sleep is activated (server sleeps and then wakes up).

 

There was no issue when I was using UNRAID version 6.10. I am currently updated to 6.11.5.

Would this be an issue on my end? If so how would I be able to fix it?

 

Here are my diagnostics.

Thank You

rynvis-diagnostics-20221219-0912.zip

[ich777]

5 hours ago, corfoto4 said:

I have an issue where the Nvidia plugin no longer works after the server exits S3 sleep. I use the plugin for hardware transcoding for Tdarr and Plex. If I restart the server then it works properly until S3 sleep is activated (server sleeps and then wakes up).

Are the Diagnostics from before or after a sleep?

 

5 hours ago, corfoto4 said:

There was no issue when I was using UNRAID version 6.10. I am currently updated to 6.11.5.

Would this be an issue on my end? If so how would I be able to fix it?

I have to say that Nvidia and sleep on Linux where never best friends anyways... You find dozens of threads (not especially Unraid related) on the Internet about Nvidia cards not working properly after waking the device from Sleep.

In general I really don't support sleeping servers because well, it's a Server and should always run, I completely get the point putting it to sleep but that is not common practice.

 

This is most likely no issue on your side, this is most likely caused because of some change in the Kernel and nothing you can do about.

 

Anyways, please post a Diagnostics file from before it was put to Sleep and a Diagnostics file after it woke from Sleep.

Maybe also a screenshot from the plugin page and also from the output from:

nvidia-smi

after it woke from sleep would be helpful.

[corfoto4]

2 hours ago, ich777 said:

Are the Diagnostics from before or after a sleep?

Yes those diagnostics were after a sleep.

 

I have attached diagnostics after a server reboot. I have also confirmed that Tdarr and Plex are using hardware transcoding successfully (after the reboot). Tomorrow I will attach another set of diagnostics after S3 sleep (server sleeps overnight to save power). On another note, is there a way to reboot the Nvidia plugin without having to reboot the server? What's usually the main issue regarding Nvidia and Linux?

 

Here is the snapshot of 

nvidia-smi

(tasks are running on the card)

 

rynvis-diagnostics-20221219-1651.zip

Edited December 19, 2022 by corfoto4

[ich777]

53 minutes ago, corfoto4 said:

Yes those diagnostics were after a sleep.

But I see nothing obvious in that Diagnostics, the card is recognized and the drivers are loaded.

 

54 minutes ago, corfoto4 said:

Tomorrow I will attach another set of diagnostics after S3 sleep (server sleeps overnight to save power).

Can you maybe try to restart (disable and enable) the Docker service after Sleep and see if this makes a difference?

 

 

55 minutes ago, corfoto4 said:

On another note, is there a way to reboot the Nvidia plugin without having to reboot the server?

You can't reboot the plugin since it's a plugin and if I'm not mistaken you should see the card just fine on the Plugin page or am I wrong about that after the server wake up from sleep?

 

56 minutes ago, corfoto4 said:

What's usually the main issue regarding Nvidia and Linux?

Nvidia and Sleep in general on Linux.

Linux is getting better and better with sleep but a few years back it was horrible and caused issues over issues, even when there was no Nvidia GPU involved and even on consumer grade Laptops when you wan't to wake devices from sleep it most of the times won't work properly anymore or won't wake up at all.

 

As said above, this is a server, I would rather recommend that you shutdown your server entirely and boot it back up on a schedule whether you do this through a BIOS schedule or through WOL instead of using Sleep.

 

59 minutes ago, corfoto4 said:

Here is the snapshot of 

The output would be interesting after it woke up from Sleep.

Oh, I forgot to ask, do you shutdown your server when processes are running on the card or do you stop the processes first?

[corfoto4]

26 minutes ago, ich777 said:

Can you maybe try to restart (disable and enable) the Docker service after Sleep and see if this makes a difference?

Although I did not restart the docker service, restarting the application (Tdarr or Plex) does not fix the issue. I have tested this.

 

26 minutes ago, ich777 said:

you should see the card just fine on the Plugin page or am I wrong about that after the server wake up from sleep?

I think that you are right about the plugins. I was just wondering if it was possible to restart the plugin.

 

26 minutes ago, ich777 said:

Linux is getting better and better

I am assuming that it may begin to work in future updates. If it worked in 6.10 then maybe it might work in the future (or if the nvidia plugin gets updated).

 

26 minutes ago, ich777 said:

As said above, this is a server, I would rather recommend that you shutdown your server entirely and boot it back up on a schedule whether you do this through a BIOS schedule or through WOL instead of using Sleep.

I originally used sleep because I had an encrypted array and I would have to enter a passphrase if i had to shut the server down overnight. I no longer have and encrypted array so I would be able to initiate a reboot with no issues. Is there any disadvantage of rebooting compared to sleep? Is it worth having it sleep, then run a command to restart the server in the morning? Then I won't have to deal with the bios's rtc config.

 

26 minutes ago, ich777 said:

The output would be interesting after it woke up from Sleep.

The server wakes up about 2 - 3h before I do to get processes back up and running so that it is all operational when I want to use it. Would this affect the diagnostics that I provide tomorrow? I.e., should I create a set of diagnostics as close to the time it wakes up or does it not matter?

 

26 minutes ago, ich777 said:

Oh, I forgot to ask, do you shutdown your server when processes are running on the card or do you stop the processes first?

Before the server sleeps, the server hits an idle stage. No data is being sent over the network, no one is logged into the unraid interface, and the array become idle.

Tdarr will finish its processing over 3h before sleep, and Plex will not be using the gpu when entering sleep (The server will stay active if plex is in use. If I am watching media the server will wait until I finish as to not interrupt me. This is monitored though the network activity of S3 sleep).

 

I have attached my settings for Dynamix S3 Sleep:

(custom commands are only used to wake the server at the specified time)

Please let me know if you want to see the custom code.

Edited December 19, 2022 by corfoto4

[ich777]

23 minutes ago, corfoto4 said:

Although I did not restart the docker service

Please try to restart the Docker service.

 

23 minutes ago, corfoto4 said:

I think that you are right about the plugins. I was just wondering if it was possible.

No, because the plugin simply downloads, installs and activates the driver for the card and as you've said the card shows up on the plugin page so it should work.

 

23 minutes ago, corfoto4 said:

If it worked in 6.10 then maybe it might work in the future (or if the nvidia plugin gets updated).

This is Kernel specific and since the Nvidia driver is a out-of-tree module, so to speak from a third party it is on Nvidia to fix this with newer Kernels, which indeed Unraid 6.11.5 uses.

The plugin doesn't need a update and there is nothing I can do about it if it's not working with Sleep and 6.11.5 this is driver specific and the driver is compiled for each individual Unraid version and then uploaded to GitHub.

The plugin simply downloads the driver package, installs and activates it.

 

23 minutes ago, corfoto4 said:

Is there any disadvantage of rebooting compared to sleep?

From my perspective not really, the boot takes a bit longer, at least I'm assuming that.

 

23 minutes ago, corfoto4 said:

Is it worth having it sleep, then run a command to restart the server in the morning? Then I won't have to deal with the bios.

powerdown -r

Is the command that you want to issue to reboot the server.

However that doesn't makes much sense, please do it over WOL or a BIOS schedule <- the BIOS Schedule shouldn't even be hard to set up.

 

23 minutes ago, corfoto4 said:

(custom commands are only used to wake the server at the specified time)

Please let me know if you want to see the custom code.

Sorry I'm not that familiar with this plugin at all and have never seen it as I usually don't support it and I personally don't see a reason why to use it on a server (even not on a home server), if possible, please use the shutdown instead.

[corfoto4]

7 minutes ago, ich777 said:

Please try to restart the Docker service.

I'll try this tomorrow morning

 

7 minutes ago, ich777 said:

third party it is on Nvidia to fix this with newer Kernels, which indeed Unraid 6.11.5 uses.

So just by updating the Nvidia drivers which are provided by the plugin would fix the issue (when Nvidia fixes the kernal issue mentioned)?

 

9 minutes ago, ich777 said:

powerdown -r

Thank you, I'll try the other options first and then give this a go. I'll send a update tomorrow morning.

[ich777]

3 minutes ago, corfoto4 said:

when Nvidia fixes the kernal issue mentioned

I'm not entirely sure what causes that in your case but I can only think of an issue with the Kernel itself because 6.11.5 uses a newer Kernel than 6.10

As said, I really don't support the Sleep plugin over here because I know the issues with the Nvidia cards and you are not the first one who experiencing this issue and posting about it here.

 

3 minutes ago, corfoto4 said:

Thank you, I'll try the other options first and then give this a go. I'll send a update tomorrow morning.

I would rather go the shutdown method because you don't stress the hardware as much as waking it up and then shutting rebooting again, at least from my perspective.

[corfoto4]

I can confirm that the issue has occurred again. Here are the diagnostics just after waking up from sleep:

rynvis-diagnostics-20221220-0942.zip

 

I have also tried restarting the docker service as suggested and that did not fix the issue. I think I will have to go down the path of shutting the server down rather than sleeping.

[ich777]

5 hours ago, corfoto4 said:

I can confirm that the issue has occurred again. Here are the diagnostics just after waking up from sleep:

it fails to initialize fully:

Dec 20 09:42:25 Rynvis kernel: NVRM: GPU at PCI:0000:09:00: GPU-e273a6ee-127f-8286-a1ea-e21a0308187c
Dec 20 09:42:25 Rynvis kernel: NVRM: GPU Board Serial Number: 1425021077950
Dec 20 09:42:25 Rynvis kernel: NVRM: Xid (PCI:0000:09:00): 31, pid=13315, name=modprobe, Ch 00000003, intr 10000000. MMU Fault: ENGINE HOST6 HUBCLIENT_HOST faulted @ 0x1_01030000. Fault is of type FAULT_PDE ACCESS_TYPE_READ

 

AFAIK this is a common issue and is around for many years.

This bug can also happen at run time while the card is in use and will usually cause most systems to hang or leave X in a hanging state.

 

Nothing I can really do about since this is a driver thing that needs to be fixed by Nvidia.

 

5 hours ago, corfoto4 said:

I have also tried restarting the docker service as suggested and that did not fix the issue. I think I will have to go down the path of shutting the server down rather than sleeping.

I would really recommend that you go that way.