Intel Core i9-13900K with SuperMicro X13SAE-F or Gigabyte MW34-SP0?

[WingmanNZ]

9 hours ago, brainreplaced said:

@WingmanNZ, hello!

I have a same build with x13sae-f + intel 12700. Did you try to start in legacy mode? I want to passthrough my gtx 1660 to vm but it is unsuccessful and I dont see any option in bios.

I'm using UEFI only. I don't have a GPU installed yet for VM pass through. I can try this week with a Quadro P2000. Do you mind checking your idle power drawer with just your cpu/ram/motherboard connected to the power supply?

[brainreplaced]

5 hours ago, WingmanNZ said:

I'm using UEFI only. I don't have a GPU installed yet for VM pass through. I can try this week with a Quadro P2000. Do you mind checking your idle power drawer with just your cpu/ram/motherboard connected to the power supply?

Please write as soon as you try. The power supply is ok. The video card is recognized in unraid, but I can’t install the driver and add it to the vm too.

[WingmanNZ]

7 hours ago, brainreplaced said:

Please write as soon as you try. The power supply is ok. The video card is recognized in unraid, but I can’t install the driver and add it to the vm too.

Try ticking all the boxes for that IOMMU group, then restarting Unraid, you can then assign the GPU to your VM and start your VM.

[brainreplaced]

13 hours ago, WingmanNZ said:

Try ticking all the boxes for that IOMMU group, then restarting Unraid, you can then assign the GPU to your VM and start your VM.

Already tried. The problem seems to be that I have the GPU installed in the second psi-e slot and not in the first one. I can’t test that because Im using noctua nh-d15, which closes the first slot. I ordered a new cooler, I'll try to rearrange it when it arrives (but it's not fast).

 

[QukI]

1 hour ago, brainreplaced said:

Already tried. The problem seems to be that I have the GPU installed in the second psi-e slot and not in the first one. I can’t test that because Im using noctua nh-d15, which closes the first slot. I ordered a new cooler, I'll try to rearrange it when it arrives (but it's not fast).

 

 Note 2

 

supermicro counts pcie slots from bottom to top

Edited January 23, 2023 by QukI

[brainreplaced]

58 minutes ago, QukI said:

 Note 2

 

supermicro counts pcie slots from bottom to top

Sorry, its my mistake. Gpu is in SLOT4

[cpthook]

On 1/3/2023 at 4:24 PM, saf said:

Couple quick comments based on my current use of the Supermicro X13sae-f.

 

• I purchased on the last week of December and it shipped with the 2.0 bios.
• I am able to use the Intel i9 13900K without issues.  This was a nice surprise to include the 2.0 BIOS I noted above

So on one had it shipped with the required BIOS to support the 13900k CPU.  

 

 

Thanks for this information about the BIOS 2.0 requirement for the 13900k CPU.  I'm curious if anyone has purchased this board more recently and if they had the same luck with the board being shipped with v2.0 BIOS?  If so, which vendor did you purchase from?  I purchased from NewEgg today so I hope I'm just as lucky.  SM want's $27 for Out of Band (OOB) license update even though as of today the board web page shows 12th/13th Generation Intel support as a Key Feature.

 

Also, is there a recommendation for known working ECC RAM for this board?

 

Thanks

 

Edited February 11, 2023 by cpthook

[FCruz2489]

I purchased it from Newegg in the past month and it came with 2.0 installed. I didn’t install a 13th gen CPU, but assume it would have worked.

 

also, supermicro has some compatible ecc memory on their US online store. 32gb sticks were in store for about $160 last I checked. 

Edited February 11, 2023 by FCruz2489
Additional info

[cpthook]

38 minutes ago, FCruz2489 said:

I purchased it from Newegg in the past month and it came with 2.0 installed. I didn’t install a 13th gen CPU, but assume it would have worked.

 

also, supermicro has some compatible ecc memory on their US online store. 32gb sticks were in store for about $160 last I checked. 

 

Thanks for the response but I checked the SM eStore already and they are out of stock.  Only RDIMM available and not validated with this board.  Was hoping for other options.  I purchased non-ECC for now until available.  

[spamalam]

Finally got round to buying this a few weeks back and just assembled it:

• i9 13900k
• 128GB ECC memory - 4x KSM48E40BD8KM-32HM (gets clocked down)
• Supermicro X13SAE-F , Version 1.02
• 3x WD 850X with Heatsyncs
• I used my old SAS card: Broadcom / LSI SAS3224 PCI-Express Fusion-MPT SAS-3 - this only worked in Slot 5
• Quad PCI 3.0 NVMe Card with 4x WD 850X - this worked fine in Slot 7 - no bifurcation on this board so needed this to use more m.2s, note its limited to 3500MB/s per drive.

The motherboard shipped with version 2.0 of the bios, so no need to flash or mess around:

Supermicro X13SAE-F , Version 1.02

American Megatrends International, LLC., Version 2.0
BIOS dated: Mon 17 Oct 2022 12:00:00 AM CEST

 

Everything worked out of the box, I enabled modprobe i915 in  /boot/config/go and  passed --device=/dev/dri  through to Plex and can see it using hw transcoding now.  Not much config needed:

 

Edited February 21, 2023 by spamalam

[scs3jb]

And I also just built out an AsRock W680D4U-2L2T/G5 with an i9 13900k and 128GB DDR.  Runs a bit hotter but ipmi / remote environment is better than supermicro and its got PCI-E 5.0, but less nvme slots and only PCI-e 3x for that.    For some reason it lists the base clock at 2970MHZ instead of 3000MHZ

 

Currently battling with the m.2 not being detected, but it seems to be compatible and working fine with unraid apart from that.

 

 

[VaiZore]

On 2/21/2023 at 7:03 PM, spamalam said:

Finally got round to buying this a few weeks back and just assembled it:

• i9 13900k
• 128GB ECC memory - 4x KSM48E40BD8KM-32HM (gets clocked down)
• Supermicro X13SAE-F , Version 1.02
• 3x WD 850X with Heatsyncs
• I used my old SAS card: Broadcom / LSI SAS3224 PCI-Express Fusion-MPT SAS-3 - this only worked in Slot 5
• Quad PCI 3.0 NVMe Card with 4x WD 850X - this worked fine in Slot 7 - no bifurcation on this board so needed this to use more m.2s, note its limited to 3500MB/s per drive.

The motherboard shipped with version 2.0 of the bios, so no need to flash or mess around:

Supermicro X13SAE-F , Version 1.02

American Megatrends International, LLC., Version 2.0
BIOS dated: Mon 17 Oct 2022 12:00:00 AM CEST

 

Everything worked out of the box, I enabled modprobe i915 in  /boot/config/go and  passed --device=/dev/dri  through to Plex and can see it using hw transcoding now.  Not much config needed:

 

 

@spamalam Thank you for posting your updates. As a fellow UK unraid user with eyes on the X13SAE-F and 13th gen, this thread has been very useful.

Where did you buy your X13SAE-F from in the UK that came with the 2.0 BIOS? Lambda-tek or somewhere else?

[mauriceatkinson@btconnect.]

With the X13SAE-F being such a powerful motherboard it is a shame there is no support for 10gbe that I can see.  

[blacklight]

I stumbled upon this post, because I was also planning to use the X13SAE-F with an i9 -13900KS, but a few questions came up:

-        Isn‘t the 125W TDP of the X13 a problem when using any 13th i9 except the 13900T ? I am planning to use a little bit stronger then the Tray variant, because I also want to run a few VMs in my homelab, but just if it is a good performance gain considering potential boost advantages  from the normal 13900  (still 219W ? problematic ?) If the advantage is just minimal I would buy the tray variant for better efficiency. Or maybe undervolt the K/KS

-        also one idea was to use all PCIe lanes (there are a few nice gaming mainboards for this – BUT no ECC ☹):

o   SLOT7: first GPU probably RTX3060ti or Quadro K4200

o   SLOT5: a SAS, I didn’t make up my mind about one, but I would prefer a x4 to hook up a bunch of HDDs and use the onboard SATA for faster sata ssds| I also wanted to try a riser cable for slot 5 bceause of the GPU. Good/bad idea ?

o   SLOT4: a second GPU: definitely a Quadro K4200

o   SLOT2: possibly a Intel X550-T2 NIC or similar, the idea was to utilize both 10Gb/s ports, which should add up over the PCIe 3 x4 if I calculated this correctly

Hopefully I didn’t overlook any potential/initial problems. As you see I want to put the NIC and SAS on x4 lanes, so that I don’t waste the two x8 lanes, where I want to put my GPU -> Why ? Because I have them lying around and I want to utilize them in my home lab (Unraid passtrough -> one for gaming one for CAD). But first I want to ensure if it isn’t a significant disadvantage to put the SAS and NIC on slower PCIe lanes ? Does someone have experience with that ? I am btw fine with two x8 GPUs instead of one x16, the difference is pretty small. 

 

I would appreciate any feedback about the CPU and PCIe layout. Thanks in advance.

 

Also I stumbled over the Gigabyte MW34-SP0 rev. 1.1 (as probably most of you guys) and I had quite a few concerns about this mainboard:

-        “only” DDR4

-        I found no max CPU TDP for this board

-        one less PCIe lane

->     ok one more m2 place, but the wifi one is useless + I have to disable four SATA ports for the fourth m2 slot-> (don't know if this is true, see below) together with one less PCIe for a potential SAS this is quite a bummer

-        one idea would be to use a m2 to sata adapter, but it is a waste of precious NVME speeds and also there could be compatibility/ connector count (there are just a small number of sata ports on it) problems

-        same goes for m2 to 10gb lan -> also seems a waste

Can anybody still recommend this board ?

Obviously, I targeted ECC as a K.O. criteria, as you can see through the mb selection (are there any other boards ? didn’t find any ATXs next to the mentioned ones):D, because I want to maximize stability. The planned machine will also run virtualized TrueNAS with 8 drives in Raid6 and will be secured with a APC UPS (planed is 24/7). I need probabbly two small m2s for the LOG cache of TrueNas so the Gigabyte would be better in this case. Or are there any 1x m2 to 2x smaller m2 boards  the ssds only need to fit 32gbs thats a huge waste for one slot each :"( (RAID1 is a K.O. criteria for these two NVMEs).

I would also appreciate comments on the general setup. And sry for the long post, but I wanted to give the full context 😊

Edited May 10, 2023 by blacklight

[Lolight]

5 hours ago, blacklight said:

I have to disable four SATA ports for the fourth m2 slot

I couldn't find any reference to this limitation in the board's manual.

[blacklight]

13 hours ago, Lolight said:

I couldn't find any reference to this limitation in the board's manual.

I am sry. It could be the case that I misunderstood the wiring diagram (in the manual and also on the website). It shows a switch for the two PCIe slots (16/0 or 8/8) and the 4x sata slots + the m2.

Assuming they both work they same it can either operate in one mode. I assumed this goes for passing through the sata if no m2 is plugged in. Or is it actually splitting the lanes, so that just the badnwith of the m2 decreases if 1 to 4 satas are plugged in ?

My assumption was based on the keyword switch that either enables a splitter or disables it and therefore either lets you access the m2 or sata.

I also found noting in the documentation about that, I just guessed at this point.

Do you use this board ? And if so, can you share some experiences with it (actual TDPs, connections -> what hardware you use)?

Thanks in advance !

[Lolight]

8 hours ago, blacklight said:

It shows a switch for the two PCIe slots (16/0 or 8/8) and the 4x sata slots + the m2.

Do you use this board ?

Yeah, I did notice that switch but couldn't really come to any conclusion since I don't have any knowledge of its exact function.

Have they somehow managed to miss the M.2/SATA-ports lane limitations and failed to mention it on both the webpage and manual??

Who knows.

It's actually quite possible judging by Gigabytes CS service responses as related to rev. 1.0 of the board.

rev. 1.0 turned out to be quite a disaster.

The question about the potentially disabled SATA ports scenario has already been asked a long while ago on this very active thread and got no responses.

 

https://forum.level1techs.com/t/gigabyte-mw34sp0-motherboard/189112

 

No, I don't own the board.

[blacklight]

Thanks for the reply. 
 

So that’s nice … both mainboards have a detailed documentation, but they still don’t answer the simplest questions  

why aren’t these workstation mainboards like normal gaming mainboards ? They are extremely simple to understand, why can’t they even communicate the TDP ? That’s the simplest detail to mention in the compatible CPU part of the documentation.

 

Anyway, if someone could explain to me how there are so many people putting the 200watt+ K cpus into the X13SAE that would be nice.

 

Thanks in advance

[NathanR]

On 5/11/2023 at 6:28 AM, blacklight said:

Anyway, if someone could explain to me how there are so many people putting the 200watt+ K cpus into the X13SAE that would be nice.

 

Thanks in advance

 

So I don't know why people do what they do.

What I can say is it is unlikely to matter what the mobo says it can support TDP wise.

 

Rationale for the above statement:

1) 125W TDP is just a classification, real metrics will change depending on workload

2) Average power draw will be ~114w. Sauce: https://www.techpowerup.com/review/intel-core-i9-13900k/22.html

3) 8-pin CPU connector can deliver 235w so the mobo isn't power limited. Sauce: https://www.overclock.net/threads/gpu-and-cpu-power-connections.1773088/

4) Power Components will work until their thermal limits are exceeded. Meaning the mosfets will deliver more current (power) than they are designed for until they reach thermal limiting. Sauce: https://www.infineon.com/dgdl/ir3550.pdf?fileId=5546d462533600a4015355cd7c831761 (note, this is a generic VRM, I do not know what VRM the X13 uses)

4A) servers usually have forced air cooling...thus the thermal headroom on the heatsink is very little because it is expected there will be hundreds of CFM moving across the mobo... Home server will cause thermal bottle-necks.

5) Bench/Stress peak power ratings are often meaningless, (albeit fun, interesting, useful to find the limits) for what we would typically use the 13900k For (server, docker, VM, storage)

 

X13SAE-F mini-review/writeup

https://www.servethehome.com/supermicro-x13sae-f-intel-w680-motherboard-mini-review/3/

 

On 2/21/2023 at 1:03 PM, spamalam said:

Finally got round to buying this a few weeks back and just assembled it:

• i9 13900k
• 128GB ECC memory - 4x KSM48E40BD8KM-32HM (gets clocked down)
• Supermicro X13SAE-F , Version 1.02
• 3x WD 850X with Heatsyncs
• I used my old SAS card: Broadcom / LSI SAS3224 PCI-Express Fusion-MPT SAS-3 - this only worked in Slot 5
• Quad PCI 3.0 NVMe Card with 4x WD 850X - this worked fine in Slot 7 - no bifurcation on this board so needed this to use more m.2s, note its limited to 3500MB/s per drive.

The motherboard shipped with version 2.0 of the bios, so no need to flash or mess around:

Supermicro X13SAE-F , Version 1.02

American Megatrends International, LLC., Version 2.0
BIOS dated: Mon 17 Oct 2022 12:00:00 AM CEST

 

Everything worked out of the box, I enabled modprobe i915 in  /boot/config/go and  passed --device=/dev/dri  through to Plex and can see it using hw transcoding now.  Not much config needed:

 

Thank you! This is awesome. You probably saved me 20+hrs of work.

 

I am considering moving my 5900X build to intel for two reasons.

1) iGPU for PLEX transcoding (my parents, wife-parents, sister, etc. all need things transcoded; only direct play is in-house to my NVIDIA shield)

2) lower power consumption (more efficient processors for dockers & server VMs)

 

Build I'm considering:

• Mobo: MBD-X13SAE-F-O
• 64GB: MEM-DR532MD-EU48
• 13700k or 13900K
• Re-use everything else from my 5900X build

Edited July 1, 2023 by NathanR
Added sauce

[mikeyosm]

On 7/1/2023 at 9:22 PM, NathanR said:

 

So I don't know why people do what they do.

What I can say is it is unlikely to matter what the mobo says it can support TDP wise.

 

Rationale for the above statement:

1) 125W TDP is just a classification, real metrics will change depending on workload

2) Average power draw will be ~114w. Sauce: https://www.techpowerup.com/review/intel-core-i9-13900k/22.html

3) 8-pin CPU connector can deliver 235w so the mobo isn't power limited. Sauce: https://www.overclock.net/threads/gpu-and-cpu-power-connections.1773088/

4) Power Components will work until their thermal limits are exceeded. Meaning the mosfets will deliver more current (power) than they are designed for until they reach thermal limiting. Sauce: https://www.infineon.com/dgdl/ir3550.pdf?fileId=5546d462533600a4015355cd7c831761 (note, this is a generic VRM, I do not know what VRM the X13 uses)

4A) servers usually have forced air cooling...thus the thermal headroom on the heatsink is very little because it is expected there will be hundreds of CFM moving across the mobo... Home server will cause thermal bottle-necks.

5) Bench/Stress peak power ratings are often meaningless, (albeit fun, interesting, useful to find the limits) for what we would typically use the 13900k For (server, docker, VM, storage)

 

X13SAE-F mini-review/writeup

https://www.servethehome.com/supermicro-x13sae-f-intel-w680-motherboard-mini-review/3/

 

 

Thank you! This is awesome. You probably saved me 20+hrs of work.

 

I am considering moving my 5900X build to intel for two reasons.

1) iGPU for PLEX transcoding (my parents, wife-parents, sister, etc. all need things transcoded; only direct play is in-house to my NVIDIA shield)

2) lower power consumption (more efficient processors for dockers & server VMs)

 

Build I'm considering:

• Mobo: MBD-X13SAE-F-O
• 64GB: MEM-DR532MD-EU48
• 13700k or 13900K
• Re-use everything else from my 5900X build

Did you end up getting the Supermicro board? Thinking of parting with my cash on this or the Asus W680-M Ace SE

[blacklight]

On 10/10/2023 at 11:30 AM, mikeyosm said:

Did you end up getting the Supermicro board? Thinking of parting with my cash on this or the Asus W680-M Ace SE

I got the  W680-M Ace SE with the IMPI card actually  it has a better pcie layout in my opinion. Unraid works without a problem.
Specs:

i9 - 13900KS

128ECC RAM (have to look up the brand if you need it - was quite a pain to find ECC -> its recognized even in VMs /Singe bit ECC)

 

PCI Layout:

- PCI x1: IPMI Card

- M2 (CPU): empty for future NVME

- PCIe x8 (gen5 - bifur): Quadro K4200 (for VM testing)

- PCIe x8 (gen5 - bifur): RTx3060TI (for VM testing)

- PCIe x4 (gen3): Broadcom 9300-16i (HBA) with 14 hard drives attached

- PCIe x4 (gen3): Mellanox ConnectX-3 Pro for max 20Gbit

- 2x M2 (x4 - chipset): optane 900P (for NAS testing, using a adapter M2 to fullsize PCIe)

 

For now everything is recognized and booting Unraid was a charm, unfortunately I have problems with virtualizing a NAS, but for now it seems like a software problem. I will update the post if I find a incompatibility (IOMMU problems or similiar). The IOMMU layout seems solid btw, no suprises there for now, just that the two onboard SATA groups (1: 4x sata, 2: SLIM-SAS which has a PCIe mode) are in the same group, but I already thought that this would be the case. 

 

And also the IPMI is a nice bonus in the combo package. Its a little bit expensive compared to a custom one (have no experience there) but it just works.

I am currently on another continent for a year and the access and controlling power over VPN with the IPMI card is a real relieve.

 

I attached a screenshot of the IOMMU groups, because I had trouble finding them online. Hope they will be useful. 

Let me know if you need more information

 

[mikeyosm]

16 hours ago, blacklight said:

I got the  W680-M Ace SE with the IMPI card actually  it has a better pcie layout in my opinion. Unraid works without a problem.
Specs:

i9 - 13900KS

128ECC RAM (have to look up the brand if you need it - was quite a pain to find ECC -> its recognized even in VMs /Singe bit ECC)

 

PCI Layout:

- PCI x1: IPMI Card

- M2 (CPU): empty for future NVME

- PCIe x8 (gen5 - bifur): Quadro K4200 (for VM testing)

- PCIe x8 (gen5 - bifur): RTx3060TI (for VM testing)

- PCIe x4 (gen3): Broadcom 9300-16i (HBA) with 14 hard drives attached

- PCIe x4 (gen3): Mellanox ConnectX-3 Pro for max 20Gbit

- 2x M2 (x4 - chipset): optane 900P (for NAS testing, using a adapter M2 to fullsize PCIe)

 

For now everything is recognized and booting Unraid was a charm, unfortunately I have problems with virtualizing a NAS, but for now it seems like a software problem. I will update the post if I find a incompatibility (IOMMU problems or similiar). The IOMMU layout seems solid btw, no suprises there for now, just that the two onboard SATA groups (1: 4x sata, 2: SLIM-SAS which has a PCIe mode) are in the same group, but I already thought that this would be the case. 

 

And also the IPMI is a nice bonus in the combo package. Its a little bit expensive compared to a custom one (have no experience there) but it just works.

I am currently on another continent for a year and the access and controlling power over VPN with the IPMI card is a real relieve.

 

I attached a screenshot of the IOMMU groups, because I had trouble finding them online. Hope they will be useful. 

Let me know if you need more information

 

Great news, thanks for the update. Do you mind showing a screenshot of the complete IPMI sensors please? I would like to see what sensors are detected.

[casperse]

On 10/14/2023 at 9:49 PM, blacklight said:

I got the  W680-M Ace SE with the IMPI card actually  it has a better pcie layout in my opinion. Unraid works without a problem.
Specs:

i9 - 13900KS

128ECC RAM (have to look up the brand if you need it - was quite a pain to find ECC -> its recognized even in VMs /Singe bit ECC)

 

PCI Layout:

- PCI x1: IPMI Card

- M2 (CPU): empty for future NVME

- PCIe x8 (gen5 - bifur): Quadro K4200 (for VM testing)

- PCIe x8 (gen5 - bifur): RTx3060TI (for VM testing)

- PCIe x4 (gen3): Broadcom 9300-16i (HBA) with 14 hard drives attached

- PCIe x4 (gen3): Mellanox ConnectX-3 Pro for max 20Gbit

- 2x M2 (x4 - chipset): optane 900P (for NAS testing, using a adapter M2 to fullsize PCIe)

 

For now everything is recognized and booting Unraid was a charm, unfortunately I have problems with virtualizing a NAS, but for now it seems like a software problem. I will update the post if I find a incompatibility (IOMMU problems or similiar). The IOMMU layout seems solid btw, no suprises there for now, just that the two onboard SATA groups (1: 4x sata, 2: SLIM-SAS which has a PCIe mode) are in the same group, but I already thought that this would be the case. 

 

And also the IPMI is a nice bonus in the combo package. Its a little bit expensive compared to a custom one (have no experience there) but it just works.

I am currently on another continent for a year and the access and controlling power over VPN with the IPMI card is a real relieve.

 

I attached a screenshot of the IOMMU groups, because I had trouble finding them online. Hope they will be useful. 

Let me know if you need more information

 

What CPU cooler did you get in order to have the IPMI in the first slot?

[ByteNick]

On 2/21/2023 at 9:03 PM, spamalam said:

Finally got round to buying this a few weeks back and just assembled it:

• i9 13900k
• 128GB ECC memory - 4x KSM48E40BD8KM-32HM (gets clocked down)
• Supermicro X13SAE-F , Version 1.02
• 3x WD 850X with Heatsyncs
• I used my old SAS card: Broadcom / LSI SAS3224 PCI-Express Fusion-MPT SAS-3 - this only worked in Slot 5
• Quad PCI 3.0 NVMe Card with 4x WD 850X - this worked fine in Slot 7 - no bifurcation on this board so needed this to use more m.2s, note its limited to 3500MB/s per drive.

The motherboard shipped with version 2.0 of the bios, so no need to flash or mess around:

Supermicro X13SAE-F , Version 1.02

American Megatrends International, LLC., Version 2.0
BIOS dated: Mon 17 Oct 2022 12:00:00 AM CEST

 

Everything worked out of the box, I enabled modprobe i915 in  /boot/config/go and  passed --device=/dev/dri  through to Plex and can see it using hw transcoding now.  Not much config needed:

 

I am trying to build something very similar.

What king of CPU cooler have used?

I need to use all PCI's, and I'm afraid buying one too big would kill the closed to the CPU one. 

[blacklight]

On 11/13/2023 at 8:19 AM, casperse said:

What CPU cooler did you get in order to have the IPMI in the first slot?

Oh man, had a lot of things to do so I wasn't able to visit the forum in the last month, sry. Probably you don't need it anymore but I used the asus ryujin 2 (Display is unfortunately useless in my chassis). I chose this AIO, because it's compact with a big radiator and it cooles the VRM with an additional fan on the head of the cooler. Also I found decent tests and reviews, which was actually not that easy for other AIOs in terms of thermal performance.

 

BTW: the IPMI is designed to be placed in the first slot. My guess is it is going to work in any PCIe, but considering the cable management I don't know if it would fit any layout. It is pretty slim tho. 

 

 

Edited December 11, 2023 by blacklight