I fixed it. Since mailing list and other forums were of no help I tried finding the solution myself. I figured that if I can't bring xfsprogs to my drive, I will bring the drive to xfsprogs.
I had an old OptiPlex laying around so I installed the latest version of Arch Linux which had xfsprogs 5.18 available. Since it only had one power cable I had to power my drive from another PSU while connected through SATA to the OptiPlex.
First run of xfs_repair -nv /dev/sdb1 ended with Segmentation fault.
I then created metadata dump of the filesystem with:
xfs_metadump /dev/sdb1 /home/p/metadump.meta
xfs_mdrestore /home/p/metadump.meta /home/p/metadump.img
and verified it with:
xfs_repair -nv /home/p/metadump.img
Test run on the image was successful so I ran xfs_repair -vL /dev/sdb1 (-L flag was required) and the program finished without any errors. I moved the drive back to my server and everything works fine again.
I also had one machine with Fedora on it with xfsprogs 5.14.2 and it was giving the same assertion failed error as 5.13