Assuming you have no bottlenecks due to port multipliers, disks are read in parallel and it is all the same very fast calculation, so number of disks should have no impact due to these factors.
Normally all HDDs are faster on the outer cylinders and slower on the inner due to data density. Same RPM but more circumference on the outer has more data per rotation than the inner lesser circumference. And, the progress goes from outer to inner so slower later on is normal.
Smaller disks are slower than larger disks as a rule due to data density. So as the smaller disks are finished there may be some increased speed due to the (slower) smaller disks not being involved, but that may be offset by the fact that the larger disks remaining are further along in progressing towards the slower inner cylinders.
And, of course, some disks just don't seem to perform as well for some reason.
I suspect you are actually seeing the effects of the slower inner cylinders and not really the effects of the smaller disks being finished. Have you tested the speed of your individual disks using the DiskSpeed docker?