I have been told by Pat that the higher voltages do work the DC-DC unit harder because of the higher switching rates.
I say don't give it a worry.
Can you define "working harder" because from my testing, it appears that for a given load, higher input voltages (to a degree) yields highest efficiency.