Allow overriding num reruns from config, added to docs