Cherry pick from dd35c27850: Cleaned up testconfig caching