Technically, whether you leave the small connector plugged in or not depends on your choice of connector type. If there are any exposed terminals where they have a chance to touch against anything, I'd just leave them connected. Other than that, it makes no difference.