It not necessarily a size issue you are grappling with but jealousy and insecurity that inevitably creeps in at the beginning of such situations. I used to worry about the size issue too when we were swinging but never did it actually matter as there is so much more to a sexual experience than size for most women. So relax, work out if both of you are on the same page and practice compersion as much as you can to overcome jealousy. Good luck!