M_Boats: braided nylon size/ stowing boom in cabin rigged

I need to replace my mainsheet and jib sheet. Anyone know what size braided nylon that should be? It looks to me like maybe 1/2 inch, and our store has 5/8 and 3/8. (Maybe the main sheet is 5/8.) The cleats that hold the jib sheets are worn and don't jam well; if I used the 5/8 (assuming it is supposed to be 1/2 inch) would that maybe work better? Also, the main sheet is worn because when I put the boom in the cabin with the rigging attached, there isn't a lot of room at the corner for the lines. I was thinking of cutting out a little bigger opening in either the teak hatch cover for them or in the vertical part of the fiberglass hatch cover. Has anyone done this? Which should I cut? (If either.) Bobbie