My friends are telling me that I am getting a little too muscular and now I don't know what workouts to do

I have been a competitive swimmer my whole life so I always had for the most part muscles so when I went on a new weight training program it didn't take long to get very muscular in the arms, chest, back area. Now I want to scale down but I don't want flabby arms because my arms are used to being a little on the big side. Right now I am just concentrating on legs, light swimming, kickboxing and doing very light weights such as 5 lbs.