| Building up to that amount of swimming is a good goal, however as you get used to it your body will quickly adapt and you will no longer reap the rewards you are looking for (physically). This is pretty much true with any activity. Its true that diet will play a major role as you are pursuing the new sport. Swimmers typically have a tad more body fat than their land sweating companions. I swam competitively most of my life (for HOURS a day) and I always did until I started adding weights/running/cycling etc. Anything non-swim related.
Swimming also tends to make you extremely hungry because the core muscles are stablized and core temp usually doesn't elevate as high as if you are sweating up a storm running. If you're not prepared for this-be prepared for some major binge sessions the minute you get home which will negate all you're trying to do physique-wise.
You say you don't have time, and that you don't really like the gym. If you're not willing to add a modest weight program to your routine then google some body weight exercises. There are plenty you can do at home. The gym simply has more accountability and you are more likely to actually DO something there then if you come up with a home workout.
If you do weights at the gym focus on 1-2 exercises per major swimming muscle group: Shoulders (include all 3 heads of the deltoid), chest, back (lats) and core muscles. Its not necessary to concentrate on legs. Find a weight thats about 75% of your max and do 2-3 sets of 12-15 per exercise. It should only really take you about 20-30 min if that and will help your swimming goals while building muscle to burn fat.  | 
