It really depends on lots of factors: ability, speed, rest, etc....If it's taking you that long to swim a mile your stroke needs some work. Think of it as the foundation of the house: a strong straight foundation will support everything about the house. A poor foundation will affect how the doors and windows open & close.
I suggest you work on your technique first. Swimming with poor technique is inefficient. Every inefficient stroke you take will reinforce the bad habits. Correct your technique and you will not only be faster you will feel better. There are a couple ways you can do this: 1. buy the book "total Immersion" and work on the skills yourself. TI also has camps. 2. Get a coach & 3.go to a camp: T.I. has camps, Crossfit endurance has camps.
Before doing CrossFit and going paleo I was swimming 3-3500M (2 miles) in just over an hour depending on the sets. Now I'm following the CrossFit endurance program and swimming less than 1/2 that but with far greater intensity. The results are awesome: I'm faster at all the distances and enjoy swimming more than ever.
Swimming is not only great exercise and tremendously enjoyable it's also a skill that might just save your life!