I was on a site the other day looking for a part for my laptop. The site may have had my part but I never waited to find out. The file size of the images uploaded for the site were humongous; making it impossible to scroll down the page. The site was so poorly constructed that I didn't stick around and went down the superhighway to a better constructed sight.
Your customers will do the same thing that I did. And you might never know you have a problem unless you have a great webteam that properly tests your website across a wide variety of platforms and browsers and you monitor your traffic for problems.
Websites are not the same as the physical locations for your business where you build correctly and leave alone. Even well built sites are subject to changing conditions. A browser update can reek the same havoc on a website that an earthquake can reek on your physical location.
Monitoring metrics like exit points and bounce rate can help you insure that visitors aren't leaving because of something you did or didn't do.