The problem comes in when all that crap floating in the fluid starts plugging things. Then valves don't actuate properly, it starts slipping, shifting funny, etc... Or of greater concern if they plug the filter you can burn up a pump due to a lack of fluid flow. (Very rare mind you.) So that garbage can cause problems. Whether or not to change it is up to you. I'm sure changing it can't hurt, but it may not be necessary. Normally (but not always) a transmission will start giving you subtle hints it's time to clean things out.
For what it's worth, after my bad experience with an automatic, I think I'm going to stick to a manual. There's a hell of a lot less than can go wrong, and a hell of a lot easier to fix. Though going with an auto if I had a killer turbo setup, or to get the 850 ft-lb rating on the Ram isn't out of the question. :shiftyeyes