Your half right, Its not so much that the seals swell but conventional oil is thicker and can actually seal small gaps where the seals may be goin bad. I hate the Castrol commercials that say synthetic wont cause leaks. Technically they are but if you have a leak that was sealed by the heavier conventional oil the synthetic can leak through when that oil seal is worn down.If your car has been used to conventional oil all its life, keep using it. If synethic, keep using that.
If you change from one to the other, the swell of all your seals changes a little bit. You have a decent chance of developing an oil leak. I'm not saying it WILL happen, but its a step towards causing leaks.