I use Microsoft Security Essentials and it doesn't slow down my 2005 Dell Notebook. I wouldn't do what you do because the last time I got a virus, it was from a safe site. Believe or not, a few deviantArt users like me got the Security Tool virus back in 2009. Some say that if you use a browser like Firefox with an add-on like AdBlock, you will be safe from it.
It does slow down your computer though - maybe you just don't notice it, but it does. Any program that runs in the background uses resources, and thus slows down your computer.
If decide to try survive without Anti-virus software, I'll keep that in mind. Anything else I should do to keep me and my PC safe? Don't worry I have common sense, I don't do anything stupid like give out my personal info or etc. I'm not sure what else I should do, the bad guys are innovating all the time.
If you really want to be safe, get rid of windows and start using a secure OS like linux or BSD. Windows (any iteration of it) is full of security holes which the malware makers are exploiting, if you use linux or BSD you won't need any antivirus because they are systems that are built with security in mind in the first place.