I'm no expert, but I'd imagine that it's less dependent on size, and more so on CFM output and things like vibration dampeners.
Above is right also some fans there blades are metal some are plastic. I have 2 6 inch ones and the 6 inch HTGSupply I got is like 10x quieter then the 6inch metal blade one.
Hi.. Check out HTGSupply's website regarding inline fans. They have db readings of the various fans. Here is the link Inline Fans According to the site, the 4" is 47 db and the 6" is 54 db... so yes, the 4" is less noisey than the 6" Good Luck!