Humph. I never noticed that it hurt the buzz any.
But you really shouldn't just keep a piece of orange peel in there either. That's just asking for mold to start chowin' on it.
Just put a piece in it until the weed's moist, then take it out.
If it gets dry again you can plop another peel in there.
If you wanted to get real fancy you could have a little "batters box" where you keep a little bit for the day's smoking or what ever. Then rotate the next day's stash into the "humidor" (bag with orange peel in it) and it's ready the next day.