Just depends on how smooth you want your wood.  I have a 4 inch tall oak board here in my office that goes all the way around my desk top.  I put on no filler.  If I run my hand over it I can feel the pours.   To each there own.  Now if I was doing a clock, table, chair, ect.  something I would be touching or handleing I would use filler.