I'm the worst for wanting to write filler all the time because I love writing fluffy interactions with my characters, but I think while we, as the writer who knows our characters to the bone, love reading it, a more casual audience don't have the same connection. I agree with @joannekwan that it should at least really dig into development or motivation.
My compromise is to tie the 'filler' in as the characters reacting to other events in the story. Say they've just had a tough fight, write them patching themselves up and cooking by the fire while they discuss what to do next. That way you get plot advancement and character development. I have a scene that I love for the interactions of my two characters where one cleans the other's wounds while they discuss how to escape from imprisonment.
Fluff is fun, but for a casual audience that isn't necessarily as in love with your characters as you, fluff can be like eating a mountain of candy floss when you don't really like sweet things.