What kind of work is carried out by your characters? Is the setting one where machines dominate or one where manual labour is still very necessary? Who carries out the manual labour by the way and, if it is a particular people grouping, how did it come about that they were the ones to do this rather than anybody else?
I would assume the likelihood is that non-magical beings would be the ones picked on for manual labour duties. If so, you would need to show why there are non-magical beings in your setting at all and why it is, with everyone else around them magical, how come it didn't seem to rub off on them?
Do any of your characters work hard deliberately to make themselves so tired they can't focus on other pressing, depressing, personal "stuff"? Do they succed in "forgetting"?