Laying off food service workers with families is the hardest part of the job, restaurant owners say.