Is it wrong of me to tell hubby that his brother has to go, that we can't afford another mouth to feed.....?

Well his brother has been here for 2 months and he is going through hard split up with his ex girlfriend) Well me and hubby have 2 boys ourselves and we both have a full time job,we have hard enough time making sure that we are taken care of... His brother is 28 and he does do side jobs to make money. He doesn't help with nothing and he doesn't even help ex girlfriend financially with their kids. All his money goes on beer and cigarettes. To me he is very selfish it's always about what he wants. He gets his kids every weekend and then tries to keep them at weeks at a time, I don't mind this at all I love them to death,But its hard to keep groceries in the house cause my Bil and his kids waste alot of food and I don't allow my boys to do that at all. He want even buy groceries for when they come and visit. I told hubby somewhat how I feel but don't want to hurt his feelings cause that is his brother. But I can't keep taking away from my kids for them. My youngest had to give up his room for him which I didn't like at all. And when his kids come over he don't discipline them at all he lets his youngest son talk to us like we are dogs and he treats my youngest that way to. So guess my question would be .... Would it be wrong to tell hubby to choose me and our kids or his brother that want help out?