How do I deal with this, or how did you deal with similar situations?

Your family does mean well.. However, yes living that close to your family does put alot of stress and just alot of crap you don't need on your relationship because everyone is always in your business I been there and it was horrible! If you and your husband are really ready for another child go for it, it's your life and you can live it the way you want to. They just gave you their opinion it doesn't mean you have to listen to it, sometimes you have to overlook the things your families say in order to make yourself happy. If you do end up having another child then they will be happy and love that child just the same so I wouldn't worry about it. Go for it!