How likely is it that my neighbor is a child predator?

I've been afraid to even ask this. I live in a new neighborhood and the man who lives next door to me is always outside with the children in the neighborhood. Especially one of our neighbors, in particular, a little girl who looks about 11 or 12. He is in his 40s and I’m sure he has no children of his own. I’m worried because he’s always alone with her and the other kids and even accompanied them on a walk at night.