It is the direction in which the objects are moving around us. They can tell that the other stars are moving in such a way that we and they are moving out from a center point.
Except there seems to be no central point. Their vectors move crossways to each other, which is why some galaxies are colliding.
Interestingly enough, IS that something that can be observed?