You’re missing the point that the universe is expanding uniformly. That means two points acceleration away from each other is dependent on their distance apart. The further they are from each other the faster they accelerate from each other.

So GP is right. We measure red shift and infer distance.