I think it’s just an assumption based on the mode of society at that time in history. If it was built in the 12th century it was built by what we would now consider slaves. In the 1100s the land was divided into fiefs and the lord of that land considered the people who lived and worked on the land as part of that land: serfs. Unless this bridge was an exception to the rule, then serfs would have undertaken all the labour that got it built.
That’s why they stopped at three. The fourth bridge always takes forever to paint.