The famous Jewish mediaeval traveler, Benjamin of Tudela, who visited Jerusalem in the year 1170, found there a community of not more than 200 Jews living “at the very end of the city near the Tower of David.”
Source: Political Report of the Executive of the Jewish Agency. Submitted to the Twenty-First Zionist Congress and the Sixth Session of the Council of the Jewish Agency at Geneva. (p. 38); “Encyclopaedia Judaica”, Vol. VIII, p1151
See also: Benjamin of Tudela and the Stones of the Holy Land, Meir Soloveichik (podcast)