In 1878 the Catholic Archbishop of Sydney, Dr Roger Bede Vaughan, a Benedictine, met with Fr Joseph Dalton SJ and asked the Jesuits to found a college for boys in the growing Catholic community. A property known as St Kilda House on the corner of Cathedral Street and Palmer Street was rented for this purpose and its 45 students were admitted on 3 February 1879. The number gradually increased during the year to 115.

In September 1883 the College moved to a property known as Auburn Villa in Darlinghurst. This building was later demolished to make way for St Margaret's Maternity Hospital. The name "Auburn Villa" was changed on purchase to ‘St Aloysius’, the patron of youth. Numbers fluctuated considerably towards the end of the century. On 2 February 1903, the College relocated to its present site at Milsons Point.

As the Community increased, a new wing was constructed and in 1916 an attractive property known as Wyalla, opposite the College in Upper Pitt Street, was purchased. In 1939 some market gardens in Tyneside Avenue, East Willoughby, were acquired to build the College Sports Ground. The number of students after the war increased rapidly and after considering various options, the Jesuits re-developed the College. This was begun in 1961 and existing buildings were demolished and rebuilt in four stages. To celebrate its one hundredth birthday, the College embarked on a fifth stage which was opened in 1981. In 1991 the College purchased the Milsons Point Primary School and constructed the Junior School Campus in Burton Street, Milsons Point. In 1995 the Jesuit community left the main building for a community house at 38 Jeffrey Street enabling the top two floors of the school to be renovated.  In 2011 the College opened a new basketball court, swimming pool and gymnasium at Dalton Hall.

History of the College