Swinburne was born at 7 Chester Street, Grosvenor Place, London, on 5 April 1837. He was the eldest of six children born to Captain (later Admiral) Charles Henry Swinburne (1797–1877) and Lady Jane Henrietta, daughter of the 3rd Earl of Ashburnham, a wealthy Northumbrian family. He grew up at East Dene in Bonchurch on the Isle of Wight.

As a child, Swinburne was "nervous" and "frail," but "was also fired with nervous energy and fearlessness to the point of being reckless."

Swinburne attended Eton College (1849–53), where he started writing poetry. At Eton, he won first prizes in French and Italian. He attended Balliol College, Oxford (1856–60) with a brief hiatus when he was rusticated from the university in 1859 for having publicly supported the attempted assassination of Napoleon III by Felice Orsini. He returned in May 1860, though he never received a degree.

Swinburne spent summer holidays at Capheaton Hall in Northumberland, the house of his grandfather, Sir John Swinburne, 6th Baronet (1762–1860), who had a famous library and was president of the Literary and Philosophical Society in Newcastle-upon-Tyne. Swinburne considered Northumberland to be his native county, an emotion reflected in poems like the intensely patriotic "Northumberland", "Grace Darling" and others. He enjoyed riding his pony across the moors, he was a daring horseman, "through honeyed leagues of the northland border", as he called the Scottish border in his Recollections.

After his grandfather's death in 1860, he stayed with William Bell Scott in Newcastle. In 1861, Swinburne visited Menton on the French Riviera, staying at the Villa Laurenti to recover from the excessive use of alcohol. From Menton, Swinburne travelled to Italy, where he journeyed extensively. In December 1862, Swinburne accompanied Scott and his guests, probably including Dante Gabriel Rossetti, on a trip to Tynemouth. Scott writes in his memoirs that, as they walked by the sea, Swinburne declaimed the as yet unpublished "Hymn to Proserpine" and "Laus Veneris" in his lilting intonation, while the waves "were running the whole length of the long level sands towards Cullercoats and sounding like far-off acclamations".

At Oxford, Swinburne met several Pre-Raphaelites, including Dante Gabriel Rossetti. He also met William Morris. After leaving college, he lived in London and started an active writing career, where Rossetti was delighted with his "little Northumbrian friend", probably a reference to Swinburne's diminutive height—he was just five foot four.

Swinburne was an alcoholic and algolagniac and highly excitable. He liked to be flogged. His health suffered, and in 1879 at the age of 42, he was taken into care by his friend, lawyer Theodore Watts, who looked after him for the rest of his life at The Pines, 11 Putney Hill, Putney. His friend took him to the Suffolk coast at the lost town of Dunwich on several occasions in the 1870s.