Agnes was the thirteenth of fourteen children of her father, the Rev. Professor Baden Powell, then aged 62.
He married his first wife, Eliza Rivaz, dau of Vincent Francis Rivaz of Hackney, on 17 July 1821 in St John's, Hackney; 14 years later she died childless on 13 March 1836 and was buried at St Peter's in the East, Oxford.
18 moths after Eliza's death, the Rev. Prof. married his second wife, Charlotte Pope, dau of William Pope of Hillindene, Middx, on 27 Sep 1837 in Speldhurst. Kent. They had four children,
Charlotte Elizabeth Powell 14 Sept 1838 - 20 Oct 1917 aged 79
Baden Henry Powell 23 Aug 1841 - 02 Jan 1901 aged 59
Louisa Anne Powell 18 Mar 1843 - 1 Aug 1896 aged 53
Laetitia Mary Powell 4 June 1844 - 2 Sept 1865 aged 21
But after but 7 years of marriage, Charlotte died 14 Oct 1844 and was buried at Speldhurst; her children kept their POWELL surname.
The Rev. Prof. was Savilian Professor of Geometry at Oxford University, but his interests extended far beyond that topic; he was the author of a number of books on various scientific subjects. He was a member of the Royal Society, and would travel down from Oxford to attend their lectures in London. He would stay with a friend, another member of the Royal Society, one Admiral William Henry Smyth at Smyth's town house in Cheyne Walk, Chelsea. Smyth normally lived in Stone, near Aylesbury. On one occasion Smyth took his 19-year-old daughter Henrietta Grace to a lecture, and she also stayed in the house, as did Baden. Baden happened to glimpse her ankle as she tended the fire, and was heard to mutter "Perfect ! Just perfect !".
Eighteen months after Charlotte's death, on 10 Mar 1846 in St Luke's, Chelsea, the Rev. Prof. married Henrietta Grace as his third wife.
His four children by his second wife ranged from 8 years old to two. In the 1851 Census, he and Henrietta Grace were registered in Upper Seymour Street, St Marylebone, and of Charlotte's four children, only 9-year-old Baden Henry - the only boy - was with them; the girls were in Ireland with their mother's family. By this time, Henrietta Grace had four boys of her own; all five boys had been born in Oxford.
Henrietta Grace went on to have six more children, of which Agnes was the ninth of the ten; two older brothers and two older sisters died before Agnes was four, so Agnes grew up as the only girl among five brothers.
Agnes's father died on the 11th June, 1860, when Agnes was only 18 months old. In the 1861 Census, Agnes's mother and five of her boys (including the youngest, Agnes's younger brother Baden, were in Stanhope Street, Kensington, but Agnes and her next elder brother Robert seem to have escaped the enumrator !
Her mother, as a tribute to Agnes's father, hyphenated his first name to the surname of herself and her children, the creation of the "Baden-Powell" family. His children by his previous marriage did not change their surname. This was regarded with some scorn by some of the family, who took to referring to her (behind her back) as "Mrs Hyphen".