Raff was born on 22 May 1822 in Lachen, Switzerland. He was the first child of the community's school teacher, a German refugee from Württemberg who had fled the Napoleonic wars. His mother Katherina, née Schmid, was the daughter of a farmer and landlord of the Ox Inn in the town. Named Joseph Joachim after his father and his maternal grandfather, he was christened on the day of his birth as was the Catholic custom in those days.
The boy came into the world to the sound of peeling bells - it was Pentecost Monday. In later life Raff was fond of saying that this welcome decided his musical vocation. As he grew up, an abiding musical stimulus was the sound of the church bells in the evening, mingling with the gentle ebb and flow of the lake.
In 1824, soon after the birth of a baby sister for Joachim, the family moved to nearby Rapperswil, where Joseph Raff's reputation had earned him promotion to the Boys High School there. His health soon deteriorated, however, and the family returned to Lachen whilst he recuperated - Katherina opening a shop selling glass and porcelain to make ends meet. Restored to health, her husband was lucky to be able to return to his former teaching job in the lakeside town.
Growing up in Lachen's beautiful location, nestling beside Lake Zürich under the foothills of the Swiss mountains, Raff aquired the sensibility to his surroundings which he was to retain in adulthood. He would confess in later years that he was unable to appreciate the attraction of a flat land and called water "the soul of a landscape".
Although blessed by his surroundings, his was not a particularly happy childhood. His father recognised early on his intellectual ability and sought to foster it with a strict and excessive study regime, aiming to turn him into a child prodigy. At six he could read German, at eight translate Latin. His father was a keen musician and the lad was soon playing the violin and organ and singing in the church choir. By ten years of age he was regularly replacing his frequently ill father as the church's organist.
Although he learned easily and was self-critical, he was still severely thrashed by his father whenever he made a mistake. Physical punishment in schools was normal in those times but Raff's treatment seems to have been unusually severe. The beatings continued until he was 15 when, after a particularly violent attack, he decided to starve himself to death rather than suffer any longer. Only when his life was clearly in danger did his father promise to end his use of the rod and Raff suffered no more after that. His father's agression was not just reserved for young Joachim. On one occasion the child intervened in an argument between his parents, when his father began to attack Katharina.
Despite the violence, there was a strong bond between father and son, founded in their common love of music and nature. When Raff pleased his father he would be rewarded with a stroll into the countryside and during these idylls they would talk about the father's Swabian homeland in Württemberg. As a result, he grew up thinking of himself as an ex-patriot, although he never lost his attachment to the country of his birth.
A strong influence on him was his maternal grandfather Joachim Schmid, who was an influential local leader and Cantonal Stattholder - chairman of the Landesgemeinde (the local assembly). He had the habit of listening in on the political discussions that frequently took place by his grandfather's hearth and it was here that he aquired an appetite for argument which never left him. His mother and grandmother Cäcelia (her name latterly regarded by Raff as another musical omen) were warm, calming influences in this masculine world.
He was in some respects a lonely boy. His closest confidant had been the Raff's second child - the little sister born just before the brief Rapperswil episode. She died in childhood, however, and his introspective bent isolated him from the younger surviving siblings (Kaspar, Maria Antonia, Alonsia, Selina and Peter) and from other children of his age. A childhood lacking the rough and tumble of robust play resulted in him growing up to be the antithesis of a sturdy country lad. An inability to swim nearly resulted in his death. He fell into the lake trying to retreive a treasured toy and was unconscious by the time he was rescued by a passing fishing boat. He was left with the quiet conviction that he must be destined for great things, otherwise God would have let him die.
At 12 his father took him to Rottenburg in his native Swabia where he was enrolled to study at the Gymnasium. Freed of Joseph Raff's overbearing guidance the boy blossomed in the intellectual atmosphere and made many friends, although he kept few in later life. He was also able to establish contact with his paternal family, who had remained in Swabia. His father's sisters and cousins doted on "Jochimle" as did his 71 year old grandfather. Once term ended, he would don a knapsack and hike on foot back to his Swiss home.
Raff's second brush with death came on one of these homeward treks. During a winter snowstorm he became exhausted and, resting, unwittingly fell asleep. Luckily, a forester found him and revived him from his potentially fatal slumber.
After four years in Rottenburg, political events in Switzerland provoked Raff's return there. Joachim Schmid became embroiled in a bitter political controversy which resulted in his son-in-law losing his teaching post in Lachen, although he was personally uninvolved in the dispute. Joseph Raff found a new position as music teacher at the Jesuit College in Schwytz, his wife took in students for food and board and his eldest son returned to live with them in Autumn 1838.
He thrived in the "harmonically balanced intellectual atmosphere" which he found there. The teaching medium was primarily Latin, in which he was already fluent. Finding homework difficult with a house full of boistrous younger brothers and sisters, he developed the habit of napping during the day and studying at night, sitting with his feet in a tub of icy water so that he kept awake.
He was evidently a good student in Schwyz, being particularly adept at mathematics, rhetoric, poetry and history. His French was poor - something he regretted in later life. Indeed, at that time even his written German was not really fluent and his handwriting also exhibited a "heavy, pedantic stroke".
Raff idolised the college's Prefect, Father Waser, who for him epitomised the ideal Catholic priest. In Lachen he had been a "dreamily pious" child and his Catholic belief was deeply instilled in him. His religion wasn't without its childish crises, however. On one occasion he could not bring himself to kiss the crucifix because so many lips had kissed it before his own. Another time he prayed to a saint for relief from a toothache and was mortified when none came.
As he developed under the stimulating regime of Schwyz, it became important to him to leave home and earn a living. In 1840, as the college's star Latin pupil, he was recommended to accompany the Papal Nuncio Monsignor Gizzi as interpreter on a visit to St. Gallen. There the 18 year old impressed the locals and, eager to keep his services in the area, the authorities awarded him a teaching qualification. On 18 October 1840 Raff was appointed teacher at the Upper Primary School in Rapperswil, near his Lachen birthplace, with a generous salary of 470 florins a year.
His childhood was over.