The Special Olympics, which recognizes the athletic potential of people with intellectual disabilities, is one of the most recognizable and respected charitable organizations in the United States. But without the tragedy sustained by a member of the Kennedy family with an intellectual disability, it may never have gotten its start.

Eunice Kennedy Shriver, one of John F. Kennedy’s sisters, founded the Special Olympics. And some of her inspiration came from a less recognized member of the Kennedy family: her older sister Rosemary.

Rosemary’s story was tinged with tragedy from the moment of her birth in 1918. During her labor, her mother Rose had opted for a home birth assisted by a doctor and a nurse. But when the doctor arrived late for the delivery, the nurse opted to push the nearly-born baby back into the birth canal, holding her there for two excruciating hours rather than allow the baby to be born without the doctor’s assistance. “I had such confidence in my obstetrician,” recalled Rose later. “I put my faith in God and tried to sublimate my discomfort in expectation of…happiness.”

In reality, the decision to hold the baby inside the birth canal had led to tragedy. The pressure had deprived Rosemary of oxygen and led to seizures, learning delays and symptoms of mental illness.

At the time, intellectual disabilities, which were referred to as “mental retardation,” were little understood and stigmatized by a society that viewed people with disabilities as shameful and different. Often, people with disabilities would be put in sanitariums or mental institutions long-term.

Though Rose never spoke of her daughter’s learning delays, which appear to have been mild, she apparently refused to send her to an institution. Ashamed of Rosemary’s difference, the Kennedys simply pretended everything was fine. They sent Rosemary to boarding schools and supervised her carefully when she was at home. Meanwhile, Rosemary continued her education and participated in some public events, but her parents were increasingly concerned about her erratic moods. When they learned she had been sneaking out of her convent school and apparently meeting men at bars, they began to seek medical advice.

Her father, Joe, got it from Walter Freeman, a neurologist known for popularizing the lobotomy. The procedure, in which some nerve pathways in the brain are cut or removed, was touted as a way to control the symptoms of mental illness. It’s unclear if Rosemary was ever informed or asked about the procedure, and Rose later denied having been asked about it, either. In 1941, when Rosemary was 23, she had two holes drilled in her brain while she was awake.

She was never the same again. “After the botched surgery,” wrote Tierney McAfee and Liz McNeil for TIME, “Rosemary was left with the mental capacity of a toddler_—_unable to walk, form a sentence or follow simple directions. She was forced to relearn the most basic of skills, but some would never be recovered.”

The lobotomy may have subdued Rosemary, but it had destroyed her life. She was transferred to a care facility by her father, and for the next 20 years, her family claimed they had no idea where she was. During John F. Kennedy’s presidential campaign, they claimed she was simply reclusive. Only after JFK’s election did they reveal that Rosemary was institutionalized—and they didn’t admit it was because of a lobotomy.

Rosemary’s ordeal “was the fuel that powered the engine that was Eunice Kennedy Shriver,” historian Eileen McNamara told the PBS News Hour. “I think there was some guilt that she was complicit in some way in letting Rosemary languish far from home.” When Joe had a stroke in 1961, Eunice brought Rosemary back into the family and began to pressure her brother to include intellectual disabilities in his policy platforms.

She also became increasingly aware of the impact of intellectual disabilities on other families. In the early 1960s, a woman who was aware of Eunice’s advocacy work for people with intellectual disabilities asked her what to do about her child, who had been rejected from summer camp because he had mental retardation.

“I said, ‘You don’t have to talk about it anymore,” Eunice later recalled. “You come here a month from today. I’ll start my own camp. No charge to go into the camp, but you have to get your kid here, and you have to come and pick your kid up.’” She set up the camp at Timberlawn, a Kennedy property in Maryland, and called it Camp Shriver.

For four years, she invited children with intellectual and other disabilities to her house, free of charge, recruited local high school students to act as counselors, and provided lessons and recreational activities. Children who had always been excluded from group activities thrived in the accepting environment, and Eunice was encouraged by their progress. “I suppose the fact that I had seen my sister swim like a deer—in swimming races—and do very, very well just always made me think that [people with disabilities] could do everything.”

Meanwhile, Eunice went public with her sister’s struggle. “We are just coming out of the dark ages in our handling of this serious national problem,” shewrote in the Saturday Evening Post in 1962. “Twenty years ago, when my sister entered an institution, it was most unusual for anyone to discuss this problem in terms of hope. But the weary fatalism of those days is no longer justified.” Though Eunice did not mention the lobotomy in the article, it is widely considered to have been a watershed for public awareness of the largely dismissed lives of people with disabilities.

Eunice went on to expand her camp, and eventually it evolved into the Special Olympics. Today, the organization serves 5.7 million athletes with intellectual disabilities, holding sports events worldwide and working to increase the visibility and health of people with intellectual disabilities. And though Eunicedenies that Rosemary, who died in 2005, was the direct reason for her involvement with the cause, there’s no doubt that her sister’s struggles and stigma deeply impacted her and shaped her views on a better way to treat people with disabilities.

“Eunice [made] sure that Rosemary felt included,” recalled their brother, Ted. “It was really that spirit that started the Special Olympics.”