Decades later, the 86-year-old is back in Germany to visit the two places where he narrowly avoided death.
Mr Ladany, who was born in 1936 in Belgrade, the former Yugoslavia, brought family members to the Bergen-Belsen concentration camp in northern Germany on Saturday to show them the place where he was imprisoned by the Nazis as an eight-year-old boy.
After that, the spry octogenarian will participate in a joint German-Israeli ceremony in Munich on Monday marking the 50th anniversary of the attack on the Olympians by Palestinian terrorists.
Mr Ladany, who competed in the Munich games as a racewalker, strode briskly in lime-green trainers and a beige sun hat as he led his granddaughter, his younger sister and her three children in Bergen-Belsen, which has been turned into a memorial site.
He pointed at a plot of land, now covered by blueberry and heather shrubs and tall birch and pine trees, where barracks No 10 used to stand.
He was held there with his parents and two sisters for about six months in 1944 before they were allowed to leave under a deal negotiated by Hungarian and Swiss Jewish foundations, which paid the Nazis ransom to free more than 1,600 Jews deported from Hungary.
“It’s not a pleasant thing to recall the period here,” Mr Ladany said in an interview at the former concentration camp.
But he said it was important to him to come back and tell relatives about the horrors he endured during the Holocaust, in which six million European Jews were killed.
It is a pilgrimage he has already made several times before with other family members.
“I always bring here one of my relatives to teach them, to educate them what happened,” Mr Ladany said.
Even though he was so young at the time, Mr Ladany still remembers the constant hunger and enduring seemingly endless roll-calls in the cold wind outside the barracks when the guards would count the camp inmates.
The Ladanys fled Belgrade in 1941 after their home was bombed by the German Luftwaffe.
They escaped to Budapest, Hungary, but were eventually captured by the Nazis and sent to Bergen-Belsen, where 52,000 mostly Jewish prisoners died at the concentration camp and more than 19,000 prisoners of war, mostly from the Soviet Union, died at the adjacent PoW camp.
After being freed the previous year in the exchange, Mr Ladany and his family travelled to Switzerland and ultimately moved in 1948 to Israel.
There he grew up to become a professor of industrial engineering and management and an accomplished racewalker – he still holds the 50-mile world record, set in 1972.
When he came to Munich for the Olympics aged 36, he tried to guess the age of every German he met, and “if in my mind he would have been age-wise in the age group that might have participated in the Third Reich’s atrocities, I prevented any contact”.
However, this time it was not the Germans who posed a threat to his life.
Early on the morning of September 5, members of the Palestinian group Black September broke into the Olympic Village, killed two athletes from the Israeli delegation and took nine more hostage, demanding the release of Palestinian prisoners in Israel as well as two left-wing extremists in West German jails.
Mr Ladany, again, narrowly escaped. A terrified roommate woke him up to say a fellow athlete was dead, and he quickly put on his trainers and ran to the door of their apartment.
Just outside he saw an Olympic official pleading with a man in a tracksuit and hat, later identified as the leader of the assailants, to be “humane” and let Red Cross officials into an adjacent apartment. The man, Mr Ladany recalled, responded: “The Jews aren’t humane either.”
He turned round, threw on some clothes over his pyjamas and joined other teammates in fleeing.
Not everyone was so lucky; all nine hostages and a police officer were killed during a failed rescue attempt by German forces.
Mr Ladany said that before the attack, the Olympics were purely “a sports meeting of joy and competition”. But today, no such event is held without strict security.
“Since then,” he said, “the world has changed.”
West Germany was criticised not only for botching the rescue but also for withholding historic files on the tragic events for decades, and for not offering enough compensation to victims’ families.
Relatives of the 11 athletes killed had threatened to boycott Monday’s anniversary but last week finally reached a deal in which they will receive a total of 28 million euro (£24.2 million) in compensation.
Mr Ladany plans to wear his original Israeli team jacket from 1972 when he attends the memorial, and he is looking forward to showing the world that both he and Israel have endured.
“Those that tried to kill me are not alive anymore,” he said. “We are still here. Not only as individuals, but also as a country.”