In 1961 aged 28, Goldberg joined the African National Congress' (ANC) military wing, Umkhonto WeSizwe, where he was able to make use of his technical skills as an engineer, including making bombs and explosives.
He died just before midnight on Wednesday, his family and foundation said in a statement.
"His was a life well-lived in the struggle for freedom in South Africa," the statement said.
Goldberg had been battling lung cancer and a heart condition. His activism meant his family was a constant target and his wife Esme Bodenstein, herself a political activist, was subjected to solitary confinement.
Goldberg spent 22 years in a whites-only jail after his arrest with Mandela and other activists including Walter Sisulu, Govan Mbeki and Andrew Mlangeni.