The explosion happened during a pro-Ukrainian march marking the one-year anniversary of the overthrow of the country's former pro-Kremlin president.
The city's prosecutors office said three people died. An AFP reporter saw two bodies lying on the ground covered with Ukrainian flags.
Kharkiv police spokeswoman Natalia Zakharova said an investigation would be opened into an apparent "terrorist explosion" caused by an unidentified device. She said at least 10 people had been wounded.
Kharkiv is located more than 200 kilometres (120 miles) from the frontline where pro-Russian rebels are fighting government forces, AFP reported.
Other marches celebrating the overthrow of former president Viktor Yanukovych a year ago, after street protests brutally repressed by snipers, were taking place in Kiev and other parts of government controlled Ukraine on Sunday.
A UN-backed ceasefire came into effect in Ukraine's east a week ago but has been largely ignored by both sides in the conflict.