Samuel Kasumu will leave the role next month. He had previously handed in his resignation in February, before retracting it.
News of his departure comes the day after the publication of a government-commissioned report on race that has been criticised by campaigners.
Downing Street has denied his exit is linked to the report’s findings.
The Commission on Race and Ethnic Disparities (CRED) – set up last year after Black Lives Matter protests last summer – claimed the UK “no longer” had a system rigged against minorities.
Racial equality campaigners have criticised the findings, and Labour accused the government of downplaying institutional racism.
Mr Kasumu, an entrepreneur who has worked for the government since 2019, has declined to comment.
In his February resignation letter, he had said he wanted to continue work he had been doing fighting misinformation on Covid vaccines, with “the view to leaving at the end of May”.
He accused the Conservatives of pursuing a “politics steeped in division,” and the party’s previous success in attracting black and Asian voters had been “eroded” after the 2015 general election.
But Mr Kasumu went on to retract this letter, following talks with vaccines minister Nadhim Zahawi.
A No 10 spokesman said on Thursday that Mr Kasumu had been planning to leave government in May “for several months”.
“Any suggestion that this decision has been made this week or that this is linked to the CRED report is completely inaccurate,” he added.
Labour’s shadow equalities secretary Marsha de Cordova called the report “divisive,” adding it was “no wonder” the government was “losing the expertise from their team”.
“To have your most senior advisor on ethnic minorities quit as you publish a so-called landmark report on race in the UK is telling of how far removed the Tories are from the everyday lived experiences of Black, Asian and ethnic minority people,” she added.
'Crisis at No 10'
Former equality and human rights commissioner Lord Simon Woolley, who knows Mr Kasumu, said he had been “disheartened” whilst at No 10.
The crossbench peer, who has criticised the commission’s findings, told BBC News there is a “crisis at No 10 when it comes to acknowledging and dealing with persistent race inequality”.
The report, which was published on Wednesday, concluded the UK was not yet a “post-racial country,” but family structure and social class had a bigger impact than race on how people’s lives turned out.
It argued that racial discrimination has often been misapplied to “account for every observed disparity” between ethnic groups.
And it said references to racism in the UK being “institutional” or “structural” had become confusing, and had sometimes been used without sufficient evidence.
Dr Maggie Aderin-Pocock, a member of the commission, said the report was not denying institutional racism existed, but said they had not discovered evidence of it in the areas they had looked.
But the report has led to criticism from unions, charities and opposition politicians – which have accused the commission of downplaying the role of wider factors in racial inequalities.