In a statement Thursday, the China Football Association said that former head coach of the Chinese women's team Hao Wei would take over the key role for the men's team.
The CFA said it was setting up a four-member group to be in charge of the Olympic qualifiers, with Hiddink's name notably absent.
The statement said the new leadership was "in view of the previous inadequate preparations for the Chinese U-22 National Men's Football Team" and was "in order to further strengthen team preparations and improve team building and management level."
The 72-year-old Hiddink was appointed as head of the Olympic team last September, joining veteran Italian coach Marcello Lippi in taking a major role in China's aggressive push to finally become a force in international football.
But after an uninspiring series of performances, Hiddink's youth team lost 2-0 against Vietnam in a friendly at home earlier this month.
China's under-21s will be expected to make up the bulk of their squad for the Olympics, where the age group is under-23.
State news agency Xinhua said Thursday night that "despite his illustrious coaching career, the Dutch veteran is far from (a) success in China" with only four wins out of the twelve matches."
Football-fan President Xi Jinping has grand ambitions of China hosting and even winning a World Cup.
But efforts to boost the performance of China's national team have been largely unsuccessful, and highlighted how much work there is to be done.
Despite its clout as a global sporting and financial power China have qualified for the World Cup only once, leading to a change in policy that now allows naturalised foreign-born players to play for the national team.