The giant banner on a high-rise building shows a semi-naked woman and the flags of the 32 countries in the World Cup, which kicks off in June.
The Pascha brothel’s owner, Armin Lobscheid, said a group of Muslims had threatened violence over the advert.
He said they had accused the brothel of insulting Islam by using the flags.
First there were telephone threats of violence, then about 30 hooded protesters armed with knives and sticks turned up outside Pascha on Friday, the Koelner Stadt-Anzeiger newspaper reported.
"The situation was explosive," Mr Lobscheid told the paper.
"Some of the people compared our ad to the Danish Mohammed cartoons," he said, referring to cartoons which sparked violent protests in several Muslim countries in February.
The Tunisian flag – bearing the Muslim crescent symbol – remains on the ad, however.
The slogan on the ad reads: "The world is a guest of female friends" – a variant of the official World Cup slogan: "The world is a guest of friends".
Mr Lobscheid said the banner had been commissioned in a normal business deal and "we certainly didn’t intend to insult anyone".
He said the significance of the flags’ symbols had been overlooked.
Prostitution is legal in Germany, where the authorities are preparing for a possible boom in the sex trade during the World Cup.