Cultural exchanges are consensual. They usually occur because one culture has consensually shared aspects of their culture with another and representatives from both cultures have a role in creating the final product.
However, cultural appropriation is not consensual, and usually ill-informed. For example, when Katy Perry dressed as a Japanese geisha for a Japanese-themed performance, despite having Japanese backup dancers on stage, the outfit wasn’t even from Japan and incorporated none of the traditional elements of Japanese attire. As such, the exchange was clearly non-consensual, and she was carelessly exploiting elements of Japanese culture. 
The presence of consent is what makes cultural appropriation distinct from cultural exchange. One is acceptable, one is not. One enriches society, one damages it and alienates minorities.