An ad optimization algorithm is an algorithm allowing automatic optimization of ad campaigns (agency and publisher side) or ad spaces (publisher side).
For advertisers, those algorithms are mostly used in the sponsored links domain (Adwords, Facebook) or for display campaign based on performance.
Ad optimization algorithms can optimize the choice of ad creatives (A/B testing, multivariable tests), of ad vehicles, targeting criteria and bids. The optimization can be done under constraint of clicks, transformations and acquisition costs goals.
For publishers the goal of optimization is to maximize the yield from a pool of ad spaces.
Optimization algorithms and softwares are integrated in ad servers and ad exchanges.