Published online by Cambridge University Press: 21 September 2018
A new stopping problem and the critical exercise price of American fractional lookback option are developed in the case where the stock price follows a special mixed jump diffusion fractional Brownian motion. By using Itô formula and Wick-Itô-Skorohod integral a new market pricing model is built, and the fundamental solutions of stochastic parabolic partial differential equations are deduced under the condition of Merton assumptions. With an optimal stopping problem and the exercise boundary, the explicit integral representation of early exercise premium and the critical exercise price are also derived. Numerical simulation illustrates the asymptotic behavior of this critical boundary.