Adaptive algorithms based on the higher order error criterion such as the LVCMS and the LMF show performance degradation if input signal contains additive noise with a heavier-tailed density. Conventional analysis often neglects higher order terms in the recursion and my not suit for prediction exact behavior of these higher order algorithms. This paper presents a new convergence analysis which contains all the higher order term in the recursion. The analysis shows that the higher order terms, which are often neglected, dose not affect the upper bound on the step size but the misadjustment. However, the effect decreases sharply proportional to the square of the step size.