IBDI (Internal Bit Depth Increase) is able to significantly improve the coding efficiency of high definition video compression by increasing the bit depth (or precision) of internal arithmetic operation. However the scheme also increases required internal memory for storing decoded reference frames and this can be significant for higher definition of video contents. So, the reference frame memory compression method is proposed to reduce such internal memory requirement. The reference memory compression is performed on 4x4 block called the processing unit to compress the decoded image using the correlation of nearby pixel values. This method has successively reduced the reference frame memory while preserving the coding efficiency of IBDI. However, additional information of each processing unit has to be stored also in internal memory, the amount of additional information could be a limitation of the effectiveness of memory compression scheme. To relax this limitation of previous memory compression scheme, we propose a selective merging-based reference frame memory compression algorithm, dramatically reducing the amount of additional information. Simulation results show that the proposed algorithm provides much smaller overhead than that of the previous algorithm while keeping the coding efficiency of IBDI.