Polymer/nanoclay composite layers were formed by a UV-curing process on plastic substrates for flexible display to enhance their mechanical and barrier properties. Sodium montmorillonite ($Na^+$-MMT) was modified for intercalation by cationic surfactants having various alkyl chain lengths. Increase of alkyl chain length achieved more intercalation of the nanoclays, and the maximum d-spacing was 4.2 nm when modified by cetyltrimethyl ammonium bromide (CTAB). The modified nanoclay was then functionalized by isophorone diisocyanate-2-hydroxyethyl acrylate (IPDI-HEA) having a reactive acrylate group, which enhanced its compatibility within the polymer matrix. The composite layer formation using functionalized nanoclay still improved the mechanical and barrier properties. The pencil hardness was improved to 3H and impact energy increased to 150 mJ, while maintaining the original optical properties of the polyethersulfone (PES) substrate. Oxygen transmittance rate (OTR) was remarkably reduced from 250 to $18.5cc/m^2$ day and water vapor transmission rate (WVTR) was gradually decreased from 60 to $37.7g/m^2$ day with the increase of nanoclay content, respectively. Composite layers with functionalized nanoclay improved the properties of the PES substrate to be used in flexible displays.