Cloud computing changes the service models of information systems and accelerates the pace of technological innovation of consumer electronics. However, it also brings new security issues. As one of the important foundations of various cloud security solutions, entity authentication is attracting increasing interest of many researchers. This article proposes a layered security architecture to provide a trust transmission mechanism among cloud systems maintained by different organizations. Based on the security architecture, four protocols are proposed to implement mutual authentication, data sharing and secure data transmission in federated cloud systems. The protocols not only can ensure the confidentiality of the data transferred, but also resist man-in-the-middle attacks and masquerading attacks. Additionally, the security properties of the four protocols have been proved by S-pi calculus formal verification. Finally, the performance of the protocols is investigated in a lab environment and the feasibility of the security architecture has been verified under a hybrid cloud system.