In this paper, a simple scheme of optical interconnections between M input and N output terminals using mode speckle patterns generated in the mulitmode optical fiber is described. It is shown that fully parallel nonblocking interconnections are accomplished by using properly generated mode speckle patterns as destination codes and dectecting the generated mode speckle patterns separately using M multiplexed Vander Lugt matched filters at each output terminal. The orthogonality condition of mode speckle patterns is investigated by either changing the launching angles or rotating the polarization direction of the beam. Preliminary experiments by use of a 4$ imes$4 graded refractive-index multimode fiber coupler whose core diameter was 50 ${mu}{ extrm}{m}$ were performed to show the feasibility of our scheme.