In order to investigate the nonlinear characteristics of structural joint, the experimental setup with a jointed mass system is established for dynamic characterization analysis and vibration prediction, and a corresponding nonlinearity identification method is studied. First, the sine-sweep vibration test with different base excitation levels are applied to the structural joint system to study the dominant modal of mass rigid motion. Then, based on the harmonic balance method principle, the measured vibration transmissibilities are utilized for nonlinearity identification using different excitation levels. Experimental results show that nonlinear spring and damping force can be represented by a polynomial order approximation. The identified nonlinear stiffness and damping force can predict the system's response, and they can reveal the shifts of resonant frequency or damping due to discontinuity of contact mechanisms within a certain range.