Estimation of the relative position between the body segments is an important task in inertial sensor-based human motion tracking. Conventionally, the relative position is determined using orientations and constant segment vectors that connect from segment to joint center, based on the assumption that the segments are rigid. However, the human body segments are non-rigid, which leads to an inaccurate relative position estimation. This paper proposes a relative position estimation method based on inertial sensor signals, considering the non-rigidity of the human bodies. Considering that the effects of non-rigidity are highly correlated with a specific variable, the proposed method uses time-varying segment vectors determined by the specific physical variable, instead of using constant segment vectors. Verification test results for an upper-body model demonstrates the superiority of the proposed method over the conventional method: The averaged root mean square error of the sternum-to-forearm estimation from the conventional method was 34.19 ㎜, while the value from the proposed method was 16.67 ㎜.