Sex steroids during developmental period are crucial to formation of sexually dimorphic nuclei, although the mechanisms are not fully understood. We previously reported that masculinization of the principal nucleus of the bed nucleus of the stria terminalis (BNSTp), which is larger and has more neurons in males than in females of mice, involves aromatized androgens acting via estrogen receptor-α (ERα), but not estrogen receptor-β (ERβ). In this study, we examined the morphology of the anteroventral periventricular nucleus (AVPV), which exhibits morphological sex differences opposite to those in the BNSTp, in aromatase knockout (AromKO), ERα knockout (αERKO), and ERβ knockout (βERKO) mice. We found that the volume and neuron number of the AVPV in AromKO and αERKO males were significantly smaller than those in wild-type males. We further examined the AVPV and BNSTp of androgen receptor knockout (ARKO) mice. The volume and neuron number of the BNSTp were significantly smaller in ARKO males than those in wild-type males, while there was no significant difference in the AVPV between ARKO and wild-type mice. We also measured aromatase, ERα, ERβ, and AR mRNA levels in the AVPV and BNSTp of mice on embryonic day 18 (ED18) and postnatal day 4 (PND4). In the male AVPV, aromatase mRNA level on ED18 was higher than that on PND4, although ERα mRNA level was higher on PND4 than on ED18. The male BNSTp expressed aromatase and ERα mRNA on ED18 and PND4 without age difference. AR mRNA in the male BNSTp was not expressed on ED18 and emerged on PND4. These results indicated that estrogen signaling via ERα during perinatal period and androgen signaling via AR during postnatal period are requisite for masculinization of the BNSTp, whereas estrogen signaling via ERα during perinatal period is sufficient for defeminization of the AVPV.