We can effortlessly know what an object is made of by seeing the object. Our recent studies suggest that such material perception involves ventral higher visual areas in humans and monkeys, and those areas represent not only visual properties (e.g., gloss) of the materials of the objects but also some tactile properties (e.g., hardness) of them. In the present study, we conducted an fMRI experiment in monkeys to investigate whether the neural representation of materials in the higher visual cortex is formed through learning of the visuo-tactile association observed in real-world materials. We scanned two awake macaque monkeys while viewing images of rods made of nine different materials before and after extensive visuo-tactile experiences of the nine materials (grasping the rods). Some of the materials (e.g., ceramic and fabric) were likely to be unfamiliar to the monkeys before the visuo-tactile experiences. Representational similarity analysis revealed that neural representation in early visual areas (typically V1) was stable before and after the visuo-tactile experiences. By contrast, neural representation in higher visual areas (typically posterior inferior temporal cortex) changed after the visuo-tactile experiences toward those reflecting the visuo-tactile properties assessed by human subjects. Our results suggest that neural representations in higher visual areas can be modified by visuo-tactile experiences and the neural representation of materials in those areas reflect visuo-tactile association established by the experiences.