When we are engaged in visual activities, we have to move our gaze position by selecting fixation targets. It is well known that the activity of the visual nervous system in visual search tasks is modulated by the top-down attention for the visual features such as color or shape. However, the information processing of the visual system during visual search has not been elucidated.A previous study suggests that the top-down attentional modulation for the visual features is generated by feedback connections of the visual area V4, the inferior temporal cortex (IT), the frontal eye field(FEF), and the prefrontal cortex (PF) by a model simulation (Hamker 2004). As this model supposed to maintain fixation and did not consider eye movements, several models based on Hamker's model have been proposed to reproduce information processing in the visual system during visual search with eye movements (Sugimoto & Kohama 2011, Nishida & Kohama 2012). Nevertheless, these models did not possess top-down or bottom-up attention integration mechanisms at the time of visual search (Ogawa & Komatsu 2006). In this study, we propose a modified version of Hamker's model considering both top-down and bottom-up integration by introducing a unit which corresponds to the functions of the temporo-parietal junction (TPJ). Furthermore, we modified the input of the V4 unit to have a surround inhibition so as to make saliency maps of earlier visual areas. Our simulation results show that the output of our proposed model closely resemble the physiological properties of TPJ and FEF cells as reported by Ogawa & Komatsu, as well as attentional modulation of V4 cells (Motter, 1994). As a result, our model was shown to be a valid neural mechanism model of visual attention. This indicates that dynamic integration of top-down and bottom-up information is mainly caused in the TPJ-FEF network.