In this position paper, we present the results of ongoing theoretical investigation in the phenomenon of “interactive architecture.” Interaction in architecture deals with the meaningful exchange of information and physical acts between building and person. This goes beyond responsive systems like automated doors, shading systems, and so on. Most examples of interactive architecture are technological explorations that probe possibilities and potential of interaction. In this paper we claim that this is not enough. The notion of interactive architecture is explored through social aspects, user experience, situatedness, and agent-based theory. From this we argue that interactive buildings need comprehensive and consistent styles of interaction rather than series of isolated and unrelated interaction events. Different people in various contexts require different sets of behavior from an interactive building. These sets are conceptualized as “interaction narratives,” following the work of Maria Lehman. We argue that such narratives can provide a better fit of the interactive building with the user, and lead to more profound understanding of such systems.