Fire detection from video is effective for most video surveillance applications. The algorithm that processes the video acquired by cameras in real time has a twofold goal: detect as many fires as possible and keep the number of false alarms low. While existing approaches obtain the first goal, they often produce many false alarms due to their inability to account for the specific characteristics of diverse application environments. This paper introduces Fire Observation and Control Using Scenarios (FOCUS), a novel configurable fire detection method designed to bridge the gap between the literature methods and the application needs by exploiting scenario-specific knowledge. FOCUS leverages scalable and configurable modules for robust fire detection, incorporating three key steps: (1) fire detection, which identifies potential fire regions using visual cues; (2) fire candidate filtering through motion analysis, to eliminate false positives by analyzing the dynamic behavior of the identified fire candidates; (3) a vision-language model, which evaluates and confirms fire alarms by correlating visual evidence with contextual knowledge. By tailoring the configuration to the scenario and integrating the advanced filtering mechanisms according to the complexity of the environment, FOCUS improves performance in all the considered application scenarios. The analysis of the results shows that the proposed approach outperforms existing methods demonstrating higher resilience on real data, which enables its usage in real-world applications. • We propose a scalable and configurable method for fire detection in videos. • It combines three modules: object detection, object tracking, and VLM. • The modules are activated depending on the needs of the specific application scenario. • The proposed method outperforms non-configurable competitors in a mixed scenario.