Automatic object detection is one of the most common and fundamental tasks in computational intelligence (CI). Neural networks (NNs) are now often the tool of choice for this task. Unlike more traditional approaches that have interpretable parameters, explaining what a NN has learned and characterizing under what conditions the model does and does not perform well is a challenging, yet important task. The most straightforward approach to evaluate performance is to run test imagery through a model. However, the gaining popularity of self-supervised methods among big players such as Tesla and Google serve as evidence that labeled data is scarce in real-world settings. On the other hand, modern high-fidelity graphics simulation is now accessible and programmable, allowing for generation of large amounts of accurately labeled training and testing data for CI. Herein, we describe a framework to assess the performance of a NN model for automatic explosive hazard detection (EHD) from an unmanned aerial vehicle using simulation. The data was generated by the Unreal Engine with Microsoft's AirSim plugin. A workflow for generating simulated data and using it to assess and understand strengths and weaknesses in a learned EHD model is demonstrated.