Applying Machine Learning for Firebrand Production Prediction

Many experiments have been conducted to study the firebrand production process [12‐15, 20, 21]. The datasets for ML algorithm training and testing (or validation) are from a firebrand production study in [12‐15]. The collected firebrands from other firebrand production experiments using full-scale building components or assemblies varied between 50 and 500. Study [13] collected and measured significantly larger firebrand data than any existing firebrand data sets. The study [13] collected and measured the physical properties of 59,820 firebrands, including 24,149 from structural components (fences, re-entrant corners, and roofs), 26,422 from wall-roof mockup structural assemblies, and 9249 from five wildland vegetative fuels (chamise, saw palmetto, loblolly pine, Leyland Cypress, and little bluestem grass). In addition, their experimental work was based on a statistics-based framework for sampling and measurement [12]. The firebrand production experiments were full-scale and were performed in a large-scale open-jet wind tunnel. The fluctuating wind speeds were used in the experiments, with nominal speeds of 5.36 m/s (idle), 11.17 m/s (medium), and 17.88 m/s (high). Three physical quantities of firebrands were measured and reported: traveling (or flying) distance, 2D projected area (or size), and mass. The units in the data are projected area in mm\(^2\), mass in grams, and traveling distance in meters. The independent parameters controlled in the experimental setup were fuel types (structural and vegetative) and wind speed. Readers should refer to the report [13] and data archives [14, 15] for details related to the experiments.

The independent factors and their designated values in the ML algorithms are listed in Table 1. For simplicity, the two fence types (privacy and lattice) were put in the same variables group as construction materials and types such as cedar/plywood, cedar/OSB (oriented strand board), recycled rubber, non-FRT (flame retardant treated) shake and FRT shake. The assigned numbers are for identification purpose only.

Firebrands are often generated in large numbers and can accumulate in a concentrated area. Here we introduce the parameters FAND and FAMD to address this issue.

FAND is defined as the number of landed firebrands per unit area. It is a measure of the concentration of landed firebrands. If the number of the landed firebrands (by default all firebrands, including both burning and extinguished firebrands) is N and the area of landed firebrands is A, then FAND (\(\rho _N\)) is,The higher the FAND, the more firebrands are accumulated in the area. FAND is an important parameter in assessing ignition potential and threat from firebrands. In modeling, FAND can help us understand spotting ignition more accurately and thus help forecast the spread of fire using appropriate metamodeling approach.

FAMD is defined as the mass of landed firebrands per unit area. FAMD denotes surface density (area density) of the landed firebrands (both burning and extinguished firebrands). If m denotes mass of the landed firebrands, then FAMD (\(\rho _m\)) is,The quantification of the area of interest in Eqs. 1 and 2 depends on the purpose of the study. For example, in a firebrand production laboratory study, the area can be determined as the pre-defined area where firebrand collection will take place. In a firebrand ignition field study, the area can be determined as the area of a structure (e.g., a house excluding yards), the area of a plot of land, or the area of a built neighborhood. In a study, FAND (or FAMD) can be global or local. In a firebrand production laboratory study, global FAND (or FAMD) is the total number (or mass) of collected firebrands divided by the total firebrand collection area. The local FAND (or FAMD) can be the number (or mass) of collected firebrands in a particular firebrand collection pan divided by the pan’s area. Local and global FANDs or FAMDs can be used for different purposes. The area in Eqs. 1 and 2 is different from the projected area in previous studies [12‐15] to represent the size of a firebrand. The projected area, discussed in the following section, is the two-dimensional (2D) projection of a firebrand.

In application, threshold values of FAMD and FAND can be introduced to indicate the possibility of spotting ignition or to classify the risk levels from firebrands based on their FAMD and FAND values. Since the FAMD and FAND are new concepts, there are no defined threshold values of FAMD and FAND in the literature for these purposes yet. Thus, in our current study, the median values of the FAND and FAMD generated from the vegetative and structural fuels in the study in [13] are used as the threshold values to classify high risk and low risk FAND and FAMD to demonstrate our framework and the process. FAMD and FAND values below the median values are classified as low risk or “low FAMD and low FAND”, and FAMD and FAND values higher than the median values are classified as high risk or “high FAMD and high FAND”. The process will be the same as long as the threshold values of low and high are determined. Further research on threshold values of FAND and FAMD causing spot fire ignition for different types of recipient fuels will help in further refining the ML model developed in this study.

Since the input variables wind speed and travel distance were not continuous, this current study focuses on predicting discrete values of FAND and FAMD (e.g., high and low values). In theory, full-scale wind tunnel experiments measuring firebrand production with continuous wind speed and travel distance are required to predict continuous FAMD and FAND values. In practice, it is unrealistic or impossible to have continuous wind speed and travel distance in firebrand production experiments. However, higher resolutions of wind speed and travel distance in experimental design will improve ML algorithms.

Firebrand projected area (i.e., size) and its mass are highly correlated. The relationship of firebrand mass (m) and its projected area (a) can be expressed as,The value of coefficient \(\alpha\) is generally between 0.5 and 1.0. It is close to 2/3 for firebrands produced from vegetative fuels [22], as these firebrands tend to be 1D. Firebrands from structural fuels are more likely to be 2D [13]. To better understand the correlation of projected area and mass in our data, a log transformation of the variables (area and mass) was performed. The results are shown in Tables 2 and 3. For structural firebrands, the coefficient \(\alpha\) varies from 0.63 to 0.77, with an average value of 0.70. For vegetative firebrands, the range of coefficient \(\alpha\) varies from 0.59 to 0.91, with an average value of 0.72.

The log–log plot of firebrand projected area and its mass has a linear trend with every combination of factors as shown in Figs. 1 and 2, thus,

where c is the intercept point on the log–log plot, coefficient \(\beta\) is the slope of the graph and the inverse of coefficient \(\alpha\) in Eq. 3, i.e.,Thus, Eq. 4 can be rewritten as,

Equation 7 is important and useful in practice, especially in firebrand experiments or post-fire fire investigations. The advancement of digital photography and computer vision has enabled us to capture images of firebrands in flight (even in real-time) and after landing. Using appropriate algorithm [12‐15], one can obtain the projected area of individual firebrands in these images at a high level of accuracy. On the contrary, measuring firebrand mass is a labor-intensive process. One must weight each brand individually to get its mass. Furthermore, since firebrands are fragile, handling the firebrand during the weighting process may break the firebrand and lead to errors in measurement and uncertainties in the whole set of firebrand data. With the projected area determined from digital imaging analysis or computer vision, Eq. 7 can then be used to calculate the mass of the corresponding firebrand. The accuracy of the calculation is mainly affected by factors \(\eta\) and \(\alpha\) (or \(\beta\)). More accurate \(\eta\) and \(\alpha\) (or \(\beta\)) values will help fine tune the ML model and can predict firebrand mass with higher accuracy.

The purpose of the ML models is to predict FAND and FAMD to estimate the chances of spot ignition occurrence since high FAND and FAMD values are directly proportional to the high probability of causing spot ignition in a wildfire or ignition of structures [23, 24]. The distance travelled and wind speed can be measured along with the identification of the type of fuel of the source of firebrands generating at the WUI fire, this enables the prediction of FAND and FAMD in real time. We will train the ML models with the independent variables and compare the accuracies after determining the importance of the predicting variables.

Firebrand mass and projected area are dropped from the independent variables for two reasons. The first is related to ML algorithms, as the high correlation between projected area and mass can cause high instability in the ML models as the adjustment of the coefficient for one predicting variable will impact the coefficient for the dependent predicting variable. This statement implies the need for full independence. The second reason is that firebrand mass and projected area are measured values after firebrands are generated. With the removal of mass and projected area, the remaining independent predicting variables are easy to obtain in a wildland or WUI fire and will not cause any variance inflation. The feature reduction allows us to use minimum possible information while maintaining high model accuracy. The independent variables used in our ML models to predict the dependent variables FAND and FAMD are: landing (or travel) distance, wind speed, and fuel type (structural and vegetative fuels).

In addition to analyzing the impact of different fuels on the spot ignition in a WUI fire, this research will analyze two hypothesis which are as follows:

The data used to predict FAND and FAMD has five dimensions which is difficult to imagine and interpret directly. To overcome this problem, we used scatter plot matrix which helps in the visualization of the data separation in 2D for each independent variable pair to decide if the binary data can be separated with linear or non-linear ML model.

The histograms (see Figs. 9, 10, 11) give some interesting insights which were not detected in the scatter plot matrix. While scatter plot matrix shows that the independent variables cannot separate high and low FAND and FAMD, histograms confirmed that the data points are overlapping and thus the separation is not linearly possible. Therefore, we need non-linear ML models to analyze further with the prediction. Due to the complexity of fire dynamics in a fire, a non-parametric ML model would generate a higher accuracy compared to a parametric model since non-parametric ML models do not make any strong assumptions for the data while creating their own function and learn the patterns in the data structure. Thus, non-parametric ML models are open for further modification in the prediction function with a possible addition of new data points in the future.

Considering the abundant data available [12‐15] to predict the firebrands and to avoid any strong assumptions about the mapping function form, the non-parametric ML model approach fits perfectly for the prediction modeling. The parametric ML models make strong linear or non-linear assumptions which works best on the data with a detectable linear or non-linear separation between the dependent variable labels. This research data does not have such patterns to use a parametric ML model for predictions and thus, any parametric ML model would either overfit or underfit the data depending on the hyperparameter we use to train the model. However, non-parametric models tend to create their own light assumptions which is flexible for a future change with the addition or removal of data points. This light assumption by the non-parametric ML model is often made after learning the patterns in the data with a tuned hyperparameter to avoid overfitting issues.

To further confirm the approach of selecting a non-parametric non-linear ML model over available ML models, we predicted firebrand data with parametric and non parametric ML models and compared their accuracies. We can observe the difference in model accuracies when compared with each other in Fig. 12. We chose the two ML models with highest accuracy and tuned the hyperparameters for the two chosen ML models to further increase the accuracy of the ML models and proceed with one of the two best prediction ML models. Among the available methods for this study, the non-parametric methods used in the ML prediction of firebrand data are KNNs and support vector machine non-linear (SVM non-linear) as described later in the Sects. 3.5 and 3.6.

The data is divided into a random training and testing splits. Training split is used for the ML model training and the testing split is used to validate the prediction efficiency of ML model after the model is trained.

The purpose of splitting the dataset into testing and training datasets is to generate a pseudo unknown dataset out of the known dataset which is split randomly with the help of an algorithm. The training and testing datasets have equal number of data points. The training dataset has the independent predicting variables as: A larger training data makes the prediction model better and accurate while more test data provides a better error estimate. This tradeoff between model accuracy and error estimation can be solved by splitting the data into half (50% training and 50% test data). The other way to get a better error estimate is to perform a cross validation. Thus to increase the model accuracy, we chose the training data split size as 70% and reduced the test data size to 30%.

All the predicting variables are independent to each other and are convenient to obtain in a WUI fire. The training data (70% of the total data points) is used to train the ML algorithm. This trained ML algorithm is used to perform prediction on the testing data (30% of the total data points). The test data, created after the random split of the dataset, is labeled. Thus, the ML prediction values on the test data and actual values of the landing firebrands will be compared and used to create a confusion matrix (accuracy evaluation matrix). This confusion matrix, shown in Fig. 4, helps in the evaluation of the ML model accuracy on a pseudo unknown data. The confusion matrix obtained on the testing data, from the ML model, will provide various other insights of its prediction efficiency, on an unknown dataset, in addition to the current ML model accuracy.

The confusion matrix obtained with the help of ML model on test data is used to measure accuracy of the ML model (Fig. 4). The predicted high FAND or FAMD is denoted by High FAND or High FAMD (simplified as HF) and the predicted low FAND or FAMD is denoted by Low FAND or Low FAMD (simplified as LF) denoting the two types of target variable class in the data. For the performance measurement of the model, we will refer the confusion matrix (Fig. 4) and use True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN) values from the matrix.

The value TP represents the total high FAND or FAMD (HF) correctly predicted, value TN represents the total low FAND or FAMD (LF) correctly predicted, value FP represents the total LF predicted as HF, and the value FN represents total HF predicted as LF. The FP and FN are the incorrect predictions which are also known as misclassifications. The range of the four values in the confusion matrix lies between 0 and the total number of classes present in the target variable of the data. The best-case scenario for the prediction model is having both FP and FN values as 0 for train data accuracy and test data accuracy which will indicate that the ML model predictions were 100% correct. The TP and TN values are supposed to be as high as possible with the max value equal to the number of the target variable classes. The four values in the confusion matrix are always positive.

The details of other performance parameters are provided in the following, while their values range from 0 to 100% and are listed in Tables 4 and 5.

Accuracy the ratio of correct predictions with respect to the total predictions performed by the model.Precision the ratio of correct positive class predictions with respect to the total positive class predictions in the data.Sensitivity (TPR) sensitivity, also known as recall and true positive rate (TPR), is the ratio of the correct positive values predicted and the total number of positive values in the data. For example total HF correctly classified as HF.False Positive Rate (FPR) the ratio of negative class values predicted as positive class and the total negative class values.Specificity (SP) specificity, also known as true negative rate (TNR), is calculated as a ratio of correct negative values predicted and the total negative values in the data. For this research SP is the ratio of correct LF predicted to the total LF values.Prevalence prevalence shows how often the positive value (HF in our case) occurs in the sample data.Positive Predictive Value (PPV) also known as precision, shows how often the predicted positive value is correct.Negative Predictive Value (NPV) negative predictive value is the ratio of actual TN values and the total negative values predicted.Balanced Accuracy the balanced accuracy is average of recall calculated in each class.F1 Score F1 Score is a harmonic mean of Sensitivity and Precision. Since F1 score accounts both false positive and false negative values, it helps in uneven class distribution.Matthews Correlation Coefficient (MCC) Matthews Correlation Coefficient produce score with all the false positive, false negative, true positive, and true negative values. MCC is also known as a balanced measure. The value of MCC varies from \(-1\) to \(+1\) where \(+1\) is considered as a perfect prediction, 0 as a random prediction, and \(-1\) as a completely wrong predictions.Confidence Interval (CI) shows the uncertainty of an estimate. The range of upper and lower accuracies of the prediction model shows that there is 95% percent chances of the accuracy of the model to lie between the upper and lower limits of the accuracy of the prediction model. For a sample size of n and z as the number of standard deviations from the Gaussian distribution, the CI is calculated as follows where z is 1.96 for 95% CI:

The confusion matrix helps us to identify the efficiency of the ML model. If the desired false negative and false positive values (present in the confusion matrix) are achieved, the hyperparameter tuning of the same ML model is conducted. Hyperparameter tuning of a ML model allows us to choose the best value of the model parameters used in the ML algorithm to control the learning rate and prevent any under fitting or over fitting, which can lead to a drop in the accuracy of the model. If the false positive and false negative values are higher than the desired value, a different ML model is chosen and trained with the available training data.

A hyperparameter, K-value, represents the number of nearest neighbors in the KNN ML model is needed to classify the data points. In Fig. 5, the optimum value of K is recognized when the error rate from the training data (solid line in the Fig. 5) gets close to the error rate from the test data (dashed line in Fig. 5). In Plot a of Fig. 5, the train error and test error are close at K-value 5. Thus, the hyperparameter K-value is 5 for the ML model predicting FAMD generated from the structural fuels. Similarly other ML models are tuned in Plots b, c, and d for the K-values.

After successful hyperparameter tuning, the sample size reduction of the model is conducted while maintaining a high F1 score of the prediction model. We analyzed the drop in error with respect to the increment in the training data’s sample size and concluded that approximately 250 data points (vertical dashed line in Fig. 6) are required to train the ML model without compromising the model accuracy. Figure 6 shows that the ML models have error rate below 10% (horizontal dashed line) at 250 data points and above. This can enable future researchers to develop an efficient machine learning model with a high level of accuracy without wasting time, money, and hardwork to generate way more than required sample size of dataset to predict FAND and FAMD of the landing firebrands.

The KNN ML model uses the Euclidean distance function as a base to measure the KNNs. Thus, the prediction can be performed with only four independent variables: distance, wind speed, component, and material for structural fuels; and distance, wind speed, species, and vegetation for vegetative fuels. This will increase the efficiency of the model and will require less data for prediction. The model uses random labeled points to label a new point. The label is decided based on the majority of the surrounding neighbors within the Euclidean distance. If p and q are the points, then the distance between p and q is calculated as,After training the models, accuracies on the testing data are derived from the confusion matrices (Table 6a, b, c, and d).

The ML models predicted 183 to 189 data points (Table 6a, b, c, and d). Their F1 score (model’s accuracy) ranges from 93.65% to 98.46%. The optimum hyperparameter (K-value) ranges from 3 to 7 for the KNN ML model (Fig. 6).

SVM requires little tuning while producing high performance [25]. Radial basis kernel (RBF) was used for the SVM ML model as the data showed non-linear patterns. The RBF uses Gaussian RBF where the distance between two points \(x_i\) and center is calculated as,where \(||x_i-x_j||^2\) is the Euclidean distance and \(\gamma\) in Eq. 21 is,The hyperparameter, \(\gamma\), decides the impact of the feature, \(\phi (x_i,center)\), on the hyperplane. With increase in value of \(\gamma\), the boundary gets curvier. Thus, an extremely small value will make the model to work like a linear model (underfit) and a large value of \(\gamma\) will make the model overfit. The optimum value of \(\gamma\), after the hyperparameter tuning of ML models predicting FAND and FAMD ranges from 0.04 to 0.23 to achieve the best possible F1 score (model’s accuracy).

Another hyperparameter tuned for the SVM model is COST (\(\zeta\)) which is responsible for tolerance of misclassifications allowed through the hyperplane. The degree of misclassifications can be explained as the softness of hyperplane (n dimensional margin) separating the dependent variables. As the \(\zeta\) increases, the separation boundary gets strict and penalize more for any misclassifications, which means the decision boundary is dependent on fewer support vectors. However, if the \(\zeta\) is small, the separation boundary depends on more support vectors because the boundary gets softer (underfit) and for the \(\zeta\) value 0, the data will not be separated at all. After the hyperparameter tuning of ML models the optimum value of \(\zeta\) ranges from 4.1 to 8.0. The tuned hyperparameters, \(\zeta\) and \(\gamma\), generate an F1 score ranging from 56.60% to 85.95% (Table 7a, b, c, and d).

Among the two top non-linear non-parametric ML models that we used, we chose KNN for a slightly higher accuracy when compared to the SVM ML model (Fig. 12). The KNN ML model performed with an accuracy greater than 90% while predicting FAND and FAMD generated from structural and vegetative fuels while SVM gave an accuracy below 90%.

The \(\sim {7\%}\) misclassification in KNN model can be due to the various independent variables which were not included in training the ML model. The model’s prediction accuracy is a result of using the independent variables which can be physically obtained within minutes, in a real WUI fire scenario, to give incident command more time to react and make better decisions accordingly. Another reason for the model’s misclassification can be the limited full scale experimental data used to train the ML model. Conducting full scale experiments is expensive and requires time and hard work. To reduce misclassification, we can train the model with a larger amount of full scale experimental data. The reduction of misclassification rate depends on the purpose of the modeling. The prediction model can be for planning an experiment, wild land fire spread modeling prediction, and fire risk assessment involving firebrands, etc. Depending on a project’s specific needs, the users can set the misclassification rate limit. We chose the three error limits (10%, 5%, and 3%) to demonstrate the process and framework but users may choose their own limits.

The purpose of comparing the area under the receiver operating characteristic curve (AUROC) of the two ML models, KNN and SVM-NL, is to compare the performance measure at various thresholds. The AUROC will help us understand the model’s measure of separability. The receiver operating characteristic curve (ROC curve) graph is plotted with FPR on x axis and TPR on y axis on different threshold values. The AUROC is then calculated using the composite trapezoidal rule along the x axis.

The KNN shows higher AUROC value when compared to the SVM-NL model. The AUROC ranges from 98.18% to 99.65% for KNN and 64.85% to 98.71% for SVM-NL. This shows that the measure of separability of KNN model is higher than SVM-NL for predicting the FAMD/FAND.

The KNN model’s performance on the test data shows that we expect the ML model to predict with at least 93.65% accuracy on an unknown dataset, generated from a fire similar to the fire created in our test facility. For a completely unknown dataset generated from a different environment (real WUI fire) it can give lower accuracy. In order to check the trained ML model’s performance on such an unknown dataset, we performed K-fold cross validation on our existing dataset. In this study, the value of K was 10.

Thus, the dataset was randomly shuffled and then split into 10 groups (K) of approximately equal size. One group was held as a validation dataset and the other nine groups (\(K-1\)) were used to train the ML model to get the model accuracy [26]. The process was repeated for all the groups individually and then an average of all accuracies was used as the model accuracy.

The accuracies for 10-fold cross validation with KNN model on our dataset ranges from 63.62% to 84.04% (Table 9), which denotes the prediction accuracy of KNN model on an unknown dataset.