Adobe Systems Deep Forest Katarína Nocarová Adobe Systems zápatí prezentace 2 My thesis ̶Deep forest and reactions ̶Results from article, newer results and new implementation ̶Compare DF to other classifiesrs ̶Experiments with different hyperparameter settings ̶AutoML ̶Improvements, future work Adobe Systems Deep Forest 3 Article All based on article named Deep Forest by Zhi-Hua Zhou, Ji Feng from Nanjing University https://arxiv.org/pdf/1702.08835.pdf (2017/2020) ̶ Adobe Systems Deep Forest 4 The ideas ̶Success of deep neural networks ̶ ̶Disadvantages of neural networks (hyperparameters, time and data consuming, blackbox, predetermined model complexity...) ̶ ̶Success of ensemble methods and Random Forest ̶ ̶Deepness rather than neurons – layer by layer processing, model complexity and in-model feature transformation as means to representation learning ability ̶ Adobe Systems Deep Forest 5 gcForest ̶Random Forest implementation ̶ ̶Cascade Forest Structure ̶ ̶Multi-grained scanning ̶ ̶ Adobe Systems Deep Forest 6 Cascade Forest Structure ̶Each level processes information and outputs results to a new layer ̶ ̶Each layer as an ensemble of different decision tree forests – ensemble of ensembles ̶ ̶Different forests for diversity (such as random forest and completely-random tree forest) ̶ ̶Each forest will produce an estimate of class distribution, by counting the percentage of different classes of training examples at the leaf node where the concerned instance falls, and then averaging across all trees in the same forest ̶ Adobe Systems Deep Forest 7 Class vector generation ̶ Adobe Systems Deep Forest 8 Example Adobe Systems Deep Forest 9 Using sliding windows to scan the raw features Multi-Grained Scanning Adobe Systems Deep Forest 10 Overall procedure ̶Multi-grained scanning ̶ ̶Cascade forests ̶ ̶Adding layers untill convergence of validation performance ̶ ̶Final prediction will be obtained by aggregating the class vectors at the last level ̶ ̶ Adobe Systems Deep Forest 11 Overall procedure Adobe Systems zápatí prezentace 12 DF21 ̶Newer implementation – improvement on efficiency ̶ ̶https://github.com/LAMDA-NJU/Deep-Forest ̶ ̶Documentation: https://deep-forest.readthedocs.io/en/stable/ Adobe Systems zápatí prezentace 13 Classification How to use Graphical user interface, text, application Description automatically generated Adobe Systems Deep Forest 14 Results ̶From article promising – comparable to neural networks ̶ ̶Several models in different fields (examples) ̶Anti-cancer drug response https://www.sciencedirect.com/science/article/abs/pii/S1046202318303232 ̶E-commerce consumers’ repurchase behavior https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0255906 ̶Detection of Cash-Out Fraud https://dl.acm.org/doi/abs/10.1145/3342241 ̶Classification of cancer subtypes https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-018-2095-4 ̶ Adobe Systems Deep Forest 15 Their results Adobe Systems Deep Forest 16 Their results Adobe Systems Deep Forest 17 Their results Adobe Systems Deep Forest 18 Their results Adobe Systems Deep Forest 19 Layers Adobe Systems Deep Forest 20 Compared to other classifiers ̶10 other classifiers, 55 datasets ̶ ̶Better than average in 85% cases ̶Average gain 5.5% ̶47% best accuracy ̶Better than Random Forest in 72%, average difference 0.019 Comparable runtime* (on average 3 times slower) Adobe Systems Deep Forest 21 Compared to Random Forest A graph of blue lines Description automatically generated Adobe Systems Deep Forest 22 Parameter tuning ̶https://deep-forest.readthedocs.io/en/stable/parameters_tunning.html# ̶ ̶For accuracy (n_estimators, n_trees, max_layers, use_predictor) ̶ ̶Faster speed (n_jobs, n_bins, max_deapth, n_estimators, n_trees, min_samples_leaf, n_tolerant_rounds) ̶ ̶Lower memory (partial_mode) ̶ ̶ Adobe Systems Deep Forest 23 Parameters ̶class deepforest.CascadeForestClassifier(n_bins=255, bin_subsample=200000, bin_type='percentile', max_layers=20, criterion='gini', n_estimators=2, n_trees=100, max_depth=None, min_samples_split=2, min_samples_leaf=1, use_predictor=False, predictor='forest', predictor_kwargs={}, backend='custom', n_tolerant_rounds=2, delta=1e-05, partial_mode=False, n_jobs=None, random_state=None, verbose=1) ̶ Adobe Systems Deep Forest 24 Experiments with parameters ̶2*2*3*4*4*4 = 768 parameter settings with grid search ̶57 datasets -> 43 776 runs ̶5-fold cross validation -> 218 880 runs ̶ ̶ Adobe Systems Deep Forest 25 Bin_type and criterion Adobe Systems Deep Forest 26 Max_layers and n_estimators Adobe Systems Deep Forest 27 N_trees Adobe Systems Deep Forest 28 Predictor and use_predictor Adobe Systems Deep Forest 29 Metalearning ̶To create a model able to recommend parameters for a selected dataset based on its metadata ̶To reduce the amount of parameter combinations needed Adobe Systems Thank you for attention Main source: https://arxiv.org/pdf/1702.08835.pdf