Differences

This shows you the differences between two versions of the page.

Link to this comparison view

online:controlflowbenchmark [2009/05/28 11:05] (current)
Line 1: Line 1:
 +====== Control Flow Benchmark ======
 +
 +In the desire to derive a “good” model for the behavior observed in the execution log, shared notions of, e.g., fitness, precision and structure can be identified in the process mining literature. ​
 +As a consequence,​ several metrics have been developed to measure the quality of a (mined) model according to these notions. The Control Flow Benchmark plug-in can be used to calculate some of these metrics and provides an integrated view on the results. See also [1,2] for further information on this topic. ​
 +
 +===== Prerequesites =====
 +
 +You need to open all the process models that should be compared (and the reference log and/or reference model) before you start the benchmark plug-in. ​
 +
 +The process models do not need to be connected to the event log (the mapping is established by the benchmark plug-in itself), but the models are expected to 1) be a Petri net and 2) specify appropriate mapping information: ​
 +
 +  - You can use existing conversion techniques if your model is specified in a different modeling formalism. ​
 +  - You can ensure that your model properly captures all mapping information (i.e., invisible tasks, and the log event belonging to each visible task) by using the PNML export in ProM before re-opening the model again for using the benchmark plug-in. ​
 +
 +Note that there are example files you can use to play with the Control Flow Benchmark contained in your ProM folder at /​examples/​benchmark/​. The same examples are used in [2] as a running example and to explain the different evaluation dimensions. ​
 +
 +===== Control Flow Benchmark Settings =====
 +
 +Before the actual benchmark is started, one can choose which kind of metrics should be calculated. Furthermore,​ the reference model and/or the reference log, and the models to be compared (benchmark items) can be selected. Note that some metrics require a reference model to be present while others don't. Similarly, there are metrics that only compare the benchmark items to the reference model, and, therefore, do not need a reference log. If the input needed for a certain metric is not provided, the metric will remain disabled (see [2] for an overview of the metrics and their requirements).  ​
 +
 +{{documentation:​benchmark:​settingsscreen.png?​567|Control Flow Benchmark Settings}}
 +**Figure 1.** Benchmark settings allow for the selection of the metrics to be calculated
 +
 +Figure 1 shows a screenshot of the settings of the Control Flow Benchmark plug-in. Note that each metric provides a short help text if you click on it. 
 +Then, the benchmark process can be started. The plug-in transparently connects each model to the reference log and calculates the chosen metrics. ​
 +
 +{{documentation:​benchmark:​progressscreen.png?​567|Control Flow Benchmark Progress}}
 +**Figure 2.** Progress view of the Control Flow Benchmark process
 +
 +You can view the progress of this operation like shown in Figure 2. If a metric takes too long to compute, it can be either skipped (and potentially approximated) or the benchmark can be aborted, which brings you back to the initial configuration screen. ​
 +
 +===== Control Flow Benchmark Results =====
 +
 +All the benchmark metrics return values between 0 (interpreted as the "​worst"​ value) and 1 (interpreted as the "​best"​ value). Furthermore,​ they may be based on assumptions (such as no duplicate tasks in the process model) that might not be fulfilled. Each metric is expected to check its assumptions and to indicate the result being "​invalid"​ if pre-conditions cannot be met. 
 +
 +{{documentation:​benchmark:​barviewscreen.png?​567|Control Flow Benchmark Bar Profile View}}
 +**Figure 3.** Bar Profile view of the Control Flow Benchmark results
 +
 +The first result view you will see once the benchmark process is finished is the //Bar Profile// view (see screenshot in Figure 3). It visualizes the calculated values along a so-called bar profile in a fixed order. Green values are "​good"​ and red values are "​problematic"​ according to the corresponding metric. Invalid values are left out (i.e., the bar profile will have a gap at this place). The profile part between two values is interpolated. This way, this view provides a graphical overview that makes it easy to spot problematic areas, which can be successively inspected in further detail. ​
 +
 +{{documentation:​benchmark:​tableviewscreen.png?​567|Control Flow Benchmark Table View}}
 +**Figure 4.** Table view of the Control Flow Benchmark results
 +
 +The view can be changed to //Table// view (see screenshot in Figure 4) for inspecting the detailed results. The exact values are shown for each benchmark item, or "​invalid"​ is given if it could not be computed for a certain case. 
 +
 +{{documentation:​benchmark:​exportcsvscreen.png?​567|Control Flow Benchmark CSV Export}}
 +**Figure 5.** CSV Export of the Control Flow Benchmark results
 +
 +Finally, the calculated values can be exported in Comma Separated Values (CSV) format. The option to export the values with semicolon separators (rather than commas) makes it easy to directly open the file in MS Excel. ​
 +
 +===== Publications =====
 +
 +
 +{{page>​blogs:​pub2007:​the_need_for_a_process_mining_evaluation_framework_in_research_and_practice&​noeditbtn&​firstseconly}}
 +
 +{{page>​blogs:​pub2007:​towards_an_evaluation_framework_for_process_mining_algorithms&​noeditbtn&​firstseconly}}