Common use of Evaluating a Modest Model to Yield Performance Artifacts Clause in Contracts

Evaluating a Modest Model to Yield Performance Artifacts. ‌ In this section, Modest models of the previous section are evaluated for each design, using different evaluation techniques. Besides discrete-event simulation, there are four ways of model checking, viz., TA-based, PTA-based, and efficient PTA-based, and efficient & scalable PTA-based. Each technique comprises three execution steps: (i) the Modest model is modified to be compatible with the given technique; (ii) a Modest tool is applied to the Modest model at least once yielding performance numbers; and, (iii) post-processing turns the performance numbers into artifacts. Next, we explain these steps for each technique. Discrete-event simulation yields latencies of services, subprocesses and resource utilizations. Latencies are obtained by enclosing each service and sub- process with stopwatches. An additional service counter and properties for each subsequent request of a given service then make it possible to retrieve individual latencies for a service. A resource utilization is obtained by adding a counter to a resource that keeps track of the total time it is processing. This counter value is divided by the total running time of the system, which is implemented as a global counter. For each run, MODES of the Modest toolset [10] is used once to perform a discrete-event simulation on the Modest model. MODES is instructed to use an as soon as possible (ASAP) scheduler for time, and uniform resolution for nondeterminism choice. This is a pragmatic and commonly used choice that does not need to reflect the real underlying structure [4]. Post-processing yields three performance artifacts. First, a latency bar chart (see Fig. 11(c)) is generated using GNUplot [20], which visually displays succeed- ing latency times. Second, a latency breakdown chart (see Fig. 11(a)) conveys the static process structure of a service extended with its dynamics, i.e., latencies and utilizations. The graph structure is derived by recursively traversing the iDSL process and resource. It is augmented with placeholders in one go wherever performance num- bers are needed. Next, these placeholders are replaced by the relevance Modest properties, after which GraphViz [6] renders the visualization. Third, a cumulative distribution graph (see Fig. 11(b)) displays latency times for different designs. Hence, they are convenient to get insight in the consequences of certain design decisions. To this end, the latency values of the different designs are gathered, combined, and turned into a plot using GNUplot [20]. TA-based model checking yields absolute service latencies. To make the model finite, real values that concern time (including loads and rates) are rounded to their nearest integer values. Additionally, probabilistic choices and infinite distributions are replaced by nondeterministic choices. Latencies of ser- vices are obtained by enclosing each service with stopwatches that reset after registering one latency. To reduce the state space size, no service counter is added; we do not retrieve which latency is the maximum one. Combined, this leads to a finite, decidable model. TA-based model checking is performed using MCTAU [1]. Via a binary search algorithm of [25], i.e., recursive function lb (in Table 8), an initial range of possible values is halved iteratively until one value Table 8. Function lb: compute lower bounds, pseudo code lb ([lbound:ubound]){ if (abs(ubound-lbound)<=1) return lbound check_value=(lbound+ubound)/2 UPPAAL (p = probability(latency<check_value)) if ( p=0 ) lb (check_value,ubound) else lb (lbound,check_value) } // case a // case b // case c O remains. This initial range is [0 : n], where n is a deliberate overestimation of the latency. Each iteration, one of the following three cases occur: (a) there is only one possible value left, which is returned; (b) model checking conveys that the probability that the value is in the lower half of the values is 0 in which case the upper half of the values is returned; or, (c) the lower half of the values is returned. The complexity of the algorithm is (log(n)), where n is the chosen overestimation. ∈ PTA-based model checking yields exact service latency distributions for each service; a Modest model is created for each service and both the minimum and maximum probability to compute the latency distribution of that particular service. In each Modest model, only the process of the given service is enclosed by stopwatches that record latencies of the service requests. The Modest models have one parameter time t R≥0, and return a probability p: the probability that the service completes within time t. In iDSL, however, a service leads to an infinite stream of service requests, each with its individual latency. Ideally, the average of this infinite stream of latencies is a measure for the performance of the whole service. Put formally: 1 PΩ(t) = lim k→∞ x x Σ Pn (t), (1) where PΩ(t) is the combined probability, n the service request number, t the latency time, Pn(t) the probability that service request n finishes within time t. This infinite sum cannot be directly computed. Instead, the computable geo- metric distribution [17] is proposed that is capable of detecting an absolute maximum latency, weighing service requests in an exponentially decreasing way: Σ n 1 ∞ PΩ(t) = (1 − ρ) − ρ Pn(t), (2) n=1 ≈ ∈ where ρ (0 : 1) is the geometric distribution parameter. The distribution is similar to (1), for ρ 0 and capable of finding the absolute latencies. 10 In Modest, the geometric distribution is implemented as a binary probabilis- tic choice every time a service request completes (as depicted in Fig. 8(a)): either the currently measured latency is returned, with probability ρ ( 1 in the figure), 10 or the next service request is evaluated, with probability 1 −ρ ( 9 in the figure). return Latency 1 return Latency 2 return Latency 3 return Latency 1/10 1/10 1/10 1/10 9/10 9/10 Latency 1 Latency 2 Latency 3 Latency Service Request 1 Service Response 1 Service Request 2 Service Response 2 Service Request 3 Service Response 3 Service Request Service Response

Evaluating a Modest Model to Yield Performance Artifacts. ‌ In this section, Modest models of the previous section are evaluated for each design, using different different evaluation techniques. Besides discrete-event simulation, there are four ways of model checking, viz., TA-based, PTA-based, and efficient efficient PTA-based, and efficient efficient & scalable PTA-based. Each technique comprises three execution steps: (i) the Modest model is modified modified to be compatible with the given technique; (ii) a Modest tool is applied to the Modest model at least once yielding performance numbers; and, (iii) post-processing turns the performance numbers into artifacts. Next, we explain these steps for each technique. Discrete-event simulation yields latencies of services, subprocesses and resource utilizations. Latencies are obtained by enclosing each service and sub- process with stopwatches. An additional service counter and properties for each subsequent request of a given service then make it possible to retrieve individual latencies for a service. A resource utilization is obtained by adding a counter to a resource that keeps track of the total time it is processing. This counter value is divided by the total running time of the system, which is implemented as a global counter. For each run, MODES of the Modest toolset [10] is used once to perform a discrete-event simulation on the Modest model. MODES is instructed to use an as soon as possible (ASAP) scheduler for time, and uniform resolution for nondeterminism choice. This is a pragmatic and commonly used choice that does not need to reflect reflect the real underlying structure [4]. Post-processing yields three performance artifacts. First, a latency bar chart (see Fig. 11(c)) is generated using GNUplot [20], which visually displays succeed- ing latency times. Second, a latency breakdown chart (see Fig. 11(a)) conveys the static process structure of a service extended with its dynamics, i.e., latencies and utilizations. The graph structure is derived by recursively traversing the iDSL process and resource. It is augmented with placeholders in one go wherever performance num- bers are needed. Next, these placeholders are replaced by the relevance Modest properties, after which GraphViz [6] renders the visualization. Third, a cumulative distribution graph (see Fig. 11(b)) displays latency times for different different designs. Hence, they are convenient to get insight in the consequences of certain design decisions. To this end, the latency values of the different different designs are gathered, combined, and turned into a plot using GNUplot [20]. TA-based model checking yields absolute service latencies. To make the model finitefinite, real values that concern time (including loads and rates) are rounded to their nearest integer values. Additionally, probabilistic choices and infinite infinite distributions are replaced by nondeterministic choices. Latencies of ser- vices are obtained by enclosing each service with stopwatches that reset after registering one latency. To reduce the state space size, no service counter is added; we do not retrieve which latency is the maximum one. Combined, this leads to a finitefinite, decidable model. TA-based model checking is performed using MCTAU [1]. Via a binary search algorithm of [25], i.e., recursive function lb (in Table 8), an initial range of possible values is halved iteratively until one value Table 8. Function lb: compute lower bounds, pseudo code lb ([lbound:ubound]){ if (abs(ubound-lbound)<=1) return lbound check_value=(lbound+ubound)/2 UPPAAL (p = probability(latency<check_value)) if ( p=0 ) lb (check_value,ubound) else lb (lbound,check_value) } // case a // case b // case c O remains. This initial range is [0 : n], where n is a deliberate overestimation of the latency. Each iteration, one of the following three cases occur: (a) there is only one possible value left, which is returned; (b) model checking conveys that the probability that the value is in the lower half of the values is 0 in which case the upper half of the values is returned; or, (c) the lower half of the values is returned. The complexity of the algorithm is (log(n)), where n is the chosen overestimation. ∈ PTA-based model checking yields exact service latency distributions for each service; a Modest model is created for each service and both the minimum and maximum probability to compute the latency distribution of that particular service. In each Modest model, only the process of the given service is enclosed by stopwatches that record latencies of the service requests. The Modest models have one parameter time t R≥0, and return a probability p: the probability that the service completes within time t. In iDSL, however, a service leads to an infinite infinite stream of service requests, each with its individual latency. Ideally, the average of this infinite infinite stream of latencies is a measure for the performance of the whole service. Put formally: 1 PΩ(t) = lim k→∞ x x Σ Pn (t), (1) where PΩ(t) is the combined probability, n the service request number, t the latency time, Pn(t) the probability that service request n finishes finishes within time t. This infinite infinite sum cannot be directly computed. Instead, the computable geo- metric distribution [17] is proposed that is capable of detecting an absolute maximum latency, weighing service requests in an exponentially decreasing way: Σ n 1 ∞ PΩ(t) = (1 − ρ) − ρ Pn(t), (2) n=1 ≈ ∈ where ρ (0 : 1) is the geometric distribution parameter. The distribution is similar to (1), for ρ 0 and capable of finding finding the absolute latencies. 10 In Modest, the geometric distribution is implemented as a binary probabilis- tic choice every time a service request completes (as depicted in Fig. 8(a)): either the currently measured latency is returned, with probability ρ ( 1 in the figurefigure), 10 or the next service request is evaluated, with probability 1 −ρ − ρ ( 9 in the figurefigure). return Latency 1 return Latency 2 return Latency 3 return Latency 1/10 1/10 1/10 1/10 9/10 9/10 Latency 1 Latency 2 Latency 3 Latency Service Request 1 Service Response 1 Service Request 2 Service Response 2 Service Request 3 Service Response 3 Service Request Service Response