InterJournal Complex Systems, 259
Status: Submitted
Manuscript Number: [259]
Submission Date: 981227
Revised On: 981227
Paper is not of general interest
Author(s): Chris Phoenix

Subject(s): CX.06, CX.17, CX.68

Category: None Specified


As I will show with specific examples, the paper presented does not provide information or insights that are valuable to most researchers. The investigation it discusses is overly specific and unrelated to the real world; further, the paper fails to make the connection to broader applications. After supporting this statement, I will describe a shift of focus that could generate a paper which could shed new light on many simulation attempts.

A simulation should provide results that are either generalizable, or related to a specific real-world domain. Although this paper makes a claim that the DASCh simulation (when run with demand higher than supply) produces results that are related to real-life behavior, this claim is unsupported and unlikely to be true. In addition, the paper describes properties of the model; however, these properties are only true of models dominated by the modulo function, and it is unclear whether many interesting models use that function. As a final attempt to provide value, the paper describes some rules of thumb for modeling systems; however, the justification for these rules of thumb is scanty since they are not very general, and the discussion does not mention a major pitfall which can result from following the advice.

The paper claims to have some connection to reality, that is, to the way supply chains work in real life. However, it gives no reason to think that the modulo function is in any way related to real-life behavior. In fact, at, the rules that generate the modulo function (in particular "An order ships only when complete" are listed as "Operating assumptions," rather than as "Observed industry practice." The DASCh model may be realistic for order sizes within the capacity of the suppliers. However, in real life a consumer who was waiting for six backlogged orders of 220 pieces would surely not limit his actions to placing yet another 220-piece order! Yet this is precisely the behavior modeled in DASCh ("the backlog of over-capacity orders"), when the supplier can only supply 100 pieces per month and the consumer wants 220. Likewise, there is no discussion of the possibility that the supplier would be asked to ship a partial order to mitigate the damage caused by late delivery. In short, the paper does not give any support to its claim that the modulo function, or likewise the rules used to implement the Inventory Oscillator, have any connection with reality in the domain under investigation.

The paper also spends considerable time describing properties of the modulo function. Without more justification for the connection between the modulo function and real-life supply chains (or indeed any real-world situation), this section is of questionable utility. It is true, for instance, that the modulo function will not go chaotic; but this does not say anything about the possibility of real supply chains (in which the modulo function, if present at all, is only one of many interacting behaviors) going chaotic.

Finally, the paper gives general recommendations for studying systems, based on the idea of developing "tools for the job." It contrasts the properties of an agent-based model, an equation-based model and an analytical formalism. It is true that these are all potentially useful tools with different properties. It is possible that the agent-based model may point out underlying simplicities in the system, which can be described and analyzed using more formal models. This is the basis for the recommendation to start with an agent-based model, and then try to develop an equation-based and an analytical model. However, the discussion of tools is incomplete because it does not address how to tell when a tool should _not_ be used. When studying a real-world system, a model should not be used unless it reproduces some behavior of that system. At each new level of abstraction, the modelers must question whether their model is still close enough to reality to be worth studying. Furthermore, there is no evidence given to support the idea that a given complex system is likely to contain either simple behaviors or behaviors that can be reproduced by simple rules. For systems which are unlikely to contain simplicities, any apparent simplicity should be treated as a warning of a possibly incorrect line of inquiry. An agent-based model, because its behaviors emerge from the interaction of several rules, may conceal the fact that its behaviors are simpler than they may first appear when plotted on a graph: the behavior of the agent-based model may already be qualitatively simpler than its real-life counterpart. Generating ever-simpler models is of questionable utility, unless there is some evidence that the real system is likely to contain simplicities.

In this case, an agent-based model with relatively simple rules developed some behavior that looked interesting. That behavior was studied, and even simpler models were developed, apparently without questioning whether the behavior or models were too simple to be realistic. From all evidence in the paper, the tools that were developed tell more about the modulo function than about supply chains or inventory levels; in other words, following these recommendations led to a misapplication of effort. It is clear, then, that these recommendations should not be followed unquestioningly, and the paper gives no guidelines for appropriate questioning. The paper makes no mention of the potential pitfalls of the agent->equation->analytical approach, and so its recommendations may actually lead modelers to pursue non-useful paths.

The paper is difficult to read overall, because it seems to be switching between two different interpretations of the same events. For example, at one point it mentions the "discovery" of an "inventory oscillator". Later it becomes clear that this is simply the modulo function, and there is no justification given for the term "inventory" (implying a connection between modulo and industry). The fact that the agent model produces more complicated graphs than the equation model does not justify the statement, "...[we] were able to discover a much wider range of interesting behaviors than in the ODE model..." For example, the difference between the single-sawtooth and double-sawtooth waveforms (figures 4 and 5) is simply a straightforward property of the modulo function. The data points P(i) = A*i mod B will cycle through several values. If A and B have a large common divisor, then there will not be many different data values generated; if they are largely incommensurate, the data will cycle through many values. At one point the paper acknowledges this: "The occurrence of multiple frequencies is stimulated... by their incommensurability." However, earlier in the same paragraph, it states, "This degree of overload generates qualitatively new dynamical behavior... a broad sawtooth with a period of eleven, modulated with a period-two oscillation." It should be clear that this behavior, though perhaps "new", is not interesting, and does not deserve a separate description. Furthermore, it has nothing to do with the magnitude of the overload, but only with the specific choice of integer. Overall, the paper reads as though it were written in two stages, where the second stage consisted of adding disclaimers at various points--without modification of the statements being disclaimed. It would benefit from a unification of the points of view or stages of composition.

In summary, this paper does not contain any insights of value to most modelers; where it claims to describe the real world, it does not give any evidence that it does so. However, the paper does provide an example of a pitfall of modeling: being distracted by a model and studying its properties without questioning whether the source of the behavior is related to the real-world system under investigation. In this case, the model (exhibiting the behavior of the modulo function) was studied extensively before it was realized that this function was responsible for the behavior. I do not mean to suggest that it would have been easy to prevent this mistake; a physicist, a computer scientist, and a Scientific Fellow studied the function for months without recognizing it. Rather, the fact that a simulation effort with this much brainpower behind it made this mistake suggests that it may be more common than we'd like to admit. Although a candid assesment of the process of study, and the risks of the generalized form of this mistake appearing in other studies, may be uncomfortable for all concerned, the alternative is worse: that many other studies may interpret behavior that seems interesting as being truly complex, and make the further assumption that the presence of complexity indicates the success of the modeling effort. Interested readers are referred to articles on the NECSI discussion list at ref1 and ref2; these discuss this simulation, another unrelated effort, and some exploratory suggestions for detecting when a simulation does not match the real-world data.

Retrieve Previous Revision's Abstract
Submit referee report/comment

Public Comments: