Chaos

With the camel-chaos JAR on the classpath, chaos is switched on with profile dev (the default of camel run), or with camel.chaos.enabled=true (for a plain Main, a test or a deterministic simulation test). It is never switched on with profile prod, also when camel.chaos.enabled=true. Without a profile, the usual case for a plain Main, Spring Boot and Quarkus, chaos stays off unless camel.chaos.enabled=true.

With camel-main, when camel.chaos.* options are set and chaos would be switched on (profile dev or camel.chaos.enabled=true), but the camel-chaos JAR is not on the classpath, camel-main logs a WARN that the options are ignored.

When chaos is switched on, it installs its hooks when the CamelContext initializes, before the routes are created, and logs a WARN that it is on. The hooks do nothing until a rule is added. When chaos is off, nothing of it is installed (only the lifecycle strategy that made this decision stays, and does nothing), and camel.chaos.rules is ignored.

camel.chaos.rules is marked insecure:dev: with camel-main and profile prod (which sets camel.security.policy=fail) an application with rules configured refuses to start. Without camel-main this check does not run, and chaos stays off with profile prod anyway. Rules set from Java (chaosConfig().withRules(…​)) are not checked either, as for every insecure:dev option.

The options are read from the properties component, and are camel-main options (ChaosConfigurationProperties).

For example, in application.properties:

Rules can also be added, changed and removed from Java while the application runs:

addRules adds rules separated by ;, as in camel.chaos.rules, all of them or none when one is not valid, so they are in place at the same time. checkRule says what a rule matches nothing of in the context now, see below.

With the Camel CLI, against a running integration (myapp is its name or pid):

chaos-add prints the id of the new rule (r1, r2, …​), and chaos-change the version of the rule that replaces it. --rule can be given more than once (or a rule can hold several separated by ;): the rules are added together, all or none, which matters when one must be in place before the other fires, as each chaos-add takes a moment. The ids are per integration, so chaos-add, chaos-change and chaos-remove need the name or pid of one integration; chaos-clear without one acts on every running integration that has chaos switched on. The rules that chaos-clear removes stay in the report; with --reset the report starts again, for a new experiment: no rules, route counts or verdict, as for an application that just started (with the same seed, and the ids of new rules going on from the last one). An id that does not exist is "not found" (exit code 3). camel get chaos lists the current rules with their id, version, state, matches and faults, the observed rate (RATE), when the rule last fired (LAST) and the effect of its faults (EFFECT), and --report shows the report, with the recent faults (also as JSON with --json); see The live state of the rules. Their NOTE column says what a lifecycle rule does to its route (route poller down, back in 12s) and why a rule did nothing when it was skipped. Without a rule to show, camel get chaos says so on stderr (no rules, chaos off, or no integration with camel-chaos).

A rule that matches nothing in the integration now is added with a warning, which chaos-add and chaos-change print, and which is logged: an endpoint= or component= pattern that matches no endpoint of the context, an at=poll for endpoints that no route consumes with a scheduled poll, a route= or step= pattern that matches no route or step id, and an expectation of a route that does not exist. Such a rule fires 0 times, and is often a typo; the warning names some of the endpoints, routes or steps there are. It is not an error, as the routes may still change (a toD creates its endpoints when it sends, and a route can be added later). The rules of camel.chaos.rules are checked when the context has started.

The commands go through the chaos action of the CLI connector, which calls ChaosManager: a rule goes through the same parser as camel.chaos.rules, and one that is not valid is refused with the reason. They work only when chaos is switched on in the integration (see Switching it on): they cannot switch it on, and with profile prod they are refused.

The chaos developer console (/q/dev/chaos with camel run --console) shows the same rules and report, as text or JSON. It only reads: a rule cannot be added, changed or removed over HTTP.

A rule is one line: the target, the fault and its parameters as name=value, then at, rate, from and until, and optionally an expectation. A value with spaces or a ; is quoted with " or ' (\", \' and \\ inside quotes are a quote and a backslash).

The patterns of the targets are matched as text, with for any characters; they are not regular expressions. The match is case-sensitive and a can stand anywhere in the pattern, unlike the patterns of interceptSendToEndpoint, which ignore case and can be regular expressions: a rule, also one from a tool, cannot make matching slow. An endpoint= pattern is matched against the endpoint URI, with or without its query parameters, and the // after the scheme does not count (kafka:orders* matches kafka://orders?brokers=b1). A component= pattern is matched against the scheme of the URI only, so a pattern with a : (component=http:localhost) is rejected, with the endpoint= pattern to use instead. Some components give their endpoints the URI of their address: undertow:http://inventory/stock has the endpoint URI http://inventory/stock. When the scheme of the endpoint URI is not a name of the component in the context, chaos puts the default name of the component in front, and matches undertow:http://inventory/stock (with component=undertow, not component=http). A camel-netty-http endpoint has the endpoint URI netty-http://http://inventory/stock, and a camel-netty-http component added under another name is matched with netty-http in front too. A route= pattern is matched against the route id, and a step= pattern against the id of the step (to1, or the id set with id("enrich")).

A rule gets an id when it is added (r1, r2, …​), which change and remove take. A change replaces the rule with a new one with the same id and the next version (a rule is version 1 when it is added), and the replaced one stays in the report: so the report has an entry per version of the id, and a tool tells them apart by the version of the entry (the latest has the highest). An until count counts the matches, so a rule with rate=every:N and an until count below N would never fire, and is rejected. A rule ends by itself after its until, at the latest after camel.chaos.maxDuration, so a forgotten rule cannot keep breaking a development instance; a stall or lifecycle rule ends when its route is back (see Route lifecycle faults). Rules stay when the routes are reloaded.

A rule with from=+30s starts 30 s after it is added (a rule of camel.chaos.rules 30 s after the context started), for faults after the warm-up of the application; until then it is waiting, and the matches do not count. Its until time counts from its start, so from=+30s until=1m fires from 30 s to 1 m 30 s after it is added. from is at most camel.chaos.maxDuration. A rule with rate=burst:5s/30s fires at every match during the first 5 s of each 30 s, counted from its start, and at none in between: outage windows, instead of the even spread of rate=0.17. The burst must be shorter than its period, and the period at most camel.chaos.maxDuration.

stall and lifecycle act on a route, not on its messages: route=poller stall duration=20s suspends route poller and resumes it 20 s later; route=orders lifecycle action=stop duration=1m stops and starts it again; route=orders lifecycle action=remove duration=10s removes it and adds the same route definition back. Each rule acts when it is added (a rule of camel.chaos.rules when the context has started), or from= later (route=poller stall duration=20s from=+1m), so at, rate and until are rejected; an expectation (expect=, within=) can be added.

A flapping route goes down more than once: route=orders stall duration=10s repeat=3 every=1m suspends route orders for 10 s three times, each time a minute after the last one went down. every must be longer than the duration, so the route is back before it goes down again, and the rule must be done within camel.chaos.maxDuration ((repeat - 1) * every + duration); repeat is at most 100. Each round acts as the first: a route that someone else stopped meanwhile is left as it is, and the rule ends there (skipped). Removing the rule between two rounds ends it, with the route as it is (back). The route= target is the id of one route, not a pattern, and the route must exist when the rule is added: a rule for a route that does not exist is rejected (with camel.chaos.rules the context does not start), and so is a duration longer than camel.chaos.maxDuration. While a lifecycle rule has a route removed, a rule for that route is still accepted (to change the rule, for example).

The action and the comeback run on a scheduled executor of the ExecutorServiceManager of the context (one thread, whose name ends with Chaos, created when the first such rule acts), not on the thread that adds the rule, so adding the rule returns at once, even while the route waits for its exchanges in flight to finish. The actions and comebacks of all rules run one after another on that thread, so a comeback can wait for the action of another rule, such as a stop that waits for its exchanges in flight; a rule that replaces one whose route is down acts after that route came back. The route is suspended, stopped, resumed and started through the route controller of the context (CamelContext.getRouteController()), and removed with CamelContext.removeRoute. A route that cannot be suspended (its consumer does not support it) is stopped and started instead, as Camel does for any suspend.

Chaos only brings back what it took down:

With the supervising route controller (camel.main.routeControllerSuperviseEnabled=true), a stop or suspend by chaos is a manual operation for the controller: it does not restart the route while it is down, and the comeback starts the route through the controller, which supervises it again (and retries a start that fails, with its back-off). A route that action=remove adds back is started through the route controller too, when adding its definition did not start it (the supervising controller takes a route that is added again under supervision, but does not start it), so it is supervised again as well. The supervising controller starts the routes in the background, so a rule of camel.chaos.rules can find its route not started yet, and then does nothing; add such a rule once the route runs.

action=remove needs the route definition of the route, so it works for the routes of the route model (Java, XML and YAML DSL). The definition that is added back is the one that was removed, as it was prepared, so the route comes back with the route configurations it had. A route created from a route template or a kamelet is rejected for action=remove (removing it closes the local beans of the template); stop and stall work for it.

The report shows what each lifecycle rule did, with the times after the rule was added, for example:

With repeat= the line tells the round, and between two rounds when the next one starts (round 1 of 3: suspended route orders at +2ms, resumed at +10s5ms (down 10s3ms); round 2 in 49s).

In JSON each action counts in fired.lifecycle, and the rule has a lifecycle object: action, route, duration, state (pending, acting, down, comingBack, back, skipped, left, failed or cancelled; pending also between two rounds), round (the times it took the route down), repeat and every (with repeat=), actsIn (the milliseconds until it acts, again), downAt and backAt (milliseconds after the rule was added, of the last round), backIn while the route is down, endedEarly when the rule was removed while the route was down, and the reason when it did not go as planned. camel get chaos --json and the chaos dev console give the same object.

ChaosManager.report() gives the report, as plain text (toText()) for a person and as JSON (toJson()) for a tool, and it is logged when the context stops (camel.chaos.reportOnShutdown), if any rule was added. It states the seed, each rule with its id and version, its matches, the faults that fired at each hook and the ones that were skipped (with the reasons, such as a stream body, counted in skippedReasons), its state (active, waiting for its start with from=, ended, removed or replaced), and per route. The rules are in the order of their ids, and the versions of an id from the latest. Per route:

The routes are counted while there are rules (also ended ones). Camel does not call the route policies while the context stops, so the exchanges that finish while the context drains them are not counted.

Without an expectation the verdict is PASS, as chaos makes exchanges fail on purpose, and the report does not say that a message was lost, except for the messages that a rule with at=dlq kept from reaching the dead letter channel, which then handled them: those count as lost (in the report of the rule as lostAtDlq), and make the verdict FAIL, with the reason. An expectation, expect=orders:100 within=2m, says that route orders completes 100 more exchanges after the rule is added and before the time is up (by default until the rule ends: its from plus its until time, else camel.chaos.maxDuration, which is also the longest within after the start of the rule). An exchange of the route counts once, the first time it completes in the route. An exchange that the error handler handled (for example one that went to the dead letter channel) completes, and so counts as completed, not as lost. The route id is not checked when the rule is added, as the route may come later (a WARN says so when a rule is added or changed, or the context starts, and no route has that id). An expectation is met when the count is reached in time, and unmet when the time is up without it: then the verdict is FAIL, with the reason, and the missing exchanges count as lost (100 expected, 97 completed: 3 lost). Before that it is pending, and so is the report (isFinal() is false): a tool must treat a PASS with a pending expectation as not decided yet. An expectation is cancelled when its rule is removed or changed first, or when the context stops first (also when the context is started again): the messages may still arrive, so it is neither met nor unmet, and it does not keep the report from being final.

The delay fault (and an error with after=) waits through TimeSource of camel-util, and the start and the end of a rule, its bursts, the time of an expectation and the times of a lifecycle rule are read through it too; the action and the comeback of a lifecycle rule are tasks of a scheduled executor of the ExecutorServiceManager. So in a deterministic simulation test (camel-test-dst) they run in virtual time: a 10 minute stall takes milliseconds, and with camel.chaos.seed set to the seed of the run, the same seed gives the same report again, times included. Chaos does not depend on camel-test-dst.

ChaosLifecycleDstTest in the tests of camel-chaos is such a test, and an example to copy: a route consumes a queue (a fake with a scheduled poll consumer) and calls a downstream system asynchronously (DstCamel.asyncStep, as a world needs for a route that is suspended or stopped while an exchange is in a downstream call). A stall or lifecycle rule, added while an exchange is in flight, takes the route down for up to 10 minutes while a step= rule fails the downstream call at a rate, and the test checks from the report that no message is lost (an expectation), that the redeliveries add up to the step faults, and that the route was down for exactly the duration.

The seed fixes the sequence of random draws of each rule (rate=<probability>, the delays of a range such as duration=100ms..2s, the random header values), not which exchange gets which draw: with several threads (concurrent consumers, parallel processing) the threads decide that, so two runs with the same seed can give different reports. With one thread and the same exchanges in the same order, the same seed gives the same faults.

The rules are also described as data, for the tools that help to write them (a form in a terminal UI, an AI that turns a request into rules, the docs): META-INF/org/apache/camel/chaos/chaos-model.json in the camel-chaos JAR, also in the Camel catalog (CamelCatalog.chaosModelJSon()). It has the targets, the hooks of each target and the faults each one offers (and whether it is reached only with at=), the parameters of each fault with their types and limits, the options, the kinds and ops of an expectation, the limits, and recipes: known rules in plain words, grouped by what goes wrong, each a rule with placeholders (${endpoint}, ${rate}) and what to pick them from (the endpoints the routes send to, the routes, the steps). To guide the one who picks a recipe, each says what it does, what it can uncover, when not to use it, and what to watch while it runs, such as:

The model is made from the tables of the parser, so it says what the parser accepts; a tool still adds a rule through ChaosManager or the CLI connector, which check it as for any other rule.