Limit to SFC instances?

What order of magnitude of sequential function chart instances starts to become “too many”?

To illustrate, I would like to begin a network transaction and wait for it to change state. As it clicks along, I want an SFC to be tracking it. But I don’t really know how many instances of this will be fired up. I expect it to be on the order of a few dozen at any given moment, but it’s not hard to imagine a pathological edge case where hundreds or even a couple thousand will be spun up in short order.

It’s possible my real question is if SFCs are limited to the overhead threads normally have. I won’t lie: this may be simple paranoia since reading Raymond Chen’s post on thread limits years ago (The Old New Thing: Does Windows have a limit of 2000 threads per process?). But it helps to look both ways down a one-way street…

Any opinions? It’s not a real problem, since I can always add some logic to do throttling, but if I can avoid it, I would like to.

It would be an interesting experiment to try to create as many SFC instances that you could just to find out how many could be generated. Of course do this on a test machine.

Turns out I really needed to test this after all, and I think it’ll be fine. Over about a minute and a half it spun up about 3000 SFCs. The deadweight SFC is very light, since I was just testing the mechanism itself; the limiting factor ended up being just how heavy computationally the SFC spawned are. Attached are two simple SFCs that do the job (along with a simply log-to-console python script).

Basically, SFCs are given enough priority in the Gateway that you really want to keep them lean, both on memory and execution load. If the driver in the test has its timer set to 1ms and the children just continually add one as fast as they can, the Gateway will become unresponsive. It doesn’t seem to wedge itself, per se, but it will basically become incapable of dealing with network traffic, and all connections will eventually fail via timeout (at least it did for me that one time). Rate limit the spawning to something like 40 a second, and make the children pausable, and the Gateway won’t break a sweat.

So I think it’s safe to assume that the Gateway is managing the SFC threads inside its own process engine, and there shouldn’t be any worry about over-loading the OS. (I may re-run the test again for a few hours just for completeness).

[code]#Place in a shared library sfc_loadtest
def dump_core(message):
from system.util import getLogger
from system.tag import read
logger = getLogger(“SFC_LoadTest”)

logger.info(message)

memUsage = read("[System]Gateway/Performance/Memory Usage").value
memMax = read("[System]Gateway/Performance/Max Memory").value

logger.info("Mem : " + str(memUsage) + "/" + str(memMax) + " (" + str(memUsage/memMax) + ")")
logger.warn("SFCs: " + str(read("[default]sfc_count")))[/code]

Result:

WARN 2015-02-17 3:39:35 PM SFC_LoadTest SFCs: [2988,Good] (I) 2015-02-17 3:39:35 PM SFC_LoadTest Mem : 110977256/1046937600 (0) (I) 2015-02-17 3:39:35 PM SFC_LoadTest Driver fully dequed! WARN 2015-02-17 3:39:35 PM SFC_LoadTest SFCs: [2980,Good] (I) 2015-02-17 3:39:35 PM SFC_LoadTest Mem : 110977256/1046937600 (0) (I) 2015-02-17 3:39:35 PM SFC_LoadTest Driver canceling... WARN 2015-02-17 3:38:09 PM SFC_LoadTest SFCs: [0,Good] (I) 2015-02-17 3:38:09 PM SFC_LoadTest Mem : 263130464/1046937600 (0) (I) 2015-02-17 3:38:09 PM SFC_LoadTest Driver starting

(Incidentally, the memory thing here is a complete red-herring. As you can see, the values don’t really make much sense. Basically, the memory will mostly be an artifact of the SFC load, I think.)
driver.xml (4.5 KB)
deadWeight.xml (2.16 KB)

Cool, thanks for the information!