I see the code in ProgressInfo::serializeStats, don't know when that was introduced, but agree it's flawed because it will miss stops/starts.
However, for most activities, a single bit each for start and stop is sufficient.
It is insufficient if activities are nested in CQ, but in Thor at least, many most are not.

So it would be a pity to serialize perhaps 2 * 8[bytes] * activities for activities which can at most start and stop once (i.e. all non CQ activities).
For those (non-CQ activities) the current serialized info is sufficient.
For CQ activities, it's going to need to send separate counts.

>Alternatively, we could make sure the row processed count (with the start stop flags) is serialized everytime the activity stops or the activity starts and we can sum up the number of start & stops on the master.

That would be prohibitively expensive especially as only applies to CQ, so you expect it to be pretty frequent.

There won't be much overhead in the slave activities to keep track of start stop counts per act.
And as long as we're serializing them periodically (as now), then it's ok, except for the additional network overhead, but I think that's okay too.

@jakesmith Thor slaves do not track the number of starts and the number of stops.  They  track the start or stop state using a flag (using the top 2 bits of row processed count). When the slave serializes the stats, it serialize the state of the flag at the point of time that the serialization occurs.  So, there are 2 problems

1) As we are using flags rather than using counters, we do not actually know the number of starts and the number of stops

2) Whilst the activity executes, the stats are serialized intermittently. That means for most activities the start and stops are missed.  i.e there could be 2 starts and 1 stop.  The only thing that would be serialized to the master is the 1 stop.

When the activity completes, the final states are serialized to the master - i.e. the stop state.

If we want to track an accurate count of the number of starts and stops, we will need to track the counts with a counter.  We'll then need to serialize the actual start and stop counts, so we won't be able to use 1 bit of the row count.  Do we (1)steal more bits from the row count to track the number of start and stops (2) serialize the start & stop count separately (independent of the row processed count).

Alternatively, we could make sure the row processed count (with the start stop flags) is serialized everytime the activity stops or the activity starts and we can sum up the number of start & stops on the master.

If we just want to track the activity to make sure the number of starts and stops match, we could do that on the slave activity. And we could serialize the top 2 bits for row processed as follows 11 - matching start & stops, 01 - too many stops, 10- too many starts and 00 - activity not started or stopped.

Reassigning to you for now, as you're working in this area.