Allowing multiple permits to send in the UDP layer may help hide the latency of the request-to-send flow control.

Currently the UDP layer contains code for supporting multiple active permits, but it has problems:
i) All slots are given to the first request
ii) when a new request comes in, it uses slots available, and does not take into account the outstanding permits.
This will potentially cause the queue to be significantly oversubscribed if the slaves are sending large quantities of data.

I think the current scheme can be adapted to work well with the following changes:

• Add a maxInflight member to the UdpSender class

• When a permit is granted, set maxInFlight to the number of slots

• Before a client sends a sequence of data packets it sends a new flow control message - send_started with the number of packets it is sending.

• When the receiver gets that flow control message it sets maxInFlight to the number of packets being sent.

• When the receiver receives a non-duplicate packet it decrements maxInFlight.

• The slots allocated in a permit are limited by the queue size and the sum of all maxInFlight values for the active permits. (Ignore -ve maxInFlight in case it has wrapped.)

I think with the flow control and data receiver on the same thread it should perform well.

comments/thoughts?

It may also be worth sending a receive_complete message from the receiver to the client once the send_completed/request_to_send_more is received. It would send back details of the packets received which would allow the sender to free up any packets that are received and possibly resend any that are missing? This needs more thought.