Serial interface traffic optimisation

I'm looking at improving my buffer management / concatenation system for the serial interface and trying to achieve a 'topup' style of operation for optimal flow management. Just wanted to double check some fundamentals..

AIUI the serial interface provides buffer space for 45 characters between the leading \ and the <CR>

If I include an acknowledgement character (g-z) within each message am I safe in assuming that once the message is acknowledged then all the space that that message occupied (length) is free again in the interface buffer ? This applies to all the variants ? . # $ % and ' ?

What's the difference between the ' and the ! below , aside from checksum issues ?

Can I also assume that should I receive a ! back from the interface, caused by overflow rather than checksum fail, that everything within that particular command (from \ to <CR> ) will have been dropped and and so can safely be resent ? I don't expect to encounter this but just checking.

Lastly - concatenation. My hardware sits as a gateway between an RS232 attached controller and C-Bus translating commands. ON receipt I am not aware if the current command from the controller is standalone or the first of a long sequence so I buffer each command and in a separate process send the commands to C-Bus as fast as possible. If there are queued commands then I check to see if I can concatenate the latest with the previous (unsent) one to speed things up, as requested by the guidelines. This has the effect however that the first command can never be concatenated with a later one as it is processed (being sent) straight away . Has anyone any suggestions to improve this ? I really don't want to enforce artificial delays though.

I do note that a very early post in this forum requests that the saturation of C-Bus cmd traffic be constrained to 5 per second peak and 2 per second sustained. I haven't as yet done this but I could add such throttling in the queue dispatcher routine.

Lastly XON/XOFF - I haven't used this as it is awkward within my existing serial libraries , plus it seems XON is only sent at totally empty buffer so it provides more start/stop than the 'topup' operation I'm implementing, which seems more optimal. AIUI XOFF is sent at 14 bytes in the buffer (28 ascii coded pairs ?) and the buffer is stated to be 20 bytes max. This doesn't seem to equate to the 45 character limit allowable between / and <CR> though..

which is possibly ...

\= 1 byte
44 ascii coded hex pairs = 22 bytes
g-z = 1 byte
<CR> = 1 byte

Total 25 bytes ?

Thanks in advance for any suggestions,

Kevin

 

I'm going to sidestep most of your questions first because the answers may not be relevant if you are only after optimising the speed of transfers.

we have looked at this problem and if you are interfacing directly to a PC Interface, it is possible to get a reliable transfer of messages and make use of a good percentage of the bus capability by taking advantage of the fact that the PC Interface is double-buffered.

To get the most out of a PC Interface, and still make full use of the message confirm (using a lower case alpha character and matching with a dot for succesful transmission), you can do the following when sending a series of commands:

1) send a message, using contatenation to build up long messages with several commands for maximum throughput
2) carriage return to issue on C-Bus
3) immediately send a second message (don't wait for reply)
4) carriage return to load into PC Interface buffer
5) wait for reply (dot and alpha relate to first message)
6) send a third message.
7) wait for reply (dot and alpha relate to second message)
etc.

note that the lower case alpha allows matching the confirmation to the message sent.

While Xon/Xoff work, they have proven to be less useful than using the message confirmation available from the PC interface.

The PC Interface has a 22 character buffer accepting serial data at 9600 baud. Every 8ms, bytes are copied from this buffer into a second 'message' buffer (also 22 bytes long, but compacted with 2 HEX characters per byte). When the carriage return or other special character is seen in processing of the message buffer, the message is parsed and either sent on to the C-Bus network, discarded, or or used to generate a reply direct from the PC Interface (if the message is addressed to the PC Interface). Any of these actions frees up the entire 22 byte message buffer.

To get best performance, turn off echo, don't use xon/xoff, and use a lower case alpha to get message confirmation.

I think your concatenation system should work fine.. perhaps send the first (short) message out followed by a carriage return, then send the concatenated message immediately after, as above.

The above method does not require any 'throttling', and makes best use of the network. If you are sending into a very busy network, you can expect occasional NACK symbols returned, and then you can make your own decision as to what to do (probably re-transmit the message after a 1s hold-off period). As long as you are not getting any errors, it is pretty safe to assume that other communication on the network is not being affected by your burst of data, but if you see any errors, that implies that some data may have been lost on the network, which may not be recoverable.. handle this situation with care. Perhaps some experimentation is required here.


Don

 

That's most helpful Don and it's great because it achieves an overlapped operation meaning I have one message queued in the interface at all times. Seems ideal. In trying my previous algorithm I had noticed a significant slowdown in speed as I had always to await confirmation of my most recent command as compared with the 'previous' one as you suggest.

Just to clarify - I am able to send say

\ <45 chars> <checksum> g-z <CR>

because after every 8ms (approx 8 chars) the first non compacted buffer is cleared and copied to the second compacted buffer where overall the whole 45 char message will now fit after compaction ?

Is there a problem though now because if I immediately send my second 45 char message I require the compacted buffer space be available by mid transmission or the first buffer will overflow ? So this relies on the compacted buffer being actioned on C-Bus, and cleared by this time ~20ms ? Is there a possibility , particularly with long concatenated commands or multiple hub bridged commands that this will fail in a non congested network - or only if the network is congested ? Is that why you're saying I must watch for ! s or is it ' s ? I'm not sure on the difference - is ! where the message was just too long (or checksum fail) and ' where it couldn't be copied to the second buffer in time perhaps.

K

<edit> Also realised that this would now mean that my first two commands to C-Bus wouldn't get compacted as I can typically send two commands now instantly before I am awaiting an acknowledge. (Before it was just one). This is because the first time I have unsent locally buffered data that I can append/compact into is message three. It's an interesting balance between message size (compacting) and number of messages / double buffering. Preliminary testing however shows that although compaction is reduced this method is significantly faster and appears to create more coincident changes on C-Bus.

 

We have tested this extensively and have found that we just can't send two consecutive messages to an idle PC Interface at 9600 baud fast enough to upset things. If you do find any problems, please me know how you achieved it, and I will try and replicate the condition in the lab.

 

Don , so far I haven't seen any problems yet either (I'm at 9600) - seems to work 100%

Another Q. Does the compacted buffer only ever hold one message , or if the messages were short enough (not concatenated at all) could it hold two or more perhaps ? Likewise I assume the first buffer might hold a couple if space permitted ?

Just asking because if so then if I am sending very short messages I could increase that gap for acknowledges to more than the penultimate one. (If I added a fancy routine to actually know how many characters were pending in buffer space).

My practical issue is that in trying to implement 'scenes' externally there are still appreciable delays between say 8 lights all coming instant ON at once. My first two messages don't (can't) get concatenated and so are short and would probably both fit in the compacted buffer , so I may be able to send the third or more as well. I am not sure where the delays are here but if I send them without handshaking it is much faster although I drop data at around 6 commands of course. This leads me to suspect the ack pacing adds signifiant speed degradation, but gains dependability of course which is far more important .

It may be that I should just accept this situation and instead try and keep scenes implemented on C-Bus itself for maximum speed and triggered from my controller.

It's a shame there is no way of 'preselecting' groups and then issuing one command to them all to achieve coincident change. Even native C-Bus scenes are noticeably sequential. I have had to work around this using 'Area's' in one application, or relay units with a common switched 'master' feed supplied from another relay or dimmer channel even.

I'm also considering adding a directive command to the controller that instructs my gateway to force concatenation of all the next n commands to achieve maximum coincident change at the expense of a slight initial delay.

e.g.
Concat 4
A instant ON
B instant ON
C instant OFF
D instant ON

(as currently implemented this would be 3 C-Bus commands and the concat will reduce it nicely to 1)

K

 

The compacted buffer can only ever hold one message at a time, so it is well worth your while to concatenate as many commands as you can if you want the closest to an 'instant' response.

 

Hi Ashleigh,

Interesting, I guess this is sort of what I was meaning when I talked about artificial delays, ie if I did change to pause ever so slightly before sending it is possible that more commands would have arrived and some would still be pending allowing concatenation.

I do much the same in terms of the buffer - ie when adding to the buffer I check to see if I can concatenate the command within the previous message if it hasn't yet been sent. I don't look beyond the previous as that would actually place things out of order, and any prior ones should have been topped up anyway.

The double buffering sort of works against this in that you can send things quicker but then you lose the concatenation ability. It's an interesting balance to achieve best visual results and given Don's response re single commands in the compacted buffer I think I'll have a play and see which mixture appears best in real world scenarios.

Kevin