How to ensure data integrity during transmission

Posted 2020-08-30 legion

1. XMODEM. It‘s old, it‘s bad, but it is widely supported both in hardware and in software, with libraries available for literally every language and market niche.

2. HDLC - High-Level Data Link Control. It‘s the protocol which has fathered lots of reliable protocols over the last 3 decades, including the TCP/IP. You can‘t use it directly, but it is a template how to develop your own protocol. Basic premise is:

   • every data byte (or packet) is numbered
   • both sides of communication maintain locally two numbers: last received and last sent
   • every packet contains the copy of two number
   • every successful transmission is confirmed by sending back an empty (or not) packet with the updated numbers
   • if transmission is not confirmed within some timeout, send again.

   For special handling (synchronization) add flags to the packet (often only one bit is sufficient, to tell that the packet is special and use). And do not forget the CRC.

Neither of the protocols has any kind of session support. But you can introduce one by simply adding another layer - a simple state machine and a timer:

• session starts with a special packet
• there should be at least one (potentially empty) packet within specified timeout
• if this side hasn‘t sent a packet within the timeout/2, send an empty packet
• if there was no packet seen from the other side of communication within the timeout, the session has been termianted
• one can use another special packet for graceful session termination

That is as simple as session control can get.

 

=----------------from StackOverflow  Dummy00001

 

 

If you want reliable data transfer, there is absolutely no way to use any USB-to-serial bridge correctly without hardware flow control, and without dedicating at least all remaining RAM in your microcontroller as the serial buffer (or at least until you can store ~1s worth of data).

I‘ve been using FTDI devices since FT232AM was a hot new thing, and here‘s how I implement them:

1. (At least) four lines go between the bridge and the MCU: RXD, TXD, RTS#, CTS#.

2. Flow control is enabled on the PC side of things.

3. Flow control is enabled on the MCU side of things.

4. MCU code is only sending communications when it can fit a complete reply packet into the buffer. Otherwise, it lets the PC side of it time out and retry the request. For requests that stream data back, the entire frame is dropped if it can‘t fit in the transmit buffer at the time the frame is ready.

5. If you wish the PC to be reliably notified of new data, say every number of completesamples/frames, you must use event characters to flush the FTDI buffers to the hist, and encode your data. HDLC works great for that purpose and is documented in free standards (RFCs and ITU X and Q series - all free!).

6. The VCP driver, or the D2XX port bring-up is set up to have transfer sizes and latencies set for the needs of the application.

7. The communication protocol is framed, with CRCs. I usually use a cut-down version if X.25/Q.921/HDLC, limited to SNRM(E) mode for simple "dumb" command-and-respond devices, and SABM(E) for devices that stream data.

The size of FTDI buffers is immaterial, your MCU should have at least an order of magnitude more storage available to buffer things.

If you‘re running hard real-time code, such as signal processing, make sure that you account for the overhead of lots of transmit interrupts running "back-to-back". Once the FTDI device purges its buffers after a USB transfer, and indicates that it‘s ready to receive more data from your MCU, your code can potentially transmit a full FTDI buffer‘s worth of data at once.

If you‘re close to running out of cycles in your realtime code, you can use a timer as a source of transmit interrupts instead of the UART interrupt. You can then set the timer rate much lower than the UART speed. This allows you to pace the transmission slower without lowering the baudrate. If you‘re running in setup/preoperational mode or with lower real-time task load, you can then trivially raise the transmit rate without changing the baudrate. You can use a similar trick to pace the receives by flipping the RTS# output on the MCU under timer control. Of course this isn‘t a problem is you use DMA or a sufficiently fast MCU.

If you‘re out of timers, note that many other peripherals can also be repurposed as a source of timer interrupts.

This advice applies no matter what is the USB host. --from StackOverFlow Kuba Ober