libftdi Archives

Subject: Re: ftdi_read loses bytes (when latency is low)

From: Hendrik <chasake@xxxxxxxxx>
To: libftdi@xxxxxxxxxxxxxxxxxxxxxxx
Date: Tue, 02 Dec 2014 14:35:01 +0100
Hello Uwe,

> - What exact device are you using?

I am using the FT232R (actually the FT232RL in 28SSOP casing).

> How large is the receive buffer?

I mentioned 128 bytes before, but it depends on what you think TX and RX means ;). The datasheet says:

>> FIFO RX Buffer (128 bytes). Data sent from the USB host controller to the UART via the USB data OUT >> endpoint is stored in the FIFO RX (receive) buffer. Data is removed from the buffer to the UART transmit >> register under control of the UART FIFO controller. (Rx relative to the USB interface). >> FIFO TX Buffer (256 bytes). Data from the UART receive register is stored in the TX buffer. The USB >> host controller removes data from the FIFO TX Buffer by sending a USB request for data from the device
>> data IN endpoint. (Tx relative to the USB interface).

So the buffer that the microcontroller sends its data to is 256 bytes. That gives a bit more margin for 'not servicing' but there is not enough room for the complete packet to stay there a while.

>  Do you use any signalling between the FTDI and your serial device?

No there is no handshaking whatsoever, as soon as my device has the requested data complete it will send it in one go out of its serial port (into the FTDI device). In Windows however, the device and this configuration has run fine for about 5 years. We even have setups connecting about 35 of these devices to USB hubs simultaneously and it 'just worked' (with some quirks from the Windows FTDI driver though). But I do think that the Windows drivers are running their own very low level processes that keep on polling the USB devices because the load on the computers increase dramatically when using more of these FTDI devices simultaneously (only opening the port already gives a lot of stress on the CPU) especially with the latency set to a low value).

> With 500 kBaud you receive about 50 bytes per ms. With a small receive buffer and no RTS/CTS signalling > on the FT232R with its 128 byte buffer, overrun will happen in 3 ms. Your programm will wait forever for the lost bytes. > The libftdi programm runs as a user program and other programms may keep your from the CPU quite for some time.

That is indeed what I suspect is happening, an overrun occurs just at a timeslot that my program is not servicing the USB read request. I don't know the internals of libusb so I thought there would be/could be some logic inside to handle this but if it is all userspace that possibly is not the case. But WHY only with a low latency and never with a latency that is high(er than the time it takes for the microcontroller to assemble and send the complete packet).

> Is the Device directly plugged to the PC or via a hub.

The device is connected to a USB HUB (as we need to connect many of these to an embedded system in later stages of development). I will test though if the problems goes away when connected to USB directly. I also tried the async method by the way but that does not help. It makes it 'worse' as the 'ftdi_transfer_data_done (tc)' function never returns (it does not meet up with the requested number of bytes, I found a patch by you for adding a timeout to the 'ftdi_read_data_submit' call but it is not in the mainline branch?).

> Is it a real FTDI device or some fake rebuild?

Our PCBs are manufactured by a Dutch company. They have a reputation to keep up and I don't think they use Chinese remakes of IC's. Also; in Windows this problem is not observed at all.

[The 0162 USB messages]
> I don't see this behaviour in th pastbin logs...

They are there but a bit hidden: http://pastebin.com/raw.php?i=3Vwk2NAs You can see that the packet with size 327 comes in with 0162 and the following packets show 0160 again.

In this post you find a larger log of the program running for some time (constantly requesting the data packet with a latency timer at 1ms, the chunk size at 4096 and my 'read' function having a timeout of 1000 ms. My read function is a wrapper that calls ftdi_read_data for as long as I did not receive the expected total number of bytes until my own time out expired).

http://pastebin.com/raw.php?i=C9qb7axf

I am not completely sure of the status bytes and their bit order (B7 ... B0 or is it B0 ... B7). We receive 62 instead of 60 for the second byte on error, this COULD be the 'overrun' bit (B1) which makes sense (however it is not directly connected to that specific 64 byte packet, the overrun is probably observed in the data after receiving about 256 correct bytes (the RX buffer size)). The first byte however indicates 01 but that a bit (B0) that is documented to be 'must be 0' which it isn't).

So, to summarize: it is probably a problem that hits me 'by design'. I should not try to send buffers larger than the RX buffer of the FTDI chip (without using some kind of flow control) under Linux, at least not in user mode and with a low latency..?

Still, I don't get why the problem does NOT appear at all when I select a latency that is (way) larger than the time it takes for my device to respond (approx 4 or 5ms for packet building and then about 10ms (522 bytes @ 500kbps) for the actual serial transmission to the FTDI device). What is really different in those cases? Where is the data stored when the latency timer is higher? And why does the latency timer seem to interfere with the ongoing transmission. Does it go off twice but isn't it serviced twice, hence the data loss? But why do I miss a random amount of data and not a 64 byte frame then? Questions, questions, questions... :(

Regards,
Hendrik

--
libftdi - see http://www.intra2net.com/en/developer/libftdi for details.
To unsubscribe send a mail to libftdi+unsubscribe@xxxxxxxxxxxxxxxxxxxxxxx
Current Thread