PPTX

Practical TDMA for Datacenter Ethernet Bhanu C. Vattikonda, George Porter, Amin Vahdat, Alex C. Snoeren

Variety of applications hosted in datacenters

All-to-all

Performance depends on throughput sensitive traffic in shuffle phase

Gather/Scatter

Generate latency sensitive traffic

Network is treated as a black-box Applications like Hadoop MapReduce perform inefficiently

Applications like Memcached experience high latency

Why does the lack of coordination hurt performance? 3

Example datacenter scenario Traffic receiver

•Bulk transfer •Latency sensitive

Bulk transfer

Latency sensitive

Bulk transfer

4

Drops and queuing lead to poor performance Traffic receiver

• Bulk transfer traffic experiences packet drops • Latency sensitive traffic gets queued in the buffers

Bulk transfer

Latency sensitive

Bulk transfer

5

Current solutions do not take a holistic approach Facebook uses a custom UDP based transport protocol Alternative transport protocols like DCTCP address TCP shortcomings

Infiniband, Myrinet offer boutique hardware solutions to address these problems but are expensive

Since the demand can be anticipated, can we coordinate hosts?

6

Taking turns to transmit packets Receiver

Bulk transfer

Latency sensitive

Bulk transfer

7

TDMA: An old technique

Enforcing TDMA is difficult It is not practical to task hosts with keeping track of time and controlling transmissions End host clocks quickly go out of synchronization

9

Existing TDMA solutions need special support Since end host clocks cannot be synchronized, special support is needed from the network FTT-Ethernet, RTL-TEP, TT-Ethernet require modified switching hardware

Even with special support, the hosts need to run real time operating systems to enforce TDMA FTT-Ethernet, RTL-TEP

Can we do TDMA with commodity Ethernet?

10

TDMA using Pause Frames Flow control packets (pause frames) can be used to control Ethernet transmissions Pause frames are processed in hardware

Very efficient processing of the flow control packets Blast UDP packets

802.3x Pause frames Measure time taken by sender to react to the pause frames 11

TDMA using Pause Frames Pause frames processed in hardware Very efficient processing of the flow control packets

• Reaction time to pause frames is 2 – 6 μs • Low variance

* Measurement done using 802.3x pause frames

12

TDMA using commodity hardware

Collect demand information from the end hosts

TDMA imposed over Ethernet using a centralized fabric manager

Compute the schedule for communication

Control end host transmissions

13

TDMA example

D1 S S –> D1: 1MB

D2

S –> D2: 1MB

S S S S

-> -> -> ->

Compute the schedule for communication Control end host transmissions

round12

Schedule • round1: • round2: • round3: • round4: •…

Collect demand information from the end hosts

D1 D2 D1 D2

Fabric manager

14

More than one host Fabric manager • Control packets should be processed with low variance round12

• Control packets should arrive at the end hosts synchronously

Synchronized arrival of control packets We cannot directly measure the synchronous arrival Difference in arrival of a pair of control packets at 24 hosts

16

Synchronized arrival of control packets Difference in arrival of a pair of control packets at 24 hosts Variation of ~15μs for different sending rates at end hosts

17

Ideal scenario: control packets arrive synchronously round2

round3

Host A

Round 1

Round 2

Round 3

Host B

Round 1

Round 2

Round 3

round2

round3 18

Experiments show that packets do not arrive synchronously round2

Host A

Round 1

Host B

Round 1

Round 2

Round 3

Round 2

Round 3

Out of sync by