I figured out a strange issue with Dell servers. They have Broadcom BCM5709C network cards. I performed large file copy test and result were like this:
Send Pri 90MByte/Sec
Rcv Pri 05 MByte/sec
Send Sec 94MByte/Sec
Rcv Sec 108 MByte/sec
So network speed seems to be pretty good in Secondary card but in Primary card it is extremely slow in one direction. Both the cards are exact same in settings.
I debugged more a found that "TCP connection offload" was causing the issue. After disabling this setting network performance was good.
Now question is why TCP connection offload did not affect Secondary card, where it is still enabled. Also I read that TCP connection offload is actually for improving network performance but it did otherway.
Voted best answer
Maybe a faulty NIC driver?
I've come across issues affecting only one out of many NICs of the same type in the same machine. Updating the driver solved the problem.
Maybe first NIC has some HW issue and the offloading caused problems with transmission / reception?
I'd try both upgrading driver and swapping positions of the two NICs; possibly also trying it in another computer.
Warranty? Try contacting the vendor or store where you bought it from and ask for advice or replacement.