iSCSI: Disk at 100% after some traffic

Question

We have a disk exported via iSCSI. When writing to this particular disk, following issue manifests:

Initial speed is decent, then after a few gigabytes of writes

i) disk becomes 100% busy

ii) disk write speed drops to 10 MB/s

iii) Disk IOPS drop to 80 transfers / sec

Some (alarming) points to consider:

1) All other disks in our network function properly, that is their speeds when exported as iSCSI targets are normal and sustained.

2) This specific disk functions properly when used locally, and not as iSCSI target.

3) Disk's behaviour was duplicated in a virtual environment. That is, iSCSI target and iSCSI initiator are two virtual machines in the same bare metal machine.

If it wasn't for point 2) we would come to the conclusion that disk is faulty. But current behaviour (problems only for 1 specific disk, only for 1 specific environment) cannot be explained.

Any thoughts?

edit:

Screnshot attached.

Top left is iSCSI target (hostname "san"). Exported disk is xvdb.

Top right is iSCSI initiator (hostname "sandbox"). Mounted disk is sdb.

Bottom right is a transfer operation (as rsync) in initiator system, from a local disk to aforementioned problematic drive.

Note: Initial speed of ~ 25 MB/s was justifiable due to initiator's CPU bottleneck. But depicted utilization differs from CPU utilization in proper functioning ( 30% user, 70% sys, 0% wait, 0% idle).

Respective screenshot

Answer 1

I experienced the same issue a while ago. After doing some tests we got to the point that the disk wasn't working well anymore. In your case one thing to consider is point #2 and based on that I'd think you might have a network path problem and you should try to check that connection (cable, MTU, switch port capacity) and see if something changes. iSCSI is a pretty good protocol but needs reliable connections.