The CTLE in GTX DFE can be used in two ways: auto adapting and fixed.
The GTX IBIS-AMI model default is set manually, not adapting CTLE.
Especially in a case of low insertion losses, you should check if the default RX settings for AGC are saturating to minimum code quickly, which could suggest that RX input is too large.
The first hint in cases like this is to reduce the TX swing.
In the case of low insertion losses, RX input amplitude is already very large and you do not need extra DC gain.
The default value for CTLE3 in AMI model is 0, which gives the max DC gain.
By setting RX_DFE_KL_CFG2[26:23] to 4b0111, you are setting CTLE3=7 and providing lower DC gain.
Since there are too many combinations of CTLE1/2/3 settings, you would need to edit the receiver AMI file to set CTLE3 in this case.
In real HW, CTLE3 is always controlled manually, there is no auto mode for it. There are two ways to freeze CTLE1/2:
Mid/High Frequency Boost = CTLE1 and CTLE2
Wide-Band Gain = CTLE3
CTLE1 = RX_DFE_KL_CFG2[8:5]
CTLE2 = RX_DFE_KL_CFG2[17:14]
CTLE3 = RX_DFE_KL_CFG2[26:23]
In IBIS-AMI model simulation, the CTLE setup is manual. There is no RX_BIAS_CFG.
There is only one parameter controlling both CTLE1 and CTLE2, that is RX_DFE_KL_CFG2_bit8_to_5.The parameter controlling CTLE3 is RX_DFE_KL_CFG2_bit23_to_26.