For the encoder, 2.4 Gbps/8 bits = 300 MHz. This can be achieved in a single Vx4 as a V2P-6 gets to 285 MHz. Failing that, it would be straightforward to use 2 encoder cores; they are around 114 slices each.
If 1.2 Gbps is the actual data rate into the RS Decoder, then the core needs to run at 1.2 Gbps/8 bits = 150 MHz. This is possible with one 2-channel core in V2P-6. A 1-channel core can achieve 197 MHz in a V4-12, so it is also possible to achieve 150 MHz in a lower speed grade.
"Shortened blocks" simply means the block is less than 2^symbol_width -1. The core can handle this automatically without the need to pad the block with 0s. If the block is padded with 0s, then the block length would be set to 2^symbol_width -1. The N_IN input is only required if multiple block lengths have to be supported by a single core.
For suggestions on how to increase the speed of your Reed Solomon Decoder, see (Xilinx Answer 21787).