[X86][MC] Report error when the instruction length exceeds 15 bytes (#83708)

The instruction-size limit of 15 bytes still applies to APX
instructions.

Note that it is possible for an EVEX-encoded legacy instruction to reach
the 15-byte instruction length limit: 4
bytes of EVEX prefix + 1 byte of opcode + 1 byte of ModRM + 1 byte of
SIB + 4 bytes of displacement + 4 bytes of immediate = 15 bytes in
total, e.g.

```
addq    $184, -96, %rax   # encoding:
[0x62,0xf4,0xfc,0x18,0x81,0x04,0x25,0xa0,0xff,0xff,0xff,0xb8,0x00,0x00,0x00]
```

If we added a segment prefix like fs, the length would be 16.

In such a case, no additional (ASIZE or segment override) prefix can be
used.

To help users find this issue earlier, especially for assembler users,
we change
the internal compiler error to error in this patch.

Diagnostic is aligned with GAS
https://sourceware.org/bugzilla/show_bug.cgi?id=31323

Author: KanRobert

llvm/lib/Target/X86/MCTargetDesc/X86MCCodeEmitter.cpp

@@ -1924,8 +1924,8 @@ void X86MCCodeEmitter::encodeInstruction(const MCInst &MI,
   if ((TSFlags & X86II::OpMapMask) == X86II::ThreeDNow)
     emitByte(X86II::getBaseOpcodeFor(TSFlags), CB);
 
-  assert(CB.size() - StartByte <= 15 &&
-         "The size of instruction must be no longer than 15.");
+  if (CB.size() - StartByte > 15)
+    Ctx.reportError(MI.getLoc(), "instruction length exceeds the limit of 15");
 #ifndef NDEBUG
   // FIXME: Verify.
   if (/*!Desc.isVariadic() &&*/ CurOp != NumOps) {

llvm/test/MC/X86/apx/long-instruction-err.s

@@ -0,0 +1,25 @@
+# RUN: not llvm-mc -triple x86_64 -show-encoding %s 2>&1 | FileCheck %s
+
+# CHECK: [[#@LINE+2]]:1: error: instruction length exceeds the limit of 15
+# CHECK: addq    $1234, %cs:-96, %rax
+addq    $1234, %cs:-96, %rax
+
+# CHECK: [[#@LINE+2]]:1: error: instruction length exceeds the limit of 15
+# CHECK: subq    $1234, %fs:257(%rbx, %rcx), %rax
+subq    $1234, %fs:257(%rbx, %rcx), %rax
+
+# CHECK: [[#@LINE+2]]:1: error: instruction length exceeds the limit of 15
+# CHECK: orq     $1234, 257(%ebx, %ecx), %rax
+orq     $1234, 257(%ebx, %ecx), %rax
+
+# CHECK: [[#@LINE+2]]:1: error: instruction length exceeds the limit of 15
+# CHECK: xorq    $1234, %gs:257(%ebx), %rax
+xorq    $1234, %gs:257(%ebx), %rax
+
+# CHECK: [[#@LINE+2]]:1: error: instruction length exceeds the limit of 15
+# CHECK: {nf} andq    $1234, %cs:-96
+{nf} andq    $1234, %cs:-96
+
+# CHECK: [[#@LINE+2]]:1: error: instruction length exceeds the limit of 15
+# CHECK: {evex} adcq    $1234, %cs:-96
+{evex} adcq    $1234, %cs:-96