Sm100 blockwise fp8 swap ab

[IwakuraRein]

Add the swap and transpose tensor A/B path to sm100 block-wise fp8 GEMM. This reduces the overhead of extra paddings.

Also add a basic heuristic.

Note: looks like 18778 is duplicate with part of this PR.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[mgoin]

Thanks @IwakuraRein, #18778 should be separate as it only touches scaled_mm_sm100_fp8_dispatch.cuh, not the blockwise fp8 path. Would appreciate help tuning for that file though, the cutlass profiler is giving me problems going to larger values.

[mgoin]

The heuristics are fairly difficult to read now due to separating the tile scalars from the MmaTileShape, but overall these changes seem reasonable. Could you share an accuracy eval as a smoke test since we don't have sm100 in CI?

[IwakuraRein]

@mgoin Hi. I have run the tests on following problem shapes and compared to the triton kernel triton_fp8_fp8_fp16_scaled_mm_blockwise. The results are all correct:

M	N	K
1-15, 64, 128, 172, 256, 384, 448	2112	7168
1-15, 64, 128, 172, 256, 384, 448	24576	1536
1-15, 64, 128, 172, 256, 384, 448	32768	512
1-15, 64, 128, 172, 256, 384, 448	7168	16384
1-15, 64, 128, 172, 256, 384, 448	4096	7168
1-15, 64, 128, 172, 256, 384, 448	7168	2048