The output port names in the test differ from the prompt and reference code requirements in task 99 (m2014_q6c)

[y-kimmy]

verilog-eval/dataset_spec-to-rtl/Prob099_m2014_q6c_test.sv

Lines 71 to 81 in c498220

	RefModule good1 (
	.y,
	.w,
	.Y2(Y2_ref),
	.Y4(Y4_ref) );
	TopModule top_module1 (
	.y,
	.w,
	.Y2(Y2_dut),
	.Y4(Y4_dut) );

The output port names are Y2 and Y4, not Y1 and Y3, so even if the solution code is provided, it cannot be compiled due to a compile error. Please check.

[thomasnormal]

Yup. Seems it should be Y1 and Y3

[stefanpie]

Same issue here. Looks like it's also addressed on an open pull request #16.

Can any of the owners (@npfet) please accept the pull request and close the issue. Solving these issues is critical for reproducibly and accurate / representative research results.

[npfet]

Hi all — apologies for the radio silence.

Regarding the pull request and recent bug fixes to the dataset: there’s a tradeoff between improving the dataset (which can increase the maximum achievable score) and maintaining consistency for already-published comparisons. We want to avoid making continual silent fixes that would retroactively penalize earlier results.

To address this, I’ll mark the current code as the v2.0 release of VerilogEval. The pull request will be incorporated into a v2.1 release, alongside a published errata. This way, future authors can explicitly reference v2.0 or explain any differences in results with respect to v2.1.

On a related note, we’re hoping to share some exciting benchmark updates later this month—one reason things have been quiet on the VerilogEval front. If you’ll be at ICLAD, feel free to reach out and connect!

[thomasahle]

Versioning makes a lot of sense!

I'm hoping your news are about a new dataset! We're getting 95%+ on verilog eval with a simple agentic loop: https://x.com/thomasahle/status/1924802446745890916
thinking about what the next challenge should be!

Somebody from our team might be at ICLAD, would be fun to meet up!

[stefanpie]

I definitely second the versioning approach as well! Mainly to merge in this pull request, #15, and #9.

I will also be presenting at ICLAD; excited to see new results!