Skip to content

FPGA: Reduce the decompress test files sizes when running simulation to make the runtime faster #1351

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 13, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ Snappy is compression format that aims for high throughput compression and decom

Unlike many compression encodings, like DEFLATE, Snappy encoding is byte oriented. The Snappy format does not use entropy encoding, such as Huffman or range encoding.

Snappy encoding is like LZ77 encoding, which replaces portions of the byte stream with {length, distance} pairs. For more information on the Snappy format, see the [Snappy](https://en.wikipedia.org/wiki/Snappy_(compression)) Wikipedia article and the [Google Snappy](https://github.com/google/snappy) GitHub repository.
Snappy encoding is like LZ77 encoding, which replaces portions of the byte stream with {length, distance} pairs. For more information on the Snappy format, see the [Snappy](https://en.wikipedia.org/wiki/Snappy_(compression)) Wikipedia article and the [Google Snappy](https://github.com/google/snappy) GitHub repository. The files being decompressed in this tutorial have been compressed using [python-snappy](https://github.com/andrix/python-snappy) which is a python library for the snappy compression library from Google.

The basic Snappy format is a *preamble* followed by the *compressed data stream*.

Expand Down
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@




ALICE'S ADVENTURES IN WONDERLAND

Lewis Carroll

THE MILLENNIUM FULCRUM EDITION 2.9




CHAPTER I

Down the Rabbit-Hole


Alice was beginning to get very tired of sitting by her sister
on the bank, and of having nothing to do: once or twice she had
peeped into the book her sister was reading, but it had no
pictures or conversations in it, `and what is the use of a book,'
thought Alice `without pictures or conversation?'
Binary file not shown.
Original file line number Diff line number Diff line change
Expand Up @@ -184,13 +184,26 @@ void PrintTestResults(std::string test_name, bool passed) {
#if defined(GZIP)
bool RunGzipTest(sycl::queue& q, GzipDecompressorT decompressor,
const std::string test_dir) {


#ifdef FPGA_SIMULATOR
// the name of the file for the simulator is fixed
std::string small_filename = test_dir + "/small.gz";

std::cout << ">>>>> Small File Test <<<<<" << std::endl;
bool small_test_pass = decompressor.DecompressFile(
q, small_filename, "", 1, false, false);
PrintTestResults("Small File Test", small_test_pass);
std::cout << std::endl;

return small_test_pass;
#else
// the name of the files for the default test are fixed
std::string uncompressed_filename = test_dir + "/uncompressed.gz";
std::string static_compress_filename = test_dir + "/static_compressed.gz";
std::string dynamic_compress_filename = test_dir + "/dynamic_compressed.gz";
std::string tp_test_filename = test_dir + "/tp_test.gz";

#ifndef FPGA_SIMULATOR
std::cout << ">>>>> Uncompressed File Test <<<<<" << std::endl;
bool uncompressed_test_pass = decompressor.DecompressFile(
q, uncompressed_filename, "", 1, false, false);
Expand All @@ -202,53 +215,61 @@ bool RunGzipTest(sycl::queue& q, GzipDecompressorT decompressor,
q, static_compress_filename, "", 1, false, false);
PrintTestResults("Statically Compressed File Test", static_test_pass);
std::cout << std::endl;
#else
std::cout << "Only running the Dynamically Compressed File Test when using "
"the simulator flow to reduce execution time." << std::endl;
bool uncompressed_test_pass = true;
bool static_test_pass = true;
#endif

std::cout << ">>>>> Dynamically Compressed File Test <<<<<" << std::endl;
bool dynamic_test_pass = decompressor.DecompressFile(
q, dynamic_compress_filename, "", 1, false, false);
PrintTestResults("Dynamically Compressed File Test", dynamic_test_pass);
std::cout << std::endl;


#ifndef FPGA_SIMULATOR
std::cout << ">>>>> Throughput Test <<<<<" << std::endl;
constexpr int kTPTestRuns = 5;
bool tp_test_pass = decompressor.DecompressFile(q, tp_test_filename, "",
kTPTestRuns, true, false);
PrintTestResults("Throughput Test", tp_test_pass);
std::cout << std::endl;
#else
bool tp_test_pass = true;
#endif

return uncompressed_test_pass && static_test_pass && dynamic_test_pass &&
tp_test_pass;
tp_test_pass;
#endif

}
#endif

#if defined(SNAPPY)
bool RunSnappyTest(sycl::queue& q, SnappyDecompressorT decompressor,
const std::string test_dir) {


#ifdef FPGA_SIMULATOR
std::cout << ">>>>> Small Alice In Wonderland Test <<<<<" << std::endl;
std::string alice_in_file = test_dir + "/alice29_small.txt.sz";
auto in_bytes = ReadInputFile(alice_in_file);
auto result = decompressor.DecompressBytes(q, in_bytes, 1, false);

std::string alice_ref_file = test_dir + "/alice29_small.ref.txt";
auto ref_bytes = ReadInputFile(alice_ref_file);
bool alice_test_pass =
(result != std::nullopt) && (result.value() == ref_bytes);

PrintTestResults("Small Alice In Wonderland Test", alice_test_pass);
std::cout << std::endl;
return alice_test_pass;

#else
std::cout << ">>>>> Alice In Wonderland Test <<<<<" << std::endl;
std::string alice_in_file = test_dir + "/alice29.txt.sz";
std::string alice_in_file = test_dir + "/alice29_small.txt.sz";
auto in_bytes = ReadInputFile(alice_in_file);
auto result = decompressor.DecompressBytes(q, in_bytes, 1, false);

std::string alice_ref_file = test_dir + "/alice29.ref.txt";
std::string alice_ref_file = test_dir + "/alice29_small.ref.txt";
auto ref_bytes = ReadInputFile(alice_ref_file);
bool alice_test_pass =
(result != std::nullopt) && (result.value() == ref_bytes);

PrintTestResults("Alice In Wonderland Test", alice_test_pass);
std::cout << std::endl;

#ifndef FPGA_SIMULATOR
std::cout << ">>>>> Only Literal Strings Test <<<<<" << std::endl;
auto test1_bytes = GenerateSnappyCompressedData(333, 3, 0, 0, 3);
auto test1_ret = decompressor.DecompressBytes(q, test1_bytes, 1, false);
Expand Down Expand Up @@ -285,8 +306,6 @@ bool RunSnappyTest(sycl::queue& q, SnappyDecompressorT decompressor,

return alice_test_pass && test1_pass && test2_pass && test3_pass &&
test_tp_pass;
#else
return alice_test_pass;
#endif

}
Expand Down