add documentation about how to put a string in progmem

[mutantbob]

I have spent the afternoon experimenting with how to put a string in progmem.

Using

#[cfg_attr(target_arch = "avr", link_section = ".progmem.data")]
static MESSAGE3: &str = "apple";

only puts the pointer and count of the slice in progmem. The actual characters still end up in RAM.

It is possible to use something like

#[cfg_attr(target_arch = "avr", link_section = ".progmem.data")]
static MESSAGE6: &[u8; 9] = b"dai kenja";

But if you try

#[cfg_attr(target_arch = "avr", link_section = ".progmem.data")]
static MESSAGE5: &[u8; 13] = b"dai 大賢者 kenja";

you get an error:

error: byte constant must be ASCII. Use a \xHH escape for a non-ASCII byte

[mutantbob]

It is also unclear how to access data from include_bytes!

#[cfg_attr(target_arch = "avr", link_section = ".progmem.data")]
static MESSAGE4: &[u8] = include_bytes!("parts1.txt");

seems like it creates a [u8;N] , because core::mem::size_of_val(MESSAGE4) reports 2031 (which is the size of the file). However, trying to unsafe{ read_value( (&MESSAGE4[0] as *const u8).offset(i) ) } does not return values that match what is in parts1.txt .

[Cryptjar]

Thanks for bringing up these pain points. This is definitely something that should be mentioned in the documentation.

Generally, you should not store any reference in progmem, because otherwise only the reference is in progmem instead of the actual data. Thus, you should rather use arrays directly, e.g.:

#[cfg_attr(target_arch = "avr", link_section = ".progmem.data")]
static MESSAGE6: [u8; 9] = *b"dai kenja";

#[cfg_attr(target_arch = "avr", link_section = ".progmem.data")]
static MESSAGE4: [u8; 2031] = *include_bytes!("parts1.txt");

Non-ASCII byte literals, well, seem not to be supported by Rust, so either use the proposed manual UTF-8 encoding or maybe a char array:

#[cfg_attr(target_arch = "avr", link_section = ".progmem.data")]
static MESSAGE5_1: [u8; 19] = *b"dai \xE5\xA4\xA7\xE8\xB3\xA2\xE8\x80\x85 kenja";

#[cfg_attr(target_arch = "avr", link_section = ".progmem.data")]
static MESSAGE5_2: [char; 13] = ['d', 'a', 'i', ' ', '大', '賢', '者', ' ', 'k', 'e', 'n', 'j', 'a'];

Neither look particularly nice, but as far as I know, there is not much else you can do. Maybe one day something like this could work, but for now this is sadly not const yet:

static MESSAGE5_3: [u8; 19] = "dai 大賢者 kenja".as_bytes().try_into().unwrap();

[mutantbob]

I have vague ideas for a procedural macro that lets you code a string literal and it turns into a wrapper around a progmem byte array. Maybe one day I'll make time to write it if no one beats me to it. Another complication is that ufmt's uWrite is all about &str and doesn't have a byte-by-byte API.

[Cryptjar]

Your suggestion to do something with macros, got me intrigue. Admittedly, I mostly use include_bytes and ASCII. But I guess, it would be quite useful, if one can simply specify a "normal" string literal, let it be stored as a byte array, while being able to use it like &str. So the first thing that come to my mind is wrapper struct around some [u8;N] and implementing Deref on it. Such a wrapper struct should work nicely with the progmem macro, something like this:

progmem! {
    static progmem TEXT: ByteString<19> = ByteString::new(
        "dai 大賢者 kenja"
    );
}
// usage:
let text_buffer = TEXT.load(); // The temporary DRAM buffer for `TEXT`
let text: &str = &text_buffer; // Just derefs to `str`

Update:

One can use that to offer convenient in-place string wrapping, e.g.:

fn print(s: &str) {
    // do something with `s`
}
// Put the literal as byte array into progmem and load it here as `&str`
print(progmem_str!("dai 大賢者 kenja"));

Which would be just syntactic sugar for an ad-hoc progmem static and load.

[mutantbob]

The best of all worlds would be if you can use ufmt to "print" a literal string that would not fit in RAM, but the least awkward way to do that would be to transfer the string byte-by-byte from progmem to the serial port or whatever, and ufmt does not have a byte interface at the moment. Workarounds would include doing 8(?)-byte slices at a time, probably using from_utf8_unchecked.

[Cryptjar]

Hmm, reading strings byte-by-byte is an interesting thought. The ByteString type is supposed to be wrapped in a ProgMem struct (i.e. yielding a static with type ProgMem<ByteString>), thus it must be read all at once, but it can be stored on stack in order to deref it as &str (as whole).

In order to support partially reading of a str, I could imagine a different struct, which itself wraps a ProgMem<[u8;N]> — let's call it: PmString. This way, a PmString could give access to arbitrary parts of the str, however, it can no longer deref to &str (because nothing has been loaded that could be referenced).

Also, due to the variable length encoding of UTF-8, reading fixed blocks of bytes (such as 8 at a time) and then using from_utf8_unchecked on it would be undefined behavior. Instead, it must be read literally byte-by-byte until a valid chunk (1-4 bytes) has been read that can be turned into a str or char. Therefore, I imagine that PmString would offer a char-iterator that lazily reads char-by-char from progmem, which seems sufficient for ufmt usage (in fact PmString could directly implement uDisplay using such an iterator).

However, at this point, I wounder whether this byte-wise reading & parsing is detrimental for performance, and whether, using a char array would be better than a UTF-8 byte-array for such a use-case. On the other hand, a char array could require up to 4x as much progmem.

Do you have further thoughts on this?

Update

Actually, it's much harder to get a char array out of a str at compile-time than a byte array. I think, the UTF-8 parsing must suffice.

[Cryptjar]

The usage style I implemented so far is:

use avr_progmem::progmem;

// Imagine having something implementing `ufmt::uWrite`
let mut writer = /* snip */;

progmem! {
    // declare a string with the additonal `string` pseudo-keyword.
    // This will wrap the given `&str` in a `PmString`
    static progmem string TEXT = "dai 大賢者 kenja";
}

// Either load at once (potential high RAM usage)
let buffer = TEXT.load();
// and use it as `&str`
writer.write_str(&*buffer).unwrap();

// Or load it lazily (one char at a time)
for c in TEXT.chars() {
    writer.write_char(c).unwrap();
}

// Or use the `uDisplay` impl which is also lazy
ufmt::uformat!(&mut writer, "{}", TEXT).unwrap();

Additionally, for single-use strings, there are two macros for in-line strings:

use avr_progmem::progmem_str as F;
use avr_progmem::progmem_display as D;

// Imagine having something implementing `ufmt::uWrite`
let mut writer = /* snip */;

// In-line progmem string accessible as temporary `&str`
// Needs enough RAM to store the entire string
writer.write_str(F!("dai 大賢者 kenja")).unwrap();

// In-line progmem string accessible as `impl Display + uDisplay`
// Needs only a minimal amount of RAM
ufmt::uwrite!(&mut writer, "{}", D!("dai 大賢者 kenja")).unwrap();

Would this cover your use-case?

[mutantbob]

I think so. The funniest part is that I wrote some of this last week (the ability to turn [u8;N] into a character iterator).

You even made your own copy of next_code_point too. Hopefully rust-lang/rust#95788 gets merged one day and you will be able to use that instead.

The final transformation will be when the pieces of the format specification to the uwrite! macro are stored in progmem.
For example

uwriteln!(serial, "would you like your {} with a side of {}?", main, side);

would become

ufmt::uDisplay::fmt(D!("would you like your "), serial);
ufmt::uDisplay::fmt(main, serial);
ufmt::uDisplay::fmt(D!(" with a side of "), serial);
ufmt::uDisplay::fmt(side, serial);
ufmt::uDisplay::fmt(D!("?\n"), serial);

which probably isn't exactly how the macro works, but you get the idea. I suspect that most (if not all) of the work in this crate is done and it's just a question of tweaking ufmt (which, given that japaric/ufmt#32 has been sitting there since January, could take a while)