Design pvlib.io

[cwhanse]

I'm assembling code to create pvlib.io.

What functionality is desired? Specifically, from which formats do we want to read, and to which formats do we want to write?

I'm going to focus first on reading a few data formats. I plan to incorporate the current pvlib.tmy which reads tmy2 and tmy3 files, returning a tuple (data, metadata) where data is a pandas DataFrame and metadata is a dict. Any comments on this design? Personally I like it.

I have code to read surfrad files so I'll include that.

[adriesse]

I often put metadata in a pandas Series rather than dict because this
gives me order. I also like the dot-style access, e.g. "meta.latitude".

[wholmgren]

As @bmu once mentioned, naming our new module io is probably asking for trouble because it conflicts with the builtin python io module. I'm open to dataio, datareader, inputoutput, or other suggestions.

So far, we've only discussed creating a new module. @mikofski already complained in #235 that our modules are too long, and adding a bunch of new code to a unified io module might make it worse. I wonder if we might be better off in the long run if we instead create a subpackage comprised of one module per data type and pull the functions into the package-level namespace. This is similar to the way that the pandas io package is structured. So, we'd have something like

pvlib/
  dataio/
    __init__.py
    surfrad.py
    tmy.py
    ...

### pvlib/dataio/__init__.py ###
from pvlib.dataio.surfrad import read_surfrad
from pvlib.dataio.tmy import read_tmy2, read_tmy3

### usage ###
import pvlib
data, metadata = pvlib.dataio.read_surfrad
data2, metadata2 = pvlib.dataio.read_tmy2

This subpackage structure might make for an easy to use API and a set of easy to read/maintain modules.

Probably a good idea to stick with the (data, metadata) pattern for new reader functions.

I cannot think of a downside to returning metadata as a Series, but my instinct is to leave it as a dict (possibly an OrderedDict) and let users convert it to a Series if they want to. I seem to remember that the pandas developers recommend against using Series as a dictionary, though I can't find the reference and could be making that up. All that being said, I'm not opposed to the change.

[cwhanse]

It’s a plan, although I prefer iotools to dataio.

Function pattern to be the same as pandas, i.e., read_(format) and to_(format) as needed.

[adriesse]

I defer to people with more experience on the code organization, but I do suspect there will be a few potentially shared elements. What I could imagine--and would really like to have--is a kind of enhanced read_csv that can read extra information in the header and/or footer in addition to the regular columnar data. Many file formats could build on that just by setting parameters and translating column or key names. An option could be to return the header and/or footer as a text block to be parsed later.

[dacoex]

@adriesse do you have an example file & code?

How would you implement with the file irradiation-0e2a19f2-abe7-11e5-a880-5254002dbd9b.csv

In most cases, the input will requrie an adaption of the dateparser to the specific data format. This usually gets you going.
Or would the scope in addition include quality checks or reformatting on the data?
I am still not clear what the output is desired to be.

1. A dataframe as read in?
2. A trying to comply closest with Variables and Symbols?

In the latter case, what do we with additional variables e.g. in spectral datasets or in BSRN files, for example?

[mikofski]

Hi all, sorry I haven't chimed in. I already have tmy2 and tmy3 readers. I'll send them when I get in.

AFA the dataio module and namespace question.

1. I also like dataio.
2. Will, I like your idea of sub packages, I think it's the easiest way to decrease the omnipotence of the god modules. Just change irradiance.py to a folder called irradiance and put an __init__.py in it. Then import the symbols that were originally imported directly from irradiance.py but are now in modules like irradiance/spa.py
3. Anton, clashes in names should be rare in Python because it uses namespaces. EG pvlib.data.io can't be confused with the built-in io because the long name that precedes it.

[wholmgren]

Mark, I am confused about your tmy readers. Do they do the same thing as pvlib's existing tmy readers?

Let's take things one step at a time and have Cliff just make us a dataio/iotools subpackage for the IO capabilities. I have been thinking about larger reorganizations, though, will eventually comment more in #235.

If necessary, the subpackage could have a core.py or its own tools.py module for code that is reused throughout the package.

I want to keep the bar for contributing to pvlib as low as possible, so it's fine with me if different modules do different amounts of processing to their respective data formats. Some might be 1000 line monsters that do QA and return data that is in "pvlib conventions," while others might not do much more than call pd.read_csv with a couple of arguments that are unique to that data format.

[mikofski]

RE: TMY

@wholmgren , sorry I misunderstood the intention of the proposed dataio, and also I wasn't aware of the readtmy2 and readtmy3 methods in tmy.py. So please disregard (2) in my comment above.

RE: reorganization

I agree with you that the bar for contributions should be low. Your ideas for core.py and tools.py sound good, although if I understand correctly (?) that approach may break some code if it shift things from say pvlib.irradiance to pvlib.core.irradiance so that's something to consider. IMHO changing the top level modules to subpackages would preserve the current API, but allow more flexibility. See my comment on #235

RE: dataio

Cliff, some ideas for formats that could be read are:

• SAM module and inverter Excel spreadsheets - I think this is already covered by pvsystem.retrieve_sam method, so similar to TMY methods, would you relocate them to dataio?
• NSRDB API for weather data
• If I can manage to spend a little time to upload the rest of the module and inverter data to pvfree it would be nice to supplement retrieve_sam.
• PAN and ONB file reader, although I don't know if there are any methods in pvlib currently that would utilize these, I know a lot of people would appreciate being able to load these into Python for use with custom tools.
• Other data sources of irradiance, atmospheric and weather measurements would be very useful. These come from numerous sources, so maybe they could be contributed by users as needed. Here's a short list of some examples:
  • AERONET
  • MERRA
  • ECMWF
  • Solar Prospector
  • Clean Power Research
  • 3-Tier Vaisala
  • Solar GIS
  • Meteonorm/Meteotest
  • SoDa-Pro

[dacoex]

@cwhanse / @mikofski many items of the last comments are already in #29 are they complementary? shall the other be closed?

I wonder if there are any observations regarding the comments in #261 (comment)

Otherwise, who will provide a prototype or will have a PR each one with the data set reader they have implemented?