pathlib.Path type

[clbarnes]

pathlib is great for interacting with paths on the local file system. It's expressive, terse, ergonomic, and typed. I'd love to use fsspec going forward, but don't want to give up the power and convenience of pathlib.

Could AbstractFileSystems contain a factory method which produced a PurePath subclass with the correct separator, prefix etc.? This concrete class could also contain the methods implemented by pathlib.Path or it could just be passed back to the filesystem.

[martindurant]

pathlib is great

(maybe my bias against pathlib keeps showing through)

Could AbstractFileSystems contain

I am not opposed in principle, but I don't immediately see how this would work. It sounds like a pain to implement?

[clbarnes]

Agreed that it may be a pain to implement all of the FS-touching operations present in Path. The instance would need to keep around a reference to the creating FileSystem which wouldn't be very tidy, and a lot of the methods wouldn't abstract well over other FSs.

However, the API of PurePath (which basically handles splitting on separators, relative paths and so on) is not too bad and IMO would add some ergonomics and type helpfulness. AbstractFileSystems could have a PurePath inner class which inherits from a fsspec.PurePath and basically just allows implementors to set their separators, disallowed characters and so on. You'd then instantiate it with my_fs.PurePath(), could do type annotations with MyFileSystem.PurePath. Downstream users could thus abstract further over fsspec implementations because they don't need to know about the FS' separators and so on to split out path components and file extensions and so on - they just use the PurePath API as they're used to doing on their own filesystem.

[martindurant]

Is this something you are proposing to take on? :)

[clbarnes]

If anyone else thinks it would be valuable and it would make it through review, I may be able to find time!

[martindurant]

If anyone else thinks it would be valuable

Let's see if other people chime in. pathlib has certainly been mentioned before in the issues.

[michcio1234]

Hello! At Data Revenue we have crafted a drfs package that does more or less what you described.

Example:

from drfs import DRPath
dir_path = DRPath("s3://bucket/key")
file_path = dir_path / "file.txt"
with file_path.open("rb") as f:
    f.read()

Another example (presenting another functionality of the package) is here.

It is an internal package and was opensourced just yesterday (actually because of this thread), so I realise that the documentation is not perfect and there are bugs here and there. The package will be probably refactored in the near future (possibly as you described - by integrating with fsspec), but maybe it can already be of use for you. :)

[raybellwaves]

I stumbled across this yesterday https://github.com/drivendataorg/cloudpathlib. Could be used as inspiration here

[martindurant]

Hurray for the extra functionality, @michcio1234 ! We would gladly see something like this included directly in fsspec, or indeed as a separate package.

@raybellwaves , I will post an issue on that repo to see if they are interested in contributing - or as inspiration, as you say.

[martindurant]

@michcio1234 , also I'd note that your second example starts to look something like an intake catalog spec

[hadim]

I would also love to have a pathlib.Path like-object for remote paths directly in fsspec. I would prefer to not rely on yet-another-lib mentioned here for this. And pathlib.PurePath does not work because it strips out the first // after the name of the protocol:

import pathlib
my_path = pathlib.Path("s3://my-bucket/experiment-1") / "results.csv"
my_path

returns:

PosixPath('s3:/my-bucket/experiment-1/results.csv')

When used with fsspec, the filesystem selected is the local one:

import fsspec
with fsspec.open(my_path, "w") as f:
    f.write("my results")

Locally a folder s3: is created.

[martindurant]

Locally a folder s3: is created.

Ooops...

Again, I'd be really happy for someone to contribute this, but it likely won't be me.

[jamesmyatt]

This is something I've been thinking about and wanting too. The particular advantage is that it then allows you to use fsspec with (almost) any function that expects a pathlib.Path object.

I think that it could be quite a simple class that acts as an adapter between the pathlib.Path API and the fsspec.AbstractFileSystem API.

[hadim]

The particular advantage is that it then allows you to use fsspec with (almost) any function that expects a pathlib.Path object.

Agree. I use pathlib everywhere but recently had to revert my code back because I also wanted to support fsspec paths...

[mraspaud]

One more vote here, we have just started looking into this. One very simple way we tried is just to implement the __fspath__ method, that goes already a long way.