API: pandas.api

[jreback]

In xref #13147

established a bit of a public api in pandas.api; ATM this only contains the type introspection routines.

1st question

Some disagreement on whether we should not do this, and rather just expose pandas.types directly.

I think pandas.api is actually very important because:

1. limits scope of what we choose to expose in the future; its not simply 'everything' that isnt nailed down (the case now). This does change pandas to make everything by default private, EXCEPT what is explicitly public.

2. a single entry point for other packages to use the pandas public API that is known / stable / maintained (will with c-API as well)

3. provide consistent naming externally (and allow us to fix / hide internally as needed)

4. namespaced. I only import what I need as a user / other package, rather than everything

2nd question

as discussed here, should these deprecated API functions should be DeprecationWrarning rather than FutureWarning?
-> done in #13990

Ideally we should resolve this before the end of 0.19.0 (or remove it).

[jreback]

@jorisvandenbossche @shoyer @wesm @TomAugspurger @sinhrks

[jreback]

@jorisvandenbossche updated the top section.

[jorisvandenbossche]

We have discussed this on some several places, recently also in the PR about the errors module (#15541) and previously when pandas.types was created (#13147 (comment) and comments below).
For the exceptions, we now have a top-level pandas.errors module, for type-related we have a private pandas.types and public pandas.api.types.

With the risk of keeping a settled discussion alive, I would still like to see this discussed some more.

To repeat the comment of @shoyer:

Ultimately, I don't think this matters too much, but given a choice between:

1. pandas.* indicates private namespace, pandas.api.* includes a public namespace, and
2. pandas.* includes public namespace, pandas._internals (and so forth) indicates private namespace

I think the later (option 2) is more user friendly and consistent with the top level namespace pandas already being public facing.

On the other hand, it's true that option 2 is a little harder to transition to (for us), so maybe it's not worth the trouble.

We have recently added new submodules to the top-level namespace (pandas.types, pandas.indexes, pandas.formats, code that was before mainly in pandas.core. ..), but towards the user these should be regarded as private. IMO this is not the good direction. For new code / refactors, I think we should stick to: only what is public can go in the top-level namespace.

Let's take the example of types. We now have a private pandas.types and a public pandas.api.types, so the less nested location is private, which I don't find ideal.
If we use the approach of pandas.api.types, I think we should at least make the actual implementation in pandas._types, so you don't have the two publicly visible locations.
The other option is to make put now what is in pandas.api.types, in pandas.types (while the exact implementation and organization in the different submodules of pandas.types remains a private implementation detail).

[jreback]

Here's my rationale:

We have 3 stakeholders.

• users: regular folks who do import pandas as pd
• library builders / advanced users: folks who need more 'library' code
• implementation

implementation: this may seem strange to mention, but pandas is a large library. We need the flexibility to move code around, expand modules, rename and generally get stuff done internally, with out having to constantly deprecate things. This would be extremely disruptive to the public (not to mention burdensome).

For example. pandas.types exists with several sub-modules that are logically defined, simply named and concise. There is not reason a for 'users' or 'library' writers to access this at all. It exists simply to organize the code. Sure they can reach in if they really want, but we want to discourage this. Making it pandas._types would surely do this, but that is an ugly name and since this is pure python code a bit misleading.

'users': they are served by the top-level pd namespace, supplemented by pandas.errors. Everything that one could need is here (or as a method).

'library': these folks need code generally for instrospection that is just too cumbersome to be in a 'user' namespace. Things like is_integer_dtype is a canonicial way to introspect things in pandas. We use these internally and provide them as an external API to library writers.

I recently added pandas.api.lib with infer_dtype, which is a useful routine, again to external library writers. This is (or another sub-module of pandas.api) can also be the home of a c-api / development api.

So, everything as public is just fine. If someone wants to reach in and use a routine from pandas.format.* or whatever they should have no expectations that this is stable across releases.

For simplicity from a user AND a library pov, its much better to have a single namespace pandas.api that say: hey they routines are what I should use, they are documented and won't change

[shoyer]

I don't see a strong division between "users" and "advanced users". There is a continuum of use cases. Certainly we should group more things into submodules: this makes it easier to find related functionality.

One choice that would make this super clear is move all internal stuff into a top level submodule called internals, e.g., pandas.internals.core, pandas.internals.indexes, etc. Then if you're importing from pandas.internals.something, you know it's an internal routine. Otherwise, as a user it's not obvious whether pandas.something exists to as a namespace to logically group together the something routines or if it's an internal implementation detail.

[jorisvandenbossche]

For example. pandas.types exists with several sub-modules that are logically defined, simply named and concise. There is not reason a for 'users' or 'library' writers to access this at all. It exists simply to organize the code.

That is not fully what I meant. The pandas.types module could still have different sub-modules, but what I meant was that only the 'top-level' of pandas.types would be public, so we are still free to implement it as we like in the submodules.
And sure, we will change things in the future, but whether we expose the public names in pandas.api.types or in pandas.types (top-level, not submodules) does not really matter for that? In both cases the functions will not actually be defined in that location.

Making it pandas._types would surely do this, but that is an ugly name and since this is pure python code a bit misleading.

This is IMO not misleading at all, it is actually very explicit that it is internal.
And I agree it is a bit uglier, but is only about a few import lines in each pandas file. I don't think it is that problematic there is a bit ugliness there, and IMO worth the clarity (if we use the pandas.api approach).
The internals namespace is also a possiblity, but this is actually more or less what core is currently. So we could use it for that I think (although less explicit in name). But we moved types, formats, indexes out of core .., so this would be putting it back (but keeping the reorganization in submodules).

'users': they are served by the top-level pd namespace, supplemented by pandas.errors. Everything that one could need is here (or as a method).

This is already not fully the case. There are public methods in pandas.io.json, pandas.tseries.offsets, pandas.plotting (when that PR is merged), ..

I recently added pandas.api.lib with infer_dtype,

Slightly of topic, but I personally would rather put this in pandas.types, as it is type related. And the fact that it is in our lib is more an implemenation detail for the user.