DOC: Discussion on new getting started page (Numfocus Grant)

[stijnvanhoey]

As currently prepared together with @jorisvandenbossche, the following document is a proposal on how to adjust the getting started page and section of the documentation: https://docs.google.com/document/d/1Rc_eql5KLrdf0c582KyWfs2ADVNxbJy4jfosnqdrVak/edit?usp=sharing

The general idea is to split the current 10 minutes into 10 x 1' topics of Pandas. Input is very welcome on:

• the topics to pick and how to introduce them
• the content of the introduction tutorial linked to each of the topics

[jorisvandenbossche]

cc @pandas-dev/pandas-core

[datapythonista]

I don't like the idea much. I'd prefer to have few conventional end to end tutorials (solving a real world problem), targeting different use cases and users types.

If I' had never heard about pandas, I think an "introductory tutorial" on how to open data (covering many options and formats) is not what I'm looking for. And it's not giving me relevant information on whether pandas is the tool I need or not. I think the user guide already addresses that quite well, and if we want tl;dr versions of the user guide pages (may be a good idea), I think that could be added at the beginning of each of them.

What I'd personally appreciate is proper tutorials, for example one could be "Forecasting the stock market", where it's shown:

• How to open data (common formats for finance, I guess csv mainly)
• Setting a datetime index
• Reindexing
• Imputing missing values (ffill and the common methods for time series/stock data)
• Windows functions
• Plotting
• ...

Of course that would only be useful for part of users, but I guess with 3 or 4 tutorials that cover the main pandas use cases, or introductory topics we can find. Finding them is surely not easy, but things that come to my mind:

• Data exploratory (stats, groupby, visualization,...)
• Data engineering (cleaning data, reshaping, joining,...)
• Time series (the above mentioned)

I'd also add a more theoretical one, to teach the concepts (dtypes, data structures,...)

[stijnvanhoey]

Good point, but I do not completely agree. I do not think that, to make the documentation more welcoming to absolute beginners, advanced case studies ('proper tutorials') will help as the complexity will easily overwhelm new users.

What we want to cover with the new getting started page is to provide different elements in line with http://third-bit.com/2019/04/16/what-docs-when.html:

• An overview by providing the 10 advertisement statements to introduce some key Pandas concepts. Purpose: makes people aware something exists
• For each of the topics a short tutorial is the way to help people build a mental model of Pandas elements and acquire a few basic skills and redirect them to sections in the user guide.
• Keeping the link to other languages in there as well provides the translation for users

Note: case studies are mentioned to help users from a competent level to expert level.

Still, your point:

if we want tl;dr versions of the user guide pages (may be a good idea), I think that could be added at the beginning of each of them.

is true. Having tutorials in the getting started section on topics of the user guide is making the documentation more scattered. But this is not the purpose. To be clear, we are actually just talking about a new version of the current 10 minutes and nothing more elaborate. So, the aim of the proposed introduction tutorials is not to cover many options and formats, but very short introductions applied on small data sets (e.g. titanic) and real questions (e.g. I 'm interested in the titanic passengers older than 18 years). When the document says 'content to cover', this is NOT to explain these functions in detail, but to create a short storyline using these functions to answer the question. By doing so, we align with the how to section described in http://third-bit.com/2019/04/16/what-docs-when.html

[jorisvandenbossche]

Thanks for the feedback!

And it's not giving me relevant information on whether pandas is the tool I need or not. I think the user guide already addresses that quite well,

I personally think the user guide is failing hugely on that aspect. IMO no beginner should read one of our pages of the user guide; they are way too overwhelming and unstructured. But that is maybe more saying something about those pages needing some love ... :)

I always had the long term idea to split all the user guide pages in multiple subpages, with one introductory page per topic, and then several pages going in more detail in one of the subtopics (eg Indexing page: one introduction page explaining the different concepts, and then pages with more detail on advanced indexing, multi-indexing, ..).
In such a setting, it is true that the introduction tutorials of the above proposal could also fit as those introductory pages of the different user guide sections. And also then we could still keep the concept of the getting started overview but directly linking to the user guide introductory pages instead to introduction tutorials of each page.

That's certainly an option to consider (and the effort Stijn is putting in those pages could be easily redirected to user guide intro's). We don't necessarily need to restrict the work of this grant strictly to the getting started section.

---

Now, then to the question: what does a beginner need? (which is of course difficult to answer, as it is subjective and you have many different "kinds" of beginners). I personally think such a full "case study" can be easily overwhelming for newcomers, and that it can be useful to have smaller, more focused tutorials.
That is not to say that I wouldn't find such case studies useful to have in our docs as well, but I see them more for people that are already have some basic knowledge.

Terminology is always difficult :-) I would call your "proper tutorial" rather a case study or showcase, but yeah, the definitions of and boundaries between tutorials, how-to's, case studies, user guides, ... is not always clear. We used the "A tutorial is a planned lesson that helps people build a mental model of a domain and acquire a few basic skills" of Greg Wilson when speaking about tutorials here.

[datapythonista]

I think I'm misunderstanding your document. What you say about covering very short introductions applied on small data sets (e.g. titanic) and real questions (e.g. I 'm interested in the titanic passengers older than 18 years) is exactly what I'm proposing. Except that I wouldn't call them very short, I think they'll have a decent size unless you really want to answer a single question.

[datapythonista]

@jorisvandenbossche I agree on what you say (splitting the user guides...). You misunderstood (I phrased poorly) the text you quote. What I think the user guide addresses correctly is providing information about a specific topic (a "tutorial" about IO), which is what I understand from the google doc. Not whether pandas is the right tool for a new user. That's exactly the opposite of what I was trying to say.

I think we need to discuss this more in practice. Write some drafts, and try to get people not familiar with pandas involved. And get their feedback as we progress.

[jorisvandenbossche]

What you say about covering very short introductions applied on small data sets (e.g. titanic) and real questions (e.g. I 'm interested in the titanic passengers older than 18 years) is exactly what I'm proposing.

@datapythonista OK, then it would be good if you (and also others of course) could look at the specific objectives/content to cover of the different tutorials proposed in the google doc. That would be very useful.
It is always a difficult exercise to pick those items out of the full pandas API that you want to show and teach in a beginner tutorial. This also depends on the domain you are coming from, your own pandas coding habits that you have grown, etc. For example, out of your example tutorial content above (#26831 (comment)), I would not include reindexing or windowing functions in a beginner tutorial.

[jorisvandenbossche]

I would not include reindexing or windowing functions in a beginner tutorial.

Although about the windowing functions, those are maybe quite common in the financial world I suppose? A reason to include after all, those kind of things would be good to get feedback on.

[jreback]

why are we trying to recreate a branch new tutorial, rather than simply copying from all of the tutorials out there & consolidate that?

[datapythonista]

My view is that the whole point of the tutorials is to let people know about what kind of things they can do with pandas. I don't expect newcomers to pandas to be experts in reindexing, window functions or whatever after reading a tutorial. But if they are not in the tutorial, how do we expect users to know they exist and use them? I'd say people go to the user guide or the API when they know what they are looking for, I'd expect the tutorial to give a quick overview of most features.

[stijnvanhoey]

@jreback that is what we are doing, e.g. the links provided by the twitter responds.

[WillAyd]

I personally like this idea and the reference to the Django docs. From a practical use case, workflows more often that look something like:

1. Ingest data from a datsource
2. Clean / reshape / merge
3. Perform calculations
4. Visualize

I would think this is particular true for new users coming through say a Jupyter notebook, which is where I think is very friendly to newcomers.

The ten points more or less cover these and can link to deeper discussions from there.

I'd probably move plotting to be the last point out of these as kind of an end goal. Specific to point 9 I'd also make some emphasis around data types preceding it (like converting from a string to a date, why that's important) and relatedly we might want to touch lightly on why dtypes are important (not from a technical perspective, but maybe show object vs numeric somehow)