About the internationalization of documentation

[kozo2]

I want to contribute to the internationalization of pandas documentation.

I created the POT files corresponding to the Sphinx source of pandas and submitted a pull request #56296 .

This PR was made following a suggestion I received from @steppi in the Scientific Python Discord.

If you have any thoughts, please let me know.

[rhshadrach]

Thanks for opening this up. Can you give a summary of what these POT files are, how they are to be used, and why they need to be added to pandas.

[steppi]

Hi @rhshadrach. I'm working on a CZI supported project to help add translations to the websites for major packages in the Scientific Python ecosystem. You may have noticed the language drop-down at https://numpy.org/. I planned to reach out to maintainers of all of the major projects shortly to ask if they'd like help setting something like this up. @kozo2 is very enthusiastic about the translation efforts and started working on the set-up independently. I'm very busy finishing a grant application at the moment, but will be able to fill you in on the details sometime next week.

[steppi]

Sorry, for taking so long to get back to this. I've had a lot on my plate, and there were some delays getting a free enterprise organization set up for Scientific Python on the localization management platform Crowdin.

I work for Quansight Labs and am helping with a CZI supported project to translate and localize the brochure websites of core scientific Python projects. The goal is to translate the websites of 8 core Scientific Python projects into at least 3 widely used languages. I'd like to gauge interest Pandas maintainers have in participating, and will give an overview of what would be involved.

For any given project

• I will need to invite someone with admin privileges on the repo containing the code for the brochure website to the Scientific Python Crowdin organization. For Pandas, this repo seems to be the main Pandas repo itself.
• A configuration file crowdin.yml needs to be added to this repo. The configuration file controls which files will be targeted for translation, and various configuration files. You can see numpy.orgs here. For Pandas, I think a restructuring of the content files may be needed. Also Add pot files for translating pandas documentation #56296 attempts to add pot files for enabling translation of all of Pandas' documentation, but this is extremely ambitious and outside of the scope of this CZI-backed project. It is not only a very large set of material, but documentation can change frequently. I recommend starting with the content of the brochure website. As far as I can tell, this is all in markdown for Pandas, which is very well supported by Crowdin, so there would be no need to bother with pot files. Some other projects use rst for their brochure websites and will need to use pot files though.
• Myself and/or a colleague(s) at Quansight will help find interested and qualified translators and there will likely be a substantial overlap in translators between projects.
• The person mentioned above with admin privileges will need to sync Crowdin with GitHub. Crowdin will automatically segment the content of interest into translatable strings and autogenerate a PR where new translations will be pushed. Changes and additions to the target content on the repo will automatically be propagated to Crowdin's UI for translation. There are some quirks in the Crowdin workflow that I've found workarounds for which I can explain later.
• Depending on the static site generator used, changes may be needed in order to add a drop down for selecting the language on the page like the one at numpy.org. I know the Scientific Python hugo theme makes this easy, but it appears that Pandas has its own custom static site generator. I imagine it would not be too difficult to make the necessary enhancements though.

Please let me know if you have any questions. I have likely missed some important information, but will read through this again to see if there's anything I should add. I will start reaching out to other projects this coming week, but starting here since @kozo2 has already got the ball rolling.

[kozo2]

Thanks for opening this up. Can you give a summary of what these POT files are, how they are to be used, and why they need to be added to pandas.

@rhshadrach Sorry for the late reply.

These POT files serve as template files for creating multilingual translations of the Sphinx rst files.
PO files are created by adding translations in languages other than English to these POT files.
And the PO files are used to generate rst files that are translations of the original English documents.

Since these POT files need to exist in the same repository as the original English documents to work, I sent this PR.

However, I understand that these POT files are numerous and their mechanism is difficult to understand, making them challenging to merge.

Therefore, I would be grateful if you could consider starting the multilingual documentation on a small scale without using the POT files, as @steppi suggests.
And for the time being, it's okay to ignore (the translations that use) these POT files.

[rhshadrach]

Thanks @steppi for filling in some of the details!

For Pandas, I think a restructuring of the content files may be needed.

Can you elaborate here?

I recommend starting with the content of the brochure website.

What is meant by "brochure website"? Searching the term gives me definitions such as "A brochure website is an informational website that is designed to look and feel like a printed brochure" but that doesn't make too much sense in this context I think.

The person mentioned above with admin privileges will need to connect sync Crowdin with GitHub.

Is this the connect sync you're referring to?

Crowdin will automatically segment the content of interest into translatable strings and autogenerate a PR where new translations will be pushed.

What repository are these PRs being put up in?

@kozo2

Therefore, I would be grateful if you could consider starting the multilingual documentation on a small scale without using the POT files, as @steppi suggests.
And for the time being, it's okay to ignore (the translations that use) these POT files.

From this, it sounds like there will be a desire to eventually consider the POT files though - is that correct? If that's the case, then I think they need to be considered upfront. I do not think pandas should start going down this path without a clear understanding of the endpoint.

cc @datapythonista

[steppi]

Good questions @rhshadrach.

Can you elaborate here?

Certainly. The changes would be minor. The content files for what I'm calling Pandas' brochure website are at https://github.com/pandas-dev/pandas/tree/main/web/pandas. It will simplify the Crowdin config to put the English content in something like web/pandas/content/en, and then translated content can go in web/pandas/content/ja, web/pandas/content/pt, etc. where en, ja, pt are two letter ISO 639-1 language codes. I think it would also work to put the English content in web/pandas/en and then put translated content in web/pandas/ja etc. The important thing is to have parallel folders for each language. (I know it probably seems obvious, but you'll also want to avoid putting folders which should not be targets for translation together with the content. That's why I suggested having a separate content folder that things like static/* robots.txt, versions.json, would sit outside of).

What is meant by "brochure website"? Searching the term gives me definitions such as "A brochure website is an informational website that is designed to look and feel like a printed brochure" but that doesn't make too much sense in this context I think.

I'm just repeating the term that was used when this task was explained to me. "brochure website" here is meant to stand in contrast to things like API documentation, tutorials, etc. Another term could be "core project website". For Pandas, this is https://pandas.pydata.org/. For NumPy, it is https://numpy.org. You may want to click around there to see the extent of the content which has been translated. A sample of other Scientific Python brochure websites includes https://scipy.org, https://scikit-learn.org/, https://matplotlib.org/, https://jupyter.org/, https://www.sympy.org.

Is this the connect sync you're referring to?

Crowdin will automatically segment the content of interest into translatable strings and autogenerate a PR where new translations will be pushed.

Yes. I had meant to include a link, and to only use the word sync. "connect sync" is a consequence of the sentence being left in an inconsistent state after editing. Sorry, I should probably compose long messages like this separately in my editor, instead of directly in the Github UI.

What repository are these PRs being put up in?

There will be single PR to the repository hosting the content files for the "brochure website". Currently for Pandas, this would be to the Pandas repo itself.

From this, it sounds like there will be a desire to eventually consider the POT files though - is that correct? If that's the case, then I think they need to be considered upfront. I do not think pandas should start going down this path without a clear understanding of the endpoint.

I think localization of the "brochure website" and localization of the sphinx documentation can be considered separately. As a SciPy maintainer, I can tell you that there is little will to create and maintain official and up-to-date translations of the large and oft-changing API documentation and tutorials for SciPy. Though these translations would be nice to have, it would involve considerable effort, and the collective bandwidth would be difficult to secure. You'll have to decide what extent of translation you all would find feasible to create and maintain. From here, the scope of the project I'm helping with is:

One such improvement is translation and localization. Development takes place in English, as reflected by project websites and documentation. While many contributors are comfortable with English as a first, second, or even third language, the language barrier excludes especially users that are very young, are new to the community, have learning disabilities, or are from the Global South—all potential future contributors and leaders in the scientific Python community! We will therefore translate key pages of core project websites, and provide translation infrastructure for the web themes.

This is the endpoint I have in mind at least.

[rhshadrach]

Thanks - I don't personally have a good understanding of the security in place with integrating Crowdin to GitHub, but other projects already using it like NumPy is certainly promising. I do think this should be better understood (at least, better than my current understanding) assuming we are to move forward with it.

cc @pandas-dev/pandas-core

[Dr-Irv]

I'm not sure of the value with doing this, given the work it may put on the pandas team. If someone needs a translated version of the pandas web site, they can use google translate. I'm not sure how good those translations are, but you then have automatic translation for many languages.

E.g., here's the pandas web site translated to German:
https://pandas-pydata-org.translate.goog/?_x_tr_sch=http&_x_tr_sl=en&_x_tr_tl=de&_x_tr_hl=en&_x_tr_pto=wapp

@steppi what is the value of what you are proposing as opposed to telling people to use Google Translate? Is it that the auto-translations are inaccurate? Or something else?

[steppi]

That's a good question @Dr-Irv. We wouldn't want to put effort into this if it's not actually helpful. From what we've seen so far, automatic translation services aren't always adequate for translating the kind of jargon-laden technical language involved. Machine translation is being used in the translation process, but it is very helpful to have human editors with subject expertise review and sign off on the machine translated content. @melissawm helped write the proposal for this grant and worked on the Portuguese translation of numpy.org. I'm sure she'd have more to share about the value of these translation efforts.

[steppi]

Thanks - I don't personally have a good understanding of the security in place with integrating Crowdin to GitHub, but other projects already using it like NumPy is certainly promising. I do think this should be better understood (at least, better than my current understanding) assuming we are to move forward with it.

I agree that this is important to consider. I wasn't involved in this project yet when Crowdin was first synced with the numpy.org repo, and so wasn't there for the discussions around security. I'll ask around and investigate a little and get back with a clear report on the security implications.

[rgommers]

Quick thought re auto-translation: it also forces the use of Google Chrome, which may not be something that's perfectly aligned with the openness of community open source. It's a trade-off of course, so just one of multiple things to take into account. Hopefully Firefox and other browsers will get there, but it looks like a long way off still.

[Dr-Irv]

Quick thought re auto-translation: it also forces the use of Google Chrome, which may not be something that's perfectly aligned with the openness of community open source. It's a trade-off of course, so just one of multiple things to take into account. Hopefully Firefox and other browsers will get there, but it looks like a long way off still.

Just did a French translation with Google translate (the web site) in Firefox. This URL should work:
https://pandas-pydata-org.translate.goog/?_x_tr_sch=http&_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp