Stop including binary translation catalogs in the source

Makes it easier to contribute and accept translations.

Does add a new dependency on installation from source, polib. This is
not made mandatory because not all users will want translations.

polib has no further dependencies and can be installed anywhere
LinkChecker can, using pip too.

Add release process step to check polib is installed

The Wheel still includes translations. Recommend install from PyPI in
the README.

Translations were being compiled in setup.py until 2014, using a bundled
copy of msgfmt.py.
9c3739f1c ("Replace msgfmt.py with local tools.", 2014-09-08)

.gitignore

@@ -4,6 +4,7 @@
 *.o
 *.lo
 *.la
+*.mo
 *~
 *.so
 *.bak

MANIFEST.in

@@ -44,7 +44,6 @@ recursive-include doc                   \
     Makefile                            \
     linkcheckerrc_*
 recursive-include po                    \
-    *.mo                                \
     *.po                                \
     *.pot                               \
     Makefile

README.rst

@@ -28,13 +28,16 @@ Features
 Installation
 -------------
 
-See `doc/install.txt`_ in the source code archive for general information. Except the given information there, please take note of the following:
+Python 3.6 or later is needed. Using pip to install LinkChecker:
+
+``pip3 install linkchecker``
+
+The version in the pip repository may be old, to find out how to get the latest
+code, plus platform-specific information and other advice see `doc/install.txt`_
+in the source code archive.
 
 .. _doc/install.txt: doc/install.txt
 
-Python 3.6 or later is needed.
-
-The version in the pip repository may be old. Instead, you can use pip to install the latest code from git: ``pip3 install git+https://github.com/linkchecker/linkchecker.git``.
 
 Usage
 ------

doc/changelog.txt

@@ -1,3 +1,11 @@
+10.x (released x)
+
+Changes:
+- Binary translation catalogs are no longer included with the source. The polib
+  package must be installed before installation from source to compile
+  translations.
+
+
 10.0.1 (released 29.1.2021)
 
 Changes:

doc/development.md

@@ -68,10 +68,12 @@ Release process
 
 5. create a new git clone
 
-6. build Python distribution files (`setup.py sdist bdist_wheel`)
+6. check Python polib package is installed
 
-7. check distribution files (`twine check dist/*`) and upload to PyPI (`twine upload dist/*`)
+7. build Python distribution files (`setup.py sdist bdist_wheel`)
 
-8. create release (vX.Y.Z) on GitHub (GitHub creates the .tar.gz and .zip archives)
+8. check distribution files (`twine check dist/*`) and upload to PyPI (`twine upload dist/*`)
 
-9. increment AppVersion to vX.Y.Z+1.dev0
+9. create release (vX.Y.Z) on GitHub (GitHub creates the .tar.gz and .zip archives)
+
+10. increment AppVersion to vX.Y.Z+1.dev0

doc/install.txt

@@ -3,10 +3,20 @@ Installation
 If you are upgrading from older versions of LinkChecker you should
 also read the upgrading documentation stored in upgrading.txt.
 
+When installing from source, for application translations to be installed
+polib_ needs to be installed before LinkChecker. After LinkChecker installation
+polib_ can be removed.
+
+.. _polib: https://pypi.org/project/polib/
+
 Setup with pip
 ------------------
-If pip_ is available, this command should install LinkChecker on
-the local system:
+pip_ can be used to install LinkChecker on the local system.
+
+If you want application translations, first:
+``pip3 install polib``
+
+Then this command will install LinkChecker from the latest source:
 ``pip3 install git+https://github.com/linkchecker/linkchecker.git``
 
 .. _pip: https://pypi.org/project/pip/
@@ -44,20 +54,23 @@ First, install the required software.
 
 3. Python Beautiful Soup package from https://pypi.org/project/beautifulsoup4/
 
-4. *Optional, for bash-completion:*
+4. *Optional, installation time only, for translations:*
+   polib Python module from https://pypi.org/project/polib/
+
+5. *Optional, for bash-completion:*
    argcomplete Python module from https://pypi.org/project/argcomplete/
 
-5. *Optional, for displaying country codes:*
+6. *Optional, for displaying country codes:*
    GeoIP from https://pypi.org/project/GeoIP/
 
-6. *Optional, used for Virus checking:*
+7. *Optional, used for Virus checking:*
    ClamAv from https://www.clamav.net/
 
-7. *Optional, for GNOME proxy setting parsing:*
+8. *Optional, for GNOME proxy setting parsing:*
    PyGObject and GIO.
    Best installed from your distribution e.g. ``python3-gi``
 
-8. *Optional, to run the WSGI web interface:*
+9. *Optional, to run the WSGI web interface:*
    Apache from https://httpd.apache.org/
    mod_wsgi from https://pypi.org/project/mod-wsgi/
 

doc/src/index.rst

@@ -19,9 +19,11 @@ Installation
 
 .. code-block:: console
 
-   $ pip3 install git+https://github.com/linkchecker/linkchecker.git
-   
-See the :doc:`installation document <install>` for more information.
+   $ pip3 install linkchecker
+
+The version in the pip repository may be old, to find out how to get the latest
+code, plus platform-specific information and other advice see the
+:doc:`installation document <install>`.
 
 Basic usage
 ------------

doc/translations.md

@@ -7,16 +7,21 @@ Translations for the man pages are stored in doc/.
 Application Translations
 ------------------------
 
-``linkchecker $ make locale``
+Makefiles using GNU gettext utilities are provided to manage .po and .pot files.
 
-is equivalent to:
+If the strings in the application change, update the .pot and .po files:
 
-``linkchecker/po $ make``
+``linkchecker/po $ rm linkchecker.pot; make``
+
+Do make a commit at this point.
 
 Translation progress and validity can be monitored with:
 
 ``linkchecker/po $ make check``
 
+.mo files are not stored in the repository and are created on building,
+using polib.
+
 Man Page Translations
 ---------------------
 

doc/upgrading.txt

@@ -1,5 +1,10 @@
 Upgrading
 =========
+Migrating from 10.0 to 10.x
+---------------------------
+If installing from source and application translations are needed the Python
+polib package is required to be installed before LinkChecker is installed.
+
 Migrating from 9.x to 10.0
 --------------------------
 Python 3.6 or newer is required.

po/Makefile

@@ -3,20 +3,13 @@ MSGFMT := msgfmt
 MSGMERGE := msgmerge
 POSOURCES = $(shell find ../linkcheck -name \*.py) \
 	../linkchecker $(shell python3 -c 'import argparse; print(argparse.__file__)')
-LDIR = ../share/locale
 PACKAGE = linkchecker
 TEMPLATE = $(PACKAGE).pot
 MYMAIL := bastian.kleineidam@web.de
 BUGSURL = https://github.com/linkchecker/linkchecker
-LFILE = LC_MESSAGES/$(PACKAGE).mo
-# defined language (add new languages here)
-LANGUAGES = de fr es
-MOFILES = $(wildcard *.mo)
+POFILES = $(wildcard *.po)
 
-all:	$(MOFILES)
-
-%.mo:	%.po
-	$(MSGFMT) -c --statistics -o $@ $<
+all:	$(POFILES)
 
 %.po:	$(TEMPLATE)
 	$(MSGMERGE) -U --suffix=.bak $@ $<
@@ -35,7 +28,6 @@ check:
 	done
 
 clean:
-	@for f in $(LANGUAGES); do rm -f $(LDIR)/$$f/$(LFILE); done
-	rm -f *.mo *.bak
+	rm -f *.bak
 
 .PHONY: check clean

setup.py

@@ -32,11 +32,12 @@ if sys.version_info < (3, 6, 0, "final", 0):
 import os
 import re
 import stat
-import glob
+from pathlib import Path
 
 # import Distutils stuff
 from setuptools import find_packages, setup
 from distutils.command.install_lib import install_lib
+from distutils.command.build import build
 from distutils.command.clean import clean
 from distutils.command.install_data import install_data
 from distutils.dir_util import remove_tree
@@ -44,6 +45,14 @@ from distutils.file_util import write_file
 from distutils import util, log
 from distutils.core import Distribution
 
+try:
+    import polib
+except ImportError:
+    print("polib package not found. Translations not compiled.")
+    COMPILE_TRANSLATIONS = False
+else:
+    COMPILE_TRANSLATIONS = True
+
 # the application version
 AppVersion = "10.0.1"
 # the application name
@@ -97,6 +106,18 @@ def get_portable():
     return os.environ.get("LINKCHECKER_PORTABLE", "0")
 
 
+class MyBuild(build):
+    """Custom build with translation compilation"""
+
+    def run(self):
+        if COMPILE_TRANSLATIONS:
+            for (src, bld_path, dst) in list_translation_files():
+                pofile = polib.pofile(src)
+                bld_path.parent.mkdir(exist_ok=True, parents=True)
+                pofile.save_as_mofile(str(bld_path))
+        super().run()
+
+
 class MyInstallLib(install_lib):
     """Custom library installation."""
 
@@ -161,38 +182,13 @@ class MyInstallData(install_data):
     """Fix file permissions."""
 
     def run(self):
-        """Adjust permissions on POSIX systems."""
-        self.install_translations()
+        """Handle translation files and adjust permissions on POSIX systems."""
+        if COMPILE_TRANSLATIONS:
+            for (src, bld_path, dst) in list_translation_files():
+                self.data_files.append((dst, [str(bld_path)]))
         super().run()
         self.fix_permissions()
 
-    def install_translations(self):
-        """Install compiled gettext catalogs."""
-        # A hack to fix https://github.com/linkchecker/linkchecker/issues/102
-        i18n_files = []
-        data_files = []
-        for dir, files in self.data_files:
-            if "LC_MESSAGES" in dir:
-                i18n_files.append((dir, files))
-            else:
-                data_files.append((dir, files))
-        self.data_files = data_files
-        # We do almost the same thing that install_data.run() does, except
-        # we can assume everything in self.data_files is a (dir, files) tuple,
-        # and all files lists are non-empty.  And for i18n files, instead of
-        # specifying the directory we instead specify the destination filename.
-        for dest, files in i18n_files:
-            dest = util.convert_path(dest)
-            if not os.path.isabs(dest):
-                dest = os.path.join(self.install_dir, dest)
-            elif self.root:
-                dest = util.change_root(self.root, dest)
-            self.mkpath(os.path.dirname(dest))
-            for data in files:
-                data = util.convert_path(data)
-                (out, _) = self.copy_file(data, dest)
-                self.outfiles.append(out)
-
     def fix_permissions(self):
         """Set correct read permissions on POSIX systems. Might also
         be possible by setting umask?"""
@@ -272,16 +268,15 @@ class MyDistribution(Distribution):
         )
 
 
-def list_message_files(package, suffix=".mo"):
-    """Return list of all found message files and their installation paths."""
-    for fname in glob.glob("po/*" + suffix):
-        # basename (without extension) is a locale name
-        localename = os.path.splitext(os.path.basename(fname))[0]
-        domainname = "%s.mo" % package.lower()
-        yield (
-            fname,
-            os.path.join("share", "locale", localename, "LC_MESSAGES", domainname),
-        )
+def list_translation_files():
+    """Return list of translation files and their build and installation paths."""
+    for po in Path("po").glob("*.po"):
+        mo = Path(
+            "share", "locale", po.stem, "LC_MESSAGES", AppName.lower()
+            ).with_suffix(".mo")
+        build_mo = Path("build", mo)
+        build_mo.parent.mkdir(exist_ok=True, parents=True)
+        yield (str(po), build_mo, str(mo.parent))
 
 
 class MyClean(clean):
@@ -322,9 +317,6 @@ data_files = [
     ),
 ]
 
-for (src, dst) in list_message_files(AppName):
-    data_files.append((dst, [src]))
-
 if os.name == "posix":
     data_files.append(("share/man/man1", ["doc/man/en/linkchecker.1"]))
     data_files.append(("share/man/man5", ["doc/man/en/linkcheckerrc.5"]))
@@ -357,6 +349,7 @@ setup(
     long_description_content_type="text/x-rst",
     distclass=MyDistribution,
     cmdclass={
+        "build": MyBuild,
         "install_lib": MyInstallLib,
         "install_data": MyInstallData,
         "clean": MyClean,