Merge pull request #563 from cjmayo/nomo

Stop including binary translation catalogs in the source

.github/workflows/build.yml

@@ -148,3 +148,17 @@ jobs:
 
       - name: Run ${{ matrix.toxenv }}
         run: python -m tox -e ${{ matrix.toxenv }}
+
+  translations:
+    name: Check Translations
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Git clone
+        uses: actions/checkout@v2
+
+      - name: Install dependency
+        run: sudo apt install -y gettext
+
+      - name: Compile translations
+        run: for po in po/*.po; do msgfmt -cv --statistics -o - ${po} >/dev/null; done

.gitignore

@@ -4,6 +4,7 @@
 *.o
 *.lo
 *.la
+*.mo
 *~
 *.so
 *.bak

MANIFEST.in

@@ -44,7 +44,6 @@ recursive-include doc                   \
     Makefile                            \
     linkcheckerrc_*
 recursive-include po                    \
-    *.mo                                \
     *.po                                \
     *.pot                               \
     Makefile

README.rst

@@ -28,13 +28,16 @@ Features
 Installation
 -------------
 
-See `doc/install.txt`_ in the source code archive for general information. Except the given information there, please take note of the following:
+Python 3.6 or later is needed. Using pip to install LinkChecker:
+
+``pip3 install linkchecker``
+
+The version in the pip repository may be old, to find out how to get the latest
+code, plus platform-specific information and other advice see `doc/install.txt`_
+in the source code archive.
 
 .. _doc/install.txt: doc/install.txt
 
-Python 3.6 or later is needed.
-
-The version in the pip repository may be old. Instead, you can use pip to install the latest code from git: ``pip3 install git+https://github.com/linkchecker/linkchecker.git``.
 
 Usage
 ------

doc/changelog.txt

@@ -1,3 +1,11 @@
+10.x (released x)
+
+Changes:
+- Binary translation catalogs are no longer included with the source. The polib
+  package must be installed before installation from source to compile
+  translations.
+
+
 10.0.1 (released 29.1.2021)
 
 Changes:

doc/development.md

@@ -68,10 +68,12 @@ Release process
 
 5. create a new git clone
 
-6. build Python distribution files (`setup.py sdist bdist_wheel`)
+6. check Python polib package is installed
 
-7. check distribution files (`twine check dist/*`) and upload to PyPI (`twine upload dist/*`)
+7. build Python distribution files (`setup.py sdist bdist_wheel`)
 
-8. create release (vX.Y.Z) on GitHub (GitHub creates the .tar.gz and .zip archives)
+8. check distribution files (`twine check dist/*`) and upload to PyPI (`twine upload dist/*`)
 
-9. increment AppVersion to vX.Y.Z+1.dev0
+9. create release (vX.Y.Z) on GitHub (GitHub creates the .tar.gz and .zip archives)
+
+10. increment AppVersion to vX.Y.Z+1.dev0

doc/install.txt

@@ -3,10 +3,20 @@ Installation
 If you are upgrading from older versions of LinkChecker you should
 also read the upgrading documentation stored in upgrading.txt.
 
+When installing from source, for application translations to be installed
+polib_ needs to be installed before LinkChecker. After LinkChecker installation
+polib_ can be removed.
+
+.. _polib: https://pypi.org/project/polib/
+
 Setup with pip
 ------------------
-If pip_ is available, this command should install LinkChecker on
-the local system:
+pip_ can be used to install LinkChecker on the local system.
+
+If you want application translations, first:
+``pip3 install polib``
+
+Then this command will install LinkChecker from the latest source:
 ``pip3 install git+https://github.com/linkchecker/linkchecker.git``
 
 .. _pip: https://pypi.org/project/pip/
@@ -21,7 +31,7 @@ Setup for macOS
 ------------------
 Python from Homebrew includes pip_. Otherwise ``get-pip.py`` can be used to
 install pip_ (untested):
-https://pip.pypa.io/en/stable/installing/
+https://pip.pypa.io/en/stable/installation/
 
 Setup for GNU/Linux
 -------------------
@@ -44,20 +54,23 @@ First, install the required software.
 
 3. Python Beautiful Soup package from https://pypi.org/project/beautifulsoup4/
 
-4. *Optional, for bash-completion:*
+4. *Optional, installation time only, for translations:*
+   polib Python module from https://pypi.org/project/polib/
+
+5. *Optional, for bash-completion:*
    argcomplete Python module from https://pypi.org/project/argcomplete/
 
-5. *Optional, for displaying country codes:*
+6. *Optional, for displaying country codes:*
    GeoIP from https://pypi.org/project/GeoIP/
 
-6. *Optional, used for Virus checking:*
+7. *Optional, used for Virus checking:*
    ClamAv from https://www.clamav.net/
 
-7. *Optional, for GNOME proxy setting parsing:*
+8. *Optional, for GNOME proxy setting parsing:*
    PyGObject and GIO.
    Best installed from your distribution e.g. ``python3-gi``
 
-8. *Optional, to run the WSGI web interface:*
+9. *Optional, to run the WSGI web interface:*
    Apache from https://httpd.apache.org/
    mod_wsgi from https://pypi.org/project/mod-wsgi/
 

doc/src/index.rst

@@ -19,9 +19,11 @@ Installation
 
 .. code-block:: console
 
-   $ pip3 install git+https://github.com/linkchecker/linkchecker.git
-   
-See the :doc:`installation document <install>` for more information.
+   $ pip3 install linkchecker
+
+The version in the pip repository may be old, to find out how to get the latest
+code, plus platform-specific information and other advice see the
+:doc:`installation document <install>`.
 
 Basic usage
 ------------

doc/translations.md

@@ -7,11 +7,20 @@ Translations for the man pages are stored in doc/.
 Application Translations
 ------------------------
 
-``linkchecker $ make locale``
+Makefiles using GNU gettext utilities are provided to manage .po and .pot files.
 
-is equivalent to:
+If the strings in the application change, update the .pot and .po files:
 
-``linkchecker/po $ make``
+``linkchecker/po $ rm linkchecker.pot; make``
+
+Do make a commit at this point.
+
+Translation progress and validity can be monitored with:
+
+``linkchecker/po $ make check``
+
+.mo files are not stored in the repository and are created on building,
+using polib.
 
 Man Page Translations
 ---------------------

doc/upgrading.txt

@@ -1,5 +1,10 @@
 Upgrading
 =========
+Migrating from 10.0 to 10.x
+---------------------------
+If installing from source and application translations are needed the Python
+polib package is required to be installed before LinkChecker is installed.
+
 Migrating from 9.x to 10.0
 --------------------------
 Python 3.6 or newer is required.

po/Makefile

@@ -3,19 +3,13 @@ MSGFMT := msgfmt
 MSGMERGE := msgmerge
 POSOURCES = $(shell find ../linkcheck -name \*.py) \
 	../linkchecker $(shell python3 -c 'import argparse; print(argparse.__file__)')
-LDIR = ../share/locale
 PACKAGE = linkchecker
 TEMPLATE = $(PACKAGE).pot
 MYMAIL := bastian.kleineidam@web.de
-LFILE = LC_MESSAGES/$(PACKAGE).mo
-# defined language (add new languages here)
-LANGUAGES = de fr es
-MOFILES = $(wildcard *.mo)
+BUGSURL = https://github.com/linkchecker/linkchecker
+POFILES = $(wildcard *.po)
 
-all:	$(MOFILES)
-
-%.mo:	%.po
-	$(MSGFMT) -c --statistics -o $@ $<
+all:	$(POFILES)
 
 %.po:	$(TEMPLATE)
 	$(MSGMERGE) -U --suffix=.bak $@ $<
@@ -24,11 +18,16 @@ template: $(TEMPLATE)
 $(TEMPLATE): $(POSOURCES)
 	$(XGETTEXT) -w 80 --default-domain=$(PACKAGE) --language=Python \
 	  --copyright-holder="Bastian Kleineidam <$(MYMAIL)>" \
-	  --msgid-bugs-address=$(MYMAIL) -o $(TEMPLATE) \
+	  --package-name=LinkChecker \
+	  --msgid-bugs-address=$(BUGSURL) -o $(TEMPLATE) \
 	  --keyword=_n:1,2 $(POSOURCES)
 
-clean:
-	@for f in $(LANGUAGES); do rm -f $(LDIR)/$$f/$(LFILE); done
-	rm -f *.mo *.bak
+check:
+	@for po in $(POFILES); do \
+		$(MSGFMT) -cv --statistics -o - $$po >/dev/null; \
+	done
 
-.PHONY: clean
+clean:
+	rm -f *.bak
+
+.PHONY: check clean

po/de.po

@@ -3,7 +3,7 @@
 #
 msgid ""
 msgstr ""
-"Project-Id-Version: $Id$\n"
+"Project-Id-Version: LinkChecker\n"
 "Report-Msgid-Bugs-To: bastian.kleineidam@web.de\n"
 "POT-Creation-Date: 2020-06-19 16:25+0100\n"
 "PO-Revision-Date: 2020-06-19 16:42+0100\n"

po/es.po

@@ -5,7 +5,7 @@
 #
 msgid ""
 msgstr ""
-"Project-Id-Version: linkchecker 2.8\n"
+"Project-Id-Version: LinkChecker\n"
 "Report-Msgid-Bugs-To: bastian.kleineidam@web.de\n"
 "POT-Creation-Date: 2020-06-19 16:25+0100\n"
 "PO-Revision-Date: 2011-02-16 15:24+0100\n"

po/fr.po

@@ -6,7 +6,7 @@
 #
 msgid ""
 msgstr ""
-"Project-Id-Version: fr\n"
+"Project-Id-Version: LinkChecker\n"
 "Report-Msgid-Bugs-To: bastian.kleineidam@web.de\n"
 "POT-Creation-Date: 2020-06-19 16:25+0100\n"
 "PO-Revision-Date: 2011-02-16 15:27+0100\n"

po/linkchecker.pot

@@ -1,12 +1,12 @@
 # SOME DESCRIPTIVE TITLE.
 # Copyright (C) YEAR Bastian Kleineidam <bastian.kleineidam@web.de>
-# This file is distributed under the same license as the PACKAGE package.
+# This file is distributed under the same license as the LinkChecker package.
 # FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
 #
 #, fuzzy
 msgid ""
 msgstr ""
-"Project-Id-Version: PACKAGE VERSION\n"
+"Project-Id-Version: LinkChecker\n"
 "Report-Msgid-Bugs-To: bastian.kleineidam@web.de\n"
 "POT-Creation-Date: 2020-06-19 16:25+0100\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"

setup.py

@@ -32,11 +32,12 @@ if sys.version_info < (3, 6, 0, "final", 0):
 import os
 import re
 import stat
-import glob
+from pathlib import Path
 
 # import Distutils stuff
 from setuptools import find_packages, setup
 from distutils.command.install_lib import install_lib
+from distutils.command.build import build
 from distutils.command.clean import clean
 from distutils.command.install_data import install_data
 from distutils.dir_util import remove_tree
@@ -44,6 +45,14 @@ from distutils.file_util import write_file
 from distutils import util, log
 from distutils.core import Distribution
 
+try:
+    import polib
+except ImportError:
+    print("polib package not found. Translations not compiled.")
+    COMPILE_TRANSLATIONS = False
+else:
+    COMPILE_TRANSLATIONS = True
+
 # the application version
 AppVersion = "10.0.1"
 # the application name
@@ -97,6 +106,18 @@ def get_portable():
     return os.environ.get("LINKCHECKER_PORTABLE", "0")
 
 
+class MyBuild(build):
+    """Custom build with translation compilation"""
+
+    def run(self):
+        if COMPILE_TRANSLATIONS:
+            for (src, bld_path, dst) in list_translation_files():
+                pofile = polib.pofile(src)
+                bld_path.parent.mkdir(exist_ok=True, parents=True)
+                pofile.save_as_mofile(str(bld_path))
+        super().run()
+
+
 class MyInstallLib(install_lib):
     """Custom library installation."""
 
@@ -161,38 +182,13 @@ class MyInstallData(install_data):
     """Fix file permissions."""
 
     def run(self):
-        """Adjust permissions on POSIX systems."""
-        self.install_translations()
+        """Handle translation files and adjust permissions on POSIX systems."""
+        if COMPILE_TRANSLATIONS:
+            for (src, bld_path, dst) in list_translation_files():
+                self.data_files.append((dst, [str(bld_path)]))
         super().run()
         self.fix_permissions()
 
-    def install_translations(self):
-        """Install compiled gettext catalogs."""
-        # A hack to fix https://github.com/linkchecker/linkchecker/issues/102
-        i18n_files = []
-        data_files = []
-        for dir, files in self.data_files:
-            if "LC_MESSAGES" in dir:
-                i18n_files.append((dir, files))
-            else:
-                data_files.append((dir, files))
-        self.data_files = data_files
-        # We do almost the same thing that install_data.run() does, except
-        # we can assume everything in self.data_files is a (dir, files) tuple,
-        # and all files lists are non-empty.  And for i18n files, instead of
-        # specifying the directory we instead specify the destination filename.
-        for dest, files in i18n_files:
-            dest = util.convert_path(dest)
-            if not os.path.isabs(dest):
-                dest = os.path.join(self.install_dir, dest)
-            elif self.root:
-                dest = util.change_root(self.root, dest)
-            self.mkpath(os.path.dirname(dest))
-            for data in files:
-                data = util.convert_path(data)
-                (out, _) = self.copy_file(data, dest)
-                self.outfiles.append(out)
-
     def fix_permissions(self):
         """Set correct read permissions on POSIX systems. Might also
         be possible by setting umask?"""
@@ -272,16 +268,15 @@ class MyDistribution(Distribution):
         )
 
 
-def list_message_files(package, suffix=".mo"):
-    """Return list of all found message files and their installation paths."""
-    for fname in glob.glob("po/*" + suffix):
-        # basename (without extension) is a locale name
-        localename = os.path.splitext(os.path.basename(fname))[0]
-        domainname = "%s.mo" % package.lower()
-        yield (
-            fname,
-            os.path.join("share", "locale", localename, "LC_MESSAGES", domainname),
-        )
+def list_translation_files():
+    """Return list of translation files and their build and installation paths."""
+    for po in Path("po").glob("*.po"):
+        mo = Path(
+            "share", "locale", po.stem, "LC_MESSAGES", AppName.lower()
+            ).with_suffix(".mo")
+        build_mo = Path("build", mo)
+        build_mo.parent.mkdir(exist_ok=True, parents=True)
+        yield (str(po), build_mo, str(mo.parent))
 
 
 class MyClean(clean):
@@ -322,9 +317,6 @@ data_files = [
     ),
 ]
 
-for (src, dst) in list_message_files(AppName):
-    data_files.append((dst, [src]))
-
 if os.name == "posix":
     data_files.append(("share/man/man1", ["doc/man/en/linkchecker.1"]))
     data_files.append(("share/man/man5", ["doc/man/en/linkcheckerrc.5"]))
@@ -357,6 +349,7 @@ setup(
     long_description_content_type="text/x-rst",
     distclass=MyDistribution,
     cmdclass={
+        "build": MyBuild,
         "install_lib": MyInstallLib,
         "install_data": MyInstallData,
         "clean": MyClean,