Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-04-27 09:34:42 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

restructuring of installation

Browse source 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@166 e7d03fd6-7b0d-0410-9947-9c21f3af8025
This commit is contained in:
calvin 2000-10-10 09:28:27 +00:00
parent 32699adc40
commit 5b1c8f9e1a
15 changed files with 148 additions and 310 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
.cvsignore
View file

@ -5,10 +5,7 @@ build-stamp
*.o
build
dist
linkchecker
linkchecker.bat
linkchecker-*.tar.gz
MANIFEST
VERSION
README
install.py
config.py

23
INSTALL
View file

@ -3,33 +3,26 @@
Requirements
------------
You need Python >= 1.5.2 from http://www.python.org/
Python >= 1.5.2 from http://www.python.org/
Distutils >= 0.9 from http://www.python.org/sigs/distutils-sig/
Optionally packages
-------------------
Distutils >= 0.9 from http://www.python.org/sigs/distutils-sig/
OpenSSL from http://www.openssl.org/
You will need Perl for Win32 (available from
http://www.activestate.com/ActivePerl) if you want to install OpenSSL
on Windows!
Install without Distutils
-------------------------
Execute "python install.py" to install LinkChecker in the distribution
directory.
To use HTTPS, compile the Pyhton module ssl.c, for example with
"gcc -c -I/usr/include/openssl -I/usr/include/python1.5 -g -O2 -fPIC \
ssl.c -o ssl.o"
Put the distribution directory in your path to execute the linkchecker
script from everywhere.
Install with Distutils
----------------------
Setup
-----
The Distutils package has a lot of install options to tweak and ease
your installation. Type "python setup.py --help" for options.
To install LinkChecker run "python setup.py install".
Debian users can build the .deb package with "debian/rules binary".
Debian users can build the .deb package with "debian/rules binary" as
root or "fakeroot debian/rules binary" as a normal user.
The system wide configuration file is in /etc/linkcheckerrc.
Running the link checker

1
MANIFEST.in
View file

@ -14,3 +14,4 @@ include DNS/README
include linkcheck/*.tmpl
include test/viewprof.py test/profiletest.py.tmpl test/*.html
recursive-include locale *.mo *.po
recursive-include lconline *

3
Makefile
View file

@ -54,7 +54,8 @@ distclean: clean
dist: mo
rm -rf debian/tmp
python setup.py sdist --formats=gztar,zip bdist_rpm
# python setup.py bdist_wininst
# extra run without SSL compilation
python setup.py bdist_wininst
fakeroot debian/rules binary
mv -f ../$(DEBPACKAGE) dist

21
README.tmpl → README
View file

@ -1,7 +1,24 @@
LinkChecker
=============
$long_description
With LinkChecker you can check your HTML documents for broken links.
Features:
---------
o recursive checking
o multithreaded
o output can be colored or normal text, HTML, SQL, CSV or a GML sitemap
graph
o HTTP/1.1, HTTPS, FTP, mailto:, news:, nntp:, Gopher, Telnet and local
file links are supported.
Javascript links are currently ignored
o restrict link checking with regular expression filters for URLs
o HTTP proxy support
o give username/password for HTTP and FTP authorization
o robots.txt exclusion protocol support
o internationalization support
o (Fast)CGI web interface
Installing, Requirements, Running
---------------------------------
@ -10,7 +27,7 @@ Read the file INSTALL.
License
--------
$appname is licensed under the GNU Public License.
LinkChecker is licensed under the GNU Public License.
Credits go to Guido van Rossum for making Python. His hovercraft is
full of eels!
As this program is directly derived from my Java link checker, additional

1
TODO
View file

@ -3,7 +3,6 @@ High priority
o DNS (and some URLs) seem to hang sometimes. This is not entirely true.
The SMTP connect has a very long timeout (>200 seconds).
But I can not use the signal module with threads.
Wait for Python 1.6 and urllib2.
o I want to be able to supply a "break" command even when multiple
threads are running.

138
Template.py
View file

@ -1,138 +0,0 @@
"""String interpolation
20 Oct 1996
Initial version called Itpl by Ka-Ping Yee
13 Jan 2000
Modified by Bastian Kleineidam to emulate Perls Text::Template
Usage:
Template data consists of normal text and python code fragments.
Code fragments start with a dollar sign followed either by
1) variable or attribute reference or subscription or function call
evaluates to the str() value of the given expression
example: $printit()
2) {python statements}
append arbitrary objects to the output list
Limitations:
1) Code fragments are arbitrary strings with matching "{}" parantheses.
No further syntax verification is done.
2) Parameters to function calls must not have parantheses inside.
Parameters to subscriptions must not have brackets inside.
So this is not allowed:
$func((tuple1,tuple2)) $func(map[key])
But this is allowed:
$func(param)[1] $a[x][y][z](blubb)
In code fragments, you can write every Python code:
${OUT = OUT + trunk[idx[1](data())]}
Example:
import Template
t = Template.Template("blabla.tmpl")
open("blabla","w").write(t.fill_in({"var1":1,"var2":'hui'}))
blabla.tmpl:
Hello, this is a demo. var1 is $var1, and var2 is $var2.
"""
import sys, string, re
from types import StringType
# type argument (we cannot distinguish between filename and string type)
FILENAME = 0
ARRAY = 1
FILE = 2
STRING = 3
# regular expressions for lexical analysis
# note we only match a subset of the language definition
# we match identifiers, attribute reference, subscriptions,
# slicings and function calls
identifier = "[a-zA-Z_][a-zA-Z0-9_]*"
attributeref = identifier + "(\." + identifier + ")*"
subscription = "\[.*\]"
callargs = "\(.*\)"
# now the whole regex
regexvar = re.compile("^"+attributeref+"("+subscription+"|"+callargs+")*")
class TemplateError(StandardError):
pass
class Template:
def __init__(self, data, argtype = FILENAME):
"load the template in self.data"
if argtype == FILENAME:
file = open(data)
self.data = string.join(file.readlines(), "")
file.close()
elif argtype == FILE:
if data.closed:
wasclosed = 1
open(data)
else:
wasclosed = 0
self.data = string.join(data.readlines(), "")
if wasclosed:
data.close()
elif argtype == ARRAY:
self.data = string.join(data, "\n")
else:
self.data = data
if type(self.data) != StringType:
raise TemplateError, "could not read data"
def fill_in(self, dict={}):
"parse the template and fill in values"
try:
namechars = 'abcdefghijklmnopqrstuvwxyz' \
'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_';
out = ""
pos = 0
while pos < len(self.data) and \
string.find(self.data, "$", pos) != -1:
dollar = string.find(self.data, "$", pos)
# append pure text without dollar
out = out + self.data[pos:dollar]
pos = dollar + 1
nextchar = self.data[dollar+1]
if nextchar == '{':
# apply a code fragment
level = 1
while level:
pos = pos + 1
if self.data[pos] == "{":
level = level + 1
elif self.data[pos] == "}":
level = level - 1
output = []
exec self.data[(dollar+2):pos] in dict, {"output":output}
for obj in output:
out = out + obj
pos = pos + 1
elif nextchar in namechars:
# scan a variable access
var = regexvar.match(self.data[pos:])
if var:
# Houston, we have a match!
# extract the string and evaluate it
var = var.group()
pos = pos + len(var)
out = out + str(eval(var, dict))
else:
if nextchar == "$":
pos = pos + 1
out = out + "$"
if pos < len(self.data):
out = out + self.data[pos:]
except:
sys.stderr.write("Template.py: parse error at pos "+`pos`+":\n"+\
out+"\n")
return out

6
debian/changelog vendored
View file

@ -1,3 +1,9 @@
linkchecker (1.2.6) unstable; urgency=low
* placeholder
-- Bastian Kleineidam <calvin@users.sourceforge.net> Sun, 17 Sep 2000 11:22:29 +0200
linkchecker (1.2.5) unstable; urgency=low
* fix /etc/resolv.conf parsing again (this time for real?)

46
install.py.tmpl
View file

@ -1,46 +0,0 @@
"""
Copyright (C) 2000 Bastian Kleineidam
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
"""
import os,sys
from Template import Template
distpath = os.getcwd()
data = {
'install_data': distpath,
'install_scripts': distpath,
"syspath": "sys.path.insert(0, '%s')" % distpath,
'author': '$author',
'version': '$version',
'url': '$url',
'appname': '$appname',
'email': '$email',
'python': os.path.normpath(sys.executable),
}
files = [
'linkchecker',
'linkchecker.bat',
'test/profiletest.py',
'linkcheck/__init__.py',
'linkcheck/Config.py',
]
for name in files:
t = Template(name+".tmpl")
f = open(name,"w")
f.write(t.fill_in(data))
f.close()
os.chmod("linkchecker", 0755)
print "Local installation ok"

1
linkcheck/.cvsignore
View file

@ -1,2 +1 @@
__init__.py
Config.py

15
linkcheck/Config.py.tmpl → linkcheck/Config.py
View file

@ -21,24 +21,23 @@ This module stores
* Other configuration options
"""
import ConfigParser,sys,os,re,UserDict,string,time
import ConfigParser,sys,os,re,UserDict,string,time,Logging,LinkCheckerConf
from os.path import expanduser,normpath,normcase,join,isfile
from types import StringType
import Logging
from linkcheck import _
Version = "$version"
AppName = "$appname"
Version = LinkCheckerConf.version
AppName = LinkCheckerConf.name
App = AppName+" "+Version
UserAgent = AppName+"/"+Version
Author = "$author"
Author = LinkCheckerConf.author
HtmlAuthor = string.replace(Author, ' ', '&nbsp;')
Copyright = "Copyright © 2000 by "+Author
HtmlCopyright = "Copyright &copy; 2000 by "+HtmlAuthor
AppInfo = App+" "+Copyright
HtmlAppInfo = App+", "+HtmlCopyright
Url = "$url"
Email = "$email"
Url = LinkCheckerConf.url
Email = LinkCheckerConf.email
Freeware = AppName+""" comes with ABSOLUTELY NO WARRANTY!
This is free software, and you are welcome to redistribute it
under certain conditions. Look at the file `LICENSE' whithin this
@ -90,7 +89,7 @@ class Configuration(UserDict.UserDict):
self.data["externlinks"] = []
self.data["internlinks"] = []
self.data["allowdeny"] = 0
self.data["authentication"] = [(re.compile(r'^.+$'),
self.data["authentication"] = [(re.compile(r'^.+'),
'anonymous',
'joe@')]
self.data["proxy"] = 0

19
linkchecker.tmpl → linkchecker
View file

@ -1,18 +1,19 @@
#!$python
#!/usr/bin/env python
# imports and checks
import sys
# no i18n at this point
if sys.version[:5] < "1.5.2":
sys.stderr.write("This program requires Python 1.5.2 or later.\n")
sys.exit(1)
# add the path to linkcheck module if you do not install with distutils
$syspath
raise SystemExit, "This program requires Python 1.5.2 or later."
import getopt,re,string,os,urlparse
import linkcheck,StringUtil
try:
import linkcheck
except ImportError:
raise SystemExit, "Could not load linkcheck module. " \
"LinkChecker is not correct installed."
import StringUtil
from linkcheck import _
Usage = _("USAGE\tlinkchecker [options] file_or_url...\n"
"\n"
"OPTIONS\n"

4
linkchecker.bat.tmpl → linkchecker.bat
View file

@ -17,5 +17,5 @@ rem Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
rem uncomment the next line to enable german output
rem set LC_MESSAGES=de
rem Limited to 9 parameters? Is there a $* for Windows?
"$python" "$install_scripts\linkchecker" %1 %2 %3 %4 %5 %6 %7 %8 %9
rem Limited to 9 parameters? Is there a * for Windows?
"python.exe" "linkchecker" %1 %2 %3 %4 %5 %6 %7 %8 %9

10
setup.cfg Normal file
View file

@ -0,0 +1,10 @@
[bdist_rpm]
release = 1
packager = Bastian Kleineidam <calvin@users.sourceforge.net>
doc_files = INSTALL
README
TODO
draft-gilman-news-url-00.txt
lconline/
test/
tests/

165
setup.py
View file

@ -19,103 +19,100 @@
from distutils.core import setup
from distutils.dist import Distribution
from distutils.extension import Extension
from Template import Template
import sys,os,string
from distutils.command.install import install
from distutils.command.config import config
import os
def change_root_inv(root, pathname):
"""the (partially) inverse function of change_root"""
if (not root) or \
(len(root)<len(pathname)) or \
(path[:len(root)]!=root):
return pathname
config_file = "config.py"
class LCInstall(install):
def run(self):
install.run(self)
self.create_conf_file()
def create_conf_file(self):
from distutils.fancy_getopt import longopt_xlate
filename = os.path.join(self.install_lib,
self.distribution.get_name() + "Conf.py")
data = []
self.execute(write_file, (filename, data),
"creating %s" % filename)
class LCConfig(config):
user_options = config.user_options + [
('ssl-include-dirs=', None,
"directories to search for SSL header files"),
('ssl-library-dirs=', None,
"directories to search for SSL library files"),
]
def initialize_options (self):
config.initialize_options(self)
self.ssl_include_dirs = None
self.ssl_library_dirs = None
def finalize_options(self):
if self.ssl_include_dirs is None:
self.ssl_include_dirs = ['/usr/include/openssl',
'/usr/local/include/openssl']
if self.ssl_library_dirs is None:
self.ssl_library_dirs = ['/usr/lib',
'/usr/local/lib']
def check_lib(self, library, library_dirs=None,
headers=None, include_dirs=None,
other_libraries=[]):
self._check_compiler()
return self.try_link("int main (vlid) { }",
headers, include_dirs,
[library]+other_libraries, library_dirs)
def run (self):
if 'bdist_wininst' in self.distribution.commands and os.name!='nt':
self.announce("bdist_wininst command found on non-Windows "
"platform. Disabling SSL compilation")
have_ssl = 0
else:
have_ssl = self.check_lib("ssl",
library_dirs = self.ssl_library_dirs,
include_dirs = self.ssl_include_dirs,
other_libraries = ["crypto"],
headers = ["ssl.h"])
f = open(config_file,'w')
f.write("have_ssl = %d\n" % (have_ssl))
f.write("ssl_library_dirs = %s\n" % `self.ssl_library_dirs`)
f.write("ssl_include_dirs = %s\n" % `self.ssl_include_dirs`)
f.write("libraries = %s\n" % `['ssl', 'crypto']`)
f.close()
path = pathname[len(root):]
if os.name == 'posix':
if path[0]!='/': path = '/'+path
elif os.name == 'nt':
if path[0]!='\\': path = '\\'+path
path = os.path.splitdrive(root)[0] + path
return path
class LCDistribution(Distribution):
default_include_dirs = ['/usr/include/openssl',
'/usr/local/include/openssl']
default_library_dirs = ['/usr/lib',
'/usr/local/lib']
def run_commands (self):
self.check_ssl()
if 'config' not in self.commands:
self.check_ssl()
self.additional_things()
Distribution.run_commands(self)
def check_ssl(self):
if 'bdist_wininst' in self.commands and os.name!='nt':
self.announce(
"""bdist_wininst command found on non-Windows platform.
Disabling SSL compilation""")
return
ok = 0
c = self.get_command_obj('build_ext')
c.ensure_finalized()
incldirs = c.include_dirs+self.default_include_dirs
libdirs = c.library_dirs+self.default_library_dirs
for d in incldirs:
if os.path.exists(os.path.join(d, "ssl.h")):
self.announce('Found %s/ssl.h' % d)
ok = ok + 1
for d in libdirs:
if os.path.exists(os.path.join(d, "libssl.so")):
self.announce('Found %s/libssl.so' % d)
ok = ok + 1
if ok==2:
self.announce("Enabling SSL compilation")
self.reinitialize_command(c)
if not os.path.exists(config_file):
raise SystemExit, 'Please configure LinkChecker by running ' \
'"python setup.py config".'
import config
if config.have_ssl:
self.ext_modules = [Extension('ssl', ['ssl.c'],
include_dirs=incldirs,
library_dirs=libdirs,
libraries=['ssl'])]
c.ensure_finalized()
else:
self.announce(
"""Some necessary SSL files are missing, disabling SSL compilation.
Use "python setup.py build_ext -I<inclpath> -L<libpath>"
where the path arguments point to your SSL installation.""")
include_dirs=config.ssl_include_dirs,
library_dirs=config.ssl_library_dirs,
libraries=config.libraries)]
def additional_things(self):
"""replace path names and program information in various files"""
self.announce("Filling template values.")
inst = self.get_command_obj("install")
inst.ensure_finalized()
data = {
'install_data': inst.install_data,
"install_scripts": change_root_inv(inst.root, inst.install_scripts),
"syspath": "# sys.path augmentation not needed",
'author': self.get_author(),
'version': self.get_version(),
'url': self.get_url(),
'appname': self.get_name(),
'email': self.get_author_email(),
'long_description': self.get_long_description(),
'python': os.path.normpath(sys.executable),
}
files = [
'linkchecker',
'linkchecker.bat',
'test/profiletest.py',
'linkcheck/__init__.py',
'linkcheck/Config.py',
'install.py',
'README',
]
for name in files:
t = Template(name+".tmpl")
f = open(name,"w")
f.write(t.fill_in(data))
f.close()
# append system specific files
if os.name=='nt':
if 'bdist_wininst' in self.commands or os.name=='nt':
# enabling windows files
self.scripts.append('linkchecker.bat')
elif os.name=='posix':
self.data_files.append(('/etc', ['linkcheckerrc']))
# for local run
os.chmod("linkchecker", 0755)
@ -146,6 +143,7 @@ o internationalization support
o (Fast)CGI web interface
""",
distclass = LCDistribution,
cmdclass = {'config': LCConfig, 'install': LCInstall},
packages = ['','DNS','linkcheck'],
scripts = ['linkchecker'],
data_files = [('share/locale/de/LC_MESSAGES',
@ -154,5 +152,6 @@ o (Fast)CGI web interface
('share/locale/fr/LC_MESSAGES',
['locale/fr/LC_MESSAGES/linkcheck.mo',
'locale/fr/LC_MESSAGES/linkcheck.po']),
('/etc', ['linkcheckerrc']),
],
)