#!/usr/bin/env perl
##!/usr/bin/perl -w
# latexdiff - differences two latex files on the word level
# and produces a latex file with the differences marked up.
#
# Copyright (C) 2004-12 F J Tilmann (tilmann@gfz-potsdam.de)
#
# Repository/issue tracker: https://github.com/ftilmann/latexdiff
# CTAN page: http://www.ctan.org/tex-archive/support/latexdiff
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program. If not, see .
#
# Detailed usage information at the end of the file
#
# ToDo:
#
# Version 1.1.1
# - patch mhchem: allow ce in equations
# - flatten now also expands \input etc. in the preamble (but not \usepackage!)
# - Better support for Japanese ( contributed by github user kshramt )
# - prevent duplicated verbatim hashes (patch contributed by github user therussianjig, issue #36)
# - disable deleted label commands (fixes issue #31)
# - introduce post-processing to reinstate most deleted environments and all needed item commands (fixes issue #1)
#
# Version 1.1.0
# - treat diacritics (\",\', etc) as safe commands
# - treat \_ and \& correctly as safe commands, even if used without spacing to the next word
# - Add a BOLD markup type that sets added text in bold face (Contribution by Victor Zabalza via pull request )
# - add append-mboxsafecmd list option to be able to specify special safe commands which need to be surrounded by mbox to avoid breaking (mostly this is needed with ulem package)
# - support for siunitx and cleveref packages: protect \SI command in siunitx package and \cref,\Cref{range}{*} in cleveref packages (thanks to Stefan Pinnow for testing)
# - experimental support for chemformula, mhchem packages: define \ch and \ce in packages as safe (but not \ch,\cee in equation array environments) - these unfortunately will not be marked up (thanks to Stefan Pinnow for testing)
# - bug fix: packages identified correctly even if \usepackage command options extend over several lines (previously \usepackage command needed to be fully contained in one line)
# - new subtype ONLYCHANGEDPAGE outputs only changed pages (might not work well for floating material)
# - new subtype ZLABEL operates similarly to LABEL but uses absolute page numbers (needs zref package)
# - undocumented option --debug/--nodebug to override default setting for debug mode (Default: 0 for release version, 1: for development version
#
# Version 1.0.4
# - introduce list UNSAFEMATHCMD, which holds list of commands which cannot be marked up with \DIFadd or \DIFdel commands (only relevant for WHOLE and COARSE math markup modes)
# - new subtype LABEL which gives each change a label. This can later be used to only display pages where changes
# have been made (instructions for that are put as comments into the diff'ed file) inspired by answer on http://tex.stackexchange.com/questions/166049/invisible-markers-in-pdfs-using-pdflatex
# - Configuration variables take into accout some commands from additional packages:
# tikzpicture environment now treated as PICTUREENV, and \smallmatrix in ARRENV (amsmath)
# - --flatten: support for \subfile command (subfiles package) (in response to http://tex.stackexchange.com/questions/167620/latexdiff-with-subfiles )
# - --flatten: \bibliography commands expand if corresponding bbl file present
# - angled bracket optional commands now parsed correctly (patch #3570) submitted by Dave Kleinschmidt (thanks)
# - \RequirePackage now treated as synonym of \usepackage with respect to setting packages
# - special rules for apacite package (redefine citation commands)
# - recognise /dev/null as 'file-like' arguments for --preamble and --config options
# - fix units package incompatibility with ulem for text maths statements $ ..$ (thanks to Stuart Prescott for reporting this)
# - amsmath environment cases treated correctly (Bug fix #19029) (thanks to Jalar)
# - {,} in comments no longer confuse latexdiff (Bug fix #19146)
# - \% in one-letter sub/Superscripts was not converted correctly
#
# Version 1.0.3
# - fix bug in add_safe_commands that made latexdiff hang on DeclareMathOperator
# command in preamble
# - \(..\) inline math expressions were not parsed correctly, if they contained a linebreak
# - applied patch contributed by tomflannaghan via Berlios: [ Patch #3431 ] Adds correct handling of \left< and \right>
# - \$ is treated correctly as a literal dollar sign (thanks to Reed Cartwright and Joshua Miller for reporting this bug
# and sketching out the solution)
# - \^ and \_ are correctly interpreted as accent and underlined space, respectively, not as superscript of subscript
# (thanks to Wail Yahyaoui for pointing out this bug)
#
# Version 1.0.1 - treat \big,\bigg etc. equivalently to \left and
# \right - include starred version in MATHENV - apply
# - flatten recursively and --flatten expansion is now
# aware of comments (thanks to Tim Connors for patch)
# - Change to post-processing for more reliability for
# deleted math environments
# - On linux systems, recognise and remove DOS style newlines
# - Provide markup for some special preamble commands (\title,
# \author,\date,
# - configurable by setting context2cmd
# - for styles using ulem package, remove \emph and \text.. from list of
# safe commands in order to allow linebreaks within the
# highlighted sections.
# - for ulem style, now show citations by enclosing them in \mbox commands.
# This unfortunately implies linebreaks within citations no longer function,
# so this functionality can be turned off (Option --disable-citation-markup).
# With --enable-citation-markup, the mbox markup is forced for other styles)
# - new substyle COLOR. This is particularly useful for marking up citations
# and some special post-processing is implemented to retain cite
# commands in deleted blocks.
# - four different levels of math-markup
# - Option --driver for choosing driver for modes employing changebar package
# - accept \\* as valid command (and other commands of form \.*). Also accept
# \ (backslashed newline)
# - some typo fixes, include commands defined in preamble as safe commands
# (Sebastian Gouezel)
# - include compared filenames as comments as line 2 and 3 of
# the preamble (can be modified with option --label, and suppressed with
# --no-label), option --visible-label to show files in generated pdf or dvi
# at the beginning of main document
#
# Version 0.5 A number of minor improvements based on feedback
# Deleted blocks are now shown before added blocks
# Package specific processing
#
# Version 0.43 unreleased typo in list of styles at the end
# Add protect to all \cbstart, \cbend commands
# More robust substitution of deleted math commands
#
# Version 0.42 November 06 Bug fixes only
#
# Version 0.4 March 06 option for fast differencing using UNIX diff command, several minor bug fixes (\par bug, improved highlighting of textcmds)
#
# Version 0.3 August 05 improved parsing of displayed math, --allow-spaces
# option, several minor bug fixes
#
# Version 0.25 October 04 Fix bug with deleted equations, add math mode commands to safecmd, add | to allowed interpunctuation signs
# Version 0.2 September 04 extension to utf-8 and variable encodings
# Version 0.1 August 04 First public release
# Inserted block for stand-alone version replaces
# use Algorithm::Diff qw(traverse_sequences);
# in standard version
# The following BEGIN block contains a verbatim copy of
# Ned Konz' Algorithm::Diff package version 1.15 except
# that all POD documentation has been stripped out.
# I encourage you to download and install the Algorithm::Diff
# package and use the standard latexdiff version instead
# (current distribution available from http://search.cpan.org/~nedkonz/Algorithm-Diff-1.15
# the most recent version can be found via http://search.cpan.org/search?module=Algorithm::Diff )
# Please note that the LICENCSE for Algorithm::Diff :
# "Copyright (c) 2000-2002 Ned Konz. All rights reserved.
# This program is free software;
# you can redistribute it and/or modify it under the same terms
# as Perl itself."
# The stand-alone version of latexdiff is provided as a convenience
# for latex users with no knowledge of PERL who do not wish to install
# additional packages to be able to use latexdiff. If you believe
# the inlining of Algorithm::Diff violates its license please contact
# me and I will modify the latexdiff distribution accordingly.
# Frederik Tilmann (tilmann@esc.cam.ac.uk)
BEGIN {
package Algorithm::Diff;
use strict;
use vars qw($VERSION @EXPORT_OK @ISA @EXPORT);
use integer; # see below in _replaceNextLargerWith() for mod to make
# if you don't use this
require Exporter;
@ISA = qw(Exporter);
@EXPORT = qw();
@EXPORT_OK = qw(LCS diff traverse_sequences traverse_balanced sdiff);
$VERSION = sprintf('%d.%02d so', (q$Revision: 1.15 $ =~ /\d+/g));
# McIlroy-Hunt diff algorithm
# Adapted from the Smalltalk code of Mario I. Wolczko,
# by Ned Konz, perl@bike-nomad.com
# Create a hash that maps each element of $aCollection to the set of positions
# it occupies in $aCollection, restricted to the elements within the range of
# indexes specified by $start and $end.
# The fourth parameter is a subroutine reference that will be called to
# generate a string to use as a key.
# Additional parameters, if any, will be passed to this subroutine.
#
# my $hashRef = _withPositionsOfInInterval( \@array, $start, $end, $keyGen );
sub _withPositionsOfInInterval
{
my $aCollection = shift; # array ref
my $start = shift;
my $end = shift;
my $keyGen = shift;
my %d;
my $index;
for ( $index = $start ; $index <= $end ; $index++ )
{
my $element = $aCollection->[$index];
my $key = &$keyGen( $element, @_ );
if ( exists( $d{$key} ) )
{
unshift ( @{ $d{$key} }, $index );
}
else
{
$d{$key} = [$index];
}
}
return wantarray ? %d : \%d;
}
# Find the place at which aValue would normally be inserted into the array. If
# that place is already occupied by aValue, do nothing, and return undef. If
# the place does not exist (i.e., it is off the end of the array), add it to
# the end, otherwise replace the element at that point with aValue.
# It is assumed that the array's values are numeric.
# This is where the bulk (75%) of the time is spent in this module, so try to
# make it fast!
sub _replaceNextLargerWith
{
my ( $array, $aValue, $high ) = @_;
$high ||= $#$array;
# off the end?
if ( $high == -1 || $aValue > $array->[-1] )
{
push ( @$array, $aValue );
return $high + 1;
}
# binary search for insertion point...
my $low = 0;
my $index;
my $found;
while ( $low <= $high )
{
$index = ( $high + $low ) / 2;
# $index = int(( $high + $low ) / 2); # without 'use integer'
$found = $array->[$index];
if ( $aValue == $found )
{
return undef;
}
elsif ( $aValue > $found )
{
$low = $index + 1;
}
else
{
$high = $index - 1;
}
}
# now insertion point is in $low.
$array->[$low] = $aValue; # overwrite next larger
return $low;
}
# This method computes the longest common subsequence in $a and $b.
# Result is array or ref, whose contents is such that
# $a->[ $i ] == $b->[ $result[ $i ] ]
# foreach $i in ( 0 .. $#result ) if $result[ $i ] is defined.
# An additional argument may be passed; this is a hash or key generating
# function that should return a string that uniquely identifies the given
# element. It should be the case that if the key is the same, the elements
# will compare the same. If this parameter is undef or missing, the key
# will be the element as a string.
# By default, comparisons will use "eq" and elements will be turned into keys
# using the default stringizing operator '""'.
# Additional parameters, if any, will be passed to the key generation routine.
sub _longestCommonSubsequence
{
my $a = shift; # array ref
my $b = shift; # array ref
my $keyGen = shift; # code ref
my $compare; # code ref
# set up code refs
# Note that these are optimized.
if ( !defined($keyGen) ) # optimize for strings
{
$keyGen = sub { $_[0] };
$compare = sub { my ( $a, $b ) = @_; $a eq $b };
}
else
{
$compare = sub {
my $a = shift;
my $b = shift;
&$keyGen( $a, @_ ) eq &$keyGen( $b, @_ );
};
}
my ( $aStart, $aFinish, $bStart, $bFinish, $matchVector ) =
( 0, $#$a, 0, $#$b, [] );
# First we prune off any common elements at the beginning
while ( $aStart <= $aFinish
and $bStart <= $bFinish
and &$compare( $a->[$aStart], $b->[$bStart], @_ ) )
{
$matchVector->[ $aStart++ ] = $bStart++;
}
# now the end
while ( $aStart <= $aFinish
and $bStart <= $bFinish
and &$compare( $a->[$aFinish], $b->[$bFinish], @_ ) )
{
$matchVector->[ $aFinish-- ] = $bFinish--;
}
# Now compute the equivalence classes of positions of elements
my $bMatches =
_withPositionsOfInInterval( $b, $bStart, $bFinish, $keyGen, @_ );
my $thresh = [];
my $links = [];
my ( $i, $ai, $j, $k );
for ( $i = $aStart ; $i <= $aFinish ; $i++ )
{
$ai = &$keyGen( $a->[$i], @_ );
if ( exists( $bMatches->{$ai} ) )
{
$k = 0;
for $j ( @{ $bMatches->{$ai} } )
{
# optimization: most of the time this will be true
if ( $k and $thresh->[$k] > $j and $thresh->[ $k - 1 ] < $j )
{
$thresh->[$k] = $j;
}
else
{
$k = _replaceNextLargerWith( $thresh, $j, $k );
}
# oddly, it's faster to always test this (CPU cache?).
if ( defined($k) )
{
$links->[$k] =
[ ( $k ? $links->[ $k - 1 ] : undef ), $i, $j ];
}
}
}
}
if (@$thresh)
{
for ( my $link = $links->[$#$thresh] ; $link ; $link = $link->[0] )
{
$matchVector->[ $link->[1] ] = $link->[2];
}
}
return wantarray ? @$matchVector : $matchVector;
}
sub traverse_sequences
{
my $a = shift; # array ref
my $b = shift; # array ref
my $callbacks = shift || {};
my $keyGen = shift;
my $matchCallback = $callbacks->{'MATCH'} || sub { };
my $discardACallback = $callbacks->{'DISCARD_A'} || sub { };
my $finishedACallback = $callbacks->{'A_FINISHED'};
my $discardBCallback = $callbacks->{'DISCARD_B'} || sub { };
my $finishedBCallback = $callbacks->{'B_FINISHED'};
my $matchVector = _longestCommonSubsequence( $a, $b, $keyGen, @_ );
# Process all the lines in @$matchVector
my $lastA = $#$a;
my $lastB = $#$b;
my $bi = 0;
my $ai;
for ( $ai = 0 ; $ai <= $#$matchVector ; $ai++ )
{
my $bLine = $matchVector->[$ai];
if ( defined($bLine) ) # matched
{
&$discardBCallback( $ai, $bi++, @_ ) while $bi < $bLine;
&$matchCallback( $ai, $bi++, @_ );
}
else
{
&$discardACallback( $ai, $bi, @_ );
}
}
# The last entry (if any) processed was a match.
# $ai and $bi point just past the last matching lines in their sequences.
while ( $ai <= $lastA or $bi <= $lastB )
{
# last A?
if ( $ai == $lastA + 1 and $bi <= $lastB )
{
if ( defined($finishedACallback) )
{
&$finishedACallback( $lastA, @_ );
$finishedACallback = undef;
}
else
{
&$discardBCallback( $ai, $bi++, @_ ) while $bi <= $lastB;
}
}
# last B?
if ( $bi == $lastB + 1 and $ai <= $lastA )
{
if ( defined($finishedBCallback) )
{
&$finishedBCallback( $lastB, @_ );
$finishedBCallback = undef;
}
else
{
&$discardACallback( $ai++, $bi, @_ ) while $ai <= $lastA;
}
}
&$discardACallback( $ai++, $bi, @_ ) if $ai <= $lastA;
&$discardBCallback( $ai, $bi++, @_ ) if $bi <= $lastB;
}
return 1;
}
sub traverse_balanced
{
my $a = shift; # array ref
my $b = shift; # array ref
my $callbacks = shift || {};
my $keyGen = shift;
my $matchCallback = $callbacks->{'MATCH'} || sub { };
my $discardACallback = $callbacks->{'DISCARD_A'} || sub { };
my $discardBCallback = $callbacks->{'DISCARD_B'} || sub { };
my $changeCallback = $callbacks->{'CHANGE'};
my $matchVector = _longestCommonSubsequence( $a, $b, $keyGen, @_ );
# Process all the lines in match vector
my $lastA = $#$a;
my $lastB = $#$b;
my $bi = 0;
my $ai = 0;
my $ma = -1;
my $mb;
while (1)
{
# Find next match indices $ma and $mb
do { $ma++ } while ( $ma <= $#$matchVector && !defined $matchVector->[$ma] );
last if $ma > $#$matchVector; # end of matchVector?
$mb = $matchVector->[$ma];
# Proceed with discard a/b or change events until
# next match
while ( $ai < $ma || $bi < $mb )
{
if ( $ai < $ma && $bi < $mb )
{
# Change
if ( defined $changeCallback )
{
&$changeCallback( $ai++, $bi++, @_ );
}
else
{
&$discardACallback( $ai++, $bi, @_ );
&$discardBCallback( $ai, $bi++, @_ );
}
}
elsif ( $ai < $ma )
{
&$discardACallback( $ai++, $bi, @_ );
}
else
{
# $bi < $mb
&$discardBCallback( $ai, $bi++, @_ );
}
}
# Match
&$matchCallback( $ai++, $bi++, @_ );
}
while ( $ai <= $lastA || $bi <= $lastB )
{
if ( $ai <= $lastA && $bi <= $lastB )
{
# Change
if ( defined $changeCallback )
{
&$changeCallback( $ai++, $bi++, @_ );
}
else
{
&$discardACallback( $ai++, $bi, @_ );
&$discardBCallback( $ai, $bi++, @_ );
}
}
elsif ( $ai <= $lastA )
{
&$discardACallback( $ai++, $bi, @_ );
}
else
{
# $bi <= $lastB
&$discardBCallback( $ai, $bi++, @_ );
}
}
return 1;
}
sub LCS
{
my $a = shift; # array ref
my $matchVector = _longestCommonSubsequence( $a, @_ );
my @retval;
my $i;
for ( $i = 0 ; $i <= $#$matchVector ; $i++ )
{
if ( defined( $matchVector->[$i] ) )
{
push ( @retval, $a->[$i] );
}
}
return wantarray ? @retval : \@retval;
}
sub diff
{
my $a = shift; # array ref
my $b = shift; # array ref
my $retval = [];
my $hunk = [];
my $discard = sub { push ( @$hunk, [ '-', $_[0], $a->[ $_[0] ] ] ) };
my $add = sub { push ( @$hunk, [ '+', $_[1], $b->[ $_[1] ] ] ) };
my $match = sub { push ( @$retval, $hunk ) if scalar(@$hunk); $hunk = [] };
traverse_sequences( $a, $b,
{ MATCH => $match, DISCARD_A => $discard, DISCARD_B => $add }, @_ );
&$match();
return wantarray ? @$retval : $retval;
}
sub sdiff
{
my $a = shift; # array ref
my $b = shift; # array ref
my $retval = [];
my $discard = sub { push ( @$retval, [ '-', $a->[ $_[0] ], "" ] ) };
my $add = sub { push ( @$retval, [ '+', "", $b->[ $_[1] ] ] ) };
my $change = sub {
push ( @$retval, [ 'c', $a->[ $_[0] ], $b->[ $_[1] ] ] );
};
my $match = sub {
push ( @$retval, [ 'u', $a->[ $_[0] ], $b->[ $_[1] ] ] );
};
traverse_balanced(
$a,
$b,
{
MATCH => $match,
DISCARD_A => $discard,
DISCARD_B => $add,
CHANGE => $change,
},
@_
);
return wantarray ? @$retval : $retval;
}
1;
}
import Algorithm::Diff qw(traverse_sequences);
# End of inserted block for stand-alone version
use Getopt::Long ;
use strict ;
use warnings;
use utf8 ;
my ($algodiffversion)=split(/ /,$Algorithm::Diff::VERSION);
my ($versionstring)=< 0,
WHOLE => 1,
COARSE => 2,
FINE => 3
};
my ($mboxcmd);
my (@configlist,@labels,
@appendsafelist,@excludesafelist,
@appendmboxsafelist,@excludemboxsafelist,
@appendtextlist,@excludetextlist,
@appendcontext1list,@appendcontext2list,
@packagelist);
my ($assign,@config);
# Hash where keys corresponds to the names of all included packages (including the documentclass as another package
# the optional arguments to the package are the values of the hash elements
my ($pkg,%packages);
# Defaults
$mathmarkup=COARSE;
$verbose=0;
# output debug and intermediate files, set to 0 in final distribution
$debug=0;
# insert preamble directly after documentclass - experimental feature, set to 0 in final distribution
# Note that this failed with mini example (or other files, where packages used in latexdiff preamble
# are called again with incompatible options in preamble of resulting file)
$earlylatexdiffpreamble=0;
# define character properties
sub IsNonAsciiPunct { return <<'END' # Unicode punctuation but excluding ASCII punctuation
+utf8::IsPunct
-utf8::IsASCII
END
}
sub IsNonAsciiS { return <<'END' # Unicode symbol but excluding ASCII
+utf8::IsS
-utf8::IsASCII
END
}
my %verbhash;
Getopt::Long::Configure('bundling');
GetOptions('type|t=s' => \$type,
'subtype|s=s' => \$subtype,
'floattype|f=s' => \$floattype,
'config|c=s' => \@configlist,
'preamble|p=s' => \$preamblefile,
'encoding|e=s' => \$encoding,
'label|L=s' => \@labels,
'no-label' => \$nolabel,
'visible-label' => \$visiblelabel,
'exclude-safecmd|A=s' => \@excludesafelist,
'replace-safecmd=s' => \$replacesafe,
'append-safecmd|a=s' => \@appendsafelist,
'exclude-textcmd|X=s' => \@excludetextlist,
'replace-textcmd=s' => \$replacetext,
'append-textcmd|x=s' => \@appendtextlist,
'replace-context1cmd=s' => \$replacecontext1,
'append-context1cmd=s' => \@appendcontext1list,
'replace-context2cmd=s' => \$replacecontext2,
'append-context2cmd=s' => \@appendcontext2list,
'exclude-mboxsafecmd=s' => \@excludemboxsafelist,
'append-mboxsafecmd=s' => \@appendmboxsafelist,
'show-preamble' => \$showpreamble,
'show-safecmd' => \$showsafe,
'show-textcmd' => \$showtext,
'show-config' => \$showconfig,
'show-all' => \$showall,
'packages=s' => \@packagelist,
'allow-spaces' => \$allowspaces,
'math-markup=s' => \$mathmarkup,
'enable-citation-markup|enforce-auto-mbox' => \$enablecitmark,
'disable-citation-markup|disable-auto-mbox' => \$disablecitmark,
'verbose|V' => \$verbose,
'ignore-warnings' => \$ignorewarnings,
'driver=s'=> \$driver,
'flatten' => \$flatten,
'version' => \$version,
'help|h|H' => \$help,
'debug!' => \$debug );
if ( $help ) {
usage() ;
}
if ( $version ) {
die $versionstring ;
}
print STDERR $versionstring if $verbose;
if (defined($showall)){
$showpreamble=$showsafe=$showtext=$showconfig=1;
}
# Default types
$type='UNDERLINE' unless defined($type);
$subtype='SAFE' unless defined($subtype);
# set floattype to IDENTICAL for LABEL and ONLYCHANGEDPAGE subtype, unless it has been set explicitly on the command line
$floattype=($subtype eq 'LABEL' || $subtype eq 'ONLYCHANGEDPAGE') ? 'IDENTICAL' : 'FLOATSAFE' unless defined($floattype);
if ( $subtype eq 'LABEL' ) {
print STDERR "Note that LABEL subtype is deprecated. If possible, use ZLABEL instead (requires zref package)";
}
if (defined($mathmarkup)) {
$mathmarkup=~tr/a-z/A-Z/;
if ( $mathmarkup eq 'OFF' ){
$mathmarkup=OFF;
} elsif ( $mathmarkup eq 'WHOLE' ){
$mathmarkup=WHOLE;
} elsif ( $mathmarkup eq 'COARSE' ){
$mathmarkup=COARSE;
} elsif ( $mathmarkup eq 'FINE' ){
$mathmarkup=FINE;
} elsif ( $mathmarkup !~ m/^[0123]$/ ) {
die "Illegal value: ($mathmarkup) for option--math-markup. Possible values: OFF,WHOLE,COARSE,FINE,0- ";
}
# else use numerical value
}
# setting extra preamble commands
if (defined($preamblefile)) {
$latexdiffpreamble=join "\n",(extrapream($preamblefile),"");
} else {
$latexdiffpreamble=join "\n",(extrapream($type,$subtype,$floattype),"");
}
if ( defined($driver) ) {
# for changebar only
$latexdiffpreamble=~s/\[dvips\]/[$driver]/sg;
}
# setting up @SAFECMDLIST and @SAFECMDEXCL
if (defined($replacesafe)) {
init_regex_arr_ext(\@SAFECMDLIST,$replacesafe);
} else {
init_regex_arr_data(\@SAFECMDLIST, "SAFE COMMANDS");
}
foreach $appendsafe ( @appendsafelist ) {
init_regex_arr_ext(\@SAFECMDLIST, $appendsafe);
}
foreach $excludesafe ( @excludesafelist ) {
init_regex_arr_ext(\@SAFECMDEXCL, $excludesafe);
}
# setting up @MBOXCMDLIST and @MBOXCMDEXCL
foreach $mboxsafe ( @appendmboxsafelist ) {
init_regex_arr_ext(\@MBOXCMDLIST, $mboxsafe);
}
foreach $mboxsafe ( @excludemboxsafelist ) {
init_regex_arr_ext(\@MBOXCMDEXCL, $mboxsafe);
}
# setting up @TEXTCMDLIST and @TEXTCMDEXCL
if (defined($replacetext)) {
init_regex_arr_ext(\@TEXTCMDLIST,$replacetext);
} else {
init_regex_arr_data(\@TEXTCMDLIST, "TEXT COMMANDS");
}
foreach $appendtext ( @appendtextlist ) {
init_regex_arr_ext(\@TEXTCMDLIST, $appendtext);
}
foreach $excludetext ( @excludetextlist ) {
init_regex_arr_ext(\@TEXTCMDEXCL, $excludetext);
}
# setting up @CONTEXT1CMDLIST ( @CONTEXT1CMDEXCL exist but is always empty )
if (defined($replacecontext1)) {
init_regex_arr_ext(\@CONTEXT1CMDLIST,$replacecontext1);
} else {
init_regex_arr_data(\@CONTEXT1CMDLIST, "CONTEXT1 COMMANDS");
}
foreach $appendcontext1 ( @appendcontext1list ) {
init_regex_arr_ext(\@CONTEXT1CMDLIST, $appendcontext1);
}
# setting up @CONTEXT2CMDLIST ( @CONTEXT2CMDEXCL exist but is always empty )
if (defined($replacecontext2)) {
init_regex_arr_ext(\@CONTEXT2CMDLIST,$replacecontext2);
} else {
init_regex_arr_data(\@CONTEXT2CMDLIST, "CONTEXT2 COMMANDS");
}
foreach $appendcontext2 ( @appendcontext2list ) {
init_regex_arr_ext(\@CONTEXT2CMDLIST, $appendcontext2);
}
# setting configuration variables
@config=();
foreach $config ( @configlist ) {
if (-f $config || lc $config eq '/dev/null' ) {
open(FILE,$config) or die ("Couldn't open configuration file $config: $!");
while () {
chomp;
next if /^\s*#/ || /^\s*%/ || /^\s*$/ ;
push (@config,$_);
}
close(FILE);
}
else {
# foreach ( split(",",$config) ) {
# push @config,$_;
# }
push @config,split(",",$config)
}
}
foreach $assign ( @config ) {
$assign=~ m/\s*(\w*)\s*=\s*(\S*)\s*$/ or die "Illegal assignment $assign in configuration list (must be variable=value)";
if ( $1 eq "MINWORDSBLOCK" ) { $MINWORDSBLOCK = $2; }
elsif ( $1 eq "FLOATENV" ) { $FLOATENV = $2 ; }
elsif ( $1 eq "ITEMCMD" ) { $ITEMCMD = $2 ; }
elsif ( $1 eq "LISTENV" ) { $LISTENV = $2 ; }
elsif ( $1 eq "PICTUREENV" ) { $PICTUREENV = $2 ; }
elsif ( $1 eq "MATHENV" ) { $MATHENV = $2 ; }
elsif ( $1 eq "MATHREPL" ) { $MATHREPL = $2 ; }
elsif ( $1 eq "MATHARRENV" ) { $MATHARRENV = $2 ; }
elsif ( $1 eq "MATHARRREPL" ) { $MATHARRREPL = $2 ; }
elsif ( $1 eq "ARRENV" ) { $ARRENV = $2 ; }
elsif ( $1 eq "COUNTERCMD" ) { $COUNTERCMD = $2 ; }
else { die "Unknown variable $1 in assignment.";}
}
if ( $mathmarkup == COARSE || $mathmarkup == WHOLE ) {
push(@MATHTEXTCMDLIST,qr/^MATHBLOCK(?:$MATHENV|$MATHARRENV|SQUAREBRACKET)$/);
}
foreach $pkg ( @packagelist ) {
map { $packages{$_}="" } split(/,/,$pkg) ;
}
if ($showpreamble) {
print "\nPreamble commands:\n";
print $latexdiffpreamble ;
}
if ($showsafe) {
print "\nCommands safe within scope of $ADDOPEN $ADDCLOSE and $DELOPEN $DELCLOSE (unless excluded):\n";
print_regex_arr(@SAFECMDLIST);
print "\nCommands not safe within scope of $ADDOPEN $ADDCLOSE and $DELOPEN $DELCLOSE :\n";
print_regex_arr(@SAFECMDEXCL);
}
if ($showtext) {
print "\nCommands with last argument textual (unless excluded) and safe in every context:\n";
print_regex_arr(@TEXTCMDLIST);
print "\nContext1 commands (last argument textual, command will be disabled in deleted passages, last argument will be shown as plain text):\n";
print_regex_arr(@CONTEXT1CMDLIST);
print "\nContext2 commands (last argument textual, command and its argument will be disabled in deleted passages):\n";
print_regex_arr(@CONTEXT2CMDLIST);
print "\nExclude list of Commands with last argument not textual (overrides patterns above):\n";
print_regex_arr(@TEXTCMDEXCL);
}
if ($showconfig) {
print "Configuration variables:\n";
print "ARRENV=$ARRENV\n";
print "COUNTERCMD=$COUNTERCMD\n";
print "FLOATENV=$FLOATENV\n";
print "ITEMCMD=$ITEMCMD\n";
print "LISTENV=$LISTENV\n";
print "MATHARRENV=$MATHARRENV\n";
print "MATHARRREPL=$MATHARRREPL\n";
print "MATHENV=$MATHENV\n";
print "MATHREPL=$MATHREPL\n";
print "MINWORDSBLOCK=$MINWORDSBLOCK\n";
print "PICTUREENV=$PICTUREENV\n";
}
if ($showconfig || $showtext || $showsafe || $showpreamble) {
exit 0; }
if ( @ARGV != 2 ) {
print STDERR "2 and only 2 non-option arguments required. Write latexdiff -h to get help\n";
exit(2);
}
# Are extra spaces between command arguments permissible?
my $extraspace;
if ($allowspaces) {
$extraspace='\s*';
} else {
$extraspace='';
}
# append context lists to text lists (as text property is implied)
push @TEXTCMDLIST, @CONTEXT1CMDLIST;
push @TEXTCMDLIST, @CONTEXT2CMDLIST;
push @TEXTCMDLIST, @MATHTEXTCMDLIST if $mathmarkup==COARSE;
# internal additions to SAFECMDLIST
push(@SAFECMDLIST, qr/^QLEFTBRACE$/, qr/^QRIGHTBRACE$/);
# Patterns. These are used by some of the subroutines, too
# I can only define them down here because value of extraspace depends on an option
my $pat0 = '(?:[^{}])*';
my $pat_n = $pat0;
# if you get "undefined control sequence MATHBLOCKmath" error, increase the maximum value in this loop
for (my $i_pat = 0; $i_pat < 20; ++$i_pat){
$pat_n = '(?:[^{}]|\{'.$pat_n.'\})*';
}
my $brat0 = '(?:[^\[\]]|\\\[|\\\])*';
my $abrat0 = '(?:[^<>])*';
my $quotemarks = '(?:\'\')|(?:\`\`)';
my $punct='[0.,\/\'\`:;\"\?\(\)\[\]!~\p{IsNonAsciiPunct}\p{IsNonAsciiS}]';
my $number='-?\d*\.\d*';
my $mathpunct='[+=<>\-\|]';
my $and = '&';
my $coords= '[\-.,\s\d]*';
# quoted underscore - this needs special treatment as perl treats _ as a letter (\w) but latex does not
# such that a\_b is interpreted as a{\_}b by latex but a{\_b} by perl
my $quotedunderscore='\\\\_';
# word: sequence of letters or accents followed by letter
my $word_ja='\p{Han}+|\p{InHiragana}+|\p{InKatakana}+';
my $word='(?:' . $word_ja . '|(?:(?:[-\w\d*]|\\\\[\"\'\`~^][A-Za-z\*])(?!(?:' . $word_ja . ')))+)';
my $cmdleftright='\\\\(?:left|right|[Bb]igg?[lrm]?|middle)\s*(?:[<>()\[\]|]|\\\\(?:[|{}]|\w+))';
my $cmdoptseq='\\\\[\w\d\*]+'.$extraspace.'(?:(?:<'.$abrat0.'>|\['.$brat0.'\]|\{'. $pat_n . '\}|\(' . $coords .'\))'.$extraspace.')*';
my $backslashnl='\\\\\n';
my $oneletcmd='\\\\.\*?(?:\['.$brat0.'\]|\{'. $pat_n . '\})*';
my $math='\$(?:[^$]|\\\$)*?\$|\\\\[(](?:.|\n)*?\\\\[)]';
## the current maths command cannot cope with newline within the math expression
my $comment='%.*?\n';
my $pat=qr/(?:\A\s*)?(?:${and}|${quotemarks}|${number}|${word}|$quotedunderscore|$cmdleftright|${cmdoptseq}|${math}|${backslashnl}|${oneletcmd}|${comment}|${punct}|${mathpunct}|\{|\})\s*/ ;
# now we are done setting up and can start working
my ($oldfile, $newfile) = @ARGV;
# check for existence of input files
if ( ! -e $oldfile ) {
die "Input file $oldfile does not exist";
}
if ( ! -e $newfile ) {
die "Input file $newfile does not exist";
}
# set the labels to be included into the file
my ($oldtime,$newtime,$oldlabel,$newlabel);
if (defined($labels[0])) {
$oldlabel=$labels[0] ;
} else {
$oldtime=localtime((stat($oldfile))[9]);
$oldlabel="$oldfile " . " "x(length($newfile)-length($oldfile)) . $oldtime;
}
if (defined($labels[1])) {
$newlabel=$labels[1] ;
} else {
$newtime=localtime((stat($newfile))[9]);
$newlabel="$newfile " . " "x(length($oldfile)-length($newfile)) . $newtime;
}
$encoding=guess_encoding($newfile) unless defined($encoding);
$encoding = "utf8" if $encoding =~ m/^utf8/i ;
if (lc($encoding) eq "utf8" ) {
binmode(STDOUT, ":utf8");
binmode(STDERR, ":utf8");
}
$old=read_file_with_encoding($oldfile,$encoding);
$new=read_file_with_encoding($newfile,$encoding);
# reset time
exetime(1);
($oldpreamble,$oldbody,$oldpost)=splitdoc($old,'\\\\begin\{document\}','\\\\end\{document\}');
($newpreamble,$newbody,$newpost)=splitdoc($new,'\\\\begin\{document\}','\\\\end\{document\}');
if ($flatten) {
$oldbody=flatten($oldbody,$oldpreamble,$oldfile,$encoding);
$newbody=flatten($newbody,$newpreamble,$newfile,$encoding);
# flatten preamble
$oldpreamble=flatten($oldpreamble,$oldpreamble,$oldfile,$encoding);
$newpreamble=flatten($newpreamble,$newpreamble,$newfile,$encoding);
}
my @auxlines;
# boolean variab
my ($ulem)=0;
if ( length $oldpreamble && length $newpreamble ) {
# pre-process preamble by looking for commands used in \maketitle (title, author, date etc commands)
# and marking up content with latexdiff markup
@auxlines=preprocess_preamble($oldpreamble,$newpreamble);
@oldpreamble = split /\n/, $oldpreamble;
@newpreamble = split /\n/, $newpreamble;
# If a command is defined in the preamble of the new file, and only uses safe commands, then it can be considered to be safe) (contribution S. Gouezel)
# Base this assessment on the new preamble
add_safe_commands($newpreamble);
# get a list of packages from preamble if not predefine
%packages=list_packages($newpreamble) unless %packages;
if ( %packages && $debug ) { my $key ; foreach $key (keys %packages) { print STDERR "DEBUG \\usepackage[",$packages{$key},"]{",$key,"}\n" ;} }
if (defined $packages{"hyperref"} ) {
# deleted lines should not generate or appear in link names:
print STDERR "hyperref package detected.\n" if $verbose ;
$latexdiffpreamble =~ s/\{\\DIFadd\}/{\\DIFaddtex}/g;
$latexdiffpreamble =~ s/\{\\DIFdel\}/{\\DIFdeltex}/g;
$latexdiffpreamble .= join "\n",(extrapream("HYPERREF"),"");
### $latexdiffpreamble .= '%DIF PREAMBLE EXTENSION ADDED BY LATEXDIFF FOR HYPERREF PACKAGE' . "\n";
### $latexdiffpreamble .= '\providecommand{\DIFadd}[1]{\texorpdfstring{\DIFaddtex{#1}}{#1}}' . "\n";
### $latexdiffpreamble .= '\providecommand{\DIFdel}[1]{\texorpdfstring{\DIFdeltex{#1}}{}}' . "\n";
### $latexdiffpreamble .= '%DIF END PREAMBLE EXTENSION ADDED BY LATEXDIFF FOR HYPERREF PACKAGE' . "\n";
}
print STDERR "Differencing preamble.\n" if $verbose;
# insert dummy first line such that line count begins with line 1 (rather than perl's line 0) - just so that line numbers inserted by linediff are correct
unshift @newpreamble,'';
unshift @oldpreamble,'';
@diffpreamble = linediff(\@oldpreamble, \@newpreamble);
# remove dummy line again
shift @diffpreamble;
# add filenames, modification time and latexdiff mark
defined($nolabel) or splice @diffpreamble,1,0,
"%DIF LATEXDIFF DIFFERENCE FILE",
,"%DIF DEL $oldlabel",
"%DIF ADD $newlabel";
if ( @auxlines ) {
push @diffpreamble,"%DIF DELETED TITLE COMMANDS FOR MARKUP";
push @diffpreamble,join("\n",@auxlines);
}
if ( $earlylatexdiffpreamble) {
# insert latexdiff command directly after documentclass at beginning of preamble
# note that grep is only run for its side effect
( grep { s/^([^%]*\\documentclass.*)$/$1$latexdiffpreamble/ } @diffpreamble )==1 or die "Could not find documentclass statement in preamble";
} else {
# insert latexdiff commands at the end of preamble (default behaviour)
push @diffpreamble,$latexdiffpreamble;
}
push @diffpreamble,'\begin{document}';
}
elsif ( !length $oldpreamble && !length $newpreamble ) {
@diffpreamble=();
} else {
print STDERR "Either both texts must have preamble or neither text must have the preamble.\n";
exit(2);
}
# Special: treat all cite commands as safe except in UNDERLINE and FONTSTRIKE mode
# (there is a conflict between citation and ulem package, see
# package documentation)
# Use post-processing
# and $packages{"apacite"}!~/natbibpapa/
$ulem = ($latexdiffpreamble =~ /\\RequirePackage(?:\[$brat0\])?\{ulem\}/ || defined $packages{"ulem"});
if (defined $packages{"units"} && $ulem ) {
# protect inlined maths environments by surrounding with an \mbox
# this is done to get around an incompatibility between the ulem and units package
# where spaces in the argument to underlined or crossed-out \unit commands cause an error message
print STDERR "units package detected at the same time as style using ulem.\n" if $verbose ;
$MBOXINLINEMATH=1;
}
if (defined $packages{"siunitx"} ) {
# protect SI command by surrounding them with an \mbox
# this is done to get around an incompatibility between the ulem and siunitx package
print STDERR "siunitx package detected.\n" if $verbose ;
my $mboxcmds='SI,ang,numlist,numrange,SIlist,SIrange';
init_regex_arr_ext(\@SAFECMDLIST,'num,si');
if ( $enablecitmark || ( $ulem && ! $disablecitmark )) {
init_regex_arr_ext(\@MBOXCMDLIST,$mboxcmds);
} else {
init_regex_arr_ext(\@SAFECMDLIST,$mboxcmds);
}
}
if (defined $packages{"cleveref"} ) {
# protect selected command by surrounding them with an \mbox
# this is done to get around an incompatibility between ulem and cleveref package
print STDERR "cleveref package detected.\n" if $verbose ;
my $mboxcmds='[Cc]ref(?:range)?\*?,labelcref,(?:lc)?name[cC]refs?' ;
if ( $enablecitmark || ( $ulem && ! $disablecitmark )) {
init_regex_arr_ext(\@MBOXCMDLIST,$mboxcmds);
} else {
init_regex_arr_ext(\@SAFECMDLIST,$mboxcmds);
}
}
if (defined $packages{"glossaries"} ) {
# protect selected command by surrounding them with an \mbox
# this is done to get around an incompatibility between ulem and glossaries package
print STDERR "glossaries package detected.\n" if $verbose ;
my $mboxcmds='[gG][lL][sS](?:|pl|disp|link|first|firstplural|desc|user[iv][iv]?[iv]?),[aA][cC][rR](?:long|longpl|full|fullpl),[aA][cC][lfp]?[lfp]?';
init_regex_arr_ext(\@SAFECMDLIST,'[gG][lL][sS](?:(?:entry)?(?:text|plural|name|symbol)|displaynumberlist|entryfirst|entryfirstplural|entrydesc|entrydescplural|entrysymbolplural|entryuser[iv][iv]?[iv]?|entrynumberlist|entrydisplaynumberlist|entrylong|entrylongpl|entryshort|entryshortpl|entryfull|entryfullpl),[gG]lossentry(?:name|desc|symbol),[aA][cC][rR](?:short|shortpl),[aA]csp?');
if ( $enablecitmark || ( $ulem && ! $disablecitmark )) {
init_regex_arr_ext(\@MBOXCMDLIST,$mboxcmds);
} else {
init_regex_arr_ext(\@SAFECMDLIST,$mboxcmds);
}
}
if (defined $packages{"chemformula"} or defined $packages{"chemmacros"} ) {
print STDERR "chemformula package detected.\n" if $verbose ;
init_regex_arr_ext(\@SAFECMDLIST,'ch');
push(@UNSAFEMATHCMD,'ch');
# The next command would be needed to allow highlighting the interior of \ch commands in math environments
# but the redefinitions in chemformula are too deep to make this viable
# push(@MATHTEXTCMDLIST,'ch');
}
if (defined $packages{"mhchem"} ) {
print STDERR "mhchem package detected.\n" if $verbose ;
init_regex_arr_ext(\@SAFECMDLIST,'ce');
push(@UNSAFEMATHCMD,'ce','cee');
# The next command would be needed to allow highlighting the interior of \cee commands in math environments
# but the redefinitions in chemformula are too deep to make this viable
# push(@MATHTEXTCMDLIST,'cee');
}
my ( $citpat);
if ( defined $packages{"apacite"} ) {
print STDERR "apacite package detected.\n" if $verbose ;
$citpat='(?:mask)?(?:full|short|no)?cite(?:A|author|year|meta)?(?:NP)?';
} else {
# citation command pattern for all other citation schemes
$citpat='(?:cite\w*|nocite)';
};
if ( ! $ulem ) {
# modes not using ulem: citation is safe
push (@SAFECMDLIST, $citpat);
} else {
### Experimental: disable text and emph commands
push(@SAFECMDEXCL, qr/^emph$/, qr/^text..$/);
# replace \cite{..} by \mbox{\cite{..}} in added or deleted blocks in post-processing
push(@MBOXCMDLIST,$citpat) unless $disablecitmark;
if ( uc($subtype) eq "COLOR" or uc($subtype) eq "DVIPSCOL" ) {
# remove \cite command again from list of safe commands
pop @MBOXCMDLIST;
# deleted cite commands
}
}
push(@MBOXCMDLIST,$citpat) if $enablecitmark ;
if (defined $packages{"amsmath"} or defined $packages{"amsart"} or defined $packages{"amsbook"} ) {
print STDERR "amsmath package detected.\n" if $verbose ;
$MATHARRREPL='align*';
}
# add commands in MBOXCMDLIST to SAFECMDLIST
foreach $mboxcmd ( @MBOXCMDLIST ) {
init_regex_arr_ext(\@SAFECMDLIST, $mboxcmd);
}
# check if \label is in SAFECMDLIST, and if yes replace "label" in $LABELCMD by something that never matches (we hope!)
if ( iscmd("label",\@SAFECMDLIST,\@SAFECMDEXCL) ) {
$LABELCMD=~ s/label/NEVERMATCHLABEL/;
}
print STDERR "Preprocessing body. " if $verbose;
preprocess($oldbody,$newbody);
# run difference algorithm
@diffbody=bodydiff($oldbody, $newbody);
$diffbo=join("",@diffbody);
if ( $debug ) {
open(RAWDIFF,">","latexdiff.debug.bodydiff");
print RAWDIFF $diffbo;
close(RAWDIFF);
}
print STDERR "(",exetime()," s)\n","Postprocessing body. \n" if $verbose;
postprocess($diffbo);
$diffall =join("\n",@diffpreamble) ;
# add visible labels
if (defined($visiblelabel)) {
# Give information right after \begin{document} (or at the beginning of the text for files without preamble
### if \date command is used, add information to \date argument, otherwise give right after \begin{document}
### $diffall=~s/(\\date$extraspace(?:\[$brat0\])?$extraspace)\{($pat_n)\}/$1\{$2 \\ LATEXDIFF comparison \\ Old: $oldlabel \\ New: $newlabel \}/ or
$diffbo = "\\begin{verbatim}LATEXDIFF comparison\nOld: $oldlabel\nNew: $newlabel\\end{verbatim}\n$diffbo" ;
}
$diffall .= "$diffbo" ;
$diffall .= "\\end{document}$newpost" if length $newpreamble ;
if ( lc($encoding) ne "utf8" && lc($encoding) ne "ascii" ) {
print STDERR "Encoding output file to $encoding\n" if $verbose;
$diffall=Encode::encode($encoding,$diffall);
binmode STDOUT;
}
print $diffall;
print STDERR "(",exetime()," s)\n","Done.\n" if $verbose;
## guess_encoding(filename)
## reads the first 20 lines of filename and looks for call of inputenc package
## if found, return the option of this package (encoding), otherwise return ascii
sub guess_encoding {
my ($filename)=@_;
my ($i,$enc);
open (FH, $filename) or die("Couldn't open $filename: $!");
$i=0;
while () {
next if /^\s*%/; # skip comment lines
if (m/\\usepackage\[(\w*?)\]\{inputenc\}/) {
close(FH);
return($1);
}
last if (++$i > 20 ); # scan at most 20 non-comment lines
}
close(FH);
return("ascii");
}
sub read_file_with_encoding {
my ($output);
my ($filename, $encoding) = @_;
if (lc($encoding) eq "utf8" ) {
open (FILE, "<:utf8",$filename) or die("Couldn't open $filename: $!");
local $/ ; # locally set record operator to undefined, ie. enable whole-file mode
$output=;
} elsif ( lc($encoding) eq "ascii") {
open (FILE, $filename) or die("Couldn't open $filename: $!");
local $/ ; # locally set record operator to undefined, ie. enable whole-file mode
$output=;
} else {
require Encode;
open (FILE, "<",$filename) or die("Couldn't open $filename: $!");
local $/ ; # locally set record operator to undefined, ie. enable whole-file mode
$output=;
print STDERR "Converting $filename from $encoding to utf8\n" if $verbose;
$output=Encode::decode($encoding,$output);
}
close FILE;
if ($^O eq "linux" ) {
$output =~ s/\r\n/\n/g ;
}
return $output;
}
## %packages=list_packages(@preamble)
## scans the arguments for \documentclass,\RequirePackage and \usepackage statements and constructs a hash
## whose keys are the included packages, and whose values are the associated optional arguments
#sub list_packages {
# my (@preamble)=@_;
# my %packages=();
# foreach $line ( @preamble ) {
# # get rid of comments
# $line=~s/(?catfile($dirname,$fname), "\n" if $debug;
# content of file becomes replacement value (use recursion), add \newpage if the command was include
###$replacement=read_file_with_encoding(File::Spec->catfile($dirname,$fname), $encoding) or die "Couldn't find file ",File::Spec->catfile($dirname,$fname),": $!";
$replacement=flatten(read_file_with_encoding(File::Spec->catfile($dirname,$fname), $encoding), $preamble,$filename,$encoding) or die "Couldn't find file ",File::Spec->catfile($dirname,$fname),": $!";
# \include always starts a new page; use explicit \newpage command to simulate this
$newpage=(defined($3)? " \\newpage " : "") ;
"$begline$newpage$replacement$newpage";
}/exgm;
# replace bibliography with bbl file if it exists
$text=~s/(^(?:[^%\n]|\\%)*)\\bibliography{(.*?)}/{
if ( -f $bblfile ){
$replacement=read_file_with_encoding(File::Spec->catfile($bblfile), $encoding);
} else {
warn "Bibliography file $bblfile cannot be found. No flattening of \\bibliography done. Run bibtex on old and new files first";
$replacement="\\bibliography{$2}";
}
$begline=(defined($1)? $1 : "") ;
"$begline$replacement";
}/exgm;
# replace subfile with contents (subfile package)
$text=~s/(^(?:[^%\n]|\\%)*)\\subfile{(.*?)}/{
$begline=(defined($1)? $1 : "") ;
$fname = $2;
# # add tex extension unless there is a three letter extension already
$fname .= ".tex" unless $fname =~ m|\.\w{3}|;
print STDERR "Include file as subfile $fname\n" if $verbose;
# content of file becomes replacement value (use recursion)
# now strip away everything outside and including \begin{document} and \end{document} pair#
# # note: no checking for comments is made
$subfile=read_file_with_encoding(File::Spec->catfile($dirname,$fname), $encoding) or die "Couldn't find file ",File::Spec->catfile($dirname,$fname),": $!";
($subpreamble,$subbody,$subpost)=splitdoc($subfile,'\\\\begin\{document\}','\\\\end\{document\}');
$replacement=flatten($subbody, $preamble,$filename,$encoding);
"$begline$replacement";
}/exgm;
return($text);
}
# print_regex_arr(@arr)
# prints regex array without x-ism expansion put in by pearl to stdout
sub print_regex_arr {
my $dumstring;
$dumstring = join(" ",@_); # PERL generates string (?-xism:^ref$) for quoted refex ^ref$
$dumstring =~ s/\(\?-xism:\^(.*?)\$\)/$1/g; # remove string and ^,$ marks before output
print $dumstring,"\n";
}
# @lines=extrapream($type)
# reads line from appendix (end of file after __END__ token)
sub extrapream {
my $type;
my @retval=("%DIF PREAMBLE EXTENSION ADDED BY LATEXDIFF") ;
my ($copy);
while (@_) {
$copy=0;
$type=shift ;
if ( -f $type || lc $type eq '/dev/null' ) {
open (FILE,$type) or die "Cannot open preamble file $type: $!";
print STDERR "Reading preamble file $type\n" if $verbose ;
while () {
chomp ;
if ( $_ =~ m/%DIF PREAMBLE/ ) {
push (@retval,"$_");
} else {
push (@retval,"$_ %DIF PREAMBLE");
}
}
}
else { # not (-f $type)
$type=uc($type); # upcase argument
print STDERR "Preamble Internal Type $type\n" if $verbose;
while () {
if ( m/^%DIF $type/ ) {
$copy=1; }
elsif ( m/^%DIF END $type/ ) {
last; }
chomp;
push (@retval,"$_ %DIF PREAMBLE") if $copy;
}
if ( $copy == 0 ) {
print STDERR "\nPreamble style $type not implemented.\n";
print STDERR "Write latexdiff -h to get help with available styles\n";
exit(2);
}
seek DATA,0,0; # rewind DATA handle to file begin
}
}
push (@retval,"%DIF END PREAMBLE EXTENSION ADDED BY LATEXDIFF") ;
return @retval;
}
# ($part1,$part2,$part3)=splitdoc($text,$word1,$word2)
# splits $text into 3 parts at $word1 and $word2.
# if neither $word1 nor $word2 exist, $part1 and $part3 are empty, $part2 is $text
# If only $word1 or $word2 exist but not the other, output an error message.
# NB this version avoids $` and $' for performance reason although it only makes a tiny difference
# (in one test gain a tenth of a second for a 30s run)
sub splitdoc {
my ($text,$word1,$word2)=@_;
my ($part1,$part2,$part3)=("","","");
my ($rest,$pos);
if ( $text =~ m/(^[^%]*)($word1)/mg ) {
$pos=pos $text;
$part1=substr($text,0,$pos-length($2));
$rest=substr($text,$pos);
if ( $rest =~ m/(^[^%]*)($word2)/mg ) {
$pos=pos $rest;
$part2=substr($rest,0,$pos-length($2));
$part3=substr($rest,$pos);
}
else {
die "$word1 and $word2 not in the correct order or not present as a pair." ;
}
} else {
$part2=$text;
die "$word2 present but not $word1." if ( $text =~ m/(^[^%]*)$word2/ms );
}
return ($part1,$part2,$part3);
}
# bodydiff($old,$new)
sub bodydiff {
my ($oldwords, $newwords) = @_;
my @retwords;
print STDERR "(",exetime()," s)\n","Splitting into latex tokens \n" if $verbose;
print STDERR "Parsing $oldfile \n" if $verbose;
my @oldwords = splitlatex($oldwords);
print STDERR "Parsing $newfile \n" if $verbose;
my @newwords = splitlatex($newwords);
if ( $debug ) {
open(TOKENOLD,">","latexdiff.debug.tokenold");
print TOKENOLD join("***\n",@oldwords);
close(TOKENOLD);
open(TOKENNEW,">","latexdiff.debug.tokennew");
print TOKENNEW join("***\n",@newwords);
close(TOKENNEW);
}
print STDERR "(",exetime()," s)\n","Pass 1: Expanding text commands and merging isolated identities with changed blocks " if $verbose;
pass1(\@oldwords, \@newwords);
print STDERR "(",exetime()," s)\n","Pass 2: inserting DIF tokens and mark up. " if $verbose;
if ( $debug ) {
open(TOKENOLD,">","latexdiff.debug.tokenold2");
print TOKENOLD join("***\n",@oldwords);
close(TOKENOLD);
open(TOKENNEW,">","latexdiff.debug.tokennew2");
print TOKENNEW join("***\n",@newwords);
close(TOKENNEW);
}
@retwords=pass2(\@oldwords, \@newwords);
return(@retwords);
}
# @words=splitlatex($string)
# split string according to latex rules
# Each element of words is either
# a word (including trailing spaces and punctuation)
# a latex command
# if there is white space in the beginning return that as first token
sub splitlatex {
my ($inputstring) = @_ ;
my $string=$inputstring ;
# if input is empty, return empty list
length($string)>0 or return ();
$string=~s/^(\s*)//s;
my $leadin=$1;
length($string)>0 or return ($leadin);
my @retval=($string =~ m/$pat/osg);
if (length($string) != length(join("",@retval))) {
print STDERR "\nWARNING: Inconsistency in length of input string and parsed string:\n This often indicates faulty or non-standard latex code.\n In many cases you can ignore this and the following warning messages.\n Note that character numbers in the following are counted beginning after \\begin{document} and are only approximate." unless $ignorewarnings;
print STDERR "DEBUG Original length ",length($string)," Parsed length ",length(join("",@retval)),"\n" if $debug;
print STDERR "DEBUG Input string: |$string|\n" if (length($string)<500) && $debug;
print STDERR "DEBUG Token parsing: |",join("+",@retval),"|\n" if (length($string)<500) && $debug ;
@retval=();
# slow way only do this if other m//sg method fails
my $last = 0;
while ( $string =~ m/$pat/osg ) {
my $match=$&;
if ($last + length $& != pos $string ) {
my $pos=pos($string);
my $offset=30<$last ? 30 : $last;
my $dum=substr($string,$last-$offset,$pos-$last+2*$offset);
my $dum1=$dum;
my $cnt=$#retval;
my $i;
$dum1 =~ s/\n/ /g;
unless ($ignorewarnings) {
print STDERR "\n$dum1\n";
print STDERR " " x 30,"^" x ($pos-$last)," " x 30,"\n";
print STDERR "Missing characters near word " . (scalar @retval) . " character index: " . $last . "-" . pos($string) . " Length: " . length($match) . " Match: |$match| (expected match marked above).\n";
}
# put in missing characters `by hand'
push (@retval, substr($dum,$offset,$pos-$last-length($match)));
# Note: there seems to be a bug in substr with utf8 that made the following line output substr which were too long,
# using dum instead appears to work
# push (@retval, substr($string,$last, pos($string)-$last-length($match)));
}
push (@retval, $match);
$last=pos $string;
}
}
unshift(@retval,$leadin) if (length($leadin)>0);
return @retval;
}
# pass1( \@seq1,\@seq2)
# Look for differences between seq1 and seq2.
# Where an common-subsequence block is flanked by deleted or appended blocks,
# and is shorter than $MINWORDSBLOCK words it is appended
# to the last deleted or appended word. If the block contains tokens other than words
# or punctuation it is not merged.
# Deleted or appended block consisting of words and safe commands only are
# also merged, to prevent break-up in pass2 (after previous isolated words have been removed)
# If there are commands with textual arguments (e.g. \caption) both in corresponding
# appended and deleted blocks split them such that the command and opening bracket
# are one token, then the rest is split up following standard rules, and the closing
# bracket is a separate token, ie. turn
# "\caption{This is a textual argument}" into
# ("\caption{","This ","is ","a ","textual ","argument","}")
# No return value. Destructively changes sequences
sub pass1 {
my $seq1 = shift ;
my $seq2 = shift ;
my $len1 = scalar @$seq1;
my $len2 = scalar @$seq2;
my $wpat=qr/^(?:[a-zA-Z.,'`:;?()!]*)[\s~]*$/; #'
my ($last1,$last2)=(-1,-1) ;
my $cnt=0;
my $block=[];
my $addblock=[];
my $delblock=[];
my $todo=[];
my $instruction=[];
my $i;
my (@delmid,@addmid,@dummy);
my ($addcmds,$delcmds,$matchindex);
my ($addtextblocks,$deltextblocks);
my ($addtokcnt,$deltokcnt,$mattokcnt)=(0,0,0);
my ($addblkcnt,$delblkcnt,$matblkcnt)=(0,0,0);
my $adddiscard = sub {
if ($cnt > 0 ) {
$matblkcnt++;
# just after an unchanged block
# print STDERR "Unchanged block $cnt, $last1,$last2 \n";
if ($cnt < $MINWORDSBLOCK
&& $cnt==scalar (
grep { /^$wpat/ || ( /^\\((?:[`'^"~=.]|[\w\d@*]+))((?:\[$brat0\]|\{$pat_n\})*)/o
&& iscmd($1,\@SAFECMDLIST,\@SAFECMDEXCL)
&& scalar(@dummy=split(" ",$2))<3 ) }
@$block) ) {
# merge identical blocks shorter than $MINWORDSBLOCK
# and only containing ordinary words
# with preceding different word
# We cannot carry out this merging immediately as this
# would change the index numbers of seq1 and seq2 and confuse
# the algorithm, instead we store in @$todo where we have to merge
push(@$todo, [ $last1,$last2,$cnt,@$block ]);
}
$block = [];
$cnt=0; $last1=-1; $last2=-1;
}
};
my $discard=sub { $deltokcnt++;
&$adddiscard; #($_[0],$_[1]);
push(@$delblock,[ $seq1->[$_[0]],$_[0] ]);
$last1=$_[0] };
my $add = sub { $addtokcnt++;
&$adddiscard; #($_[0],$_[1]);
push(@$addblock,[ $seq2->[$_[1]],$_[1] ]);
$last2=$_[1] };
my $match = sub { $mattokcnt++;
if ($cnt==0) { # first word of matching sequence after changed sequence or at beginning of word sequence
$deltextblocks = extracttextblocks($delblock);
$delblkcnt++ if scalar @$delblock;
$addtextblocks = extracttextblocks($addblock);
$addblkcnt++ if scalar @$addblock;
$delcmds = extractcommands($delblock);
$addcmds = extractcommands($addblock);
# keygen(third argument of _longestCommonSubsequence) implies to sort on command (0th elements of $addcmd elements)
# the calling format for longestCommonSubsequence has changed between versions of
# Algorithm::Diff so we need to check which one we are using
if ( $algodiffversion > 1.15 ) {
### Algorithm::Diff 1.19
$matchindex=Algorithm::Diff::_longestCommonSubsequence($delcmds,$addcmds, 0, sub { $_[0]->[0] } );
} else {
### Algorithm::Diff 1.15
$matchindex=Algorithm::Diff::_longestCommonSubsequence($delcmds,$addcmds, sub { $_[0]->[0] } );
}
for ($i=0 ; $i<=$#$matchindex ; $i++) {
if (defined($matchindex->[$i])){
$j=$matchindex->[$i];
@delmid=splitlatex($delcmds->[$i][3]);
@addmid=splitlatex($addcmds->[$j][3]);
while (scalar(@$deltextblocks) && $deltextblocks->[0][0]<$delcmds->[$i][1]) {
my ($index,$block,$cnt)=@{ shift(@$deltextblocks) };
push(@$todo, [$index,-1,$cnt,@$block]);
}
push(@$todo, [ $delcmds->[$i][1],-1,-1,$delcmds->[$i][2],@delmid,$delcmds->[$i][4]]);
while (scalar(@$addtextblocks) && $addtextblocks->[0][0]<$addcmds->[$j][1]) {
my ($index,$block,$cnt)=@{ shift(@$addtextblocks) };
push(@$todo, [-1,$index,$cnt,@$block]);
}
push(@$todo, [ -1,$addcmds->[$j][1],-1,$addcmds->[$j][2],@addmid,$addcmds->[$j][4]]);
}
}
# mop up remaining textblocks
while (scalar(@$deltextblocks)) {
my ($index,$block,$cnt)=@{ shift(@$deltextblocks) } ;
push(@$todo, [$index,-1,$cnt,@$block]);
}
while (scalar(@$addtextblocks)) {
my ($index,$block,$cnt)=@{ shift(@$addtextblocks) };
push(@$todo, [-1,$index,$cnt,@$block]);
}
$addblock=[];
$delblock=[];
}
push(@$block,$seq2->[$_[1]]);
$cnt++ };
my $keyfunc = sub { join(" ",split(" ",shift())) };
traverse_sequences($seq1,$seq2, { MATCH=>$match, DISCARD_A=>$discard, DISCARD_B=>$add }, $keyfunc );
# now carry out the merging/splitting. Refer to elements relative from
# the end (with negative indices) as these offsets don't change before the instruction is executed
# cnt>0: merged small unchanged groups with previous changed blocks
# cnt==-1: split textual commands into components
foreach $instruction ( @$todo) {
($last1,$last2,$cnt,@$block)=@$instruction ;
if ($cnt>=0) {
splice(@$seq1,$last1-$len1,1+$cnt,join("",$seq1->[$last1-$len1],@$block)) if $last1>=0;
splice(@$seq2,$last2-$len2,1+$cnt,join("",$seq2->[$last2-$len2],@$block)) if $last2>=0;
} else {
splice(@$seq1,$last1-$len1,1,@$block) if $last1>=0;
splice(@$seq2,$last2-$len2,1,@$block) if $last2>=0;
}
}
if ($verbose) {
print STDERR "\n";
print STDERR " $mattokcnt matching tokens in $matblkcnt blocks.\n";
print STDERR " $deltokcnt discarded tokens in $delblkcnt blocks.\n";
print STDERR " $addtokcnt appended tokens in $addblkcnt blocks.\n";
}
}
# extracttextblocks(\@blockindex)
# $blockindex has the following format
# [ [ token1, index1 ], [token2, index2],.. ]
# where index refers to the index in the original old or new word sequence
# Returns: reference to an array of the form
# [[ $index, $textblock, $cnt ], ..
# where $index index of block to be merged
# $textblock contains all the words to be merged with the word at $index (but does not contain this word)
# $cnt is length of block
#
# requires: iscmd
#
sub extracttextblocks {
my $block=shift;
my ($i,$token,$index);
my $textblock=[];
my $last=-1;
my $wpat=qr/^(?:[a-zA-Z.,'`:;?()!]*)[\s~]*$/; #'
my $retval=[];
for ($i=0;$i< scalar @$block;$i++) {
($token,$index)=@{ $block->[$i] };
# store pure text blocks
if ($token =~ /$wpat/ || ( $token =~/^\\((?:[`'^"~=.]|[\w\d@\*]+))((?:${extraspace}\[$brat0\]${extraspace}|${extraspace}\{$pat_n\})*)/o
&& iscmd($1,\@SAFECMDLIST,\@SAFECMDEXCL)
&& !iscmd($1,\@TEXTCMDLIST,\@TEXTCMDEXCL))) {
# we have text or a command which can be treated as text
if ($last<0) {
# new pure-text block
$last=$index;
} else {
# add to pure-text block
push(@$textblock, $token);
}
} else {
# it is not text
if (scalar(@$textblock)) {
push(@$retval,[ $last, $textblock, scalar(@$textblock) ]);
}
$textblock=[];
$last=-1;
}
}
# finish processing a possibly unfinished block before returning
if (scalar(@$textblock)) {
push(@$retval,[ $last, $textblock, scalar(@$textblock) ]);
}
return($retval)
}
# extractcommands( \@blockindex )
# $blockindex has the following format
# [ [ token1, index1 ], [token2, index2],.. ]
# where index refers to the index in the original old or new word sequence
# Returns: reference to an array of the form
# [ [ "\cmd1", index, "\cmd1[optarg]{arg1}{", "arg2" ,"} " ],..
# where index is just taken from input array
# command must have a textual argument as last argument
#
# requires: iscmd
#
sub extractcommands {
my $block=shift;
my ($i,$token,$index,$cmd,$open,$mid,$closing);
my $retval=[];
for ($i=0;$i< scalar @$block;$i++) {
($token,$index)=@{ $block->[$i] };
# check if token is an alphanumeric command sequence with at least one non-optional argument
# \cmd[...]{...}{last argument}
# Capturing in the following results in these associations
# $1: \cmd[...]{...}{
# $2: \cmd
# $3: last argument
# $4: } + trailing spaces
if ( ( $token =~ m/^(\\([\w\d\*]+)(?:${extraspace}\[$brat0\]|${extraspace}\{$pat_n\})*${extraspace}\{)($pat_n)(\}\s*)$/so )
&& iscmd($2,\@TEXTCMDLIST,\@TEXTCMDEXCL) ) {
# push(@$retval,[ $2,$index,$1,$3,$4 ]);
($cmd,$open,$mid,$closing) = ($2,$1,$3,$4) ;
$closing =~ s/\}/\\RIGHTBRACE/ ;
push(@$retval,[ $cmd,$index,$open,$mid,$closing ]);
}
}
return $retval;
}
# iscmd($cmd,\@regexarray,\@regexexcl) checks
# return 1 if $cmd matches any of the patterns in the
# array $@regexarray, and none of the patterns in \@regexexcl, otherwise return 0
sub iscmd {
my ($cmd,$regexar,$regexexcl)=@_;
my ($ret)=0;
foreach $pat ( @$regexar ) {
if ( $cmd =~ m/^${pat}$/ ) {
$ret=1 ;
last;
}
}
return 0 unless $ret;
foreach $pat ( @$regexexcl ) {
return 0 if ( $cmd =~ m/^${pat}$/ );
}
return 1;
}
# pass2( \@seq1,\@seq2)
# Look for differences between seq1 and seq2.
# Mark begin and end of deleted and appended sequences with tags $DELOPEN and $DELCLOSE
# and $ADDOPEN and $ADDCLOSE, respectively, however exclude { } & and all comands, unless
# they match an element of the whitelist (SAFECMD)
# For words in TEXTCMD but not in SAFECMD, enclose interior with $ADDOPEN and $ADDCLOSE brackets
# Deleted comment lines are marked with %DIF <
# Added comment lines are marked with %DIF >
sub pass2 {
my $seq1 = shift ;
my $seq2 = shift ;
my ($addtokcnt,$deltokcnt,$mattokcnt)=(0,0,0);
my ($addblkcnt,$delblkcnt,$matblkcnt)=(0,0,0);
my $retval = [];
my $delhunk = [];
my $addhunk = [];
my $discard = sub { $deltokcnt++;
push ( @$delhunk, $seq1->[$_[0]]) };
my $add = sub { $addtokcnt++;
push ( @$addhunk, $seq2->[$_[1]]) };
my $match = sub { $mattokcnt++;
if ( scalar @$delhunk ) {
$delblkcnt++;
# mark up changes, but comment out commands
push @$retval,marktags($DELMARKOPEN,$DELMARKCLOSE,$DELOPEN,$DELCLOSE,$DELCMDOPEN,$DELCMDCLOSE,$DELCOMMENT,$delhunk);
$delhunk = [];
}
if ( scalar @$addhunk ) {
$addblkcnt++;
# we mark up changes, but simply quote commands
push @$retval,marktags($ADDMARKOPEN,$ADDMARKCLOSE,$ADDOPEN,$ADDCLOSE,"","",$ADDCOMMENT,$addhunk);
$addhunk = [];
}
push(@$retval,$seq2->[$_[1]]) };
my $keyfunc = sub { join(" ",split(" ",shift())) };
traverse_sequences($seq1,$seq2, { MATCH=>$match, DISCARD_A=>$discard, DISCARD_B=>$add }, $keyfunc );
# clear up unprocessed hunks
push @$retval,marktags($DELMARKOPEN,$DELMARKCLOSE,$DELOPEN,$DELCLOSE,$DELCMDOPEN,$DELCMDCLOSE,$DELCOMMENT,$delhunk) if scalar @$delhunk;
push @$retval,marktags($ADDMARKOPEN,$ADDMARKCLOSE,$ADDOPEN,$ADDCLOSE,"","",$ADDCOMMENT,$addhunk) if scalar @$addhunk;
if ($verbose) {
print STDERR "\n";
print STDERR " $mattokcnt matching tokens. \n";
print STDERR " $deltokcnt discarded tokens in $delblkcnt blocks.\n";
print STDERR " $addtokcnt appended tokens in $addblkcnt blocks.\n";
}
return(@$retval);
}
# marktags($openmark,$closemark,$open,$close,$opencmd,$closecmd,$comment,\@block)
# returns ($openmark,$open,$block,$close,$closemark) if @block contains no commands (except white-listed ones),
# braces, ampersands, or comments
# mark comments with $comment
# exclude all other exceptions from scope of open, close like this
# ($openmark, $open,...,$close, $opencmd,command, command,$closecmd, $open, ..., $close, $closemark)
# If $opencmd begins with "%" marktags assumes it is operating on a deleted block, otherwise on an added block
sub marktags {
my ($openmark,$closemark,$open,$close,$opencmd,$closecmd,$comment,$block)=@_;
my $word;
my (@argtext);
my $retval=[];
my $noncomment=0;
my $cmd=-1; # -1 at beginning 0: last token written is a ordinary word
# 1: last token written is a command
# for keeping track whether we are just in a command sequence or in a word sequence
my $cmdcomment= ($opencmd =~ m/^%/); # Flag to indicate whether opencmd is a comment (i.e. if we intend to simply comment out changed commands)
my ($command,$commandword,$closingbracket) ; # temporary variables needed below to remember sub-pattern matches
# split this block to flatten out sequences joined in pass1
@$block=splitlatex(join "",@$block);
### print STDERR "DEBUG: marktags $openmark,$closemark,$open,$close,$opencmd,$closecmd,$comment\n" if $debug;
### print STDERR "DEBUG: marktags blocksplit ",join("|",@$block),"\n" if $debug;
foreach (@$block) {
$word=$_;
### print STDERR "DEBUG MARKTAGS: |$word|\n" if $debug;
if ( $word =~ s/^%/%$comment/ ) {
# a comment
if ($cmd==1) {
push (@$retval,$closecmd) ;
$cmd=-1;
}
push (@$retval,$word);
next;
}
if ( $word =~ m/^\s*$/ ) {
### print STDERR "DEBUG MARKTAGS: whitespace detected |$word| cmdcom |$cmdcomment| |$opencmd|\n" if $debug;
# a sequence of white-space characters - this should only ever happen for the first element of block.
# in deleted block, omit, otherwise just copy it in
if ( ! $cmdcomment) { # ignore in deleted blocks
push(@$retval,$word);
}
next;
}
if (! $noncomment) {
push (@$retval,$openmark);
$noncomment=1;
}
# negative lookahead pattern (?!) in second clause is put in to avoid matching \( .. \) patterns
# also note that second pattern will match \\
### print STDERR "DEBUG marktags: Considering word |$word|\n";
if ( $word =~ /^[&{}\[\]]/ || ( $word =~ /^\\(?!\()(\\|[`'^"~=.]|[\w*@]+)/ && !iscmd($1,\@SAFECMDLIST,\@SAFECMDEXCL)) ) {
###print STDERR "DEBUG MARKTAGS is a non-safe command ($1)\n" if $debug;
### if ( $word =~ /^[&{}\[\]]/ || ( $word =~ /^\\([\w*@\\% ]+)/ && !iscmd($1,\@SAFECMDLIST,\@SAFECMDEXCL)) ) {
# word is a command or other significant token (not in SAFECMDLIST)
## same conditions as in subroutine extractcommand:
# check if token is an alphanumeric command sequence with at least one non-optional argument
# \cmd[...]{...}{last argument}
# Capturing in the following results in these associations
# $1: \cmd[...]{...}{
# $2: cmd
# $3: last argument
# $4: } + trailing spaces
### pre-0.3 if ( ( $token =~ m/^(\\([\w\d\*]+)(?:\[$brat0\]|\{$pat_n\})*\{)($pat_n)(\}\s*)$/so )
if ( ( $word =~ m/^(\\([\w\d\*]+)(?:${extraspace}\[$brat0\]|${extraspace}\{$pat_n\})*${extraspace}\{)($pat_n)(\}\s*)$/so )
&& (iscmd($2,\@TEXTCMDLIST,\@TEXTCMDEXCL)|| iscmd($2,\@MATHTEXTCMDLIST,\@MATHTEXTCMDEXCL))
&& ( !$cmdcomment || !iscmd($2,\@CONTEXT2CMDLIST, \@CONTEXT2CMDEXCL) ) ) {
# Condition 1: word is a command? - if yes, $1,$2,.. will be set as above
# Condition 2: word is a text command - we mark up the interior of the word. There is a separate check for MATHTEXTCMDLIST
# because for $mathmarkup=WHOLE, the commands should not be split in pass1 (ie. math mode commands are not in
# TEXTCMDLIST, but the interior of MATHTEXT commands should be highlighted in both deleted and added blocks
# Condition 3: But if we are in a deleted block ($cmdcomment=1) and
# $2 (the command) is in context2, just treat it as an ordinary command (i.e. comment it open with $opencmd)
# Because we do not want to disable this command
# here we do not use $opencmd and $closecmd($opencmd is empty)
if ($cmd==1) {
push (@$retval,$closecmd) ;
} elsif ($cmd==0) {
push (@$retval,$close) ;
}
$command=$1; $commandword=$2; $closingbracket=$4;
@argtext=splitlatex($3); # split textual argument into tokens
# and mark it up (but we do not need openmark and closemark)
# insert command with initial arguments, marked-up final argument, and closing bracket
if ( $cmdcomment && iscmd($commandword,\@CONTEXT1CMDLIST, \@CONTEXT1CMDEXCL) ) {
# context1cmd in a deleted environment; delete command itself but keep last argument, marked up
push (@$retval,$opencmd);
$command =~ s/\n/\n${opencmd}/sg ; # repeat opencmd at the beginning of each line
# argument, note that the additional comment character is included
# to suppress linebreak after opening parentheses, which is important
# for latexrevise
push (@$retval,$command,"%\n{$AUXCMD\n",marktags("","",$open,$close,$opencmd,$closecmd,$comment,\@argtext),$closingbracket);
} elsif ( iscmd($commandword,,\@MATHTEXTCMDLIST, \@MATHTEXTCMDEXCL) ) {
# MATHBLOCK pseudo command: consider all commands safe, except & and \\
# Keep these commands even in deleted blocks, hence set $opencmd and $closecmd (5th and 6th argument of marktags) to
# ""
local @SAFECMDLIST=(".*");
local @SAFECMDEXCL=('\\','\\\\',@UNSAFEMATHCMD);
push(@$retval,$command,marktags("","",$open,$close,"","",$comment,\@argtext)#@argtext
,$closingbracket);
} else {
# normal textcmd or context1cmd in an added block
push (@$retval,$command,marktags("","",$open,$close,$opencmd,$closecmd,$comment,\@argtext),$closingbracket);
}
push (@$retval,$AUXCMD,"\n") if $cmdcomment ;
$cmd=-1 ;
} else {
# ordinary command
push (@$retval,$opencmd) if $cmd==-1 ;
push (@$retval,$close,$opencmd) if $cmd==0 ;
$word =~ s/\n/\n${opencmd}/sg if $cmdcomment ; # if opencmd is a comment, repeat this at the beginning of every line
### print STDERR "MARKTAGS: Add command |$word|\n";
push (@$retval,$word);
$cmd=1;
}
} else {
###print STDERR "DEBUG MARKTAGS is an ordinary word or SAFECMD command \n" if $debug;
# just an ordinary word or command in SAFECMD
push (@$retval,$open) if $cmd==-1 ;
push (@$retval,$closecmd,$open) if $cmd==1 ;
###TODO: check here if it is a command in MBOXCMD list, and surround it with \mbox{...}
### $word =~ /^\\(?!\()(\\|[`'^"~=.]|[\w*@]+)/ && iscmd($1,\@MBOXCMDLIST,\@MBOXCMDEXCL))
### but actually this check has been carried out already so can simply check if word begins with backslash
if ( $word =~ /^\\(?!\()(\\|[`'^"~=.]|[\w*@]+)(.*?)(\s*)$/s && iscmd($1,\@MBOXCMDLIST,\@MBOXCMDEXCL)) {
# $word is a safe command in MBOXCMDLIST
###print STDERR "DEBUG Mboxsafecmd detected:$word:\n" if $debug ;
push(@$retval,"\\mbox{$AUXCMD\n\\" . $1 . $2 . $3 ."}$AUXCMD\n" );
} else {
# $word is a normal word or a safe command (not in MBOXCMDLIST)
push (@$retval,$word);
}
$cmd=0;
}
}
push (@$retval,$close) if $cmd==0;
push (@$retval,$closecmd) if $cmd==1;
push (@$retval,$closemark) if ($noncomment);
return @$retval;
}
#used in preprocess
sub take_comments_and_enter_from_frac() {
###*************take the \n and % between frac and {}***********
###notice all of the substitution are made none global
while( m/\\begin{($MATHENV|$MATHARRENV|SQUAREBRACKET)}(.*?)\\frac(([\s]*%[^\n]*?)*[\r\n|\r|\n])+\{(.*?)\\end{\1}/s ) {
### if there isn't any % or \n in the pattern $2 then there should be an \\end{...} in $2
if( $2 !~ m/\\end{$1}/s ) {
### take out % and \n from the next match only (none global)
s/\\begin{($MATHENV|$MATHARRENV|SQUAREBRACKET)}(.*?)\\frac(([\s]*%[^\n]*?)*[\r\n|\r|\n])+\{(.*?)\\end{\1}/\\begin{$1}$2\\frac{$5\\end{$1}/s;
}
else{
###there are no more % and \n in $2, we want to find the next one so we clear the begin-end from the pattern
s/\\begin{($MATHENV|$MATHARRENV|SQUAREBRACKET)}(.*?)\\end{\1}/MATHBLOCK$1$2MATHBLOCKEND/s;
}
}
###cleaning up
while( s/MATHBLOCK($MATHENV|$MATHARRENV|SQUAREBRACKET)(.*?)MATHBLOCKEND/\\begin{$1}$2\\end{$1}/s ){}
###*************take the \n and % between frac and {}***********
###**********take the \n and % between {} and {} of the frac***************
while( m/\\begin\{($MATHENV|$MATHARRENV|SQUAREBRACKET)\}(.*?)\\frac\{(.*?)\\end\{\1\}/s ) {
### if there isn't any more //frac before the first //end in the pattern $2 then there should be an \\end{...} in $2
if( $2 !~ m/\\end\{$1\}/s ) {
### from now on CURRFRAC is the frac we are looking at
s/\\begin\{($MATHENV|$MATHARRENV|SQUAREBRACKET)\}(.*?)\\frac\{(.*?)\\end\{\1\}/\\begin\{$1\}$2CURRFRAC\{$3\\end\{$1\}/s;
while( m/\\begin\{($MATHENV|$MATHARRENV|SQUAREBRACKET)\}(.*?)CURRFRAC\{(.*?)\\end\{\1\}/s ) {
if( m/\\begin\{($MATHENV|$MATHARRENV|SQUAREBRACKET)\}(.*?)CURRFRAC\{($pat_n)\}([\s]*(%[^\n]*?)*[\r\n|\r|\n])+[\s]*\{(.*?)\\end{\1}/s ) {
s/\\begin\{($MATHENV|$MATHARRENV|SQUAREBRACKET)\}(.*?)CURRFRAC\{($pat_n)\}([\s]*(%[^\n]*?)*[\r\n|\r|\n])+[\s]*\{(.*?)\\end\{\1\}/\\begin\{$1\}$2CURRFRAC\{$3\}\{$6\\end\{$1\}/s;
}
else { # there is no comment or \n between the two brackets {}{}
### change CURRFRAC to FRACSTART so we can change them all back to //frac{ when we finish
s/\\begin{($MATHENV|$MATHARRENV|SQUAREBRACKET)}(.*?)CURRFRAC\{(.*?)\\end{\1}/\\begin{$1}$2FRACSTART\{$3\\end{$1}/s;
}
}
}
else{
###there are no more frac in $2, we want to find the next one so we clear the begin-end from the pattern
s/\\begin{($MATHENV|$MATHARRENV|SQUAREBRACKET)}(.*?)\\end{\1}/MATHBLOCK$1$2MATHBLOCKEND/s;
}
}
###cleaning up
while( s/MATHBLOCK($MATHENV|$MATHARRENV|SQUAREBRACKET)(.*?)MATHBLOCKEND/\\begin{$1}$2\\end{$1}/s ){}
s/FRACSTART/\\frac/g;
###***************take the \n and % between {} and {} of the frac*********************
}
# preprocess($string, ..)
# carry out the following pre-processing steps for all arguments:
# 1. Remove leading white-space
# Change \{ to \QLEFTBRACE and \} to \QRIGHTBRACE and \& to \AMPERSAND
# #. Change {,} in comments to \CLEFTBRACE, \CRIGHTBRACE
# 2. mark all first empty line (in block of several) with \PAR tokens
# 3. Convert all '\%' into '\PERCENTAGE ' and all '\$' into \DOLLAR to make parsing regular expressions easier
# 4. Convert all \verb|some verbatim text| commands (where | can be an arbitrary character)
# into \verb{hash}
# 5. Convert \begin{verbatim} some verbatim text \end{verbatim} into \verbatim{hash}
# 6. Convert _n into \SUBSCRIPTNB{n} and _{nnn} into \SUBSCRIPT{nn}
# 7. Convert ^n into \SUPERSCRIPTNB{n} and ^{nnn} into \SUPERSCRIPT{nn}
# 8. a. Convert $$ $$ into \begin{DOLLARDOLLAR} \end{DOLLARDOLLAR}
# b. Convert \[ \] into \begin{SQUAREBRACKET} \end{SQUAREBRACKET}
# 9. Convert all picture environmentent (\begin{PICTUREENV} .. \end{PICTUREENV} \PICTUREBLOCKenv
# For --block-math-markup option -convert all \begin{MATH} .. \end{MATH}
# into \MATHBLOCKmath{...} commands, where MATH/math is any valid math environment
# 10. Add final token STOP to the very end. This is put in because the algorithm works better if the last token is identical. This is removed again in postprocessing.
#
# NB: step 6 and 7 is likely to convert some "_" inappropriately, e.g. in file
# names or labels but it does not matter because they are converted back in the postprocessing step
# Returns: leading white space removed in step 1
sub preprocess {
for (@_) {
# Change \{ to \QLEFTBRACE, \} to \QRIGHTBRACE, and \& to \AMPERSAND
s/(?{$hstr}) && $string ne $hash->{$hstr});
# else found a duplicate HASH need to repeat for a higher hash value
}
$hash->{$hstr}=$string;
### print STDERR "Hash:$hstr: Content:$string:\n";
return($hstr);
}
#string=fromhash(\%hash,$fromstring)
# restores string value stored in hash
#string=fromhash(\%hash,$fromstring,$prependstring)
# additionally begins each line with prependstring
sub fromhash {
my ($hash,$hstr)=($_[0],$_[1]);
my $retstr=$hash->{$hstr};
if ( $#_ >= 2) {
$retstr =~ s/^/$_[2]/mg;
}
return $retstr;
}
# postprocess($string, ..)
# carry out the following post-processing steps for all arguments:
# * Remove STOP token from the end
# * Replace \RIGHTBRACE by }
# * change citation commands within comments to protect from processing (using marker CITEDIF)
# 1. Check all deleted blocks:
# a.where a deleted block contains a matching \begin and
# \end environment (these will be disabled by a %DIFDELCMD statements), for selected environments enable
# these commands again (such that for example displayed math in a deleted equation
# is properly within math mode. For math mode environments replace numbered equation
# environments with their display only variety (so that equation numbers in new file and
# diff file are identical). Where the correct type of math environment cannot be determined
# use a place holder MATHMODE
# b.where one of the commands matching $COUNTERCMD is used as a DIFAUXCMD, add a statement
# subtracting one from the respective counter to keep numbering consistent with new file
# Replace all MATHMODE environment commands by the correct environment to achieve matching
# pairs
# c. Convert MATHBLOCKmath commands to their uncounted numbers (e.g. convert equation -> displaymath
# (environments defined in $MATHENV will be replaced by $MATHREPL, and environments in $MATHARRENV
# will be replaced by $MATHARRREPL
# d. If in-line math mode contains array environment, enclose the whole environment in \mbox'es
# d. place \cite commands in mbox'es (for UNDERLINE style)
#
# For added blocks:
# c. If in-line math mode contains array environment, enclose the whole environment in \mbox'es
# d. place \cite commands in mbox'es (for UNDERLINE style)
#
# 2. If --block-math-markup option set: Convert \MATHBLOCKmath{..} commands back to environments
#
# Convert all PICTUREblock{..} commands back to the appropriate environments
# 3. Convert DIFadd, DIFdel, DIFFaddbegin , ... into FL varieties
# within floats (currently recognised float environments: plate,table,figure
# plus starred varieties).
# 4. Remove empty %DIFDELCMD < lines
# 4. Convert \begin{SQUAREBRACKET} \end{SQUAREBRACKET} into \[ \]
# Convert \begin{DOLLARDOLLAR} \end{DOLLARDOLLAR} into $$ $$
# 5. Convert \SUPERSCRIPTNB{n} into ^n and \SUPERSCRIPT{nn} into ^{nnn}
# 6. Convert \SUBSCRIPTNB{n} into _n and \SUBCRIPT{nn} into _{nnn}
# 7. Expand hashes of verb and verbatim environments
# 8. Convert '\PERCENTAGE ' back into '\%' and '\DOLLAR ' into '\$'
# 9.. remove all \PAR tokens
# 10. package specific processing: endfloat: make sure \begin{figure} and \end{figure} are always
# on a line by themselves, similarly for table environment
# 4, undo renaming of the \begin, \end,{,} in comments
# Change \QLEFTBRACE, \QRIGHTBRACE,\AMPERSAND to \{,\},\&
#
# Note have to manually synchronize substitution commands below and
# DIF.. command names in the header
sub postprocess {
my ($begin,$len,$cnt,$float,$delblock,$addblock);
# second level blocks
my ($begin2,$cnt2,$len2,$eqarrayblock,$mathblock);
my (@textparts,@newtextparts,@liststack,$listtype,$listlast);
for (@_) {
# change $'s in comments to something harmless
1 while s/(%.*)\$/$1DOLLARDIF/mg ;
# Remove final STOP token
s/ STOP$//;
# Replace \RIGHTBRACE by }
s/\\RIGHTBRACE/}/g;
# Check all deleted blocks: where a deleted block contains a matching \begin and
# \end environment (these will be disabled by a %DIFDELCMD statements), enable
# these commands again (such that for example displayed math in a deleted equation
# is properly within math mode. For math mode environments replace numbered equation
# environments with their display only variety (so that equation numbers in new file and
# diff file are identical
while ( m/\\DIFdelbegin.*?\\DIFdelend/sg ) {
$cnt=0;
$len=length($&);
$begin=pos($_) - $len;
$delblock=$&;
### (.*?[^\n]?)\n? construct is necessary to avoid empty lines in math mode, which result in
### an error
# displayed math environments
if ($mathmarkup == FINE ) {
$delblock=~ s/(\%DIFDELCMD < \s*\\begin\{((?:$MATHENV)|SQUAREBRACKET)\}.*?(?:$DELCMDCLOSE|\n))(.*?[^\n]?)\n?(\%DIFDELCMD < \s*\\end\{\2\})/\\begin{$MATHREPL}$AUXCMD\n$1$3\n\\end{$MATHREPL}$AUXCMD\n$4/sg;
# also transform the opposite pair \end{displaymath} .. \begin{displaymath} but we have to be careful not to interfere with the results of the transformation in the line directly above
### pre-0.42 obsolete version which did not work on eqnarray test $delblock=~ s/(? displaymath
# (environments defined in $MATHENV will be replaced by $MATHREPL, and environments in $MATHARRENV
# will be replaced by $MATHARRREPL
$delblock=~ s/\\MATHBLOCK($MATHENV)\{($pat_n)\}/\\MATHBLOCK$MATHREPL\{$2\}/sg;
$delblock=~ s/\\MATHBLOCK($MATHARRENV)\{($pat_n)\}/\\MATHBLOCK$MATHARRREPL\{$2\}/sg;
}
# Reinstate completely deleted list environments. note that items within the
# environment will still be commented out. They will be restored later
$delblock=~ s/(\%DIFDELCMD < \s*\\begin\{($LISTENV)\}\s*?(?:\n|$DELCMDCLOSE))(.*?)(\%DIFDELCMD < \s*\\end\{\2\})/{
### # block within the search; replacement environment
### "$1\\begin{$2}$AUXCMD\n". restore_item_commands($3). "\n\\end{$2}$AUXCMD\n$4";
"$1\\begin{$2}$AUXCMD\n$3\n\\end{$2}$AUXCMD\n$4";
}/esg;
# b.where one of the commands matching $COUNTERCMD is used as a DIFAUXCMD, add a statement
# subtracting one from the respective counter to keep numbering consistent with new file
$delblock=~ s/\\($COUNTERCMD)((?:${extraspace}\[$brat0\]${extraspace}|${extraspace}\{$pat_n\})*\s*${AUXCMD}\n)/\\$1$2\\addtocounter{$1}{-1}${AUXCMD}\n/sg ;
# bb. disable active labels within deleted blocks (as these are not safe commands, this should normally only
# happen within deleted maths blocks
$delblock=~ s/(\\$LABELCMD(?:${extraspace})\{(?:[^{}])*\}[\t ]*)\n?/${DELCMDOPEN}$1${DELCMDCLOSE}/smg ;
# c. If in-line math mode contains array environment, enclose the whole environment in \mbox'es
while ( $delblock =~ m/($math)(\s*)/sg ) {
$cnt2=0;
$len2=length($&);
$begin2=pos($delblock) - $len2;
$mathblock="%\n\\mbox{$AUXCMD\n$1\n}$AUXCMD\n";
next unless $mathblock =~ m/\{$ARRENV\}/ ;
substr($delblock,$begin2,$len2)=$mathblock;
pos($delblock) = $begin2 + length($mathblock);
}
# if MBOXINLINEMATH is set, protect inlined math environments with an extra mbox
if ( $MBOXINLINEMATH ) {
# note additional \newline after command is omitted from output if right at the end of deleted block (otherwise a spurious empty line is generated)
$delblock=~s/($math)(?:[\s\n]*)?/\\mbox{$AUXCMD\n$1\n}$AUXCMD\n/sg;
}
# splice in modified delblock
substr($_,$begin,$len)=$delblock;
pos = $begin + length($delblock);
}
# make the array modification in added blocks
while ( m/\\DIFaddbegin.*?\\DIFaddend/sg ) {
$cnt=0;
$len=length($&);
$begin=pos($_) - $len;
$addblock=$&;
while ( $addblock =~ m/($math)(\s*)/sg ) {
$cnt2=0;
$len2=length($&);
$begin2=pos($addblock) - $len2;
$mathblock="%\n\\mbox{$AUXCMD\n$1\n}$AUXCMD\n";
next unless $mathblock =~ m/\{$ARRENV\}/ ;
substr($addblock,$begin2,$len2)=$mathblock;
pos($addblock) = $begin2 + length($mathblock);
}
# if MBOXINLINEMATH is set, protect inlined math environments with an extra mbox
if ( $MBOXINLINEMATH ) {
##$addblock=~s/($math)/\\mbox{$AUXCMD\n$1\n}$AUXCMD\n/sg;
$addblock=~s/($math)(?:[\s\n]*)?/\\mbox{$AUXCMD\n$1\n}$AUXCMD\n/sg;
}
# splice in modified addblock
substr($_,$begin,$len)=$addblock;
pos = $begin + length($addblock);
}
# Go through whole text, and by counting list environment commands, find out when we are within a list environment.
# Within those restore deleted \item commands
@textparts=split /(? 0) {
$listlast=pop(@liststack);
($listtype eq $listlast) or warn "Invalid nesting of list environments: $listlast environment closed by \\end{$listtype}.";
} else {
warn "Invalid nesting of list environments: \\end{$listtype} encountered without matching \\begin{$listtype}.";
}
} else {
print STDERR "DEBUG: postprocess \@liststack=(",join(",",@liststack),")\n" if $debug;
if (scalar @liststack > 0 ) {
# we are within a list environment and should replace all item commands
$_=restore_item_commands($_);
}
# else: we are outside a list environment and do not need to do anything
}
$_ } @textparts; # end of map command
# replace the main text with the modified version
$_= "@newtextparts";
# Replace MATHMODE environments from step 1a above by the correct Math environment
# The next line is complicated. The negative look-ahead insertion makes sure that no \end{$MATHENV} (or other mathematical
# environments) are between the \begin{$MATHENV} and \end{MATHMODE} commands. This is necessary as the minimal matching
# is not globally minimal but only 'locally' (matching is beginning from the left side of the string)
if ( $mathmarkup == FINE ) {
1 while s/\\begin{((?:$MATHENV)|(?:$MATHARRENV)|SQUAREBRACKET)}((?:.(?!(?:\\end{(?:(?:$MATHENV)|(?:$MATHARRENV)|SQUAREBRACKET)}|\\begin{MATHMODE})))*?)\\end{MATHMODE}/\\begin{$1}$2\\end{$1}/s;
1 while s/\\begin{MATHMODE}((?:.(?!\\end{MATHMODE}))*?)\\end{((?:$MATHENV)|(?:$MATHARRENV)|SQUAREBRACKET)}/\\begin{$2}$1\\end{$2}/s;
# convert remaining \begin{MATHMODE} \end{MATHMODE} (and not containing & or \\ )into MATHREPL environments
s/\\begin{MATHMODE}((?:(.(?!(?)?)((?:\[$brat0\])?)\s*?(?:\n|$DELCMDCLOSE))/
# if \item has an []argument, then mark up the argument as deleted)
if (length($4)>0) {
# use substr to exclude square brackets at end points
@itemargs=splitlatex(substr($4,1,length($4)-2));
$itemarg="[".join("",marktags("","",$DELOPEN,$DELCLOSE,$DELCMDOPEN,$DELCMDCLOSE,$DELCOMMENT,\@itemargs))."]";
} else {
$itemarg="";
}
"$1$2$3$itemarg$AUXCMD\n"/sge;
return($string);
}
# @auxlines=preprocess_preamble($oldpreamble,$newpreamble);
# pre-process preamble by looking for commands used in \maketitle (title, author, date etc commands)
# if found then use a bodydiff to mark up content, and replace the corresponding commands
# in both preambles by marked up version to 'fool' the linediff (such that only body is marked
# up.
# A special case are e.g. author commands being added (or removed)
# 1. If commands are added, then the entire content is marked up as new, but also the lines are marked as new in the linediff
# 2. If commands are removed, then the linediff will mark the line as deleted. The program returns
# with $auxlines a text to be appended at the end of the preamble, which shows the respective fields as deleted
sub preprocess_preamble {
my ($oldpreambleref,$newpreambleref)=(\$_[0],\$_[1]) ;
my @auxlines=();
# Remember to use $$oldpreambleref to refer to oldpreamble
my ($titlecmd,$titlecmdpat);
my (@oldtitlecommands,@newtitlecommands );
my %oldhash = ();
my %newhash = ();
my ($line,$cmd,$optarg,$arg,$optargnew,$optargold,$optargdiff,$argold,$argnew,$argdiff,$auxline);
# resuse context2cmdlist to define these commands to look out for in preamble
$titlecmd = "(?:".join("|",@CONTEXT2CMDLIST).")";
# as context2cmdlist is stored as regex, e.g. ((?-xism:^title$), we need to remove ^- fo
# resue in a more complex regex
$titlecmd =~ s/[\$\^]//g;
# make sure to not match on comment lines:
$titlecmdpat=qr/^(?:[^%\n]|\\%)*(\\($titlecmd)$extraspace(?:\[($brat0)\])?(?:\{($pat_n)\}))/ms;
@oldtitlecommands= ( $$oldpreambleref =~ m/$titlecmdpat/g );
@newtitlecommands= ( $$newpreambleref =~ m/$titlecmdpat/g );
while ( @oldtitlecommands ) {
$line=shift @oldtitlecommands;
$cmd=shift @oldtitlecommands;
$optarg=shift @oldtitlecommands;
$arg=shift @oldtitlecommands;
if ( defined($oldhash{$cmd})) {
warn "$cmd is used twice in preamble of old file. Reverting to pure line diff mode for preamble.\n";
return;
}
$oldhash{$cmd}=[ $line, $optarg, $arg ];
}
while ( @newtitlecommands ) {
$line=shift @newtitlecommands;
$cmd=shift @newtitlecommands;
$optarg=shift @newtitlecommands;
$arg=shift @newtitlecommands;
if ( defined($newhash{$cmd})) {
warn "$cmd is used twice in preamble of new file. Reverting to pure line diff mode for preamble.\n";
return;
}
$newhash{$cmd}=[ $line, $optarg, $arg ];
}
foreach $cmd ( keys %newhash ) {
if ( defined($newhash{$cmd}->[1])) {
$optargnew=$newhash{$cmd}->[1];
} else {
$optargnew="";
}
if ( defined($oldhash{$cmd}->[1])) {
$optargold=$oldhash{$cmd}->[1];
} else {
$optargold="";
}
if ( defined($oldhash{$cmd}) ) {
$argold=$oldhash{$cmd}->[2];
} else {
$argold="";
}
$argnew=$newhash{$cmd}->[2];
$argdiff="{" . join("",bodydiff($argold,$argnew)) ."}";
if ( length $optargnew ) {
$optargdiff="[".join("",bodydiff($optargold,$optargnew))."]" ;
$optargdiff =~ s/\\DIFaddbegin /\\DIFaddbeginFL /g;
$optargdiff =~ s/\\DIFaddend /\\DIFaddendFL /g;
$optargdiff =~ s/\\DIFadd\{/\\DIFaddFL{/g;
$optargdiff =~ s/\\DIFdelbegin /\\DIFdelbeginFL /g;
$optargdiff =~ s/\\DIFdelend /\\DIFdelendFL /g;
$optargdiff =~ s/\\DIFdel\{/\\DIFdelFL{/g;
} else {
$optargdiff="";
}
### print STDERR "DEBUG s/\\Q$newhash{$cmd}->[0]\\E/\\$cmd$optargdiff$argdiff/s\n";
# Note: \Q and \E force literal interpretation of what it between them but allow
# variable interpolation, such that e.g. \title matches just that and not TAB-itle
$$newpreambleref=~s/\Q$newhash{$cmd}->[0]\E/\\$cmd$optargdiff$argdiff/s;
# replace this in old preamble if necessary
if ( defined($oldhash{$cmd}->[0])) {
$$oldpreambleref=~s/\Q$oldhash{$cmd}->[0]\E/\\$cmd$optargdiff$argdiff/s ;
}
### print STDERR "DEBUG NEW PRE ".$$newpreambleref."\n";
}
foreach $cmd ( keys %oldhash ) {
# if this has already been dealt with above can just skip
next if defined($newhash{$cmd}) ;
if ( defined($oldhash{$cmd}->[1])) {
$optargold=$oldhash{$cmd}->[1];
$optargdiff="[".join("",bodydiff($optargold,""))."]" ;
$optargdiff =~ s/\\DIFdelbegin /\\DIFdelbeginFL /g;
$optargdiff =~ s/\\DIFdelend /\\DIFdelendFL /g;
$optargdiff =~ s/\\DIFdel\{/\\DIFdelFL{/g;
} else {
$optargdiff="";
}
$argdiff="{" . join("",bodydiff($argold,"")) ."}";
$auxline = "\\$cmd$optargdiff$argdiff";
$auxline =~s/$/$AUXCMD/sg;
push @auxlines,$auxline;
}
# add auxcmd comment to highlight added lines
return(@auxlines);
}
# @diffs=linediff(\@seq1, \@seq2)
# mark up lines like this
#%DIF mm-mmdnn
#%< old deleted line(s)
#%DIF -------
#%DIF mmann-nn
#new appended line %<
#%DIF -------
# Future extension: mark change explicitly
# Assumes: traverse_sequence traverses deletions before insertions in changed sequences
# all line numbers relative to line 0 (first line of real file)
sub linediff {
my $seq1 = shift ;
my $seq2 = shift ;
my $block = [];
my $retseq = [];
my @begin=('','',''); # dummy initialisation
my $instring ;
my $discard = sub { @begin=('d',$_[0],$_[1]) unless scalar @$block ;
push(@$block, "%DIF < " . $seq1->[$_[0]]) };
my $add = sub { if (! scalar @$block) {
@begin=('a',$_[0],$_[1]) ;}
elsif ( $begin[0] eq 'd' ) {
$begin[0]='c'; $begin[2]=$_[1];
push(@$block, "%DIF -------") }
push(@$block, $seq2->[$_[1]] . " %DIF > " ) };
my $match = sub { if ( scalar @$block ) {
if ( $begin[0] eq 'd' && $begin[1]!=$_[0]-1) {
$instring = sprintf "%%DIF %d-%dd%d",$begin[1],$_[0]-1,$begin[2]; }
elsif ( $begin[0] eq 'a' && $begin[2]!=$_[1]-1) {
$instring = sprintf "%%DIF %da%d-%d",$begin[1],$begin[2],$_[1]-1; }
elsif ( $begin[0] eq 'c' ) {
$instring = sprintf "%%DIF %sc%s",
($begin[1]==$_[0]-1) ? "$begin[1]" : $begin[1]."-".($_[0]-1) ,
($begin[2]==$_[1]-1) ? "$begin[2]" : $begin[2]."-".($_[1]-1) ; }
else {
$instring = sprintf "%%DIF %d%s%d",$begin[1],$begin[0],$begin[2]; }
push @$retseq, $instring,@$block, "%DIF -------" ;
$block = [];
}
push @$retseq, $seq2->[$_[1]]
};
# key function: remove multiple spaces (such that insertion or deletion of redundant white space is not reported)
my $keyfunc = sub { join(" ",split(" ",shift())) };
traverse_sequences($seq1,$seq2, { MATCH=>$match, DISCARD_A=>$discard, DISCARD_B=>$add }, $keyfunc );
push @$retseq, @$block if scalar @$block;
return wantarray ? @$retseq : $retseq ;
}
# init_regex_arr_data(\@array,"TOKEN INIT")
# scans DATA file handel for line "%% TOKEN INIT" line
# then appends each line not beginning with % into array (as a quoted regex)
sub init_regex_arr_data {
my ($arr,$token)=@_;
my ($copy);
while () {
if ( m/^%%BEGIN $token\s*$/ ) {
$copy=1; }
elsif ( m/^%%END $token\s*/ ) {
last; }
chomp;
push (@$arr,qr/^$_$/) if ( $copy && !/^%/ ) ;
}
seek DATA,0,0; # rewind DATA handle to file begin
}
# init_regex_arr_ext(\@array,$arg)
# fills array with regular expressions.
# if arg is a file name, then read in list of regular expressions from that file
# (one expression per line)
# Otherwise treat arg as a comma separated list of regular expressions
sub init_regex_arr_ext {
my ($arr,$arg)=@_;
my $regex;
if ( -f $ arg ) {
open(FILE,"$arg") or die ("Couldn't open $arg: $!");
while () {
chomp;
next if /^\s*#/ || /^\s*%/ || /^\s*$/ ;
push (@$arr,qr/^$_$/);
}
close(FILE);
}
else {
# assume it is a comma-separated list of reg-ex
foreach $regex (split(qr/(?=1) {
$reset=shift;
}
if ($reset) {
$lasttime=times();
}
else {
$retval=times()-$lasttime;
$lasttime=$lasttime+$retval;
return($retval);
}
}
sub usage {
die <<"EOF";
Usage: $0 [options] old.tex new.tex > diff.tex
Compares two latex files and writes tex code to stdout, which has the same
format as new.tex but has all changes relative to old.tex marked up or commented.
--type=markupstyle
-t markupstyle Add code to preamble for selected markup style
Available styles: UNDERLINE CTRADITIONAL TRADITIONAL CFONT FONTSTRIKE INVISIBLE
CHANGEBAR CCHANGEBAR CULINECHBAR CFONTCBHBAR BOLD
[ Default: UNDERLINE ]
--subtype=markstyle
-s markstyle Add code to preamble for selected style for bracketing
commands (e.g. to mark changes in margin)
Available styles: SAFE MARGIN DVIPSCOL COLOR ZLABEL ONLYCHANGEDPAGE (LABEL)*
[ Default: SAFE ]
* LABEL subtype is deprecated
--floattype=markstyle
-f markstyle Add code to preamble for selected style which
replace standard marking and markup commands within floats
(e.g., marginal remarks cause an error within floats
so marginal marking can be disabled thus)
Available styles: FLOATSAFE IDENTICAL
[ Default: FLOATSAFE ]
--encoding=enc
-e enc Specify encoding of old.tex and new.tex. Typical encodings are
ascii, utf8, latin1, latin9. A list of available encodings can be
obtained by executing
perl -MEncode -e 'print join ("\\n",Encode->encodings( ":all" )) ;'
[Default encoding is utf8 unless the first few lines of the preamble contain
an invocation "\\usepackage[..]{inputenc} in which case the
encoding chosen by this command is asssumed. Note that ASCII (standard
latex) is a subset of utf8]
--preamble=file
-p file Insert file at end of preamble instead of auto-generating
preamble. The preamble must define the following commands
\\DIFaddbegin,\\DIFaddend,\\DIFadd{..},
\\DIFdelbegin,\\DIFdelend,\\DIFdel{..},
and varieties for use within floats
\\DIFaddbeginFL,\\DIFaddendFL,\\DIFaddFL{..},
\\DIFdelbeginFL,\\DIFdelendFL,\\DIFdelFL{..}
(If this option is set -t, -s, and -f options
are ignored.)
--exclude-safecmd=exclude-file
--exclude-safecmd="cmd1,cmd2,..."
-A exclude-file
--replace-safecmd=replace-file
--append-safecmd=append-file
--append-safecmd="cmd1,cmd2,..."
-a append-file Exclude from, replace or append to the list of regex
matching commands which are safe to use within the
scope of a \\DIFadd or \\DIFdel command. The file must contain
one Perl-RegEx per line (Comment lines beginning with # or % are
ignored). A literal comma within the comma-separated list must be
escaped thus "\\,", Note that the RegEx needs to match the whole of
the token, i.e., /^regex\$/ is implied and that the initial
"\\" of the command is not included. The --exclude-safecmd
and --append-safecmd options can be combined with the --replace-safecmd
option and can be used repeatedly to add cumulatively to the lists.
--exclude-textcmd=exclude-file
--exclude-textcmd="cmd1,cmd2,..."
-X exclude-file
--replace-textcmd=replace-file
--append-textcmd=append-file
--append-textcmd="cmd1,cmd2,..."
-x append-file Exclude from, replace or append to the list of regex
matching commands whose last argument is text. See
entry for --exclude-safecmd directly above for further details.
--replace-context1cmd=replace-file
--append-context1cmd=append-file
--append-context1cmd="cmd1,cmd2,..."
Replace or append to the list of regex matching commands
whose last argument is text but which require a particular
context to work, e.g. \\caption will only work within a figure
or table. These commands behave like text commands, except when
they occur in a deleted section, when they are disabled, but their
argument is shown as deleted text.
--replace-context2cmd=replace-file
--append-context2cmd=append-file
--append-context2cmd="cmd1,cmd2,..."
As corresponding commands for context1. The only difference is that
context2 commands are completely disabled in deleted sections, including
their arguments.
--exclude-mboxsafecmd=exclude-file
--exclude-mboxsafecmd="cmd1,cmd2,..."
--append-mboxsafecmd=append-file
--append-mboxsafecmd="cmd1,cmd2,..."
Define safe commands, which additionally need to be protected by encapsulating
in an \\mbox{..}. This is sometimes needed to get around incompatibilities
between external packages and the ulem package, which is used for highlighting
in the default style UNDERLINE as well as CULINECHBAR CFONTSTRIKE
--config var1=val1,var2=val2,...
-c var1=val1,.. Set configuration variables.
-c configfile Available variables:
ARRENV (RegEx)
COUNTERCMD (RegEx)
FLOATENV (RegEx)
ITEMCMD (RegEx)
LISTENV (RegEx)
MATHARRENV (RegEx)
MATHARRREPL (String)
MATHENV (RegEx)
MATHREPL (String)
MINWORDSBLOCK (integer)
PICTUREENV (RegEx)
This option can be repeated.
--packages=pkg1,pkg2,..
Tell latexdiff that .tex file is processed with the packages in list
loaded. This is normally not necessary if the .tex file includes the
preamble, as the preamble is automatically scanned for \\usepackage commands.
Use of the --packages option disables automatic scanning, so if for any
reason package specific parsing needs to be switched off, use --packages=none.
The following packages trigger special behaviour:
endfloat hyperref amsmath apacite siunitx cleveref glossaries mhchem chemformula/chemmacros
[ Default: scan the preamble for \\usepackage commands to determine
loaded packages.]
--show-preamble Print generated or included preamble commands to stdout.
--show-safecmd Print list of regex matching and excluding safe commands.
--show-textcmd Print list of regex matching and excluding commands with text argument.
--show-config Show values of configuration variables
--show-all Show all of the above
NB For all --show commands, no old.tex or new.tex file needs to be given, and no
differencing takes place.
Other configuration options:
--allow-spaces Allow spaces between bracketed or braced arguments to commands
[Default requires arguments to directly follow each other without
intervening spaces]
--math-markup=level Determine granularity of markup in displayed math environments:
Possible values for level are (both numerical and text labels are acceptable):
off or 0: suppress markup for math environments. Deleted equations will not
appear in diff file. This mode can be used if all the other modes
cause invalid latex code.
whole or 1: Differencing on the level of whole equations. Even trivial changes
to equations cause the whole equation to be marked changed. This
mode can be used if processing in coarse or fine mode results in
invalid latex code.
coarse or 2: Detect changes within equations marked up with a coarse
granularity; changes in equation type (e.g.displaymath to equation)
appear as a change to the complete equation. This mode is recommended
for situations where the content and order of some equations are still
being changed. [Default]
fine or 3: Detect small change in equations and mark up and fine granularity.
This mode is most suitable, if only minor changes to equations are
expected, e.g. correction of typos.
--disable-citation-markup
--disable-auto-mbox Suppress citation markup and markup of other vulnerable commands in styles
using ulem (UNDERLINE,FONTSTRIKE, CULINECHBAR)
(the two options are identical and are simply aliases)
--enable-citation-markup
--enforce-auto-mbox Protect citation commands and other vulnerable commands in changed sections
with \\mbox command, i.e. use default behaviour for ulem package for other packages
(the two options are identical and are simply aliases)
Miscelleneous options
--label=label
-L label Sets the labels used to describe the old and new files. The first use
of this option sets the label describing the old file and the second
use of the option sets the label for the new file.
[Default: use the filename and modification dates for the label]
--no-label Suppress inclusion of old and new file names as comment in output file
--visible-label Include old and new filenames (or labels set with --label option) as
visible output
--flatten Replace \\input and \\include commands within body by the content
of the files in their argument. If \\includeonly is present in the
preamble, only those files are expanded into the document. However,
no recursion is done, i.e. \\input and \\include commands within
included sections are not expanded. The included files are assumed to
be located in the same directories as the old and new master files,
respectively, making it possible to organise files into old and new directories.
--flatten is applied recursively, so inputted files can contain further
\\input statements.
--help
-h Show this help text.
--ignore-warnings Suppress warnings about inconsistencies in length between input
and parsed strings and missing characters.
--verbose
-V Output various status information to stderr during processing.
Default is to work silently.
--version Show version number.
EOF
}
=head1 NAME
latexdiff - determine and markup differences between two latex files
=head1 SYNOPSIS
B [ B ] F F > F
=head1 DESCRIPTION
Briefly, I is a utility program to aid in the management of
revisions of latex documents. It compares two valid latex files, here
called C and C, finds significant differences
between them (i.e., ignoring the number of white spaces and position
of line breaks), and adds special commands to highlight the
differences. Where visual highlighting is not possible, e.g. for changes
in the formatting, the differences are
nevertheless marked up in the source.
The program treats the preamble differently from the main document.
Differences between the preambles are found using line-based
differencing (similarly to the Unix diff command, but ignoring white
spaces). A comment, "S>>" is appended to each added line, i.e. a
line present in C but not in C. Discarded lines
are deactivated by prepending "S>>". Changed blocks are preceded by
comment lines giving information about line numbers in the original files. Where there are insignificant
differences, the resulting file C will be similar to
C. At the end of the preamble, the definitions for I markup commands are inserted.
In differencing the main body of the text, I attempts to
satisfy the following guidelines (in order of priority):
=over 3
=item 1
If both C and C are valid LaTeX, then the resulting
C should also be valid LateX. (NB If a few plain TeX commands
are used within C or C then C is not
guaranteed to work but usually will).
=item 2
Significant differences are determined on the level of
individual words. All significant differences, including differences
between comments should be clearly marked in the resulting source code
C.
=item 3
If a changed passage contains text or text-producing commands, then
running C through LateX should produce output where added
and discarded passages are highlighted.
=item 4
Where there are insignificant differences, e.g. in the positioning of
line breaks, C should follow the formatting of C
=back
For differencing the same algorithm as I is used but words
instead of lines are compared. An attempt is made to recognize
blocks which are completely changed such that they can be marked up as a unit.
Comments are differenced line by line
but the number of spaces within comments is ignored. Commands including
all their arguments are generally compared as one unit, i.e., no mark-up
is inserted into the arguments of commands. However, for a selected
number of commands (for example, C<\caption> and all sectioning
commands) the last argument is known to be text. This text is
split into words and differenced just as ordinary text (use options to
show and change the list of text commands, see below). As the
algorithm has no detailed knowledge of LaTeX, it assumes all pairs of
curly braces immediately following a command (i.e. a sequence of
letters beginning with a backslash) are arguments for that command.
As a restriction to condition 1 above it is thus necessary to surround
all arguments with curly braces, and to not insert
extraneous spaces. For example, write
\section{\textem{This is an emphasized section title}}
and not
\section {\textem{This is an emphasized section title}}
or
\section\textem{This is an emphasized section title}
even though all varieties are the same to LaTeX (but see
B<--allow-spaces> option which allows the second variety).
For environments whose content does not conform to standard LaTeX or
where graphical markup does not make sense all markup commands can be
removed by setting the PICTUREENV configuration variable, set by
default to C and C environments; see B<--config>
option). The latter environment (C) can be used to
protect parts of the latex file where the markup results in illegal
markup. You have to surround the offending passage in both the old and
new file by C<\begin{DIFnomarkup}> and C<\end{DIFnomarkup}>. You must
define the environment in the preambles of both old and new
documents. I prefer to define it as a null-environment,
C<\newenvironment{DIFnomarkup}{}{}>
but the choice is yours. Any markup within the environment will be
removed, and generally everything within the environment will just be
taken from the new file.
It is also possible to difference files which do not have a preamble.
In this case, the file is processed in the main document
mode, but the definitions of the markup commands are not inserted.
All markup commands inserted by I begin with "C<\DIF>". Added
blocks containing words, commands or comments which are in C
but not in C are marked by C<\DIFaddbegin> and C<\DIFaddend>.
Discarded blocks are marked by C<\DIFdelbegin> and C<\DIFdelend>.
Within added blocks all text is highlighted with C<\DIFadd> like this:
C<\DIFadd{Added text block}>
Selected `safe' commands can be contained in these text blocks as well
(use options to show and change the list of safe commands, see below).
All other commands as well as braces "{" and "}" are never put within
the scope of C<\DIFadd>. Added comments are marked by prepending
"S >>".
Within deleted blocks text is highlighted with C<\DIFdel>. Deleted
comments are marked by prepending "S >>". Non-safe command
and curly braces within deleted blocks are commented out with
"S >>".
=head1 OPTIONS
=head2 Preamble
The following options determine the visual markup style by adding the appropriate
command definitions to the preamble. See the end of this section for a description of
available styles.
=over 4
=item B<--type=markupstyle> or
B<-t markupstyle>
Add code to preamble for selected markup style. This option defines
C<\DIFadd> and C<\DIFdel> commands.
Available styles:
C
[ Default: C ]
=item B<--subtype=markstyle> or
B<-s markstyle>
Add code to preamble for selected style for bracketing
commands (e.g. to mark changes in margin). This option defines
C<\DIFaddbegin>, C<\DIFaddend>, C<\DIFdelbegin> and C<\DIFdelend> commands.
Available styles: C
[ Default: C ]
* Subtype C