Add files using upload-large-folder tool

mosesdecoder/COPYING

@@ -0,0 +1,460 @@
+
+                  GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 2.1, February 1999
+
+ Copyright (C) 1991, 1999 Free Software Foundation, Inc.
+	51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+[This is the first released version of the Lesser GPL.  It also counts
+ as the successor of the GNU Library Public License, version 2, hence
+ the version number 2.1.]
+
+                            Preamble
+
+  The licenses for most software are designed to take away your
+freedom to share and change it.  By contrast, the GNU General Public
+Licenses are intended to guarantee your freedom to share and change
+free software--to make sure the software is free for all its users.
+
+  This license, the Lesser General Public License, applies to some
+specially designated software packages--typically libraries--of the
+Free Software Foundation and other authors who decide to use it.  You
+can use it too, but we suggest you first think carefully about whether
+this license or the ordinary General Public License is the better
+strategy to use in any particular case, based on the explanations
+below.
+
+  When we speak of free software, we are referring to freedom of use,
+not price.  Our General Public Licenses are designed to make sure that
+you have the freedom to distribute copies of free software (and charge
+for this service if you wish); that you receive source code or can get
+it if you want it; that you can change the software and use pieces of
+it in new free programs; and that you are informed that you can do
+these things.
+
+  To protect your rights, we need to make restrictions that forbid
+distributors to deny you these rights or to ask you to surrender these
+rights.  These restrictions translate to certain responsibilities for
+you if you distribute copies of the library or if you modify it.
+
+  For example, if you distribute copies of the library, whether gratis
+or for a fee, you must give the recipients all the rights that we gave
+you.  You must make sure that they, too, receive or can get the source
+code.  If you link other code with the library, you must provide
+complete object files to the recipients, so that they can relink them
+with the library after making changes to the library and recompiling
+it.  And you must show them these terms so they know their rights.
+
+  We protect your rights with a two-step method: (1) we copyright the
+library, and (2) we offer you this license, which gives you legal
+permission to copy, distribute and/or modify the library.
+
+  To protect each distributor, we want to make it very clear that
+there is no warranty for the free library.  Also, if the library is
+modified by someone else and passed on, the recipients should know
+that what they have is not the original version, so that the original
+author's reputation will not be affected by problems that might be
+introduced by others.
+
+  Finally, software patents pose a constant threat to the existence of
+any free program.  We wish to make sure that a company cannot
+effectively restrict the users of a free program by obtaining a
+restrictive license from a patent holder.  Therefore, we insist that
+any patent license obtained for a version of the library must be
+consistent with the full freedom of use specified in this license.
+
+  Most GNU software, including some libraries, is covered by the
+ordinary GNU General Public License.  This license, the GNU Lesser
+General Public License, applies to certain designated libraries, and
+is quite different from the ordinary General Public License.  We use
+this license for certain libraries in order to permit linking those
+libraries into non-free programs.
+
+  When a program is linked with a library, whether statically or using
+a shared library, the combination of the two is legally speaking a
+combined work, a derivative of the original library.  The ordinary
+General Public License therefore permits such linking only if the
+entire combination fits its criteria of freedom.  The Lesser General
+Public License permits more lax criteria for linking other code with
+the library.
+
+  We call this license the "Lesser" General Public License because it
+does Less to protect the user's freedom than the ordinary General
+Public License.  It also provides other free software developers Less
+of an advantage over competing non-free programs.  These disadvantages
+are the reason we use the ordinary General Public License for many
+libraries.  However, the Lesser license provides advantages in certain
+special circumstances.
+
+  For example, on rare occasions, there may be a special need to
+encourage the widest possible use of a certain library, so that it
+becomes a de-facto standard.  To achieve this, non-free programs must
+be allowed to use the library.  A more frequent case is that a free
+library does the same job as widely used non-free libraries.  In this
+case, there is little to gain by limiting the free library to free
+software only, so we use the Lesser General Public License.
+
+  In other cases, permission to use a particular library in non-free
+programs enables a greater number of people to use a large body of
+free software.  For example, permission to use the GNU C Library in
+non-free programs enables many more people to use the whole GNU
+operating system, as well as its variant, the GNU/Linux operating
+system.
+
+  Although the Lesser General Public License is Less protective of the
+users' freedom, it does ensure that the user of a program that is
+linked with the Library has the freedom and the wherewithal to run
+that program using a modified version of the Library.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.  Pay close attention to the difference between a
+"work based on the library" and a "work that uses the library".  The
+former contains code derived from the library, whereas the latter must
+be combined with the library in order to run.
+
+                  GNU LESSER GENERAL PUBLIC LICENSE
+   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+  0. This License Agreement applies to any software library or other
+program which contains a notice placed by the copyright holder or
+other authorized party saying it may be distributed under the terms of
+this Lesser General Public License (also called "this License").
+Each licensee is addressed as "you".
+
+  A "library" means a collection of software functions and/or data
+prepared so as to be conveniently linked with application programs
+(which use some of those functions and data) to form executables.
+
+  The "Library", below, refers to any such software library or work
+which has been distributed under these terms.  A "work based on the
+Library" means either the Library or any derivative work under
+copyright law: that is to say, a work containing the Library or a
+portion of it, either verbatim or with modifications and/or translated
+straightforwardly into another language.  (Hereinafter, translation is
+included without limitation in the term "modification".)
+
+  "Source code" for a work means the preferred form of the work for
+making modifications to it.  For a library, complete source code means
+all the source code for all modules it contains, plus any associated
+interface definition files, plus the scripts used to control
+compilation and installation of the library.
+
+  Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope.  The act of
+running a program using the Library is not restricted, and output from
+such a program is covered only if its contents constitute a work based
+on the Library (independent of the use of the Library in a tool for
+writing it).  Whether that is true depends on what the Library does
+and what the program that uses the Library does.
+
+  1. You may copy and distribute verbatim copies of the Library's
+complete source code as you receive it, in any medium, provided that
+you conspicuously and appropriately publish on each copy an
+appropriate copyright notice and disclaimer of warranty; keep intact
+all the notices that refer to this License and to the absence of any
+warranty; and distribute a copy of this License along with the
+Library.
+
+  You may charge a fee for the physical act of transferring a copy,
+and you may at your option offer warranty protection in exchange for a
+fee.
+
+  2. You may modify your copy or copies of the Library or any portion
+of it, thus forming a work based on the Library, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+    a) The modified work must itself be a software library.
+
+    b) You must cause the files modified to carry prominent notices
+    stating that you changed the files and the date of any change.
+
+    c) You must cause the whole of the work to be licensed at no
+    charge to all third parties under the terms of this License.
+
+    d) If a facility in the modified Library refers to a function or a
+    table of data to be supplied by an application program that uses
+    the facility, other than as an argument passed when the facility
+    is invoked, then you must make a good faith effort to ensure that,
+    in the event an application does not supply such function or
+    table, the facility still operates, and performs whatever part of
+    its purpose remains meaningful.
+
+    (For example, a function in a library to compute square roots has
+    a purpose that is entirely well-defined independent of the
+    application.  Therefore, Subsection 2d requires that any
+    application-supplied function or table used by this function must
+    be optional: if the application does not supply it, the square
+    root function must still compute square roots.)
+
+These requirements apply to the modified work as a whole.  If
+identifiable sections of that work are not derived from the Library,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works.  But when you
+distribute the same sections as part of a whole which is a work based
+on the Library, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote
+it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Library.
+
+In addition, mere aggregation of another work not based on the Library
+with the Library (or with a work based on the Library) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+  3. You may opt to apply the terms of the ordinary GNU General Public
+License instead of this License to a given copy of the Library.  To do
+this, you must alter all the notices that refer to this License, so
+that they refer to the ordinary GNU General Public License, version 2,
+instead of to this License.  (If a newer version than version 2 of the
+ordinary GNU General Public License has appeared, then you can specify
+that version instead if you wish.)  Do not make any other change in
+these notices.
+
+  Once this change is made in a given copy, it is irreversible for
+that copy, so the ordinary GNU General Public License applies to all
+subsequent copies and derivative works made from that copy.
+
+  This option is useful when you wish to copy part of the code of
+the Library into a program that is not a library.
+
+  4. You may copy and distribute the Library (or a portion or
+derivative of it, under Section 2) in object code or executable form
+under the terms of Sections 1 and 2 above provided that you accompany
+it with the complete corresponding machine-readable source code, which
+must be distributed under the terms of Sections 1 and 2 above on a
+medium customarily used for software interchange.
+
+  If distribution of object code is made by offering access to copy
+from a designated place, then offering equivalent access to copy the
+source code from the same place satisfies the requirement to
+distribute the source code, even though third parties are not
+compelled to copy the source along with the object code.
+
+  5. A program that contains no derivative of any portion of the
+Library, but is designed to work with the Library by being compiled or
+linked with it, is called a "work that uses the Library".  Such a
+work, in isolation, is not a derivative work of the Library, and
+therefore falls outside the scope of this License.
+
+  However, linking a "work that uses the Library" with the Library
+creates an executable that is a derivative of the Library (because it
+contains portions of the Library), rather than a "work that uses the
+library".  The executable is therefore covered by this License.
+Section 6 states terms for distribution of such executables.
+
+  When a "work that uses the Library" uses material from a header file
+that is part of the Library, the object code for the work may be a
+derivative work of the Library even though the source code is not.
+Whether this is true is especially significant if the work can be
+linked without the Library, or if the work is itself a library.  The
+threshold for this to be true is not precisely defined by law.
+
+  If such an object file uses only numerical parameters, data
+structure layouts and accessors, and small macros and small inline
+functions (ten lines or less in length), then the use of the object
+file is unrestricted, regardless of whether it is legally a derivative
+work.  (Executables containing this object code plus portions of the
+Library will still fall under Section 6.)
+
+  Otherwise, if the work is a derivative of the Library, you may
+distribute the object code for the work under the terms of Section 6.
+Any executables containing that work also fall under Section 6,
+whether or not they are linked directly with the Library itself.
+
+  6. As an exception to the Sections above, you may also combine or
+link a "work that uses the Library" with the Library to produce a
+work containing portions of the Library, and distribute that work
+under terms of your choice, provided that the terms permit
+modification of the work for the customer's own use and reverse
+engineering for debugging such modifications.
+
+  You must give prominent notice with each copy of the work that the
+Library is used in it and that the Library and its use are covered by
+this License.  You must supply a copy of this License.  If the work
+during execution displays copyright notices, you must include the
+copyright notice for the Library among them, as well as a reference
+directing the user to the copy of this License.  Also, you must do one
+of these things:
+
+    a) Accompany the work with the complete corresponding
+    machine-readable source code for the Library including whatever
+    changes were used in the work (which must be distributed under
+    Sections 1 and 2 above); and, if the work is an executable linked
+    with the Library, with the complete machine-readable "work that
+    uses the Library", as object code and/or source code, so that the
+    user can modify the Library and then relink to produce a modified
+    executable containing the modified Library.  (It is understood
+    that the user who changes the contents of definitions files in the
+    Library will not necessarily be able to recompile the application
+    to use the modified definitions.)
+
+    b) Use a suitable shared library mechanism for linking with the
+    Library.  A suitable mechanism is one that (1) uses at run time a
+    copy of the library already present on the user's computer system,
+    rather than copying library functions into the executable, and (2)
+    will operate properly with a modified version of the library, if
+    the user installs one, as long as the modified version is
+    interface-compatible with the version that the work was made with.
+
+    c) Accompany the work with a written offer, valid for at least
+    three years, to give the same user the materials specified in
+    Subsection 6a, above, for a charge no more than the cost of
+    performing this distribution.
+
+    d) If distribution of the work is made by offering access to copy
+    from a designated place, offer equivalent access to copy the above
+    specified materials from the same place.
+
+    e) Verify that the user has already received a copy of these
+    materials or that you have already sent this user a copy.
+
+  For an executable, the required form of the "work that uses the
+Library" must include any data and utility programs needed for
+reproducing the executable from it.  However, as a special exception,
+the materials to be distributed need not include anything that is
+normally distributed (in either source or binary form) with the major
+components (compiler, kernel, and so on) of the operating system on
+which the executable runs, unless that component itself accompanies
+the executable.
+
+  It may happen that this requirement contradicts the license
+restrictions of other proprietary libraries that do not normally
+accompany the operating system.  Such a contradiction means you cannot
+use both them and the Library together in an executable that you
+distribute.
+
+  7. You may place library facilities that are a work based on the
+Library side-by-side in a single library together with other library
+facilities not covered by this License, and distribute such a combined
+library, provided that the separate distribution of the work based on
+the Library and of the other library facilities is otherwise
+permitted, and provided that you do these two things:
+
+    a) Accompany the combined library with a copy of the same work
+    based on the Library, uncombined with any other library
+    facilities.  This must be distributed under the terms of the
+    Sections above.
+
+    b) Give prominent notice with the combined library of the fact
+    that part of it is a work based on the Library, and explaining
+    where to find the accompanying uncombined form of the same work.
+
+  8. You may not copy, modify, sublicense, link with, or distribute
+the Library except as expressly provided under this License.  Any
+attempt otherwise to copy, modify, sublicense, link with, or
+distribute the Library is void, and will automatically terminate your
+rights under this License.  However, parties who have received copies,
+or rights, from you under this License will not have their licenses
+terminated so long as such parties remain in full compliance.
+
+  9. You are not required to accept this License, since you have not
+signed it.  However, nothing else grants you permission to modify or
+distribute the Library or its derivative works.  These actions are
+prohibited by law if you do not accept this License.  Therefore, by
+modifying or distributing the Library (or any work based on the
+Library), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Library or works based on it.
+
+  10. Each time you redistribute the Library (or any work based on the
+Library), the recipient automatically receives a license from the
+original licensor to copy, distribute, link with or modify the Library
+subject to these terms and conditions.  You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties with
+this License.
+
+  11. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Library at all.  For example, if a patent
+license would not permit royalty-free redistribution of the Library by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Library.
+
+If any portion of this section is held invalid or unenforceable under
+any particular circumstance, the balance of the section is intended to
+apply, and the section as a whole is intended to apply in other
+circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system which is
+implemented by public license practices.  Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+
+  12. If the distribution and/or use of the Library is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Library under this License
+may add an explicit geographical distribution limitation excluding those
+countries, so that distribution is permitted only in or among
+countries not thus excluded.  In such case, this License incorporates
+the limitation as if written in the body of this License.
+
+  13. The Free Software Foundation may publish revised and/or new
+versions of the Lesser General Public License from time to time.
+Such new versions will be similar in spirit to the present version,
+but may differ in detail to address new problems or concerns.
+
+Each version is given a distinguishing version number.  If the Library
+specifies a version number of this License which applies to it and
+"any later version", you have the option of following the terms and
+conditions either of that version or of any later version published by
+the Free Software Foundation.  If the Library does not specify a
+license version number, you may choose any version ever published by
+the Free Software Foundation.
+
+  14. If you wish to incorporate parts of the Library into other free
+programs whose distribution conditions are incompatible with these,
+write to the author to ask for permission.  For software which is
+copyrighted by the Free Software Foundation, write to the Free
+Software Foundation; we sometimes make exceptions for this.  Our
+decision will be guided by the two goals of preserving the free status
+of all derivatives of our free software and of promoting the sharing
+and reuse of software generally.
+
+                            NO WARRANTY
+
+  15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO
+WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW.
+EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR
+OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY
+KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE
+LIBRARY IS WITH YOU.  SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME
+THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN
+WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY
+AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU
+FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR
+CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE
+LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING
+RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A
+FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF
+SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
+DAMAGES.
+

mosesdecoder/bjam

@@ -0,0 +1,23 @@
+#!/bin/bash
+set -e
+top="$(dirname "$0")"
+if
+  bjam="$(which bjam 2>/dev/null)" && #exists
+  [ ${#bjam} != 0 ] && #paranoia about which printing nothing then returning true
+  ! grep UFIHGUFIHBDJKNCFZXAEVA "${bjam}" </dev/null >/dev/null && #bjam in path isn't this script
+  "${bjam}" --sanity-test 2>/dev/null |grep Sane >/dev/null && #The test in jam-files/sanity.jam passes
+  (cd "${top}/jam-files/fail" && ! "${bjam}") >/dev/null #Returns non-zero on failure
+then
+  #Delegate to system bjam
+  exec "${bjam}" "$@"
+fi
+
+if [ ! -x "$top"/jam-files/bjam ] || "$top"/jam-files/bjam -v |grep 2011.4 >/dev/null; then
+  pushd "$top/jam-files/engine"
+  ./build.sh
+  cp -f bin.*/bjam ../bjam
+  popd
+fi
+
+export BOOST_BUILD_PATH="$top"/jam-files/boost-build 
+exec "$top"/jam-files/bjam "$@"

mosesdecoder/moses2/AlignmentInfoCollection.h

@@ -0,0 +1,81 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#pragma once
+
+#include "AlignmentInfo.h"
+
+#include <set>
+
+#ifdef WITH_THREADS
+#include <boost/thread/shared_mutex.hpp>
+#include <boost/thread/locks.hpp>
+#endif
+
+namespace Moses2
+{
+
+/** Singleton collection of all AlignmentInfo objects.
+ *  Used as a cache of all alignment info to save space.
+ */
+class AlignmentInfoCollection
+{
+public:
+  static AlignmentInfoCollection &Instance() {
+    return s_instance;
+  }
+
+  /** Returns a pointer to an AlignmentInfo object with the same source-target
+    * alignment pairs as given in the argument.  If the collection already
+    * contains such an object then returns a pointer to it; otherwise a new
+    * one is inserted.
+   */
+private:
+  const AlignmentInfo* Add(AlignmentInfo const& ainfo);
+
+public:
+  template<typename ALNREP>
+  AlignmentInfo const *
+  Add(ALNREP const & aln) {
+    return this->Add(AlignmentInfo(aln));
+  }
+
+  //! Returns a pointer to an empty AlignmentInfo object.
+  const AlignmentInfo &GetEmptyAlignmentInfo() const;
+
+private:
+  typedef std::set<AlignmentInfo, AlignmentInfoOrderer> AlignmentInfoSet;
+
+
+  //! Only a single static variable should be created.
+  AlignmentInfoCollection();
+  ~AlignmentInfoCollection();
+
+  static AlignmentInfoCollection s_instance;
+
+#ifdef WITH_THREADS
+  //reader-writer lock
+  mutable boost::shared_mutex m_accessLock;
+#endif
+
+  AlignmentInfoSet m_collection;
+  const AlignmentInfo *m_emptyAlignmentInfo;
+};
+
+}

mosesdecoder/moses2/ArcLists.h

@@ -0,0 +1,44 @@
+/*
+ * ArcList.h
+ *
+ *  Created on: 26 Oct 2015
+ *      Author: hieu
+ */
+#pragma once
+#include <vector>
+#include <unordered_map>
+#include <unordered_set>
+
+namespace Moses2
+{
+class System;
+
+class HypothesisBase;
+
+typedef std::vector<const HypothesisBase*> ArcList;
+
+class ArcLists
+{
+public:
+  ArcLists();
+  virtual ~ArcLists();
+
+  void AddArc(bool added, const HypothesisBase *currHypo,
+              const HypothesisBase *otherHypo);
+  void Sort();
+  void Delete(const HypothesisBase *hypo);
+
+  const ArcList &GetArcList(const HypothesisBase *hypo) const;
+
+  std::string Debug(const System &system) const;
+protected:
+  typedef std::unordered_map<const HypothesisBase*, ArcList*> Coll;
+  Coll m_coll;
+
+  ArcList &GetArcList(const HypothesisBase *hypo);
+  ArcList &GetAndDetachArcList(const HypothesisBase *hypo);
+
+};
+
+}
+

mosesdecoder/moses2/EstimatedScores.cpp

@@ -0,0 +1,117 @@
+// $Id$
+// vim:tabstop=2
+
+/***********************************************************************
+ Moses - factored phrase-based language decoder
+ Copyright (C) 2006 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+ ***********************************************************************/
+
+#include <string>
+#include <iostream>
+#include "EstimatedScores.h"
+
+using namespace std;
+
+namespace Moses2
+{
+/**
+ * Calculate future score estimate for a given coverage bitmap
+ *
+ * /param bitmap coverage bitmap
+ */
+
+float EstimatedScores::CalcEstimatedScore(Bitmap const &bitmap) const
+{
+  const size_t notInGap = numeric_limits<size_t>::max();
+  size_t startGap = notInGap;
+  float estimatedScore = 0.0f;
+  for (size_t currPos = 0; currPos < bitmap.GetSize(); currPos++) {
+    // start of a new gap?
+    if (bitmap.GetValue(currPos) == false && startGap == notInGap) {
+      startGap = currPos;
+    }
+    // end of a gap?
+    else if (bitmap.GetValue(currPos) == true && startGap != notInGap) {
+      estimatedScore += GetValue(startGap, currPos - 1);
+      startGap = notInGap;
+    }
+  }
+  // coverage ending with gap?
+  if (startGap != notInGap) {
+    estimatedScore += GetValue(startGap, bitmap.GetSize() - 1);
+  }
+
+  return estimatedScore;
+}
+
+/**
+ * Calculare future score estimate for a given coverage bitmap
+ * and an additional span that is also covered. This function is used
+ * to compute future score estimates for hypotheses that we may want
+ * build, but first want to check.
+ *
+ * Note: this function is implemented a bit more complex than
+ * the basic one (w/o additional phrase) for speed reasons,
+ * which is probably overkill.
+ *
+ * /param bitmap coverage bitmap
+ * /param startPos start of the span that is added to the coverage
+ * /param endPos end of the span that is added to the coverage
+ */
+
+float EstimatedScores::CalcEstimatedScore(Bitmap const &bitmap, size_t startPos,
+    size_t endPos) const
+{
+  const size_t notInGap = numeric_limits<size_t>::max();
+  float estimatedScore = 0.0f;
+  size_t startGap = bitmap.GetFirstGapPos();
+  if (startGap == NOT_FOUND) return estimatedScore; // everything filled
+
+  // start loop at first gap
+  size_t startLoop = startGap + 1;
+  if (startPos == startGap) { // unless covered by phrase
+    startGap = notInGap;
+    startLoop = endPos + 1; // -> postpone start
+  }
+
+  size_t lastCovered = bitmap.GetLastPos();
+  if (endPos > lastCovered || lastCovered == NOT_FOUND) lastCovered = endPos;
+
+  for (size_t currPos = startLoop; currPos <= lastCovered; currPos++) {
+    // start of a new gap?
+    if (startGap == notInGap && bitmap.GetValue(currPos) == false
+        && (currPos < startPos || currPos > endPos)) {
+      startGap = currPos;
+    }
+    // end of a gap?
+    else if (startGap != notInGap
+             && (bitmap.GetValue(currPos) == true
+                 || (startPos <= currPos && currPos <= endPos))) {
+      estimatedScore += GetValue(startGap, currPos - 1);
+      startGap = notInGap;
+    }
+  }
+  // coverage ending with gap?
+  if (lastCovered != bitmap.GetSize() - 1) {
+    estimatedScore += GetValue(lastCovered + 1, bitmap.GetSize() - 1);
+  }
+
+  return estimatedScore;
+}
+
+}
+

mosesdecoder/moses2/InputPathBase.h

@@ -0,0 +1,32 @@
+/*
+ * InputPath.h
+ *
+ *  Created on: 23 Oct 2015
+ *      Author: hieu
+ */
+
+#pragma once
+
+#include <iostream>
+#include <vector>
+#include "SubPhrase.h"
+#include "legacy/Range.h"
+
+namespace Moses2
+{
+
+class PhraseTable;
+
+class InputPathBase
+{
+public:
+  const InputPathBase *prefixPath;
+  Range range;
+
+  InputPathBase(MemPool &pool, const Range &range,
+                size_t numPt, const InputPathBase *prefixPath);
+
+};
+
+}
+

mosesdecoder/moses2/InputType.h

@@ -0,0 +1,86 @@
+/*
+ * InputType.h
+ *
+ *  Created on: 14 Dec 2015
+ *      Author: hieu
+ */
+
+#pragma once
+
+#include "PhraseBased/ReorderingConstraint.h"
+#include "TypeDef.h"
+
+namespace Moses2
+{
+
+class InputType
+{
+public:
+  //////////////////////////////////////////////////////////////////////////////
+  class XMLOption
+  {
+  public:
+    size_t startPos, phraseSize;
+
+    SCORE prob;
+
+    XMLOption(MemPool &pool, const std::string &nodeName, size_t vStartPos);
+
+    const char *GetNodeName() const {
+      return m_nodeName;
+    }
+
+    const char *GetTranslation() const {
+      return m_translation;
+    }
+
+    const char *GetEntity() const {
+      return m_entity;
+    }
+
+    void SetTranslation(MemPool &pool, const std::string &val);
+    void SetEntity(MemPool &pool, const std::string &val);
+
+    std::string Debug(const System &system) const;
+  public:
+    char *m_nodeName;
+    char *m_translation;
+    char *m_entity;
+
+  };
+
+  //////////////////////////////////////////////////////////////////////////////
+
+  InputType(MemPool &pool);
+  virtual ~InputType();
+
+  virtual void Init(const System &system, size_t size, int max_distortion);
+
+  ReorderingConstraint &GetReorderingConstraint() {
+    return m_reorderingConstraint;
+  }
+
+  const ReorderingConstraint &GetReorderingConstraint() const {
+    return m_reorderingConstraint;
+  }
+
+  const Vector<const XMLOption*> &GetXMLOptions() const {
+    return m_xmlOptions;
+  }
+
+  void AddXMLOption(const System &system, const XMLOption *xmlOption);
+
+  //! Returns true if there were any XML tags parsed that at least partially covered the range passed
+  bool XmlOverlap(size_t startPos, size_t endPos) const;
+
+  virtual std::string Debug(const System &system) const;
+
+protected:
+  ReorderingConstraint m_reorderingConstraint; /**< limits on reordering specified either by "-mp" switch or xml tags */
+  Vector<const XMLOption*> m_xmlOptions;
+  Vector<bool> m_xmlCoverageMap;
+
+};
+
+} /* namespace Moses2 */
+

mosesdecoder/moses2/Moses2Wrapper.cpp

@@ -0,0 +1,70 @@
+#include "Moses2Wrapper.h"
+#include "System.h"
+#include "legacy/Parameter.h"
+#include "TranslationTask.h"
+#include <string.h>
+
+using namespace std;
+namespace Moses2 {
+	//summary ::  need to update the LM path at runtime with complete artifact path.
+	void Moses2Wrapper::UpdateLMPath(const std::string& filePath) {
+
+		char sep = '/';
+
+		#ifdef _WIN32
+				sep = '\\';
+		#endif
+		auto file = filePath.substr(filePath.find_last_of(sep) + 1);
+		auto path = filePath.substr(0, filePath.find_last_of(sep));
+		auto a = m_param->GetParam("feature");
+		std::vector<std::string> feature;
+		for (int i = 0; i < a->size(); i++) {
+			auto abc = Tokenize(a->at(i));
+			if (*abc.begin() == "KENLM") {
+				string s = "";
+				for (int k = 0; k < abc.size(); k++) {
+					if (abc.at(k).find("path=") != string::npos) {
+						auto lm = abc.at(k).substr(abc.at(k).find_last_of("=") + 1);
+						s = s + "path=" + path + sep + lm + " ";
+					}
+					else {
+						s = s + abc.at(k) + " ";
+					}
+				}
+				feature.push_back(s.erase(s.find_last_not_of(" \n\r\t") + 1));
+			}
+			else {
+				feature.push_back(a->at(i));
+			}
+		}
+		m_param->OverwriteParam("feature", feature);
+	}
+
+	Moses2Wrapper::Moses2Wrapper(const std::string &filePath) {
+		m_param = new Parameter();
+		m_param->LoadParam(filePath);
+		UpdateLMPath(filePath);
+		m_system = new System(*m_param);
+	}
+
+	std::string Moses2Wrapper::Translate(const std::string &input , long id, bool nbest) {
+		TranslationTask task(*m_system, input, id);
+		return task.ReturnTranslation(nbest);
+	}
+	Moses2Wrapper::~Moses2Wrapper() {
+		delete m_param;
+		delete  m_system;
+	}
+
+	char* Moses2Wrapper::CopyString(const char* str) {
+		int32_t size = (int32_t)strlen(str);
+		char* obj = (char*)malloc(size + 1);
+		memcpy(obj, str, size);
+		obj[size] = '\0';
+		return obj;
+	}
+	void Moses2Wrapper::Free(void* ptr) {
+		free(ptr);
+	}
+
+}

mosesdecoder/moses2/Phrase.cpp

@@ -0,0 +1,23 @@
+/*
+ * PhraseImpl.cpp
+ *
+ *  Created on: 23 Oct 2015
+ *      Author: hieu
+ */
+#include <boost/functional/hash.hpp>
+#include "Phrase.h"
+#include "Word.h"
+#include "MemPool.h"
+#include "Scores.h"
+#include "System.h"
+
+using namespace std;
+
+namespace Moses2
+{
+
+
+
+
+} // namespace
+

mosesdecoder/moses2/Scores.cpp

@@ -0,0 +1,283 @@
+/*
+ * Scores.cpp
+ *
+ *  Created on: 23 Oct 2015
+ *      Author: hieu
+ */
+
+#include <boost/foreach.hpp>
+#include <vector>
+#include <cstddef>
+#include <stdio.h>
+#include "Scores.h"
+#include "Weights.h"
+#include "System.h"
+#include "FF/FeatureFunction.h"
+#include "FF/FeatureFunctions.h"
+#include "legacy/Util2.h"
+
+using namespace std;
+
+namespace Moses2
+{
+
+Scores::Scores(const System &system, MemPool &pool, size_t numScores) :
+  m_total(0)
+{
+  if (system.options.nbest.nbest_size) {
+    m_scores = new (pool.Allocate<SCORE>(numScores)) SCORE[numScores];
+    Init<SCORE>(m_scores, numScores, 0);
+  } else {
+    m_scores = NULL;
+  }
+}
+
+Scores::Scores(const System &system, MemPool &pool, size_t numScores,
+               const Scores &origScores) :
+  m_total(origScores.m_total)
+{
+  if (system.options.nbest.nbest_size) {
+    m_scores = new (pool.Allocate<SCORE>(numScores)) SCORE[numScores];
+    memcpy(m_scores, origScores.m_scores, sizeof(SCORE) * numScores);
+  } else {
+    m_scores = NULL;
+  }
+}
+
+Scores::~Scores()
+{
+
+}
+
+const SCORE *Scores::GetScores(const FeatureFunction &featureFunction) const
+{
+  assert(m_scores);
+  size_t ffStartInd = featureFunction.GetStartInd();
+  const SCORE &scores = m_scores[ffStartInd];
+  return &scores;
+}
+
+void Scores::Reset(const System &system)
+{
+  if (system.options.nbest.nbest_size) {
+    size_t numScores = system.featureFunctions.GetNumScores();
+    Init<SCORE>(m_scores, numScores, 0);
+  }
+  m_total = 0;
+}
+
+void Scores::PlusEquals(const System &system,
+                        const FeatureFunction &featureFunction, const SCORE &score)
+{
+  assert(featureFunction.GetNumScores() == 1);
+
+  const Weights &weights = system.weights;
+
+  size_t ffStartInd = featureFunction.GetStartInd();
+  if (system.options.nbest.nbest_size) {
+    m_scores[ffStartInd] += score;
+  }
+  SCORE weight = weights[ffStartInd];
+  m_total += score * weight;
+}
+
+void Scores::PlusEquals(const System &system,
+                        const FeatureFunction &featureFunction, const SCORE &score, size_t offset)
+{
+  assert(offset < featureFunction.GetNumScores());
+
+  const Weights &weights = system.weights;
+
+  size_t ffStartInd = featureFunction.GetStartInd();
+  if (system.options.nbest.nbest_size) {
+    m_scores[ffStartInd + offset] += score;
+  }
+  SCORE weight = weights[ffStartInd + offset];
+  m_total += score * weight;
+}
+
+void Scores::PlusEquals(const System &system,
+                        const FeatureFunction &featureFunction, const std::vector<SCORE> &scores)
+{
+  assert(scores.size() == featureFunction.GetNumScores());
+
+  const Weights &weights = system.weights;
+
+  size_t ffStartInd = featureFunction.GetStartInd();
+  for (size_t i = 0; i < scores.size(); ++i) {
+    SCORE incrScore = scores[i];
+    if (system.options.nbest.nbest_size) {
+      m_scores[ffStartInd + i] += incrScore;
+    }
+    //cerr << "ffStartInd=" << ffStartInd << " " << i << endl;
+    SCORE weight = weights[ffStartInd + i];
+    m_total += incrScore * weight;
+  }
+}
+
+void Scores::PlusEquals(const System &system,
+                        const FeatureFunction &featureFunction, SCORE scores[])
+{
+  //assert(scores.size() == featureFunction.GetNumScores());
+
+  const Weights &weights = system.weights;
+
+  size_t ffStartInd = featureFunction.GetStartInd();
+  for (size_t i = 0; i < featureFunction.GetNumScores(); ++i) {
+    SCORE incrScore = scores[i];
+    if (system.options.nbest.nbest_size) {
+      m_scores[ffStartInd + i] += incrScore;
+    }
+    //cerr << "ffStartInd=" << ffStartInd << " " << i << endl;
+    SCORE weight = weights[ffStartInd + i];
+    m_total += incrScore * weight;
+  }
+}
+
+void Scores::PlusEquals(const System &system, const Scores &other)
+{
+  size_t numScores = system.featureFunctions.GetNumScores();
+  if (system.options.nbest.nbest_size) {
+    for (size_t i = 0; i < numScores; ++i) {
+      m_scores[i] += other.m_scores[i];
+    }
+  }
+  m_total += other.m_total;
+}
+
+void Scores::MinusEquals(const System &system, const Scores &other)
+{
+  size_t numScores = system.featureFunctions.GetNumScores();
+  if (system.options.nbest.nbest_size) {
+    for (size_t i = 0; i < numScores; ++i) {
+      m_scores[i] -= other.m_scores[i];
+    }
+  }
+  m_total -= other.m_total;
+}
+
+void Scores::Assign(const System &system,
+                    const FeatureFunction &featureFunction, const SCORE &score)
+{
+  assert(featureFunction.GetNumScores() == 1);
+
+  const Weights &weights = system.weights;
+
+  size_t ffStartInd = featureFunction.GetStartInd();
+
+  if (system.options.nbest.nbest_size) {
+    assert(m_scores[ffStartInd] == 0);
+    m_scores[ffStartInd] = score;
+  }
+  SCORE weight = weights[ffStartInd];
+  m_total += score * weight;
+
+}
+
+void Scores::Assign(const System &system,
+                    const FeatureFunction &featureFunction, const std::vector<SCORE> &scores)
+{
+  assert(scores.size() == featureFunction.GetNumScores());
+
+  const Weights &weights = system.weights;
+
+  size_t ffStartInd = featureFunction.GetStartInd();
+  for (size_t i = 0; i < scores.size(); ++i) {
+    SCORE incrScore = scores[i];
+
+    if (system.options.nbest.nbest_size) {
+      assert(m_scores[ffStartInd + i] == 0);
+      m_scores[ffStartInd + i] = incrScore;
+    }
+    //cerr << "ffStartInd=" << ffStartInd << " " << i << endl;
+    SCORE weight = weights[ffStartInd + i];
+    m_total += incrScore * weight;
+  }
+}
+
+void Scores::CreateFromString(const std::string &str,
+                              const FeatureFunction &featureFunction, const System &system,
+                              bool transformScores)
+{
+  vector<SCORE> scores = Tokenize<SCORE>(str);
+  if (transformScores) {
+    std::transform(scores.begin(), scores.end(), scores.begin(),
+                   TransformScore);
+    std::transform(scores.begin(), scores.end(), scores.begin(), FloorScore);
+  }
+
+  /*
+   std::copy(scores.begin(),scores.end(),
+   std::ostream_iterator<SCORE>(cerr," "));
+   */
+
+  PlusEquals(system, featureFunction, scores);
+}
+
+std::string Scores::Debug(const System &system) const
+{
+  stringstream out;
+  out << "total=" << m_total;
+
+  if (system.options.nbest.nbest_size) {
+    out << ", ";
+    BOOST_FOREACH(const FeatureFunction *ff, system.featureFunctions.GetFeatureFunctions()) {
+      out << ff->GetName() << "= ";
+      for (size_t i = ff->GetStartInd(); i < (ff->GetStartInd() + ff->GetNumScores()); ++i) {
+        out << m_scores[i] << " ";
+      }
+    }
+  }
+
+  return out.str();
+}
+
+void Scores::OutputBreakdown(std::ostream &out, const System &system) const
+{
+  if (system.options.nbest.nbest_size) {
+    BOOST_FOREACH(const FeatureFunction *ff, system.featureFunctions.GetFeatureFunctions()) {
+      if (ff->IsTuneable()) {
+        out << ff->GetName() << "= ";
+        for (size_t i = ff->GetStartInd(); i < (ff->GetStartInd() + ff->GetNumScores()); ++i) {
+          out << m_scores[i] << " ";
+        }
+      }
+    }
+  }
+}
+
+// static functions to work out estimated scores
+SCORE Scores::CalcWeightedScore(const System &system,
+                                const FeatureFunction &featureFunction, SCORE scores[])
+{
+  SCORE ret = 0;
+
+  const Weights &weights = system.weights;
+
+  size_t ffStartInd = featureFunction.GetStartInd();
+  for (size_t i = 0; i < featureFunction.GetNumScores(); ++i) {
+    SCORE incrScore = scores[i];
+
+    //cerr << "ffStartInd=" << ffStartInd << " " << i << endl;
+    SCORE weight = weights[ffStartInd + i];
+    ret += incrScore * weight;
+  }
+
+  return ret;
+}
+
+SCORE Scores::CalcWeightedScore(const System &system,
+                                const FeatureFunction &featureFunction, SCORE score)
+{
+  const Weights &weights = system.weights;
+  assert(featureFunction.GetNumScores() == 1);
+
+  size_t ffStartInd = featureFunction.GetStartInd();
+  SCORE weight = weights[ffStartInd];
+  SCORE ret = score * weight;
+
+  return ret;
+}
+
+}
+

mosesdecoder/moses2/System.cpp

@@ -0,0 +1,262 @@
+/*
+ * System.cpp
+ *
+ *  Created on: 23 Oct 2015
+ *      Author: hieu
+ */
+#include <string>
+#include <iostream>
+#include <boost/foreach.hpp>
+#include <boost/thread.hpp>
+#include <boost/thread/mutex.hpp>
+#include "System.h"
+#include "FF/FeatureFunction.h"
+#include "TranslationModel/UnknownWordPenalty.h"
+#include "legacy/Util2.h"
+#include "util/exception.hh"
+
+using namespace std;
+
+namespace Moses2
+{
+#ifndef WIN32
+  thread_local MemPool System::m_managerPool;
+  thread_local MemPool System::m_systemPool;
+  thread_local Recycler<HypothesisBase*> System::m_hypoRecycler;
+#endif // WIN32
+
+System::System(const Parameter &paramsArg) :
+  params(paramsArg), featureFunctions(*this)
+{
+  options.init(paramsArg);
+  IsPb();
+
+  bestCollector.reset(new OutputCollector());
+
+  params.SetParameter(cpuAffinityOffset, "cpu-affinity-offset", -1);
+  params.SetParameter(cpuAffinityOffsetIncr, "cpu-affinity-increment", 1);
+
+  const PARAM_VEC *section;
+
+  // output collectors
+  if (options.nbest.nbest_size && options.nbest.output_file_path != "-") {
+    nbestCollector.reset(new OutputCollector(options.nbest.output_file_path));
+  }
+
+  if (!options.output.detailed_transrep_filepath.empty()) {
+    detailedTranslationCollector.reset(new OutputCollector(options.output.detailed_transrep_filepath));
+  }
+
+  featureFunctions.Create();
+  LoadWeights();
+
+  if (params.GetParam("show-weights")) {
+    cerr << "Showing weights then exit" << endl;
+    featureFunctions.ShowWeights(weights);
+    //return;
+  }
+
+  cerr << "START featureFunctions.Load()" << endl;
+  featureFunctions.Load();
+  cerr << "START LoadMappings()" << endl;
+  LoadMappings();
+  cerr << "END LoadMappings()" << endl;
+  LoadDecodeGraphBackoff();
+  cerr << "END LoadDecodeGraphBackoff()" << endl;
+
+  UTIL_THROW_IF2(options.input.xml_policy == XmlConstraint, "XmlConstraint not supported");
+
+  // max spans for scfg decoding
+  if (!isPb) {
+    section = params.GetParam("max-chart-span");
+    if (section && section->size()) {
+      maxChartSpans = Scan<size_t>(*section);
+      maxChartSpans.resize(mappings.size(), DEFAULT_MAX_CHART_SPAN);
+
+      /*
+      cerr << "maxChartSpans=" << maxChartSpans.size();
+      for (size_t i = 0; i < maxChartSpans.size(); ++i) {
+          cerr << " " << mappings[i]->GetName() << "=" << maxChartSpans[i];
+      }
+      cerr << endl;
+      */
+    }
+  }
+
+}
+
+System::~System()
+{
+}
+
+void System::LoadWeights()
+{
+  weights.Init(featureFunctions);
+
+  //cerr << "Weights:" << endl;
+  typedef std::map<std::string, std::vector<float> > WeightMap;
+  const WeightMap &allWeights = params.GetAllWeights();
+
+  // check all weights are there for all FF
+  const std::vector<FeatureFunction*> &ffs = featureFunctions.GetFeatureFunctions();
+  BOOST_FOREACH(const FeatureFunction *ff, ffs) {
+    if (ff->IsTuneable()) {
+      const std::string &ffName = ff->GetName();
+      WeightMap::const_iterator iterWeight = allWeights.find(ffName);
+      UTIL_THROW_IF2(iterWeight == allWeights.end(), "Must specify weight for " << ffName);
+    }
+  }
+
+
+  // set weight
+  BOOST_FOREACH(const WeightMap::value_type &valPair, allWeights) {
+    const string &ffName = valPair.first;
+    const std::vector<float> &ffWeights = valPair.second;
+    /*
+    cerr << ffName << "=";
+    for (size_t i = 0; i < ffWeights.size(); ++i) {
+      cerr << ffWeights[i] << " ";
+    }
+    cerr << endl;
+    */
+    weights.SetWeights(featureFunctions, ffName, ffWeights);
+  }
+}
+
+void System::LoadMappings()
+{
+  const PARAM_VEC *vec = params.GetParam("mapping");
+  UTIL_THROW_IF2(vec == NULL, "Must have [mapping] section");
+
+  BOOST_FOREACH(const std::string &line, *vec) {
+    vector<string> toks = Tokenize(line);
+    assert( (toks.size() == 2 && toks[0] == "T") || (toks.size() == 3 && toks[1] == "T") );
+
+    size_t ptInd;
+    if (toks.size() == 2) {
+      ptInd = Scan<size_t>(toks[1]);
+    } else {
+      ptInd = Scan<size_t>(toks[2]);
+    }
+    const PhraseTable *pt = featureFunctions.GetPhraseTableExcludeUnknownWordPenalty(ptInd);
+    mappings.push_back(pt);
+  }
+
+// unk pt
+  const UnknownWordPenalty *unkWP = featureFunctions.GetUnknownWordPenalty();
+  if (unkWP) {
+    mappings.push_back(unkWP);
+  }
+}
+
+void System::LoadDecodeGraphBackoff()
+{
+  const PARAM_VEC *vec = params.GetParam("decoding-graph-backoff");
+
+  for (size_t i = 0; i < mappings.size(); ++i) {
+    PhraseTable *pt = const_cast<PhraseTable*>(mappings[i]);
+
+    if (vec && vec->size() < i) {
+      pt->decodeGraphBackoff = Scan<int>((*vec)[i]);
+    } else if (pt == featureFunctions.GetUnknownWordPenalty()) {
+      pt->decodeGraphBackoff = 1;
+    } else {
+      pt->decodeGraphBackoff = 0;
+    }
+  }
+}
+
+void System::IsPb()
+{
+  switch (options.search.algo) {
+  case Normal:
+  case NormalBatch:
+  case CubePruning:
+  case CubePruningPerMiniStack:
+  case CubePruningPerBitmap:
+  case CubePruningCardinalStack:
+  case CubePruningBitmapStack:
+  case CubePruningMiniStack:
+    isPb = true;
+    break;
+  case CYKPlus:
+    isPb = false;
+    break;
+  default:
+    throw std::runtime_error("Unknown search algorithm " + options.search.algo);
+    break;
+  }
+}
+
+FactorCollection& System::GetVocab() const
+{
+  return m_vocab;
+}
+
+//////////////////////////////////////////////////////
+// thread local stuff
+Batch& System::GetBatch(MemPool& pool) const
+{
+  Batch* obj;
+  obj = m_batch.get();
+  if (obj == NULL) {
+    obj = new Batch(pool);
+    m_batch.reset(obj);
+  }
+  assert(obj);
+  return *obj;
+}
+
+#ifdef WIN32
+template<class C>
+C& GetThreadSpecificObj(boost::thread_specific_ptr<C> &threadSpecificPtr)
+{
+  C* obj;
+  obj = threadSpecificPtr.get();
+  if (obj == NULL) {
+    obj = new C();
+    threadSpecificPtr.reset(obj);
+  }
+  assert(obj);
+  return *obj;
+}
+
+MemPool& System::GetManagerPool() const
+{
+  MemPool &obj = GetThreadSpecificObj<MemPool>(m_managerPool);
+  return obj;
+}
+
+MemPool& System::GetSystemPool() const
+{
+  MemPool& obj = GetThreadSpecificObj<MemPool>(m_systemPool);
+  return obj;
+}
+
+Recycler<HypothesisBase*>& System::GetHypoRecycler() const
+{
+  Recycler<HypothesisBase*>& obj = GetThreadSpecificObj<Recycler<HypothesisBase*> >(m_hypoRecycler);
+  return obj;
+}
+
+#else
+MemPool& System::GetManagerPool() const
+{
+  return m_managerPool;
+}
+
+MemPool& System::GetSystemPool() const
+{
+  return m_systemPool;
+}
+
+Recycler<HypothesisBase*>& System::GetHypoRecycler() const
+{
+  return m_hypoRecycler;
+}
+
+#endif
+
+
+}
+

mosesdecoder/moses2/System.h

@@ -0,0 +1,90 @@
+/*
+ * System.h
+ *
+ *  Created on: 23 Oct 2015
+ *      Author: hieu
+ */
+
+#pragma once
+#include <vector>
+#include <deque>
+#include <boost/thread/tss.hpp>
+#include <boost/pool/object_pool.hpp>
+#include <boost/shared_ptr.hpp>
+#include "FF/FeatureFunctions.h"
+#include "Weights.h"
+#include "MemPool.h"
+#include "Recycler.h"
+#include "legacy/FactorCollection.h"
+#include "legacy/Parameter.h"
+#include "TypeDef.h"
+#include "legacy/Bitmaps.h"
+#include "legacy/OutputCollector.h"
+#include "parameters/AllOptions.h"
+
+namespace Moses2
+{
+namespace NSCubePruning
+{
+class Stack;
+}
+
+class FeatureFunction;
+class StatefulFeatureFunction;
+class PhraseTable;
+class HypothesisBase;
+
+class System
+{
+public:
+  const Parameter &params;
+  AllOptions options;
+  FeatureFunctions featureFunctions;
+  Weights weights;
+  std::vector<const PhraseTable*> mappings;
+
+  std::vector<size_t> maxChartSpans;
+  bool isPb;
+
+  mutable boost::shared_ptr<OutputCollector> bestCollector, nbestCollector, detailedTranslationCollector;
+
+  // moses.ini params
+  int cpuAffinityOffset;
+  int cpuAffinityOffsetIncr;
+
+  System(const Parameter &paramsArg);
+  virtual ~System();
+
+  MemPool &GetSystemPool() const;
+  MemPool &GetManagerPool() const;
+  FactorCollection &GetVocab() const;
+
+  Recycler<HypothesisBase*> &GetHypoRecycler() const;
+
+  Batch &GetBatch(MemPool &pool) const;
+
+protected:
+  mutable FactorCollection m_vocab;
+
+  mutable boost::thread_specific_ptr<Batch> m_batch;
+
+#ifdef WIN32
+  mutable boost::thread_specific_ptr<MemPool> m_managerPool;
+  mutable boost::thread_specific_ptr<MemPool> m_systemPool;
+  mutable boost::thread_specific_ptr <Recycler<HypothesisBase*> > m_hypoRecycler;
+#else
+  thread_local static MemPool m_managerPool;
+  thread_local static MemPool m_systemPool;
+  thread_local static Recycler<HypothesisBase*> m_hypoRecycler;
+#endif
+
+  void LoadWeights();
+  void LoadMappings();
+  void LoadDecodeGraphBackoff();
+
+  void IsPb();
+
+};
+
+}
+

mosesdecoder/moses2/TargetPhrase.h

@@ -0,0 +1,164 @@
+/*
+ * TargetPhrase.h
+ *
+ *  Created on: 26 Apr 2016
+ *      Author: hieu
+ */
+
+#pragma once
+#include <sstream>
+#include "PhraseImplTemplate.h"
+#include "System.h"
+#include "Scores.h"
+#include "AlignmentInfoCollection.h"
+#include "TranslationModel/PhraseTable.h"
+
+namespace Moses2
+{
+class AlignmentInfo;
+
+template<typename WORD>
+class TargetPhrase: public PhraseImplTemplate<WORD>
+{
+public:
+  typedef PhraseImplTemplate<WORD> Parent;
+  const PhraseTable &pt;
+  mutable void **ffData;
+  SCORE *scoreProperties;
+
+  TargetPhrase(MemPool &pool, const PhraseTable &pt, const System &system, size_t size)
+    : PhraseImplTemplate<WORD>(pool, size)
+    , pt(pt)
+    , scoreProperties(NULL)
+    , m_alignTerm(&AlignmentInfoCollection::Instance().GetEmptyAlignmentInfo()) {
+    m_scores = new (pool.Allocate<Scores>()) Scores(system, pool,
+        system.featureFunctions.GetNumScores());
+  }
+
+  Scores &GetScores() {
+    return *m_scores;
+  }
+
+  const Scores &GetScores() const {
+    return *m_scores;
+  }
+
+  virtual SCORE GetScoreForPruning() const = 0;
+
+  SCORE *GetScoresProperty(int propertyInd) const {
+    return scoreProperties ? scoreProperties + propertyInd : NULL;
+  }
+
+  const AlignmentInfo &GetAlignTerm() const {
+    return *m_alignTerm;
+  }
+
+  void SetAlignTerm(const AlignmentInfo &alignInfo) {
+    m_alignTerm = &alignInfo;
+  }
+
+  // ALNREP = alignment representation,
+  // see AlignmentInfo constructors for supported representations
+  template<typename ALNREP>
+  void
+  SetAlignTerm(const ALNREP &coll) {
+    m_alignTerm = AlignmentInfoCollection::Instance().Add(coll);
+  }
+
+  virtual void SetAlignmentInfo(const std::string &alignString) {
+    AlignmentInfo::CollType alignTerm;
+
+    std::vector<std::string> toks = Tokenize(alignString);
+    for (size_t i = 0; i < toks.size(); ++i) {
+      std::vector<size_t> alignPair = Tokenize<size_t>(toks[i], "-");
+      UTIL_THROW_IF2(alignPair.size() != 2, "Wrong alignment format");
+
+      size_t sourcePos = alignPair[0];
+      size_t targetPos = alignPair[1];
+
+      alignTerm.insert(std::pair<size_t,size_t>(sourcePos, targetPos));
+    }
+
+    SetAlignTerm(alignTerm);
+    //    cerr << "TargetPhrase::SetAlignmentInfo(const StringPiece &alignString) this:|" << *this << "|\n";
+
+    //cerr << "alignTerm=" << alignTerm.size() << endl;
+    //cerr << "alignNonTerm=" << alignNonTerm.size() << endl;
+
+  }
+
+  void OutputToStream(const System &system, const Phrase<WORD> &inputPhrase, std::ostream &out) const {
+    // get placeholders
+    FactorType placeholderFactor = system.options.input.placeholder_factor;
+    std::map<size_t, const Factor*> placeholders;
+    if (placeholderFactor != NOT_FOUND) {
+      // creates map of target position -> factor for placeholders
+      placeholders = GetPlaceholders(system, inputPhrase);
+    }
+
+    size_t size = PhraseImplTemplate<WORD>::GetSize();
+    for (size_t i = 0; i < size; ++i) {
+      // output placeholder, if any
+      std::map<size_t, const Factor*>::const_iterator iter = placeholders.find(i);
+      if (iter == placeholders.end()) {
+        const WORD &word = (*this)[i];
+        word.OutputToStream(system, out);
+      } else {
+        const Factor *factor = iter->second;
+        out << *factor;
+      }
+
+      out << " ";
+    }
+  }
+
+  std::map<size_t, const Factor*> GetPlaceholders(const System &system, const Phrase<WORD> &inputPhrase) const {
+    FactorType placeholderFactor = system.options.input.placeholder_factor;
+    std::map<size_t, const Factor*> ret;
+    //std::cerr << "inputPhrase=" << inputPhrase.Debug(system) << std::endl;
+
+    for (size_t sourcePos = 0; sourcePos < inputPhrase.GetSize(); ++sourcePos) {
+      const Factor *factor = inputPhrase[sourcePos][placeholderFactor];
+      if (factor) {
+        //std::cerr << "factor=" << *factor << std::endl;
+        //std::cerr << "tp=" << Debug(system) << std::endl;
+        std::set<size_t> targetPos = GetAlignTerm().GetAlignmentsForSource(sourcePos);
+        UTIL_THROW_IF2(targetPos.size() != 1,
+                       "Placeholder should be aligned to 1, and only 1, word:" << targetPos.size() << "!=1");
+        ret[*targetPos.begin()] = factor;
+      }
+    }
+
+    return ret;
+  }
+
+  virtual std::string Debug(const System &system) const {
+    std::stringstream out;
+    out << Phrase<WORD>::Debug(system);
+    out << " pt=" << pt.GetName() << " ";
+    out << " SCORES:" << GetScores().Debug(system);
+    out << " ALIGN-T:";
+    out << GetAlignTerm().Debug(system);
+
+    return out.str();
+  }
+
+protected:
+  Scores *m_scores;
+  const AlignmentInfo *m_alignTerm;
+};
+
+///////////////////////////////////////////////////////////////////////
+template<typename TP>
+struct CompareScoreForPruning {
+  bool operator()(const TP *a, const TP *b) const {
+    return a->GetScoreForPruning() > b->GetScoreForPruning();
+  }
+
+  bool operator()(const TP &a, const TP &b) const {
+    return a.GetScoreForPruning() > b.GetScoreForPruning();
+  }
+};
+
+} /* namespace Moses2a */
+

mosesdecoder/moses2/TrellisPaths.cpp

@@ -0,0 +1,14 @@
+/*
+ * TrellisPaths.cpp
+ *
+ *  Created on: 16 Mar 2016
+ *      Author: hieu
+ */
+#include "TrellisPaths.h"
+#include "legacy/Util2.h"
+
+namespace Moses2
+{
+
+
+} /* namespace Moses2 */

mosesdecoder/moses2/TypeDef.cpp

@@ -0,0 +1,11 @@
+#include "TypeDef.h"
+#include "util/exception.hh"
+#include <typeinfo>
+
+
+namespace Moses2
+{
+
+
+
+}

mosesdecoder/moses2/Word.h

@@ -0,0 +1,63 @@
+/*
+ * Word.h
+ *
+ *  Created on: 23 Oct 2015
+ *      Author: hieu
+ */
+
+#pragma once
+
+#include <iostream>
+#include "TypeDef.h"
+#include "legacy/Factor.h"
+#include "legacy/FactorCollection.h"
+
+namespace Moses2
+{
+
+class Word
+{
+public:
+  explicit Word();
+  Word(const Word &copy);
+
+  virtual ~Word();
+
+  void CreateFromString(FactorCollection &vocab, const System &system,
+                        const std::string &str);
+
+  virtual size_t hash() const;
+  virtual size_t hash(const std::vector<FactorType> &factors) const;
+
+  int Compare(const Word &compare) const;
+
+  virtual bool operator==(const Word &compare) const {
+    int cmp = Compare(compare);
+    return cmp == 0;
+  }
+
+  virtual bool operator!=(const Word &compare) const {
+    return !((*this) == compare);
+  }
+
+  virtual bool operator<(const Word &compare) const;
+
+  const Factor* operator[](size_t ind) const {
+    return m_factors[ind];
+  }
+
+  const Factor*& operator[](size_t ind) {
+    return m_factors[ind];
+  }
+
+  virtual void OutputToStream(const System &system, std::ostream &out) const;
+  virtual std::string Debug(const System &system) const;
+
+  std::string GetString(const FactorList &factorTypes) const;
+protected:
+  const Factor *m_factors[MAX_NUM_FACTORS];
+
+};
+
+}
+

mosesdecoder/phrase-extract/extract-ghkm/AlignmentGraph.cpp

@@ -0,0 +1,418 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#include "AlignmentGraph.h"
+
+#include <algorithm>
+#include <cassert>
+#include <cstdlib>
+#include <memory>
+#include <stack>
+
+#include "SyntaxTree.h"
+
+#include "ComposedRule.h"
+#include "Node.h"
+#include "Options.h"
+#include "Subgraph.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+AlignmentGraph::AlignmentGraph(const SyntaxTree *t,
+                               const std::vector<std::string> &s,
+                               const Alignment &a)
+{
+  // Copy the parse tree nodes and add them to m_targetNodes.
+  m_root = CopyParseTree(t);
+
+  // Create a node for each source word.
+  m_sourceNodes.reserve(s.size());
+  for (std::vector<std::string>::const_iterator p(s.begin());
+       p != s.end(); ++p) {
+    m_sourceNodes.push_back(new Node(*p, SOURCE));
+  }
+
+  // Connect source nodes to parse tree leaves according to the given word
+  // alignment.
+  std::vector<Node *> targetTreeLeaves;
+  GetTargetTreeLeaves(m_root, targetTreeLeaves);
+  for (Alignment::const_iterator p(a.begin()); p != a.end(); ++p) {
+    Node *src = m_sourceNodes[p->first];
+    Node *tgt = targetTreeLeaves[p->second];
+    src->AddParent(tgt);
+    tgt->AddChild(src);
+  }
+
+  // Attach unaligned source words (if any).
+  AttachUnalignedSourceWords();
+
+  // Populate node spans.
+  std::vector<Node *>::const_iterator p(m_sourceNodes.begin());
+  for (int i = 0; p != m_sourceNodes.end(); ++p, ++i) {
+    (*p)->PropagateIndex(i);
+  }
+
+  // Calculate complement spans.
+  CalcComplementSpans(m_root);
+}
+
+AlignmentGraph::~AlignmentGraph()
+{
+  for (std::vector<Node *>::iterator p(m_sourceNodes.begin());
+       p != m_sourceNodes.end(); ++p) {
+    delete *p;
+  }
+  for (std::vector<Node *>::iterator p(m_targetNodes.begin());
+       p != m_targetNodes.end(); ++p) {
+    delete *p;
+  }
+}
+
+Subgraph AlignmentGraph::ComputeMinimalFrontierGraphFragment(
+  Node *root,
+  const std::set<Node *> &frontierSet)
+{
+  std::stack<Node *> expandableNodes;
+  std::set<const Node *> expandedNodes;
+
+  if (root->IsSink()) {
+    expandedNodes.insert(root);
+  } else {
+    expandableNodes.push(root);
+  }
+
+  while (!expandableNodes.empty()) {
+    Node *n = expandableNodes.top();
+    expandableNodes.pop();
+
+    const std::vector<Node *> &children = n->GetChildren();
+
+    for (std::vector<Node *>::const_iterator p(children.begin());
+         p != children.end(); ++p) {
+      Node *child = *p;
+      if (child->IsSink()) {
+        expandedNodes.insert(child);
+        continue;
+      }
+      std::set<Node *>::const_iterator q = frontierSet.find(child);
+      if (q == frontierSet.end()) { //child is not from the frontier set
+        expandableNodes.push(child);
+      } else if (child->GetType() == TARGET) { // still need source word
+        expandableNodes.push(child);
+      } else {
+        expandedNodes.insert(child);
+      }
+    }
+  }
+
+  return Subgraph(root, expandedNodes);
+}
+
+void AlignmentGraph::ExtractMinimalRules(const Options &options)
+{
+  // Determine which nodes are frontier nodes.
+  std::set<Node *> frontierSet;
+  ComputeFrontierSet(m_root, options, frontierSet);
+
+  // Form the minimal frontier graph fragment rooted at each frontier node.
+  std::vector<Subgraph> fragments;
+  fragments.reserve(frontierSet.size());
+  for (std::set<Node *>::iterator p(frontierSet.begin());
+       p != frontierSet.end(); ++p) {
+    Node *root = *p;
+    Subgraph fragment = ComputeMinimalFrontierGraphFragment(root, frontierSet);
+    assert(!fragment.IsTrivial());
+    // Can it form an SCFG rule?
+    // FIXME Does this exclude non-lexical unary rules?
+    if (root->GetType() == TREE && !root->GetSpan().empty()) {
+      root->AddRule(new Subgraph(fragment));
+    }
+  }
+}
+
+void AlignmentGraph::ExtractComposedRules(const Options &options)
+{
+  ExtractComposedRules(m_root, options);
+}
+
+void AlignmentGraph::ExtractComposedRules(Node *node, const Options &options)
+{
+  // Extract composed rules for all children first.
+  const std::vector<Node *> &children = node->GetChildren();
+  for (std::vector<Node *>::const_iterator p(children.begin());
+       p != children.end(); ++p) {
+    ExtractComposedRules(*p, options);
+  }
+
+  // If there is no minimal rule for this node then there are no composed
+  // rules.
+  const std::vector<const Subgraph*> &rules = node->GetRules();
+  assert(rules.size() <= 1);
+  if (rules.empty()) {
+    return;
+  }
+
+  // Construct an initial composition candidate from the minimal rule.
+  ComposedRule cr(*(rules[0]));
+  if (!cr.GetOpenAttachmentPoint()) {
+    // No composition possible.
+    return;
+  }
+
+  std::queue<ComposedRule> queue;
+  queue.push(cr);
+  while (!queue.empty()) {
+    ComposedRule cr = queue.front();
+    queue.pop();
+    const Node *attachmentPoint = cr.GetOpenAttachmentPoint();
+    assert(attachmentPoint);
+    assert(attachmentPoint != node);
+    // Create all possible rules by composing this node's minimal rule with the
+    // existing rules (both minimal and composed) rooted at the first open
+    // attachment point.
+    const std::vector<const Subgraph*> &rules = attachmentPoint->GetRules();
+    for (std::vector<const Subgraph*>::const_iterator p = rules.begin();
+         p != rules.end(); ++p) {
+      assert((*p)->GetRoot()->GetType() == TREE);
+      ComposedRule *cr2 = cr.AttemptComposition(**p, options);
+      if (cr2) {
+        node->AddRule(new Subgraph(cr2->CreateSubgraph()));
+        if (cr2->GetOpenAttachmentPoint()) {
+          queue.push(*cr2);
+        }
+        delete cr2;
+      }
+    }
+    // Done with this attachment point.  Advance to the next, if any.
+    cr.CloseAttachmentPoint();
+    if (cr.GetOpenAttachmentPoint()) {
+      queue.push(cr);
+    }
+  }
+}
+
+Node *AlignmentGraph::CopyParseTree(const SyntaxTree *root)
+{
+  NodeType nodeType = (root->IsLeaf()) ? TARGET : TREE;
+
+  std::auto_ptr<Node> n(new Node(root->value().label, nodeType));
+
+  if (nodeType == TREE) {
+    float score = 0.0f;
+    SyntaxNode::AttributeMap::const_iterator p =
+      root->value().attributes.find("pcfg");
+    if (p != root->value().attributes.end()) {
+      score = std::atof(p->second.c_str());
+    }
+    n->SetPcfgScore(score);
+  }
+
+  const std::vector<SyntaxTree *> &children = root->children();
+  std::vector<Node *> childNodes;
+  childNodes.reserve(children.size());
+  for (std::vector<SyntaxTree *>::const_iterator p(children.begin());
+       p != children.end(); ++p) {
+    Node *child = CopyParseTree(*p);
+    child->AddParent(n.get());
+    childNodes.push_back(child);
+  }
+  n->SetChildren(childNodes);
+
+  Node *p = n.release();
+  m_targetNodes.push_back(p);
+  return p;
+}
+
+// Recursively constructs the set of frontier nodes for the tree (or subtree)
+// rooted at the given node.
+void AlignmentGraph::ComputeFrontierSet(Node *root,
+                                        const Options &options,
+                                        std::set<Node *> &frontierSet) const
+{
+  // Non-tree nodes and unaligned target subtrees are not frontier nodes (and
+  // nor are their descendants).  See the comment for the function
+  // AlignmentGraph::IsFrontierNode().
+  if (root->GetType() != TREE || root->GetSpan().empty()) {
+    return;
+  }
+
+  if (IsFrontierNode(*root, options)) {
+    frontierSet.insert(root);
+  }
+
+  // Recursively check descendants.
+  const std::vector<Node *> &children = root->GetChildren();
+  for (std::vector<Node *>::const_iterator p(children.begin());
+       p != children.end(); ++p) {
+    ComputeFrontierSet(*p, options, frontierSet);
+  }
+}
+
+// Determines whether the given node is a frontier node or not. The definition
+// of a frontier node differs from Galley et al's (2004) in the following ways:
+//
+// 1. A node with an empty span is not a frontier node (this is to exclude
+//    unaligned target subtrees).
+// 2. Target word nodes are not frontier nodes.
+// 3. Source word nodes are not frontier nodes.
+// 4. Unless the --AllowUnary option is used, a node is not a frontier node if
+//    it has the same span as its parent.
+bool AlignmentGraph::IsFrontierNode(const Node &n, const Options &options) const
+{
+  // Don't include word nodes or unaligned target subtrees.
+  if (n.GetType() != TREE || n.GetSpan().empty()) {
+    return false;
+  }
+  // This is the original GHKM definition of a frontier node.
+  if (SpansIntersect(n.GetComplementSpan(), Closure(n.GetSpan()))) {
+    return false;
+  }
+  // Unless unary rules are explicitly allowed, we use Chung et al's (2011)
+  // modified defintion of a frontier node to eliminate the production of
+  // non-lexical unary rules.
+  assert(n.GetParents().size() <= 1);
+  if (!options.allowUnary &&
+      !n.GetParents().empty() &&
+      n.GetParents()[0]->GetSpan() == n.GetSpan()) {
+    return false;
+  }
+  return true;
+}
+
+void AlignmentGraph::CalcComplementSpans(Node *root)
+{
+  Span compSpan;
+  std::set<Node *> siblings;
+
+  const std::vector<Node *> &parents = root->GetParents();
+  for (std::vector<Node *>::const_iterator p(parents.begin());
+       p != parents.end(); ++p) {
+    const Span &parentCompSpan = (*p)->GetComplementSpan();
+    compSpan.insert(parentCompSpan.begin(), parentCompSpan.end());
+    const std::vector<Node *> &c = (*p)->GetChildren();
+    siblings.insert(c.begin(), c.end());
+  }
+
+  for (std::set<Node *>::iterator p(siblings.begin());
+       p != siblings.end(); ++p) {
+    if (*p == root) {
+      continue;
+    }
+    const Span &siblingSpan = (*p)->GetSpan();
+    compSpan.insert(siblingSpan.begin(), siblingSpan.end());
+  }
+
+  root->SetComplementSpan(compSpan);
+
+  const std::vector<Node *> &children = root->GetChildren();
+  for (std::vector<Node *>::const_iterator p(children.begin());
+       p != children.end(); ++p) {
+    CalcComplementSpans(*p);
+  }
+}
+
+void AlignmentGraph::GetTargetTreeLeaves(Node *root,
+    std::vector<Node *> &leaves)
+{
+  if (root->IsSink()) {
+    leaves.push_back(root);
+  } else {
+    const std::vector<Node *> &children = root->GetChildren();
+    for (std::vector<Node *>::const_iterator p(children.begin());
+         p != children.end(); ++p) {
+      GetTargetTreeLeaves(*p, leaves);
+    }
+  }
+}
+
+void AlignmentGraph::AttachUnalignedSourceWords()
+{
+  // Find the unaligned source words (if any).
+  std::set<int> unaligned;
+  for (size_t i = 0; i < m_sourceNodes.size(); ++i) {
+    const Node &sourceNode = (*m_sourceNodes[i]);
+    if (sourceNode.GetParents().empty()) {
+      unaligned.insert(i);
+    }
+  }
+
+  // Determine the attachment point for each one and attach it.
+  for (std::set<int>::iterator p = unaligned.begin();
+       p != unaligned.end(); ++p) {
+    int index = *p;
+    Node *attachmentPoint = DetermineAttachmentPoint(index);
+    Node *sourceNode = m_sourceNodes[index];
+    attachmentPoint->AddChild(sourceNode);
+    sourceNode->AddParent(attachmentPoint);
+  }
+}
+
+Node *AlignmentGraph::DetermineAttachmentPoint(int index)
+{
+  // Find the nearest aligned neighbour to the left, if any.
+  int i = index;
+  while (--i >= 0) {
+    if (!m_sourceNodes[i]->GetParents().empty()) {
+      break;
+    }
+  }
+  // No aligned neighbours to the left, so attach to the root.
+  if (i == -1) {
+    return m_root;
+  }
+  // Find the nearest aligned neighbour to the right, if any.
+  size_t j = index;
+  while (++j < m_sourceNodes.size()) {
+    if (!m_sourceNodes[j]->GetParents().empty()) {
+      break;
+    }
+  }
+  // No aligned neighbours to the right, so attach to the root.
+  if (j == m_sourceNodes.size()) {
+    return m_root;
+  }
+  // Construct the set of target nodes that are aligned to the left and right
+  // neighbours.
+  const std::vector<Node *> &leftParents = m_sourceNodes[i]->GetParents();
+  assert(!leftParents.empty());
+  const std::vector<Node *> &rightParents = m_sourceNodes[j]->GetParents();
+  assert(!rightParents.empty());
+  std::set<Node *> targetSet;
+  targetSet.insert(leftParents.begin(), leftParents.end());
+  targetSet.insert(rightParents.begin(), rightParents.end());
+  // The attachment point is the lowest common ancestor of the target word
+  // nodes, unless the LCA is itself a target word, in which case the LCA
+  // is the parent.  This is to avoid including introducing new word alignments.
+  // It assumes that the parse tree uses preterminals for parts of speech.
+  Node *lca = Node::LowestCommonAncestor(targetSet.begin(), targetSet.end());
+  if (lca->GetType() == TARGET) {
+    assert(lca->GetParents().size() == 1);
+    return lca->GetParents()[0];
+  }
+  return lca;
+}
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining

mosesdecoder/phrase-extract/extract-ghkm/AlignmentGraph.h

@@ -0,0 +1,87 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#pragma once
+#ifndef EXTRACT_GHKM_ALIGNMENT_GRAPH_H_
+#define EXTRACT_GHKM_ALIGNMENT_GRAPH_H_
+
+#include <set>
+#include <string>
+#include <vector>
+
+#include "SyntaxTree.h"
+
+#include "Alignment.h"
+#include "Options.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+class Node;
+class Subgraph;
+
+class AlignmentGraph
+{
+public:
+  AlignmentGraph(const SyntaxTree *,
+                 const std::vector<std::string> &,
+                 const Alignment &);
+
+  ~AlignmentGraph();
+
+  Node *GetRoot() {
+    return m_root;
+  }
+  const std::vector<Node *> &GetTargetNodes() {
+    return m_targetNodes;
+  }
+
+  void ExtractMinimalRules(const Options &);
+  void ExtractComposedRules(const Options &);
+
+private:
+  // Disallow copying
+  AlignmentGraph(const AlignmentGraph &);
+  AlignmentGraph &operator=(const AlignmentGraph &);
+
+  Node *CopyParseTree(const SyntaxTree *);
+  void ComputeFrontierSet(Node *, const Options &, std::set<Node *> &) const;
+  bool IsFrontierNode(const Node &, const Options &) const;
+  void CalcComplementSpans(Node *);
+  void GetTargetTreeLeaves(Node *, std::vector<Node *> &);
+  void AttachUnalignedSourceWords();
+  Node *DetermineAttachmentPoint(int);
+  Subgraph ComputeMinimalFrontierGraphFragment(Node *,
+      const std::set<Node *> &);
+  void ExtractComposedRules(Node *, const Options &);
+
+  Node *m_root;
+  std::vector<Node *> m_sourceNodes;
+  std::vector<Node *> m_targetNodes;
+};
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining
+
+#endif

mosesdecoder/phrase-extract/extract-ghkm/ComposedRule.cpp

@@ -0,0 +1,134 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#include "ComposedRule.h"
+
+#include <set>
+#include <vector>
+#include <queue>
+
+#include "Node.h"
+#include "Options.h"
+#include "Subgraph.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+ComposedRule::ComposedRule(const Subgraph &baseRule)
+  : m_baseRule(baseRule)
+  , m_depth(baseRule.GetDepth())
+  , m_size(baseRule.GetSize())
+  , m_nodeCount(baseRule.GetNodeCount())
+{
+  const std::set<const Node *> &leaves = baseRule.GetLeaves();
+  for (std::set<const Node *>::const_iterator p = leaves.begin();
+       p != leaves.end(); ++p) {
+    if ((*p)->GetType() == TREE) {
+      m_openAttachmentPoints.push(*p);
+    }
+  }
+}
+
+ComposedRule::ComposedRule(const ComposedRule &other, const Subgraph &rule,
+                           int depth)
+  : m_baseRule(other.m_baseRule)
+  , m_attachedRules(other.m_attachedRules)
+  , m_openAttachmentPoints(other.m_openAttachmentPoints)
+  , m_depth(depth)
+  , m_size(other.m_size+rule.GetSize())
+  , m_nodeCount(other.m_nodeCount+rule.GetNodeCount()-1)
+{
+  m_attachedRules.push_back(&rule);
+  m_openAttachmentPoints.pop();
+}
+
+const Node *ComposedRule::GetOpenAttachmentPoint()
+{
+  return m_openAttachmentPoints.empty() ? 0 : m_openAttachmentPoints.front();
+}
+
+void ComposedRule::CloseAttachmentPoint()
+{
+  assert(!m_openAttachmentPoints.empty());
+  m_attachedRules.push_back(0);
+  m_openAttachmentPoints.pop();
+}
+
+ComposedRule *ComposedRule::AttemptComposition(const Subgraph &rule,
+    const Options &options) const
+{
+  // The smallest possible rule fragment should be rooted at a tree node.
+  // Note that this differs from the original GHKM definition.
+  assert(rule.GetRoot()->GetType() == TREE);
+
+  // Check the node count of the proposed rule.
+  if (m_nodeCount+rule.GetNodeCount()-1 > options.maxNodes) {
+    return 0;
+  }
+
+  // Check the size of the proposed rule.
+  if (m_size+rule.GetSize() > options.maxRuleSize) {
+    return 0;
+  }
+
+  // Determine the depth of the proposed rule and test whether it exceeds the
+  // limit.
+  int attachmentPointDepth = 0;
+  const Node *n = rule.GetRoot();
+  while (n != m_baseRule.GetRoot()) {
+    assert(n->GetParents().size() == 1);
+    n = n->GetParents()[0];
+    ++attachmentPointDepth;
+  }
+  int newDepth = std::max(m_depth, attachmentPointDepth+rule.GetDepth());
+  if (newDepth > options.maxRuleDepth) {
+    return 0;
+  }
+
+  return new ComposedRule(*this, rule, newDepth);
+}
+
+Subgraph ComposedRule::CreateSubgraph()
+{
+  std::set<const Node *> leaves;
+  const std::set<const Node *> &baseLeaves = m_baseRule.GetLeaves();
+  size_t i = 0;
+  for (std::set<const Node *>::const_iterator p = baseLeaves.begin();
+       p != baseLeaves.end(); ++p) {
+    const Node *baseLeaf = *p;
+    if (baseLeaf->GetType() == TREE && i < m_attachedRules.size()) {
+      const Subgraph *attachedRule = m_attachedRules[i++];
+      if (attachedRule) {
+        leaves.insert(attachedRule->GetLeaves().begin(),
+                      attachedRule->GetLeaves().end());
+        continue;
+      }
+    }
+    leaves.insert(baseLeaf);
+  }
+  return Subgraph(m_baseRule.GetRoot(), leaves);
+}
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining

mosesdecoder/phrase-extract/extract-ghkm/ComposedRule.h

@@ -0,0 +1,75 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#pragma once
+#ifndef EXTRACT_GHKM_COMPOSED_RULE_H_
+#define EXTRACT_GHKM_COMPOSED_RULE_H_
+
+#include <vector>
+#include <queue>
+
+#include "Subgraph.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+class Node;
+struct Options;
+
+class ComposedRule
+{
+public:
+  // Form a 'trivial' ComposedRule from a single existing rule.
+  ComposedRule(const Subgraph &baseRule);
+
+  // Returns the first open attachment point if any exist or 0 otherwise.
+  const Node *GetOpenAttachmentPoint();
+
+  // Close the first open attachment point without attaching a rule.
+  void CloseAttachmentPoint();
+
+  // Attempts to produce a new composed rule by attaching a given rule at the
+  // first open attachment point.  This will fail if the proposed rule violates
+  // the constraints set in the Options object, in which case the function
+  // returns 0.
+  ComposedRule *AttemptComposition(const Subgraph &, const Options &) const;
+
+  // Constructs a Subgraph object corresponding to the composed rule.
+  Subgraph CreateSubgraph();
+
+private:
+  ComposedRule(const ComposedRule &, const Subgraph &, int);
+
+  const Subgraph &m_baseRule;
+  std::vector<const Subgraph *> m_attachedRules;
+  std::queue<const Node *> m_openAttachmentPoints;
+  int m_depth;
+  int m_size;
+  int m_nodeCount;
+};
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining
+
+#endif

mosesdecoder/phrase-extract/extract-ghkm/ExtractGHKM.cpp

@@ -0,0 +1,902 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#include "ExtractGHKM.h"
+
+#include <cassert>
+#include <cstdlib>
+#include <fstream>
+#include <iostream>
+#include <iterator>
+#include <string>
+#include <sstream>
+#include <vector>
+
+#include <boost/program_options.hpp>
+
+#include "syntax-common/exception.h"
+#include "syntax-common/xml_tree_parser.h"
+
+#include "InputFileStream.h"
+#include "OutputFileStream.h"
+#include "SyntaxNode.h"
+#include "SyntaxNodeCollection.h"
+#include "SyntaxTree.h"
+#include "tables-core.h"
+#include "XmlException.h"
+#include "XmlTree.h"
+
+#include "Alignment.h"
+#include "AlignmentGraph.h"
+#include "Node.h"
+#include "Options.h"
+#include "PhraseOrientation.h"
+#include "ScfgRule.h"
+#include "ScfgRuleWriter.h"
+#include "Span.h"
+#include "StsgRule.h"
+#include "StsgRuleWriter.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+int ExtractGHKM::Main(int argc, char *argv[])
+{
+  using Moses::InputFileStream;
+  using Moses::OutputFileStream;
+
+  // Process command-line options.
+  Options options;
+  ProcessOptions(argc, argv, options);
+
+  // Open input files.
+  //
+  // The GHKM algorithm is neutral about whether the model is string-to-tree or
+  // tree-to-string.  This implementation assumes the model to be
+  // string-to-tree, but if the -t2s option is given then the source and target
+  // input files are switched prior to extraction and then the source and
+  // target of the extracted rules are switched on output.
+  std::string effectiveTargetFile = options.t2s ? options.sourceFile
+                                    : options.targetFile;
+  std::string effectiveSourceFile = options.t2s ? options.targetFile
+                                    : options.sourceFile;
+  InputFileStream targetStream(effectiveTargetFile);
+  InputFileStream sourceStream(effectiveSourceFile);
+  InputFileStream alignmentStream(options.alignmentFile);
+
+  // Open output files.
+  OutputFileStream fwdExtractStream;
+  OutputFileStream invExtractStream;
+  OutputFileStream glueGrammarStream;
+  OutputFileStream targetUnknownWordStream;
+  OutputFileStream sourceUnknownWordStream;
+  OutputFileStream sourceLabelSetStream;
+  OutputFileStream unknownWordSoftMatchesStream;
+
+  std::string fwdFileName = options.extractFile;
+  std::string invFileName = options.extractFile + std::string(".inv");
+  if (options.gzOutput) {
+    fwdFileName += ".gz";
+    invFileName += ".gz";
+  }
+  OpenOutputFileOrDie(fwdFileName, fwdExtractStream);
+  OpenOutputFileOrDie(invFileName, invExtractStream);
+
+  if (!options.glueGrammarFile.empty()) {
+    OpenOutputFileOrDie(options.glueGrammarFile, glueGrammarStream);
+  }
+  if (!options.targetUnknownWordFile.empty()) {
+    OpenOutputFileOrDie(options.targetUnknownWordFile, targetUnknownWordStream);
+  }
+  if (!options.sourceUnknownWordFile.empty()) {
+    OpenOutputFileOrDie(options.sourceUnknownWordFile, sourceUnknownWordStream);
+  }
+  if (!options.sourceLabelSetFile.empty()) {
+    if (!options.sourceLabels) {
+      Error("SourceLabels should be active if SourceLabelSet is supposed to be written to a file");
+    }
+    OpenOutputFileOrDie(options.sourceLabelSetFile, sourceLabelSetStream); // note that this is not a global source label set if extraction is parallelized
+  }
+  if (!options.unknownWordSoftMatchesFile.empty()) {
+    OpenOutputFileOrDie(options.unknownWordSoftMatchesFile, unknownWordSoftMatchesStream);
+  }
+
+  // Word count statistics for producing unknown word labels.
+  std::map<std::string, int> targetWordCount;
+  std::map<std::string, std::string> targetWordLabel;
+
+  // Word count statistics for producing unknown word labels: source side.
+  std::map<std::string, int> sourceWordCount;
+  std::map<std::string, std::string> sourceWordLabel;
+
+  std::string targetLine;
+  std::string sourceLine;
+  std::string alignmentLine;
+  Alignment alignment;
+  XmlTreeParser targetXmlTreeParser;
+  XmlTreeParser sourceXmlTreeParser;
+  ScfgRuleWriter scfgWriter(fwdExtractStream, invExtractStream, options);
+  StsgRuleWriter stsgWriter(fwdExtractStream, invExtractStream, options);
+  size_t lineNum = options.sentenceOffset;
+  while (true) {
+    std::getline(targetStream, targetLine);
+    std::getline(sourceStream, sourceLine);
+    std::getline(alignmentStream, alignmentLine);
+
+    if (targetStream.eof() && sourceStream.eof() && alignmentStream.eof()) {
+      break;
+    }
+
+    if (targetStream.eof() || sourceStream.eof() || alignmentStream.eof()) {
+      Error("Files must contain same number of lines");
+    }
+
+    ++lineNum;
+
+    // Parse target tree.
+    if (targetLine.size() == 0) {
+      std::cerr << "skipping line " << lineNum << " with empty target tree\n";
+      continue;
+    }
+    std::auto_ptr<SyntaxTree> targetParseTree;
+    try {
+      targetParseTree = targetXmlTreeParser.Parse(targetLine);
+      assert(targetParseTree.get());
+    } catch (const Exception &e) {
+      std::ostringstream oss;
+      oss << "Failed to parse target XML tree at line " << lineNum;
+      if (!e.msg().empty()) {
+        oss << ": " << e.msg();
+      }
+      Error(oss.str());
+    }
+
+    // Read source tokens (and parse tree if using source labels).
+    std::vector<std::string> sourceTokens;
+    std::auto_ptr<SyntaxTree> sourceParseTree;
+    if (!options.sourceLabels) {
+      sourceTokens = ReadTokens(sourceLine);
+    } else {
+      try {
+        sourceParseTree = sourceXmlTreeParser.Parse(sourceLine);
+        assert(sourceParseTree.get());
+      } catch (const Exception &e) {
+        std::ostringstream oss;
+        oss << "Failed to parse source XML tree at line " << lineNum;
+        if (!e.msg().empty()) {
+          oss << ": " << e.msg();
+        }
+        Error(oss.str());
+      }
+      sourceTokens = sourceXmlTreeParser.words();
+    }
+
+    // Read word alignments.
+    try {
+      ReadAlignment(alignmentLine, alignment);
+    } catch (const Exception &e) {
+      std::ostringstream oss;
+      oss << "Failed to read alignment at line " << lineNum << ": ";
+      oss << e.msg();
+      Error(oss.str());
+    }
+    if (alignment.size() == 0) {
+      std::cerr << "skipping line " << lineNum << " without alignment points\n";
+      continue;
+    }
+    if (options.t2s) {
+      FlipAlignment(alignment);
+    }
+
+    // Record word counts.
+    if (!options.targetUnknownWordFile.empty()) {
+      CollectWordLabelCounts(*targetParseTree, options, targetWordCount,
+                             targetWordLabel);
+    }
+
+    // Record word counts: source side.
+    if (options.sourceLabels && !options.sourceUnknownWordFile.empty()) {
+      CollectWordLabelCounts(*sourceParseTree, options, sourceWordCount,
+                             sourceWordLabel);
+    }
+
+    // Form an alignment graph from the target tree, source words, and
+    // alignment.
+    AlignmentGraph graph(targetParseTree.get(), sourceTokens, alignment);
+
+    // Extract minimal rules, adding each rule to its root node's rule set.
+    graph.ExtractMinimalRules(options);
+
+    // Extract composed rules.
+    if (!options.minimal) {
+      graph.ExtractComposedRules(options);
+    }
+
+    // Initialize phrase orientation scoring object
+    PhraseOrientation phraseOrientation(sourceTokens.size(),
+                                        targetXmlTreeParser.words().size(), alignment);
+
+    // Write the rules, subject to scope pruning.
+    const std::vector<Node *> &targetNodes = graph.GetTargetNodes();
+    for (std::vector<Node *>::const_iterator p = targetNodes.begin();
+         p != targetNodes.end(); ++p) {
+
+      const std::vector<const Subgraph *> &rules = (*p)->GetRules();
+
+      PhraseOrientation::REO_CLASS l2rOrientation=PhraseOrientation::REO_CLASS_UNKNOWN, r2lOrientation=PhraseOrientation::REO_CLASS_UNKNOWN;
+      if (options.phraseOrientation && !rules.empty()) {
+        int sourceSpanBegin = *((*p)->GetSpan().begin());
+        int sourceSpanEnd   = *((*p)->GetSpan().rbegin());
+        l2rOrientation = phraseOrientation.GetOrientationInfo(sourceSpanBegin,sourceSpanEnd,PhraseOrientation::REO_DIR_L2R);
+        r2lOrientation = phraseOrientation.GetOrientationInfo(sourceSpanBegin,sourceSpanEnd,PhraseOrientation::REO_DIR_R2L);
+        // std::cerr << "span " << sourceSpanBegin << " " << sourceSpanEnd << std::endl;
+        // std::cerr << "phraseOrientation " << phraseOrientation.GetOrientationInfo(sourceSpanBegin,sourceSpanEnd) << std::endl;
+      }
+
+      for (std::vector<const Subgraph *>::const_iterator q = rules.begin();
+           q != rules.end(); ++q) {
+        // STSG output.
+        if (options.stsg) {
+          StsgRule rule(**q);
+          if (rule.Scope() <= options.maxScope) {
+            stsgWriter.Write(rule);
+          }
+          continue;
+        }
+        // SCFG output.
+        ScfgRule *r = 0;
+        if (options.sourceLabels) {
+          r = new ScfgRule(**q, &sourceXmlTreeParser.node_collection());
+        } else {
+          r = new ScfgRule(**q);
+        }
+        // TODO Can scope pruning be done earlier?
+        if (r->Scope() <= options.maxScope) {
+          scfgWriter.Write(*r,lineNum,false);
+          if (options.treeFragments) {
+            fwdExtractStream << " {{Tree ";
+            (*q)->PrintTree(fwdExtractStream);
+            fwdExtractStream << "}}";
+          }
+          if (options.partsOfSpeech) {
+            fwdExtractStream << " {{POS";
+            (*q)->PrintPartsOfSpeech(fwdExtractStream);
+            fwdExtractStream << "}}";
+          }
+          if (options.phraseOrientation) {
+            fwdExtractStream << " {{Orientation ";
+            phraseOrientation.WriteOrientation(fwdExtractStream,l2rOrientation);
+            fwdExtractStream << " ";
+            phraseOrientation.WriteOrientation(fwdExtractStream,r2lOrientation);
+            fwdExtractStream << "}}";
+            phraseOrientation.IncrementPriorCount(PhraseOrientation::REO_DIR_L2R,l2rOrientation,1);
+            phraseOrientation.IncrementPriorCount(PhraseOrientation::REO_DIR_R2L,r2lOrientation,1);
+          }
+          fwdExtractStream << std::endl;
+          invExtractStream << std::endl;
+        }
+        delete r;
+      }
+    }
+  }
+
+  if (options.phraseOrientation) {
+    std::string phraseOrientationPriorsFileName = options.extractFile + std::string(".phraseOrientationPriors");
+    OutputFileStream phraseOrientationPriorsStream;
+    OpenOutputFileOrDie(phraseOrientationPriorsFileName, phraseOrientationPriorsStream);
+    PhraseOrientation::WritePriorCounts(phraseOrientationPriorsStream);
+  }
+
+  std::map<std::string,size_t> sourceLabels;
+  if (options.sourceLabels && !options.sourceLabelSetFile.empty()) {
+    std::set<std::string> extendedLabelSet = sourceXmlTreeParser.label_set();
+    extendedLabelSet.insert("XLHS"); // non-matching label (left-hand side)
+    extendedLabelSet.insert("XRHS"); // non-matching label (right-hand side)
+    extendedLabelSet.insert("TOPLABEL");  // as used in the glue grammar
+    extendedLabelSet.insert("SOMELABEL"); // as used in the glue grammar
+    size_t index = 0;
+    for (std::set<std::string>::const_iterator iter=extendedLabelSet.begin();
+         iter!=extendedLabelSet.end(); ++iter, ++index) {
+      sourceLabels.insert(std::pair<std::string,size_t>(*iter,index));
+    }
+    WriteSourceLabelSet(sourceLabels, sourceLabelSetStream);
+  }
+
+  std::set<std::string> strippedTargetLabelSet;
+  std::map<std::string, int> strippedTargetTopLabelSet;
+  if (options.stripBitParLabels &&
+      (!options.glueGrammarFile.empty() || !options.unknownWordSoftMatchesFile.empty())) {
+    StripBitParLabels(targetXmlTreeParser.label_set(),
+                      targetXmlTreeParser.top_label_set(),
+                      strippedTargetLabelSet, strippedTargetTopLabelSet);
+  }
+
+  if (!options.glueGrammarFile.empty()) {
+    if (options.stripBitParLabels) {
+      WriteGlueGrammar(strippedTargetLabelSet, strippedTargetTopLabelSet, sourceLabels, options, glueGrammarStream);
+    } else {
+      WriteGlueGrammar(targetXmlTreeParser.label_set(),
+                       targetXmlTreeParser.top_label_set(),
+                       sourceLabels, options, glueGrammarStream);
+    }
+  }
+
+  if (!options.targetUnknownWordFile.empty()) {
+    WriteUnknownWordLabel(targetWordCount, targetWordLabel, options, targetUnknownWordStream);
+  }
+
+  if (options.sourceLabels && !options.sourceUnknownWordFile.empty()) {
+    WriteUnknownWordLabel(sourceWordCount, sourceWordLabel, options, sourceUnknownWordStream, true);
+  }
+
+  if (!options.unknownWordSoftMatchesFile.empty()) {
+    if (options.stripBitParLabels) {
+      WriteUnknownWordSoftMatches(strippedTargetLabelSet, unknownWordSoftMatchesStream);
+    } else {
+      WriteUnknownWordSoftMatches(targetXmlTreeParser.label_set(),
+                                  unknownWordSoftMatchesStream);
+    }
+  }
+
+  return 0;
+}
+
+void ExtractGHKM::ProcessOptions(int argc, char *argv[],
+                                 Options &options) const
+{
+  namespace po = boost::program_options;
+  namespace cls = boost::program_options::command_line_style;
+
+  // Construct the 'top' of the usage message: the bit that comes before the
+  // options list.
+  std::ostringstream usageTop;
+  usageTop << "Usage: " << name()
+           << " [OPTION]... TARGET SOURCE ALIGNMENT EXTRACT\n\n"
+           << "SCFG rule extractor based on the GHKM algorithm described in\n"
+           << "Galley et al. (2004).\n\n"
+           << "Options";
+
+  // Construct the 'bottom' of the usage message.
+  std::ostringstream usageBottom;
+  usageBottom << "\nImplementation Notes:\n"
+              << "\nThe parse tree is assumed to contain part-of-speech preterminal nodes.\n"
+              << "\n"
+              << "For the composed rule constraints: rule depth is the "
+              "maximum distance from the\nrule's root node to a sink "
+              "node, not counting preterminal expansions or word\n"
+              "alignments.  Rule size is the measure defined in DeNeefe "
+              "et al (2007): the\nnumber of non-part-of-speech, non-leaf "
+              "constituent labels in the target tree.\nNode count is the "
+              "number of target tree nodes (excluding target words).\n"
+              << "\n"
+              << "Scope pruning (Hopkins and Langmead, 2010) is applied to both minimal and\ncomposed rules.\n"
+              << "\n"
+              << "Unaligned source words are attached to the tree using the "
+              "following heuristic:\nif there are aligned source words to "
+              "both the left and the right of an unaligned\nsource word "
+              "then it is attached to the lowest common ancestor of its "
+              "nearest\nsuch left and right neighbours.  Otherwise, it is "
+              "attached to the root of the\nparse tree.\n"
+              << "\n"
+              << "Unless the --AllowUnary option is given, unary rules containing no lexical\nsource items are eliminated using the method described in Chung et al. (2011).\nThe parsing algorithm used in Moses is unable to handle such rules.\n"
+              << "\n"
+              << "References:\n"
+              << "Galley, M., Hopkins, M., Knight, K., and Marcu, D. (2004)\n"
+              << "\"What's in a Translation Rule?\", In Proceedings of HLT/NAACL 2004.\n"
+              << "\n"
+              << "DeNeefe, S., Knight, K., Wang, W., and Marcu, D. (2007)\n"
+              << "\"What Can Syntax-Based MT Learn from Phrase-Based MT?\", In Proceedings of\nEMNLP-CoNLL 2007.\n"
+              << "\n"
+              << "Hopkins, M. and Langmead, G. (2010)\n"
+              << "\"SCFG Decoding Without Binarization\", In Proceedings of EMNLP 2010.\n"
+              << "\n"
+              << "Chung, T. and Fang, L. and Gildea, D. (2011)\n"
+              << "\"Issues Concerning Decoding with Synchronous Context-free Grammar\", In\nProceedings of ACL/HLT 2011.";
+
+  // Declare the command line options that are visible to the user.
+  po::options_description visible(usageTop.str());
+  visible.add_options()
+  //("help", "print this help message and exit")
+  ("AllowUnary",
+   "allow fully non-lexical unary rules")
+  ("ConditionOnTargetLHS",
+   "write target LHS instead of \"X\" as source LHS")
+  ("GlueGrammar",
+   po::value(&options.glueGrammarFile),
+   "write glue grammar to named file")
+  ("GZOutput",
+   "write gzipped extract files")
+  ("IncludeSentenceId",
+   "include sentence ID")
+  ("MaxNodes",
+   po::value(&options.maxNodes)->default_value(options.maxNodes),
+   "set maximum number of tree nodes for composed rules")
+  ("MaxRuleDepth",
+   po::value(&options.maxRuleDepth)->default_value(options.maxRuleDepth),
+   "set maximum depth for composed rules")
+  ("MaxRuleSize",
+   po::value(&options.maxRuleSize)->default_value(options.maxRuleSize),
+   "set maximum size for composed rules")
+  ("MaxScope",
+   po::value(&options.maxScope)->default_value(options.maxScope),
+   "set maximum allowed scope")
+  ("Minimal",
+   "extract minimal rules only")
+  ("PartsOfSpeech",
+   "output parts-of-speech as property (preterminals from the parse tree)")
+  ("PartsOfSpeechFactor",
+   "output parts-of-speech as factor (preterminals from the parse tree)")
+  ("PCFG",
+   "include score based on PCFG scores in target corpus")
+  ("PhraseOrientation",
+   "output phrase orientation information")
+  ("StripBitParLabels",
+   "strip suffix starting with a hyphen symbol (\"-\") from non-terminal labels")
+  ("STSG",
+   "output STSG rules (default is SCFG)")
+  ("T2S",
+   "enable tree-to-string rule extraction (string-to-tree is assumed by default)")
+  ("TreeFragments",
+   "output parse tree information")
+  ("SourceLabels",
+   "output source syntax label information")
+  ("SourceLabelSet",
+   po::value(&options.sourceLabelSetFile),
+   "write source syntax label set to named file")
+  ("SentenceOffset",
+   po::value(&options.sentenceOffset)->default_value(options.sentenceOffset),
+   "set sentence number offset if processing split corpus")
+  ("UnknownWordLabel",
+   po::value(&options.targetUnknownWordFile),
+   "write unknown word labels to named file")
+  ("SourceUnknownWordLabel",
+   po::value(&options.sourceUnknownWordFile),
+   "write source syntax unknown word labels to named file")
+  ("UnknownWordMinRelFreq",
+   po::value(&options.unknownWordMinRelFreq)->default_value(
+     options.unknownWordMinRelFreq),
+   "set minimum relative frequency for unknown word labels")
+  ("UnknownWordSoftMatches",
+   po::value(&options.unknownWordSoftMatchesFile),
+   "write dummy value to unknown word label file, and mappings from dummy value to other labels to named file")
+  ("UnknownWordUniform",
+   "write uniform weights to unknown word label file")
+  ("UnpairedExtractFormat",
+   "do not pair non-terminals in extract files")
+  ;
+
+  // Declare the command line options that are hidden from the user
+  // (these are used as positional options).
+  po::options_description hidden("Hidden options");
+  hidden.add_options()
+  ("TargetFile",
+   po::value(&options.targetFile),
+   "target file")
+  ("SourceFile",
+   po::value(&options.sourceFile),
+   "source file")
+  ("AlignmentFile",
+   po::value(&options.alignmentFile),
+   "alignment file")
+  ("ExtractFile",
+   po::value(&options.extractFile),
+   "extract file")
+  ;
+
+  // Compose the full set of command-line options.
+  po::options_description cmdLineOptions;
+  cmdLineOptions.add(visible).add(hidden);
+
+  // Register the positional options.
+  po::positional_options_description p;
+  p.add("TargetFile", 1);
+  p.add("SourceFile", 1);
+  p.add("AlignmentFile", 1);
+  p.add("ExtractFile", 1);
+
+  // Process the command-line.
+  po::variables_map vm;
+  try {
+    po::store(po::command_line_parser(argc, argv).style(MosesOptionStyle()).
+              options(cmdLineOptions).positional(p).run(), vm);
+    po::notify(vm);
+  } catch (const std::exception &e) {
+    std::ostringstream msg;
+    msg << e.what() << "\n\n" << visible << usageBottom.str();
+    Error(msg.str());
+  }
+
+  if (vm.count("help")) {
+    std::cout << visible << usageBottom.str() << std::endl;
+    std::exit(0);
+  }
+
+  // Check all positional options were given.
+  if (!vm.count("TargetFile") ||
+      !vm.count("SourceFile") ||
+      !vm.count("AlignmentFile") ||
+      !vm.count("ExtractFile")) {
+    std::ostringstream msg;
+    std::cerr << visible << usageBottom.str() << std::endl;
+    std::exit(1);
+  }
+
+  // Process Boolean options.
+  if (vm.count("AllowUnary")) {
+    options.allowUnary = true;
+  }
+  if (vm.count("ConditionOnTargetLHS")) {
+    options.conditionOnTargetLhs = true;
+  }
+  if (vm.count("GZOutput")) {
+    options.gzOutput = true;
+  }
+  if (vm.count("IncludeSentenceId")) {
+    options.includeSentenceId = true;
+  }
+  if (vm.count("Minimal")) {
+    options.minimal = true;
+  }
+  if (vm.count("PartsOfSpeech")) {
+    options.partsOfSpeech = true;
+  }
+  if (vm.count("PartsOfSpeechFactor")) {
+    options.partsOfSpeechFactor = true;
+  }
+  if (vm.count("PCFG")) {
+    options.pcfg = true;
+  }
+  if (vm.count("PhraseOrientation")) {
+    options.phraseOrientation = true;
+  }
+  if (vm.count("StripBitParLabels")) {
+    options.stripBitParLabels = true;
+  }
+  if (vm.count("STSG")) {
+    options.stsg = true;
+  }
+  if (vm.count("T2S")) {
+    options.t2s = true;
+  }
+  if (vm.count("TreeFragments")) {
+    options.treeFragments = true;
+  }
+  if (vm.count("SourceLabels")) {
+    options.sourceLabels = true;
+  }
+  if (vm.count("UnknownWordUniform")) {
+    options.unknownWordUniform = true;
+  }
+  if (vm.count("UnpairedExtractFormat")) {
+    options.unpairedExtractFormat = true;
+  }
+
+  // Workaround for extract-parallel issue.
+  if (options.sentenceOffset > 0) {
+    options.targetUnknownWordFile.clear();
+  }
+  if (options.sentenceOffset > 0) {
+    options.sourceUnknownWordFile.clear();
+    options.unknownWordSoftMatchesFile.clear();
+  }
+}
+
+std::vector<std::string> ExtractGHKM::ReadTokens(const std::string &s) const
+{
+  std::vector<std::string> tokens;
+
+  std::string whitespace = " \t";
+
+  std::string::size_type begin = s.find_first_not_of(whitespace);
+  assert(begin != std::string::npos);
+  while (true) {
+    std::string::size_type end = s.find_first_of(whitespace, begin);
+    std::string token;
+    if (end == std::string::npos) {
+      token = s.substr(begin);
+    } else {
+      token = s.substr(begin, end-begin);
+    }
+    tokens.push_back(token);
+    if (end == std::string::npos) {
+      break;
+    }
+    begin = s.find_first_not_of(whitespace, end);
+    if (begin == std::string::npos) {
+      break;
+    }
+  }
+
+  return tokens;
+}
+
+void ExtractGHKM::WriteGlueGrammar(
+  const std::set<std::string> &labelSet,
+  const std::map<std::string, int> &topLabelSet,
+  const std::map<std::string,size_t> &sourceLabels,
+  const Options &options,
+  std::ostream &out) const
+{
+  // choose a top label that is not already a label
+  std::string topLabel = "QQQQQQ";
+  for(size_t i = 1; i <= topLabel.length(); i++) {
+    if (labelSet.find(topLabel.substr(0,i)) == labelSet.end() ) {
+      topLabel = topLabel.substr(0,i);
+      break;
+    }
+  }
+
+  const size_t sourceLabelGlueTop = 0;
+  const size_t sourceLabelGlueX = 1;
+  const size_t sourceLabelSentenceStart = 2;
+  const size_t sourceLabelSentenceEnd = 3;
+//  const size_t partOfSpeechSentenceStart = 0;
+//  const size_t partOfSpeechSentenceEnd = 1;
+
+#ifndef BOS_
+#define BOS_ "<s>" //Beginning of sentence symbol
+#endif
+#ifndef EOS_
+#define EOS_ "</s>" //End of sentence symbol
+#endif
+
+  std::string sentenceStartSource = BOS_;
+  std::string sentenceEndSource   = EOS_;
+  std::string sentenceStartTarget = BOS_;
+  std::string sentenceEndTarget   = EOS_;
+  if (options.partsOfSpeech) {
+    sentenceStartTarget = sentenceStartTarget + "|" + BOS_;
+    sentenceEndTarget   = sentenceEndTarget   + "|" + EOS_;
+  }
+  if (options.partsOfSpeechFactor) {
+    sentenceStartTarget = sentenceStartTarget + "|" + BOS_;
+    sentenceEndTarget   = sentenceEndTarget   + "|" + EOS_;
+  }
+
+  // basic rules
+  out << sentenceStartSource << " [X] ||| " << sentenceStartTarget << " [" << topLabel << "] ||| 1 ||| 0-0 ||| ||| |||";
+  if (options.treeFragments) {
+    out << " {{Tree [" << topLabel << " [SSTART <s>]]}}";
+  }
+//  if (options.partsOfSpeech) {
+//    out << " {{POS " << partOfSpeechSentenceStart << "}}";
+//  }
+  if (options.sourceLabels) {
+    out << " {{SourceLabels 2 1 " << sourceLabelSentenceStart << " 1 1 " << sourceLabelGlueTop << " 1}}";
+  }
+  if (options.phraseOrientation) {
+    out << " {{Orientation 1 1 0.5 0.5 1 1 0.5 0.5}}";
+  }
+  out << std::endl;
+
+  out << "[X][" << topLabel << "] " << sentenceEndSource << " [X] ||| [X][" << topLabel << "] " << sentenceEndTarget << " [" << topLabel << "] ||| 1 ||| 0-0 1-1 ||| ||| |||";
+  if (options.treeFragments) {
+    out << " {{Tree [" << topLabel << " [" << topLabel << "] [SEND </s>]]}}";
+  }
+//  if (options.partsOfSpeech) {
+//    out << " {{POS " << partOfSpeechSentenceEnd << "}}";
+//  }
+  if (options.sourceLabels) {
+    out << " {{SourceLabels 4 1 " << sourceLabelSentenceStart << " " << sourceLabelGlueTop << " " << sourceLabelSentenceEnd << " 1 1 " << sourceLabelGlueTop << " 1}}";
+  }
+  if (options.phraseOrientation) {
+    out << " {{Orientation 1 1 0.5 0.5 1 1 0.5 0.5}}";
+  }
+  out << std::endl;
+
+  // top rules
+  for (std::map<std::string, int>::const_iterator i = topLabelSet.begin();
+       i != topLabelSet.end(); ++i) {
+    out << sentenceStartSource << " [X][" << i->first << "] " << sentenceEndSource << " [X] ||| " << sentenceStartTarget << " [X][" << i->first << "] " << sentenceEndTarget << " [" << topLabel << "] ||| 1 ||| 0-0 1-1 2-2 ||| ||| |||";
+    if (options.treeFragments) {
+      out << " {{Tree [" << topLabel << " [SSTART <s>] [" << i->first << "] [SEND </s>]]}}";
+    }
+//    if (options.partsOfSpeech) {
+//      out << " {{POS " << partOfSpeechSentenceStart << " " << partOfSpeechSentenceEnd << "}}";
+//    }
+    if (options.sourceLabels) {
+      out << " {{SourceLabels 4 1 " << sourceLabelSentenceStart << " " << sourceLabelGlueX << " " << sourceLabelSentenceEnd << " 1 1 " << sourceLabelGlueTop << " 1}}";
+    }
+    if (options.phraseOrientation) {
+      out << " {{Orientation 1 1 0.5 0.5 1 1 0.5 0.5}}";
+    }
+    out << std::endl;
+  }
+
+  // glue rules
+  for(std::set<std::string>::const_iterator i = labelSet.begin();
+      i != labelSet.end(); i++ ) {
+    out << "[X][" << topLabel << "] [X][" << *i << "] [X] ||| [X][" << topLabel << "] [X][" << *i << "] [" << topLabel << "] ||| 2.718 ||| 0-0 1-1 ||| ||| |||";
+    if (options.treeFragments) {
+      out << " {{Tree [" << topLabel << " ["<< topLabel << "] [" << *i << "]]}}";
+    }
+    if (options.sourceLabels) {
+      out << " {{SourceLabels 3 1 " << sourceLabelGlueTop << " " << sourceLabelGlueX << " 1 1 " << sourceLabelGlueTop << " 1}}";
+    }
+    if (options.phraseOrientation) {
+      out << " {{Orientation 1 1 0.5 0.5 1 1 0.5 0.5}}";
+    }
+    out << std::endl;
+  }
+
+  // glue rule for unknown word...
+  out << "[X][" << topLabel << "] [X][X] [X] ||| [X][" << topLabel << "] [X][X] [" << topLabel << "] ||| 2.718 ||| 0-0 1-1 ||| ||| |||";
+  if (options.treeFragments) {
+    out << " {{Tree [" << topLabel << " [" << topLabel << "] [X]]}}";
+  }
+  if (options.sourceLabels) {
+    out << " {{SourceLabels 3 1 " << sourceLabelGlueTop << " " << sourceLabelGlueX << " 1 1 " << sourceLabelGlueTop << " 1}}";
+  }
+  if (options.phraseOrientation) {
+    out << " {{Orientation 1 1 0.5 0.5 1 1 0.5 0.5}}";
+  }
+  out << std::endl;
+}
+
+void ExtractGHKM::WriteSourceLabelSet(
+  const std::map<std::string,size_t> &sourceLabels,
+  std::ostream &out) const
+{
+  out << sourceLabels.size() << std::endl;
+  for (std::map<std::string,size_t>::const_iterator iter=sourceLabels.begin();
+       iter!=sourceLabels.end(); ++iter) {
+    out << iter->first << " " << iter->second << std::endl;
+  }
+}
+
+void ExtractGHKM::CollectWordLabelCounts(
+  SyntaxTree &root,
+  const Options &options,
+  std::map<std::string, int> &wordCount,
+  std::map<std::string, std::string> &wordLabel)
+{
+  for (SyntaxTree::ConstLeafIterator p(root);
+       p != SyntaxTree::ConstLeafIterator(); ++p) {
+    const SyntaxTree &leaf = *p;
+    const std::string &word = leaf.value().label;
+    const SyntaxTree *ancestor = leaf.parent();
+    // If unary rule elimination is enabled and this word is at the end of a
+    // chain of unary rewrites, e.g.
+    //    PN-SB -> NE -> word
+    // then record the constituent label at the top of the chain instead of
+    // the part-of-speech label.
+    while (!options.allowUnary &&
+           ancestor->parent() &&
+           ancestor->parent()->children().size() == 1) {
+      ancestor = ancestor->parent();
+    }
+    const std::string &label = ancestor->value().label;
+    ++wordCount[word];
+    wordLabel[word] = label;
+  }
+}
+
+std::vector<std::string> ExtractGHKM::ReadTokens(const SyntaxTree &root) const
+{
+  std::vector<std::string> tokens;
+  for (SyntaxTree::ConstLeafIterator p(root);
+       p != SyntaxTree::ConstLeafIterator(); ++p) {
+    const SyntaxTree &leaf = *p;
+    const std::string &word = leaf.value().label;
+    tokens.push_back(word);
+  }
+  return tokens;
+}
+
+void ExtractGHKM::WriteUnknownWordLabel(
+  const std::map<std::string, int> &wordCount,
+  const std::map<std::string, std::string> &wordLabel,
+  const Options &options,
+  std::ostream &out,
+  bool writeCounts) const
+{
+  if (!options.unknownWordSoftMatchesFile.empty()) {
+    out << "UNK 1" << std::endl;
+    return;
+  }
+
+  std::map<std::string, int> labelCount;
+  int total = 0;
+  for (std::map<std::string, int>::const_iterator p = wordCount.begin();
+       p != wordCount.end(); ++p) {
+    // Only consider singletons.
+    if (p->second == 1) {
+      std::map<std::string, std::string>::const_iterator q =
+        wordLabel.find(p->first);
+      assert(q != wordLabel.end());
+      if (options.stripBitParLabels) {
+        size_t pos = q->second.find('-');
+        if (pos == std::string::npos) {
+          ++labelCount[q->second];
+        } else {
+          ++labelCount[q->second.substr(0,pos)];
+        }
+      } else {
+        ++labelCount[q->second];
+      }
+      ++total;
+    }
+  }
+  if ( writeCounts ) {
+    for (std::map<std::string, int>::const_iterator p = labelCount.begin();
+         p != labelCount.end(); ++p) {
+      out << p->first << " " << p->second << std::endl;
+    }
+  } else {
+    for (std::map<std::string, int>::const_iterator p = labelCount.begin();
+         p != labelCount.end(); ++p) {
+      double ratio = static_cast<double>(p->second) / static_cast<double>(total);
+      if (ratio >= options.unknownWordMinRelFreq) {
+        float weight = options.unknownWordUniform ? 1.0f : ratio;
+        out << p->first << " " << weight << std::endl;
+      }
+    }
+  }
+}
+
+void ExtractGHKM::WriteUnknownWordSoftMatches(
+  const std::set<std::string> &labelSet,
+  std::ostream &out) const
+{
+  for (std::set<std::string>::const_iterator p = labelSet.begin(); p != labelSet.end(); ++p) {
+    std::string label = *p;
+    out << "UNK " << label << std::endl;
+  }
+}
+
+void ExtractGHKM::StripBitParLabels(
+  const std::set<std::string> &labelSet,
+  const std::map<std::string, int> &topLabelSet,
+  std::set<std::string> &outLabelSet,
+  std::map<std::string, int> &outTopLabelSet) const
+{
+  for (std::set<std::string>::const_iterator it=labelSet.begin();
+       it!=labelSet.end(); ++it) {
+    size_t pos = it->find('-');
+    if (pos == std::string::npos) {
+      outLabelSet.insert(*it);
+    } else {
+      outLabelSet.insert(it->substr(0,pos));
+    }
+  }
+  for (std::map<std::string,int>::const_iterator it=topLabelSet.begin();
+       it!=topLabelSet.end(); ++it) {
+    size_t pos = it->first.find('-');
+    std::string stripped;
+    if (pos == std::string::npos) {
+      stripped = it->first;
+    } else {
+      stripped = it->first.substr(0,pos);
+    }
+    std::map<std::string, int>::iterator found=outTopLabelSet.find(stripped);
+    if (found != outTopLabelSet.end()) {
+      found->second += it->second;
+    } else {
+      outTopLabelSet.insert(std::pair<std::string,int>(stripped,it->second));
+    }
+  }
+}
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining

mosesdecoder/phrase-extract/extract-ghkm/ExtractGHKM.h

@@ -0,0 +1,82 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#pragma once
+
+#include <map>
+#include <ostream>
+#include <set>
+#include <string>
+#include <vector>
+
+#include "OutputFileStream.h"
+#include "SyntaxTree.h"
+
+#include "syntax-common/tool.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+struct Options;
+
+class ExtractGHKM : public Tool
+{
+public:
+  ExtractGHKM() : Tool("extract-ghkm") {}
+
+  virtual int Main(int argc, char *argv[]);
+
+private:
+  void RecordTreeLabels(const SyntaxTree &, std::set<std::string> &);
+  void CollectWordLabelCounts(SyntaxTree &,
+                              const Options &,
+                              std::map<std::string, int> &,
+                              std::map<std::string, std::string> &);
+  void WriteUnknownWordLabel(const std::map<std::string, int> &,
+                             const std::map<std::string, std::string> &,
+                             const Options &,
+                             std::ostream &,
+                             bool writeCounts=false) const;
+  void WriteUnknownWordSoftMatches(const std::set<std::string> &,
+                                   std::ostream &) const;
+  void WriteGlueGrammar(const std::set<std::string> &,
+                        const std::map<std::string, int> &,
+                        const std::map<std::string,size_t> &,
+                        const Options &,
+                        std::ostream &) const;
+  void WriteSourceLabelSet(const std::map<std::string,size_t> &,
+                           std::ostream &) const;
+  void StripBitParLabels(const std::set<std::string> &labelSet,
+                         const std::map<std::string, int> &topLabelSet,
+                         std::set<std::string> &outLabelSet,
+                         std::map<std::string, int> &outTopLabelSet) const;
+
+  std::vector<std::string> ReadTokens(const std::string &) const;
+  std::vector<std::string> ReadTokens(const SyntaxTree &root) const;
+
+  void ProcessOptions(int, char *[], Options &) const;
+};
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining

mosesdecoder/phrase-extract/extract-ghkm/Jamfile

@@ -0,0 +1 @@
+exe extract-ghkm : [ glob *.cpp ] ..//syntax-common ..//deps ../..//boost_iostreams ../..//boost_program_options ../..//z : <include>.. ;

mosesdecoder/phrase-extract/extract-ghkm/Main.cpp

@@ -0,0 +1,26 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#include "ExtractGHKM.h"
+
+int main(int argc, char *argv[])
+{
+  MosesTraining::Syntax::GHKM::ExtractGHKM tool;
+  return tool.Main(argc, argv);
+}

mosesdecoder/phrase-extract/extract-ghkm/Node.cpp

@@ -0,0 +1,76 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#include "Node.h"
+
+#include "Subgraph.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+Node::~Node()
+{
+  for (std::vector<const Subgraph*>::const_iterator p(m_rules.begin());
+       p != m_rules.end(); ++p) {
+    delete *p;
+  }
+}
+
+bool Node::IsPreterminal() const
+{
+  return (m_type == TREE
+          && m_children.size() == 1
+          && m_children[0]->m_type == TARGET);
+}
+
+void Node::PropagateIndex(int index)
+{
+  m_span.insert(index);
+  for (std::vector<Node *>::const_iterator p(m_parents.begin());
+       p != m_parents.end(); ++p) {
+    (*p)->PropagateIndex(index);
+  }
+}
+
+std::vector<std::string> Node::GetTargetWords() const
+{
+  std::vector<std::string> targetWords;
+  GetTargetWords(targetWords);
+  return targetWords;
+}
+
+void Node::GetTargetWords(std::vector<std::string> &targetWords) const
+{
+  if (m_type == TARGET) {
+    targetWords.push_back(m_label);
+  } else {
+    for (std::vector<Node *>::const_iterator p(m_children.begin());
+         p != m_children.end(); ++p) {
+      (*p)->GetTargetWords(targetWords);
+    }
+  }
+}
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining

mosesdecoder/phrase-extract/extract-ghkm/Node.h

@@ -0,0 +1,223 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#pragma once
+#ifndef EXTRACT_GHKM_NODE_H_
+#define EXTRACT_GHKM_NODE_H_
+
+#include <cassert>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "Span.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+class Subgraph;
+
+enum NodeType { SOURCE, TARGET, TREE };
+
+class Node
+{
+public:
+  Node(const std::string &label, NodeType type)
+    : m_label(label)
+    , m_type(type)
+    , m_pcfgScore(0.0f) {}
+
+  ~Node();
+
+  const std::string &GetLabel() const {
+    return m_label;
+  }
+  NodeType GetType() const {
+    return m_type;
+  }
+  const std::vector<Node*> &GetChildren() const {
+    return m_children;
+  }
+  const std::vector<Node*> &GetParents() const {
+    return m_parents;
+  }
+  float GetPcfgScore() const {
+    return m_pcfgScore;
+  }
+  const Span &GetSpan() const {
+    return m_span;
+  }
+  const Span &GetComplementSpan() const {
+    return m_complementSpan;
+  }
+  const std::vector<const Subgraph*> &GetRules() const {
+    return m_rules;
+  }
+
+  void SetChildren(const std::vector<Node*> &c) {
+    m_children = c;
+  }
+  void SetParents(const std::vector<Node*> &p) {
+    m_parents = p;
+  }
+  void SetPcfgScore(float s) {
+    m_pcfgScore = s;
+  }
+  void SetSpan(const Span &s) {
+    m_span = s;
+  }
+  void SetComplementSpan(const Span &cs) {
+    m_complementSpan = cs;
+  }
+
+  void AddChild(Node *c) {
+    m_children.push_back(c);
+  }
+  void AddParent(Node *p) {
+    m_parents.push_back(p);
+  }
+  void AddRule(const Subgraph *s) {
+    m_rules.push_back(s);
+  }
+
+  bool IsSink() const {
+    return m_children.empty();
+  }
+  bool IsPreterminal() const;
+
+  void PropagateIndex(int);
+
+  std::vector<std::string> GetTargetWords() const;
+
+  // Gets the path from this node's parent to the root.  This node is
+  // required to be part of the original parse tree (i.e. not a source word,
+  // which can have multiple parents).
+  template<typename OutputIterator>
+  void GetTreeAncestors(OutputIterator result, bool includeSelf=false);
+
+  // Returns the lowest common ancestor given a sequence of nodes belonging to
+  // the target tree.
+  template<typename InputIterator>
+  static Node *LowestCommonAncestor(InputIterator first, InputIterator last);
+
+private:
+  // Disallow copying
+  Node(const Node &);
+  Node &operator=(const Node &);
+
+  void GetTargetWords(std::vector<std::string> &) const;
+
+  std::string m_label;
+  NodeType m_type;
+  std::vector<Node*> m_children;
+  std::vector<Node*> m_parents;
+  float m_pcfgScore;
+  Span m_span;
+  Span m_complementSpan;
+  std::vector<const Subgraph*> m_rules;
+};
+
+template<typename OutputIterator>
+void Node::GetTreeAncestors(OutputIterator result, bool includeSelf)
+{
+  // This function assumes the node is part of the parse tree.
+  assert(m_type == TARGET || m_type == TREE);
+
+  if (includeSelf) {
+    *result++ = this;
+  }
+
+  Node *ancestor = !(m_parents.empty()) ? m_parents[0] : 0;
+  while (ancestor != 0) {
+    *result++ = ancestor;
+    ancestor = !(ancestor->m_parents.empty()) ? ancestor->m_parents[0] : 0;
+  }
+}
+
+template<typename InputIterator>
+Node *Node::LowestCommonAncestor(InputIterator first, InputIterator last)
+{
+  // Check for an empty sequence.
+  if (first == last) {
+    return 0;
+  }
+
+  // Check for the case that the sequence contains only one distinct node.
+  // Also check that every node belongs to the target tree.
+  InputIterator p = first;
+  Node *lca = *p++;
+  for (; p != last; ++p) {
+    Node *node = *p;
+    assert(node->m_type != SOURCE);
+    if (node != lca) {
+      lca = 0;
+    }
+  }
+  if (lca) {
+    return lca;
+  }
+
+  // Now construct an ancestor path for each node, from itself to the root.
+  size_t minPathLength = 0;
+  std::vector<std::vector<Node *> > paths;
+  for (p = first; p != last; ++p) {
+    paths.resize(paths.size()+1);
+    (*p)->GetTreeAncestors(std::back_inserter(paths.back()), true);
+    size_t pathLength = paths.back().size();
+    assert(pathLength > 0);
+    if (paths.size() == 1 || pathLength < minPathLength) {
+      minPathLength = pathLength;
+    }
+  }
+
+  // Search for the start of the longest common suffix by working forward from
+  // the the earliest possible starting point to the root.
+  for (size_t i = 0; i < minPathLength; ++i) {
+    bool match = true;
+    for (size_t j = 0; j < paths.size(); ++j) {
+      size_t index = paths[j].size() - minPathLength + i;
+      assert(index >= 0);
+      assert(index < paths[j].size());
+      if (j == 0) {
+        lca = paths[j][index];
+        assert(lca);
+      } else if (lca != paths[j][index]) {
+        match = false;
+        break;
+      }
+    }
+    if (match) {
+      return lca;
+    }
+  }
+
+  // A lowest common ancestor should have been found.
+  assert(false);
+  return 0;
+}
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining
+
+#endif

mosesdecoder/phrase-extract/extract-ghkm/Options.h

@@ -0,0 +1,95 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#pragma once
+
+#include <string>
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+struct Options {
+public:
+  Options()
+    : allowUnary(false)
+    , conditionOnTargetLhs(false)
+    , gzOutput(false)
+    , includeSentenceId(false)
+    , maxNodes(15)
+    , maxRuleDepth(3)
+    , maxRuleSize(3)
+    , maxScope(3)
+    , minimal(false)
+    , partsOfSpeech(false)
+    , partsOfSpeechFactor(false)
+    , pcfg(false)
+    , phraseOrientation(false)
+    , sentenceOffset(0)
+    , sourceLabels(false)
+    , stripBitParLabels(false)
+    , stsg(false)
+    , t2s(false)
+    , treeFragments(false)
+    , unknownWordMinRelFreq(0.03f)
+    , unknownWordUniform(false)
+    , unpairedExtractFormat(false) {}
+
+  // Positional options
+  std::string targetFile;
+  std::string sourceFile;
+  std::string alignmentFile;
+  std::string extractFile;
+
+  // All other options
+  bool allowUnary;
+  bool conditionOnTargetLhs;
+  std::string glueGrammarFile;
+  bool gzOutput;
+  bool includeSentenceId;
+  int maxNodes;
+  int maxRuleDepth;
+  int maxRuleSize;
+  int maxScope;
+  bool minimal;
+  bool partsOfSpeech;
+  bool partsOfSpeechFactor;
+  bool pcfg;
+  bool phraseOrientation;
+  int sentenceOffset;
+  bool sourceLabels;
+  std::string sourceLabelSetFile;
+  std::string sourceUnknownWordFile;
+  bool stripBitParLabels;
+  bool stsg;
+  bool t2s;
+  std::string targetUnknownWordFile;
+  bool treeFragments;
+  float unknownWordMinRelFreq;
+  std::string unknownWordSoftMatchesFile;
+  bool unknownWordUniform;
+  bool unpairedExtractFormat;
+};
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining

mosesdecoder/phrase-extract/extract-ghkm/Rule.cpp

@@ -0,0 +1,44 @@
+#include "Rule.h"
+
+#include "Node.h"
+#include "Subgraph.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+int Rule::Scope(const std::vector<Symbol> &symbols)
+{
+  int scope = 0;
+  bool predIsNonTerm = false;
+  if (symbols[0].GetType() == NonTerminal) {
+    ++scope;
+    predIsNonTerm = true;
+  }
+  for (std::size_t i = 1; i < symbols.size(); ++i) {
+    bool isNonTerm = symbols[i].GetType() == NonTerminal;
+    if (isNonTerm && predIsNonTerm) {
+      ++scope;
+    }
+    predIsNonTerm = isNonTerm;
+  }
+  if (predIsNonTerm) {
+    ++scope;
+  }
+  return scope;
+}
+
+bool Rule::PartitionOrderComp(const Node *a, const Node *b)
+{
+  const Span &aSpan = a->GetSpan();
+  const Span &bSpan = b->GetSpan();
+  assert(!aSpan.empty() && !bSpan.empty());
+  return *(aSpan.begin()) < *(bSpan.begin());
+}
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining

mosesdecoder/phrase-extract/extract-ghkm/Rule.h

@@ -0,0 +1,62 @@
+#pragma once
+#ifndef EXTRACT_GHKM_RULE_H_
+#define EXTRACT_GHKM_RULE_H_
+
+#include <string>
+#include <vector>
+
+#include "Alignment.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+class Node;
+
+enum SymbolType { Terminal, NonTerminal };
+
+class Symbol
+{
+public:
+  Symbol(const std::string &v, SymbolType t) : m_value(v) , m_type(t) {}
+
+  const std::string &GetValue() const {
+    return m_value;
+  }
+  SymbolType GetType() const {
+    return m_type;
+  }
+
+private:
+  std::string m_value;
+  SymbolType m_type;
+};
+
+// Base class for ScfgRule and StsgRule.
+class Rule
+{
+public:
+  virtual ~Rule() {}
+
+  const Alignment &GetAlignment() const {
+    return m_alignment;
+  }
+
+  virtual int Scope() const = 0;
+
+protected:
+  static bool PartitionOrderComp(const Node *, const Node *);
+
+  static int Scope(const std::vector<Symbol>&);
+
+  Alignment m_alignment;
+};
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining
+
+#endif

mosesdecoder/phrase-extract/extract-ghkm/ScfgRule.cpp

@@ -0,0 +1,203 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#include "ScfgRule.h"
+
+#include <algorithm>
+
+#include "Node.h"
+#include "Subgraph.h"
+#include "SyntaxNode.h"
+#include "SyntaxNodeCollection.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+ScfgRule::ScfgRule(const Subgraph &fragment,
+                   const SyntaxNodeCollection *sourceNodeCollection)
+  : m_graphFragment(fragment)
+  , m_sourceLHS("X", NonTerminal)
+  , m_targetLHS(fragment.GetRoot()->GetLabel(), NonTerminal)
+  , m_pcfgScore(fragment.GetPcfgScore())
+  , m_hasSourceLabels(sourceNodeCollection)
+{
+
+  // Source RHS
+
+  const std::set<const Node *> &leaves = fragment.GetLeaves();
+
+  std::vector<const Node *> sourceRHSNodes;
+  sourceRHSNodes.reserve(leaves.size());
+  for (std::set<const Node *>::const_iterator p(leaves.begin());
+       p != leaves.end(); ++p) {
+    const Node &leaf = **p;
+    if (!leaf.GetSpan().empty()) {
+      sourceRHSNodes.push_back(&leaf);
+    }
+  }
+
+  std::sort(sourceRHSNodes.begin(), sourceRHSNodes.end(), PartitionOrderComp);
+
+  // Build a mapping from target nodes to source-order indices, so that we
+  // can construct the Alignment object later.
+  std::map<const Node *, std::vector<int> > sourceOrder;
+
+  m_sourceRHS.reserve(sourceRHSNodes.size());
+  m_numberOfNonTerminals = 0;
+  int srcIndex = 0;
+  for (std::vector<const Node *>::const_iterator p(sourceRHSNodes.begin());
+       p != sourceRHSNodes.end(); ++p, ++srcIndex) {
+    const Node &sinkNode = **p;
+    if (sinkNode.GetType() == TREE) {
+      m_sourceRHS.push_back(Symbol("X", NonTerminal));
+      sourceOrder[&sinkNode].push_back(srcIndex);
+      ++m_numberOfNonTerminals;
+    } else {
+      assert(sinkNode.GetType() == SOURCE);
+      m_sourceRHS.push_back(Symbol(sinkNode.GetLabel(), Terminal));
+      // Add all aligned target words to the sourceOrder map
+      const std::vector<Node *> &parents(sinkNode.GetParents());
+      for (std::vector<Node *>::const_iterator q(parents.begin());
+           q != parents.end(); ++q) {
+        if ((*q)->GetType() == TARGET) {
+          sourceOrder[*q].push_back(srcIndex);
+        }
+      }
+    }
+    if (sourceNodeCollection) {
+      // Source syntax label
+      PushSourceLabel(sourceNodeCollection,&sinkNode,"XRHS");
+    }
+  }
+
+  // Target RHS + alignment
+
+  std::vector<const Node *> targetLeaves;
+  fragment.GetTargetLeaves(targetLeaves);
+
+  m_alignment.reserve(targetLeaves.size());  // might be too much but that's OK
+  m_targetRHS.reserve(targetLeaves.size());
+
+  for (std::vector<const Node *>::const_iterator p(targetLeaves.begin());
+       p != targetLeaves.end(); ++p) {
+    const Node &leaf = **p;
+    if (leaf.GetSpan().empty()) {
+      // The node doesn't cover any source words, so we can only add
+      // terminals to the target RHS (not a non-terminal).
+      std::vector<std::string> targetWords(leaf.GetTargetWords());
+      for (std::vector<std::string>::const_iterator q(targetWords.begin());
+           q != targetWords.end(); ++q) {
+        m_targetRHS.push_back(Symbol(*q, Terminal));
+      }
+    } else if (leaf.GetType() == SOURCE) {
+      // Do nothing
+    } else {
+      SymbolType type = (leaf.GetType() == TREE) ? NonTerminal : Terminal;
+      m_targetRHS.push_back(Symbol(leaf.GetLabel(), type));
+
+      int tgtIndex = m_targetRHS.size()-1;
+      std::map<const Node *, std::vector<int> >::iterator q(sourceOrder.find(&leaf));
+      assert(q != sourceOrder.end());
+      std::vector<int> &sourceNodes = q->second;
+      for (std::vector<int>::iterator r(sourceNodes.begin());
+           r != sourceNodes.end(); ++r) {
+        int srcIndex = *r;
+        m_alignment.push_back(std::make_pair(srcIndex, tgtIndex));
+      }
+    }
+  }
+
+  if (sourceNodeCollection) {
+    // Source syntax label for root node (if sourceNodeCollection available)
+    PushSourceLabel(sourceNodeCollection,fragment.GetRoot(),"XLHS");
+    // All non-terminal spans (including the LHS) should have obtained a label
+    // (a source-side syntactic constituent label if the span matches, "XLHS" otherwise)
+//    assert(m_sourceLabels.size() == m_numberOfNonTerminals+1);
+  }
+}
+
+void ScfgRule::PushSourceLabel(const SyntaxNodeCollection *sourceNodeCollection,
+                               const Node *node,
+                               const std::string &nonMatchingLabel)
+{
+  ContiguousSpan span = Closure(node->GetSpan());
+  if (sourceNodeCollection->HasNode(span.first,span.second)) { // does a source constituent match the span?
+    std::vector<SyntaxNode*> sourceLabels =
+      sourceNodeCollection->GetNodes(span.first,span.second);
+    if (!sourceLabels.empty()) {
+      // store the topmost matching label from the source syntax tree
+      m_sourceLabels.push_back(sourceLabels.back()->label);
+    }
+  } else {
+    // no matching source-side syntactic constituent: store nonMatchingLabel
+    m_sourceLabels.push_back(nonMatchingLabel);
+  }
+}
+
+// TODO: rather implement the method external to ScfgRule
+void ScfgRule::UpdateSourceLabelCoocCounts(std::map< std::string, std::map<std::string,float>* > &coocCounts, float count) const
+{
+  std::map<int, int> sourceToTargetNTMap;
+  std::map<int, int> targetToSourceNTMap;
+
+  for (Alignment::const_iterator p(m_alignment.begin());
+       p != m_alignment.end(); ++p) {
+    if ( m_sourceRHS[p->first].GetType() == NonTerminal ) {
+      assert(m_targetRHS[p->second].GetType() == NonTerminal);
+      sourceToTargetNTMap[p->first] = p->second;
+    }
+  }
+
+  size_t sourceIndex = 0;
+  size_t sourceNonTerminalIndex = 0;
+  for (std::vector<Symbol>::const_iterator p=m_sourceRHS.begin();
+       p != m_sourceRHS.end(); ++p, ++sourceIndex) {
+    if ( p->GetType() == NonTerminal ) {
+      const std::string &sourceLabel = m_sourceLabels[sourceNonTerminalIndex];
+      int targetIndex = sourceToTargetNTMap[sourceIndex];
+      const std::string &targetLabel = m_targetRHS[targetIndex].GetValue();
+      ++sourceNonTerminalIndex;
+
+      std::map<std::string,float>* countMap = NULL;
+      std::map< std::string, std::map<std::string,float>* >::iterator iter = coocCounts.find(sourceLabel);
+      if ( iter == coocCounts.end() ) {
+        std::map<std::string,float> *newCountMap = new std::map<std::string,float>();
+        std::pair< std::map< std::string, std::map<std::string,float>* >::iterator, bool > inserted =
+          coocCounts.insert( std::pair< std::string, std::map<std::string,float>* >(sourceLabel, newCountMap) );
+        assert(inserted.second);
+        countMap = (inserted.first)->second;
+      } else {
+        countMap = iter->second;
+      }
+      std::pair< std::map<std::string,float>::iterator, bool > inserted =
+        countMap->insert( std::pair< std::string,float>(targetLabel, count) );
+      if ( !inserted.second ) {
+        (inserted.first)->second += count;
+      }
+    }
+  }
+}
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining

mosesdecoder/phrase-extract/extract-ghkm/ScfgRule.h

@@ -0,0 +1,99 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <list>
+#include <memory>
+#include <iostream>
+
+#include "Alignment.h"
+#include "Rule.h"
+#include "SyntaxNodeCollection.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+class Node;
+class Subgraph;
+
+class ScfgRule : public Rule
+{
+public:
+  ScfgRule(const Subgraph &fragment,
+           const SyntaxNodeCollection *sourceNodeCollection = 0);
+
+  const Subgraph &GetGraphFragment() const {
+    return m_graphFragment;
+  }
+  const Symbol &GetSourceLHS() const {
+    return m_sourceLHS;
+  }
+  const Symbol &GetTargetLHS() const {
+    return m_targetLHS;
+  }
+  const std::vector<Symbol> &GetSourceRHS() const {
+    return m_sourceRHS;
+  }
+  const std::vector<Symbol> &GetTargetRHS() const {
+    return m_targetRHS;
+  }
+  float GetPcfgScore() const {
+    return m_pcfgScore;
+  }
+  bool HasSourceLabels() const {
+    return m_hasSourceLabels;
+  }
+  void PrintSourceLabels(std::ostream &out) const {
+    for (std::vector<std::string>::const_iterator it = m_sourceLabels.begin();
+         it != m_sourceLabels.end(); ++it) {
+      out << " " << (*it);
+    }
+  }
+  void UpdateSourceLabelCoocCounts(std::map< std::string, std::map<std::string,float>* > &coocCounts,
+                                   float count) const;
+
+  int Scope() const {
+    return Rule::Scope(m_sourceRHS);
+  }
+
+private:
+  void PushSourceLabel(const SyntaxNodeCollection *sourceNodeCollection,
+                       const Node *node, const std::string &nonMatchingLabel);
+
+  const Subgraph& m_graphFragment;
+  Symbol m_sourceLHS;
+  Symbol m_targetLHS;
+  std::vector<Symbol> m_sourceRHS;
+  std::vector<Symbol> m_targetRHS;
+  float m_pcfgScore;
+  bool m_hasSourceLabels;
+  std::vector<std::string> m_sourceLabels;
+  unsigned m_numberOfNonTerminals;
+};
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining

mosesdecoder/phrase-extract/extract-ghkm/ScfgRuleWriter.cpp

@@ -0,0 +1,235 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#include "ScfgRuleWriter.h"
+
+#include <cassert>
+#include <cmath>
+#include <ostream>
+#include <map>
+#include <sstream>
+#include <vector>
+
+#include "Alignment.h"
+#include "Options.h"
+#include "ScfgRule.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+void ScfgRuleWriter::Write(const ScfgRule &rule, size_t lineNum, bool printEndl)
+{
+  std::ostringstream sourceSS;
+  std::ostringstream targetSS;
+
+  if (m_options.unpairedExtractFormat) {
+    WriteUnpairedFormat(rule, sourceSS, targetSS);
+  } else {
+    WriteStandardFormat(rule, sourceSS, targetSS);
+  }
+
+  // Write the rule to the forward and inverse extract files.
+  if (m_options.t2s) {
+    // If model is tree-to-string then flip the source and target.
+    m_fwd << targetSS.str() << " ||| " << sourceSS.str() << " |||";
+    m_inv << sourceSS.str() << " ||| " << targetSS.str() << " |||";
+  } else {
+    m_fwd << sourceSS.str() << " ||| " << targetSS.str() << " |||";
+    m_inv << targetSS.str() << " ||| " << sourceSS.str() << " |||";
+  }
+
+  const Alignment &alignment = rule.GetAlignment();
+  for (Alignment::const_iterator p = alignment.begin();
+       p != alignment.end(); ++p) {
+    if (m_options.t2s) {
+      // If model is tree-to-string then flip the source and target.
+      m_fwd << " " << p->second << "-" << p->first;
+      m_inv << " " << p->first << "-" << p->second;
+    } else {
+      m_fwd << " " << p->first << "-" << p->second;
+      m_inv << " " << p->second << "-" << p->first;
+    }
+  }
+
+  if (m_options.includeSentenceId) {
+    if (m_options.t2s) {
+      m_inv << " ||| " << lineNum;
+    } else {
+      m_fwd << " ||| " << lineNum;
+    }
+  }
+
+  // Write a count of 1.
+  m_fwd << " ||| 1";
+  m_inv << " ||| 1";
+
+  // Write the PCFG score (if requested).
+  if (m_options.pcfg) {
+    m_fwd << " ||| " << std::exp(rule.GetPcfgScore());
+  }
+
+  m_fwd << " |||";
+
+  if (m_options.sourceLabels && rule.HasSourceLabels()) {
+    m_fwd << " {{SourceLabels";
+    rule.PrintSourceLabels(m_fwd);
+    m_fwd << "}}";
+  }
+
+  if (printEndl) {
+    m_fwd << std::endl;
+    m_inv << std::endl;
+  }
+}
+
+void ScfgRuleWriter::WriteStandardFormat(const ScfgRule &rule,
+    std::ostream &sourceSS,
+    std::ostream &targetSS)
+{
+  const std::vector<Symbol> &sourceRHS = rule.GetSourceRHS();
+  const std::vector<Symbol> &targetRHS = rule.GetTargetRHS();
+
+  std::map<int, int> sourceToTargetNTMap;
+  std::map<int, int> targetToSourceNTMap;
+
+  const Alignment &alignment = rule.GetAlignment();
+
+  for (Alignment::const_iterator p(alignment.begin());
+       p != alignment.end(); ++p) {
+    if (sourceRHS[p->first].GetType() == NonTerminal) {
+      assert(targetRHS[p->second].GetType() == NonTerminal);
+      sourceToTargetNTMap[p->first] = p->second;
+      targetToSourceNTMap[p->second] = p->first;
+    }
+  }
+
+  // If parts-of-speech as a factor requested: retrieve preterminals from graph fragment
+  std::vector<std::string> partsOfSpeech;
+  if (m_options.partsOfSpeechFactor) {
+    const Subgraph &graphFragment = rule.GetGraphFragment();
+    graphFragment.GetPartsOfSpeech(partsOfSpeech);
+  }
+
+  // Write the source side of the rule to sourceSS.
+  int i = 0;
+  for (std::vector<Symbol>::const_iterator p(sourceRHS.begin());
+       p != sourceRHS.end(); ++p, ++i) {
+    WriteSymbol(*p, sourceSS);
+    if (p->GetType() == NonTerminal) {
+      int targetIndex = sourceToTargetNTMap[i];
+      WriteSymbol(targetRHS[targetIndex], sourceSS);
+    }
+    sourceSS << " ";
+  }
+  if (m_options.conditionOnTargetLhs) {
+    WriteSymbol(rule.GetTargetLHS(), sourceSS);
+  } else {
+    WriteSymbol(rule.GetSourceLHS(), sourceSS);
+  }
+
+  // Write the target side of the rule to targetSS.
+  i = 0;
+  int targetTerminalIndex = 0;
+  for (std::vector<Symbol>::const_iterator p(targetRHS.begin());
+       p != targetRHS.end(); ++p, ++i) {
+    if (p->GetType() == NonTerminal) {
+      int sourceIndex = targetToSourceNTMap[i];
+      WriteSymbol(sourceRHS[sourceIndex], targetSS);
+    }
+    WriteSymbol(*p, targetSS);
+    // If parts-of-speech as a factor requested: write part-of-speech
+    if (m_options.partsOfSpeechFactor && (p->GetType() != NonTerminal)) {
+      assert(targetTerminalIndex<partsOfSpeech.size());
+      targetSS << "|" << partsOfSpeech[targetTerminalIndex];
+      ++targetTerminalIndex;
+    }
+    targetSS << " ";
+  }
+  WriteSymbol(rule.GetTargetLHS(), targetSS);
+}
+
+void ScfgRuleWriter::WriteUnpairedFormat(const ScfgRule &rule,
+    std::ostream &sourceSS,
+    std::ostream &targetSS)
+{
+  const std::vector<Symbol> &sourceRHS = rule.GetSourceRHS();
+  const std::vector<Symbol> &targetRHS = rule.GetTargetRHS();
+
+  // If parts-of-speech as a factor requested: retrieve preterminals from graph fragment
+  std::vector<std::string> partsOfSpeech;
+  if (m_options.partsOfSpeechFactor) {
+    const Subgraph &graphFragment = rule.GetGraphFragment();
+    graphFragment.GetPartsOfSpeech(partsOfSpeech);
+  }
+
+  // Write the source side of the rule to sourceSS.
+  for (std::vector<Symbol>::const_iterator p(sourceRHS.begin());
+       p != sourceRHS.end(); ++p) {
+    WriteSymbol(*p, sourceSS);
+    sourceSS << " ";
+  }
+  if (m_options.conditionOnTargetLhs) {
+    WriteSymbol(rule.GetTargetLHS(), sourceSS);
+  } else {
+    WriteSymbol(rule.GetSourceLHS(), sourceSS);
+  }
+
+  // Write the target side of the rule to targetSS.
+  int targetTerminalIndex = 0;
+  for (std::vector<Symbol>::const_iterator p(targetRHS.begin());
+       p != targetRHS.end(); ++p) {
+    WriteSymbol(*p, targetSS);
+    // If parts-of-speech as a factor requested: write part-of-speech
+    if (m_options.partsOfSpeechFactor && (p->GetType() != NonTerminal)) {
+      assert(targetTerminalIndex<partsOfSpeech.size());
+      targetSS << "|" << partsOfSpeech[targetTerminalIndex];
+      ++targetTerminalIndex;
+    }
+    targetSS << " ";
+  }
+  WriteSymbol(rule.GetTargetLHS(), targetSS);
+}
+
+void ScfgRuleWriter::WriteSymbol(const Symbol &symbol, std::ostream &out)
+{
+  if (symbol.GetType() == NonTerminal) {
+    out << "[";
+    if (m_options.stripBitParLabels) {
+      size_t pos = symbol.GetValue().find('-');
+      if (pos == std::string::npos) {
+        out << symbol.GetValue();
+      } else {
+        out << symbol.GetValue().substr(0,pos);
+      }
+    } else {
+      out << symbol.GetValue();
+    }
+    out << "]";
+  } else {
+    out << symbol.GetValue();
+  }
+}
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining

mosesdecoder/phrase-extract/extract-ghkm/ScfgRuleWriter.h

@@ -0,0 +1,63 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#pragma once
+
+#include <ostream>
+
+#include "Subgraph.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+struct Options;
+class ScfgRule;
+class Symbol;
+
+class ScfgRuleWriter
+{
+public:
+  ScfgRuleWriter(std::ostream &fwd, std::ostream &inv, const Options &options)
+    : m_fwd(fwd)
+    , m_inv(inv)
+    , m_options(options) {}
+
+  void Write(const ScfgRule &rule, size_t lineNum, bool printEndl=true);
+
+private:
+  // Disallow copying
+  ScfgRuleWriter(const ScfgRuleWriter &);
+  ScfgRuleWriter &operator=(const ScfgRuleWriter &);
+
+  void WriteStandardFormat(const ScfgRule &, std::ostream &, std::ostream &);
+  void WriteUnpairedFormat(const ScfgRule &, std::ostream &, std::ostream &);
+  void WriteSymbol(const Symbol &, std::ostream &);
+
+  std::ostream &m_fwd;
+  std::ostream &m_inv;
+  const Options &m_options;
+};
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining

mosesdecoder/phrase-extract/extract-ghkm/Span.cpp

@@ -0,0 +1,51 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#include "Span.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+bool SpansIntersect(const Span &a, const ContiguousSpan &b)
+{
+  for (Span::const_iterator p = a.begin(); p != a.end(); ++p) {
+    if (*p >= b.first && *p <= b.second) {
+      return true;
+    }
+  }
+  return false;
+}
+
+ContiguousSpan Closure(const Span &s)
+{
+  ContiguousSpan result(-1,-1);
+  if (!s.empty()) {
+    result.first = *(s.begin());
+    result.second = *(s.rbegin());
+  }
+  return result;
+}
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining

mosesdecoder/phrase-extract/extract-ghkm/Span.h

@@ -0,0 +1,45 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#pragma once
+#ifndef EXTRACT_GHKM_SPAN_H_
+#define EXTRACT_GHKM_SPAN_H_
+
+#include <map>
+#include <set>
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+typedef std::set<int> Span;
+typedef std::pair<int, int> ContiguousSpan;
+
+bool SpansIntersect(const Span &, const ContiguousSpan &);
+
+ContiguousSpan Closure(const Span &);
+
+}  // namespace MosesTraining
+}  // namespace Syntax
+}  // namespace GHKM
+
+#endif

mosesdecoder/phrase-extract/extract-ghkm/StsgRule.cpp

@@ -0,0 +1,97 @@
+#include "StsgRule.h"
+
+#include <algorithm>
+
+#include "Node.h"
+#include "Subgraph.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+StsgRule::StsgRule(const Subgraph &fragment)
+  : m_targetSide(fragment, true)
+{
+  // Source side
+
+  const std::set<const Node *> &sinkNodes = fragment.GetLeaves();
+
+  // Collect the subset of sink nodes that excludes target nodes with
+  // empty spans.
+  std::vector<const Node *> productiveSinks;
+  productiveSinks.reserve(sinkNodes.size());
+  for (std::set<const Node *>::const_iterator p = sinkNodes.begin();
+       p != sinkNodes.end(); ++p) {
+    const Node *sink = *p;
+    if (!sink->GetSpan().empty()) {
+      productiveSinks.push_back(sink);
+    }
+  }
+
+  // Sort them into the order defined by their spans.
+  std::sort(productiveSinks.begin(), productiveSinks.end(), PartitionOrderComp);
+
+  // Build a map from target nodes to source-order indices, so that we
+  // can construct the Alignment object later.
+  std::map<const Node *, std::vector<int> > sinkToSourceIndices;
+  std::map<const Node *, int> nonTermSinkToSourceIndex;
+
+  m_sourceSide.reserve(productiveSinks.size());
+  int srcIndex = 0;
+  int nonTermCount = 0;
+  for (std::vector<const Node *>::const_iterator p = productiveSinks.begin();
+       p != productiveSinks.end(); ++p, ++srcIndex) {
+    const Node &sink = **p;
+    if (sink.GetType() == TREE) {
+      m_sourceSide.push_back(Symbol("X", NonTerminal));
+      sinkToSourceIndices[&sink].push_back(srcIndex);
+      nonTermSinkToSourceIndex[&sink] = nonTermCount++;
+    } else {
+      assert(sink.GetType() == SOURCE);
+      m_sourceSide.push_back(Symbol(sink.GetLabel(), Terminal));
+      // Add all aligned target words to the sinkToSourceIndices map
+      const std::vector<Node *> &parents(sink.GetParents());
+      for (std::vector<Node *>::const_iterator q = parents.begin();
+           q != parents.end(); ++q) {
+        if ((*q)->GetType() == TARGET) {
+          sinkToSourceIndices[*q].push_back(srcIndex);
+        }
+      }
+    }
+  }
+
+  // Alignment
+
+  std::vector<const Node *> targetLeaves;
+  m_targetSide.GetTargetLeaves(targetLeaves);
+
+  m_alignment.reserve(targetLeaves.size());
+  m_nonTermAlignment.resize(nonTermCount);
+
+  for (int i = 0, j = 0; i < targetLeaves.size(); ++i) {
+    const Node *leaf = targetLeaves[i];
+    assert(leaf->GetType() != SOURCE);
+    if (leaf->GetSpan().empty()) {
+      continue;
+    }
+    std::map<const Node *, std::vector<int> >::iterator p =
+      sinkToSourceIndices.find(leaf);
+    assert(p != sinkToSourceIndices.end());
+    std::vector<int> &sourceNodes = p->second;
+    for (std::vector<int>::iterator r = sourceNodes.begin();
+         r != sourceNodes.end(); ++r) {
+      int srcIndex = *r;
+      m_alignment.push_back(std::make_pair(srcIndex, i));
+    }
+    if (leaf->GetType() == TREE) {
+      m_nonTermAlignment[nonTermSinkToSourceIndex[leaf]] = j++;
+    }
+  }
+}
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining

mosesdecoder/phrase-extract/extract-ghkm/StsgRule.h

@@ -0,0 +1,47 @@
+#pragma once
+#ifndef EXTRACT_GHKM_STSG_RULE_H_
+#define EXTRACT_GHKM_STSG_RULE_H_
+
+#include <vector>
+
+#include "Rule.h"
+#include "Subgraph.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+class Node;
+
+class StsgRule : public Rule
+{
+public:
+  StsgRule(const Subgraph &fragment);
+
+  const std::vector<Symbol> &GetSourceSide() const {
+    return m_sourceSide;
+  }
+  const Subgraph &GetTargetSide() const {
+    return m_targetSide;
+  }
+  const std::vector<int> &GetNonTermAlignment() const {
+    return m_nonTermAlignment;
+  }
+  int Scope() const {
+    return Rule::Scope(m_sourceSide);
+  }
+
+private:
+  std::vector<Symbol> m_sourceSide;
+  Subgraph m_targetSide;
+  std::vector<int> m_nonTermAlignment;
+};
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining
+
+#endif

mosesdecoder/phrase-extract/extract-ghkm/StsgRuleWriter.cpp

@@ -0,0 +1,98 @@
+#include "StsgRuleWriter.h"
+
+#include <cassert>
+#include <cmath>
+#include <ostream>
+#include <map>
+#include <sstream>
+#include <vector>
+
+#include "Alignment.h"
+#include "Options.h"
+#include "StsgRule.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+void StsgRuleWriter::Write(const StsgRule &rule)
+{
+  std::ostringstream sourceSS;
+  std::ostringstream targetSS;
+
+  // Write the source side of the rule to sourceSS.
+  const std::vector<Symbol> &sourceSide = rule.GetSourceSide();
+  for (std::size_t i = 0; i < sourceSide.size(); ++i) {
+    const Symbol &symbol = sourceSide[i];
+    if (i > 0) {
+      sourceSS << " ";
+    }
+    if (symbol.GetType() == NonTerminal) {
+      sourceSS << "[X]";
+    } else {
+      sourceSS << symbol.GetValue();
+    }
+  }
+
+  // Write the target side of the rule to targetSS.
+  rule.GetTargetSide().PrintTree(targetSS);
+
+  // Write the rule to the forward and inverse extract files.
+  if (m_options.t2s) {
+    // If model is tree-to-string then flip the source and target.
+    m_fwd << targetSS.str() << " ||| " << sourceSS.str() << " |||";
+    m_inv << sourceSS.str() << " ||| " << targetSS.str() << " |||";
+  } else {
+    m_fwd << sourceSS.str() << " ||| " << targetSS.str() << " |||";
+    m_inv << targetSS.str() << " ||| " << sourceSS.str() << " |||";
+  }
+
+  // Write the non-terminal alignments.
+  const std::vector<int> &nonTermAlignment = rule.GetNonTermAlignment();
+  for (int srcIndex = 0; srcIndex <  nonTermAlignment.size(); ++srcIndex) {
+    int tgtIndex = nonTermAlignment[srcIndex];
+    if (m_options.t2s) {
+      // If model is tree-to-string then flip the source and target.
+      m_fwd << " " << tgtIndex << "-" << srcIndex;
+      m_inv << " " << srcIndex << "-" << tgtIndex;
+    } else {
+      m_fwd << " " << srcIndex << "-" << tgtIndex;
+      m_inv << " " << tgtIndex << "-" << srcIndex;
+    }
+  }
+  m_fwd << " |||";
+  m_inv << " |||";
+
+  // Write the symbol alignments.
+  const Alignment &alignment = rule.GetAlignment();
+  for (Alignment::const_iterator p = alignment.begin();
+       p != alignment.end(); ++p) {
+    if (m_options.t2s) {
+      // If model is tree-to-string then flip the source and target.
+      m_fwd << " " << p->second << "-" << p->first;
+      m_inv << " " << p->first << "-" << p->second;
+    } else {
+      m_fwd << " " << p->first << "-" << p->second;
+      m_inv << " " << p->second << "-" << p->first;
+    }
+  }
+
+  // Write a count of 1.
+  m_fwd << " ||| 1";
+  m_inv << " ||| 1";
+
+  // Write the PCFG score (if requested).
+  if (m_options.pcfg) {
+    m_fwd << " ||| " << std::exp(rule.GetTargetSide().GetPcfgScore());
+  }
+
+  m_fwd << std::endl;
+  m_inv << std::endl;
+}
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining

mosesdecoder/phrase-extract/extract-ghkm/StsgRuleWriter.h

@@ -0,0 +1,44 @@
+#pragma once
+#ifndef EXTRACT_GHKM_STSG_RULE_WRITER_H_
+#define EXTRACT_GHKM_STSG_RULE_WRITER_H_
+
+#include <ostream>
+
+#include "Subgraph.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+struct Options;
+class StsgRule;
+class Symbol;
+
+class StsgRuleWriter
+{
+public:
+  StsgRuleWriter(std::ostream &fwd, std::ostream &inv, const Options &options)
+    : m_fwd(fwd)
+    , m_inv(inv)
+    , m_options(options) {}
+
+  void Write(const StsgRule &rule);
+
+private:
+  // Disallow copying
+  StsgRuleWriter(const StsgRuleWriter &);
+  StsgRuleWriter &operator=(const StsgRuleWriter &);
+
+  std::ostream &m_fwd;
+  std::ostream &m_inv;
+  const Options &m_options;
+};
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining
+
+#endif

mosesdecoder/phrase-extract/extract-ghkm/Subgraph.cpp

@@ -0,0 +1,206 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#include <iostream>
+
+#include "Node.h"
+#include "Subgraph.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+void Subgraph::GetTargetLeaves(std::vector<const Node *> &result) const
+{
+  result.clear();
+  GetTargetLeaves(m_root, result);
+}
+
+void Subgraph::GetTargetLeaves(const Node *root,
+                               std::vector<const Node *> &result) const
+{
+  if (root->GetType() == TARGET || m_leaves.find(root) != m_leaves.end()) {
+    result.push_back(root);
+  } else {
+    const std::vector<Node*> &children = root->GetChildren();
+    for (std::vector<Node *>::const_iterator p(children.begin());
+         p != children.end(); ++p) {
+      GetTargetLeaves(*p, result);
+    }
+  }
+}
+
+int Subgraph::CountNodes(const Node *n) const
+{
+  if (n->GetType() != TREE) {
+    return 0;
+  }
+  if (IsTrivial()) {
+    return 1;
+  }
+  int count = 1;
+  const std::vector<Node*> &children = n->GetChildren();
+  for (std::vector<Node *>::const_iterator p = children.begin();
+       p != children.end(); ++p) {
+    const Node *child = *p;
+    if (m_leaves.find(child) == m_leaves.end()) {
+      count += CountNodes(child);
+    } else if (child->GetType() == TREE) {
+      ++count;
+    }
+  }
+  return count;
+}
+
+int Subgraph::CalcSize(const Node *n) const
+{
+  if (n->GetType() != TREE || n->IsPreterminal()) {
+    return 0;
+  }
+  if (IsTrivial()) {
+    return 1;
+  }
+  int count = 1;
+  const std::vector<Node*> &children = n->GetChildren();
+  for (std::vector<Node *>::const_iterator p = children.begin();
+       p != children.end(); ++p) {
+    if (m_leaves.find(*p) == m_leaves.end()) {
+      count += CalcSize(*p);
+    }
+  }
+  return count;
+}
+
+int Subgraph::CalcDepth(const Node *n) const
+{
+  if (n->GetType() != TREE || n->IsPreterminal() || m_leaves.empty()) {
+    return 0;
+  }
+  int maxChildDepth = 0;
+  const std::vector<Node*> &children = n->GetChildren();
+  for (std::vector<Node *>::const_iterator p = children.begin();
+       p != children.end(); ++p) {
+    if (m_leaves.find(*p) == m_leaves.end()) {
+      maxChildDepth = std::max(maxChildDepth, CalcDepth(*p));
+    }
+  }
+  return maxChildDepth + 1;
+}
+
+float Subgraph::CalcPcfgScore() const
+{
+  if (m_root->GetType() != TREE || m_leaves.empty()) {
+    return 0.0f;
+  }
+  float score = m_root->GetPcfgScore();
+  for (std::set<const Node *>::const_iterator p = m_leaves.begin();
+       p != m_leaves.end(); ++p) {
+    const Node *leaf = *p;
+    if (leaf->GetType() == TREE) {
+      score -= leaf->GetPcfgScore();
+    }
+  }
+  return score;
+}
+
+void Subgraph::PrintTree(std::ostream &out) const
+{
+  RecursivelyPrintTree(m_root,out);
+}
+
+void Subgraph::RecursivelyPrintTree(const Node *n, std::ostream &out) const
+{
+  NodeType nodeType = n->GetType();
+  if (nodeType == TREE) {
+    out << "[" << n->GetLabel();
+    if (m_leaves.find(n) == m_leaves.end()) {
+      const std::vector<Node *> &children = n->GetChildren();
+      for (std::vector<Node *>::const_iterator p(children.begin());
+           p != children.end(); ++p) {
+        Node *child = *p;
+        if (child->GetType() == SOURCE) {
+          // This is possible due to the heuristic for attaching unaligned
+          // source words.
+          continue;
+        }
+        out << " ";
+        RecursivelyPrintTree(child,out);
+      }
+    }
+    out << "]";
+  } else if (nodeType == TARGET) {
+    out << n->GetLabel();
+  }
+}
+
+void Subgraph::PrintPartsOfSpeech(std::ostream &out) const
+{
+  RecursivelyPrintPartsOfSpeech(m_root,out);
+}
+
+void Subgraph::RecursivelyPrintPartsOfSpeech(const Node *n, std::ostream &out) const
+{
+  NodeType nodeType = n->GetType();
+  if (nodeType == TREE) {
+    if (m_leaves.find(n) == m_leaves.end()) {
+      const std::vector<Node *> &children = n->GetChildren();
+      for (std::vector<Node *>::const_iterator p(children.begin());
+           p != children.end(); ++p) {
+        Node *child = *p;
+        if (child->GetType() == TARGET) {
+          out << " " << n->GetLabel();
+        } else {
+          RecursivelyPrintPartsOfSpeech(child,out);
+        }
+      }
+    }
+  }
+}
+
+void Subgraph::GetPartsOfSpeech(std::vector<std::string> &out) const
+{
+  out.clear();
+  RecursivelyGetPartsOfSpeech(m_root,out);
+}
+
+void Subgraph::RecursivelyGetPartsOfSpeech(const Node *n, std::vector<std::string> &out) const
+{
+  NodeType nodeType = n->GetType();
+  if (nodeType == TREE) {
+    if (m_leaves.find(n) == m_leaves.end()) {
+      const std::vector<Node *> &children = n->GetChildren();
+      for (std::vector<Node *>::const_iterator p(children.begin());
+           p != children.end(); ++p) {
+        Node *child = *p;
+        if (child->GetType() == TARGET) {
+          out.push_back(n->GetLabel());
+        } else {
+          RecursivelyGetPartsOfSpeech(child,out);
+        }
+      }
+    }
+  }
+}
+
+}  // namespace MosesTraining
+}  // namespace Syntax
+}  // namespace GHKM

mosesdecoder/phrase-extract/extract-ghkm/Subgraph.h

@@ -0,0 +1,143 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#pragma once
+
+#include <set>
+#include <vector>
+
+#include "Node.h"
+
+namespace MosesTraining
+{
+namespace Syntax
+{
+namespace GHKM
+{
+
+class Node;
+
+class Subgraph
+{
+public:
+  Subgraph(const Node *root)
+    : m_root(root)
+    , m_depth(0)
+    , m_size(root->GetType() == TREE ? 1 : 0)
+    , m_nodeCount(1)
+    , m_pcfgScore(0.0f) {}
+
+  Subgraph(const Node *root, const std::set<const Node *> &leaves)
+    : m_root(root)
+    , m_leaves(leaves)
+    , m_depth(-1)
+    , m_size(-1)
+    , m_nodeCount(-1)
+    , m_pcfgScore(0.0f) {
+    m_depth = CalcDepth(m_root);
+    m_size = CalcSize(m_root);
+    m_nodeCount = CountNodes(m_root);
+    m_pcfgScore = CalcPcfgScore();
+  }
+
+  Subgraph(const Subgraph &other, bool targetOnly=false)
+    : m_root(other.m_root)
+    , m_leaves(other.m_leaves)
+    , m_depth(other.m_depth)
+    , m_size(other.m_size)
+    , m_nodeCount(other.m_nodeCount)
+    , m_pcfgScore(other.m_pcfgScore) {
+    if (targetOnly && m_root->GetType() != SOURCE) {
+      // Replace any source-word sink nodes with their parents (except for
+      // the special case where the parent is a non-word tree node -- see
+      // below).
+      std::set<const Node *> targetLeaves;
+      for (std::set<const Node *>::const_iterator p = m_leaves.begin();
+           p != m_leaves.end(); ++p) {
+        const Node *leaf = *p;
+        if (leaf->GetType() != SOURCE) {
+          targetLeaves.insert(leaf);
+        } else {
+          const std::vector<Node*> &parents = leaf->GetParents();
+          for (std::vector<Node*>::const_iterator q = parents.begin();
+               q != parents.end(); ++q) {
+            const Node *parent = *q;
+            // Only add parents that are words, not tree nodes since those
+            // are never sink nodes.  (A source word can have a tree node as
+            // its parent due to the heuristic for handling unaligned source
+            // words).
+            if (parent->GetType() == TARGET) {
+              targetLeaves.insert(*q);
+            }
+          }
+        }
+      }
+      m_leaves.swap(targetLeaves);
+    }
+  }
+
+  const Node *GetRoot() const {
+    return m_root;
+  }
+  const std::set<const Node *> &GetLeaves() const {
+    return m_leaves;
+  }
+  int GetDepth() const {
+    return m_depth;
+  }
+  int GetSize() const {
+    return m_size;
+  }
+  int GetNodeCount() const {
+    return m_nodeCount;
+  }
+  float GetPcfgScore() const {
+    return m_pcfgScore;
+  }
+
+  bool IsTrivial() const {
+    return m_leaves.empty();
+  }
+
+  void GetTargetLeaves(std::vector<const Node *> &) const;
+  void PrintTree(std::ostream &out) const;
+  void PrintPartsOfSpeech(std::ostream &out) const;
+  void GetPartsOfSpeech(std::vector<std::string> &out) const;
+
+private:
+  void GetTargetLeaves(const Node *, std::vector<const Node *> &) const;
+  int CalcDepth(const Node *) const;
+  int CalcSize(const Node *) const;
+  float CalcPcfgScore() const;
+  int CountNodes(const Node *) const;
+  void RecursivelyPrintTree(const Node *n, std::ostream &out) const;
+  void RecursivelyPrintPartsOfSpeech(const Node *n, std::ostream &out) const;
+  void RecursivelyGetPartsOfSpeech(const Node *n, std::vector<std::string> &out) const;
+
+  const Node *m_root;
+  std::set<const Node *> m_leaves;
+  int m_depth;
+  int m_size;
+  int m_nodeCount;
+  float m_pcfgScore;
+};
+
+}  // namespace GHKM
+}  // namespace Syntax
+}  // namespace MosesTraining

mosesdecoder/phrase-extract/extract-mixed-syntax/AlignedSentence.cpp

@@ -0,0 +1,195 @@
+/*
+ * AlignedSentence.cpp
+ *
+ *  Created on: 18 Feb 2014
+ *      Author: s0565741
+ */
+
+#include <sstream>
+#include "moses/Util.h"
+#include "AlignedSentence.h"
+#include "Parameter.h"
+
+using namespace std;
+
+
+/////////////////////////////////////////////////////////////////////////////////
+AlignedSentence::AlignedSentence(int lineNum,
+                                 const std::string &source,
+                                 const std::string &target,
+                                 const std::string &alignment)
+  :m_lineNum(lineNum)
+{
+  PopulateWordVec(m_source, source);
+  PopulateWordVec(m_target, target);
+  PopulateAlignment(alignment);
+}
+
+AlignedSentence::~AlignedSentence()
+{
+  Moses::RemoveAllInColl(m_source);
+  Moses::RemoveAllInColl(m_target);
+}
+
+void AlignedSentence::PopulateWordVec(Phrase &vec, const std::string &line)
+{
+  std::vector<string> toks;
+  Moses::Tokenize(toks, line);
+
+  vec.resize(toks.size());
+  for (size_t i = 0; i < vec.size(); ++i) {
+    const string &tok = toks[i];
+    Word *word = new Word(i, tok);
+    vec[i] = word;
+  }
+}
+
+void AlignedSentence::PopulateAlignment(const std::string &line)
+{
+  vector<string> alignStr;
+  Moses::Tokenize(alignStr, line);
+
+  for (size_t i = 0; i < alignStr.size(); ++i) {
+    vector<int> alignPair;
+    Moses::Tokenize(alignPair, alignStr[i], "-");
+    assert(alignPair.size() == 2);
+
+    int sourcePos = alignPair[0];
+    int targetPos = alignPair[1];
+
+    if (sourcePos >= m_source.size()) {
+      cerr << "ERROR1:AlignedSentence=" << Debug() << endl;
+      cerr << "m_source=" << m_source.size() << endl;
+      abort();
+    }
+    assert(sourcePos < m_source.size());
+    assert(targetPos < m_target.size());
+    Word *sourceWord = m_source[sourcePos];
+    Word *targetWord = m_target[targetPos];
+
+    sourceWord->AddAlignment(targetWord);
+    targetWord->AddAlignment(sourceWord);
+  }
+}
+
+std::string AlignedSentence::Debug() const
+{
+  stringstream out;
+  out << "m_lineNum:";
+  out << m_lineNum;
+  out << endl;
+
+  out << "m_source:";
+  out << m_source.Debug();
+  out << endl;
+
+  out << "m_target:";
+  out << m_target.Debug();
+  out << endl;
+
+  out << "consistent phrases:" << endl;
+  out << m_consistentPhrases.Debug();
+  out << endl;
+
+  return out.str();
+}
+
+std::vector<int> AlignedSentence::GetSourceAlignmentCount() const
+{
+  vector<int> ret(m_source.size());
+
+  for (size_t i = 0; i < m_source.size(); ++i) {
+    const Word &word = *m_source[i];
+    ret[i] = word.GetAlignmentIndex().size();
+  }
+  return ret;
+}
+
+void AlignedSentence::Create(const Parameter &params)
+{
+  CreateConsistentPhrases(params);
+  m_consistentPhrases.AddHieroNonTerms(params);
+}
+
+void AlignedSentence::CreateConsistentPhrases(const Parameter &params)
+{
+  int countT = m_target.size();
+  int countS = m_source.size();
+
+  m_consistentPhrases.Initialize(countS);
+
+  // check alignments for target phrase startT...endT
+  for(int lengthT=1;
+      lengthT <= params.maxSpan && lengthT <= countT;
+      lengthT++) {
+    for(int startT=0; startT < countT-(lengthT-1); startT++) {
+
+      // that's nice to have
+      int endT = startT + lengthT - 1;
+
+      // find find aligned source words
+      // first: find minimum and maximum source word
+      int minS = 9999;
+      int maxS = -1;
+      vector< int > usedS = GetSourceAlignmentCount();
+      for(int ti=startT; ti<=endT; ti++) {
+        const Word &word = *m_target[ti];
+        const std::set<int> &alignment = word.GetAlignmentIndex();
+
+        std::set<int>::const_iterator iterAlign;
+        for(iterAlign = alignment.begin(); iterAlign != alignment.end(); ++iterAlign) {
+          int si = *iterAlign;
+          if (si<minS) {
+            minS = si;
+          }
+          if (si>maxS) {
+            maxS = si;
+          }
+          usedS[ si ]--;
+        }
+      }
+
+      // unaligned phrases are not allowed
+      if( maxS == -1 )
+        continue;
+
+      // source phrase has to be within limits
+      size_t width = maxS - minS + 1;
+
+      if( width < params.minSpan )
+        continue;
+
+      if( width > params.maxSpan )
+        continue;
+
+      // check if source words are aligned to out of bound target words
+      bool out_of_bounds = false;
+      for(int si=minS; si<=maxS && !out_of_bounds; si++)
+        if (usedS[si]>0) {
+          out_of_bounds = true;
+        }
+
+      // if out of bound, you gotta go
+      if (out_of_bounds)
+        continue;
+
+      // done with all the checks, lets go over all consistent phrase pairs
+      // start point of source phrase may retreat over unaligned
+      for(int startS=minS;
+          (startS>=0 &&
+           startS>maxS - params.maxSpan && // within length limit
+           (startS==minS || m_source[startS]->GetAlignment().size()==0)); // unaligned
+          startS--) {
+        // end point of source phrase may advance over unaligned
+        for(int endS=maxS;
+            (endS<countS && endS<startS + params.maxSpan && // within length limit
+             (endS==maxS || m_source[endS]->GetAlignment().size()==0)); // unaligned
+            endS++) {
+
+          // take note that this is a valid phrase alignment
+          m_consistentPhrases.Add(startS, endS, startT, endT, params);
+        }
+      }
+    }
+  }
+}

mosesdecoder/phrase-extract/extract-mixed-syntax/AlignedSentenceSyntax.h

@@ -0,0 +1,46 @@
+/*
+ * AlignedSentenceSyntax.h
+ *
+ *  Created on: 26 Feb 2014
+ *      Author: hieu
+ */
+
+#pragma once
+
+#include "AlignedSentence.h"
+#include "SyntaxTree.h"
+#include "pugixml.hpp"
+
+class AlignedSentenceSyntax : public AlignedSentence
+{
+public:
+  AlignedSentenceSyntax(int lineNum,
+                        const std::string &source,
+                        const std::string &target,
+                        const std::string &alignment);
+  virtual ~AlignedSentenceSyntax();
+
+  void Create(const Parameter &params);
+
+  //virtual std::string Debug() const;
+protected:
+  std::string m_sourceStr, m_targetStr, m_alignmentStr;
+  SyntaxTree m_sourceTree, m_targetTree;
+
+  void XMLParse(Phrase &output,
+                SyntaxTree &tree,
+                const std::string input,
+                const Parameter &params);
+  void XMLParse(Phrase &output,
+                SyntaxTree &tree,
+                const pugi::xml_node &parentNode,
+                const Parameter &params);
+  void CreateNonTerms();
+  void CreateNonTerms(ConsistentPhrase &cp,
+                      const SyntaxTree::Labels &sourceLabels,
+                      const SyntaxTree::Labels &targetLabels);
+  void Populate(bool isSyntax, int mixedSyntaxType, const Parameter &params,
+                std::string line, Phrase &phrase, SyntaxTree &tree);
+
+};
+

mosesdecoder/phrase-extract/extract-mixed-syntax/ConsistentPhrase.cpp

@@ -0,0 +1,67 @@
+/*
+ * ConsistentPhrase.cpp
+ *
+ *  Created on: 20 Feb 2014
+ *      Author: hieu
+ */
+#include <sstream>
+#include "ConsistentPhrase.h"
+#include "Word.h"
+#include "NonTerm.h"
+#include "Parameter.h"
+
+using namespace std;
+
+ConsistentPhrase::ConsistentPhrase(
+  int sourceStart, int sourceEnd,
+  int targetStart, int targetEnd,
+  const Parameter &params)
+  :corners(4)
+  ,m_hieroNonTerm(*this, params.hieroNonTerm, params.hieroNonTerm)
+{
+  corners[0] = sourceStart;
+  corners[1] = sourceEnd;
+  corners[2] = targetStart;
+  corners[3] = targetEnd;
+}
+
+ConsistentPhrase::~ConsistentPhrase()
+{
+  // TODO Auto-generated destructor stub
+}
+
+bool ConsistentPhrase::operator<(const ConsistentPhrase &other) const
+{
+  return corners < other.corners;
+}
+
+void ConsistentPhrase::AddNonTerms(const std::string &source,
+                                   const std::string &target)
+{
+  m_nonTerms.push_back(NonTerm(*this, source, target));
+}
+
+bool ConsistentPhrase::TargetOverlap(const ConsistentPhrase &other) const
+{
+  if ( other.corners[3] < corners[2] || other.corners[2] > corners[3])
+    return false;
+
+  return true;
+}
+
+std::string ConsistentPhrase::Debug() const
+{
+  stringstream out;
+  out << "[" << corners[0] << "-" << corners[1]
+      << "][" << corners[2] << "-" << corners[3] << "]";
+
+  out << "NT:";
+  for (size_t i = 0; i < m_nonTerms.size(); ++i) {
+    const NonTerm &nonTerm = m_nonTerms[i];
+    out << nonTerm.GetLabel(Moses::Input) << ":" << nonTerm.GetLabel(Moses::Output);
+  }
+
+  return out.str();
+}
+
+

mosesdecoder/phrase-extract/extract-mixed-syntax/ConsistentPhrases.cpp

@@ -0,0 +1,104 @@
+/*
+ * ConsistentPhrases.cpp
+ *
+ *  Created on: 20 Feb 2014
+ *      Author: hieu
+ */
+#include <sstream>
+#include <cassert>
+#include "ConsistentPhrases.h"
+#include "NonTerm.h"
+#include "Parameter.h"
+#include "moses/Util.h"
+
+using namespace std;
+
+ConsistentPhrases::ConsistentPhrases()
+{
+}
+
+ConsistentPhrases::~ConsistentPhrases()
+{
+  for (size_t start = 0; start < m_coll.size(); ++start) {
+    std::vector<Coll> &allSourceStart = m_coll[start];
+
+    for (size_t size = 0; size < allSourceStart.size(); ++size) {
+      Coll &coll = allSourceStart[size];
+      Moses::RemoveAllInColl(coll);
+    }
+  }
+}
+
+void ConsistentPhrases::Initialize(size_t size)
+{
+  m_coll.resize(size);
+
+  for (size_t sourceStart = 0; sourceStart < size; ++sourceStart) {
+    std::vector<Coll> &allSourceStart = m_coll[sourceStart];
+    allSourceStart.resize(size - sourceStart);
+  }
+}
+
+void ConsistentPhrases::Add(int sourceStart, int sourceEnd,
+                            int targetStart, int targetEnd,
+                            const Parameter &params)
+{
+  Coll &coll = m_coll[sourceStart][sourceEnd - sourceStart];
+  ConsistentPhrase *cp = new ConsistentPhrase(sourceStart, sourceEnd,
+      targetStart, targetEnd,
+      params);
+
+  assert(coll.find(cp) == coll.end());
+  coll.insert(cp);
+}
+
+const ConsistentPhrases::Coll &ConsistentPhrases::GetColl(int sourceStart, int sourceEnd) const
+{
+  const std::vector<Coll> &allSourceStart = m_coll[sourceStart];
+  const Coll &ret = allSourceStart[sourceEnd - sourceStart];
+  return ret;
+}
+
+ConsistentPhrases::Coll &ConsistentPhrases::GetColl(int sourceStart, int sourceEnd)
+{
+  std::vector<Coll> &allSourceStart = m_coll[sourceStart];
+  Coll &ret = allSourceStart[sourceEnd - sourceStart];
+  return ret;
+}
+
+std::string ConsistentPhrases::Debug() const
+{
+  std::stringstream out;
+  for (size_t start = 0; start < m_coll.size(); ++start) {
+    const std::vector<Coll> &allSourceStart = m_coll[start];
+
+    for (size_t size = 0; size < allSourceStart.size(); ++size) {
+      const Coll &coll = allSourceStart[size];
+
+      Coll::const_iterator iter;
+      for (iter = coll.begin(); iter != coll.end(); ++iter) {
+        const ConsistentPhrase &consistentPhrase = **iter;
+        out << consistentPhrase.Debug() << endl;
+      }
+    }
+  }
+
+  return out.str();
+}
+
+void ConsistentPhrases::AddHieroNonTerms(const Parameter &params)
+{
+  // add [X] labels everywhere
+  for (size_t i = 0; i < m_coll.size(); ++i) {
+    vector<Coll> &inner = m_coll[i];
+    for (size_t j = 0; j < inner.size(); ++j) {
+      ConsistentPhrases::Coll &coll = inner[j];
+      ConsistentPhrases::Coll::iterator iter;
+      for (iter = coll.begin(); iter != coll.end(); ++iter) {
+        ConsistentPhrase &cp = **iter;
+        cp.AddNonTerms(params.hieroNonTerm, params.hieroNonTerm);
+      }
+    }
+  }
+}
+

mosesdecoder/phrase-extract/extract-mixed-syntax/Jamfile

@@ -0,0 +1,2 @@
+exe extract-mixed-syntax : Main.cpp AlignedSentence.cpp AlignedSentenceSyntax.cpp ConsistentPhrase.cpp ConsistentPhrases.cpp NonTerm.cpp Parameter.cpp Phrase.cpp pugixml.cpp Rule.cpp RulePhrase.cpp Rules.cpp RuleSymbol.cpp SyntaxTree.cpp Word.cpp ..//deps ../..//z ../..//boost_iostreams ../..//boost_program_options ../../moses//moses : <include>..  ;
+

mosesdecoder/phrase-extract/extract-mixed-syntax/Parameter.cpp

@@ -0,0 +1,74 @@
+/*
+ * Parameter.cpp
+ *
+ *  Created on: 17 Feb 2014
+ *      Author: hieu
+ */
+#include "Parameter.h"
+#include "moses/Util.h"
+#include "util/exception.hh"
+
+using namespace std;
+
+Parameter::Parameter()
+  :maxSpan(10)
+  ,minSpan(0)
+  ,maxNonTerm(2)
+  ,maxHieroNonTerm(999)
+  ,maxSymbolsTarget(999)
+  ,maxSymbolsSource(5)
+  ,minHoleSource(2)
+  ,minHoleSourceSyntax(1)
+  ,sentenceOffset(0)
+  ,nonTermConsecSource(false)
+  ,requireAlignedWord(true)
+  ,fractionalCounting(true)
+  ,gzOutput(false)
+
+  ,hieroNonTerm("[X]")
+  ,sourceSyntax(false)
+  ,targetSyntax(false)
+
+  ,mixedSyntaxType(0)
+  ,multiLabel(0)
+  ,nonTermConsecSourceMixed(true)
+  ,hieroSourceLHS(false)
+  ,maxSpanFreeNonTermSource(0)
+  ,nieceTerminal(true)
+  ,maxScope(UNDEFINED)
+  ,minScope(0)
+
+  ,spanLength(false)
+  ,ruleLength(false)
+  ,nonTermContext(false)
+  ,nonTermContextTarget(false)
+  ,nonTermContextFactor(0)
+
+  ,numSourceFactors(1)
+  ,numTargetFactors(1)
+
+  ,nonTermConsecSourceMixedSyntax(1)
+{}
+
+Parameter::~Parameter()
+{
+  // TODO Auto-generated destructor stub
+}
+
+void Parameter::SetScopeSpan(const std::string &str)
+{
+  scopeSpanStr = str;
+  vector<string> toks1;
+  Moses::Tokenize(toks1, str, ":");
+
+  for (size_t i = 0; i < toks1.size(); ++i) {
+    const string &tok1 = toks1[i];
+
+    vector<int> toks2;
+    Moses::Tokenize<int>(toks2, tok1, ",");
+    UTIL_THROW_IF2(toks2.size() != 2, "Format is min,max:min,max... String is " << tok1);
+
+    std::pair<int,int> values(toks2[0], toks2[1]);
+    scopeSpan.push_back(values);
+  }
+}

mosesdecoder/phrase-extract/extract-mixed-syntax/Parameter.h

@@ -0,0 +1,66 @@
+/*
+ * Parameter.h
+ *
+ *  Created on: 17 Feb 2014
+ *      Author: hieu
+ */
+#pragma once
+
+#include <string>
+#include <limits>
+#include <vector>
+
+#define UNDEFINED	std::numeric_limits<int>::max()
+
+class Parameter
+{
+public:
+  Parameter();
+  virtual ~Parameter();
+
+  int maxSpan;
+  int minSpan;
+  int maxNonTerm;
+  int maxHieroNonTerm;
+  int maxSymbolsTarget;
+  int maxSymbolsSource;
+  int minHoleSource;
+  int minHoleSourceSyntax;
+
+  long sentenceOffset;
+
+  bool nonTermConsecSource;
+  bool requireAlignedWord;
+  bool fractionalCounting;
+  bool gzOutput;
+
+  std::string hieroNonTerm;
+  std::string gluePath;
+
+  bool sourceSyntax, targetSyntax;
+
+  int mixedSyntaxType, multiLabel;
+  bool nonTermConsecSourceMixed;
+  bool hieroSourceLHS;
+  int maxSpanFreeNonTermSource;
+  bool nieceTerminal;
+  int maxScope, minScope;
+
+  // properties
+  bool spanLength;
+  bool ruleLength;
+  bool nonTermContext;
+  bool nonTermContextTarget;
+  int nonTermContextFactor;
+
+  int numSourceFactors, numTargetFactors;
+
+  int nonTermConsecSourceMixedSyntax;
+
+  std::string scopeSpanStr;
+  std::vector<std::pair<int,int> > scopeSpan;
+
+  void SetScopeSpan(const std::string &str);
+
+};
+

mosesdecoder/phrase-extract/extract-mixed-syntax/Phrase.cpp

@@ -0,0 +1,14 @@
+#include <sstream>
+#include "Phrase.h"
+
+std::string Phrase::Debug() const
+{
+  std::stringstream out;
+
+  for (size_t i = 0; i < size(); ++i) {
+    Word &word = *at(i);
+    out << word.Debug() << " ";
+  }
+
+  return out.str();
+}

mosesdecoder/phrase-extract/extract-mixed-syntax/RulePhrase.cpp

@@ -0,0 +1,50 @@
+/*
+ * RulePhrase.cpp
+ *
+ *  Created on: 26 Feb 2014
+ *      Author: hieu
+ */
+
+#include <sstream>
+#include "RulePhrase.h"
+#include "RuleSymbol.h"
+
+using namespace std;
+
+extern bool g_debug;
+
+int RulePhrase::Compare(const RulePhrase &other) const
+{
+  if (GetSize() != other.GetSize()) {
+    return GetSize() < other.GetSize() ? -1 : +1;
+  }
+
+  for (size_t i = 0; i < m_coll.size(); ++i) {
+    const RuleSymbol &symbol = *m_coll[i];
+    const RuleSymbol &otherSymbol = *other.m_coll[i];
+    int compare = symbol.Compare(otherSymbol);
+
+    if (compare) {
+      return compare;
+    }
+  }
+
+  return 0;
+}
+
+void RulePhrase::Output(std::ostream &out) const
+{
+  for (size_t i =  0; i < m_coll.size(); ++i) {
+    const RuleSymbol &symbol = *m_coll[i];
+    symbol.Output(out);
+    out << " ";
+  }
+}
+
+std::string RulePhrase::Debug() const
+{
+  std::stringstream out;
+  Output(out);
+  return out.str();
+}
+