List of Japanese NLP tools

I haven't tried out all of these so I don't have comments for everything, but hopefully this list will come in useful for someone.

Morphological analyzers/tokenizers

Itadaki: a Japanese processing module for OpenOffice. I've done a tiny bit of work and issue documentation on a fork here, and someone forked that to work with a Japanese/German dictionary here.

GoSen: Uses sen as a base, and is part of Itadaki; a pure Java version of ChaSen. See my previous post on where to download it from.

MeCab: This page also contains a comparison of MeCab, ChaSen, JUMAN, and Kakasi.

ChaSen

JUMAN

Cabocha: Uses support vector machines for morphological and dependency structure analysis.

Gomoku

Igo

Kuromoji: Donated to Apache and used in Solr. Looks nice.

Corpora

Hypermedia Corpus

TüBa-J/S: Japanese treebank from universityu of Tübingen. Not as heavily annotated as I'd hoped. You have to send them an agreement to download it, but it's free.

GSK: Not free, but very cheap.

LDC: Expensive unless your institution is a member

Other lexical resources

Kakasi: Gives readings for kanji compounds.

WordNet: Stil under development by NiCT. The sense numbers are cross-indexed with those in the English WordNet, so it could be useful for translation. Also, there are no verb frames like there are in English.

LCS Database: From Okayama University

Framenet: Unfortunately you can only do online browsing.

Chakoshi: Online collocation search engine.

Itadaki GoSen and IPADIC 2.7

Update3: I've forked the Itadaki project on GitHub to keep track of it better.
Update2: I made an executable JAR for GoSen that runs the ReadingProcessorDemo. It requires Java 6; just unzip the contents of this zip file to your computer and click on the jar file.
Update1: The IPADIC dictionary is no longer available from its original location. It has been replaced by the NAIST dictionary. I have edited the following post to reflect the needed changes.

Itadaki is a software suite for processing Japanese in OpenOffice. GoSen, part of the Itadaki project, is a pure Java morphological analysis tool for Japanese, and I have found it extremely useful in my research. Unfortunately, the page for this project went down recently, making the tools harder to find. Itadaki is still available through Google code here, but I can't find a separate installment of GoSen. The old GoSen website can still be accessed through the way-back-machine here. The other problem is that GoSen hasn't been updated since 2007, and in it's current release cannot handle the latest release of IPADIC. I'll describe how to fix it in this post.
Why does it matter that we can't use the latest version of IPADIC? Well, here's an example. I am using GoSen in my thesis work right now, and I put in a sentence which included a negative, past tense verb, such as 行かなかった. It analyzed it as な being used for negation, and かった being the past tense of the verb かう. That is indeed a problem! Using the newer IPADIC fixed it for me, though. To do that, download this modified version of GoSen. The explanation for the fix is here. Basically, a change in the new IPADIC versions to work better with MeCab adds a bunch of commas that break GoSen.


Edit: Once you've downloaded and unzipped GoSen, run ant in the top directory to build an executable JAR file. Note that if you want javadoc, you'll have to change build.xml so that the javadoc command has 'encoding="utf-8"'. Next, you must download the IPADIC dictionary from its legacy repository, here. Unpack the contents into testdata/dictionary. Change testdata/dictionary/build.xml so that the value of "ipadic.version" is "2.7.0" (the version that you downloaded). Now run ant in this directory to build the dictionary. [If you had errors, you may have forgotten to run ant in the top level directory first.]


Then, to run a demo and see what amazing things GoSen can do, copy the dictionary.xml file from the testdata/dictionary directory to the dictionary/dictionary directory, go back to the root directory of GoSen, and then run java -cp bin examples.ReadingProcessorDemo testData/dictionary/dictionary.xml. The GoSen site says to run using the testdata folder, but that means you'll have to download the dictionary twice, which is dumb. When you run the above command, you'll get this GUI:

Notice that it tokenizes the sentence, gives readings, and allows you to choose among alternatives analyses. It also gives information on part of speech and inflection.
To use GoSen in an Eclipse project, add gosen-1.0beta.jar to the project build path. You also need to have the dictionary directory somewhere, along with the dictionary.xml file. This code will get you started:

package edu.byu.xnlsoar.jp.lexacc;

import java.io.IOException;
import java.util.List;

import edu.byu.xnlsoar.utils.Constants;



import net.java.sen.SenFactory;
import net.java.sen.StringTagger;
import net.java.sen.dictionary.Morpheme;
import net.java.sen.dictionary.Token;

public class GoSenInterface {
 public List tokenize(String sentence){
  StringTagger tagger = SenFactory.getStringTagger(Constants.getProperty("GOSEN_DICT_CONFIG"));
  try {
   return tagger.analyze(sentence);
  } catch (IOException e) {
   e.printStackTrace();
   System.exit(-1);
  }
  return null;
 }
 public static void main(String[] args){
  String sentence = "やっぱり日本語情報処理って簡単に出来ちゃうんだもんな。";
  GoSenInterface dict = new GoSenInterface();
  System.out.println("tokenizing " + sentence);
  List tokens = dict.tokenize(sentence);
  System.out.println(tokens);
  Morpheme m;
  System.out.println("surface, lemma, POS, conjugation");
  for(Token t : tokens){
   System.out.print(t + ", ");
   m = t.getMorpheme();
   System.out.print(m.getBasicForm() + ", ");
   System.out.print(m.getPartOfSpeech() + ", ");
   System.out.println(m.getConjugationalType());
  }
 }
}

If you run that you will get:
tokenizing やっぱり日本語情報処理って簡単に出来ちゃうんだもんな。
[やっぱり, 日本語, 情報処理, って, 簡単, に, 出来, ちゃう, ん, だ, もん, な, 。]
surface, lemma, POS, conjugation
やっぱり, やっぱり, 副詞-一般, *
日本語, 日本語, 名詞-一般, *
情報処理, 情報処理, 名詞-一般, *
って, って, 助詞-格助詞-連語, *
簡単, 簡単, 名詞-形容動詞語幹, *
に, に, 助詞-副詞化, *
出来, 出来る, 動詞-自立, 一段
ちゃう, ちゃう, 動詞-非自立, 五段・ワ行促音便
ん, ん, 名詞-非自立-一般, *
だ, だ, 助動詞, 特殊・ダ
もん, もん, 名詞-非自立-一般, *
な, だ, 助動詞, 特殊・ダ
。, 。, 記号-句点, *


You have plenty of other options while processing, like grabbing alternate readings, etc. Notice that it got one wrong here: ちゃう is a contraction of てしまう, not a verb whose lemma is ちゃう. It doesn't seem to work on contractions because every token needs a surface form. So this might not work well on informal registers such as tweets or blogs unless some pre-preprocessing is done.
Feel free to leave any questions or comments.

String Allignment with Edit Operations

A common way to measure the distance between two strings is using Levenshtein distance. Levenshtein distance is the minimum number of deletions, insertions, and substitutions needed to transform one string into another. Finding the distance between two strings is useful in certain applications such as spell checking (a word processor will suggest dictionary words that are close to your misspelled word). See wikipedia for more details and an example of Levenshtein distance calculation.
Another related and also important operation is to find the minimum edit alignment; that is, once the minimum edit distance between the two strings is found, output the sequence of operations that can be used to change the one string into the other. For example, if we let C mean "correct", S mean "substitution", D mean "deletion" and I mean "insertion", then the edit alignment between the characters in "construction" and "distortions" would be ISSCCISSCCCCD. Here is an explanation of the alignment:

• I: Insert a "c" ->cdistortions
• S: Substitute "d" for "o" ->coistortions
• S: Substitute "i" for "n" -> constortions
• CC: Leave the "st" alone
• I: Insert "r" -> constrortion
• S: Substitute "u" for "o" -> contrurtion
• S: Substitute "c" for "r" -> constructions
• CCCC: Leave "tion" alone
• D: delete "s" -> construction

This sort of an allignment operation is commonly used in grading the output of machine translation and speech recognition systems, though the operations are done at the word level instead of the character level. When I needed a quick piece of code to do the alignment in my own application, though, there was none to be found on the internet! So I made my own and am posting it here. The logic for determining the proper string of edit operations is taken from a StackOverflow answer here. And now the code:

	/**
	 * @return List of Operations representing allignment between list1 and
	 * list2. The allignment represents operations to change list2 into list1.
	 */
	
	public static List levenshteinAllignment(Object[] list1, Object[] list2) {
		int[][] distanceMatrix = getDistanceMatrix(list1, list2);
		List ops= new ArrayList(
				list1.length > list2.length ? list1.length : list2.length);
		//think of distance chart as going from bottom left to top right;
		//current position coordinates; start at top right.
		int row = list1.length;
		int col = list2.length;
		//could have moved to current position from three others; store their scores here.
		int diag;
		int left;
		int below;
		int current;
		
		while (row != 0 || col != 0) {
			diag = getVal(row-1,col-1,distanceMatrix);
			left = getVal(row,col-1,distanceMatrix);
			below = getVal(row-1,col,distanceMatrix);
			current = distanceMatrix[row][col];

//			   if the value in the diagonal cell (going up+left) is smaller or equal to the
//			      values found in the other two cells
				
//			   AND 
//			      if this is same or 1 minus the value of the current cell 
			if(diag <= left && diag <= below && (diag == current || diag == current - 1)){
//				   then  "take the diagonal cell"
//		         if the value of the diagonal cell is one less than the current cell:
				if(diag == current - 1)
//		            Add a SUBSTITUTION operation (from the letters corresponding to
//		            the _current_ cell)
					ops.add(new Operation(Operation.Type.SUBSTITUTION,list1[row-1],list2[col-1]));
				else
//			        otherwise: do not add an operation this was a no-operation.
					ops.add(new Operation(Operation.Type.CORRECT,list1[row-1]));
				//move diagonally
				row--;
				col--;
			}
//
//		    elseif the value in the cell to the left is smaller or equal to the value of
//		  		the cell below current cell
//		   	AND 
//	       	if this value is same or 1 minus the value of the current cell 
			else if(left < below && (left == current || left == current - 1)){
//		        add an INSERTION of the cell to the left
				ops.add(new Operation(Operation.Type.INSERTION,list2[col-1]));
				//move left
				col--;
			}
//		   	else
			else{
//		       take the cell below, add
//		       Add a DELETION operation
				ops.add(new Operation(Operation.Type.DELETION,list1[row-1]));
				//move down
				row--;
			}
		}
		Collections.reverse(ops);
		return ops;
	}
	private static int getVal(int row, int col, int[][] distanceMatrix){
		if(row < 0 || row > distanceMatrix.length)
			return Integer.MAX_VALUE;
		if(col < 0 || col > distanceMatrix.length)
			return Integer.MAX_VALUE;
		else return distanceMatrix[row][col];
	}
	public static class Operation{
		private Type type;
		private Object object1;
		private Object object2;
		public enum Type{
			CORRECT,SUBSTITUTION,DELETION,INSERTION
		}
		public Operation(Type t, Object o1){
			type = t;
			object1 = o1;
			object2 = null;
		}
		public Operation(Type t, Object o1, Object o2){
			type = t;
			object1 = o1;
			object2 = o2;
		}
		@Override
		public String toString(){
			if(type == Type.SUBSTITUTION)
				return "Substitution: " + object1.toString() + " -> " + object2.toString();
			if(type == Type.CORRECT)
				return "Correct:      " + object1.toString();
			if(type == Type.DELETION)
				return "Deletion:     " + object1.toString();
			if(type == Type.INSERTION)
				return "Insertion:    " + object1.toString();
			return null;
		}
	}

	/**
	 * 
	 * @param array of objects to compare
	 * @param array of objects to compare
	 * @return Levenshtein distance between arrays.
	 * This method uses the equals(Object o) method to compare the
	 * objects in the two arrays, returning the Levenshtein distance between them.
	 */
	public static int levenshteinDistance(Object[] list1, Object[] list2) {

		int[][] distance = getDistanceMatrix(list1, list2);
		return distance[list1.length][list2.length];
	}

	/**
	 * 
	 * @param list1
	 * @param list2
	 * @return A completely filled distance matrix; movement from [i-1][j]
	 *         represents insertion, from [i][j-1] represents deletion, and from
	 *         [i-1][j-1] represents substitution or no operation.
	 */
	private static int[][] getDistanceMatrix(Object[] list1, Object[] list2) {
		int[][] distanceMatrix = new int[list1.length + 1][list2.length + 1];

		for (int i = 0; i <= list1.length; i++)
			distanceMatrix[i][0] = i;
		for (int j = 0; j <= list2.length; j++)
			distanceMatrix[0][j] = j;

		for (int i = 1; i <= list1.length; i++)
			for (int j = 1; j <= list2.length; j++)
				distanceMatrix[i][j] = minimum(distanceMatrix[i - 1][j] + 1,// insertion
						distanceMatrix[i][j - 1] + 1,// deletion
						distanceMatrix[i - 1][j - 1]// substitution or correct
								+ ((list1[i - 1].equals(list2[j - 1])) ? 0 : 1));
		return distanceMatrix;
	}

	/**
	 * Same as Math.min, but returns the minimum of three arguments instead of
	 * two.
	 */
	private static int minimum(int a, int b, int c) {
		return Math.min(Math.min(a, b), c);
	}

For word level alignment, you can call it on Strings using the split function like so:

for(Operation o : levenshteinAllignment(
			"What My house gleams with the light of the moon and your face"
					.split(" "),
			"Your house with the light of the the moon and my face"
					.split(" "))
		)
			System.out.println(o);

And the output would be:
Deletion: What
Substitution: My -> Your
Correct: house
Deletion: gleams
Correct: with
Correct: the
Correct: light
Correct: of
Correct: the
Insertion: the
Correct: moon
Correct: and
Substitution: your -> my
Correct: face
Feel free to use and edit this as you like. Many applications disregard the strings that are correct and only output the edit operations, and that should be an easy edit.