• Java API Updates in com.basistech.rlp.RLPResultAccess: Added access to NamedEntityData. Replaced getIntegerData() with getDetectedLanguage() and getDetectedScript, which return the appropriate com.basistech.util Enum type (LanguageCode and ISO639).

• Added API to guarantee that multiple instances of a given entity are returned with the same entity type

• Added facility for creating a blacklist of entities that are not to be returned by the Named Entity Extractor for the specified entity type.

• For the Base Linguistics Language Analyzer, added caching of morphological data about commonly used words in English and German to accelerate linguistic processing.

• Named Entities. Expanded named entity support to include Russian and three new entity types: TITLE, NUMBER, and DISTANCE. For this release, we have acquired, annotated, and performed statistical training with new data; improved and added more regular expressions; and introduced internal language-specific binary gazetteers (created by Basis Technology). To improve performance, the PERCENT entity type has been removed.

  Added Regular Expression support for naming and reusing regular expressions (including expression fragments).

  Added Named Entity Redactor support for joining adjacent named entities into a single named entity. By default, adjacent TITLE and PERSON entities are joined into a PERSON entity.

  Added the com.basistech.neredact.prefer_length property. When set to true (the default), this property instructs the Named Entity Redactor to resolve a conflict between overlapping candidate entities in favor of the longer candidate.

  Added the com.basistech.neredact.max_entity_tokens property. When a named entity returned by NamedEntityExtractor contains more than this number of tokens (the default is 8), Named Entity Redactor discards the entity.

  Added the FragmentBoundaryDetector, which uses whitespace to separate items in fragmentary text (such as lists and tables), so that the NamedEntityExtractor will not combine a series of fragments into a single entity. The Regular Expression processor also contains a new context property (com.basistech.regexp.respect_boundaries) that you can set to instruct the processor not to cross fragment boundaries when matching text.

  Added support for language-specific text gazetteers.

  For a problem and workaround applying regular expressions to Japanese text with "unit in square" characters, see Known Problems in 6.5.0.

• Expanded the .NET API to provide complete access to RLP functionality. The .NET API is modeled on the Java API.

• Enhanced the Base Linguistics (BL1) processor to run simultaneously in multiple threads..

• Replaced the Japanese Orthographic Normalizer (JON) with the ManyToOneNormalizer, which provides a multi-language utility for using language-specific normalization dictionaries to provide normalized tokens. We continue to distribute a Japanese normalization dictionary. Users can add their own normalization dictionaries for any of the languages we support.

  Important

  If you are using a context configuration file or string that contains <languageprocessor>JON</languageprocessor>, you must change this entry to <languageprocessor>ManyToOneNormalizer</languageprocessor>.

  For each token, JON returned a NORMALIZED_TOKEN, whereas ManyToOneNormalizer returns a MANY_TO_ONE_NORMALIZED_TOKEN.

• Moved sample code for integrating RLP with Lucene and Solr into a separate RLP For Lucene package (rlplucene-6.0.0-sdk-unix.tar.gz or rlplucene-6.0.0-sdk-win.zip).

• The sample Japanese normalization dictionary has been updated to include place name variants.

• Added support for instantiating multiple Environment objects in the same process. Each of these Environment objects is a wrapper for the same underlying Environment. Accordingly all Environment objects must be initialized with the same environment configuration (normally rlp-global.xml).

• Replaced Unix Make files and Windows Visual Studio Solution and Project files with scripts for building the C++ , C, and .NET sample applications. The Unix .sh scripts are designed to be run in a Bash Shell. The Windows(.bat scripts should be run in the Command Prompt. We continue to provide Ant scripts for building and running the Java sample applications, as well as Unix .shscripts and Windows .bat scripts for running all the sample applications.

• RLI can now detect UTF-16LE and UTF-16BE, even if the endianness does not match the endianness of the host operating system. If the endianness of the file matches the endiannnes of the host, RLI reports the encoding as UTF-16. If the endianness of file and host do not match, RLI reports the full encoding: UTF-16LE or UTF-16BE.

• Approximately 24,000 traditional Chinese words have been added to the Chinese dictionary.

• Replaced the Mac OS 10.4 (Darwin 8.9.1) platform for 32-bit Intel platform with the universal Mac OS 10.5 (Darwin 9) platform for 32-bit and 64-bit Intel.

• The sparc-solaris9-gcc345 platform has been renamed to sparc-solaris9-gcc34.

• To obtain the Rosette Demo, download and install rlpdemo-6.0.3.4.msi.

• RLP is no longer being built on the amd64-glibc23-gcc32 platform.

• This release removes dynamic dependencies on libgcc_s.so and libstdc++.so for the following Linux and Solaris platforms. The first three platforms are new; the last three have been changed to eliminate these dependencies. The compressed software installation packages are

  • rlp-6.0.2-sdk-amd64-glibc25-gcc41.tar.gz
  • rlp-6.0.2-sdk-amd64-solaris10-gcc41.tar.gz
  • rlp-6.0.2-sdk-sparc-solaris9-gcc41-64.tar.gz
  • rlp-6.0.2-sdk-amd64-glibc23-gcc34.tar.gz
  • rlp-6.0.2-sdk-ia32-glibc23-gcc34.tar.gz
  • rlp-6.0.2-sdk-amd64-solaris10-gcc41 (amd64-solaris10-gcc41 was formerly called amd64-solaris10-gcc412)

• Added named entity support for Korean, Farsi (Persian), and Urdu.

• Added support for new named entity types: RELIGION, NATIONALITY, GPE (a geo-political entity), and FACILITY (a man-made structure or architectural entity).

• 366 neologisms have been added to the Japanese dictionary.

• 171 neologisms have been added to the Chinese dictionary.

• A preliminary .NET API that provides limited coverage of RLP functionality.

• Enhancements to the Windows GUI Demo.

• Merging of the Korean Hangul and compound noun dictionaries into a single compiled Korean user dictionary that users can edit and recompile.

• The Rosette Language Identifier (RLI) returns DETECTED_SCRIPT, the ISO15924 code for the writing script of the text to be processed.

• The Rosette Language Identifier (RLI) is able to detect UTF-16 encoding.

• Moved the routine for scanning the RLP license and generating a list of supported features from the introductory RLP sample applications to separate C++, Java, and C sample applications.

• Removed deprecated Java interface and classes from com.basistech package: ISO15924Constants, BTLanguageCodes, and ISO15924. Use com.basistech.util.ISO15924 and com.basistech.util.LanguageCode.

• Expanded the scope of Tokenizer to tokenize all languages. In a context configuration, Tokenizer should be placed after processors that provide their own language-specific tokenization (BL1, CLA, JLA, and KLA ), and (along with Sentence Boundary Detector) before processors that use the tokenization it provides (ARBL, FABL, and URBL).

• Enabled use of the same RLP context with a Chinese Script converter (CSC) to convert simplified Chinese to traditional Chinese or vice versa.

• The Arabic Base Linguistics processor (ARBL) returns a new result type: lemma, and also returns multiple analyses of each token. Each analysis contains a stem, lemma, normalized token, root, and part-of-speech tag. See also Known Problems in 5.4.1.

• Sample code to integrate RLP with Lucene and Solr, for applications that index and search English and Japanese documents. This code provides a starting point for creating analyzers to index and search documents in any of the languages that RLP supports.

• Base linguistics support for Farsi (Persian).

• Base linguistics support for Urdu.

• A C API.

• Adoption of Java 1.5.

• Java clients are no longer required to set the java.library.path system property. Additionaly this directory is no longer required on the Windows path.

• Use of the BT_ROOT environment variable to set the root directory is no longer supported. New APIs to set the Basis root directory (the installation directory) have been introduced to replace deprecated APIs for setting the rlp root directory (a child of BT_ROOT). Utilities that took a BT_ROOT argument interpreted as the rlp root directory now take a BT_ROOT argument meaning the Basis root directory.

• Deprecated BT_RLP_Environment::InitializeFromBuffer in favor of BT_RLP_Environment::InitializeFromFile.

• In the com.basistech package, we have deprecated the com.basistech.ISO15924 interface and the ISOLanguageCodes and ISO15924 classes in favor of the new com.basistech.util.ISO15924 Enum.

• In the com.basistech.rlp package, we have deprecated the RLPNEConstants interface and RLPNEUtils class in favor of com.basistech.util.NEConstants

• Utilities shared by RLP and other Basis Technology products have been segregated from RLP. If using Java, you must include both btrlp.jar and btutil.jar on the classpath. As noted above, some Java APIs have been deprecated in favor of new APIs in the utilities package: com.basistech.util.

• Support for Chinese user dictionaries has been extended to include part-of-speech tags, decomposition patterns, and the compilation of binary dictionaries.

• The Japanese Language Analyzer (JLA) identifies the dictionary (the standard dictionary or a user dictionary) in which each token is found.

• Special syntax for entering Private Use Area (PUA) Unicode characters in Japanese user dictionaries.

• Addition of 1403 nouns, 2657 proper nouns, and 4 verbs to the Chinese dictionary used by the Chinese Language Analyzer (CLA).

• Addition of 68 nouns, 579 proper nouns, 5 adjectives, and 2 verbs to the Japanese dictionary used by the Japanese Language Analyzer (JLA).

• RCLU returns a new result type if com.basistech.rclu.mapoffsets is set to true. MAP_OFFSETS is an array of unsigned integers that map the characters in normalized text to their location in the input text prior to normalization.

• Optional builds with statically linked libraries for Windows.

• Three processors have been removed from RLP: Arabic Transliteration/Transcription (artrans), Arabic Name Vocalizer (arvoc), and Arabic Name Variations (arvars). The functionality provided by these processors is in a separate product, the Rosette Name Translator (RNT).

• Improved Named Entity accuracy for English.

• RCLU support for mapping token offsets in text that undergoes character transformations (normalization) to offsets in the original text prior to the normalization. Normalization may transform one Unicode character into two or three characters or vice versa. If you set the com.basistech.rclu.mapoffsets property to "true" or "yes" and apply com.basistech.rclu.FormCNormalization, com.basistech.rclu.FormDNormalization, com.basistech.rclu.FormKDNormalization, or com.basistech.rclu.FormKCNormalization, RCLU returns an array of unsigned integers in addition to the transformed text. Each element in the array is the original text character index corresponding to the character of transformed text indicated by the element's position in the array. If com.basistech.rclu.mapoffsets is set to "false" or "no" (the default), or a transformation other than the ones listed above is performed, this result is null.

• Support for reading the RLP license directly from the RLP environment configuration, rather than from a separate file. The RLP environment configuration, which may be instantiated from a file or buffer, may now contain a license element with the license.

• Support for Chinese user dictionaries has been extended to include part-of-speech tags, decomposition patterns, and the compilation of binary dictionaries.

• Support for European language (BL1) user dictionaries.

• Rosette Language Identifier (RLI) support for identifying a number of additional languages: Hindi, Tamil, Kannada, Telugi, Bengali, Gujarati, Malayalam, Ukranian (Cyrillic), Serbian (Cyrillic and Latin), Uzbek (Cyrllic and Latin), transliterated Farsi, and transliterated Urdu (Farsi and Urdu in Arabic script already supported).

• RLI support for the Traditional Chinese Big5-HKSCS encoding.

• RLI can identify upper-case English as such, enabling more accurate processing of upper-case English when RLI identifies the language.

• RLI has been retrained to improve accuracy detecting Russian and Bulgarian.

• Named Entity Extractor accuracy improvements identifying named entities in Arabic, Chinese, English, and Japanese.

• Addition of a C++ named entity iterator (BT_RLP_NE_Iterator) to streamline access to named entities.

• Addition of 1,632 Chinese names and 89 non-Chinese names to the Chinese dictionary used by the Chinese Language Analyzer (CLA).

• Addition of 1,758 nouns to the Japanese dictionary used by the Japanese Language Analyzer (JLA).

• All language processors (including Unicode Converter, which used to be built in to the RLP environment) are dynamically loaded as needed.

• C++ API changes for BT_RLP_Context. GetStringResult and GetUTF16StringResult now return pointers as constants. GetUTF16StringResult now takes a second argument, which returns the length of the non-null-terminated string. The new signatures are as follows:

  • BT_Char8 const *GetStringResult(BT_RLP_EntityType type)

  • BT_Char16 const *GetUTF16StringResult(BT_RLP_EntityType type, BT_UInt32& resultLength)

• Improved performance and accuracy for Arabic base linguistics and named entity extraction. For efficiency, folded all 12 pronoun POS tags (PRON_XX) into a single POS tag (PRONOUN), and all 7 demonstrative pronoun tags (DEM_PRON_XX) into a single POS tag (DEM_PRON).

• Added a context property (com.basistech.bl.query) for processing queries (search terms) more accurately and efficiently.

• Enhanced support for user-defined named entity types.

• Added API support for supplying MIME type and encoding when processing input (you can also include language processors in the RLP context that detect MIME type and encoding).

• Refactoring of language processors.

  • The Raw Input processor no longer exists. In its place, you simply start your context with RLI and RCLU.

  • The formal distinction between processor types (input, language, output) no longer exists.

  • RDIF functionality for extracting plain text from markup up input (such as HTML, XML, PDF, and Microsoft Office documents) has been folded into new language processors: mime_detector, iFilter, and HTML Stripper.

• Addition of the Chinese Script Converter (CSC). CSC is a language processor that converts text in Simplified Chinese script (used in the People's Republic of China) to text in Traditional Chinese script (used in Taiwan, Hong Kong, and Macau), or vice versa.

• RLI has added support for GB 18030.

• To enhance performance, RLP has switched from the PCRE regular expression engine to the Tcl regular expression engine.

• This release adds support for Mac OS 10.4. The compressed software and documentation installation packages for this platform are

  • rlp-5.0.4-sdk-ia32-darwin891-gcc40.tar.gz
  • rlp-5.0.4-doc-unix.tar.gz

  In addition to the SDK (RLP) and release number (5.0.4), the software package filename identifies the architecture (IA32), the Unix core environment (Darwin 8.9.1), and the C++ compiler (gcc 4.0). Using the Basis Technology naming convention, BT_BUILD is ia32-darwin891-gcc40; see "Getting Started: Supported Platforms" in the RLP Application Developer's Guide.

  Note: This platform uses .dylib instead of .so as the file extension for shared objects, and DYLD_LIBRARY_PATH instead of LD_LIBRARY_PATH as the environment variable for setting the load library path. See Known Problems in RLP 5.0.4.

• If you are using the Java API, Java 1.5 is required.

C++ BT_RLP_Environment functions:

• bool HasLicenses() const = 0;

• bool HasLicenseForLanguage(BT_LanguageID lid, BT_UInt32 functionality) const = 0;

• virtual bool HasLicenseForNamedFeature(const char *feature, BT_UInt32 functionality) const = 0;

Java RLPEnvironment methods:

• boolean hasLicenses() throws RLPException

• boolean hasLicenseForLanguage(int language_id, int functionality) throws RLPException

• virtual boolean hasLicenseForNamedFeature(String feature, int functionality) throws RLPException

• Rosette Language Identifier (RLI) enhancements: improved support for identifying language with short text streams, ability to detect Shift-JIS-2004 encoding, and support for identifying the following additional languages: Pashto, Urdu, Kurdish, Somali, transliterated (Romanized) Arabic, and transliterated (Romanized) Farsi. Added language codes to distinguish transliterated Arabic and transliterated Farsi from Arabic and Farsi in Arabic script.

• Streamlining of input processors. The Raw Input processor, in conjunction with the RLI and RCLU language processors, can now handle Unicode and non-Unicode input encodings. The Latin 1 input processor is no longer required to handle US-ASCII and ISO 8859-1, so it has been eliminated.

• Lexicon updates for the Japanese Language Analyzer (JLA) and the Chinese Language Analyzer (CLA):

  For Japanese, added 3 verbs, 2 given personal names, 3 foreign place names, 187 general proper nouns, 1731 common nouns, and 7 place names.

  For Chinese, added 15 adjectives, 47 adverbs, 395 phrases, 35 abbreviations, 125 noun phrases, 105 numeric entries, 196 proper nouns, 78 pronouns,34 temporal nouns, and 194 verbs.

• Speed improvements for the NameEntityExtractor, improved support for handling upper-case English input, and added support for traditional Chinese.

• Enhanced API support for accessing RLP result data. The C++ BT_RLP_ResultIterator class now includes an At function, which provide random accesss to individual results.

• The default logging level has been changed from "none" to "error". See "Capturing Log Output" in Chapter 3 of the RLP Application Developer's Guide.

• RLP Demo (GUI) fixes/enhancements.

• Removal of support for the .NET interface. In a subsequent release, we plan to reintroduce the .NET interface with additional API to provide complete access to RLP functionality.

Java API Changes

• RLPResultRandomAccess, RLPIntegerPair, and RLPIntegerTriple have been deprecated. Use RLPResultAccess to access result data. Depending on the result type, RLPResultAccess returns a List, Map, Integer, or String.

• It is no longer necessary to call RLPInitialize.init() to initialize the JNI. Accordingly, the RLPInitialize class has been removed.

• Introduction of the EnvironmentParameters class for defining the environment configuration, handling the logging callback object and log level, and setting the RLP root directory. The empty constructor RLPEnvironment() is deprecated. Use the new constructor after you have set up the EnvironmentParameters object: RLPEnvironment(EnvironmentParameters params).

• The Raw Input Processor has been updated to handle Unicode encodings properly.

• The performance of English, French, Italian, German, and Spanish named-entity processing has been improved by approximately 200%.

• The Gazetteer XML DTD and corresponding parser have been updated to support new features of the Global Name Matcher. Note: Apart from this readme file, the RLP documentation has not changed since Release 4.3.

• The token normalization processor, bt_lp_m1norm, is a many to one normalizer. It reads the normalized token vector or the token vector if the normailzed vector is not present. For each input token, it returns the normalized token, if any are found in the data source. Results are written to the normailzed token vector.

  RLP is preconfigured with a Japanese Orthographic Normalization data source for use with the token normailzation processor. See the JON processor listing in the rlp-global.xml file.

• The following processors are new in 4.3:

  • Raw Input Processor

    To enable the processing of non-Unicode input data, the Raw Input processor takes any raw text data and performs no encoding detection or conversion. This is only relevant for use with the RCLU language processor, which can take non-Unicode data and convert them to UTF-16 for succeeding language processors to use. In that case, the context definition must begin with RCLU and an encoding must be passed to the ProcessBuffer method. If you do not wish to provide an encoding, RLI must precede RCLU to detect the encoding.

  • Core Library for Unicode Language Processor (RCLU)

    The RCLU LP performs two functions. First, it will convert the input text to UTF-16 as required by other language processors. Second, it can perform certain character transformations on the input text. Encoding conversion will only occur if RCLU is used in conjunction with the Raw Input processor. Transformations are controlled using the context properties described above and are performed in the order they are listed in the context definition. If you do not provide an encoding, RLI must precede RCLU to detect the encoding. For a complete list of dependencies and context properties, please see the RLP Programming and API Guide.

  • Arabic Name Vocalizer Language Processor (ARVOC)

    The ARVOC language processor attempts to vocalize Arabic words under the assumption that they are likely to be personal names or parts of personal names. The selected text will be replaced with its fully-vocalized version under the prejudice that each word is likely to be part of a personal name. Words for which the proper vocalization could not be determined will be left intact (i.e., in their nonvocalized form). Words are vocalized on a one-by-one (context-free) basis. See the RLP Programming and API Guide for further details.

• Significant accuracy improvements have been attained for Arabic base linguistics and English named entity extraction.

• Support has been added for Czech, Greek, Hungarian, and Polish. See the Features by Language table above, and the BL1 section and POS Tags Appendix of the RLP Programming and API Guide for more information. In addition, BNP and NE are now available for Portuguese, and NE is now available for Dutch.

• A new sample program for Windows, rlpW32demo.exe, has been added. The previous demo, RLPDemo.exe, has been removed.

• bt_ip_htmlstripper, which does the following:

  1. Reads HTML document and identifies encoding via RLI.

  2. Converts to UTF-16, transforms all SGML entities to UTF-16 characters.

  3. Strips out HTML tags and post the stripped text as a BT_RLP_RAW_TEXT notice.

  4. Posts UTF-16 HTML document as BT_RLP_HTML_DOC notice on the blackboard.

  5. Posts HTML tags information as BT_RLP_HTML_TAGS notice on the blackboard.

• bt_lp_htmljoiner, which does the following:

  1. Annotates the BT_RLP_HTML_DOC with Named Entity data.

  2. If com.basistech.htmljoiner.do_artrans property is true, performs Arabic vocalization and transliteration of the Named Entity data.

  3. Posts the annotated HTML document as BT_RLP_HTML_DOC notice on the blackboard.

  Annotated Named Entity in the output HTML has the following format:

  <OBJECT title="NE type">Arabic NE Text<BDO dir="LTR">[transliterated 
  text]</BDO></OBJECT>

  In the BT_RLP_HTML_DOC notice, all meta tags with http-equiv="Content-Type" will be convert to:

  <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> 

• bt_op_dumphtml, which

  Reads the BT_RLP_HTML_DOC notice and dumps the output as UTF-8.

  The following is a sample context configuration (included in the SDK package as samples/etc/rlp-html-context.xml). It defines a processing chain that will consume an HTML document and output annotated HTML document via the DumpHTML output processor.

         <?xml version="1.0" encoding="utf-8" standalone="no"?>
          <!DOCTYPE contextconfig SYSTEM "http://www.basistech.com/dtds/2003/contextconfig.dtd">
          <contextconfig>   
            <properties>
              <property name="com.basistech.dumphtml.output_pathname" value="rlp-output.htm"/>
              <property name="com.basistech.htmljoiner.do_artrans" value="true"/>
            </properties>
            <inputprocessor>HTML Stripper</inputprocessor>
            <outputprocessor>DumpHTML</outputprocessor>
            <languageprocessors>
              <languageprocessor>RLI</languageprocessor>
              <languageprocessor>BL1</languageprocessor>
              <languageprocessor>Tokenizer</languageprocessor>
              <languageprocessor>JLA</languageprocessor>
              <languageprocessor>CLA</languageprocessor>
              <languageprocessor>KLA</languageprocessor>
              <languageprocessor>SentenceBoundaryDetector</languageprocessor>
              <languageprocessor>ARBL</languageprocessor>
              <languageprocessor>Stopwords</languageprocessor>
              <languageprocessor>BaseNounPhrase</languageprocessor> 
              <languageprocessor>NamedEntityExtractor</languageprocessor>
              <languageprocessor>Gazetteer</languageprocessor>
              <languageprocessor>RegExpLP</languageprocessor>
              <languageprocessor>NERedactLP</languageprocessor>
              <languageprocessor>HTML Joiner</languageprocessor>
            </languageprocessors>
          </contextconfig>

• Processing functionality for Dutch, Portuguese, and Russian has been added to the BL1 language processor. This new functionality includes changes to the bl1config.dtd file and new POS tags for each new language. See the RLP Programming and API Guide for full information.

• Named Entity Extraction is now available for French, Italian, and Spanish.

• Improvements were made for German decompounding and disambiguation.

• Beta release of improved European language performance. See the description of the BL1 processor in the RLP Programming and API Guide. A side effect of the introduction of this processor is that base noun phrase functionality and named entity extraction for French, Italian, and Spanish are temporarily unavailable. They will be available in the next release of RLP.

• Improved AR-BL performance (now approximately 30 times faster).

• Improved logging functionality:

  • BT_RLP_Environment::SetLogLevel and BT_RLP_Environment::SetLogCallbackFunction can now be called multiple times.

  • The logging callback now takes a "channel" parameter, allowing the function to examine the channel number and decide how to handle the message based on its severity. The new prototype is:

    typedef void (*BT_Log_callback_function)(void * callback_data, int channel, char const *string_to_log);

• Improved gazetteer and regular expression (regex) files for demonstrating custom named entity extraction: The default configuration files shipped with RLP have been updated to facilitate experimenting with these features. The instructions below assume you are using the RLPDemo program that is part of the Windows distribution. The instructions concerning edits to the configuration files also apply if you are using the samples\scripts\go.sh script that comes with UNIX distributions or the samples\scripts\go.bat script that comes with Windows distributions.

  How to demo the gazetteer

  1. Edit the file rlp\samples\etc\rlpdemo-gazetteer.txt; insert any demo words or phrases at the end of the file

  2. Open a text file with those phrases in the RLPDemo GUI, or type them in, and press the "Named Entities" button.

     You should see your entries highlighted in grey as "GazDemo" type entities. The standard demo text that appears when you open RLPDemo has the words "identification", "morphological", and "extraction". These will be highlighted in grey.

  How to demo the regex entity locator:

  The standard demo has three types of regex entities defined to show up as "OTHER" in purple highlight.

  1. Type in one of these three types:

     • A US Social Security number (xxx-xx-xxxx)

     • A credit card number (xxxx-xx-xxxx)

     • The emoticon (smiley) ":)"

  2. Push "Named Entities."

     You will see valid SS#'s, credit card numbers and the smiley highlighted in purple.

  3. You can edit the file rlp\etc\regex-config.xml to add your own regex entities. Look at the demo entity types near the top of the file.

  Notes

  This version will classify all regex entities as "NONE" which is shown as "OTHER" in the demo.

  Regex entity definition will be simplified in a future release to make it easier to define entities.

• JLA now includes the capability to create and use one or more user dictionaries for words specific to an industry or application. See the JLA Processor section of the RLP Programming and API Guide for full details.

• All European language processors (English, French, German, Italian, and Spanish) have been replaced by a beta version of a single new language processor called Base Linguistics (BL1). This new processor renders the old POSTagger processor obsolete and has changed all POS tags for these languages. In addition, tokenizing functionality is built in, and thus the Tokenizer processor will produce no results when run with BL1. For complete specifications, see the RLP Programming and API Guide.

• Support for French, Italian, and Spanish named entities.

• Python is no longer used by RLP.

• The distributed directory structure has been changed. All binaries are now in the top-level bin directory and all libraries are now in the top-level lib directory. Also, the rex directory has been renamed to rlp.

• "REX" in the names of classes and structures has been replaced with "RLP". The include file bt_rex.h provides backward compatibility for the old names.

• The Japanese lexicon has been improved. A comparison with the RLP 2.3 lexicon is given below.

• The Chinese lexicon has been improved. A comparison with the RLP 2.3 lexicon is given below.

  This release includes Basis Technology Chinese lexicons with data from new providers. This may affect the Chinese Language Analyzer tokenization results in your application. Basis internal tests show that overall, segmentation accuracy has improved compared to the lexicons used in the 3.x release of CLA. These tests also show that over a very large corpus, the segmentations do not differ at all in 88% of cases. Of the cases where there are differences, CLA 4.0 was correct 53% of the time and CLA 3.x was correct 34% of the time.

  The following is a summary of the major differences.

  	CLA 4.0.0	CLA 3.x.x
  verb phrases	Many treated as one token Examples: 坐在, 去了, Ŋ到, 抄起	Many treated as multiple tokens Examples: 坐 在, 去 了, Ŋ 到, 抄 起
  adverb + verb	Treated as one token Example: 也有	Treated as multiple tokens Example: 也   有
  popular noun phrases	Treated as one token Examples: 本报讯, 党支部书记, 资产阶级自	Treated as multiple tokens Examples: 本报   讯, 党支部   书记, 资产阶级   自由化
  abbreviations	Treated as one token Examples: 女足, 世锦赛	Treated as multiple tokens Examples: 女   足, 世   锦  赛
  not	Combinations treated as one token Examples: 不是, 不要,   不会	Combinations treated as two tokens Examples: 不   是, 不   要, 不   会
  book	Combinations treated as two tokens Examples: 华文   书, 英文   书	Combinations treated as one token Examples: 华文书, 英文书
  one, some	Combinations treated as two tokens Examples: 一   个	Combinations treated as one token Example: 一个
  的 (particle)	Not appended to some adjectives Example: 红红	Appended to some adjectives Example: 红红的
  biggest	Treated as one token Example: 最大	Treated as two tokens Example: 最   大

  In 4.0.0, idioms and proper nouns are considerably more up-to-date. The most significant decrease in version 4.0 accuracy is in the recognition of proper nouns, particularly Chinese personal names. Basis Technology is committed to making product improvements to deliver better accuracy, and a project is underway now to improve proper noun coverage in an upcoming release.

• Japanese Lexicon Improvements:

  • New Dictionary: The Japanese dictionary has been revised extensively. New words have been added. Internal parts-of-speech have been updated.

  • Katakana segmentation: Unknown Katakana words will be segmented as one token.

  • Compound Verbs: Where possible, compound verbs will be segmented as one token.

  • SeparatePlaceNameFromSuffix Option: This options will apply to Japanese Postal Address Level 1 and Level 2 words only.

  • IgnoreSeparators Option: In JLA 4.1, when set to false, JLA will always treat whitespace separators as token delimiters. In prior versions, this behavior was not always observed.

  • New Segmentation: Because of major lexicon change plus other significant changes in segmentation of Katakana words and compound verbs, it is highly recommended that existing customers consider re-segmenting existing data.

• Arabic and Chinese statistical algorithms have been implemented for named-entity extraction.

• Chinese base noun phrase identification has been added.

• Chinese part-of-speech algorithms have been improved.

• Four new entity types have been added to RLP (ACE/TIDES entity types): FACILITY (FAC), GEOPOLITICAL ENTITY (GPE), VEHICLE (VEH), WEAPON (WEA).

• Entity subtypes are now allowed.

• Korean dictionary comments which were previously only in Korean are now also in English. (Files: ham-cnn.dic, ham-rma.dic, ham-usr.dic)

• Named Entity Maximum Entropy Labeling has been implemented.

• For this release, there is a new component for named entity tagging. This is expected to increase the accuracy of entity labeling for the languages that are already covered: English, German and Japanese. It also lays the groundwork for the addition of new languages in the near future. Other than the accuracy improvements, there should be no visible differences between the previous output and the new output.

• Delay loading enables more efficient loading of dynamic libraries, as well as the ability to locate dependent libraries without setting the PATH environment variable. This feature is for Windows environments with MSVC 7.1.

• The document type definition neconfig.dtd has changed. It is now:

  <!ELEMENT neconfig (ne-options)+>
  <!ELEMENT ne-options (nerfeasturedatapath | labelweightsdatapath}>
  <!ATTLIST ne-options language CDATA #REQUIRED>
  <!ELEMENT env EMPTY>
  <!ATTLIST env name CDATA #REQUIRED>
  <!ELEMENT nerfeasturedatapath (#PCDATA | env)*>
  <!ELEMENT labelweightsdatapath (#PCDATA | env)*>

• featuredatapath and classdatapath are no longer used. labelweightsdatapath is used for named entity labeling. nerfeasturedatapath is used for named entity recognition (the spelling error is intentional for compatibility reasons).

• Two new Chinese POS tags have been added to the Chinese Language Processor: NP - Noun Proper and U - Unknown.

• The Gazetteer Text Processor Module allows users to create custom Named Entity definitions from their own data lists. See Chapter 4: RLP Processors of the RLP Programming and API Guide for documentation.

• This release introduces two new APIs for setting and handling the root directory: BT_REX_Environment::SetRootDirectory and BT_REX_Environment::RootDirectory.

• A new context property for the REXML Output Processor, suppress_header_comment, allows the option of turning off the header comment that includes the REX version used to generate the file.

• RLP Arabic capabilities have been expanded to handle variant Arabic numeric orthography such as the use of 'reh' (U+0631) as the decimal (U+066B) separator.

• This release adds support for Korean and Chinese to the Sentence Boundary Detector (SBD).

• There is now a .NET API and a .NET sample program for RLP.

• Three new POS tags appear in the Korean Language Processor: NN for numbers, FW for foreign (non-Korean) words and Unknown for words not recognized.

• A new Japanese LP option, IgnoreSeparators (default TRUE), can be set in the jla-options.xml file.

• go.sh
• go-cpp-samples.sh
• go-java-samples.sh

• Configuration: All European language processors (DELA, ENLA, ESLA, FRLA, and ITLA) have been replaced by the single language processor, BL1, and thus these older processors no longer appears in configuration files such as rlp-global.xml, rlp-context.xml, and rlp-context-no-op.xml.

• POS Tags: All POS tags for European languages have changed. See the RLP Programming and API Guide for full details.

• Threads: Currently, you can only run one analysis for BL1 at a time.