Pass The Text And Detect Its Language Automatically

The classification or grouping of text is an area of active research and the different Universities of Informatic Sciences are reaching great relevance for science in general. Many of these algorithms use a model that is developed for language detection based on those used in algorithms for information retrieval. A particular area within text classification is language determination. Some public applications such as the services provided by Google make obvious use of this type of classification. At Pass The Text And Detect Its Language Automaticallywe are interested in telling you about an alternative; Text Translation and Language Detector API.

Before you can perform word and sentence segmentation, character normalization, or lemmatization, you must determine the language of the source document.

Linguistic processes detect the language of a source document during parsing and indexing, not during query processing.

You can specify whether you want to use automatic language detection when you define data sniffers. You can also specify which language should be used if the parser cannot determine the source language. If you do not enable automatic language detection for a data crawler, the analyzer uses the language that you specify. If you do not specify a language, the parser uses the English language.

When you create the collection you can limit the languages that are returned by the automatic language detection. If a document is in a language that you have not selected for the collection; the first selected language will be used to parse the document. You can select languages in order of priority when you create a collection.

For example, if you specify that you want to use English and French when you create the collection, and the automatic language detection process identifies a document as written in German, the parser uses English when processing the document for the index.

Documents for which there are no language-specific dictionaries are processed using a basic language-independent method, such as whitespace segmentation and n-gram segmentation.

When you search a collection, you can restrict your search results to only documents that are written in a certain language. For example, if you search for documents about Jacques Chirac in a collection of documents written in multiple languages, you can limit your search results to include only documents that are written in French.

Language detection technology is best suited for documents written in only one language. If a document is written in multiple languages, an attempt is made to determine the most dominant language used in the document. But the results of the analysis are not always satisfactory.