Standardizing Tweets with Character-level Machine Translation (CROSBI ID 55173)
Prilog u knjizi | izvorni znanstveni rad
Podaci o odgovornosti
Ljubešić, Nikola ; Erjavec, Tomaž ; Fišer, Darja
engleski
Standardizing Tweets with Character-level Machine Translation
This paper presents the results of the standardization procedure of Slovene tweets that are full of colloquial, dialectal and foreign-language elements. With the aim of minimizing the human input required we produced a manually normalized lexicon of the most salient out-of-vocabulary (OOV) tokens and used it to train a character-level statistical machine translation system (CSMT). Best results were obtained by combining the manually constructed lexicon and CSMT as fallback with an overall improvement of 9.9% increase on all tokens and 31.3% on OOV tokens. Manual preparation of data in a lexicon manner has proven to be more efficient than normalizing running text for the task at hand. Finally we performed an extrinsic evaluation where we automatically lemmatized the test corpus taking as input either original or automatically standardized wordforms, and achieved 75.1% per-token accuracy with the former and 83.6% with the latter, thus demonstrating that standardization has significant benefits for upstream processing.
twitterese, standardization, character-level machine translation
nije evidentirano
nije evidentirano
nije evidentirano
nije evidentirano
nije evidentirano
nije evidentirano
Podaci o prilogu
164-175.
objavljeno
Podaci o knjizi
Computational Linguistics and Intelligent Text Processing
Gelbukh, Alexander
Berlin: Springer
2014.
978-3-642-54903-8