MixLID

Token-Level Language Identification in Mixed-Language Text
Hugging Face arXiv
Identification mode

Text
Examples

Supported languages (111)

afr_Latn als_Latn amh_Ethi ara_Arab asm_Beng azj_Latn bam_Latn ben_Beng bod_Tibt bul_Cyrl cat_Latn ceb_Latn ces_Latn ckb_Arab cmn_Hans cmn_Hant dan_Latn deu_Latn ekk_Latn ell_Grek eng_Latn eus_Latn fas_Arab fil_Latn fin_Latn fra_Latn fuv_Latn gaz_Latn gug_Latn guj_Gujr hat_Latn hau_Latn heb_Hebr hin_Deva hrv_Latn hun_Latn hye_Armn ibo_Latn ilo_Latn ind_Latn isl_Latn ita_Latn jav_Latn jpn_Jpan kac_Latn kan_Knda kat_Geor kaz_Cyrl kea_Latn khk_Cyrl khm_Khmr kin_Latn kir_Cyrl kor_Hang lao_Laoo lin_Latn lit_Latn lug_Latn luo_Latn lvs_Latn mal_Mlym mar_Deva mkd_Cyrl mlt_Latn mri_Latn mya_Mymr nld_Latn nob_Latn npi_Deva nso_Latn nya_Latn ory_Orya pan_Guru pbt_Arab plt_Latn pol_Latn por_Latn ron_Latn rus_Cyrl shn_Mymr sin_Sinh slk_Latn slv_Latn sna_Latn snd_Arab som_Latn sot_Latn spa_Latn srp_Cyrl ssw_Latn sun_Latn swe_Latn swh_Latn tam_Taml tel_Telu tgk_Cyrl tha_Thai tir_Ethi tsn_Latn tso_Latn tur_Latn ukr_Cyrl urd_Arab uzn_Latn vie_Latn war_Latn wol_Latn xho_Latn yor_Latn zsm_Latn zul_Latn