Rencontrer Bilbo en 2012

Plusieurs présentations et démonstrations de Bilbo sont prévues pour cette année :

  • CORIA 2012 (COnférence en Recherche d’Information et Applications) qui s’est tenue les 21, 22 et 23 mars à Bordeaux1
  • les 19 et 20 avril, une démonstration du web-service Bilbo aura lieu au stand du CNRS lors de la 21ème conférence internationale World Wide Web ( WWW’2012)  qui se tiendra à Lyon du 16 au 20 avril 2012
  • une présentation de deux corpus issus de Revues.org à Istanbul lors du colloque LREC 2012 qui aura lieu du 22 mai au 27 mai2
  • le colloque sur les digital humanities qui se tiendra à Hambourg (DH2012) du 16 au 22 juillet3

Young-Min Kim et Patrice Bellot présenteront la mise en place du projet, les évolutions techniques du web-service Bilbo et les premiers résultats avec des données externes.

  1. programme disponible ici : http://cifed-coria2012.labri.fr/programme.php []
  2. liste des papiers acceptés : http://www.lrec-conf.org/lrec2012/?List-of-accepted-papers []
  3. liste des posters : http://www.dh2012.uni-hamburg.de/conference/programme/poster-presentation []
Publié dans News | Un commentaire

Upper bound of annotation with proper noun features

An idea, which naturally stuck us, is to try out learning a model with a complete proper noun feature set for each strategy. It will give us an objective upper bound on the annotation performance. And from the expected upper bounds, we can decide which strategy is preferable that the others.

 The question here is how to complete proper noun lists. We figure out this problem by artificially attaching proper noun features from the real labels. For example, if a token has ‘surname’ label we attach ‘surnamelist’ feature to the token. In this way we testify the upper bound of different strategies. In the following table, we compare the upper bound of S1 and S3.

The result well depicts the limit of ‘namelist’ feature. Even though S1 works better than the baseline and produces a similar performance with S2 and S3, the upper bounds of S1 and S3(or S2) are quite different. When we apply distinct artificial features for all three fields, these fields are almost perfectly predicted whereas there is no improvement in surname and forename for the artificial feature ‘namelist’. It justifies our decision to continue to separate the ‘surnamelist’ and ‘forenamelsit’. Moreover the perfect annotation accuracy motivates us to explore in detail the completeness of proper noun features.

Publié dans Building Bilbo | Laisser un commentaire

Proper noun features III (corpus 1)

We extract seven different learning sets according to the defined strategies in the previous post. Several important fields are selected for the comparison including the surname, forename and place fields that are our main concern here.

First, we compare the baseline model and the strategy S1, which attaches the ‘namelist’ feature to both surname and forename found in author name list. We emphasize that surname and forename are searched in pairs in S1.

Compared to the baseline, S1 obtains a gain about 0.4 points in general accuracy. The three main fields improved a little especially the recall of surname and precision of place. The other fields also have some change, which is influenced by the added features.

Now we continue comparing other CRF models constructed by the strategies S2 and S3. Here we separate ‘surnamelist’ and ‘forenamelist’ features for their corresponding token. We also search them in pairs as in S1. In S2 we attach just the name-concerned features whereas in S3 we also find ‘placelist’.

S2 gives the same performance with S1 in general accuracy(87.43 vs. 87.46). However there are anyway some improvements in precision and recall of most fields. We assume that this separation between two name types induces the model to learn more intensively the name fields. On the other hand, the ‘placelist’ feature in S3 brings only a tiny difference in all fields. It is may because of the comparatively few number of tokens having ‘place’ label or the incompleteness of the used place lists.

Strategies S4 and S5 are designed to verify if searching surname and forename in pairs (as in S1, S2, S3) is effective or not. Therefore in S4 and S5, surname and forename are searched independently in the name list.

While the general accuracies do not change much, the detailed performance shows a somewhat different pattern with the previous strategies. In both S4 and S5, the improvement of recall of forename is noteworthy. But at the same time, the recall of surname rather decreases. It means that the separation brings a favorable influence to finding forename but not for surname. We also notice an interesting result in the place field, which is better revealed in terms of precision when applying ‘placelist’ feature (S5), but gives opposite result in recall. According to our analysis, when the ‘surnamelist’ and ‘forenamelist’ are scattered here and there in learning data, an incomplete ‘placelist’ feature can disturb modeling by estimating ‘place’ label to the tokens having ‘placelist’ feature. That is why the precision is more correct but the learned model can find less tokens, which have place label.

To see a pure effect of ‘placelist’ feature on accuracy, we add just ‘placelist’ feature in S6. Like the above diagnosis, the precision for ‘place’ field is much better than that of baseline (88.41 vs. 72.51) but its recall is worse as shown in the following table (85.80 vs. 90.53) :

These results lead us to some temporary conclusions. First, inserting proper noun features improves the performance but not much. Second, we can not guarantee that imposing different features to surname and forename will obtain a better annotation result. Third, ‘placelist’ feature works for the precision of place field but not for recall. Forth, the decision about searching in pair the ‘surnamelist’ and ‘forenamelist’ does not influence on general accuracy but it makes anyway some difference depending on fields. We assume that the independence assumption between two features will be more sensible to the quality of used external list than the pair assumption.

This synthesis inspires us to tackle the incompleteness of proper noun lists. And we treat this issue as a missing feature problem.

Publié dans Building Bilbo | Laisser un commentaire

Proper noun features II (corpus 1)

In the previous experiments on proper noun lists, certain author names were omitted because of parsing error. We deduce from this inadvertent mistake and the degradation of performance that we would have an improved result on the newly revised corpus with the same settings of proper noun feature as before. By modifying the way to match feature to token, we can prepare different learning data from same lists. So we design several detailed proper noun feature-matching strategies described in the following table. Same lists with the previous experiments are also used.

Strategy Type of external list Property Feature name
Baseline Without any proper noun lists
S1 People name list No distinction between surname and forename NAMELIST
S2 People name list Distinction between surname and forename SURNAMELIST FORENAMELIST
S3 People name list, Place lists Distinction between surname and forename, Place lists (city and country) SURNAMELIST FORENAMELIST PLACELIST
S4 People name list Surname and forename are separately found SURNAMELIST FORENAMELIST
S5 People name list, Place lists Surname and forename are separately found, Place lists (city and country) SURNAMELIST FORENAMELIST PLACELIST
S6 Place lists Place lists (city and country) PLACELIST

One solution against the restriction of name list, we allow the combination of new name from surnames and forenames in the list. It means that even a full author name in our data is not found in the list, if surname and forename are matched with their own list respectively, we accept them as name. This is possible because our author name list distinguishes between surname and forename.

An important rule in the matching strategies S1, S2 and S3 is we prohibit the separation of surname and forename in searching. That is, we first check if a given token is surname and if yes, we examine if one of surrounding token forename. For that we check three different expressions of full name: [surname forename], [forename surname], [surname , forename]. In this way we sequentially check all the tokens during feature extraction process. For the comparison, we also separately find surname and forename in the strategy S4 and S5. For the placelist feature, we also check city and country names, which consist of more than a word. A part of a reference in learning data with strategy S2 (or S3) is shown in the following figure.

Since the first four name tokens are found in the name list but the two latters are not, the presentation of the reference is like above. In this case, even though ‘René’ is found in the name list, as ‘OTAYEK’ is not, finally the matching rule does not attach ‘surnamelist’ or ‘forenamelist’ to them. On the contrary to this, we attach the ‘forenamelist’ feature to ‘René’ token when applying S4 and S5. In the next post we compare the performance of these seven (baseline + 6) strategies.

Publié dans Building Bilbo | Laisser un commentaire

Manual annotation revision (corpus 1)

Excluding any errors in manual annotation is hard to be achieved especially when the annotation structure is complex and data size is not small as ours. But the quality of manual annotation is one of the core factors in sequence learning because the estimation ability of a statistical model considerably depends on learning data extracted from the annotation.

With the progress of our experiments, we go on correcting any small errors in terms of both manual annotation and original input string. An example of the latter is inserting unnecessary space in URL when typing online a reference that provokes a wrong tokenization in Bilbo system. Since the first attempt of revision, which was intended to modify directly learning and test data, we have continued to correct dispersed tiny errors. After several attempt, we found that correcting them directly in our corpus was more practical in case we change feature-extracting strategy even though searching errors in the corpus is more annoying than that in learning and test data.

These little changes in the corpus always influence final annotation result and we currently have the following accuracies with the same settings on feature and field types with the previous experiments.

 There is a light difference by comparison with the first data revision. Especially, total number of tokens and the number of punctuation marks decrease (6632 vs. 6569 and 2090 vs. 2024) that comes mostly from the change of tokenization. The correction also allows us automatically remove several unintended field labels such as ‘author’, ‘name’ and ‘region’, whereas we keep ‘ref’ and ‘genname’ in spite of their week frequency (because they occur often in corpus 2). With this revised version of corpus 1, we try out a series of experiments for the incorporation of external resources such as proper noun list.

Publié dans Building Bilbo | Laisser un commentaire

Finding DOI through CrossRef

One of the main services that we have planed using automatic annotation is to provide a unique identifier such as DOI (Digital Object Identifier) to each reference. DOI is assigned by CrossRef, the registration agency of the International DOI Foundation (IDF). We use the CrossRef’s query service that allows us to extract DOI with some basic information about a target digital document.

To submit queries into CrossRef from our system, we need a registered user id. However, CrossRef also provides several guest query interfaces without inscription on the site. So before applying the service in our system, we analyzed the guest query interfaces. First, we need to have at least title and author name, especially surname of first author. If we enter full name, the system does not find an item. Second, the reference string parsing service in the site does not always work. For example, with the reference string, [WACQUANT, Loïc, "Deadly Symbiosis: When Ghetto and Prison Meet and Mesh", in Punishment and Society 3, 2000, pp. 95-134.], we could not find the DOI even though it actually exists. Third, when we add metadata besides the basic information for searching, it should be exact. If not the system cannot find DOI even though it could by just submitting basic information (title and author). Fourth, title with several beginning words works well.

Finally, we decided to use xml queries with some beginning words of title and surname of first author. For each bibliographical reference in test data of corpus 1, these two items are extracted from the automatic annotation result. As a result, among 215 references 22 are revealed as having a DOI. Interestingly, 10 DOIs over 22 have not been discovered using the auto parsing service of CrossRef at the guest query interface (marked with ‘***’ in  the following examples). We sampled 6 references that reflect the diversity of form.

(Reference string  | automatically extracted items | found DOI | particularity)

CHAUVEAU, J-P, DOZON J-P. et RICHARD J., 1981, Histoires de riz, histoires d’igname. Le cas de la moyenne Côte d’Ivoire, Africa, 51 (2), 621-658. | First author :   CHAUVEAU  Start of title :   Histoires de riz | DOI : 10.2307/1158830 |

JOHNSTON-FELLER, R., Color Science in the Examination of Museum Objects. Nondestructive Procedures, Los Angeles, The Getty Conservation Institute, 2001.  | First author :   JOHNSTON-FELLER    Start of title :   Color Science in the Examination of Museum Objects | DOI : 10.1002/(ISSN)1520-6378 | ***NO DOI when AUTO PARSING in the site

ORGANSKI Kenneth, 1958, World Politics, New York, Knopf. | First author :   ORGANSKI     Start of title :   World Politics | DOI : 10.2307/1338221 | ***WRONG DOI when AUTO PARSING in the site

KRUEGER Dirk & Fabrizio PERRI, “Does Income Inequality Lead to Consumption Inequality? Evidence and Theory”, Review of Economic Studies, 73 (1), 2006. | First author :   KRUEGER      Start of title :   Does Income Inequality Lead to Consumption Inequality | DOI : 10.1111/roes.2006.73.issue-1 |

Halperin David M., Winkler John J. et Zeitlin Froma I. (éd.) (1990), Before Sexuality. The Construction of Erotic Experience in the Ancient World, Princeton. | First author :   Halperin    Start of title :   Before Sexuality | DOI : 10.1080/03612759.1991.9949408 | ***NO DOI when AUTO PARSING in the site

Veyne Paul (1981), «L’homosexualité à Rome», Histoire 30, p. 71-78 (= légèrement augmenté, « L’homosexualité à Rome », Communication 35, 1982, p. 26-33), réédité dans Duby G. (éd.), Amour et sexualité en Occident (1991), Paris, Seuil, p. 69-77). | First author :   Veyne    Start of title :   L’homosexualité à Rome | DOI : 10.3406/comm.1982.1519 | 

 

Publié dans Building Bilbo | Laisser un commentaire

Several practical and learning issues for future work

First step to use the annotation result:

  • Finding a unique id such as DOI for each reference via the crossref site. It can be used as an important metadata that describes the reference.
  • Finding also a unique id such as ISBN for a reviewed book in an article of Revues.org, using external services such as amazon.org. Then a link can be created toward the corresponding page of the external site.

Practical construction of Bilbo system:

  • Expanding the testified result to other real references by integrating the test codes into the Revues.org platform.

To improve the accuracy of automatic annotation:

  • Incorporation of the external resources at the level of modeling. A kind of semi-supervised learning seems to be appropriate for this issue.
  • Applying rule based post-processing.

Processing of third corpus:

  • Another main tool for the treatment of corpus 3 will be topic model for the context analysis.
  • Named entity disambiguation is necessary because the initial expression of forename confuses the author identification.

Each issue will involve a number of detailed statistical or heuristic techniques.

 

Publié dans Building Bilbo | Laisser un commentaire

Note annotation on Corpus level 2

We continue our experiments with the corpus 2. In the previous post, we showed that a well-defined set of three different feature types improves note classification accuracy. Now we verify the usefulness of note classification on automatic annotation of bibliographical reference in notes.

 Two CRF models are constructed on two different sets respectively. First set is ‘NotesCL’ that consists of the classified notes into positive category after applying a SVM with the strategy 10. Second set is ‘NotesOR’ that contains all 1532 notes without classification. We randomly selected 70% of a set as learning data and the remaining 30% as test data just as the previous experiments.

 The above figure shows the annotation result of these two models with some selected fileds. The gray cells are three most important fields, surname, forename and title of article. Embolden value means that the corresponding CRF model better estimates on the field than the other model. Total accuracy of our approach is 87.28% and it outperforms the baseline (85.16%). Considering the baseline model is learned on a larger number of notes (1532 vs. 1185), this result is quite encouraging because in general classification result improves with number of instances.

Consequently, our approach of sequence classification then applying a CRF significantly improved the final annotation accuracy.

Publié dans Building Bilbo | Laisser un commentaire

Réunion du 8 décembre 2011

Cette dernière réunion de l’année a permis de faire le point sur les avancées et sur les futures évolutions du programme de R&D Bilbo.

Jusqu’à maintenant, nous avons travaillé sur les modèles CRFs permettant d’annoter automatiquement les références bibliographiques présentes dans les articles de Revues.org. Une phase d’évaluation est prévue pour le deuxième corpus, en cours de développement actuellement. Pour le traitement du troisième corpus, les différentes techniques d’apprentissage statistique vont être appliquées parallèlement au modèle CRF. Il est aussi prévu, pour le début de l’année 2012, d’intégrer le système Bilbo dans la plateforme de Revues.org.

Plusieurs corpus vont être créés d’une part pour la détection des langues, d’autre part sur les comptes-rendus de lecture. Ce deuxième corpus est destiné à la création de liens externes. Comme la partie bibliographique du compte-rendu est souvent sous la forme d’une phrase ressemblant aux  références du corpus 1, nous pourrons tester la performance de la segmentation du système avec ce nouveau corpus.

Publié dans LIA and Cléo meetings | Laisser un commentaire

A paper accepted in BooksOnline11 workshop

A paper “Automatic Annotation of Bibliographical References in Digital Humanities Books, Articles and Blogs” which describes our work is accepted in the workshop BooksOnline11 (Oct. 24, 2011. Glasgow, UK.) organized at CIKM 2011.

The list of accepted papers in this workshop is:

http://research.microsoft.com/en-us/events/booksonline11/acceptedpapers.aspx

Our paper has been allocated both oral presentation and poster slot. The progress of Bilbo system is summarized in the following poster (click image for larger view).

 

Publié dans News | Laisser un commentaire

Rencontre du 4 octobre

Pour cette réunion du mois d’octobre, nous avons fait le point sur les avancées de Bilbo, des projets en cours et des possibilités de communication à son sujet.

La présentation de Bilbo, lors de l’atelier Enrichir des métadonnées – méthode manuelle et automatique à l’Université d’été du Cléo (12 au 16 septembre 2011), a été bien accueillie  par les participants, surpris des premiers résultats des tests de Bilbo, mais qui y ont vu un intérêt certain dans la recherche d’informations. Au vu de ces retours, nous avons décidé de présenter notre projet lors du prochain THATCamp de Paris (septembre 2012). Par ailleurs, Young-Min Kim exposera son travail lors du BooksOnline 2011 qui se tiendra le 24 octobre à Glasgow (un prochain billet reprendra cet évènement).

Au cours de cette réunion, une présentation-explication de l’avancée des travaux a été faite par les deux équipes, d’une part sur la constitution du troisième corpus (voir Les observations et Identification et annotation des références implicites), d’autre part sur les expérimentations menées sur les deux premiers corpus (voir notamment Les stratégies d’exploitation du corpus 2 et Les expériences sur le corpus 1).

De nouvelles pistes vont être explorées pour les mois à venir :

  • le problème du multilinguisme dans les revues : il s’agira de signaler les différentes langues présentes sur l’ensemble des plateformes du Cléo afin de les prendre en compte dans la construction de Bilbo.
  • les référentiels : la recherche d’une utilisation plus pertinente des référentiels pour améliorer le travail de détection de Bilbo.
Publié dans LIA and Cléo meetings | Laisser un commentaire

Identification et annotation des références implicites

Comme nous avons pu l’observer pour le premier et deuxième corpus, la plateforme Revues.org compte plus de 300 revues qui, en fonction des disciplines scientifiques, présentent des styles bibliographiques particuliers. Pour ce troisième corpus, le style bibliographique importe peu car nous nous appuyons sur le troisième niveau de difficulté, les références imbriquées dans du contenu. Nous avons fait le choix de représenter dans ce corpus d’apprentissage, des références présentes dans le texte, mais aussi des références qui sont en partie dans le texte et dans les notes de bas de page, et des références présentes entre parenthèses.

Nous avons, à partir des observations, constituer deux sous-groupes :

  • un groupe de 553 références bibliographiques implicites.
  • un groupe de 449 références particulières entre parenthèses comme : (auteur, date) ou (auteur, date : pages) qui sont des renvois à des références situées dans la partie bibliographie de l’article.

Nous avons complété cette nouvelle base d’apprentissage par des fichiers XML issus du deuxième corpus qui présentaient des références implicites (43 références au total).

Exemple 1 :

<bibl><author><surname>Gipi</surname></author>, surtout après qu’il a remporté à Angoulême le prix du meilleur album de l’année <date>2005</date> pour <title level=”m”><hi rend=”italic”>Notes pour</hi> <hi rend=”italic”>une histoire de guerre. </hi></title></bibl>

Exemple 2 :

<hi rend=”italic”><title level=”m”>Extérieur nuit</title> </hi>est un exemple remarquable de cette esthétique : il<hi rend=”italic”> </hi>s’agit d’un recueil de récits courts parus en <pubPlace>Italie</pubPlace> chez <publisher>Coconino Press</publisher> en <date>2003</date>, puis <edition>réimprimés</edition> en <date>2005</date></bibl>

Nous pouvons noter que dans l’usage des références bibliographiques, les auteurs privilégient les références courtes et entre parenthèses qui permettent un renvoi à la partie bibliographie de l’article, ce qui nous a conduit à créer un deuxième sous-groupe dans ce troisième corpus et à réaliser des liens entre ces références. Nous avons attribué un identifiant unique à chaque référence bibliographique :

<bibl> @xml :id= “nom de (des) auteurs + date“

Puis nous avons créé des liens entre les références en utilisant :

<link> @target=“xml :id “

Ce qui nous donne par exemple :

  • Pour la référence courte :

<c type=”parenthesis_left”>(</c><bibl><author><surname>Zérah</surname></author><c type=”comma”>,</c><date>2005</date><c type=”colon”>:</c><biblScope type=”pp”>177</biblScope><link target=”zerah2005″/></bibl><c type=”parenthesis_right”>)</c>

  • Pour la référence longue (non annotée mais attribution d’un identifiant) :

<bibl xml:id=”zerah2005″>Zérah Marie-Hélène (2005), « Enjeux d’eau en Inde. Des effets allocatifs et redistributifs complexes entre usagers et territoires », <hi rend=”italic”>Sciences de la société</hi>, n°64, février, pp. 174-193.</bibl>

Publié dans Annotation in XML TEI | Laisser un commentaire

Quelques observations sur les références implicites de Revues.org

Pour la constitution de ce nouveau corpus, nous avons dû nous poser plusieurs questions sur l’identification et l’annotation de références implicites (notamment pour le premier sous-groupe). Il nous a fallu dans un premier temps, identifier les éléments importants qui composent une référence bibliographique (du plus important au moins important) :

  • le titre
  • le ou les auteurs
  • la date
  • le lieu et la maison d’édition

La question de l’importance des éléments d’une référence bibliographique peut paraitre simple mais elle est nécessaire car elle permet de maintenir une certaine cohérence  dans le repérage des références avec les balises <bibl> et dans son annotation. En effet, la question posée de manière implicite est celle de savoir à partir de combien d’éléments une référence est considérée comme bibliographique et quels sont les éléments qui font d’elle une référence. Autrement dit, le titre d’un article ou d’un ouvrage est suffisant pour lancer une recherche dans un moteur de recherche, et les éléments comme auteur, date, lieu et maison d’édition viennent préciser la recherche. Alors qu’une recherche à partir de l’auteur serait  inefficace par exemple.

Ce corpus pose une autre question, quand débute et quand finit une référence bibliographique implicite ? Cette question est importante car de nombreuses difficultés sont apparus dans le repérage des références. En effet, nous pouvons observer que les références peuvent se trouver à deux endroits, d’une part dans le corps du texte, d’autre part à cheval entre le texte et la note de bas de page. Le problème devient plus complexe quand le nom de l’auteur se situe deux à trois lignes au-dessus ou en-dessous du titre de l’ouvrage ou de l’article. Nous avons fait le choix, dans ce cas de considérer qu’une référence commence au premier élément bibliographique (auteur, titre, etc.) jusqu’à la dernière indication bibliographique (date, page, ou autre marque de publication).

Publié dans Annotation in XML TEI | Laisser un commentaire

Proper noun features on corpus level 1

This part of experiments constitutes the use of external proper noun lists. To overcome the miss-annotation between people name and place, we think of using a set of proper noun lists. People name and country lists are provided by the Revues.org article collection and place list is completed from top 3000 largest city list (http://www.mongabay.com/cities_pop_01.htm) and 215 livable city list (http://www.businessweek.com/interactive_reports/livable_cities_worldwide.html).

 In fact, the previous study (Accurate Information Extraction from Research Papers using Conditional Random Fields, 2004) reports that the use of lexicon features does not influence much on the performance. But because of the uniqueness of our corpus that leads to some confusion between proper noun types (name and place), we expect that an appropriate use of proper noun lists would help the distinction of these two types.

 As the first attempt to apply proper nouns in a CRF model, we simply use them as features like in the earlier study. Considering that the name list is far from enough, we separate the surname and forename. It means that for a given token, we first check if the token is in the surname list and then we check if a nearby token is in the forename list. In this way, even a full name separately found in the list can be detected. The place lists are simpler. We just search if the given token is in the lists.

 We constructed four different models applying different lists as follows:

  •  People name list
  • City list
  • Country list + City list
  • People name list + City list

In this version, we verified again the learning and test data and made some correction. In addition, the “PUNC” feature, which was ignored in the previous experiments, is counted.

As we can see in the above table, all the attempts to apply proper noun lists have been failed. The proper noun features may somewhat effective to detect the corresponding proper noun types however the incomplete lists rather reduce the overall annotation performance.

Consequently, we are now interested in developing a new method to insert these lists. Since the Revues.org data includes several useful additional lists, we expect to find an effective way to use them that would contribute to the annotation performance.

Publié dans Building Bilbo | Laisser un commentaire

CRF model reconstruction on the revised data (Corpus 1)

For the moment, we leave the note classification problem alone and return to the CRF learning on the corpus 1. After the detailed evaluation and revision of the automatic reference annotation result, we now try to reconstruct a CRF model on the revised learning data. As written in the completeness verification report, the main correction of learning and test data had been carried out on title and book title. So we expect that the reconstructed CRF model is able to separate better the title and book title than before. The following table shows the estimation result of this newly constructed CRF model.

This result is not exactly what we expected. In spite of the data revision, general accuracy rather decreased about 1% compared to the final version on corpus 1. Considering that the 1% difference is acceptable with the change of learning data, we conclude that the automatic annotation quality is not really changed.

Some small errors of manual annotation do not influence the automatic annotation quality. Instead, the limit of the current version of Bilbo system should be overcome using more informative features or by modifying the existing CRF model. Named entity lists can be used as the former approach and in the next posting, we show the results of experiments that use these lists.

Publié dans Building Bilbo | Laisser un commentaire