Remove CitationInfo and use Citation instead

[ronald]

Citations are still being generated by loading the article xml, transforming that into an xml description of CitationInfo, and deserializing that xml description. Almost all of CitationInfo's info is available already in Citation - what's missing (is this really needed?) is the concept of collaborative authors and the marking of primary vs other authors.

Please see #742 for additional description. It is marked as a duplicate of this.

[amit]

Clean up citation-info caching. Please see r5681.

[amit]

Reassigning...

[dragisak]

Citation class is missing some properties that CitationInfo has:

1. Title formatting: Foo <italic>bar</italic> needs to be converted to Foo <i>bar</i> the way it is done in XSL (Same issue is in Article class). It can be done either during ingestion or on the fly, during the display.
2. Journal short name (as in XML: article/front/journal-meta/journal-id[@journal-id-type='nlm-ta']). Journal property seems to be always null.
3. Authors are listed as UserProfile objects, not as Author objects. UserProfile is missing suffix property.
4. startPage (article/front/article-meta/elocation-id) is missing in Citation

This will require data migration.

[amit]

The change from <italic>bar</italic> to <i>bar</i> should be done on the fly at display time.

[dragisak]

(In [6692]) Remove CitationInfo? that was generated from article XML by XSLT transformation and use Citation entity from triplet store. Modify article ingestion so it includes new properties in Citation and UserProfile?. Will require data migration. Addresses #921

[amit]

Reassigning to Ronald to do migration as part of search also.

[amit]

Assigning migration to Pradeep.

[dragisak]

(In [6741]) Should have been done in r6692. Addresses #921

[pradeep]

(In [6953]) Migrator for ambra v0.9 Citations. See the applicationContext.xml for tuning information. By default this runs in a back-ground thread allowing web-traffic to go thru. The migrations are cache aware and so there is no need to restart peers after the migrations are done.

If 'background' strategy is chosen, tune the txn time outs and blobThrottle to adjust maximum concurrency. The migration requires a write txn from mulgara - and so it is not expected that ingests etc. are attempted till migrations are all complete. Also note that a mulgara txn related error will terminate the migration operation and a restart of ambra is required to continue the rest of the migrations.

If 'background' strategy is not chosen, ambra start up will succeed only when all Citations in the database are migrated.

Addresses #921

[amit]

Re-assigning for testing.

[russ]

citation migrator is failing on many articles (21/44 on branch).

pradeep explained on the list that this is due to authors being out of order in mulgara, and that the solution is to reingest by hand.

i don't think it will be possible to reingest 50% of our articles by hand.

perhaps this belongs to dragisa?

[pradeep]

Replying to russ:

citation migrator is failing on many articles (21/44 on branch). pradeep explained on the list that this is due to authors being out of order in mulgara, and that the solution is to reingest by hand. i don't think it will be possible to reingest 50% of our articles by hand.

Hmm. Something is strange then. The serverbackup.gz that I used for running couple of rounds of migration tests, succeeded on moody.topazproject.org. (Barring just one article that failed because the duplicate citation key). My understanding was that this was a backup of the production data from a few weeks back. But if that is not the case and you suspect 50% of the articles in the production have the author order wrong, that also can be addressed during this migration.

Not sure how 50% of articles have the author order all wrong. Was the article XMLs on production edited after ingestion to correct some of these things - but a re-ingestion was not done?

The current version of Migration is flagging this as an error to bring attention to this so that it can be fixed by the admin - mainly by doing a re-ingest. But if the mismatch is as high as 50% as you say, then the Migrator can be modified to take care of this easily.

[pradeep]

Replying to pradeep:

Realized what the confusion was with the 50% number. (See the message on the mailing list) Since this is not a migration issue, assigning back to you to figure out what is going on with the data used in the test.

[russ]

pretty much confirmed that branch has corrupt data. migrations on stage corpus and clean small corpus did not reproduce this issue.

[anonymous]

Milestone 0.9.1 deleted