Making our System Smarter

Computers are amazing at following instructions. So amazing, in fact, that a seemingly harmless instruction can potentially lead to an entirely false conclusion.

At our recent BiSciCol meeting at the University of Florida, we had a discussion about just such a case.

At its core, BiSciCol is all about connecting objects to each other. In order to accomplish connecting object identifiers to other objects, we have been using a simple relationship expression called “leadsTo” that indicates a direction in the nature of the relationship between one object and another. To illustrate how “leadsTo” works, lets provide a simple example. Suppose we have a collecting event, which we join to a specimen object using our relationship predicate “leadsTo”. The specimen object could then “lead to” a taxonomic determination, which could in turn “lead to” a scientist, and so on.

This is certainly useful as we can express an endless chain of objects and their derivatives , even if they exist in different databases. However, what if we extended the above example just a bit further, using our “leadsTo” relationship?

Uh oh--- By successively following the leadsTo relationships, we could now erroneously conclude that spec2-t1 came from spec3! This is not good! Fortunately, there is a solution.

We realized that the directional “leadsTo” relationship simply doesn't make very much sense in some situations, such as the connection between spec3 and person1 in the diagram above. Consequently, instead of the single “leadsTo” relationship, we actually need two relationship terms: one that has a distinct direction and one that implies no direction. Two terms are in use currently that do just this from the Dublin core standard: 1) relation (no direction) and 2) source (has direction).


In the first example above, we could avoid the problem entirely by describing the link between the taxonomic determination and the scientist as a non-directional relation. Using our new terminology, the graph would look something like this:


The computers involved in figuring out how to traverse the graph of relationships would know not to follow non-directional relationships and we would no longer infer that spec2-t1 came from spec3. Problem solved!


This post written by John Deck and Brian Stucky with input from Hilmar Lapp, Steve Baskauf, Andrea Thomer, Rob Guralnick, Lukasz Ziemba, Tim Robertson, Reed Beaman, and Nico Cellinese, summarizes a discussion that took place at the BiSciCol development meeting held on the March 31, 2012 at the University of Florida.

Development Meeting, Boulder, Colorado, 2-4 February, 2012

Two Days of Triplifying

The Boulder contingent of BiSciCol hosted a short two day "developer's meeting" that included John Deck, Brian Stucky, Lukasz Ziemba, Bryan Heidorn and his students Alyssa Janning and Qianjin Zhang.  As luck would have it, the BiSciCol crew arrived on a blustery morning just ahead of the biggest single February snowfall on record.  It proceeded to snow from Thursday afternoon to Saturday morning without much pause, lending a surreal quality to the proceedings.  It also fubared some plans to use campus meeting facilities, since that Friday was the first snow day in many a moon.  Enough about the weather!  Lets talk about what we did!

All participants were very pleased with the outcome of this meeting. Brian had been hard at work developing a generic plug-in interface so that anyone can write some simple code to connect whatever kinds of record sets they have and begin an import into the Triplifier.  OH! WAIT.  WAIT.  First things first!  What the heck is Triplifier, you ask?  And why do we think this Triplifier is such a good idea?

The BiSciCol project works by linking data in data sources based on logical relationships independent of a particular implementation. This is a different kettle of fish compared to a data standard. BiSciCol works where standards stop.  We particularly want to, for example, represent how a sequence is related to a specimen which is related to an event and a location.  The problem is creating a common "format" for expressing simple relationships and then using a set of those simple ones to build more complex "graphs" of these relationships. So what is that "common format"?  In the world of the Resource Description Framework (RDF), the format is called a "triple".

A triple is not that complicated; it basically expresses a unique fact about how things are related to one another.  The format of a triple is subject - predicate - object (thus the "triple" - three pieces of data). The triple format is not all that different from what is expressed in a database or spreadsheet or other structured data document.  The set of relationships that allow joins to happen in relational databases are in theory very similar.  So similar that one can convert a database or other document into triples.  And thus the point and value of a "Triplifier" -- a way to convert any set of documents into triples so that we can begin to compile a larger set of resources.

So back where we started...Brian Stucky has developed a generic plug-in for ingesting different types of data into the Triplifier. And Lukasz has used that generic plug-in to build a Darwin Core Archive ingester.  The big news, however, has to do with a platform called D2RQ.  Basically, D2RQ does the heavy lifting of representing relational databases (or your own declarations of relationships between objects) as triples.  At the heart of D2RQ are: 1) the "ClassMap" which represents classes from a schema; 2)"PropertyBridge" which basically defines the properties in RDF using the class map.; 3) Joins that link tables.  In a nutshell, a user can specify (or pass along a relational database) with the right information about the database and its foreign keys, and dump out RDF triples.

The good news is that we were able to test D2RQ with a very simple relational database that relates collectors and specimens to verify that we get the right outputs.  After some trial and error, and specifying the right class maps, we succeeded in generating meaningful RDF output.  Given this, we are ready to rock and roll with developing the Triplifier fully, and will be cranking on this over the next few months.  Lukasz has already made progress on a Web interface, and we are preparing to test the system with data from the Moorea Biocode and HERBIS project.

- Rob Guralnick reports

Development meeting at UF, Sept. 12-13, 2011 - A quick summary

Our core development team met again, and although a few key people were missing, there were enough to generate a healthy discussion and set the course for more progress on BiSciCol development. The following people attended: Reed Beaman (UF), Nico Cellinese (UF), Brian Stucky (University of Colorado, Boulder), John Deck (UC Berkeley), Bryan Heidorn (UA), Tom Orrell (SI), Kate Rachwal (UF), Russell Watkins (UF), Lukasz Ziemba (UF). Overall, we discussed a number of topics to include user scenarios and potential queries, output of query results, coding requirements, how to represent geographic information, how to handle taxonomic names, tasks to be completed, timetable. Consistent threads throughout these discussions were the form of BiSciCol, from a structural and user interface perspective, and its role with respect to data providers (clients), processing and service. On Monday 12th, we reviewed the user scenarios we generated in the past and discussed potential queries derived from the workflows presented in these scenarios. It quickly became clear that there was a convergence of common queries across all scenarios. Potential query types are listed below followed by a brief example in parentheses. It should be noted that the data schema is based on transitive relations between objects, including “relatedTo” and “sameAs” inferences between objects. Also, filters can be applied to any Object Id with respect to: date, source database, and level or depth of relatedness. These filters can be applied in combination within a single query (see below). We will have to spend more time thinking about scalability and how to handle very large and/or complex queries.

Enumeration of Query Types 

Relations = The graph of transitive relations attached to a particular object. Including “sameAs” inferences between this object and all related objects.

DateFilters = All queries below can be modified with a date filter for any Object ID Source DB Filters = Ability to filter on source database.

 Potential Query Types:

 1. WHERE Object=X, return relations (Given a specimen ID, return eventID, tissueIDs, etc… that are related)

 2. WHERE Object=X, return List of Objects that are "sameAs" (From Use Case #4: Given a collectors specimen ObjectID, return internal IDs, and other institution IDs that refer to this same ID)

 3. WHERE Object=X, and has duplicate non-matching literals (A GUID has two different taxonomic names "Jim" and "Fred")

 Filtering Queries (on rdf:Type):

 Recursive filters can be applied to derived graphs (returned output.).

We need to support multiple types of filters, which means (1) multiple filters at one level of relatedness and/or/not (2) multiple filters across several levels of relatedness.

1. Given an Object, return all related objects that are a certain rdf:type (e.g. Show all images that are attached to a Specimen ID).

 2. Given an Object, return related objects that are a certain rdf:type AND related objects that are another rdf:type (e.g. given an EventID, only search for specimens with identifications so we don't also include tissues, extractions, sequences with identifications).

A consideration that arose from the query conversation was database updates and tracking of changes. A discrete timetable schedule for database updates needs to be determined. The mechanism for updates will involve applying an inferencing process to data delivered from partners to create new relations tables. Currently, we think that the process will proceed as follows:

1. Clients publish data to a Virtuoso Database for example, (perhaps via GBIF’s IPT) into a. client specific relations table and b. client specific sameAs table

2. BiSciCol will periodically re-infer all relations (all previously submitted data is still accessible)

3. BiSciCol publishes results of user queries periodically (e.g. through RSS) Regarding real-time changes/incorporation of individual object records we think this is easily done for relatedObjects (e.g. putting in a newRelationsTable) but we need more discussion on the inferencing process to complete (1x / month) for sameAs relations

So, having stated all of the above we still need to: 

- Think about complex queries and scalability
- Expressing “Darwin-Triples” (institution code, collection code, catalog #) as a GUID. E.g. http://mydarwincore.org/resolver/ic=MVZ;cc=VERTS;cn=12345 OR LSID --- this is a project relegated to some implementation, probably HERBIS test case.
- Update Test case to put Multiple Literals for same Object
- Pre-inferred relations as a separate table?

We then discussed various forms of output of query results (see summary listed below). Suggestions included some standard forms of visualization such as summary statistics, charting (e.g. bar, pie, etc.), browsable lists, sortable HTML tables, a timeline of object updates and changes, and mapping of geolocated objects. Other ideas included RDF triples, various JSON outputs, some sort of standards compliance measure(s), incremental levels (e.g. list a specified depth or level of the sameAs or relatedTo). One interesting idea was to provide a means to evaluate the completeness and quality of collector efforts and databases. Two suggested approaches were user vetting, with Facebook-like “up or down” voting buttons, or a possibly more objective ranking algorithm (e.g. Google’s page ranking). For now, we are going to provide outputs in the form of tables, hierarchical (tree view), maps, and by rdf:Type. However, we agreed to seek out other interesting and innovative forms of visualizing results.

 0. pie charts, statistics on data that is returned (e.g. Sencha)
1. RDF Output (integrate w/ freebase)
2. JSON Output
3. Standards Compliance Measure (e.g. MIMARKS score in percent)
4. Sortable HTML Table 

5. Summarize results by rdf:Type (either modified HTML Table or modified JSTree e.g. see http://library.conservefloridawater.org/WCC?act=view&oid=11964077&lng=1)

• dwc:Identifications 
• 1 http://www.domain.com/ID1 
• 1 http://www.domain.com/ID2 
• dwc:Taxon 
• 2 http://www.domain.com/ID1 
• 2 http://www.domain.com/ID2 

6. Search for interesting visualizations
7. Service renders incremental portions of graphs (e.g. 1 transitive level deep)
8. Timeline – what has changed over a period (graph of date last modified)
9. Browse lists: institutions, collectors, etc …
10. Mapping results
11. Vote Up or Down on Datasets or Collectors (e.g. see http://answers.semanticweb.com/questions/1581/whats-missing-from-rdf-databases) - for example, consider facebook like buttons, or google page rank on output methods for vetting, objective algorithms.

Tuesday morning was largely occupied with technical coding discussions. Specific tasks were identified and prioritized relative to project milestones (see summary below). This included user interface development, documentation of code, REST service tuning, data ingestion filters, and RDF-specific enhancements such as indexing and inferencing. One major component identified for development has been dubbed “TriSciCol”, which establishes and expresses the triple-store object relations from client-supplied databases. Details on the design and functionality of this component will be circulated soon.

Focus areas

Lukasz Ziemba (UF) User Interface Development
Brian Stucky (CU) REST Service / Tweeks
Brian Stucky (CU) Code-Base / Documentation of Code / Unit Testing
John Deck Data Ingest (GBIF data?, Cam Web)
Deck/Stucky Data Indexing / RDF Structure / Inferencing Engines / Reasoners
Bryan Heidron (UA) Herbis Use Case / Arctos Integration
Orrell (SI), Deck, Cellinese (UF) TriSciCol – flesh out specifications and submit for review by BiSciCol collaborators

 How to store and represent geographic information in BiSciCol 

 “Location” is a reference to the verbatim location information by the original collector. This GUID points to the Darwin Core Location class except for the parts of that class that refer to a georeference. The “Georeference” class expresses a georeferencing instance that is related to “Location”. This allows us to express multiple georeferencing instances for an individual “Location” and track DateLastModified dates for georeferencing instances atomically.

 The result is that “Locations” are never mapped by BiSciCol, only “Georeferences” are. In this way, we can define types of “Georeferences” that correspond to the manner in which they will be mapped or utilized in consuming applications. E.g. geo:lat/geo:lng, SpatialThing, MGRS, etc.

 A note for consideration here is that currently georeference terms are embedded in the Location class in Darwin Core and we will probably want to make a reccomendation to move the georeferencing terms into their own class.

 Terms

 dwc:Location is an object type that indicates the associated identifier is a location (E.g. text description, coordinate, any direct reading from an instrument, or observed). The content linked by this identifier is normally verbatim location information. This identifier cannot contain references to location literals. 

:Georeference is an object type that indicates the associated identifier is a geo-referenced location. This is the only object type that can reference spatial representations as literals. Need update from DwC folks on status of Georeference being its own DwC Class?

:hasGeoreference is a predicate that joins an identifier of type GeoreferenceID to a literal representation of a spatial representation. :hasGeoreference can have the following subProperties:

• :hasMGRSGeoreference 
• :hasSpatialThingGeoreference 
• :hasWKTGeoreference 

 Example N3 file

:ObjectX a dwc:Event  

:ObjectX dwc:DateLastModified “2011-01-01” 

:ObjectY a dwc:Location 

:ObjectY dwc:DateLastModified “2011-01-02” 

:ObjectZ1 a dwc:Georeference 

:ObjectZ1 :hasSpatialThingGeoreference “48.198634,16.371648;crs=wgs84;u=40” 

:ObjectZ1 dwc:DateLastModified “2011-01-03” 

:ObjectZ2 a dwc:Georeference 

:ObjectZ2 :hasMGRSGeoreference “4QFJ1234567890” 

:ObjectZ2 dwc:DateLastModified “2011-01-03” 

:ObjectX relatedTo :ObjectY 

:ObjectY relatedTo :ObjectZ1 

:ObjectY relatedTo :ObjectZ2 

Java Code 

Georeference class implements Location interface. SpatialThingGeoreference, MGRSGeoreference, WKTGeoreference classes extend the Georeference class. A georeference is returned by BSCObject with the method getGeoreference. 

 We’ll meet again as a much larger group on the 19th of October during the TDWG meeting in New Orleans. If you happen to be there and interested in our discussion, do join us and/or stay tuned for more news.

Tagging use case 585

Tagging use case 585. requires semantic linking to taxonomy of the shark, geospatial track and place names and likely a tracking methods ontology.

BiSciCol core software architecture

This simplified UML diagram illustrates the current architecture of the core BiSciCol classes. These classes are responsible for most of the lower-level BiSciCol functionality: interacting with data, working with queries, traversing the BiSciCol data structures, and converting BiSciCol data to other representations, such as XML. Note that this diagram does not depict all relationships or dependencies among the classes. Instead, relationships most important for a general understanding of the code are included, such as inheritance, interface implementation, and aggregation. Further, most private class methods and members are excluded for simplicity.

In general, we've been working to develop an architecture characterized by classes with clearly-defined responsibilities and a high degree of flexibility. We've also tried to develop objects that map naturally to the BiSciCol problem domain. For instance, both BiSciCol data objects and models are represented by corresponding abstract data types. Furthermore, the code that implements BiSciCol objects and data operations is completely separated from the code responsible for converting BiSciCol data to other formats, such as XML.

This is simply a snapshot of the state of the code at this time. Some of the classes illustrated above are incomplete and/or will almost certainly be redesigned. Collaborations between classes will likely change as well. Nevertheless, the diagram shows not only progress we've made, but also many of the design ideas we've been working with.

BioSciCol VertNet Integration Meeting

Recent big news is the funding of VertNet! Given this, some of us got together to discuss ways we can work together towards some common goals.

Location:”Jupiter Cafe”
John Deck (Kolsch)
Aaron Steele (Jupiter Red)
John Wieczorek (Hot Chocolate)

VertNet will publish to PubSubHubbub. BiSciCol can subscribe to the hub and receive updates on what is going on.

JohnD:
- VN can encourage, promote, implement use of GUIDs
- Links to GUIDs of Annotation references (of all kinds like data quality stuff e.g. Creation of WGS84/dec.lat/lng instance of DMS data)
- BiSciCol can help VertNet in augmenting search (relationship graph search)

JohnW:
- BiSciCol is base study for annotation use cases (query on annotation relations?)
- VertNet won’t look at relationships between objects, so BiSciCol could be FTW
- Recommends Twitter dialog between BiSciCol and VN

Aaron:
- Annotations
- PubSubHubbub
- Twitter dialog

More to come. If 2 of the attendees can figure out how to use (and if they really want to use) twitter we can stay plugged in that way (per JohnW's suggestion). Otherwise, we'll be forced to meet over beers and use our speech.

Revised Prototype Implementation Diagram

This is a refined diagram describing our preliminary implementation