Marine mammal sightings (MMS) database

To view older data through the admin interface, use link https://data.npolar.no/sighting

The marine mammal sightings database can be found here:

The mms data from the schemas, which is what most people are looking for:
https://v2-api.npolar.no/sighting/marine-mammal/observation/?page=..&includeData=true
A database with the links to the original data (the Excel schemas):
https://v2-api.npolar.no/sighting/marine-mammal/source/?page=..&includeData=true

The database has been opened for access.

Some of the newer data might be added behind a login - this means that you will not be able to access the database through the browser, but use tools like Curl or Rlang. Login is that same as for all data accessed from the older version of https://data.npolar.no.

Contact person(s): Env Data Section: siri.uldal@npolar.no, Research: magnus.andersen@npolar.no

1. File storage prior to database storage

The original mms forms can be found under \\NPDATA\PROJECT\MMS\.

The current MMS excel schema and 2024 candidate: https://gitlab.npolar.no/npdc/other/npdc-docs-public-repo/templates/mms

2. Getting data back

Get the data, using https (Curl and Rlang descriptions below)

2.1 Using Kibana to get data (obsolete)

NP has a version of the Kibana visualization software set up with the mms database, available only through NPI’s network: http://hanna.npolar.no:5601

NOTE! Server uses the http protocol, not the secure https. Some browsers, f.ex. Firefox may have problems showing pages with http requests without additional configuration. Edge or Chrome usually presents the data with a warning asking if you want to continue to load an unsecure connection. Say yes as it is an NP webpage.

This server can be used to seach the MMS database for data without the hassle of passwords. Go to the “hamburger” menu in the upper left corner and select Analytics - Discover.

Choose database sighting-marine-mammal. To get data back you also need to set the time frame in the upper right corner to years back rather than the last 15 minutes. Choose Absolute and set the calendar dates.

Now you should see that the number of Available fields in the left side has changed. Choose the fields that applies to your query, f.ex. “scientificName”.

F.ex. if you want the scientificName to be “ursus maritimus”, use the search field in the upper left corner. Write “scientificName.keyword : “Ursus maritimus”” and press the blue refresh button on the upper right side. If there are any entries these should now turn up. Similarly, you can choose other fields.

See the map of all entries in database

If you wish, you can edit the Layer sighting-marine-mammal-geojson f.ex. by adding tooltip to see the species’ name (properties.scientificName) or date (properties.eventDate) in the map. To get a subset, you may use the search in the upper left corner.

See a dashboard with statistics examples

2.2 Get data back by using curl/https

From a Windows PC, download and install Curl.

MMS database example with login:

curl -X GET "https://v2-api.npolar.no/sighting/marine-mammal/observation/?page=..&includeData=true > mmsfile

Now you will get a json files downloaded. You can get these converted into Excel by following the description on Windows.

2.3 Get data back using Rlang

Code snippet to get all data into R lang:

in the map
   # Download the mms database

   #if you havn't install package jsonlite do this first
   install.packages('jsonlite')

   # Then fetch the library
   library(jsonlite)

   mms_json = fromJSON("https://v2-api.npolar.no/sighting/marine-mammal/observation/?page=..&includeData=true")

   # Traverse JSON hierarchy
   mms_df = mms_json$items$data

   # Ways of viewing and see columns by using names
   View(mms_df)
   head(mms_df, 5)
   names(mms_df)
   field_df_flat = flatten(mms_df, recursive = TRUE)

   # You can remove columns in R, but perhaps the easiest approach is to download the database as tsv and
   # remove unwanted columns in Excel afterwards. Comma is not recommended as delimiter as it is often included in the text.
   # If you want something else than tab, use the sep option, but tab should work in most if not all cases for mms.
   write.table(field_df_flat, file='mms.tsv', quote=FALSE, row.names = FALSE, sep='\t', fileEncoding = "UTF-8")

2.4 How to locate and search data in the database

In order to find what you are looking for you need to know the parameter names.

The parameter names follows the Darwin Core standard, however the MMS database also have parameters under dynamicProperties that are tailored to NPI’s needs. Some of the most used fields in the database:

scientificName: Species’ name in latin.
locality: Placename.
dynamicProperties.startDate:  Project startdate.
eventDate: Usually date of observation.
recordedByName: Name of observer.
individualCount: The actual count itself.
decimalLatitude: Latitude, point or part of box.
decimalLongitude: Longitude, point or part of box.

Missing a variable? JSON schemas for MMS holds all varibles with short descriptions (you need to log in with your NPI logon using LDAP)::: https://gitlab.npolar.no/npdc/other/v2-jsonschema/-/blob/development/v2-jsonschema/sighting/marine-mammal/observation-darwin-core.v1.0.0.json

Database example: https://v2-api.npolar.no/sighting/marine-mammal/observation/?page=..&_search=scientificName=ursus+maritimus&includeData=true

&page=.. means all data, not just the first page.

&verbose=true means include all metadata as well.

&includeData=true means get the data itself as well.

&type=feed means download as nd-json.

3. How to prepare and clean the incoming excel sheets

The original spreadsheets from the users may have a number of errors that makes it difficult to read direcly. Please check:

Columns and rows should not have moved. Some people try to alter the schema although we have locked it.

The rows to read should start from row 20.

Names should be in English or Latin in the schema. Norwegian names should be translated.

If someone registers a species that is not in the select list, it should be registered as “Other species” and the species added in the comments in Latin or English.

Lat and lng should be in decimalDegrees, not geoLocations (min, sec) or other formats. Separator should be dot, not comma.

Check spelling of placename that it is correct according to Svalbard placenames.

Date should be written consistently in the same format across all files and file entries.

Comment of the observation should not contain any names or ship of the observer as all info about names are stripped from the database before submission to Artsdatabanken.

During the schema cleaning the assessment should be added to the schema, these are:

col Y  - editorAssessment
col Z  - editorComment
col AA - editorDate
col AB - editorName
col AC - editorQualityTest

See

MMS field explanations

for explanation and controlled vocabulary available for these fields. If needed, additional vocabulary can be added if agreed upon as needed.

Try to avoid adding comments to the fields written by the user, rather add comments to the editorComment field and include your name and date.

To unlock the schema, store a version of it under a different name.

How to handle missing coord/placenames: If only a comment tells something about the placename, f.eks. “South of Moffen” for an animal with ocean as the only habitat. - Set Placename to “Moffen” - Lat and lon is set to a place directly south of Moffen - coordinateUncertaintyInMeters set to more than a kilometer. - editorQualityTest: geopointMissing, geopointOutsideHabitat.