RESTful, Group and Nodeaccess - an efficient combination (part 2/4)

Hi again! My name is George Baev and I am a Drupal developer at Sangre. In my previous blog post, I described the context of this project and background for moving into a Decoupled system in Drupal. I also explained the back end system and how we solved user roles and permissions. In this post, I will continue to describe the Drupal content types, their main fields and interconnections. The data migration functionality is explained also.

Drupal Content Types

The Diagram, Marker and Diagram Template are the main content types which are used in the project. These content types' nodes organize the information in a meaningful and flexible way. These are Drupal content types and they provide endless possibilities for expansion and new business logic implementation.

Diagram

The Diagram is the basic data structure in the project. Before describing the Diagram's structure, let's describe its relation to Groups and user access permissions in more details.

As we explained in the previous post, there are Diagram related quota limits which are applied in a Group. The following functionality checks if the user has the permission to create a Diagram in at least one Group:

/**
 * Checks if the current user could create Diagrams.
 *
 * @return bool
 *    The result of the check.
 */
function my_module_account_can_create_diagram() {
  global $user;

  $accountGroups = group_load_by_member($user->uid);

  if (empty($accountGroups)) {
    // The user doesn't belong to a group.
    return FALSE;
  }

  foreach ($accountGroups as $group) {
    // Check if diagrams quota has been reached.
    $diagrams_allowed = $group->field_diagrams_allowed[LANGUAGE_NONE][0]['value'] > $group->my_diagrams_used;
    if ($diagrams_allowed == FALSE) {
      continue;
    }

    if ($group->userHasPermission($user->uid, 'can create diagrams')) {
      // The user can create a diagram in at least one group.
      return TRUE;
    }
  }

  return FALSE;
}

The user should belong to at least one group in order to be able to create a Diagram. Then, we try to find at least one group where the quota limit hasn't been reached and the user has the appropriate permissions to create Diagrams. Once the user was allowed to create a Diagram, the particular Group user permissions where the Diagram has to be created are checked also (not shown here).

The Diagram's fields are based on the Paragraphs contributed module. The following description is taken from the module's landing page and it determines the Paragraphs major advantages:

Paragraphs module comes with a new "paragraphs" field type that works like Entity Reference's. Simply add a new paragraphs field on any Content Type you want and choose which Paragraph Types should be available to end-users. They can then add as many Paragraph items as you allowed them to and reorder them at will.

Another Paragraphs advantage is the automatic Parent-Child relationship. Each Paragraph entity is connected to its host entity and all Paragraphs entities are deleted automatically when the host entity is deleted.

There are Paragraphs disadvantages also. The interconnection between the Sectors and Markers Paragraphs entities is very hard to maintain because it is very resource consuming. We are still trying to implement efficient solution which clones a Diagram with relatively large number of Markers for reasonable time. The relatively heavy user interface disadvantage is solved successfully with the React front end which allows Diagrams manipulation such as zoom, drag and drop.

The three most important Diagram content type fields are Time Ranges, Sectors and Markers .

The Time Ranges are the concentric circles which determine the time periods in a diagram. They contain the Title and Year fields. The Title field is a text field and the Year field is a Date with the Year set as the Date attribute to collect.

The Sector is another relatively simple Paragraphs entity. It contains the Title and Description fields. Both these fields are text fields.

The Markers is a complex Paragraphs entity with several fields such as Marker, Sector, X Offset, Time. These fields describe the precise Marker position in the Diagram.

The Marker field is represented as an Entity reference to a connector node. This connector node contains the Marker's UUID as its title and no other fields. All Marker data is stored in an external system (Elasticsearch) which will be described later. We avoid data duplication with this organization and allow external applications to access the markers data directly through the Search API. The various user interactions such as voting and commenting are related to the connector node's Drupal Node Id. To summarize, the connector node joins the Elasticsearch stored Marker data with its Drupal votes, ratings and comments.

The Sector is an integrer field which stores the Sector paragraph entity id. This value is used by the React front end application as one parameter for the Marker positioning in the Diagram.

Both X Offset and Time are decimal fields which participate in the Marker's positioning in the Diagrams. Their values are used by the front end React application also.

Marker

The complex Marker structure is maintained by Elasticsearch. As described on the Elasticsearch landing page,

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected.

This is a sample Marker data representation in JSON format:

{
  "owner": 1,
  "visibility": "PUBLIC",
  "created": "2018-04-20T14:04:08+00:00",
  "language": "en",
  "title": "Sample title",
  "body": """
<p>This is the body.</p>

""",
  "uuid": "be1ba4e5-2dec-4d6f-9462-be3841af4c70",
  "feed_tag": [
    {
      "id": "536dbcb7-98e1-475d-87d1-942dcc396d35",
      "title": "tag1"
    }
  ],
  "time_range": {
    "id": "d0b91db9-6d3a-4975-a899-10c78dfe8fd4",
    "title": "2020"
  },
  "related_markers": [
    {
      "id": "38eb6b56-18ac-4344-803a-9ddce999ff3e",
      "title": "Marker 1"
    },
    {
      "id": "91acf5b9-5152-4ab4-b3a9-8a40c1ace966",
      "title": "Marker 2"
    }
  ],
  "state": {
    "id": "43fa863e-26ca-470c-8588-cf162cba08b5",
    "title": "State 1"
  },
  "original_id": 1,
  "group": 0
}

The owner property matches the author's user ID in Drupal. The visibility property, together with the group property, determine if the Marker could be added to a Diagram in any Group. There could be Markers which are visible in one Group only and this Group ID is stored in the group property.

The uuid property uniquely identifies the Marker. Its value matches the Drupal connector node's title as described above. The same uuid values are used in the related_markers property to identify the Markers related to this one.

One interesting aspect is the Drupal taxonomy terms representation in the Marker's data structure. The Universally Unique IDentifier Drupal contributed module allows automatic generation of UUID values for each taxonomy term. These UUID values are stored in the time_range, feed_tag and state properties. The UUID as taxonomy term identifier helps us to "disconnect" the term entity from its Drupal core and allow the external applications to work with these fields also.

Diagram Template

The Diagram Template content type is used to quickly create Diagrams with predefined set of Sectors, Time Ranges and Markers. This content type consists of the Paragraphs fields only. The Drupal administrator can define unlimited number of Diagram Template nodes. These nodes are displayed in the front-end interface to allow the user to auto-generate a new Diagram. It is possible to set taxonomy terms to these templates in order to improve the templates search in case the number of templates grow significantly.

Data Migration

This is the Drush command callback which we used to migrate the old content into the new Markers (some source code parts are removed):

/**
 * Drush command callback.
 *
 * @param int $uid
 *    The owner UID.
 */
function drush_migrate_markers($uid = NULL) {
  if (!is_numeric($uid)) {
    return drush_set_error(dt('Please submit the User ID!'));
  }

  $account = user_load($uid);

  if (!is_object($account)) {
    return drush_set_error(dt('The owner was not found!'));
  }

  $limit = drush_get_option('limit', 1);

  $query = new EntityFieldQuery();
  $entities = $query->entityCondition('entity_type', 'node')
    ->propertyCondition('type', 'marker')
    ->propertyCondition('status', NODE_PUBLISHED)
    ->propertyCondition('uid', $account->uid)
    ->propertyOrderBy('nid')
    ->range(0, $limit)
    ->execute();

  if (!isset($entities['node'])) {
    return drush_set_error(dt('No markers found'));
  }

  // Get RESTful module request handler to the Ingestion API.
  $handler = restful()->getResourceManager()->getPlugin('markers:1.0');

  $nids = array_keys($entities['node']);

  $taxonomy_fields = [
    // Contains the taxonomy fields dictionary.
  ];

  foreach ($nids as $marker_nid) {
    $ingestion_data = [];

    $marker = node_load($marker_nid);
    $wrapper = entity_metadata_wrapper('node', $marker);

    $data_arr = [
      // Contains the fields -> value mapping.
    ];

    $data_arr['body'] = 'N/A';
    if (!empty($wrapper->body->raw()) && is_array($wrapper->body->raw())) {
      $data_arr['body'] = $wrapper->body->raw()['value'];
    }

    foreach ($taxonomy_fields as $key => $tf) {
      if (empty($wrapper->{$key}->raw())) {
        continue;
      }

      $taxonomy_terms = $wrapper->{$key}->value();
      if (!empty($taxonomy_terms)) {
        if (is_object($taxonomy_terms)) {
          $data_arr[$tf] = [
            // Set the taxonomy term UUID.
            'id' => $taxonomy_terms->uuid,
            'title' => $taxonomy_terms->name,
          ];
        }
        elseif (is_array($taxonomy_terms)) {
          $values = [];
          foreach ($taxonomy_terms as $tt) {
            if (is_object($tt)) {
              // Set the related marker UUID.
              $values[] = ['id' => $tt->uuid, 'title' => $tt->title];
            }
          }

          $data_arr[$tf] = $values;
        }
      }
    }

    $ingestion_data[] = $data_arr;

    // Execute the API call.
    $result = $handler->doPost($ingestion_data);

    if (!empty($result['op']) && $result['op'] == 'create') {
      $marker->status = NODE_NOT_PUBLISHED;
      node_save($marker);

      drush_print(t('A marker was created with NID=@nid and UUID=@uuid', [
        '@nid' => $result['markers'][0]['nid'],
        '@uuid' => $result['markers'][0]['uuid'],
      ]));
    }

    unset($marker, $wrapper, $result);
  }
}

This Drush command retrieves the old "Marker" content nodes, sets the field values for the new Markers and executes a call to the Ingestion API to store the Marker in Elasticsearch. We shall describe the Ingestion API in more details in the next post.

Some interesting parts in this source code section are the lines:

$handler = restful()->getResourceManager()->getPlugin('markers:1.0');

and

$result = $handler->doPost($ingestion_data);

The RESTful Drupal contributed module allows the API usage from the Drupal source code. This helps us keep the data manipulation functionality on one place and avoid unnecessary code duplication.

The project RESTful APIs will be described in our next post. There are very interesting solutions there which use the PHP Traits. We shall understand how to define the API endpoints and create compound response data structures. See you soon!