markdown 迁移模块 - 迁移CSV

Posted 2021-05-05

# Migrate Module - Migrate CSV

Notes on how to migrate a CSV doc into drupal maping the CSV data to fields using a custom module.

## Module requirements

* Migrate
* Migrate Plus
* Migrate Source CSV
* Migrate Tools

## Instructions

In the [module_name].info.yml include the needed dependcies like so:

```
name: 'Phasellus ac.'
type: module
description: 'Integer facilisis.'
core: 8.x
package: 'Custom'
dependencies:
  - drupal:migrate
  - migrate_plus:migrate_plu
```

Create a `config\install` directory. The files inside this will be where the logic for the migration is kept.

Within `config\install` create these three files.

* [module_name].config.yml
* migrate_plus.migration_group.[module_name].yml
* migrate_plus.migration.[module_name].yml

### [module_name].config.yml

For this example we can keep this very simple. Just this:

```
[module_name]:
```

### migrate_plus.migration_group.[module_name].yml

This is where we setup the migration group that this migration will be assigned to.

```
id: amvac_retailers
label: AMVAC Retailer 
description: Migrates Retailers for Find a Retailer map.
source_type: CSV
dependencies:
  enforced:
    module:
      - amvac_retailers
```

* id: A unique ID for the migration. This is usually the NAME part of the migration group declaration file name as discussed above.
* label: A human-friendly name of the migration group as it would appear in the UI.
* description: A brief description about the migration group.
* source_type: This would appear in the UI to provide a general hint as to where the data for this migration comes from.
* dependencies: Though this might sound a bit strange to Drupal 7 users, this segment is used to define modules on which the migration depends. When one of these required modules are missing / removed, the migration group is also automatically removed.

### migrate_plus.migration.[module_name].yml

This is where all of the logic/maping goes for the migration. Depending on what documentation or guide you're reading the keys/values in this code can differ as I think a good amount of this can be specified optionally (it will fall back to default values).

In this example I'm using the base code from the Migrate CSV yaml example. I'm also including some exra things like depenencies. Note that I am not using the `fields:` data that the evolvingweb.ca site uses but you could. I think it might be similar to `column_names:`, maybe not sure.

```
id: amvac_retailers
label: Amvac retailers.
migration_group: amvac_retailers
dependencies:
  enforced:
    module:
      - amvac_retailers
source:
  plugin: csv
  # Full path to the file.
  path: 'sites/default/files/retailer_locations.csv'
  # Column delimiter. Comma (,) by default.
  delimiter: ','
  # Field enclosure. Double quotation marks (") by default.
  enclosure: '"'
  # The number of rows at the beginning which are not data.
  header_row_count: 1
  # The column(s) to use as a key. Each column specified will 
  # create an index in the migration table and too many columns 
  # may throw an index size error.
  keys:
    - id
  # Here we identify the columns of interest in the source file. 
  # Each numeric key is the 0-based index of the column. 
  # For each column, the key below is the field name assigned to 
  # the data on import, to be used in field mappings below. 
  # The value is a user-friendly string for display by the 
  # migration UI.
  column_names:
    0:
      id: 'Unique Id'
    1:
      name: 'Dealer Name'
process:
  title: name
  type:
    plugin: default_value
    default_value: retailer_location
  field_retailer_name: name
destination:
  plugin: 'entity:node'
  default_bundle: retailer_location
migration_dependencies: null
```

* id: A unique identifier for the migration. In this example, I allocated the ID amvac_retailers, hence, the migration declaration file has been named migrate_plus.migration.amvac_retailers.yml. We can specifically execute this with the command `drush migrate-import amvac_retailers`.
* label: A human-friendly name of the migration as it would appear in the UI.
* migration_group: This puts the migration into the migration group amvac_retailers we created above. We can execute all migrations in a given group with the command `drush migrate-import --group=amvac_retailers`.
* dependencies: Just like in case of migration groups, this segment is used to define modules on which the migration depends. When one of these required modules are missing / removed, the migration is automatically removed.
* plugin: The plugin responsible for reading the source data. In our case we use the [custom module name] module which provides the source plugin csv. There are other modules available for other data sources like JSON, XML, etc.
* path: Path to the data source file - in this case, the amvac_retailers.csv file.
* header_row_count: This is a plugin-specific parameter which allows us to skip a number of rows from the top of the CSV.
* keys: This parameter defines a number of columns in the source data which form a unique key in the source data. This unique key will be used by the migrate module to relate records from the source with the records created in our Drupal site. With this relation, the migrate module can interpret changes in the source data and update the relevant data on the site. To execute an update, we use the parameter --update with our drush migrate-import command, for example `drush migrate-import --all --update`.
* title: Assign the Title column of the CSV as the title property of the node. Though we do not explicitly mention any plugin for this, in the background, Drupal uses the get plugin to handle this property.
* `field_retailer_name: name`: This is where we map the drupal field (left) with the csv data column name (right). For this example we're just doing simple text fields but depending on the field you're mapping too this are a lot of options.

## Process

Once the above is in place you interact with the migration tools via drush commands. I found I always need to run `drush migrate-import [migration_name] --update` after activating the module.

* `drush migrate-import [migration_name]` - Kicks off a migration.
* `drush migrate-import [migration_name] --update` - Kicks off a migration for new items as well as previously migrated items. I found that without the `--update` flag my new content wasn't getting pulled in.
* `drush config-import --partial --source=path/to/module/config/install` - When you install a custom module the `config/install` files are evaulated once on install not again. If you modify any of those files after enabling the module you can use this command to pull in those new changes.

So the steps in general.

* Setup custom module
* Enable custom module
* run `drush migrate-import [migration_name] --update` (needs to be run whenever the csv data changes).

## Reference

1. [Drupal 8 Migration: Migrating Basic Data (Part 1)](https://evolvingweb.ca/blog/drupal-8-migration-migrating-basic-data-part-1)
2. [Migrating data from a CSV source](https://www.drupal.org/docs/8/api/migrate-api/migrate-source-plugins/migrating-data-from-a-csv-source)
3. [Migrate Tools Commands](https://www.drupal.org/docs/8/upgrade/upgrade-using-drush)
4. [Migrate API](https://www.drupal.org/docs/8/api/migrate-api/migrate-api-overview)