2.2.5.2. AWS MarketPlace Cloudformation Specifications

creation-day:

Aug. 1, 2022

update date:

February 2, 2025

The AWS MarketPlace service, mijin Catapult(v.2), is deployed using Cloudformation, an orchestration tool.
This chapter describes the AWS resources created using Cloudformation.

The Cloudformation Template (CFT) consists of multiple files, where the parent Stack calls each of its child Stacks. The child Stacks to be invoked depend on the parameters of the parent Stack.

../../_images/cft.jpg

2.2.5.2.1. macroNestStack

macroNestStack creates a Cloudformation Macro. If the parameter mijinStackAlreadyExist is Yes, a stack that does not create a Macro will call (empty-macro). This is because the Cloudformation Macro is created with a unique name, so it is not possible to create multiple Macro’s with the same name.

Macro is created in Lambda (Node.js) and converts the CFT of mijinNestStack that comes after this by specified parameters.
Assign the following IAM roles and policies to allow the created Lambda to read and write to Amazon CloudWatch Logs.
Resources:
  PeerUnitsExecutionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: /
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource:
                  - Fn::Join:
                    - ':'
                    -
                      - 'arn:aws:logs'
                      - Ref: 'AWS::Region'
                      - Ref: 'AWS::AccountId'
                      - 'log-group:/aws/lambda/*:*:*'

The Cloudformation Macro is created in Lambda (Node.js) and converts the CFT of the mijinNestStack that follows by the specified parameters.The Macro can replicate the stack of EC2 instances and dynamically change the EC2 instances to be launched, depending on the number of the parameter PeerNumberOfUnits.

Warning

To create multiple mijin Catapult(v.2) in the same region, mijinStackAlreadyExist must be YES.

2.2.5.2.2. vpcNestStack

vpcNestStack creates a new VPC. A multi-AZ environment is created, with public and private subnets placed in each AZ. The default gateway for private network routing can also go out to the Internet using the Nat Gateway placed on the public network in the same AZ.

../../_images/vpc.jpg

Note

This stack is not used when deploying mijin in an existing network.

2.2.5.2.3. iamNestStack

iamNestStack creates IAM roles and the IAM policies associated with them for use by EC2 instances.

The following are roles to be assigned to each API node and PEER node.

AWSApiAccessRole:
  Type: 'AWS::IAM::Role'
  Properties:
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - ec2.amazonaws.com
          Action:
            - 'sts:AssumeRole'
    Path: /
AWSPeerAccessRole:
  Type: 'AWS::IAM::Role'
  Properties:
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - ec2.amazonaws.com
          Action:
            - 'sts:AssumeRole'
    Path: /
Grant permissions to AWS Systems Manager Session Manager
AWS Systems Manager Session Manager grants privileges to EC2 instances to allow remote login with IAM privileges and ties them to IAM roles.
AWSSSMRolePolicies:
  Type: 'AWS::IAM::Policy'
  Properties:
    PolicyName: AWSSSMAccessPolicy
    PolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: Allow
          Action:
            - 'ssm:DescribeAssociation'
            - 'ssm:GetDeployablePatchSnapshotForInstance'
            - 'ssm:GetDocument'
            - 'ssm:GetManifest'
            - 'ssm:GetParameters'
            - 'ssm:ListAssociations'
            - 'ssm:ListInstanceAssociations'
            - 'ssm:PutInventory'
            - 'ssm:PutComplianceItems'
            - 'ssm:PutConfigurePackageResult'
            - 'ssm:UpdateAssociationStatus'
            - 'ssm:UpdateInstanceAssociationStatus'
            - 'ssm:UpdateInstanceInformation'
          Resource: '*'
        - Effect: Allow
          Action:
            - 'ssmmessages:CreateControlChannel'
            - 'ssmmessages:CreateDataChannel'
            - 'ssmmessages:OpenControlChannel'
            - 'ssmmessages:OpenDataChannel'
          Resource: '*'
Grant read/write permissions to AWS Systems Manager Parameter Store.
The AWS Systems Manager parameter store stores dynamically generated data for the first node of mijin Catapult(v.2), grants permissions that can be referenced by each of the other nodes, and ties them to IAM roles.
AWSPSAccessRolePolicies:
  Type: 'AWS::IAM::Policy'
  Properties:
    PolicyName: AWSPSAccessRole
    PolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: Allow
          Action:
            - 'ssm:PutParameter'
            - 'ssm:GetParameter'
            - 'ssm:GetParametersByPath'
Grant Security Token Service (STS) from the resource of the same account same service name IAM role and link the IAM role.
With the STS granted, the AWSApiAccessRole and AWSPeerAccessRole IAM roles can operate the AWS services specified in the policy.
AWSAssumeAccessRolePolicies:
  Type: 'AWS::IAM::Policy'
  Properties:
    PolicyName: AWSAssumeAccessRole
    PolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: Allow
          Action:
            - 'sts:AssumeRole'
          Resource:
            - Fn::Join:
                    - ':'
                    -
                      - 'arn:aws:iam:'
                      - Ref: 'AWS::AccountId'
                      - !Sub "role/${ServiceName}*"
    Roles:
      - !Ref AWSApiAccessRole
      - !Ref AWSPeerAccessRole

2.2.5.2.4. s3NestStack

Warning

This feature is currently disabled and only the parameter store can be saved.

s3NestStack allows you to choose whether to place mijin data in the AWS Systems Manager parameter store or in an S3 bucket, and if you choose S3, it creates an S3 bucket.

2.2.5.2.5. mijinNestStack

mijinNestStack creates private DNS, security groups, EBS, and EC2 instances with Route53.

mijin uses DNS names to communicate between each EC2 instance. The domain is fixed at mijin.internal, and each instance name is set to the A record as follows.

  1. api1.mijin.internal

  2. api2.mijin.internal

  3. peer1.mijin.internal

  4. peer2.mijin.internal

  5. peer3.mijin.internal

  6. peer4.mijin.internal

  7. peer5.mijin.internal

  8. peer6.mijin.internal

  9. peer7.mijin.internal

  10. peer8.mijin.internal

  11. peer9.mijin.internal

Security groups are applied below, with the minimum security group used for communication as shown in the image.

Security group name

Description

attach-ssm_ssh-client

This security group can be set up as a stepping stone, etc. to allow SSH login. It is not used for new VPCs. (If created for an existing VPC, assign it to an existing stepping stone, etc.)

attach-node-client

For inter-node communication.

sv-api
This is for API nodes.
3000 port / for REST access.
7900 / For mijin node communication.

sv-peer
For PEER node.
7900 port / for mijin node communication.
../../_images/sg_network.en.jpg
Since EBS stores mijin’s data area, two EBSs are attached to the API node in addition to the root partition, and one EBS is attached to the PEER node.
VolumeType is fixed GP3 and no disk encryption.
The EBS is designed to be XFS formatted and disk mounted using cloud-init only when the EC2 instance is built for the first time.
The mijin data is stored in block, the API node stores the block and the mongo data used by rest, but can restore the mongo data if there is data in block.
../../_images/ebs.jpg

The EC2 instance is started using a custom AMI with the mijin package installed. UserData runs cloud-init and executes the initial package configuration. It receives the results of the UserData execution and rolls back if the configuration fails.

  1. Unix user settings specified by parameter

  2. Host Name Settings

  3. OS package updates

  4. Installing pip

  5. Installing cloudformation helper script

  6. Setup of mijin

    1. EBS Format

    2. mijin setup (api1 uploads data to parameter store)

In addition, the order in which instances are created is as follows

  1. ApiInstance1

  2. ApiInstance2 PeerInstanceX Concurrency

At ApiInstance1, create the configuration to be used for all nodes and store the data in the AWS Systems Manager parameter store. Other instances will retrieve data from this parameter store and create mijin.
The contents stored in the parameter store are as follows

Parameter name

Description

/Crown name specified at deploy time/shares/api_node.json

Public key used by the API node

/Crown name specified at deploy time/shares/generation_hash.json

mijin Catapult(v.2) blockchain’s Genesis Hash (GenerationHash)

/Crown name specified at deploy time/shares/harvest_fee_sink_public_key.json

Address to receive Harvest (not required for mijin)

/Crown name specified at deploy time/shares/init_host_count.json

Number of nodes created during deployment

/Crown name specified at deploy time/shares/mosaic_rental_fee_sink_public_key.json

Address to receive Mosaic rental fees

/Crown name specified at deploy time/shares/namespace_rental_fee_sink_public_key.json

Address to receive Namespace rental fees

/Crown name specified at deploy time/shares/nemesis_addresses.json

Empty address not used in particular (can be used)

/Crown name specified at deploy time/shares/nemesis_addresses_harvesting.json

Address to receive harvest, etc.

/Crown name specified at deploy time/shares/nemesis_addresses_harvesting_voting.json

Address used for authorization to finalize

/Crown name specified at deploy time/shares/nemesis_addresses_harvesting_vrf.json

Addresses for enhanced security (to obscure the state in which blocks can be generated)

/Crown name specified at deploy time/shares/peer_node.json

Public key used by the PEER node

/Crown name specified at deploy time/shares/rest_gateway_private_key.json

Address for REST used by API node

/Crown name specified at deploy time/shares/signer_private_key.json

Address to sign Nemesis (Genesis) block

/Crown name specified at deploy time/shares/new-cert/each node/CA/[*].pem

SSL certificate to encrypt communications between nodes

Note

The values in this parameter store are stored as the values to be created in the first block of the blockchain and are not called from the parameter store after deployment.
Therefore, if you want to delete data as security, etc., you can delete this data.
In addition, if there is a failure or a need for a new expansion, etc., the data can be recovered from this data.

2.2.5.2.6. loadbalanceNestStack

loadbalanceNestStack creates an ELB (load balancer).
If the parameter UseLoadBalancer is No, the stack (empty-elb) is called without creating the ELB.
The ELB distributes to the REST access port 3000 port of the API node.
The ELB Type uses NLB (Network Load Balancer) and the connection to the API node is configured to use the same node for a certain period of time for the same session by sticky session.

Note

ELBs are created with Network Load Balancer (NLB) only.
To understand the NLB, please refer to the following
https://docs.aws.amazon.com/ja_jp/elasticloadbalancing/latest/network/introduction.html
TargetGroupAttributes:
  - Key: stickiness.enabled
    Value: 'true'

NLBs can be parameterized for inward placement for private use only or outward placement for connection via the Internet.

../../_images/elb.jpg
Health checks are performed on the REST access port 3000 of the API node under the following conditions
Health checks are monitoring http://API-NODE:3000/chain/info for dead/ alive.
Properties:
  HealthCheckIntervalSeconds: 10 # 10秒間隔でチェックする Check at 10-second intervals.
  UnhealthyThresholdCount: 3 # 異常とみなす回数 Number of times considered abnormal
  HealthyThresholdCount: 3 # 正常とみなす回数 Number of times considered normal
  HealthCheckPath: /chain/info # ヘルスチェックをするURL URL for health check
  HealthCheckProtocol: HTTP
  Port: 3000 # ヘルスチェックポート health check port