[AWS/EMR] AMI v4.x 대의 포트 변경

AWS의 EMR 서비스의 AMI 버전이 4.x 대로 변경되면서

하둡의 기본 포트가 변경되었다.

 

기존과 같이 사용하면 오류가 발생하므로 수정해야 한다.

 

Service Ports

In Amazon EMR AMI versions 2.x and 3.x, some services used custom ports. emr-4.0.0 hosts these services on open source community defaults in most cases.

Changes in Port Settings

Setting	AMI Version 3.x	Release Label emr-4.x
fs.default.name	hdfs://emrDeterminedIP:9000	default (hdfs://emrDeterminedIP:8020)
dfs.datanode.address	0.0.0.0:9200	default (0.0.0.0:50010)
dfs.datanode.http.address	0.0.0.0:9102	default (0.0.0.0:50075)
dfs.datanode.https.address	0.0.0.0:9402	default (0.0.0.0:50475)
dfs.datanode.ipc.address	0.0.0.0:9201	default (0.0.0.0:50020)
dfs.http.address	0.0.0.0:9101	default (0.0.0.0:50070)
dfs.https.address	0.0.0.0:9202	default (0.0.0.0:50470)
dfs.secondary.http.address	0.0.0.0:9104	default (0.0.0.0:50090)
yarn.nodemanager.address	0.0.0.0:9103	default (${yarn.nodemanager.hostname}:0)
yarn.nodemanager.localizer.address	0.0.0.0:9033	default (${yarn.nodemanager.hostname}:8040)
yarn.nodemanager.webapp.address	0.0.0.0:9035	default (${yarn.nodemanager.hostname}:8042)
yarn.resourcemanager.address	emrDeterminedIP:9022	default (${yarn.resourcemanager.hostname}:8032)
yarn.resourcemanager.admin.address	emrDeterminedIP:9025	default (${yarn.resourcemanager.hostname}:8033)
yarn.resourcemanager.resource-tracker.address	emrDeterminedIP:9023	default (${yarn.resourcemanager.hostname}:8031)
yarn.resourcemanager.scheduler.address	emrDeterminedIP:9024	default (${yarn.resourcemanager.hostname}:8030)
yarn.resourcemanager.webapp.address	0.0.0.0:9026	default (${yarn.resourcemanager.hostname}:8088)
yarn.web-proxy.address	emrDeterminedIP:9046	default (no-value)
yarn.resourcemanager.hostname	0.0.0.0 (default)	emrDeterminedIP

 

Note

The term emrDeterminedIP is an IP address that is generated by the Amazon EMR control plane. In the newer version, this convention has been eliminated except for the yarn.resourcemanager.hostname and fs.default.name settings.

 

<참고>

http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-release-differences.html