大数据环境搭建.docxVIP

本文记录zookeeper+hadoop+hbase+kafka+storm集群的搭建过程。 准备工作 硬件环境 3台物理主机 91 – dc01 92 – dc02 93 – dc03 系统全部为Ubuntu16.04 软件环境 1) jdk1.8.0_121 2) zookeeper-3.4.10 3) hadoop-2.6.5 4) hbase-1.2.6 5) kafka_2.11- 6) apache-storm-1.1.0 jdk的安装在此不做赘述，有疑问可自行百度。 设置主机名 三台机器都需设置，后面所有操作都通过主机名连接。 sudo gedit /etc/sysconfig/network ip为91的机器修改为如下内容： NETWORKING=yes HOSTNAME=dc01 其他两台机器分别设置为dc02和dc03。 修改host文件 sudo gedit /etc/hosts 在末尾追加如下内容： 91 dc01 92 dc02 93 dc03 三台机器做相同操作。 设置ssh免密码登录 以dc01为例，执行以下命令 sssh-keygen -t rsa # 连续三次回车,即在本地生成了公钥和私钥,不设置密码 ssh-copy-id dc01 # 按提示输入dc01的密码 ssh-copy-id dc02 # 按提示输入dc02的密码 ssh-copy-id dc03 # 按提示输入dc03的密码 dc02和dc03重复dc01的操作，设置完成后任意两台机器之间不需要密码可直接登录，例如从dc01登录到dc03,只需在dc01输入”ssh dc03”即可。 开始安装 所有软件安装目录为/usr/local/ 安装zookeeper 下载zookeeper安装文件zookeeper-3.4.10.tar.gz到/usr/local/下 执行以下命令： #解压文件 tar -zxvf zookeeper-3.4.10.tar.gz #修改文件夹名称 mv zookeeper-3.4.10 zookeeper cd zookeeper/ #zookeeper配置文件名为zoo.cfg cp conf/zoo_sample.cfg conf/zoo.cfg 7 编辑zoo.cfg vim conf/zoo.cfg 修改后文件内容如下： # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/usr/local/zookeeper/data dataLogDir=/usr/local/zookeeper/log # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # /doc/current/zookeeperAdmin.html#sc_maintenance # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to 0 to disable auto purge feature #autopurge.purgeInterval=1 server.1=dc01:2888:3888 server.2=dc02:2888:3888 s