Home PC Games Linux Windows Database Network Programming Server Mobile  
           
  Home \ Server \ Based Docker build stand-alone high-availability cluster Hadoop2.7.1 Spark1.7     - 12 novice Linux command must learn (Linux)

- How to defragment the hard disk in Linux (Linux)

- Python regular expressions: how to use regular expressions (Programming)

- OGG-01496 OGG-01031 Error Resolution (Database)

- Using PPA to install the lightweight theme software HotShots 2.1.0 under Ubuntu (Linux)

- Summary of Docker mounted directory (Server)

- RabbitMQ tutorial examples: RabbitMQ installation under Windows (Linux)

- C # C ++ Java interface type conversion (Programming)

- iptables allow only specific ip address to access the specified port (Linux)

- Vi syntax highlighting settings (Linux)

- C ++ stderr / stdout redirected to a file (Programming)

- Linux environment installation of rvm and ruby (Linux)

- J2EE Example of Filter (Programming)

- Linux Regular expressions grep and egrep (Linux)

- Spring AOP (Programming)

- Wildcards and special symbols usage comments under Linux (Linux)

- Linux more command Detailed (Linux)

- Oracle 11g maintenance partitions (Seven) - Modifying Real Attributes of Partitions (Database)

- Spring declarative transaction management (Programming)

- Linux 64-bit porting (Programming)

 
         
  Based Docker build stand-alone high-availability cluster Hadoop2.7.1 Spark1.7
     
  Add Date : 2018-11-21      
         
         
         
  Based Docker build stand-alone high-availability cluster Hadoop2.7.1 Spark1.7

Get Ubuntu mirror

sudo docker pull ubuntu

Place the local file are downloaded spark1.7 hadoop2.7.1 scala1.1 zookeeper3.4.6 jdk1.8 unzipped folder for mounting to the vessel

And create files in the folder

authorized_keys

hosts

In this case the use of directory / home / docker / config

Starting container

sudo docker run --name installspark -v / home / docker / config /: / config -it ubuntu: 14.04

installation
After starting in the container / config folder to see the installation files placed

Install jdk, scala:

vim ~ / .bashrc

Append:

/ Usr / sbin / sshd
cat / config / hosts> / etc / hosts
cat / config / authorized_keys> /root/.ssh/authorized_keys
export JAVA_HOME = / usr / lib / jvm / java-8-sun
export PATH = $ {JAVA_HOME} / bin: $ PATH
export HADOOP_HOME = / opt / hadoop
export PATH = $ {HADOOP_HOME} / bin: $ PATH
export SCALA_HOME = / opt / scala
export PATH = $ {SCALA_HOME} / bin: $ PATH
export SPARK_HOME = / opt / spark
export PATH = $ {SPARK_HOME} / bin: $ PATH

Copy spark / hadoop / zookeeper to under / opt

Installation hadoop:

Create a folder: / opt / hadoop / namenode / opt / hadoop / datanode / opt / hadoop / tmp / opt / hadoop / journal

root @ nn1: / opt / hadoop / etc / hadoop # vim hadoop-env.sh

modify:

export JAVA_HOME = / usr / lib / jvm / java-8-sun

root @ nn1: / opt / hadoop / etc / hadoop # vim core-site.xml

Add to:

< Property>
< Name> fs.defaultFS < / name>
< Value> hdfs: // ns1 < / value>
< / Property>
< Property>
< Name> hadoop.tmp.dir < / name>
< Value> / opt / hadoop / tmp < / value>
< / Property>
< Property>
< Name> ha.zookeeper.quorum < / name>
< Value> dnzk1: 2181, dnzk2: 2181, dnzk3: 2181 < / value>
< / Property>

root @ nn1: / opt / hadoop / etc / hadoop # vim hdfs-site.xml
Add to:

< Property>
 < Name> dfs.datanode.data.dir < / name>
 < Value> file: / opt / hadoop / datanode < / value>
 < / Property>
 < Property>
 < Name> dfs.namenode.name.dir < / name>
 < Value> file: / opt / hadoop / namenode < / value>
 < / Property>
< Property>
< Name> dfs.nameservices < / name>
< Value> ns1 < / value>
< / Property>
< Property>
< Name> dfs.ha.namenodes.ns1 < / name>
< Value> nn1, nn2 < / value>
< / Property>
< Property>
< Name> dfs.namenode.rpc-address.ns1.nn1 < / name>
< Value> nn1: 9000 < / value>
< / Property>
< Property>
< Name> dfs.namenode.http-address.ns1.nn1 < / name>
< Value> nn1: 50070 < / value>
< / Property>
< Property>
< Name> dfs.namenode.rpc-address.ns1.nn2 < / name>
< Value> nn2: 9000 < / value>
< / Property>
< Property>
< Name> dfs.namenode.http-address.ns1.nn2 < / name>
< Value> nn2: 50070 < / value>
< / Property>
< Property>
< Name> dfs.namenode.shared.edits.dir < / name>
< Value> qjournal: // dnzk1: 8485; dnzk2: 8485; dnzk3: 8485 / ns1 < / value>
< / Property>
< Property>
< Name> dfs.journalnode.edits.dir < / name>
< Value> / opt / hadoop / journal < / value>
< / Property>
< Property>
        < Name> dfs.journalnode.http-address < / name>
        < Value> 0.0.0.0:8480 < / value>
< / Property>
< Property>
        < Name> dfs.journalnode.rpc-address < / name>
        < Value> 0.0.0.0:8485 < / value>
< / Property>
< Property>
< Name> dfs.ha.automatic-failover.enabled < / name>
< Value> true < / value>
< / Property>
< Property>
< Name> dfs.client.failover.proxy.provider.ns1 < / name>
< Value> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider < / value>
< / Property>
< Property>
< Name> dfs.ha.fencing.methods < / name>
< Value>
        sshfence
        shell (/ bin / true)
< / Value>
< / Property>
< Property>
< Name> dfs.ha.fencing.ssh.private-key-files < / name>
< Value> /root/.ssh/id_rsa < / value>
< / Property>
< Property>
< Name> dfs.ha.fencing.ssh.connect-timeout < / name>
< Value> 30000 < / value>
< / Property>
< Property>
        < Name> dfs.permissions < / name>
        < Value> false < / value>
< / Property>

< Name > yarn.resourcemanager.store.class < / name> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore < / value> < / property> < property> < name> yarn.resourcemanager. zk-address < / name> < value> dnzk1: 2181, dnzk2: 2181, dnzk3: 2181 < / value> < / property> < property > < name > yarn.nodemanager.aux-services < / name> < value> mapreduce_shuffle < / value> < / property>

root @ nn1: / opt / hadoop # vim / opt / hadoop / etc / hadoop / slaves

Add to:

dnzk1
dnzk2
dnzk3

Install spark
root @ nn1: / opt / spark / conf # vim spark-env.sh
Add to:

export SPARK_MASTER_IP = nn1
export SPARK_WORKER_MEMORY = 256m
export JAVA_HOME = / usr / lib / jvm / java-8-sun
export SCALA_HOME = / opt / scala
export SPARK_HOME = / opt / spark
export HADOOP_CONF_DIR = / opt / hadoop / etc / hadoop
export SPARK_LIBRARY_PATH = $$ SPARK_HOME / lib
export SCALA_LIBRARY_PATH = $ SPARK_LIBRARY_PATH
export SPARK_WORKER_CORES = 1
export SPARK_WORKER_INSTANCES = 1
export SPARK_MASTER_PORT = 7077

root @ nn1: / opt / spark / conf # vim slaves

Add to:

Installation zookeeper

Create a folder / opt / zookeeper / tmp
Create the file / opt / zookeeper / tmp / myid
echo 1> / opt / zookeeper / tmp / myid
root @ nn1: / opt / zookeeper / conf # vim zoo.cfg

modify

dataDir = / opt / zookeeper / tmp
server.1 = dnzk1: 2888: 3888
server.2 = dnzk2: 2888: 3888
server.3 = dnzk3: 2888: 3888

Generate the key

ssh-keygen -t dsa

Append id_dsa.pub to host the / home / docker / config / authorized_keys file

root @ nn1: / opt / hadoop # cat ~ / .ssh / id_dsa.pub

carried out

sudo docker commit -m "namenode1" installspark ubuntu: ns1

Modify the local host / home / docker / config / hosts file
Add to

172.17.0.11 nn1
172.17.0.12 nn2
172.17.0.13 rm1
172.17.0.14 rm2
172.17.0.15 dnzk1
172.17.0.16 dnzk2
172.17.0.17 dnzk3

Start docker

sudo docker run --name dnzk1 -h dnzk1 --net = none -p 2185: 2181 -p 50075: 50070 -p 9005: 9000 -p 8485: 8485 -p 7075: 7077 -p 2885: 2888 -v / home / docker / config /: / config -it spark1_7-hadoop2_7_1-scala1_1: basic
sudo docker run --name dnzk2 -h dnzk2 --net = none -p 2186: 2181 -p 50076: 50070 -p 9006: 9000 -p 8486: 8485 -p 7076: 7077 -p 2886: 2888 -v / home / docker / config /: / config -it spark1_7-hadoop2_7_1-scala1_1: basic
sudo docker run --name dnzk3 -h dnzk3 --net = none -p 2186: 2181 -p 50076: 50070 -p 9006: 9000 -p 8486: 8485 -p 7076: 7077 -p 2887: 2888 -v / home / docker / config /: / config -it spark1_7-hadoop2_7_1-scala1_1: basic
sudo docker run --name nn1 -h nn1 --net = none -p 2181: 2181 -p 50071: 50070 -p 9001: 9000 -p 8481: 8485 -p 7071: 7077 -p 2881: 2888 -v / home / docker / config /: / config -it spark1_7-hadoop2_7_1-scala1_1: basic
sudo docker run --name nn2 -h nn2 --net = none -p 2182: 2181 -p 50072: 50070 -p 9002: 9000 -p 8482: 8485 -p 7072: 7077 -p 2882: 2888 -v / home / docker / config /: / config -it spark1_7-hadoop2_7_1-scala1_1: basic
sudo docker run --name rm1 -h rm1 --net = none -p 2183: 2181 -p 50073: 50070 -p 9003: 9000 -p 8483: 8485 -p 7073: 7077 -p 2883: 2888 -v / home / docker / config /: / config -it spark1_7-hadoop2_7_1-scala1_1: basic
sudo docker run --name rm2 -h rm2 --net = none -p 2184: 2181 -p 50074: 50070 -p 9004: 9000 -p 8484: 8485 -p 7074: 7077 -p 2884: 2888 -v / home / docker / config /: / config -it spark1_7-hadoop2_7_1-scala1_1: basic

dnzk2 (Executive echo 2> / opt / zookeeper / tmp / myid), dnzk2 (execute echo 3> / opt / zookeeper / tmp / myid)

Network Configuration

sudo pipework docker0 -i eth0 nn1 172.17.0.11/16
sudo pipework docker0 -i eth0 nn2 172.17.0.12/16
sudo pipework docker0 -i eth0 rm1 172.17.0.13/16
sudo pipework docker0 -i eth0 rm2 172.17.0.14/16
sudo pipework docker0 -i eth0 dnzk1 172.17.0.15/16
sudo pipework docker0 -i eth0 dnzk2 172.17.0.16/16
sudo pipework docker0 -i eth0 dnzk3 172.17.0.17/16

Start hadoop cluster
Start zookeeper and hadoop journal on dnzk1 / dnzk2 / dnzk3

/opt/zookeeper/bin/zkServer.sh start
/opt/hadoop/sbin/hadoop-daemon.sh start journalnode

Formatting zookeeper start and format hadoop on nn1

 

/ Opt / hadoop / bin / hdfs namenode -format

/ Opt / hadoop / bin / hdfs namenode -format

scp -r / opt / hadoop / namenode / nn2: / opt / hadoop /

or

/ Opt / hadoop / bin / hdfs namenode -bootstrapStandby

/ Opt / hadoop / bin / hdfs zkfc -formatZK

/opt/hadoop/sbin/start-dfs.sh

Start yarn on rm1

/opt/hadoop/sbin/start-yarn.sh

Started on rm2

/opt/hadoop/sbin/yarn-daemon.sh start resourcemanager

Start spark

/opt/spark/sbin/start-all.sh

Views:
http://172.17.0.11:50070 (active)
http://172.17.0.12:50070(standby)

After starting the cluster service in

nn1 172.17.0.11 jdk, hadoop NameNode, DFSZKFailoverController (zkfc)
nn2 172.17.0.12 jdk, hadoop NameNode, DFSZKFailoverController (zkfc)
rm1 172.17.0.13 jdk, hadoop ResourceManager
rm2 172.17.0.14 jdk, hadoop ResourceManager
dnzk1 172.17.0.15 jdk, hadoop, zookeeper DataNode, NodeManager, JournalNode, QuorumPeerMain
dnzk2 172.17.0.16 jdk, hadoop, zookeeper DataNode, NodeManager, JournalNode, QuorumPeerMain
dnzk3 172.17.0.17 jdk, hadoop, zookeeper DataNode, NodeManager, JournalNode, QuorumPeerMain
     
         
         
         
  More:      
 
- Node.js developers must know four JavaScript concepts (Programming)
- C ++ in the elimination Wunused (Programming)
- MySQL script incremental backups (innobackupex) (Database)
- Linux how to handle file names that contain spaces and special characters (Linux)
- Hadoop upload files error solved (Server)
- About Git (Linux)
- Oracle 12C RAC on temporary table space Enlighten (Database)
- Linux server data backup (Server)
- Share Java-based multithreading file case (Programming)
- Will Laravel become the most successful PHP framework? (Programming)
- How to fix apt-get update can not add a new CD-ROM error (Linux)
- Ubuntu 12.04 LTS installation configuration JDK1.6.0_45 (Linux)
- Install minimize RHEL / CentOS 7 things to do (Linux)
- grep regular expression (Linux)
- grep command output highlighted word (Linux)
- Summarize small problems encountered in the use Lua (Programming)
- Linux server Php injection prevention (Linux)
- Write perfect printf (Programming)
- Getting Started with Linux: Learn how to upgrade Docker in Ubuntu (Server)
- Linux Command study manual - GPG command (Linux)
     
           
     
  CopyRight 2002-2022 newfreesoft.com, All Rights Reserved.