Installation step:
1) JDK - Hadoop is written in Java, the program does not install the Java virtual machine how to run the Hadoop;
2) create a user specifically for linux and run hadoop perform tasks (such as map and reduce task), just under the windows service accounts, and authorizing him to have access to running JDK directory permissions, so that he can perform the java virtual machine. Finally, this account is used to run the bin .start_all.sh start all services hadoop, that account must have run him, he must have sufficient permissions. Furthermore, we need to configure the environment variable personal account, the home directory of the Java virtual machine environment variables one address, or run behind hadoop task is certainly being given, could not find the java command to run in the current environment variables.
2) modify / etc / hosts file and / etc / hostname - clustered mode machine needs to communicate with each other, by the IP address. And usually we use a host name, and then to the target host communication by mapping IP addresses and host names. Then we naturally have to change the / etc / hosts file and / etc / hostname, the former is the mapping between the host and the IP address, which is the local host names;
4) SSH - between the cluster machine to access each other's resources need to establish a connection, using the SSH protocol is a secure channel for data exchange, moreover also need to use the SSH authorized certificate approach to achieve the target host password Free;
4) and it is to install a hadoop. We need to modify a few configuration files. hadoop configuration file there are many.
read-only configuration file: src / core / core-default.xml, src / hdfs / hdfs-default.xml, src / mapred / mapred-default.xml, conf / mapred-queues.xml.
Location settings: conf / core-site.xml, conf / hdfs-site.xml, conf / mapred-site.xml, conf / mapred-queues.xml. Such documents are generally used to configure hadoop some core features such hdfs mapred and directory information.
environment configuration: conf / Hadoop-env.sh
Back to the topic, since the core hadoop is hdfs and mapreduce, you at least want to configure hdfs of namenode directory location, datanode directory location, mapreduce the job tracker and task tracker port number, the system directory and the local directory to-peer communication Wait.
The master and slave on the machine are the same.
5) After the configuration is to format the hdfs. Formatting hdfs system on the master machine.
6) End hdfs formatting system start all hdfs process.
built environment using VMWare Workstation 12. Here with three Linux virtual machine: master, slave1, slave2.
Specific configuration:
master
slave1
slave2
OS
Ubuntu 14.04 LTE x64
Ubuntu 14.04 LTE x64
Ubuntu 14.04 LTE x64
memory
1GB
1GB
1GB
hard drive space
20GB
20GB
20GB
processors
2
2
2
IP Address
192.168.2.110
192.168.2.111
192.168.2.112
Roles
NameNode DataNode JobTracker TaskTracker SecondaryNameNode
DataNode TaskTracker
DataNode TaskTracker
hadoop directory
/ Opt / hadoop
/ Opt / hadoop
/ Opt / hadoop
jdk version
JDK 1.8
JDK 1.8
JDK 1.8
be installed on the master machine JDK and SSH
1. JDK installation
jerry @ ubuntu: / run / network $ scp jerry@192.168.2.100: /home/jerry/Download/jdk-8u65-linux-x64.tar.gz ~ The authenticity of host '192.168.2.100 (192.168.2.100) 'can not be established ECDSA key fingerprint is da:. b7: c3: 2a: ea: a2: 76: 4c: c3: c1: 68: ca: 0e: c2: ea: 92. Are you sure you want to continue connecting (yes / no) yes Warning:? Permanently added '192.168.2.100' (ECDSA) to the list of known hosts jerry @. 192.168.2.100's password: scp: /home/jerry/Download/jdk-8u65-linux-x64.tar.gz: No such file or directory jerry @ ubuntu: / run / network $ scp jerry@192.168.2.100: /home/jerry/Downloads/jdk-8u65-linux-x64.tar.gz ~ jerry@192.168.2.100's password: jdk-8u65-linux-x64 .tar.gz 100% 173MB 21.6MB / s 00: 08 jerry @ ubuntu: / run / network $ cd ~ jerry @ ubuntu: ~ $ ls Desktop Downloads jdk-8u65-linux-x64.tar.gz Pictures Templates Documents examples.desktop Music Public Videos jerry @ ubuntu:. ~ $ cd jerry @ ubuntu: ~ $ cd / jerry @ ubuntu: / $ ls bin dev initrd.img lost + found opt run sys var boot etc lib media proc sbin tmp vmlinuz cdrom home lib64 mnt root srv usr jerry @ ubuntu: / $ sudo mkdir jvm [sudo] password for jerry: jerry @ ubuntu: / $ rm jvm / rm: can not remove ‘ jvm / ’: Is a directory jerry @ ubuntu: / $ rm -d jvm / rm: remove write-protected directory ‘ jvm / ’? y rm: can not remove ‘ jvm / ’: Permission denied jerry @ ubuntu: / $ ls bin dev initrd.img lib64 mnt root srv usr boot etc jvm lost + found opt run sys var cdrom home lib media proc sbin tmp vmlinuz jerry @ ubuntu: / $ sudo mkdir / usr / lib / jvm jerry @ ubuntu: / $ cd ~ jerry @ ubuntu: ~ $ sudo tar zxf ./jdk-8u65-linux -x64.tar.gz -C / usr / lib / jvm / jerry @ ubuntu: ~ $ cd / usr / lib / jvm / jerry @ ubuntu: / usr / lib / jvm $ sudo mv jdk1.8.0_65 java jerry @ ubuntu: / usr / lib / jvm $ cd java / jerry @ ubuntu: / usr / lib / jvm / java $ sudo vim ~ / .bashrc jerry @ ubuntu: / usr / lib / jvm / java $ tail -n 4 ~ / .bashrc export JAVA_HOME = / usr / lib / jvm / java export JRE_HOME = $ {JAVA_HOME} / jre export CLASSPATH =: $ {JAVA_HOME} / lib:. $ {JRE_HOME} / lib export PATH = $ {JAVA_HOME} / bin: $ PATH jerry @ ubuntu: / usr / lib / jvm / java $ java -version The program 'java' can be found in the following packages: * default-jre * gcj-4.8-jre-headless * openjdk-7-jre-headless * gcj-4.6-jre-headless * openjdk-6-jre-headless Try: sudo apt -get install < selected package > jerry @ ubuntu: / usr / lib / jvm / java $ sudo update-alternatives --install / usr / bin / java java / usr / lib / jvm / java / bin / java 300 update-alternatives: using / usr / lib / jvm / java / bin / java to provide / usr / bin / java (java) in auto mode jerry @ ubuntu: / usr / lib / jvm / java $ sudo update-alternatives --install / usr / bin / javac javac / usr / lib / jvm / java / bin / javac 300 update-alternatives: using / usr / lib / jvm / java / bin / javac to provide / usr / bin / javac (javac) in auto mode jerry @ ubuntu: / usr / lib / jvm / java $ sudo update-alternatives --install / usr / bin / jar jar / usr / lib / jvm / java / bin / jar 300 update-alternatives: using / usr / lib / jvm / java / bin / jar to provide / usr / bin / jar (jar) in auto mode jerry @ubuntu: / usr / lib / jvm / java $ sudo update-alternatives --install / usr / bin / javah javah / usr / lib / jvm / java / bin / javah 300 update-alternatives: using / usr / lib / jvm / java / bin / javah to provide / usr / bin / javah (javah) in auto mode jerry @ ubuntu: / usr / lib / jvm / java $ sudo update-alternatives --install / usr / bin / javap javap / usr / lib / jvm / java / bin / javap 300 update-alternatives: using / usr / lib / jvm / java / bin / javap to provide / usr / bin / javap (javap) in auto mode jerry @ ubuntu: / usr / lib / jvm / java $ sudo update-alternatives --config java There is only one alternative in link group java (providing / usr / bin / java) : / usr / lib / jvm / java / bin / java Nothing to configure jerry @ ubuntu:. / usr / lib / jvm / java $ java -version java version "1.8. 0_65 " Java (TM) SE Runtime Environment (build 1.8.0_65-b17) Java HotSpot (TM) 64-Bit Server VM (build 25.65-b01, mixed mode) jerry @ ubuntu: / usr / lib / jvm / java $
. 2 hadoop add users and groups
jerry @ ubuntu: / usr / lib / jvm / java $ sudo groupadd hadoop [sudo] password for jerry: jerry @ ubuntu: / usr / lib / jvm / java $ useradd hadoop - g hadoop useradd: Permission denied useradd:. can not lock / etc / passwd; try again later jerry @ ubuntu:. / usr / lib / jvm / java $ sudo useradd hadoop -g hadoop jerry @ ubuntu: / usr / lib / jvm / java $
3. Modify / etc / hosts file configuration master, slave1, slave2 host IP address and host name mappings. Modify / etc / hostname file to configure local computer name.
jerry @ ubuntu: ~ $ cat / etc / hosts 127.0.0.1 localhost 127.0.1.1 master
# The following lines are desirable for IPv6 capable hosts :: 1 ip6-localhost ip6-loopback fe00 :: 0 ip6-localnet ff00 :: 0 ip6- mcastprefix ff02 :: 1 ip6-allnodes ff02 :: 2 ip6-allrouters master 192.168.2.110 slave1 192.168.2.111 slave2 192.168.2.112 jerry @ ubuntu: ~ $ cat / etc / hostname master jerry @ ubuntu: ~ $
4. Installing SSH
jerry @ ubuntu: / usr / lib / jvm / java $ sudo apt-get install ssh Reading package lists ... Done Building dependency tree Reading state information ... Done The following extra packages will be installed: libck-connector0 ncurses-term openssh-client openssh-server openssh-sftp-server ssh-import -id Suggested packages: libpam-ssh keychain monkeysphere rssh molly-guard The following NEW packages will be installed: libck-connector0 ncurses-term openssh- server openssh-sftp-server ssh ssh-import-id The following packages will be upgraded: openssh-client 1 upgraded, 6 newly installed, 0 to remove and 239 not upgraded. Need to get 617 kB / 1,181 kB of archives. After this operation, 3,450 kB of additional disk space will be used. Do you want to continue? [Y / n] Y Get: 1 http://us.archive.ubuntu.com/ubuntu/ trusty / main libck-connector0 amd64 0.4.5-3.1ubuntu2 [10.5 kB] Get: 2 http://us.archive.ubuntu.com/ubuntu/ trusty / main ncurses-term all 5.9 + 20140118-1ubuntu1 [243 kB] Get: 3 http://us.archive.ubuntu.com/ ubuntu / trusty-updates / main openssh-sftp-server amd64 1: 6.6p1-2ubuntu2.3 [34.1 kB] Get: 4 http://us.archive.ubuntu.com/ubuntu/ trusty-updates / main openssh-server amd64 1: 6.6p1-2ubuntu2.3 [319 kB] Get: 5 http://us.archive.ubuntu.com/ubuntu/ trusty-updates / main ssh all 1: 6.6p1- 2ubuntu2.3 [1,114 B] Get: 6 http://us.archive.ubuntu.com/ubuntu/ trusty / main ssh-import-id all 3.21-0ubuntu1 [9,624 B] fetched 617 kB in 16s (37.3 kB / s) Preconfiguring packages ... Selecting previously unselected package libck-connector0: amd64 (Reading database ... 166519 files and directories currently installed.) . Preparing to unpack ... / libck-connector0_0.4.5-3.1ubuntu2_amd64.deb ... Unpacking libck-connector0: amd64 (0.4.5-3.1ubuntu2) ... Preparing to unpack .. ./openssh-client_1%3a6.6p1-2ubuntu2.3_amd64.deb ... Unpacking openssh-client (1: 6.6p1-2ubuntu2.3) over (1: 6.6p1-2ubuntu2) ... Selecting previously unselected package ncurses-term. Preparing to unpack ... / ncurses-term_5.9 + 20140118-1ubuntu1_all.deb ... Unpacking ncurses-term (5.9 + 20140118-1ubuntu1) ... Selecting previously unselected package openssh-sftp-server. Preparing to unpack ... / openssh-sftp-server_1% 3a6.6p1-2ubuntu2.3_amd64.deb ... Unpacking openssh-sftp-server. (1: 6.6p1-2ubuntu2.3) ... Selecting previously unselected package openssh-server Preparing to unpack ... / openssh-server_1% 3a6.6p1- 2ubuntu2.3_amd64.deb ... Unpacking openssh-server (1: 6.6p1-2ubuntu2.3) ... Selecting previously unselected package ssh Preparing to unpack ... /. ssh_1% 3a6.6p1-2ubuntu2.3_all.deb ... Unpacking ssh (1: 6.6p1-2ubuntu2.3) ... Selecting previously unselected package ssh-import-id Preparing to unpack ... / ssh-import-id_3.21-0ubuntu1_all.deb ... Unpacking ssh-import-id (3.21-0ubuntu1) ... Processing triggers for man-db (2.6.7.1-1ubuntu1) ... Processing triggers for ureadahead (0.100.0-16) ... ureadahead will be reprofiled on next reboot Processing triggers for ufw (0.34 ~ rc-0ubuntu2) ... Setting up libck-connector0: amd64 (0.4.5-3.1ubuntu2) ... Setting up openssh-client (1: 6.6p1-2ubuntu2.3) .. . Setting up ncurses-term (5.9 + 20140118-1ubuntu1) ... Setting up openssh-sftp-server (1: 6.6p1-2ubuntu2.3) ... Setting up openssh-server (1: 6.6p1-2ubuntu2.3) ... Creating SSH2 RSA key; this may take some time ... Creating SSH2 DSA key; this may take some time ... Creating SSH2 ECDSA key; this may take some time ... Creating SSH2 ED25519 key; this may take some time ... ssh start / running, process 7611 Setting up ssh-import-id (3.21-0ubuntu1) ... Processing triggers for ureadahead (0.100.0-16) ... Processing triggers for ufw (0.34 ~ rc-0ubuntu2) ... Setting up ssh (1: 6.6p1-2ubuntu2.3) ... Processing triggers for libc-bin (2.19-0ubuntu6.6) ...
5. Decompression hadoop archive locally, modify the configuration file
jerry @ ubuntu: ~ $ sudo tar zxf hadoop-1.2.1.tar.gz -C / opt / [sudo] password for jerry: jerry @ ubuntu: ~ $ cd / opt /hadoop-1.2.1/ jerry @ ubuntu: /opt/hadoop-1.2.1$ ls bin hadoop-ant-1.2.1.jar ivy sbin build.xml hadoop-client-1.2.1.jar ivy.xml share c ++ hadoop-core-1.2.1.jar lib src CHANGES .txt hadoop-examples-1.2.1.jar libexec webapps conf hadoop-minicluster-1.2.1.jar LICENSE.txt contrib hadoop-test-1.2.1.jar NOTICE.txt docs hadoop-tools-1.2.1.jar README.txt jerry @ ubuntu: /opt/hadoop-1.2.1$ cd conf / jerry @ ubuntu: /opt/hadoop-1.2.1/conf$ ls capacity-scheduler.xml hadoop-policy.xml slaves configuration.xsl hdfs-site.xml ssl-client.xml.example core-site.xml log4j.properties ssl-server.xml.example fair-scheduler.xml mapred-queue-acls.xml taskcontroller.cfg hadoop-env.sh mapred-site .xml task-log4j.properties hadoop-metrics2.properties masters jerry @ ubuntu: /opt/hadoop-1.2.1/conf$ vim hadoop-env.sh jerry @ ubuntu: /opt/hadoop-1.2.1/conf$ sudo vim hadoop-env.sh jerry @ ubuntu: /opt/hadoop-1.2.1/conf$ tail -n 1 hadoop -env.sh export JAVA_HOME = / usr / lib / jvm / java jerry @ ubuntu: /opt/hadoop-1.2.1/conf$ sudo vim core-site.xml jerry @ubuntu: /opt/hadoop-1.2.1/conf$ cat core-site.xml < xml version = "1.0" > < xml-stylesheet type = "text? ? / xsl "href =" configuration.xsl ">
< -! Put site-specific property overrides in this file - >.
< configuration > < property > < name > fs.default.name < / name > < value > hdfs: // master: 9000 < / value > < / property > < property > < name > hadoop.tmp.dir < / name > < value > / hadoop / hadooptmp < / value > < / property > < / configuration > jerry @ ubuntu: /opt/hadoop-1.2.1/conf$ sudo vim hdfs-site.xml jerry @ ubuntu :? /opt/hadoop-1.2.1/conf$ cat hdfs-site.xml < xml version = "1.0" > < xml-stylesheet type = "text / xsl "href =" configuration.xsl ">?
< -! Put site-specific property overrides in this file - >.
< configuration > < property > < name > dfs.replication < / name > < value > 3 < / value > < / property > < property > < name > dfs.name.dir < / name > < value > / hadoop / hdfs / name < / value > < / property > < property > < name > dfs.data.dir < / name > < value > / hadoop / hdfs / data < / value > < / property > < / configuration > jerry @ ubuntu: / opt / hadoop- 1.2.1 / conf $ sudo vim mapred- jerry @ ubuntu: /opt/hadoop-1.2.1/conf$ sudo vim mapred-site.xml jerry @ ubuntu: /opt/hadoop-1.2 .1 / conf $ cat mapred-site.xml xml version = "1.0" >? xml-stylesheet type = "text / xsl" href = "configuration.xsl ? ">
< -! Put site-specific property overrides in this file - >.
< configuration > < property > < name > mapred.job.tracker < / name > < value > master: 9001 < / value > < / property > < / configuration > jerry @ ubuntu: /opt/hadoop-1.2.1/conf$ sudo vim mapred-site.xml jerry @ ubuntu: / opt / hadoop-1.2.1 / conf $ cat mapred-site.xml xml version = "1.0" >? xml-stylesheet type = "text / xsl" href = "configuration.xsl" >?
< -! Put site-specific property overrides in this file - >.
< configuration > < property > < name > mapred.job.tracker < / name > < value > master: 9001 < / value > < / property > < property > < name > mapred.system.dir < / name > < value > / hadoop / mapred / mapred_system < / value > < / property > < property > < name > mapred.local. dir < / name > < value > / hadoop / mapred / mapred_local < / value > < / property > < / configuration > jerry @ ubuntu: /opt/hadoop-1.2.1/conf$ ls capacity-scheduler.xml hadoop-policy.xml slaves configuration.xsl hdfs-site.xml ssl-client.xml.example core-site.xml log4j.properties ssl-server.xml.example fair-scheduler.xml mapred-queue-acls.xml taskcontroller.cfg hadoop -env.sh mapred-site.xml task-log4j.properties hadoop-metrics2.properties masters jerry @ubuntu: /opt/hadoop-1.2.1/conf$ sudo vim masters jerry @ ubuntu: /opt/hadoop-1.2.1/conf$ sudo vim slaves jerry @ ubuntu: / opt / hadoop-1.2.1 / conf $ cat masters master jerry @ ubuntu: /opt/hadoop-1.2.1/conf$ cat slaves master slave1 slave2 jerry @ ubuntu: /opt/hadoop-1.2.1/conf $
.ssh file is created under the master, slave1 and slave2 of hadoop user directory folder
jerry @ slave1: ~ $ sudo mkdir -p / hadoop / hdfs / name [sudo] password for jerry: jerry @ slave1: ~ $ sudo mkdir -p / hadoop / hdfs / data jerry @ slave1: ~ $ sudo mkdir -p / hadoop / mapred / mapred_system jerry @ slave1: ~ $ sudo mkdir -p / hadoop / mapred / mapred_local jerry @ slave1: ~ $ sudo mkdir -p / hadoop / hadooptmp jerry @ slave1: ~ $ chown -R hadoop: hadoop / hadoop / chown: changing ownership of ‘ / hadoop / mapred / mapred_local ’: Operation not permitted chown: changing ownership of ‘ / hadoop / mapred / mapred_system ’: Operation not permitted chown: changing ownership of ‘ / hadoop / mapred ’: Operation not permitted chown: changing ownership of ‘ / hadoop / hdfs / data ’: Operation not permitted chown: changing ownership of ‘ / hadoop / hdfs / name ’: Operation not permitted chown: changing ownership of ‘ / hadoop / hdfs ’: Operation not permitted chown: changing ownership of ‘ / hadoop / hadooptmp ’: Operation not permitted chown: changing ownership of ‘ / hadoop / ’: Operation not permitted jerry @ slave1: ~ $ sudo chown -R hadoop: hadoop / hadoop / jerry @ slave1: ~ $
On the master secret key survival ssh then copied to the .ssh directory hadoop user's home directory on slave1 and slave2 under
hadoop @ master: ~ $ sudo mkdir ~ / .ssh [sudo] password for hadoop: hadoop @ master: ~ $ sudo chown -R hadoop: hadoop ~ / .ssh hadoop @ master: ~ $ ssh-keygen -t rsa -P '' -f ~ / .ssh / id_rsa Generating public / private rsa key pair Your identification has been saved in / home. . /hadoop/.ssh/id_rsa Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub The key fingerprint is:. 36: 99: e0: e9: aa: b8: c3: c6: cd: 97: 59: 9f: 49: 87: 0d: bf hadoop @ master The key's randomart image is: + - [RSA 2048] - --- + | | | | | . | | o + |. | o S = | | . o + + | | o o . = o + | . | = o = + E | | + o..o | + ---------- ------- + hadoop @ master: ~ / .ssh $ scp authorized_keys slave1: /home/hadoop/.ssh The authenticity of host 'slave1 (192.168.2.111)' can ' t be established ECDSA key fingerprint is 48:. 93: 30: 0d: bb: 3a: 85: da: 46: 3f: 75: 76: 3e: b7: 42:. 6a Are you sure you want to continue connecting (yes / no) yes Warning:? Permanently added 'slave1,192.168.2.111' (ECDSA) to the list of known hosts hadoop @ slave1's password:. scp: /home/hadoop/.ssh/authorized_keys: Permission denied hadoop @ master: ~ / .ssh $ scp authorized_keys slave1: /home/hadoop/.ssh hadoop @ slave1's password: authorized_keys 100% 395 0.4KB / s 00: 00 hadoop @ master: ~ / .ssh $ scp authorized_keys slave2: /home/hadoop/.ssh The authenticity of host 'slave2 (192.168.2.112)' can not be established ECDSA key fingerprint is 48: 93: 30: 0d: bb: 3a: 85: da: 46: 3f: 75: 76: 3e: b7: 42: 6a Are you sure you want to continue connecting (. ?. yes / no) yes Warning: Permanently added 'slave2,192.168.2.112' (ECDSA) to the list of known hosts hadoop @ slave2's password: authorized_keys 100% 395 0.4KB / s 00: 00 hadoop @ master : ~ / .ssh $
then test again at whether ssh connection when you need to type a password. The result is not required.
hadoop @ master: ~ / .ssh $ ssh slave1 Welcome to Ubuntu 14.04.3 LTS (GNU / Linux 3.19.0-42-generic x86_64)
* Documentation: https://help.ubuntu.com/
0 packages can be updated. 0 updates are security updates.
$ ls -al total 239436 drwxr-xr-x 12 hadoop hadoop 4096 Jan . 9 22:37 drwxr-xr-x 4 root root 4096 Jan 9 22:25 .. drwx ------ 2 hadoop hadoop 4096 Jan 9 22:37 .cache drwxr-xr-x 2 hadoop hadoop 4096 Dec 27 04:19 Desktop drwxr-xr-x 2 hadoop hadoop 4096 Dec 27 04:19 Documents drwxr-xr-x 2 hadoop hadoop 4096 Dec 27 04:19 Downloads -rw-r - r - 1 hadoop hadoop 8980 Dec 27 03 : 27 examples.desktop -rw-rw-r - 1 hadoop hadoop 63851630 Dec 30 05:25 hadoop-1.2.1.tar.gz -rw-rw-r-- 1 hadoop hadoop 181260798 Dec 27 06:49 jdk-8u65-linux-x64.tar.gz drwxr-xr-x 2 hadoop hadoop 4096 Dec 27 04:19 Music drwxr-xr-x 2 hadoop hadoop 4096 Dec 27 04:19 Pictures drwxr-xr-x 2 hadoop hadoop 4096 Dec 27 04:19 Public drwxr-xr-x 2 hadoop hadoop 4096 Jan 9 22:39 .ssh drwxr-xr-x 2 hadoop hadoop 4096 Dec 27 04:19 Templates drwxr-xr-x 2 hadoop hadoop 4096 Dec 27 04:19 Videos $
hadoop @ master: ~ / .ssh $ ssh slave2 Welcome to Ubuntu 14.04.3 LTS (GNU / Linux 3.19.0-42-generic x86_64)
* Documentation: https://help.ubuntu.com/
27 packages can be updated. 23 updates are security updates.
$ ls -l total 239424 drwxr-xr-x 2 hadoop hadoop 4096 Dec 27 04:19 Desktop drwxr-xr-x 2 hadoop hadoop 4096 Dec 27 04:19 Documents drwxr-xr-x 2 hadoop hadoop 4096 Dec 27 04:19 Downloads -rw-r - r-- 1 hadoop hadoop 8980 Dec 27 03:27 examples.desktop -rw-rw-r-- 1 hadoop hadoop 63851630 Dec 30 05:25 hadoop-1.2.1.tar.gz -rw-rw-r-- 1 hadoop hadoop 181260798 Dec 27 06:49 jdk-8u65-linux-x64.tar.gz drwxr-xr-x 2 hadoop hadoop 4096 Dec 27 04: 19 Music drwxr-xr-x 2 hadoop hadoop 4096 Dec 27 04:19 Pictures drwxr-xr-x 2 hadoop hadoop 4096 Dec 27 04:19 Public drwxr-xr-x 2 hadoop hadoop 4096 Dec 27 04:19 Templates drwxr-xr-x 2 hadoop hadoop 4096 Dec 27 04:19 Videos $ exit Connection to slave2 closed hadoop @ master:. ~ / .ssh $
Format hdfs of namenode
hadoop @ master: /opt/hadoop-1.2.1/bin$ ./hadoop namenode -format16 / 01/09 23:33:18 INFO namenode.NameNode: STARTUP_MSG: / ***** ************************************************** ***** STARTUP_MSG: Starting NameNode STARTUP_MSG: host = master / 127.0.1.1 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 1.2.1 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013 STARTUP_MSG: java = 1.8.0_65 ************************* *********************************** / Re-format filesystem in / hadoop / hdfs / ? name (Y or N) Y 16/01/09 23:34:09 INFO util.GSet: Computing capacity for map BlocksMap 16/01/09 23:34:09 INFO util.GSet : VM type = 64-bit 16/01/09 23:34:09 INFO util.GSet: 2.0% max memory = 1013645312 16/01/09 23:34: 09 INFO util.GSet: capacity = 2 ^ 21 = 2097152 entries 16/01/09 23:34:09 INFO util.GSet: recommended = 2097152, actual = 2097152 16 / 01/09 23:34:10 INFO namenode.FSNamesystem: fsOwner = hadoop 16/01/09 23:34:10 INFO namenode.FSNamesystem: supergroup = supergroup 16/01/09 23: 34:10 INFO namenode.FSNamesystem: isPermissionEnabled = true 16/01/09 23:34:10 INFO namenode.FSNamesystem: dfs.block.invalidate.limit = 100 16/01/09 23: 34:10 INFO namenode.FSNamesystem: isAccessTokenEnabled = false accessKeyUpdateInterval = 0 min (s), accessTokenLifetime = 0 min (s) 16/01/09 23:34:10 INFO namenode.FSEditLog: dfs.namenode.edits .toleration.length = 0 16/01/09 23:34:10 INFO namenode.NameNode: Caching file names occuring more than 10 times 16/01/09 23:34:11 INFO common. Storage: Image file / hadoop / hdfs / name / current / fsimage of size 112 bytes saved in 0 seconds 16/01/09 23:34:11 INFO namenode.FSEditLog:. closing edit log: position = 4, editlog = / hadoop / hdfs / name / current / edits 16/01/09 23:34:11 INFO namenode.FSEditLog: close success: truncate to 4, editlog = / hadoop / hdfs / name / current / edits 16/01/09 23:34:11 INFO common.Storage: Storage directory / hadoop / hdfs / name has been successfully formatted 16/01/09 23:34:11 INFO namenode.NameNode. : SHUTDOWN_MSG: / ****************************************** ****************** SHUTDOWN_MSG: Shutting down NameNode at master / 127.0.1.1 ************* *********************************************** / hadoop@master:/opt/hadoop-1.2.1/bin $
start hadoop master on all services
hadoop @ master: /opt/hadoop-1.2.1/bin$ ./start-all.sh starting namenode, logging to /opt/hadoop-1.2.1/libexec/../logs/ hadoop-hadoop-namenode-master.out slave2: starting datanode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-slave2.out slave1 : starting datanode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-slave1.out master: starting datanode, logging to /opt/hadoop-1.2.1 /libexec/../logs/hadoop-hadoop-datanode-master.out master: starting secondarynamenode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-secondarynamenode- master.out starting jobtracker, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-jobtracker-master.out slave2: starting tasktracker, logging to / opt /hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-slave2.out slave1: starting tasktracker, logging to /opt/hadoop-1.2.1/libexec/../logs/ hadoop-hadoop-tasktracker-slave1.out master: starting tasktracker, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-master.out hadoop @master: /opt/hadoop-1.2.1/bin$ jps 5088 JobTracker 5010 SecondaryNameNode 4871 DataNode 4732 NameNode 5277 Jps 5230 TaskTracker hadoop @ master: /opt/hadoop-1.2.1/bin $
Observation slave1
hadoop @ slave1: ~ / .ssh $ jps 3669 TaskTracker 3718 Jps 3560 DataNode hadoop @ slave1: ~ / .ssh $
Observation slave2
hadoop @ slave2: ~ $ jps 3216 Jps 2982 TaskTracker 3095 DataNode
process issues that require attention:
Since you are using hadoop hadoop user to start the service, at least that was the user of java, hadoop program has access privileges to perform it, this points to note.
Second, the user hadoop of hdfs files and mapreduce file needs to have write access to it
One problem:
hadoop @ master: / usr / lib / jvm / java / bin $ jps The program 'jps' can be found in the following packages: * openjdk-7-jdk * openjdk-6-jdk Ask your administrator to install one of them
obviously have been installed jdk, and environmental variables did not appear wrong path, why not find jps order? The results still have to use update-alternative command to resolve this issue |