Home PC Games Linux Windows Database Network Programming Server Mobile  
           
  Home \ Server \ Hadoop 2.6.0 standalone configuration and pseudo-distributed configuration under Ubuntu 14.04     - Linux-du and df command (Linux)

- Modify MySQL character encoding under Linux (Database)

- The PostgreSQL database pg_dump command line does not enter a password method (Database)

- CentOS 7 RHEL 7 to reset the root password (Linux)

- CentOS 6.5 opens the Xmanager Remote Desktop login (Linux)

- YUM install desktop environment in CentOS (Linux)

- DRBD installation configuration, working principle and Recovery (Server)

- Use LKM change the default linux security level (Linux)

- Detailed installation OpenCV2.3.1 under CentOS 6.5 (Linux)

- Efficient Linux Shell - Shell special characters Summary (Linux)

- Installation Yarock 1.1.4 Music Player in Ubuntu (Linux)

- Root of AVL Tree- achieve balanced search trees AVL tree (Programming)

- VMware clone Linux find eth0 (Linux)

- Linux daemon (Linux)

- Java and Python use make way dictionary word search script (Programming)

- Java memory area Explanation (Programming)

- Lambda expressions of evolution (Programming)

- OpenSSH server configuration file for each Common Definition (Server)

- mysqldump MySQL command-line tool (Database)

- How to use OpenVPN and PrivacyIDEA build two-factor authentication for remote access (Server)

 
         
  Hadoop 2.6.0 standalone configuration and pseudo-distributed configuration under Ubuntu 14.04
     
  Add Date : 2018-11-21      
         
         
         
  Hadoop create user groups and users in Ubuntu

Hadoop is the best administrator after logon desktop environment to run eclipse users. Otherwise, there will be behind the refusal literacy problems. Of course not, there are ways solution.

1. Create hadoop user group;

sudo addgroup hadoop
2. Create hadoop users;

sudo adduser -ingroup hadoop hadoop
3. hadoop user to add permissions, open the / etc / sudoers file;

sudo gedit / etc / sudoers
In the root ALL =: add the next (ALL ALL) ALL hadoop ALL = (ALL: ALL) ALL.

Install JDK in Ubuntu
See in particular: http: //blog.csdn.net/ggz631047367/article/details/42366687 //JAVA_HOME=/usr/lib/jvm/jdk1.8.0_25

Install ssh service

sudo apt-get install ssh openssh-server

Establish ssh without password this machine
Switch to hadoop user, execute the following command:

su - hadoopssh generate rsa and dsa keys have two ways to generate mode, using rsa mode by default.

1. Create ssh-key ,, here we use rsa way;

ssh-keygen -t rsa -P "" (Note: The two files are generated in ~ / .ssh / post-carriage: id_rsa and id_rsa.pub these two files are in pairs)

2. Go to ~ / .ssh / directory, appends the id_rsa.pub to authorized_keys authorization file, the start is no authorized_keys files;

cd ~ / .ssh
cat id_rsa.pub >> authorized_keys (after the completion of the unit can no password up.)
3. Log localhost;

ssh localhost

4. Run the quit command;

exit
Install hadoop

Download: http: //apache.fayea.com/hadoop/common/stable/hadoop-2.6.0.tar.gz

1. hadoop extract to / usr / local:

sudo tar -zxvf hadoop-2.6.0.tar.gz
sudo mv hadoop-2.6.0 / usr / local / hadoop
sudo chmod -R 775 / usr / local / hadoop
sudo chown -R hadoop: hadoop / usr / local / hadoop // otherwise ssh will refuse access

Configuring

Modify bashrc configuration:

sudo gedit ~ / .bashrc

At the end of the file add:

#HADOOP VARIABLES START

export JAVA_HOME = / usr / lib / jvm / jdk1.8.0_25

export HADOOP_INSTALL = / usr / local / hadoop

export PATH = $ PATH: $ HADOOP_INSTALL / bin

export PATH = $ PATH: $ HADOOP_INSTALL / sbin

export HADOOP_MAPRED_HOME = $ HADOOP_INSTALL

export HADOOP_COMMON_HOME = $ HADOOP_INSTALL

export HADOOP_HDFS_HOME = $ HADOOP_INSTALL

export YARN_HOME = $ HADOOP_INSTALL

export HADOOP_COMMON_LIB_NATIVE_DIR = $ HADOOP_INSTALL / lib / native

export HADOOP_OPTS = "- Djava.library.path = $ HADOOP_INSTALL / lib"

#HADOOP VARIABLES END
If you do not know JAVA_HOME command can be obtained by:

update-alternatives --config java java directory to the root directory can get.

Run the following command to make the changes take effect:

source ~ / .bashrc
Modify hadoop-env.sh configurations:

sudo gedit /usr/local/hadoop/etc/hadoop/hadoop-env.sh
Find JAVA_HOME to the above values.

test
By performing hadoop own instance WordCount verify that the installation was successful

 Create input folder / usr / local / hadoop path

mkdir input
cp README.txt input
Performed in hadoop directory WordCount:

bin / hadoop jar share / hadoop / mapreduce / sources / hadoop-mapreduce-examples-2.6.0-sources.jar
org.apache.hadoop.examples.WordCount input output

Hadoop pseudo-distributed configuration

sudo gedit /usr/local/hadoop/etc/hadoop/core-site.xml

< Configuration>
    < Property>
        < Name> hadoop.tmp.dir < / name>
        < Value> / usr / local / hadoop / tmp < / value>
        < Description> Abase for other temporary directories. < / Description>
    < / Property>
    < Property>
        < Name> fs.defaultFS < / name>
        < Value> hdfs: // localhost: 9000 < / value>
    < / Property>
< / Configuration>
 sudo gedit /usr/local/hadoop/etc/hadoop/mapred-site.xml // this unnecessary

< Configuration>
 < Property>
      < Name> mapred.job.tracker < / name>
      < Value> localhost: 9001 < / value>
    < / Property>
< / Configuration> sudo gedit /usr/local/hadoop/etc/hadoop/yarn-site.xml

< Configuration>
< Property>
< Name> mapreduce.framework.name < / name>
< Value> yarn < / value>
< / Property>

< Property>
< Name> yarn.nodemanager.aux-services < / name>
< Value> mapreduce_shuffle < / value>
< / Property>
< / Configuration>

 sudo gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml

< Configuration>
< Property>
        < Name> dfs.replication < / name>
        < Value> 1 < / value>
    < / Property>
    < Property>
        < Name> dfs.namenode.name.dir < / name>
        < Value> file: / usr / local / hadoop / dfs / name < / value>
    < / Property>
    < Property>
        < Name> dfs.datanode.data.dir < / name>
        < Value> file: / usr / local / hadoop / dfs / data < / value>
    < / Property>
    < Property> // This attribute node is to prevent the rear eclopse Denial reader settings
            < Name> dfs.permissions < / name>
            < Value> false < / value>
    < / Property>
 < / Configuration>
sudo gedit / usr / local / hadoop / etc / hadoop / masters add: localhost

sudo gedit / usr / local / hadoop / etc / hadoop / slaves added: localhost

A note on configuration: as long as the above configuration fs.defaultFS and dfs.replication can run, but there is no such argument is hadoop.tmp.dir configuration parameters, when the temporary directory is used by default Hadoop / tmp / hadoo- hadoop, and this directory will be restarted after each kill, you must re-execute format for the job (not verified), the pseudo-distributed configuration is best to set it.

After the configuration, first create a temporary directory required under Hadoop directory:

cd / usr / local / hadoop
mkdir tmp dfs dfs / name dfs / data

Then initialize the file system HDFS.
bin / hdfs namenode -format successful final tips below, Exitting with status 0 indicates success, Exitting with status 1: it is wrong.

sbin / start-dfs.sh
sbin / start-yarn.sh

Unable to load native-hadoop library for your platform this prompt solution:
1, the new lib will recompile the source / native to replace the original cluster lib / native
2, modify hadoop-env.sh, increase
export HADOOP_OPTS = "- Djava.library.path = $ HADOOP_PREFIX / lib: $ HADOOP_PREFIX / lib / native"

Namenode information: http: // localhost: 50070 to view the Hadoop information.

All Applications: http: // http: //2xx.81.8x.1xx: 8088 /, where 2xx.81.8x.1xx will replace your actual IP address.

Run the examples:

1. First, build a folder bin / hdfs dfs -mkdir -p / user / ha1 / input on hdfs

                                  bin / hdfs dfs -mkdir -p / user / ha1 / output

2. Upload files: bin / hdfs dfs -put etc / hadoop / / user / ha1 / input the etc / hadoop hdfs upload files to the / user / ha1 / input in

3. Instruction

bin / hadoop jar share / hadoop / mapreduce / hadoop-mapreduce-examples-2.6.0.jar grep / user / ha1 / input / hadoop / user / ha1 / output / temp 'dfs [a-z.] +'

4. Check results

bin / hdfs dfs -cat / user / ha1 / output / temp / *

8 dfs.audit.logger
4 dfs.class
3 dfs.server.namenode.
2 dfs.audit.log.maxbackupindex
2 dfs.period
2 dfs.audit.log.maxfilesize
1 dfsmetrics.log
1 dfsadmin
1 dfs.servers
1 dfs.replication
1 dfs.file
1 dfs.datanode.data.dir
1 dfs.namenode.name.dir
     
         
         
         
  More:      
 
- Python decorators to learn and practice the actual usage scenarios (Programming)
- Upload the project to GitHub, synchronous remote repository Github (Linux)
- Security of data to create a safe .mdb database (Linux)
- How to remove the Linux memory Cache, Buffer and swap space (Linux)
- Use GLOBK batch command network adjustment (Linux)
- Redis-2.8.17 installation and configuration process some errors (Linux)
- Linux Mint 17 set up the Ruby environment (Linux)
- Shell Programming points to note about the function (Programming)
- Ubuntu disable graphics card (Linux)
- To use slay kill user processes (Linux)
- Nodejs mysql pool Example (Programming)
- Try to use Lets Encrypt (Linux)
- The formatted Linux hard drive and mount (Linux)
- How to make Linux a non-root user uses less than 1024 ports (Linux)
- ORA-01157 & ORA-01110 Troubleshooting (Database)
- [SHELL] MySQL primary recovery solution from + Keepalived online (Server)
- Redis master-slave replication switch (Database)
- Do you know how to build the Linux kernel (Programming)
- Multipath configuration under Linux (Linux)
- Verify the character set on MyCAT (Database)
     
           
     
  CopyRight 2002-2022 newfreesoft.com, All Rights Reserved.