Home PC Games Linux Windows Database Network Programming Server Mobile  
           
  Home \ Server \ Ubuntu under Spark development environment to build     - Docker Basic and Advanced (Linux)

- Improve the Ubuntu SSH login authentication approach speed (Linux)

- Seven kinds of NIC binding mode Detail (Linux)

- CentOS 6.5 can not connect to the network under VMware (Linux)

- Difference in MySQL VARCHAR and CHAR data format (Database)

- MySQL remote connection settings (Database)

- Linux command Detailed chpasswd bulk edit user password (Linux)

- Use nice, cpulimit and cgroups limit cpu usage (Linux)

- C ++, overloading, cover, hide (Programming)

- PostgreSQL Stream Configuration (Database)

- Linux find command usage practices (Linux)

- Zookeeper Rights Management and Quota Management (Server)

- Using C / C ++ extensions Python (Programming)

- Ubuntu install video playback software SMPlayer 14.9.0.7042 (Linux)

- How to upgrade the kernel to 3.13.7 on Ubuntu / Linux Mint (Linux)

- pkg-config to use (Linux)

- JITwatch installation under Linux (Linux)

- To install Ganglia configuration of experience under CentOS 5.5 (Linux)

- Sniffer install applications in Linux supplement (Linux)

- CentOS 6.5 boot automatically mount the hard drive (Linux)

 
         
  Ubuntu under Spark development environment to build
     
  Add Date : 2018-11-21      
         
         
         
  Use the following configuration Ubuntu Python application development Spark

Ubuntu 64-bit Basic environment configuration

Install JDK, download jdk-8u45-linux-x64.tar.gz, extract to /opt/jdk1.8.0_45

Download: http: //www.Oracle.com/technetwork/java/javase/downloads/index.html

Install scala, download scala-2.11.6.tgz, extract to /opt/scala-2.11.6

Shimoji Address: http://www.scala-lang.org/

Install Spark, download spark-1.3.1-bin-Hadoop2.6.tgz, extract it to / opt / spark-hadoop

Download: http: //spark.apache.org/downloads.html,



Configuration environment variable, edit the / etc / profile, execute the following command

python @ ubuntu: ~ $ sudo gedit / etc / profile

                      In most file add:

          #Seeting JDK JDK environment variable

            export JAVA_HOME = / opt / jdk1.8.0_45

            export JRE_HOME = $ {JAVA_HOME} / jre

            export CLASSPATH =:. $ {JAVA_HOME} / lib: $ {JRE_HOME} / lib

            export PATH = $ {JAVA_HOME} / bin: $ {JRE_HOME} / bin: $ PATH

         #Seeting Scala Scala environment variable

            export SCALA_HOME = / opt / scala-2.11.6

            export PATH = $ {SCALA_HOME} / bin: $ PATH

         #setting Spark Spark environment variable

           export SPARK_HOME = / opt / spark-hadoop /

        #PythonPath The Spark in pySpark module increases Python environment

         export PYTHONPATH = / opt / spark-hadoop / python

 Restart the computer, so that / etc / profile permanent, temporary take effect, open a command window and execute source / etc / profile to take effect in the current window

Test results of the installation

Open a command window, switch to the root directory of Spark


Executive ./bin/spark-shell, Scala to open the connection window Spark


    Startup error message appears scala>, a successful start

  Executive ./bin/pyspark, open the Python connection to Spark window

    Ubuntu under Spark development environment to build


 Startup error, as shown above appears when the successful launch.

   Browser accessed via: The following page

   

           Test SPark available.

Python An application made Spark

Previously set PYTHONPATH, will pyspark added to Python's search path

Open Spark installation directory, Python- "build folder py4j, complex Python to the next directory, as shown:


Open a command line window, enter the python, Python version 2.7.6, as shown in note Spark does not support Python3


Enter the import pyspark, as shown below, prove that the development work is completed before


  Use Pycharm new key projects, the use of red box code testing
     
         
         
         
  More:      
 
- Upgrade Goldengate 11.1.1.1.2 to 11.2.1.0.1 (Database)
- How do I upgrade from Ubuntu 15.04 to Ubuntu 15.10 (Linux)
- Installation and configuration of phpMyAdmin under CentOS (Database)
- Linux remote connectivity tools -OpenSSH (Linux)
- CentOS How to mount the hard drive (Linux)
- iOS persistence of data archiving NSKeyedArchiver (Programming)
- JDK comes with tools JPS (Linux)
- Linear table with a Java implementation of the iterator (Programming)
- To install Python-Pip and Fabric under CentOS / Ubuntu (Linux)
- How to use tmpfs in RHEL / CentOS 7.0 (Linux)
- Vi (Vim) keyboard map and its basic commands (Linux)
- Ubuntu 14.04 Configuring cuda-convnet (Linux)
- Linux Nginx installation and configuration instructions (Server)
- Oracle common internal event tracking number (Database)
- VNC configuration detailed analysis under Linux (Linux)
- Supervisor Installation and Configuration (Server)
- Linux and hardware (Linux)
- Zabbix monitors the status of TCP connections (Server)
- Using DOS command to change UNIX administrator password (Linux)
- Installation of Python2.7.10 under CentOS 6.4 (Linux)
     
           
     
  CopyRight 2002-2022 newfreesoft.com, All Rights Reserved.