Home PC Games Linux Windows Database Network Programming Server Mobile  
           
  Home \ Server \ Ubuntu under Spark development environment to build     - Analysis RabbitMQ cluster (Server)

- Fedora 21 setting boot script (Linux)

- Use Aptik Battery Monitor monitoring Ubuntu laptop battery usage (Linux)

- Ten linux version of the system (Linux)

- Android Custom View step (Programming)

- Source MongoDB 3.2.1 installed on CentOS6.5 (Database)

- Ubuntu achieve initialization iptables (Linux)

- HTTPS Encryption Algorithm (Linux)

- Experience RHEL7 new features (Linux)

- TNS-03505 name could not be resolved (Database)

- Mounting kit under Fedora Linux (Linux)

- How to Install 3.16.7 CKT2 kernel in Ubuntu 14.10, Ubuntu 14.04 and its derivative versions (Linux)

- OGG-03510 Problem (Database)

- EXP-00091 Error resolved (Database)

- CentOS install Redis (Database)

- CentOS 5.x install Lua 5.2.3 error (LIBS = -lncurses) (Linux)

- DIY security of Linux platform (Linux)

- Under Ubuntu 15.04 installation JDK8 (Linux)

- Use libpq under Ubuntu 14.04 (Linux)

- Let 32 Linux / CentOS system to support more than 4G memory (Linux)

 
         
  Ubuntu under Spark development environment to build
     
  Add Date : 2018-11-21      
         
         
         
  Use the following configuration Ubuntu Python application development Spark

Ubuntu 64-bit Basic environment configuration

Install JDK, download jdk-8u45-linux-x64.tar.gz, extract to /opt/jdk1.8.0_45

Download: http: //www.Oracle.com/technetwork/java/javase/downloads/index.html

Install scala, download scala-2.11.6.tgz, extract to /opt/scala-2.11.6

Shimoji Address: http://www.scala-lang.org/

Install Spark, download spark-1.3.1-bin-Hadoop2.6.tgz, extract it to / opt / spark-hadoop

Download: http: //spark.apache.org/downloads.html,



Configuration environment variable, edit the / etc / profile, execute the following command

python @ ubuntu: ~ $ sudo gedit / etc / profile

                      In most file add:

          #Seeting JDK JDK environment variable

            export JAVA_HOME = / opt / jdk1.8.0_45

            export JRE_HOME = $ {JAVA_HOME} / jre

            export CLASSPATH =:. $ {JAVA_HOME} / lib: $ {JRE_HOME} / lib

            export PATH = $ {JAVA_HOME} / bin: $ {JRE_HOME} / bin: $ PATH

         #Seeting Scala Scala environment variable

            export SCALA_HOME = / opt / scala-2.11.6

            export PATH = $ {SCALA_HOME} / bin: $ PATH

         #setting Spark Spark environment variable

           export SPARK_HOME = / opt / spark-hadoop /

        #PythonPath The Spark in pySpark module increases Python environment

         export PYTHONPATH = / opt / spark-hadoop / python

 Restart the computer, so that / etc / profile permanent, temporary take effect, open a command window and execute source / etc / profile to take effect in the current window

Test results of the installation

Open a command window, switch to the root directory of Spark


Executive ./bin/spark-shell, Scala to open the connection window Spark


    Startup error message appears scala>, a successful start

  Executive ./bin/pyspark, open the Python connection to Spark window

    Ubuntu under Spark development environment to build


 Startup error, as shown above appears when the successful launch.

   Browser accessed via: The following page

   

           Test SPark available.

Python An application made Spark

Previously set PYTHONPATH, will pyspark added to Python's search path

Open Spark installation directory, Python- "build folder py4j, complex Python to the next directory, as shown:


Open a command line window, enter the python, Python version 2.7.6, as shown in note Spark does not support Python3


Enter the import pyspark, as shown below, prove that the development work is completed before


  Use Pycharm new key projects, the use of red box code testing
     
         
         
         
  More:      
 
- Installation Sublime Text 3 (Build 3065) text editor in Ubuntu (Linux)
- Linux System shutdown procedures (Linux)
- Struts2 dynamic call DMI and error Solution (Programming)
- Gitblit adopted Ticket collaborative development model (Linux)
- Android system source code and compile the kernel source code (Programming)
- Btrfs file system repair techniques (Linux)
- Linux, Firefox unloading and installation and upgrade (Linux)
- Top command: the Task Manager under linux (Linux)
- JavaScript prototype and prototype chain and project combat (Programming)
- Introduction to thread pooling and simple implementation (Programming)
- PPA on Ubuntu Linux installation Plank 0.8.0 (Linux)
- CentOS 6.3 compile and install LNMP environment (Server)
- Xshell configure SSH free password (Server)
- RabbitMQ installation, configuration, monitoring (Linux)
- Struts2 : combobox label use (Programming)
- Linux Basics Tutorial: Linux Kickstart automated installation (Linux)
- Teach you how to protect the Linux operating system security tips (Linux)
- Linux operating system set up to effectively prevent ARP attacks (Linux)
- MySQL performance view and configure finishing Daquan (Database)
- Bash job control (Linux)
     
           
     
  CopyRight 2002-2022 newfreesoft.com, All Rights Reserved.