Home PC Games Linux Windows Database Network Programming Server Mobile  
           
  Home \ Server \ Ubuntu under Spark development environment to build     - Command-line based tools for better alternatives (Linux)

- Spring JDBC Comments (Programming)

- Go performed using iOS and Android programming (Programming)

- MySQL Error Code Complete (Database)

- Linux System Getting Started Learning: the Linux Wireshark interface dead solve (Linux)

- How to Install Apache on Linux servers (Server)

- Ubuntu 14.04 / 14.10 how to install Mate 1.10.0 (Linux)

- CentOS 6.6 command-line automatic completion (Linux)

- Linux installation is larger than 2TB (UEFI interface) hard disk solution (Linux)

- 32-bit and 64-bit integer in PHP and MySQL range (Database)

- MySQL remote connection settings (Database)

- Debian 7.6 install Nvidia graphics driver (Linux)

- Oracle Database ORA-01555 snapshot too old (Database)

- Vim useful plugin: vundle (Linux)

- Preview function to confirm the stop resource Oracle 12c new feature crsctl (Database)

- Smooth upgrade to OpenSSH 6.1 Procedure (Linux)

- Linux install Samba file sharing server (Server)

- Forbid screen change the window size when creating a new window under CentOS (Linux)

- Linux installed Cisco Packet Tracer (Linux)

- RHEL 6.4 installed MySQL 5.6.27 (Database)

 
         
  Ubuntu under Spark development environment to build
     
  Add Date : 2018-11-21      
         
         
         
  Use the following configuration Ubuntu Python application development Spark

Ubuntu 64-bit Basic environment configuration

Install JDK, download jdk-8u45-linux-x64.tar.gz, extract to /opt/jdk1.8.0_45

Download: http: //www.Oracle.com/technetwork/java/javase/downloads/index.html

Install scala, download scala-2.11.6.tgz, extract to /opt/scala-2.11.6

Shimoji Address: http://www.scala-lang.org/

Install Spark, download spark-1.3.1-bin-Hadoop2.6.tgz, extract it to / opt / spark-hadoop

Download: http: //spark.apache.org/downloads.html,



Configuration environment variable, edit the / etc / profile, execute the following command

python @ ubuntu: ~ $ sudo gedit / etc / profile

                      In most file add:

          #Seeting JDK JDK environment variable

            export JAVA_HOME = / opt / jdk1.8.0_45

            export JRE_HOME = $ {JAVA_HOME} / jre

            export CLASSPATH =:. $ {JAVA_HOME} / lib: $ {JRE_HOME} / lib

            export PATH = $ {JAVA_HOME} / bin: $ {JRE_HOME} / bin: $ PATH

         #Seeting Scala Scala environment variable

            export SCALA_HOME = / opt / scala-2.11.6

            export PATH = $ {SCALA_HOME} / bin: $ PATH

         #setting Spark Spark environment variable

           export SPARK_HOME = / opt / spark-hadoop /

        #PythonPath The Spark in pySpark module increases Python environment

         export PYTHONPATH = / opt / spark-hadoop / python

 Restart the computer, so that / etc / profile permanent, temporary take effect, open a command window and execute source / etc / profile to take effect in the current window

Test results of the installation

Open a command window, switch to the root directory of Spark


Executive ./bin/spark-shell, Scala to open the connection window Spark


    Startup error message appears scala>, a successful start

  Executive ./bin/pyspark, open the Python connection to Spark window

    Ubuntu under Spark development environment to build


 Startup error, as shown above appears when the successful launch.

   Browser accessed via: The following page

   

           Test SPark available.

Python An application made Spark

Previously set PYTHONPATH, will pyspark added to Python's search path

Open Spark installation directory, Python- "build folder py4j, complex Python to the next directory, as shown:


Open a command line window, enter the python, Python version 2.7.6, as shown in note Spark does not support Python3


Enter the import pyspark, as shown below, prove that the development work is completed before


  Use Pycharm new key projects, the use of red box code testing
     
         
         
         
  More:      
 
- Nginx load balancing configuration (http proxy) (Server)
- See Shell Script Linux Server network traffic (Server)
- Compile and install Memcached can not find GCC (Programming)
- Linux user login ban (Linux)
- Basic Java JNI (Programming)
- Redhat 7 can only be read after installation Samba service catalog approach could not be written (Server)
- VNC connection VMware vSphere ESXi 5.5 (Linux)
- Linux operating system security management skills (Linux)
- Let Linux operating system more secure (Linux)
- Linux raw socket (Programming)
- Samhain: Powerful intrusion detection system under Linux (Linux)
- Linux Basic Course: Install the software from source code (Linux)
- There are three ways to run a Linux operating system from a USB stick (Linux)
- Make full use of the Raspberry Pi SD card space (Linux)
- Install the latest Pinta graphics editing software on Ubuntu 14.04 (Linux)
- Oracle11g build physical standby database (Database)
- Linux to achieve a simple cp command (Linux)
- Bash difference in single quotes and double quotes (Programming)
- How to set up FTP server on Linux (Server)
- Linux installation Jetty deployment under RedHat5 8 (Linux)
     
           
     
  CopyRight 2002-2020 newfreesoft.com, All Rights Reserved.