Home PC Games Linux Windows Database Network Programming Server Mobile  
  Home \ Server \ Ubuntu under Spark development environment to build     - Redhat 5 prohibit IPv6 (Linux)

- Zabbix API and PHP configuration (Server)

- Ubuntu mysql stop fail to solve the problem (Database)

- rpm package specify the installation path (Linux)

- B-tree - ideas and implementation of C language code (Programming)

- Linux file permissions chmod chown (Linux)

- Linux system security configuration Collection (Linux)

- SQL Server 2012 failover looksalive check and is alive check (Database)

- Oracle inline view updates problems encountered (Database)

- C ++ stderr / stdout redirected to a file (Programming)

- PostgreSQL 9.3.5 database installation under Ubuntu Server 14.04 (Database)

- Wine 1.7 is installed on a system based on RedHat or Debian (Linux)

- To use slay kill user processes (Linux)

- Linux operating system security settings initial understanding (Linux)

- Linux vi command list (Linux)

- Linux vi command list (Linux)

- How to back up Debian system backupninja (Linux)

- Oracle SQL statement to retrieve data paging table (Database)

- SA weak password security system of the security risks posed (Linux)

- GRUB and Linux system repair (Linux)

  Ubuntu under Spark development environment to build
  Add Date : 2018-11-21      
  Use the following configuration Ubuntu Python application development Spark

Ubuntu 64-bit Basic environment configuration

Install JDK, download jdk-8u45-linux-x64.tar.gz, extract to /opt/jdk1.8.0_45

Download: http: //www.Oracle.com/technetwork/java/javase/downloads/index.html

Install scala, download scala-2.11.6.tgz, extract to /opt/scala-2.11.6

Shimoji Address: http://www.scala-lang.org/

Install Spark, download spark-1.3.1-bin-Hadoop2.6.tgz, extract it to / opt / spark-hadoop

Download: http: //spark.apache.org/downloads.html,

Configuration environment variable, edit the / etc / profile, execute the following command

python @ ubuntu: ~ $ sudo gedit / etc / profile

                      In most file add:

          #Seeting JDK JDK environment variable

            export JAVA_HOME = / opt / jdk1.8.0_45

            export JRE_HOME = $ {JAVA_HOME} / jre

            export CLASSPATH =:. $ {JAVA_HOME} / lib: $ {JRE_HOME} / lib

            export PATH = $ {JAVA_HOME} / bin: $ {JRE_HOME} / bin: $ PATH

         #Seeting Scala Scala environment variable

            export SCALA_HOME = / opt / scala-2.11.6

            export PATH = $ {SCALA_HOME} / bin: $ PATH

         #setting Spark Spark environment variable

           export SPARK_HOME = / opt / spark-hadoop /

        #PythonPath The Spark in pySpark module increases Python environment

         export PYTHONPATH = / opt / spark-hadoop / python

 Restart the computer, so that / etc / profile permanent, temporary take effect, open a command window and execute source / etc / profile to take effect in the current window

Test results of the installation

Open a command window, switch to the root directory of Spark

Executive ./bin/spark-shell, Scala to open the connection window Spark

    Startup error message appears scala>, a successful start

  Executive ./bin/pyspark, open the Python connection to Spark window

    Ubuntu under Spark development environment to build

 Startup error, as shown above appears when the successful launch.

   Browser accessed via: The following page


           Test SPark available.

Python An application made Spark

Previously set PYTHONPATH, will pyspark added to Python's search path

Open Spark installation directory, Python- "build folder py4j, complex Python to the next directory, as shown:

Open a command line window, enter the python, Python version 2.7.6, as shown in note Spark does not support Python3

Enter the import pyspark, as shown below, prove that the development work is completed before

  Use Pycharm new key projects, the use of red box code testing
- Production environment Linux bonding method standby mode slave card switch (Server)
- tar command to extract a file error: stdin has more than one entry (Linux)
- SSH service configuration to use a certificate login Linux server (Server)
- Oracle in the add & split partition on the impact of global & local index (Database)
- Ubuntu 15.10 15.04 14.10 14.04 Install Ubuntu Tweak (Linux)
- Nodejs mysql pool Example (Programming)
- To use Android RecyclerView (Programming)
- Using PHP MySQL library (Programming)
- Ubuntu Linux installation GAMIT10.6 (Linux)
- Ubuntu resolve sudo: source: command not found error (Linux)
- MongoDB common optimization settings in Linux (Database)
- Oracle index visible and hidden (visible / invisible) (Database)
- SQL in the specific internal Oracle process (Database)
- Simple comparison of MySQL and Oracle in a sql analytical details (Database)
- Ubuntu 14.04 / 13.10 users how to install Ubuntu Touch core applications (Linux)
- To access an Oracle database using Instant Client (Database)
- Linux module mechanism of (Programming)
- HAproxy let IP recording back-end RS (Server)
- OpenDaylight Helium version installed (Linux)
- MySQL uses mysqld_multi to deploy stand-alone multi-instance detail procedures (Database)
  CopyRight 2002-2022 newfreesoft.com, All Rights Reserved.