Spark local development environment to build - server - newfreesoft.com

Home

PC Games

Linux

Windows

Database

Network

Programming

Server

Mobile


	Home \ Server \ Spark local development environment to build			- Computer security protection remove local and remote system log files (Linux) - Running into the site-wide HTTPS (Server) - User rights management system under Linux (Linux) - Analyzing Linux server architecture is 32-bit / 64-bit (Server) - using the ssh command to check the socket / Network Connections (Linux) - How to use tmpfs in RHEL / CentOS 7.0 (Linux) - Depth understanding of Python character set encoding (Programming) - CentOS 7 source code to compile and install Nginx process record (Server) - Linux user management (Linux) - Quick paging ROW_NUMBER conducted (Database) - 10 important Linux ps command combat (Linux) - Nginx load balancing configuration (http proxy) (Server) - IP configuration under Linux (Linux) - Making Linux root file system problems on-link library (Programming) - Reported too many open files Linux solutions (Server) - Ubuntu Linux use MAC binding against ARP attacks (Linux) - To_teach you three strategies to prevent the LAN IP address theft (Linux) - Linux filtration empty file command summary (Linux) - Ubuntu users install the Download Manager software Xdman 5.0 (Linux) - Linux find command detailing (Linux)

	Spark local development environment to build
	Add Date : 2018-11-21



	Now many online build spark development environment is based on the idea, the personal habits of the eclipse, or use eclipse to build the development environment. Preparatory work, download Scala IDE for Eclipse version: http: //scala-ide.org/ Scala project version This method is similar to Java and engineering. New scala Engineering Remove scala repository comes in Engineering Add spark library spark-assembly-1.1.0-cdh5.2.0-Hadoop2.5.0-cdh5.2.0.jar Modify Project scala compiler version Right-click -> Scala -> set the Scala Installation You can also Right-click the project -> Properties -> Scala Compiler -> Use project Setting, select the corresponding spark scala version, select here Lastest2.10 bundle Code written in scala export jar package Maven project version Currently the project communities have adopted the basic Maven manage, so Maven to develop ways of learning is still very necessary. To create a spark project, we first learn to create a scala project under maven it! Scala Maven project Create a maven project, modify pom.xml <Project xmlns = "http://maven.apache.org/POM/4.0.0" xmlns: xsi = "http://www.w3.org/2001/XMLSchema-instance" xsi: schemaLocation = "http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <ModelVersion> 4.0.0 </ modelVersion> <GroupId> com.wankun.scala </ groupId> <ArtifactId> test3 </ artifactId> <Version> 1.0 </ version> <Packaging> jar </ packaging> <Name> test3 </ name> <Url> http://maven.apache.org </ url> <Properties> <Project.build.sourceEncoding> UTF-8 </project.build.sourceEncoding> <Maven-compiler-plugin.version> 3.1 </maven-compiler-plugin.version> <Org.scala-tools.version> 2.3.2 </org.scala-tools.version> <Scala.library.version> 2.10.4 </scala.library.version> </ Properties> <Dependencies> <Dependency> <GroupId> org.scala-lang </ groupId> <ArtifactId> scala-library </ artifactId> <Version> $ {scala.library.version} </ version> </ Dependency> <Dependency> <GroupId> junit </ groupId> <ArtifactId> junit </ artifactId> <Version> 4.10 </ version> <Scope> test </ scope> </ Dependency> </ Dependencies> <Repositories> <Repository> <Id> scala-tools.org </ id> <Name> Scala-tools Maven2 Repository </ name> <Url> http://scala-tools.org/repo-releases </ url> </ Repository> </ Repositories> <PluginRepositories> <PluginRepository> <Id> scala-tools.org </ id> <Name> Scala-tools Maven2 Repository </ name> <Url> http://scala-tools.org/repo-releases </ url> </ PluginRepository> </ PluginRepositories> <Build> <Plugins> <Plugin> <ArtifactId> maven-compiler-plugin </ artifactId> <Version> $ {maven-compiler-plugin.version} </ version> <Configuration> <Source> 1.6 </ source> <Target> 1.6 </ target> <Encoding> UTF-8 </ encoding> </ Configuration> </ Plugin> <Plugin> <GroupId> org.scala-tools </ groupId> <ArtifactId> maven-scala-plugin </ artifactId> <Executions> <Execution> <Goals> <Goal> compile </ goal> <Goal> testCompile </ goal> </ Goals> </ Execution> </ Executions> <Configuration> <SourceDir> src / main / scala </ sourceDir> <JvmArgs> <JvmArg> -Xms64m </ jvmArg> <JvmArg> -Xmx1024m </ jvmArg> </ JvmArgs> </ Configuration> </ Plugin> </ Plugins> </ Build> </ Project> </td> <td background="../../c6.jpg"> </td> <td background="../../c4.jpg"> </td> <td background="../../c6.jpg"> </td> </tr> <tr> <td height="20" background="../../c4.jpg"> </td> <td align="center" valign="middle"> </td> <td background="../../c6.jpg"> </td> <td background="../../c4.jpg"> </td> <td background="../../c6.jpg"> </td> </tr> <tr> <td height="20" background="../../c4.jpg"> </td> <td align="center" valign="middle"> </td> <td background="../../c6.jpg"> </td> <td background="../../c4.jpg"> </td> <td background="../../c6.jpg"> </td> </tr> <tr> <td height="20" background="../../c4.jpg"> </td> <td align="center" valign="middle"> </td> <td background="../../c6.jpg"> </td> <td background="../../c4.jpg"> </td> <td background="../../c6.jpg"> </td> </tr> <tr> <td height="20" background="../../c4.jpg"> </td> <td align="left" valign="middle">More:</td> <td background="../../c6.jpg"> </td> <td background="../../c4.jpg"> </td> <td background="../../c6.jpg"> </td> </tr> <tr> <td height="20" background="../../c4.jpg"> </td> <td align="center" valign="middle"><div align="left"> <a href='http://www.newfreesoft.com/programming/the_hill_sorting_algorithm_1086/' target='_blank' border=0> - The Hill sorting algorithm</a> (<a href='http://www.newfreesoft.com/programming/' target='_self'>Programming</a>)<br> <a href='http://www.newfreesoft.com/database/mysql_56_open_full_query_log_257/' target='_blank' border=0> - MySQL 5.6 Open full query log</a> (<a href='http://www.newfreesoft.com/database/' target='_self'>Database</a>)<br> <a href='http://www.newfreesoft.com/linux/installation_of_jdk_and_tomcat_under_linux_centos_6522/' target='_blank' border=0> - Installation of JDK and Tomcat under Linux (CentOS)</a> (<a href='http://www.newfreesoft.com/linux/' target='_self'>Linux</a>)<br> <a href='http://www.newfreesoft.com/server/based_kubernetes_construction_docker_cluster_management_comments_379/' target='_blank' border=0> - Based kubernetes Construction Docker Cluster Management Comments</a> (<a href='http://www.newfreesoft.com/server/' target='_self'>Server</a>)<br> <a href='http://www.newfreesoft.com/linux/centos_installed_jdk8_71/' target='_blank' border=0> - CentOS installed JDK8</a> (<a href='http://www.newfreesoft.com/linux/' target='_self'>Linux</a>)<br> <a href='http://www.newfreesoft.com/database/oracle_table_space_rename_and_delete_table_space_5556/' target='_blank' border=0> - Oracle table space rename and delete table space</a> (<a href='http://www.newfreesoft.com/database/' target='_self'>Database</a>)<br> <a href='http://www.newfreesoft.com/database/oracle_first_automated_installation_packages_130/' target='_blank' border=0> - Oracle first Automated Installation Packages</a> (<a href='http://www.newfreesoft.com/database/' target='_self'>Database</a>)<br> <a href='http://www.newfreesoft.com/linux/five_useful_commands_to_manage_file_types_and_system_time_in_linux_2909/' target='_blank' border=0> - Five useful commands to manage file types and system time in linux</a> (<a href='http://www.newfreesoft.com/linux/' target='_self'>Linux</a>)<br> <a href='http://www.newfreesoft.com/linux/grub_how_to_load_linux_kernel_1995/' target='_blank' border=0> - GRUB how to load Linux kernel</a> (<a href='http://www.newfreesoft.com/linux/' target='_self'>Linux</a>)<br> <a href='http://www.newfreesoft.com/server/activemq_configuration_getting_started_tutorial_5342/' target='_blank' border=0> - ActiveMQ configuration Getting Started Tutorial</a> (<a href='http://www.newfreesoft.com/server/' target='_self'>Server</a>)<br> <a href='http://www.newfreesoft.com/programming/binary_search_java_achieve_4469/' target='_blank' border=0> - Binary search -Java achieve</a> (<a href='http://www.newfreesoft.com/programming/' target='_self'>Programming</a>)<br> <a href='http://www.newfreesoft.com/linux/ubuntu_adsl_dial_up_internet_access_372/' target='_blank' border=0> - Ubuntu ADSL dial-up Internet access</a> (<a href='http://www.newfreesoft.com/linux/' target='_self'>Linux</a>)<br> <a href='http://www.newfreesoft.com/server/gitlab_guide_installation_under_ubuntu_1404_4330/' target='_blank' border=0> - GitLab Guide installation under Ubuntu 14.04</a> (<a href='http://www.newfreesoft.com/server/' target='_self'>Server</a>)<br> <a href='http://www.newfreesoft.com/linux/linux_system_started_to_learn_how_to_solve_the_xxx_is_not_in_the_sudoers_file_error_6096/' target='_blank' border=0> - Linux system started to learn: how to solve the xxx is not in the sudoers file error</a> (<a href='http://www.newfreesoft.com/linux/' target='_self'>Linux</a>)<br> <a href='http://www.newfreesoft.com/server/construction_ca_certificate_using_openssl_command_line_3663/' target='_blank' border=0> - Construction CA certificate using OpenSSL command line</a> (<a href='http://www.newfreesoft.com/server/' target='_self'>Server</a>)<br> <a href='http://www.newfreesoft.com/linux/linux_raid_levels_and_concepts_introduced_776/' target='_blank' border=0> - linux raid levels and concepts introduced</a> (<a href='http://www.newfreesoft.com/linux/' target='_self'>Linux</a>)<br> <a href='http://www.newfreesoft.com/linux/how_utorrent_download_the_iso_image_from_the_command_line_4437/' target='_blank' border=0> - How UTorrent download the ISO image from the command line</a> (<a href='http://www.newfreesoft.com/linux/' target='_self'>Linux</a>)<br> <a href='http://www.newfreesoft.com/linux/virtualbox_43_can_not_open_a_new_task_for_a_virtual_computer_solution_2748/' target='_blank' border=0> - VirtualBox 4.3 Can not open a new task for a virtual computer solution</a> (<a href='http://www.newfreesoft.com/linux/' target='_self'>Linux</a>)<br> <a href='http://www.newfreesoft.com/linux/securecrt_use_the_configuration_detailed_tutorial_2310/' target='_blank' border=0> - SecureCRT use the configuration detailed tutorial</a> (<a href='http://www.newfreesoft.com/linux/' target='_self'>Linux</a>)<br> <a href='http://www.newfreesoft.com/linux/linux_host_dual_lan_transceiver_package_arp_problem_6120/' target='_blank' border=0> - Linux host dual LAN transceiver package ARP problem</a> (<a href='http://www.newfreesoft.com/linux/' target='_self'>Linux</a>)<br> </div></td> <td background="../../c6.jpg"> </td> <td background="../../c4.jpg"> </td> <td background="../../c6.jpg"> </td> </tr> <tr> <td width="11" height="20" background="../../c7.jpg"> </td> <td background="../../c8.jpg"> </td> <td width="19" background="../../c9.jpg"> </td> <td width="19" background="../../c7.jpg"> </td> <td width="178" background="../../c8.jpg"> </td> <td width="19" background="../../c9.jpg"> </td> </tr> </table></td> </tr> </table></td> </tr> <tr> <td height="80"> <table width="100%" border="0" align="center" cellpadding="0" cellspacing="0"> <tr> <td width="18" height="20"> </td> <td width="947"> </td> <td width="18"> </td> </tr> <tr> <td height="20"> </td> <td align="center" valign="middle"><font size="-1">CopyRight 2002-2022 newfreesoft.com, All Rights Reserved.</font> </td> <td> </td> </tr> <tr> <td width="18" height="19"> </td> <td align="center" valign="middle"> <!-- Start of StatCounter Code for Default Guide --> <script type="text/javascript"> var sc_project=10901094; var sc_invisible=0; var sc_security="ba62ed46"; var scJsHost = (("https:" == document.location.protocol) ? "https://secure." : "http://www."); document.write("<sc"+"ript type='text/javascript' src='" + scJsHost+ "statcounter.com/counter/counter.js'></"+"script>"); </script> <noscript><div class="statcounter"><a title="shopify analytics tool" href="http://statcounter.com/shopify/" target="_blank"><img class="statcounter" src="http://c.statcounter.com/10901094/0/ba62ed46/0/" alt="shopify analytics tool"></a></div></noscript> <!-- End of StatCounter Code for Default Guide --> </td> <td width="18"> </td> </tr> <tr> <td width="18" height="19"> </td> <td> </td> <td width="18"> </td> </tr> </table></td> </tr> </table> <script type="text/javascript"> var Effect = (function() { var Slider = function(o) { this.setting = typeof o === 'object' ? o : {}; this.target = this.setting.target \|\| 'slider'; this.showMarkers = this.setting.showMarkers \|\| false; this.showControls = this.setting.showControls \|\| false; this.timer = null; this.currentTime = null; this.ms = 35; this.autoMs = 3000; this.iTarget = 0; this.nextTarget = 0; this.speed = 0; this.init(); this.handleEvent(); }; Slider.prototype = { init: function() { this.obj = document.getElementById(this.target); this.oUl = this.obj.getElementsByTagName('ul')[0]; this.aUlLis = this.oUl.getElementsByTagName('li'); this.width = this.aUlLis[0].offsetWidth; this.number = this.aUlLis.length; this.oUl.style.width = this.width * this.number + 'px'; if(this.showMarkers) { var oDiv = document.createElement('div'); var aLis = []; for(var i = 0; i < this.number; i++) { aLis.push('<li>'+ (i+1) +'<\/li>'); }; oDiv.innerHTML = '<ol>'+ aLis.join('') +'<\/ol>'; this.obj.appendChild(oDiv.firstChild); this.aLis = this.obj.getElementsByTagName('ol')[0].getElementsByTagName('li'); this.aLis[0].className = 'active'; oDiv = null; }; if(this.showControls) { this.oPrev = document.createElement('p'); this.oNext = document.createElement('p'); this.oPrev.className = 'prev'; this.oPrev.innerHTML = '«'; this.oNext.className = 'next'; this.oNext.innerHTML = '»'; this.obj.appendChild(this.oPrev); this.obj.appendChild(this.oNext); }; }, handleEvent: function() { var that = this; this.currentTime = setInterval(function() { that.autoPlay(); }, this.autoMs); this.addEvent(this.obj, 'mouseover', function() { clearInterval(that.currentTime); }); this.addEvent(this.obj, 'mouseout', function() { that.currentTime = setInterval(function() { that.autoPlay(); }, that.autoMs); }); if(this.showMarkers) { for(var i = 0; i < this.number; i++) { var el = this.aLis[i]; (function(index) { that.addEvent(el, 'mouseover', function() { that.goTime(index); }); })(i); }; }; if(this.showControls) { this.addEvent(this.oPrev, 'click', function() { that.fnPrev(); }); this.addEvent(this.oNext, 'click', function() { that.autoPlay(); }); }; }, addEvent: function(el, type, fn) { if(window.addEventListener) { el.addEventListener(type, fn, false); } else if(window.attachEvent) { el.attachEvent('on' + type, fn); }; }, fnPrev: function() { this.nextTarget--; if(this.nextTarget < 0) { this.nextTarget = this.number - 1; }; this.goTime(this.nextTarget); }, autoPlay: function() { this.nextTarget++; if(this.nextTarget >= this.number) { this.nextTarget = 0; }; this.goTime(this.nextTarget); }, goTime: function(index) { var that = this; if(this.showMarkers) { for(var i = 0; i < this.number; i++) { i == index ? this.aLis[i].className = 'active' : this.aLis[i].className = ''; }; }; this.iTarget = -index * this.width; if(this.timer) { clearInterval(this.timer); }; this.timer = setInterval(function() { that.doMove(that.iTarget); }, this.ms); }, doMove: function(target) { this.oUl.style.left = this.speed + 'px'; this.speed += (target - this.oUl.offsetLeft) / 3; if(Math.abs(target - this.oUl.offsetLeft) === 0) { this.oUl.style.left = target + 'px'; clearInterval(this.timer); this.timer = null; }; } }; return { slider: function(o) { var tt = new Slider(o); } }; })(); Effect.slider({ 'targetElement': 'slider', 'showMarkers': true, 'showControls': true }); </script> </body> </html>