Pre-requisites 1. Java 2. Maven 3. Intellij or STS (Optional but recommended)
Follow below steps to complete your first spark application
Create a new maven project without any specific archetype. I am using IntelliJ editor but you may choose any other suitable editor as well. I have created a project with name “SparkExample”
Navigate to File-> New Project
Select Maven from Left Panel
Do not select any archetype
Click on “Next”
Name the project “SparkExample”
Click on “Finish” This should create a new maven project like below
Next we update the pom.xml with spark-core dependency as below.
3. Now we create a new Class “WordCount” in “com.examples” package and copy below contents.
package com.examples;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import java.util.Arrays;
import java.util.Map;
public class WordCount {
public static void main(String[] args) throws Exception {
SparkConf conf = new SparkConf().setAppName("wordCounts").setMaster("local[3]");
JavaSparkContext sc = new JavaSparkContext(conf);
JavaRDD<String> lines = sc.textFile("in/word_count.txt");
JavaRDD<String> words = lines.flatMap(line -> Arrays.asList(line.split(" ")).iterator());
Map<String, Long> wordCounts = words.countByValue();
for (Map.Entry<String, Long> entry : wordCounts.entrySet()) {
System.out.println(entry.getKey() + " : " + entry.getValue());
}
}
}
4. create a new directory folder “in” at the project root and add below file into the “in” directory. In this example we are going to read this file and use spark to count the occurrence of each word in the file.
5. Now we have to build our project to see our output. Since we are using maven, we can run the “mvn clean install” from the command prompt or we can use the rebuild from the intellij. both works and once that is done we can run our application. So basically we have to run the WordCount class so right click on the class and run “WordCount.main()”
6. This should fire up a standalone spark application and run our job of “WordCount”. This job basically counts for the occurrences of words in the file “word_count.txt”. The output should look like below
7. Now that we have successfully ran the program, lets learn what really happened The below code configures the name of our spark application and we set the master to be local, which basically tells spark to run this application locally and run it on 3 cores.
SparkConf conf = new SparkConf().setAppName("wordCounts").setMaster("local[3]");
8. Initializes the spark context
JavaSparkContext sc = new JavaSparkContext(conf);
9. The below code reads the file and converts it into what is called as Resilient Distributed Dataset (RDD) . This will distribute our file into 3 cores to be processed further and returns us with a single reference to the RDD for manipulation
10. Rest of the code is self explanatory. The RDD api provides with certain apis like the one we have used. The countByValue as name suggest counts the occurrence of values in our text file and then when we print the values from the map, we get a consolidated view of the aggregation.
So as can be seen, writing a spark application is really easy and its only with a single class we can start writing a spark application. Please comment if you have faced any issue following this tutorial or like if you would like to see more.
In plain simple computing language, spark is an open source cluster computing framework. It is used to solve big data problems. Spark distributes the data on the cluster nodes and then will process the distributed data on each of the nodes against the local data and then send the consolidated response back to the requested of the spark job. If someone is going to ask you about spark, the above explanation is good enough.
Big data has gained lot of traction in last decade or so as users of the internet are continuously creating huge amount of data and processing huge data was something our older frameworks were not capable of handling. Such huge amount of data requires special handling and that was provided initially by Hadoop. What spark provides over Hadoop is the speed. Spark in most cases will perform better than Hadoop. Spark does all the processing in-memory whereas Hadoop writes it on the disk. When it is in-memory processing spark performs upto 100 times better than Hadoop and upto 10 times faster when we write to disk in spark.
It does appear that it will be complex and difficult to follow and understand spark however, most of the complexity is abstracted by the spark and it is extremely easy to start coding in spark. If you know basics of Java, Scala, Python or R, then you can easily write a spark job. In terms of java, we are supposed to write everything inside a main program and we can submit the same to a spark cluster.
A spark cluster typically looks like below. Suppose we are having a spark cluster of 3 nodes. So one of the node will become the master node and rest as worker nodes. Spark has a standalone cluster manager which will basically drive your program across the cluster and act as master. A spark job is submitted to a spark cluster ( a spark job is nothing but a main program bundled in a jar), the node on which we submit a spark job is called the driver program and will have the instance of spark context. All the processing will happen on the worker nodes. Spark can work with HDFS, Hive, Cassandra, Hbase as its storage. We will come to know why do we need storage in future posts.
This is a small introduction to Apache Spark. I will be writing more about the Spark Architecture, Core components, Running your first spark application in future posts. Let me know what you would like to see first.
Also please comment or like if you liked the explanation. Thank you for spending time reading this post.
Lets see how to add pagination to viewset. As will all other things in django, this is super easy
Assuming you have already done the the tutorial P6 – Django – Viewsets it is possible that we might need pagination on some of the rest calls. So we may have a lot of data on the server and sending it all at once back to the client may not be required and may not be the efficient way to handle the data. The client on the other hand might need only a handful of data and would not want all the data which will become difficult to process. So Lets see how do we set up pagination on view sets.
lets take example of rest services which returns “Ads” and it may have thousands of ads. At any point of time the client will need only a handful of them to process. So the client must be able to pass the rest service some parameter identifying the page number it desires.
For our example. We create a pagination.py file and add below content.
from rest_framework.pagination import PageNumberPagination
class PaginationMeta(PageNumberPagination):
page_size = 10
page_size_query_param = 'page_size'
max_page_size = 10000
Here we have set three parameters. One is the “page_size” which means the amount of data the server will send if requested with page number i.e with parameter “page” . The next is “page_size_query_param” which can be used if want a specified number of records to be retrieved and override the page_size. “max_page_size” which is the max page size for our request.
The next step is to set this class in our viewset as below.
from django.shortcuts import render
from .models import Ads
from .serializers import AdsSerializer
from rest_framework import viewsets
from .pagination import PaginationMeta
# Create your views here.
class AdsView(viewsets.ModelViewSet):
queryset = Ads.objects.all()
serializer_class = AdsSerializer
pagination_class = PaginationMeta
Here we set the pagination_class to the PaginationMeta we created. And this is all we will have to do to set up pagination. I am providing other classes required for this example.
from django.db import models
# Create your models here.
class Ads(models.Model):
category = models.CharField(max_length=50)
city= models.CharField(max_length=50)
email = models.CharField(max_length=50)
brand = models.CharField(max_length=50)
make = models.CharField(max_length=50)
capacity = models.CharField(max_length=50)
desc = models.CharField(max_length=150)
price = models.FloatField(max_length=20)
from rest_framework import serializers
from .models import Ads
class AdsSerializer(serializers.ModelSerializer):
class Meta:
model = Ads
fields = ['id','category', 'city','email','brand','make','capacity','price','desc']
from rest_framework.routers import DefaultRouter
from . import views
from django.urls import path, include
router = DefaultRouter()
router.register('ads', views.AdsView)
urlpatterns = [
path('', include(router.urls))
]
Now we have set the url pattern for our views to “/ads” so to access the views so a call to http://localhost:8000/api/ads/ will return the first page as below.
As you can see, it has returned with the first page and also various others attributes to navigate to next page. We can jump to any page we wish by providing the page parameter to the url e.g.http://localhost:8000/api/ads/?page=4
Please let us know if you liked this post and if it helped you. Thank you for spending time reading my post.
This directive is getting used to apply multiple styles at a same time as array from variable or it can be also used to change style base on conditions
The syntax is:
<h3 [ngStyle]="{'color': flag ? 'red':'green' }">Change Style</h3>
Prerequisites Angularj JS new version is using nodejs for setting up application and for running applicaion you will also need any code editor tool. For current setup we are using visual code studio Kindly down load latest version of nodejs and visual code studio on your machine and complete setup Node Installation 1 Go to…
We have declared variable which is boolean and initial value is false and it will get changed when use clicks on button from UI and flag will become true
In this code we are dynamically applying class and style to h1 element base on “flag” declared in component above. Initially its false so no class will be applied but style color will be green for false value.
Once use click button and “changeClass()” function will called value of flag will be true and class will get applied as “blue-text text-darken-2′” and style color will be red.
app.module.ts
import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core';
import { AppRoutingModule } from './app-routing.module';
import { AppComponent } from './app.component';
import { TwowaybindingwithNGModelComponent } from './twowaybindingwith-ngmodel/twowaybindingwith-ngmodel.component';
import { FormsModule } from '@angular/forms';
import { NgclassandngstyleComponent } from './ngclassandngstyle/ngclassandngstyle.component';
@NgModule({
declarations: [
AppComponent,
TwowaybindingwithNGModelComponent,
NgclassandngstyleComponent
],
imports: [
BrowserModule,
AppRoutingModule,
FormsModule
],
providers: [],
bootstrap: [AppComponent]
})
export class AppModule { }
Add Materialize css in index.html
In Index.html we will add materialize css and js file to use elegant UI
In main html we have imbedded all three components.
Lets see output of this
Now click button and check for class and style
After clicking on button it changes style and color of text
We have combined all above mentioned examples on single page in app.component.html and it will look like below on browser so you can play with all 3 components
In this Example we will see inbuilt angularjs Directives
NgIf
NgFor
NgSwitch
NgIf ,NgFor and NgSwitch
Lets go through basics first
NgIf
This directive is used when you want to display or remove an element based on a condition.
The syntax is:
*ngIf=”<condition>”
NgFor
This directive is getting used to iterate through collection
The syntax is:
<ul *ngFor =" let n of newcars">
NgSwitch
NgSwitch is like the JavaScript switch statement. It displays one element from among several possible elements, based on a switch condition. Angular puts only the selected element into the DOM.
NgSwitch is actually a set of three, cooperating directives: NgSwitch, NgSwitchCase, and NgSwitchDefault as in the following example.
Prerequisites Angularj JS new version is using nodejs for setting up application and for running applicaion you will also need any code editor tool. For current setup we are using visual code studio Kindly down load latest version of nodejs and visual code studio on your machine and complete setup Node Installation 1 Go to…
<p>ng-if-ng-for-ng-switch works!</p>
<div class="row">
<div class="col s3" style="border: brown; border-style: dashed; margin: 10px;">
<h4>{{title}}</h4>
<ul *ngFor=" let c of cars">
<li> {{c}}</li>
</ul>
</div>
<div class="col s3" style="border: brown; border-style: dashed; margin: 10px;">
<h4>Display data with ngFor and ngIf</h4>
<ul *ngFor=" let n of newcars">
<li style="color:red" *ngIf="n.average < 7 "> {{n.name}}</li>
</ul>
</div>
<div class="col s4" style="border: brown; border-style: dashed; margin: 10px;">
<h4>Display data with Switch Directive</h4>
<ul *ngFor=" let c of newcars" [ngSwitch]="c.name">
<li *ngSwitchCase="'city'" class="blue-text text-darken-2">{{ c.name }} {{ c.average }}
</li>
<li *ngSwitchCase="'polo'" class="blue-text text-darken-2">{{ c.name }} {{ c.average }}
</li>
<li *ngSwitchDefault class="text-success">{{ c.name }} {{ c.average }}
</li>
</ul>
</div>
</div>
In this code we are first printing all car names using for loop
In second for loop we are filtering cars whose average is less than 7.
In third block we are using switch case so we can apply different class to different cars based on name comparison
ng-if-ng-for-ng-switch.component.ts
import { Component, OnInit } from '@angular/core';
import { Car } from '../car';
@Component({
selector: 'app-ng-if-ng-for-ng-switch',
templateUrl: './ng-if-ng-for-ng-switch.component.html',
styleUrls: ['./ng-if-ng-for-ng-switch.component.css']
})
export class NgIfNgForNgSwitchComponent implements OnInit {
constructor() { }
title = 'Display Cars';
cars = ['city', 'nexon' , 'vento'];
newcars = [
new Car('city', 10),
new Car('polo', 5),
new Car('vento', 6),
];
ngOnInit() {
}
}
In this component we are populating array of cars.
car.ts
export class Car {
name: string;
average: number;
constructor( name , average) {
this.name = name;
this.average = average;
}
}
In Above code we have declared car as class and with if we are filtering out some of cars based on use of different combination of ngIf,ngfor and ngswitch directives
After running this it will show below output
app.module.ts
import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core';
import { AppRoutingModule } from './app-routing.module';
import { AppComponent } from './app.component';
import { TwowaybindingwithNGModelComponent } from './twowaybindingwith-ngmodel/twowaybindingwith-ngmodel.component';
import { FormsModule } from '@angular/forms';
import { NgclassandngstyleComponent } from './ngclassandngstyle/ngclassandngstyle.component';
@NgModule({
declarations: [
AppComponent,
TwowaybindingwithNGModelComponent,
NgclassandngstyleComponent
],
imports: [
BrowserModule,
AppRoutingModule,
FormsModule
],
providers: [],
bootstrap: [AppComponent]
})
export class AppModule { }
Add Materialize css in index.html
In Index.html we will add materialize css and js file to use elegant UI
This example demonstrated that ngIF and NgFor working fine
We have combined all above mentioned examples on single page in in git repo in app.component.html and it will look like below on browser so you can play with all 3 components
Prerequisites Angularj JS new version is using nodejs for setting up application and for running applicaion you will also need any code editor tool. For current setup we are using visual code studio Kindly down load latest version of nodejs and visual code studio on your machine and complete setup Node Installation 1 Go to…
This directive is used when you want to display or remove an element based on a condition.
The syntax is:
*ngIf=”<condition>”
NgFor
This directive is getting used to iterate through collection
The syntax is:
<ul *ngFor =" let n of newcars">
NgSwitch
NgSwitch is like the JavaScript switch statement. It displays one element from among several possible elements, based on a switch condition. Angular puts only the selected element into the DOM.
NgSwitch is actually a set of three, cooperating directives: NgSwitch, NgSwitchCase, and NgSwitchDefault as in the following example.
<p>ng-if-ng-for-ng-switch works!</p>
<div class="row">
<div class="col s3" style="border: brown; border-style: dashed; margin: 10px;">
<h4>{{title}}</h4>
<ul *ngFor=" let c of cars">
<li> {{c}}</li>
</ul>
</div>
<div class="col s3" style="border: brown; border-style: dashed; margin: 10px;">
<h4>Display data with ngFor and ngIf</h4>
<ul *ngFor=" let n of newcars">
<li style="color:red" *ngIf="n.average < 7 "> {{n.name}}</li>
</ul>
</div>
<div class="col s4" style="border: brown; border-style: dashed; margin: 10px;">
<h4>Display data with Switch Directive</h4>
<ul *ngFor=" let c of newcars" [ngSwitch]="c.name">
<li *ngSwitchCase="'city'" class="blue-text text-darken-2">{{ c.name }} {{ c.average }}
</li>
<li *ngSwitchCase="'polo'" class="blue-text text-darken-2">{{ c.name }} {{ c.average }}
</li>
<li *ngSwitchDefault class="text-success">{{ c.name }} {{ c.average }}
</li>
</ul>
</div>
</div>
In this code we are first printing all car names using for loop
IN second for loop we are filtering cars whose average is less than 7.
In third block we are using switch case so we can apply different class to different cars based on name comparison
ng-if-ng-for-ng-switch.component.ts
import { Component, OnInit } from '@angular/core';
import { Car } from '../car';
@Component({
selector: 'app-ng-if-ng-for-ng-switch',
templateUrl: './ng-if-ng-for-ng-switch.component.html',
styleUrls: ['./ng-if-ng-for-ng-switch.component.css']
})
export class NgIfNgForNgSwitchComponent implements OnInit {
constructor() { }
title = 'Display Cars';
cars = ['city', 'nexon' , 'vento'];
newcars = [
new Car('city', 10),
new Car('polo', 5),
new Car('vento', 6),
];
ngOnInit() {
}
}
In this component we are populating array of cars.
car.ts
export class Car {
name: string;
average: number;
constructor( name , average) {
this.name = name;
this.average = average;
}
}
In Above code we have declared car as class and with if we are filtering out some of cars based on use of different combination of ngIf,ngfor and ngswitch directives
After running this it will show below output
NgClass and NgStyle
Lets go through basics first
NgClass
This directive is used when you want to select class of element on certain conditions
This directive is getting used to apply multiple styles at a same time as array from variable or it can be also used to change style base on conditions
The syntax is:
<h3 [ngStyle]="{'color': flag ? 'red':'green' }">Change Style</h3>
For demonstrating this we have created below component in git project
We have declared variable which is boolean and initial value is false and it will get changed when use clicks on button from UI and flag will become true
In this code we are dynamically applying class and style to h1 element base on “flag” declared in component above. Initially its false so no class will be applied but style color will be green for false value.
Once use click button and “changeClass()” function will called value of flag will be true and class will get applied as “blue-text text-darken-2′” and style color will be red.
Lets see output of this
Now click button and check for class and style
You can see NgClass and NgStyle are working as expected
NgModel
NgModel Overview
This directive allows you to display a data property and update that property when the user makes changes. You have to use FormModule and need to import in your module to use this.
This supports two way data binding in forms
2. Create Child Component
ng generate component twowaybindingwithngmodel
3. Edit twowaybindingwith-ngmodel.component.ts
In this component we have declared variables to demonstrate one way and two way binding initially both will have red color
When user will click button it will log color and we will see if it reflects in console as per change.
4. Edit HTML File twowaybindingwith-ngmodel.component.html
Here we are adding button which will be get clicked to print color on console.
IN first block we have used two way binding with
Enter Color : <input [(ngModel)] ='twowaycolor'>
And in second blod we are using it for one way data binding
Enter Color : <input [ngModel] ='onewaycolor'>
When in first block twowaycolor value get changed and will click button it will be get printed because data will be get passed from component to view and view to Component in bi-direction.
When in second block onewaycolor value get changed and will click button it will be print old value because new value from view is not getting passed to component as its using unidirectional data binding with [ngModel
twowaybindingwith-ngmodel.component.html
<p>One and Two way Binding with NG Model</p>
<div class="card-panel blue ">
<h2>Two way Binding</h2>
Enter Color : <input [(ngModel)]='twowaycolor'>
<br>
Color value is: {{twowaycolor}}
<br>
<button (click)="click()"> CLickme</button>
</div>
<div class="card-panel blue lighten-3">
<h2>One way Binding</h2>
Enter Color : <input [ngModel]='onewaycolor'>
<br>
Color value is: {{onewaycolor}}
<br>
<button (click)="clickme()"> CLickme</button>
</div>
Ng Model need to declare form module in app.module.ts else it will give you error .Kindly add FormsModule in app.module.ts so kindly add like below code base
app.module.ts
import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core';
import { AppRoutingModule } from './app-routing.module';
import { AppComponent } from './app.component';
import { TwowaybindingwithNGModelComponent } from './twowaybindingwith-ngmodel/twowaybindingwith-ngmodel.component';
import { FormsModule } from '@angular/forms';
import { NgclassandngstyleComponent } from './ngclassandngstyle/ngclassandngstyle.component';
@NgModule({
declarations: [
AppComponent,
TwowaybindingwithNGModelComponent,
NgclassandngstyleComponent
],
imports: [
BrowserModule,
AppRoutingModule,
FormsModule
],
providers: [],
bootstrap: [AppComponent]
})
export class AppModule { }
5. Add Materialize css in index.html
In Index.html we will add materialize css and js file to use elegant UI
After editing color in both input boxes. Change only reflected in first color and not in second one that says first one is having two way data binding and second one is having one way data binding.
Now we will click on button to check value in component.
Below screenshot displays One and Two way data binding between variables using ng model
This example demonstrated that ng-Model can be used for one and two way binding.
We have combined all above mentioned examples on single page in app.component.html and it will look like below on browser so you can play with all 3 components
letsblogcontent.wordpress.com 