In plain simple computing language, spark is an open source cluster computing framework. It is used to solve big data problems. Spark distributes the data on the cluster nodes and then will process the distributed data on each of the nodes against the local data and then send the consolidated response back to the requested of the spark job. If someone is going to ask you about spark, the above explanation is good enough.
Big data has gained lot of traction in last decade or so as users of the internet are continuously creating huge amount of data and processing huge data was something our older frameworks were not capable of handling. Such huge amount of data requires special handling and that was provided initially by Hadoop. What spark provides over Hadoop is the speed. Spark in most cases will perform better than Hadoop. Spark does all the processing in-memory whereas Hadoop writes it on the disk. When it is in-memory processing spark performs upto 100 times better than Hadoop and upto 10 times faster when we write to disk in spark.
It does appear that it will be complex and difficult to follow and understand spark however, most of the complexity is abstracted by the spark and it is extremely easy to start coding in spark. If you know basics of Java, Scala, Python or R, then you can easily write a spark job. In terms of java, we are supposed to write everything inside a main program and we can submit the same to a spark cluster.
A spark cluster typically looks like below. Suppose we are having a spark cluster of 3 nodes. So one of the node will become the master node and rest as worker nodes. Spark has a standalone cluster manager which will basically drive your program across the cluster and act as master. A spark job is submitted to a spark cluster ( a spark job is nothing but a main program bundled in a jar), the node on which we submit a spark job is called the driver program and will have the instance of spark context. All the processing will happen on the worker nodes. Spark can work with HDFS, Hive, Cassandra, Hbase as its storage. We will come to know why do we need storage in future posts.
This is a small introduction to Apache Spark. I will be writing more about the Spark Architecture, Core components, Running your first spark application in future posts. Let me know what you would like to see first.
Also please comment or like if you liked the explanation. Thank you for spending time reading this post.
Lets begin with creation of the project using command django-admin startproject djangoauth. I will be using pycharm to edit files. You can edit in pycharm or anyother ide or your choice.
Step 2 –
On the terminal, in our project root directory, execute commands “python manage.py makemigrations” and “python manage.py migrate”
Step 3 – Lets create a template directory where we will place our html files. I have created a templates directory in the root of the project. You may create the same structure. Also we need to register this directory in the settings.py file in the templates object. Against the DIRS add the following path. This will register our templates directory with the django project
'DIRS': [os.path.join(BASE_DIR, 'templates')],
Lets add a home page for our application by adding home.html under the templates directory and add some sample content as below.
Step 4 – Add url mapping to enable authentication and the home page in the urls.py of djangoauth project (/djangoauth/djangoauth) file. The “django.contrib.auth.urls” is where we set up authentication
from django.contrib import admin
from django.urls import path, include
from django.views.generic.base import TemplateView
urlpatterns = [
path('admin/', admin.site.urls),
]
urlpatterns += [
path('', include('django.contrib.auth.urls')),
path('', TemplateView.as_view(template_name='home.html'))
]
Step 5 – Now lets run and check what is happening on the browser. Run our application by execution “python manage.py runserver”. You should see following code
Step6 – Django provides an admin page where we can add modify users. We will first create a superuser so that we can add more users to log-in. To create a superuser execute command “python maange.py createsuperuser”
To login with this user, On the browser navigate tohttp://localhost:8000/admin, this will show up the login page. Login with the user created above, you should see the below page
Now create a new user by clicking on add button. you will be presented with below page. fill in the details and click save. On the next page you can just click save.
Step 7 Next up is we need to setup the login page. Django looks for login page under the registration directory of templates. So we create a login.html file as below in templates/registrations directory.
Next we update our home.html to show login and logout. Copy the below content to home.html
{% if user.is_authenticated %}
<h1>Welcome to home</h1>
{% else %}
<h1> Login to the application</h1>
{% endif %}
<a href = "http://localhost:8000/login/">login</a>
{% if user.is_authenticated %}
<a href = "http://localhost:8000/logout/">logout</a>
{% endif %}
Add below variables to settings.py file. Here we are setting the login url and where should it redirect upon login and logout. Save the file and now navigate to http://localhost:8000/
Click on login and enter with the user you created to see below page. And now you can click on logout to check if it works. It should take you back to the login screen.
In this tutorial we will go through integration of django and react in just 10 min
There aren’t many tutorials clearly identifying changes required in django app to run with a react app. Most tutorials seem complex and steps mentioned are often are not necessary. We will look at steps to integrate django rest service with a react web application and all this will be served from the django project. We will also demonstrate hot deployment of react changes when working in django when we are developing and there are constant changes that we would want to make to the UI pages and look at result of those changes. You wont believe, it hardly takes 10 mins.
I am making a few assumptions here.
Django is installed
DjangoRestFramework is installed
Basic understanding of Django
npm and npx are installed
PyCharm IDE – Not mandatory but recommended
The above installations are fairly simple and extremely easy. We wont go into details of those installations as you would find bunch of those on google search
Lets begin by creating a django project. As you can see below, I have created a new django project “DjangoReactProject“
If you open the directory, you should find below files/directories created by django
Step 2 – Create App
Next up we are going to create a new Django app, lets call this as service as we are going to serve rest calls from this service.
Step 3 – Register the app & Create url mapping
I have opened my project in pycharm but you may as well edit the files in any editor. The current project structure should look like
Next is to register the service app to the project, this is done by adding the app config in the settings.py file of the project.
Now that we have registered our app to the project. Lets add a function in the views.py file of our app. This function is what we will respond from the service app for our project. If you see below In the views.py file, I have added a new function which responds with a dummy json string having name and city attributes.
Next, Create a urls.py file in the our service app and create a mapping between a default http request with the function we have created just now.
Step 4 – Test the url
Now lets run our project to see if we receive response for our mapped url. Run the command “python manage.py runserver” and you should see below console output.
Lets create a react app inside our project folder with npx. This should take a few mins. This will download all the packages required for react. Once this is complete, You should see directory structure like below
Steps 6 – Add Component and Update App.js
Lets create a new directory named “components” under src in our react app just created and create a new file “MyComponent.js” and add below content .
This is a very basic code in React and most of you must already understand what is going on here.
We have created a new React component and initialized state of the component with name and city. In the render function, we are passing an html with the content of the state inside a div element. We have overriden the component “componentDidMount” in which we are making an http request to the url we created in Django and setting it into the state of the component
Now lets add this component to App.js file in the react application under the src folder. React must have pre-populated some content. Remove all of that and copy below code to the App.js file
import React from 'react';
import logo from './logo.svg';
import './App.css';
import MyComponent from './components/MyComponent.js'
function App() {
return (
<div className="App">
<h1>React App - Served from Django</h1>
<MyComponent/>
</div>
);
}
export default App;
Step 7 – Run the react build
Navigate to the react app folder and create react build by running npm run build. A build directory will be created after successful execution of the build with all the files required for the react app. The starting point of the application being index.html
Step 8 – Configure Django to serve react build
We now just need to point Django to the build directory of react so that it starts serving from there. In our DjangoReactProject directory present in the root of our project, navigate to settings.py
Find Templates in the settings.py file and add the build directory of our react application as show in the screenshot in the DIRS. We are telling python where to find templates in the react application
In the same file add a new variable “STATICFILES_DIRS” if not present already and add variable which points to the static content of the react app as below
Now run our django server and it should start serving our react application and call the rest service /api to render the response
C:\DjangoReactProject>python manage.py runserver
This should deploy our django app with react @ http://localhost:8000/. Navigate to the url to see below page.
Step 9 – Hot Deployment
So the above is fine structure to deploy our react application once all of our react code is ready, but during development it is going to be very difficult to always having to build our react app for small changes in our react app. So to overcome this, we have to make a small change in the react app so that when we are developing all the requests are redirected to django app.
So when we are developing we will run the inbuild react server to server our react application, which will redirect calls to django server when required.
To configure this we just need to add proxy configure in the package.json file of the react app.
Here we have given the address of server
"proxy": "http://127.0.0.1:8000",
Once this is done, lets run the react server by running command “npm run start”. This will start the react application @ http://localhost:3000/ and you should see the below page. Any changes to react will be hot deployed for development.
At this point any changes in the Django or the react app are hot deployed and we can make changes to quickly see the results.
Ohh. We have still not reached Step 10, that was really easy. Let us know if you have any queries or like the page if you have like this tutorial, post any feedback if you have. Thank you!
We will quickly go through a simple tutorial to create django rest project. It is extremely easy to configure a rest project work with Djange rest framework. Lets go through the steps. I am using PyCharm but it is not required but it will make development much easier with PyCharm. So lets begin.
Step1
Create a new project in PyCharm. Those not using PyCharm can skip this step. Create a directory otherwise.
Step 2
Install django using command “pip install django==3.1” on the terminal. You should see a successful installation of the django library.
Step 3
Now create a django project using command on terminal “django-admin startproject restproject .” “.” is used to create project in the current directory
Step 4
Now that we have the project ready, we will create a django app. The project consist only the configuration information. The app contains the application code. Execute the command “python manage.py startapp products“
Step 5
Lets now create a model to store and retrieve. Lets create a model with name “Product” with fields “name”, “price”, “desc” as per the below screenshot.
Now, execute the command “python manage.py makemigrations” and “python manage.py migrate” . You will see following output. This will create a table in sqlite3 db being used in django.
Step 6
Install the django rest framework executing the command “pip install djangorestframework”
Step 7
Create a new file with name “serializers.py” under the products app as shown in the snapshot below. The “serializers” are used to transform between various formats between the rest client and the server so that communication can happen seamlessly. Here we create a class ProductSerializer class which extends from the ModelSerializer class. The ModelSerializer class maps the fields between the model and serializer and performs validation based on the model defined under the meta class. We don’t need to think much here, just we need to understand that this is required for transofmation between an object to json/xml or vice versa.
Step 8
Now update the views.py file under the products app as below. Here we create a new function “product_list” which takes request as input parameter. We retrieve all the elements in the products table and then use the ProductSerializer created in the previous step to create a json response. Please see 1,2,3 points in the snapshot. We annotate the function to process “GET” request with api_view
Step 9
Now create a urls.py file in the products app if not present already and map “products” url to “product_list” function we created in the views.py file
Now update the urls.py file in the restproject as below to map urls starting with api to the products app
Step 10
We are done with the rest framework basic setup. To demonstrate its working, lets add some values to the products object/table using the admin functionality provided by django.
First register your “product” object to the admin. For this create a file admin.py file under products app and update with below content.
Lets create a superuser to manage admin functionality provided by django. Execute below commands. Provide details of username and password to create a super user
Now lets run our server with command “python manage.py runserver“. This will run the server at http://localhost/8000
Now navigate to url http://127.0.01:8000/admin. you will see below below screenshot and provide username and password created in the step above. for me it is admin/admin. Provide the details and press login.
Now you should see below screenshot
Now lets add a few products to the list, click on add and fill the details
This should add to the products table. This is really easy and fast way to populate some data. The admin functionality can do a lot of this, but will see more of it in future tutorials.
Step 11
Now we are all set to retrieve the list of the products using django rest framework. Hit the url http://127.0.0.1:8000/api/products to see the products we have created in previous step. You should response in json for the products you have created.
We have just now successfully created a basic setup of django restframework. We will look at some advanced features in the next tutorial. The git link for this project is “https://github.com/letsblogcontent/SampleRestProject“
Learn Django project creation. Its is extremely easy and fast to develop with django
We will quickly go through a simple tutorial to create django rest project. It is extremely easy to configure a rest project work with Djange rest framework. Lets go through the steps. I am using PyCharm but it is not required but it will make development much easier with PyCharm. So lets begin.
Step1
Create a new project in PyCharm. Those not using PyCharm can skip this step. Create a directory otherwise.
Step 2
Install django using command “pip install django==3.1” on the terminal. You should see a successful installation of the django library.
Step 3
Now create a django project using command on terminal “django-admin startproject restproject .” “.” is used to create project in the current directory
Step 4
Now that we have the project ready, we will create a django app. The project consist only the configuration information. The app contains the application code. Execute the command “python manage.py startapp products“
Add the newly created app to the settings.py file of the project, in our case it is our restproject.
Step 5
Lets now create a model to store and retrieve. Lets create a model with name “Product” with fields “name”, “price”, “desc” as per the below screenshot.
Now, execute the command “python manage.py makemigrations” and “python manage.py migrate” . You will see following output. This will create a table in sqlite3 db being used in django.
Step 6
Install the django rest framework executing the command “pip install djangorestframework”
Import this into the installed apps of the project’s settings.py
Step 7
Create a new file with name “serializers.py” under the products app as shown in the snapshot below. The “serializers” are used to transform between various formats between the rest client and the server so that communication can happen seamlessly. Here we create a class ProductSerializer class which extends from the ModelSerializer class. The ModelSerializer class maps the fields between the model and serializer and performs validation based on the model defined under the meta class. We don’t need to think much here, just we need to understand that this is required for transofmation between an object to json/xml or vice versa.
Step 8
Now update the views.py file under the products app as below. Here we create a new function “product_list” which takes request as input parameter. We retrieve all the elements in the products table and then use the ProductSerializer created in the previous step to create a json response. Please see 1,2,3 points in the snapshot. We annotate the function to process “GET” request with api_view
Step 9
Now create a urls.py file in the products app if not present already and map “products” url to “product_list” function we created in the views.py file
Now update the urls.py file in the restproject as below to map urls starting with api to the products app
Step 10
We are done with the rest framework basic setup. To demonstrate its working, lets add some values to the products object/table using the admin functionality provided by django.
First register your “product” object to the admin. For this create a file admin.py file under products app and update with below content.
Lets create a superuser to manage admin functionality provided by django. Execute below commands. Provide details of username and password to create a super user
Now lets run our server with command “python manage.py runserver“. This will run the server at http://localhost/8000
Now navigate to url http://127.0.01:8000/admin. you will see below below screenshot and provide username and password created in the step above. for me it is admin/admin. Provide the details and press login.
Now you should see below screenshot
Now lets add a few products to the list, click on add and fill the details
This should add to the products table. This is really easy and fast way to populate some data. The admin functionality can do a lot of this, but will see more of it in future tutorials.
Step 11
Now we are all set to retrieve the list of the products using django rest framework. Hit the url http://127.0.0.1:8000/api/products to see the products we have created in previous step. You should response in json for the products you have created.
We have just now successfully created a basic setup of django restframework. We will look at some advanced features in the next tutorial. The git link for this project is “https://github.com/letsblogcontent/SampleRestProject“
It hardly takes a minute to create a usable django app. It is simple and quick
Now that we have created an initalized a Django project. It is time to initialize a django app. An app in django is a web application that performs some operation. The project creation is just an initialized of the project with configurations for a website.
1. Navigate to you project directory and install “pip install django” 2. Create a Django App “webapp” by executing below command on the terminal “python manage.py startapp webapp”
Add the app to the projects settings.py under the “INSTALLED_APPS” array.
3. Navigate to views.py under the newly created app “webapp”
4. In the views.py is a file where we create functions that we want to respond to an http request. Lets create a function index as displayed below. The index function has parameter request, which will be supplied by django. We will also import the HttpResponse class from django.http so that we can use it to respond to the http requesting client. We have returned a message “Welcome to myapp” in the response.
5. Now create a new file “urls.py” file in the same directory and add the import “from . import views”. Here we are importing the functions we have created in views.py. Now add a new array urlPatterns as shown in below image with entries as mentioned. Here we telling django the mapping between request url and the corresponding function.
6. We have created the mapping in the webapp app but now we need to tell the project about the mappings. Navigate to the project “urls.py” file as shown below. Add a new path in the “urlpatterns” and import “include” as shown. Here on the line 21 we have told django, whenever request is for a url with pattern “/api” it should look for the mapping in the urlpatterns mentioned in the webapp app.
7. Now run the server with “python3 manage.py runserver”. Run the url http://127.0.01:8000/api . Similarly we can add multiple paths to the webapp application. We will more about it in the next tutorial
Your application is running and rented “Hello World” in browser. Same way you can initialize variables and pass data from component to UI.
Create Array and String in
Here we have declared title as string and cars as array. Internally typescript will understand data types
import { Component } from '@angular/core';
@Component({
selector: 'app-root',
templateUrl: './app.component.html',
styleUrls: ['./app.component.css']
})
export class AppComponent {
title = 'Display data of Cars';
cars = [ 'city' , 'nexon', 'scorpio' ];
}
Edit app.component.html
Map variables in html using below code
with {{ }} we have mapped title in below html ..
For showing array we have to iterate through it so we have used “let”,which will help to get value one by one from array and store in “c” variable,
We can now simply prin
<div>
<h1> {{ title }} </h1>
<ul *ngFor = "let c of cars ">
<h1> Car Name:{{c}} </h1>
<li *ngIf = " c == 'nexon' ">
<h2 style="color: brown;">
This is new car: {{c}}
</h2>
</li>
</ul>
</div>
Once you save your changes just run below command
ng serve --open
Output:
Below output will get printed on HTML page..For loop will rotate array and if will add conditions to it.
Now will see how we can use type script class and print data
Create class using below command
ng generate class car
this will create car class and its test class as well like below screen shot in your project
Add below snippet in car.ts. here we have declared two variables in it
name =>datatype is string
average=>datatype is number
we have also added constructor which will help to initialize these two variable while creating instance of this class
export class Car {
name : string;
average : number;
constructor( name , average)
{
this.name = name;
this.average = average;
}
}
Edit app.component.ts
In this class will create instance of car and create array of new cars.
import { Component } from '@angular/core';
import { Car } from './car';
@Component({
selector: 'app-root',
templateUrl: './app.component.html',
styleUrls: ['./app.component.css']
})
export class AppComponent {
title = 'Display Cars';
cars = ['city', 'nexon' , 'vento'];
newcars =[
new Car('city', 10),
new Car('polo', 5),
new Car('vento', 6),
];
}
Edit app.component.html
Map variables in html using below code
with {{ }} we have mapped “title” in below html ..
For showing array we have to iterate through it so we have used “let”,which will help to get value one by one from array and store in “n” and “c”variable,
In below code first we are printing all cars and then filtering them using below condition
*ngIf =”n.average < 7″
<div>
<h1>{{title}}</h1>
<ul *ngFor =" let c of cars">
<li> {{c}}</li>
</ul>
<h1>Display data with Class</h1>
<ul *ngFor =" let n of newcars">
<li style="color:red" *ngIf ="n.average < 7 "> {{n.name}}</li>
</ul>
</div>
Once you save your changes just run below command
ng serve --open
Output:
Below output will get printed on HTML page..For loop will rotate array and if will add conditions to it. In second loop all cars having less than 7 average will get printed
Your application is running and rented “Hello World” in browser. Same way you can initialize variables and pass data from component to UI.
Create Array and String in
Here we have declared title as string and cars as array. Internally typescript will understand data types
import { Component } from '@angular/core';
@Component({
selector: 'app-root',
templateUrl: './app.component.html',
styleUrls: ['./app.component.css']
})
export class AppComponent {
title = 'Display data of Cars';
cars = [ 'city' , 'nexon', 'scorpio' ];
}
Edit app.component.html
Map variables in html using below code
with {{ }} we have mapped title in below html ..
For showing array we have to iterate through it so we have used “let”,which will help to get value one by one from array and store in “c” variable,
We can now simply prin
<div>
<h1> {{ title }} </h1>
<ul *ngFor = "let c of cars ">
<h1> Car Name:{{c}} </h1>
<li *ngIf = " c == 'nexon' ">
<h2 style="color: brown;">
This is new car: {{c}}
</h2>
</li>
</ul>
</div>
Once you save your changes just run below command
ng serve --open
Output:
Below output will get printed on HTML page..For loop will rotate array and if will add conditions to it.
With Angular Js you can create libraries which can be easily imported in other angularjs projects and used.
This gives you advantage to reuse code or put boilerplate code in common library that can be used repetitively in different project
Below Command is getting used to create library in angular js
ng generate library my-lib
Below is example of structure of library project
2. Multiple Projects
Angularjs given you option to create workspace which can hold multiple small projects including libraries which give you tremendous advantage to manage your source code efficiently.
ng new my-workspace --createApplication="false"
cd my-workspace
ng generate application my-first-app
ng generate application my-second-app
ng generate library my-lib
As shown in above command you can create workspace first
ng new my-workspace –createApplication=”false”
then go to workspace
cd my-workspace
Now create your first application in workspace
ng generate application my-first-app
Now create your Second application in workspace
You can also create library in workspace with below command
ng generate library my-lib.
Below is project strucutre of workspace
3. Angular JS Application
This is most common way to use angularjs most of beginners straight away creates project with angular with this type of project.
This will create workspace with your angularjs project and all necessary things will be configured so you can directly start coding with this appraoch.
Lets see how to create it
Create a workspace and initial application
ng new my-app
Run the application
cd my-app
ng serve --open
Below is project structure for this kind of project
letsblogcontent.wordpress.com 