Course Content

1. Overview of data science and machine learning at scale

2. Overview of the Hadoop ecosystem

3. Working with HDFS data and Hive tables using Hue

4. Introduction to Cloudera Data Science Workbench

5. Overview of Apache Spark 2

6. Reading and writing data

7. Inspecting data quality

8. Cleansing and transforming data

9. Summarizing and grouping data

10. Combining, splitting, and reshaping data

11. Exploring data

12. Configuring, monitoring, and troubleshooting Spark applications

13. Overview of machine learning in Spark MLlib

14. Extracting, transforming, and selecting features

15. Building and evaluating regression models

16. Building and evaluating classification models

17. Building and evaluating clustering models

18. Cross-validating models and tuning hyperparameters

19. Building machine learning pipelines

20. Deploying machine learning models

Trainer Profile

Pankaj Kumar Pathak

• Having 15+ years of Experience in Big Data Hadoop
• Successfullyimplemented and migrated data on demand from existing Traditional(RDBMS,Oracle/Sql server) to Nosql (Cassandra, Mongo db etc) on Hadoop Cluster and provided ample training in India’s and oversees. The topmost Big Corporate houses where I have delivered such things from last 4 and half years are: -
• Times internet: - Implemented Apcahe Spark with Cassandra on @ Hadoop RAC server for collectingMultiple log files.             
• Amar Ujala: -For Hadoop cluster planning and sizing with data migration from Sql server to Cassandra.       
• TCS: - 3 corporate batches for Hadoop admin and Data warehousing Cassandra Mongodb (Cloudera, Hortonworks).
• HCL info System: - Hadoop Cluster implementing and migration from DB2.
• HCL Technologies: - Hadoop, Spark-Scala, FlumeCassandra Nosql.
• IBM: - 2 Corporate batches for Hadoop clustering, Cloudera Manager and others.                                  
• Dish TV: - Implemented Ware housing on Hadoop cluster. 
• UHG: - Implemented Hadoop cluster20 node cluster for Warehousing using Hive/Impala,Mapreduces.
• Genpact:- Hadoop, Spark-Scala, Flume- and R.
• Nucleus software: -For Hadoop cluster planning and sizing for data warehouses through Cassandra.
• Tech Mahindra:- Implemented Spark with Cassandra on @ RAC server for collectingMultiple log files.         Migrated db2 data.
• BARC Mumbai:- Hadoop clustering with Spark and Cassandra.
• Providing consultancy to UK base 2 clients for Data Science implementation

Interview Questions & Answer

1) What is Python? What are the benefits of using Python?

Python is a programming language. It includes objects, modules, threads, exceptions and automatic memory management. The benefits of pythons are that it is simple and easy, portable, extensible, build-in data structure. it is an open source automation tool.

2) What are Python decorators?

Python decorators is a specific change that we make in Python syntax in order to alter functions easily.

3) What is unittest in Python?

A unit testing framework in Python is known as unittest. It supports sharing of setups, automation testing, shutdown code for tests, aggregation of tests into collections etc.

4) How can you copy an object in Python?

In order to copy an object in Python, you can try copy.copy () or copy.deepcopy() for the general case. You cannot copy all objects but most of them.

5) Explain how to delete a file in Python?

We can delete a file in Python ,By using a command os.remove (filename) or os.unlink(filename)

6) Explain how can you access a module written in Python from C?

You can access a module written in Python from C by following method,

Module = =PyImport_ImportModule("");

7) Mention five benefits of using Python?

• Easy-to-learn for beginners.

• It Provide easy readability due to use of square brackets.

• Python comprises of a huge standard library for most Internet platforms like Email, HTML, etc.

• It  does not require explicit memory management as the interpreter itself allocates the memory to new variables and free them automatically

• Having the built-in data types saves programming time and effort from declaring variables.

8) What is the syntax for “for” loop if we have to run for loop for 3 times?

• Answer:

• for 1 in 5:

• Print(“)

9) What Is The Statement That Can Be Used In Python If The Program Requires No Action But Requires It Syntactically?

The pass statement is a null operation. Nothing happens when it executes. You should use “pass” keyword in lowercase. If you write “Pass,” you’ll face an error like “NameError: name Pass is not defined.” Python statements are case sensitive.

letter = "hai sethuraman"

for i in letter:

    if i == "a":

        pass

        print("pass statement is execute ..............")

    else:

        print(i)

10) What is The Process To Get The Home Directory Using ‘~’ In Python?

In order to get Home Directory , You need to import the os module, and then just a single line would do the rest.

import os

print (os.path.expanduser('~'))

Output:

/home/runner

11) Write A Reg Expression That Confirms An Email Id Using The Python Reg Expression Module “Re”?

Python has a regular expression module “re.”

Check out the “re” expression that can check the email id for .com and .co.in subdomain.

import re

print(re.search(r"[0-9a-zA-Z.]+@[a-zA-Z]+\.(com|co\.in)$","micheal.pages@mp.com"))

12) What Is A Function In Python Programming?

A function in Python Programming  is an object which represents a block of code and is a reusable entity. It brings modularity to a program and a higher degree of code reusability.

Python has given us many built-in functions such as print() and provides the ability to create user-defined functions.

13) What Does The *Args Do In Python?

We use *args as a parameter in the function header. It gives us the ability to pass N (variable) number of arguments.

Please note that this type of argument syntax doesn’t allow passing a named argument to the function.

Example of using the *args:

# Python code to demonstrate 

# *args for dynamic arguments 

def fn(*argList):  

    for argx in argList:  

        print (argx) 

fn('I', 'am', 'Learning', 'Python')

The output:

I

am

Learning

Python

14) What Does The **Kwargs Do In Python?

We can also use the **kwargs syntax in a Python function declaration. It let us pass N (variable) number of arguments which can be named or keyworded.

Example of using the **kwargs:

# Python code to demonstrate 

# **kwargs for dynamic + named arguments 

def fn(**kwargs):  

    for emp, age in kwargs.items(): 

        print ("%s's age is %s." %(emp, age)) 

fn(John=25, Kalley=22, Tom=32)

The output:

John's age is 25.

Kalley's age is 22.

Tom's age is 32

Blog