SQOOP

-

 

What is Apache Sqoop?

Many of us still wonder what Apache Sqoop is, its architecture, features, uses, and how it is related to big data. In this Sqoop write up, we will talk about everything along with its requirements. Let’s get started! Apache Sqoop is a big data tool for transferring data between Hadoop and relational database servers. Sqoop is used to transfer data from RDBMS (relational database management system) like MySQL and Oracle to HDFS (Hadoop Distributed File System). Big Data Sqoop can also be used to transform data in Hadoop MapReduce and then export it into RDBMS. Sqoop is a data collection and ingestion tool used to import and export data between RDBMS and HDFS.

SQOOP = SQL + HADOOP

Why do we need Big Data Sqoop?

Sqoop Big Data Tool is primarily used for bulk data transfer to and from relational databases or mainframes. Sqoop in Big Data can import from entire tables or allow the user to specify predicates to restrict data selection. You can write directly to HDFS as Sequence files or Avro. Big Data Sqoop can take data directly into Hive or Hbase with proper command line arguments. Finally, you can also export data back out to relational databases using Sqoop in Big Data. A typical workflow with Big Data Hadoop Sqoop is where data is brought in Hive, so intermediate processing and transformation tasks can be done on Apache Hadoop. Once processing is done, data can be exported back to a database. This is one of many ways to perform "data warehouse offloading," where Hadoop is used for ETL purposes.

Features of Big Data Sqoop

  • Parallel Import/Export: Sqoop Big Data Tool uses the YARN framework to import and export data. This provides fault tolerance on top of parallelism
  • Import Results of SQL Query: Big Data Hadoop Sqoop allows us to import the result returned from an SQL query into HDFS
  • Connectors for All Major RDBMS Databases: Sqoop in Big Data provides connectors for multiple Relational Database Management System (RDBMS) databases such as MySQL and MS SQL Server
  • Offers Full and Incremental Load: Big Data Sqoop can load the whole table or parts of the table by a single command. Hence, it supports the full and incremental load.

Sqoop Import

Big Data Hadoop Sqoop imports every single table from RDBMS to HDFS. Each row within a table is treated as a single record in HDFS. All records are then stored as text data in the format of text files or as binary data in Avro and Sequence files.

Sqoop Export

Sqoop Big Data Tool exports files from HDFS back to an RDBMS. All the files given as input to Sqoop contain records, which are called rows in the table. Later, those are read and parsed into a set of records and delimited with a user-specified delimiter.

Where Can I Use Sqoop?

Relational database systems are used to interact with business applications. One of the biggest and largest sources of big data in the modern world is the relational database system. In Hadoop, different processing frameworks like MapReduce, Hive, HBase, Cassandra, Pig, etc. and storage frameworks like HDFS can benefit distributed computing and distributed storage. Data needs to be transferred among database systems and Hadoop Distributed File System (HDFS) to store and analyze Big Data from relational databases. Here, Sqoop in Big Data comes into the picture. Big Data Sqoop acts like a middle man/ a layer between Hadoop and relational database systems. One can import data and export data between relational database systems and Hadoop and its eco-systems directly using Sqoop.
Sqoop in Big Data provides a command-line interface to the end-users. Sqoop can also be accessed through Java APIs. Sqoop command submitted by the user is parsed by Sqoop and launches Hadoop Map only to import or export data because the Reduce phase is only required when aggregations are required. Sqoop just imports and exports the information; it does not do aggregations. Big Data Sqoop parses the arguments given in the command line and prepares the Map job. A map job launching multiple mappers depends on the number defined by the user in the command line. During Sqoop import, each mapper task is assigned with part of the data to be imported based on the command line. Data is distributed equally by Sqoop among the mappers to get high performance. After that, each mapper creates a connection with the database using JDBC and then imports the part of the data given by Sqoop, and then it is written into the HDFS or Hive or Apache HBase according to the option provided in the command line.

Comments

Post a Comment

Popular posts from this blog

SPARK- DATAFRAME DSL

-
Apache Spark is a lightning-fast cluster computing framework designed for fast computation. With the advent of real-time processing framework in the Big Data Ecosystem, companies are using Apache Spark rigorously in their solutions. Spark SQL is a new module in Spark which integrates relational processing with Spark’s functional programming API. It supports querying data either via SQL or via the Hive Query Language.  Through this blog, I will introduce you to this new exciting domain of Spark SQL. What is Spark SQL? Spark SQL integrates relational processing with Spark’s functional programming. It provides support for various data sources and makes it  possible to weave SQL queries with code transformations thus resulting in a very powerful tool.  Why is Spark SQL used? Spark SQL originated as Apache Hive to run on top of Spark and is now integrated with the Spark stack. Apache Hive had certain limitations as mentioned below. Spark SQL was built to overcom...
Read more

MongoDB - Data Modelling

-
Image
Data in MongoDB has a flexible schema.documents in the same collection. They do not need to have the same set of fields or structure Common fields in a collection’s documents may hold different types of data. Data Model Design MongoDB provides two types of data models: — Embedded data model and Normalized data model. Based on the requirement, you can use either of the models while preparing your document. Embedded Data Model In this model, you can have (embed) all the related data in a single document, it is also known as de-normalized data model. For example, assume we are getting the details of employees in three different documents namely, Personal_details, Contact and, Address, you can embed all the three documents in a single one as shown below − { _id : , Emp_ID : "10025AE336" Personal_details :{ First_Name : "Kishan" , Last_Name : "choudhary" , Date_Of_Birth : "1995-09-26" }, Contact : { e - mail : ...
Read more