Wednesday 30 June 2021

Reading JSON file and loading into in the table using SSIS Using Data Flow Task

 If we want to perform any transformation in the JSON file data in this case we will use the Data Flow task to load the data into the table.

Below is the JSON file.  

This file is having 4 Elements.

Below is the table.

CREATE TABLE Posts_details(

                userID int NULL,

                Id int NULL,

                Post_title varchar(100) NULL,

                Post_body varchar(500) NULL

)

In this post we will see how we load the JSON file using Data Flow  Task.

In this Post we will see how we will Read the JSON file data and loading into the database.

Taking Data flow task.  

Taking Source as OLEDB Source.  

Providing OLEDB connection manager and selecting Data access mode as Sql command and writing the below code.

declare @json varchar(max);

set @json = (SELECT *

FROM OPENROWSET (BULK 'H:\SSIS1\Source\Posts.JSON', SINGLE_CLOB) as import)

 

 SELECT *

     FROM OPENJSON(@json)

          WITH (

                 userId NVARCHAR(30) ,

                 id NVARCHAR(50) ,

                 title NVARCHAR(50),

                 body NVARCHAR(50)

               );


    

Now taking data conversion to convert the data as per the destination data type.  

Now taking OLEDB destination and doing the column mapping.  

Now package is ready to run

Before running this package see the records in the table.

  

Running this package.  

Package is executed successfully.

See the records in the table.  

Data Loaded successfully.

Reading JSON file and loading into in the table using SSIS Using Script Task

 If we want to validate any data from the JSON file like Email id validation. In this case we will read the JSON file parse the data and do the validation then load it into the database.

Below is the JSON file.   

This file is having 4 Elements.

Below is the table.

CREATE TABLE Posts_details(

                userID int NULL,

                Id int NULL,

                Post_title varchar(100) NULL,

                Post_body varchar(500) NULL

)

In this post we will see how we load the JSON file using Script Task.

Taking the script task. 

We are writing the below code

public class lPosts

        {

 

            public List<posts> data { get; set; }

        }

 

        public class posts

        {

            public string userid { get; set; }

            public string id { get; set; }

            public string Post_title { get; set; }

            public string Post_body { get; set; }

        }

 

 

        public void Main()

        {    

 

            try

            {

                string connetionString = null;

                string sql = null;

                var serializer = new JavaScriptSerializer();

             

                connetionString = "Data Source=BAGESH\\BAGESHDB;Initial Catalog=TestDB; Trusted_Connection=True;";

 

                string jsonFilePath = @"H:\SSIS1\Source\Posts.json";

 

                string json = File.ReadAllText(jsonFilePath);

 

                var dt = serializer.Deserialize<Dictionary<string,string>[]>(json);

 

                    using (SqlConnection cnn = new SqlConnection(connetionString))

                    {

 

                        cnn.Open();

 

                        sql = "insert into Posts_details (userID, Id,Post_title,Post_body) values(@userID, @Id,@Post_title,@Post_body)";

 

                        foreach (var item in dt)

                        {

 

                         

                            using (SqlCommand cmd = new SqlCommand(sql, cnn))

                            {

                              

                                cmd.Parameters.Add("@userID", SqlDbType.NVarChar).Value = item["userId"].ToString();

                                cmd.Parameters.Add("@Id", SqlDbType.NVarChar).Value = item["id"].ToString();

                                cmd.Parameters.Add("@Post_title", SqlDbType.NVarChar).Value = item["title"].ToString();

                                cmd.Parameters.Add("@Post_body", SqlDbType.NVarChar).Value = item["body"].ToString();

                             

                                int rowsAdded = cmd.ExecuteNonQuery();

 

                            }

                        }

                        cnn.Close();

 

                    }

 

                    Dts.TaskResult = (int)ScriptResults.Success;

             

            }

            catch (Exception ex)

            {

                Dts.Events.FireError(0, "Fire Error", "An error occurred: " + ex.Message.ToString(), "", 0);

                Dts.TaskResult = (int)ScriptResults.Failure;

            }

        }

Now package is ready to run.

See the records in the table.  

Now running this package.   

See the records in the table.

  

Record inserted successfully.

Calling API from SSIS and storing the JSON result into the table (Using Execute Sql Task)

 We can call the API in SSIS Package in multiple ways.

·         Using Execute Sql Task

·         Using Script task

We are taking execute sql task.   

Creating the oledb connection and writing the below script.

DECLARE @status int

DECLARE @responseText as table(responseText nvarchar(max))

DECLARE @res as Int;

DECLARE @url as nvarchar(max) = 'https://jsonplaceholder.typicode.com/posts'

EXEC sp_OACreate 'MSXML2.ServerXMLHTTP', @res OUT

EXEC sp_OAMethod @res, 'open', NULL, 'GET',@url,'false'

EXEC sp_OAMethod @res, 'send'

EXEC sp_OAGetProperty @res, 'status', @status OUT

INSERT INTO @ResponseText (ResponseText) EXEC sp_OAGetProperty @res, 'responseText'

 IF exists(SELECT  responseText FROM @responseText)

BEGIN

     DECLARE @json NVARCHAR(MAX) = (SELECT  responseText FROM @responseText)

                 

                 insert into Posts_details

                 (

                                userID ,

                                Id ,

                                Post_title ,

                                Post_body 

                 )

     SELECT userId,

                                                id,

                                                title,

                                                body

     FROM OPENJSON(@json)

          WITH (

                 userId NVARCHAR(30) '$.userId',

                 id NVARCHAR(50) '$.id',

                 title NVARCHAR(50) '$.title',

                 body NVARCHAR(50) '$.body'

               );

END

ELSE

BEGIN

     DECLARE @ErroMsg NVARCHAR(30) = 'No data found.';

     Print @ErroMsg;

END

EXEC sp_OADestroy @res

  

Click Ok . Now the package is ready to run.

Records in the table.  

Running the package now.

Package executed successfully.

See the records in the table.  

Data loaded successfully.

Calling API from SSIS and storing the JSON result into the table (Using Script Task)

In this post, we will see how we will call the API and extracting the data from API and load it into the table using Script task.

In the previous post, we seen that Calling API from SQL Server and storing the JSON result into the table

Read here:  Calling API from SQL Server and storing theJSON result into the table

https://bageshkumarbagi-msbi.blogspot.com/2021/06/calling-api-from-sql-server-and-storing.html

We are taking Script task. 

Click ok the Edit Script. Here we are writing the code in C#. We are including the below name spaces.

using System;

using System.Data;

using Microsoft.SqlServer.Dts.Runtime;

using System.Windows.Forms;

using System.Net;

using System.Net.Http;

using System.Net.Http.Headers;

using System.IO;

using System.Collections.Generic;

using System.Text;

using System.Web.Script.Serialization;

using System.Data.SqlClient;

Highlighted namespace we need  to add from the reference.

Below is the code to calling the API and extracting the JSON data from it and parsing it and finally we are loading it into the database table.

For testing we are using the Below API which is available for online for the free testing.

https://jsonplaceholder.typicode.com/posts

Below is the JSON data. 

Now we are creating property class

public class lPosts

        {

 

            public List<posts> data { get; set; }

        }

 

        public class posts

        {

            public string userid { get; set; }

            public string id { get; set; }

            public string Post_title { get; set; }

            public string Post_body { get; set; }

        }

 

In main we are writing the below code.

  public void Main()

        {    

 

            try

            {

                string connetionString = null;

                string sql = null;

                string serviceUrl = "https://jsonplaceholder.typicode.com/posts";

                HttpClient client = new HttpClient();

                client.BaseAddress = new Uri(serviceUrl);

                var serializer = new JavaScriptSerializer();

                // Add an Accept header for JSON format.   

                client.DefaultRequestHeaders.Accept.Add(

                  new MediaTypeWithQualityHeaderValue("application/json"));

                connetionString = "Data Source=BAGESH\\BAGESHDB;Initial Catalog=TestDB; Trusted_Connection=True;";

 

                string APIUrl = string.Format(serviceUrl);

                var response = client.GetAsync(APIUrl).Result;

                if (response.IsSuccessStatusCode)

                {

                    var result = response.Content.ReadAsStringAsync().Result;

 

                    var dt = serializer.Deserialize<Dictionary<string,

                      string>[]>(result);

                    using (SqlConnection cnn = new SqlConnection(connetionString))

                    {

                        cnn.Open();

                        sql = "insert into Posts_details (userID, Id,Post_title,Post_body) values(@userID, @Id,@Post_title,@Post_body)";

                        foreach (var item in dt)

                       {                         

                            using (SqlCommand cmd = new SqlCommand(sql, cnn))

                            {                              

    cmd.Parameters.Add("@userID", SqlDbType.NVarChar).Value = item["userId"].ToString();

   cmd.Parameters.Add("@Id", SqlDbType.NVarChar).Value = item["id"].ToString();

    cmd.Parameters.Add("@Post_title", SqlDbType.NVarChar).Value = item["title"].ToString();

  cmd.Parameters.Add("@Post_body", SqlDbType.NVarChar).Value = item["body"].ToString();

                                   int rowsAdded = cmd.ExecuteNonQuery();

                            } }

                        cnn.Close();

                    }

                    Dts.TaskResult = (int)ScriptResults.Success;

                } }

            catch (Exception ex)

            {

                Dts.Events.FireError(0, "Fire Error", "An error occurred: " + ex.Message.ToString(), "", 0);

                Dts.TaskResult = (int)ScriptResults.Failure;

            }

        }

 

Compile the code and build it save and close it.

Here we are not using Newtonsoft.Json

Now our package is ready to run.

Before running the package see the records in the table.

  

Running the package now.  

Package executed successfully.

See the records in the table.

  

Data loaded successfully.

Popular Posts