Showing posts with label SSIS. Show all posts
Showing posts with label SSIS. Show all posts

Friday 30 September 2022

Load the data from SQL server to SNOWFLAKE table using SSIS

We can load the SQL server table data into the SNOWFLAKE using SSIS. To load the data we are using the ODBC destination to connect the SNOWFLAKE and loading it.

Before creating the package we need to create the ODBC DSN.

Read here: Creating ODBC Connection for the SNOWFLAKE

Taking Data flow task in SSIS.  

Now taking ODEDB Source to get the data from the SQL server table. 

Now taking ODBC Destination.

Before that we are creating the ODBC connection.

Right click on the connection manager pan  

Click on the new connection and select the ODBC Connection and click on the add button.

Read here: Creating ODBC Connection for the SNOWFLAKE

https://bageshkumarbagi-msbi.blogspot.com/2022/09/creating-odbc-connection-for-snowflake.html

Select the DSN name and provide the SNOWFLAKE User name and password and click on the Test Connection button to test the connection. 

Click Ok. ODBC Connection is created successfully.

Now taking Destination as ODBC destination and selecting connect as ODBC.  

Providing the table name  

Doing the column mapping.  

Click ok.

Our package is ready to run.

See the source data. 

 See the destination table 

                         

Now running the package.

Package executed successfully.    

Now see the records into the SNOWFLAKE table.  

1k rows loaded successfully.

Creating ODBC Connection for the SNOWFLAKE

 When we are creating the DSN for ODBC connection to connect the Snowflake, First time we need to check that snowflake odbc driver install in our system or not.  Without this driver we can’t create the DSN.

First we need to install the Driver. Below is the official website to download the driver.

https://sfc-repo.snowflakecomputing.com/odbc/win64/latest/index.html    

After downloading the driver we need to install this.

Let’s create the DSN

Open ODBC Data Source Administrator and select User DSN and click on the Add button.    

Select SnowflakeDSIIDriver  

Click finish button                

Here we need to provide all information. After filling all details click on the Test button.  

Now click ok.


DSN Created successfully.

Saturday 13 August 2022

AlwaysCheckForRowDelimiters in SSIS

We have a file as below                      

  

We need to load this file into the database. See in this file we are getting irregular number of columns. While we are loading the data into the database it is not loaded in the correct format.

See this demo

We have below table 

Now we are taking source as flat file source and oledb destination and doing the mapping. 

Now we are running this package.   

Package executed successfully. It load 3 records only but in the file there are 5 rows.

Let’s see the records in the table.

 

It is not loaded in correct format.

In this case, what needs to be done is:-

SET AlwaysCheckForRowDelimiters Property of connection manager to true (If it's not set by default) and we will see all rows behaving normally when we do the preview. 

By default, the flat file connection manager always checks for a row delimiter in unquoted data and starts a new row when a row delimiter is found. This enables connection manager to correctly parse files with rows that are missing column fields.

In some cases, disabling this feature may improve the package performance. You can disable this feature by setting the flat file connection manager property, AlwaysCheckForRowDelimiters, to False.

Currently AlwaysCheckForRowDelimiters (see the file connection manager property) is false due to that data is not loaded properly.

 

Now we are setting it true. 

Now we need to run this package.

Before running the package see the data in the table. 

Now running the package. 

Package executed successfully and 5 records are loaded into the table. 

See those column don’t have value for that the value will be null. 

AlwaysUseDefaultCodePage property in OLEDB Source in SSIS

Recently we are getting the below error.

Data Flow Task: The column “ABCD” cannot be processed because more than one code page (XXXX and YYYY) are specified for it.

By default  AlwaysUseDefaultCodePage property value is false. 

It means the conversion from Unicode (65001) to ANSI (1252) is not done automatically within the data source step.

By setting it true it will done automatically. 


Thursday 14 July 2022

DTUTIL exit codes

 The Dtutil command prompt utility is used to manage SQL Server Integration Services packages. The utility can copy, move, delete, or verify the existence of a package. These actions can be performed on any SSIS package that is stored in one of three locations:

·         Microsoft SQL Server database

·         SSIS Package Store, and

·         File system

Read more here:

 https://bageshkumarbagi-msbi.blogspot.com/2018/12/dtutil-utility-command-in-ssis.html

We can run DTUTIL from other applications and it will return some codes to show what happened after running the command; these are DTUTIL exit codes.

 

                Value

Description

0

The utility executed successfully.

1

The utility failed.

4

The utility cannot locate the requested package.

5

The utility cannot load the requested package.

6

The utility cannot resolve the command line because it contains either syntactic or semantic errors.

 

Tuesday 21 June 2022

This SP is failing in the database when we are running it manually but when we are calling this SP in the package our package is executed successfully

Recently we get a production issue. Our issue is that In the SSIS package using execute SQL task we are calling the SP. This SP is failing in the database when we are running it manually but when we are calling this SP in the package our package is executed successfully. Ideally, our package needs to be failed.

Let’s replicate this issue after that we will see the solution to fix this issue.

We are creating an SP that inserts the record in the table.

Table scrip

create table emp

(id int identity(1,1) primary key,

emp_name varchar(50),

emp_add varchar(50),

emp_mob varchar(10)

)

--Log table

create table Error_log

(

log_id int identity(1,1) primary key,

Error_msg varchar(max)

)

Creating SP to insert the record.

CREATE PROCEDURE P_insert_emp

                (

                @emp_name varchar(50),

    @emp_add varchar(50),

    @emp_mob varchar(100)

                )

AS

BEGIN

declare @rt int;

set @rt=1;

begin try

insert into emp (emp_name,emp_add,emp_mob)values (@emp_name,@emp_add,@emp_mob);

end try

begin catch

set @rt=-1;

insert into Error_log(error_msg) select ERROR_MESSAGE();

end catch;

return @rt;

END;

 

Now executing the Sp

declare @result int;

execute @result= P_insert_emp 'Bagesh','Noida','88888XXXXX';

select @result;

 Record inserted successfully.  

Now calling this SP in the Package

Taking Execute SQL task and doing the configuration.  

Now running the package.  

Executed successfully.  See the records in the table. 

Now we are producing the error in the SP.

declare @result int;

 execute @result= P_insert_emp 'Bagesh','Noida','88888XXXXX678905';

Passing the emp_mob value more than ten characters.  It is throwing an error which we have logged in the Error_log table.  

Now we are calling this SP with the same parameter in SSIS package.  

Now we are executing this package.  

Package executed successfully. Even though our SP is failing.

See the records in the table.  

We need to fail the package if the SP is failed in the DB.

To fail the package we will we the raiserror Function.

To use this function our SP must have either return statement or Output parameter then only we can fail our package.

In this demo SP we have return statement.

Writing below script in the Execute SQL task.

declare @result int,

        @error_msg varchar(max); 

execute @result= P_insert_emp 'Bagesh','Noida','88888XXXXX678905';

 if (@result=-1)

begin

set @error_msg= (select Error_msg from Error_log where log_id=(select @@IDENTITY));

 set @error_msg =' Error while inserting records in the Emp table. Error message :' + @error_msg;

 raiserror(@error_msg , 16, 1);

end;


  

Now we are running this package.

  

Package executed successfully.

See the error.

[Execute SQL Task] Error: Executing the query "declare @result int,

        @error_msg varchar(ma..." failed with the following error: "Error while inserting records in the Emp table. Error message :String or binary data would be truncated in table 'Test_DB.dbo.emp', column 'emp_mob'. Truncated value: '88888XXXXX'.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.

 This is the best way to use the raise error function when we are calling the SP or function in the Execute SQL task in the SSIS package. 

Sunday 22 May 2022

Implementation of slowly changing Dimension Type-2 Creating a new additional record to maintain the history (Using Start Date and End Date) using SCD Transformation in SSIS



In this demo we will see how to implement the SCD type 2 using SCD transformation.

Read: Implementationof slowly changing Dimension Type-2 Creating a new additional record tomaintain the history (Using Start Date and End Date) using SQL Merge statement.

https://bageshkumarbagi-msbi.blogspot.com/2022/05/implementation-of-slowly-changing_17.html

 Taking data flow task 

Taking source as OLEDB source and doing the configuration.

  

Now we are taking SCD transformation.

  

Select the business key and click on the next button.

 

Selecting historical changed attribute.

Click on the next button.

  

In case if we are selecting the flag then we are using above radio button.

Here we are using the start date and end date and also selecting the start date as Start time.

Click on ok button.

  

We are not enabling the inferred member support. Click on the next button.

  

Click on finish button.

Our package looks like below

  

Our package is ready to run.

Before running the package records in the Stage and main table as below.

  

Now running this package.

     


Package is executed successfully. See the records in the table.

  

Get the expected result.

Popular Posts